2014-04-01 14:32:44,582.582 DEBUG:teuthology.report:No results_server set in /home/loic/.teuthology.yaml; not attempting to push results 2014-04-01 14:32:44,590.590 DEBUG:teuthology.run:Config: machine_type: plana os_type: ubuntu overrides: admin_socket: branch: master ceph: conf: global: ms inject socket failures: 5000 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 20 fs: ext4 log-whitelist: - slow request sha1: e672c52b4f8b945a516f2eec006e33665a08f045 ceph-deploy: branch: dev: master conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: debug mon: 1 debug ms: 20 debug paxos: 20 osd default pool size: 2 install: ceph: sha1: e672c52b4f8b945a516f2eec006e33665a08f045 s3tests: branch: master workunit: sha1: e672c52b4f8b945a516f2eec006e33665a08f045 owner: scheduled_teuthology@teuthology roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - client.0 - - mon.b - mds.a - osd.3 - osd.4 - osd.5 - client.1 targets: ubuntu@plana78.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZxXbzgCaQZsjUFXMXBP6gCnCguoRfZ8dxbV71vaRpMX8p7avh1cFB2qeXDQMBDYukxJZdVBSEomkW/bPygRa+/vlvj7XhcOroy7SkpVY9ZAFkUKScANybxglH2qo+7TFGNB0Yq9GPBQ6z/3sbB50Uzod6gKPrAhGS/uggrSj2KbXK5UUrUQNsSvMtAJqYPDDWRWtwMH/Ezr76FEM//cveTXoViNh557CgJVB9HqmKHTtk9c9iPIi4DBdH5cGWe2f1Lh3HHVO4KOEepI9/2ItMIhoODEobvNJi62Vh76ZR2lcQJw4njyv3GF4c4khK47xQx6vPttvPxMLprClEM/J1 ubuntu@plana79.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDGJX9/cZFlm+ll32X716yKrmR/RE94iH9TusIFLY5vLoE8CBCupjAKdkn4mqOB5eNvwqapMG63Vww5Cl9zo0wKPEHi3jsZCwbAxByc9sBFivZeBnmTUrYNesQvAs1Izr49/4h71oCt98hfX3hl81iEhIEGjjj7XoD3blyYHlyR9LBNoYMllLfi9Pw4KD1snEikGKlR8zMyAgndlZ5ODU4usiCMXLypAa4wFKfR6w17nZI2/Q2xkUy7l59oap50bOR9mSNMOPqpN1KgsK717JCKMU4jHHw+zN0oWvQPmDZK2ckzbyftBfpOqKyidGHpLjpXfuQ1Xk4R9fgUkpE6gcD5 tasks: - chef: null - clock.check: null - install: null - ceph: log-whitelist: - wrongly marked me down - objects unfound and apparently lost - thrashosds: chance_pgnum_grow: 2 chance_pgpnum_fix: 1 timeout: 1200 - radosbench: clients: - client.0 ec_pool: true time: 1800 unique_pool: true teuthology_branch: master verbose: true worker_log: /var/lib/teuthworker/archive/worker_logs/worker.plana.16939 2014-04-01 14:32:44,591.591 INFO:teuthology.run_tasks:Running task internal.save_config... 2014-04-01 14:32:44,591.591 INFO:teuthology.task.internal:Saving configuration 2014-04-01 14:32:44,591.591 INFO:teuthology.run_tasks:Running task internal.check_lock... 2014-04-01 14:32:44,591.591 INFO:teuthology.task.internal:Checking locks... 2014-04-01 14:32:46,016.016 DEBUG:teuthology.task.internal:machine status is {u'sshpubkey': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDGJX9/cZFlm+ll32X716yKrmR/RE94iH9TusIFLY5vLoE8CBCupjAKdkn4mqOB5eNvwqapMG63Vww5Cl9zo0wKPEHi3jsZCwbAxByc9sBFivZeBnmTUrYNesQvAs1Izr49/4h71oCt98hfX3hl81iEhIEGjjj7XoD3blyYHlyR9LBNoYMllLfi9Pw4KD1snEikGKlR8zMyAgndlZ5ODU4usiCMXLypAa4wFKfR6w17nZI2/Q2xkUy7l59oap50bOR9mSNMOPqpN1KgsK717JCKMU4jHHw+zN0oWvQPmDZK2ckzbyftBfpOqKyidGHpLjpXfuQ1Xk4R9fgUkpE6gcD5', u'vpshost': None, u'name': u'ubuntu@plana79.front.sepia.ceph.com', u'type': u'plana', u'description': u'None', u'locked_by': u'loic@dachary.org', u'up': 1, u'mac': u'78:2b:cb:55:dc:4e', u'locked': 1, u'distro': u'ubuntu', u'arch': u'x86_64', u'locked_since': u'2014-04-01T05:26:14'} 2014-04-01 14:32:46,328.328 DEBUG:teuthology.task.internal:machine status is {u'sshpubkey': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZxXbzgCaQZsjUFXMXBP6gCnCguoRfZ8dxbV71vaRpMX8p7avh1cFB2qeXDQMBDYukxJZdVBSEomkW/bPygRa+/vlvj7XhcOroy7SkpVY9ZAFkUKScANybxglH2qo+7TFGNB0Yq9GPBQ6z/3sbB50Uzod6gKPrAhGS/uggrSj2KbXK5UUrUQNsSvMtAJqYPDDWRWtwMH/Ezr76FEM//cveTXoViNh557CgJVB9HqmKHTtk9c9iPIi4DBdH5cGWe2f1Lh3HHVO4KOEepI9/2ItMIhoODEobvNJi62Vh76ZR2lcQJw4njyv3GF4c4khK47xQx6vPttvPxMLprClEM/J1', u'vpshost': None, u'name': u'ubuntu@plana78.front.sepia.ceph.com', u'type': u'plana', u'description': u'None', u'locked_by': u'loic@dachary.org', u'up': 1, u'mac': u'78:2b:cb:5b:fc:20', u'locked': 1, u'distro': u'ubuntu', u'arch': u'x86_64', u'locked_since': u'2014-04-01T05:26:14'} 2014-04-01 14:32:46,328.328 INFO:teuthology.run_tasks:Running task internal.connect... 2014-04-01 14:32:46,328.328 INFO:teuthology.task.internal:Opening connections... 2014-04-01 14:32:46,332.332 DEBUG:teuthology.task.internal:connecting to ubuntu@plana79.front.sepia.ceph.com 2014-04-01 14:32:48,136.136 DEBUG:teuthology.task.internal:connecting to ubuntu@plana78.front.sepia.ceph.com 2014-04-01 14:32:49,912.912 INFO:teuthology.task.internal:roles: ubuntu@plana79.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2014-04-01 14:32:49,913.913 INFO:teuthology.task.internal:roles: ubuntu@plana78.front.sepia.ceph.com - ['mon.b', 'mds.a', 'osd.3', 'osd.4', 'osd.5', 'client.1'] 2014-04-01 14:32:49,913.913 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2014-04-01 14:32:49,913.913 INFO:teuthology.task.internal:Checking for old test directory... 2014-04-01 14:32:49,914.914 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "test '!' -e /home/ubuntu/cephtest" 2014-04-01 14:32:50,237.237 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "test '!' -e /home/ubuntu/cephtest" 2014-04-01 14:32:50,697.697 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2014-04-01 14:32:50,698.698 INFO:teuthology.task.internal:Checking for old /var/lib/ceph... 2014-04-01 14:32:50,698.698 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "test '!' -e /var/lib/ceph" 2014-04-01 14:32:51,019.019 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "test '!' -e /var/lib/ceph" 2014-04-01 14:32:51,397.397 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2014-04-01 14:32:51,398.398 INFO:teuthology.run_tasks:Running task internal.base... 2014-04-01 14:32:51,398.398 INFO:teuthology.task.internal:Creating test directory... 2014-04-01 14:32:51,399.399 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'mkdir -m0755 -- /home/ubuntu/cephtest' 2014-04-01 14:32:51,736.736 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'mkdir -m0755 -- /home/ubuntu/cephtest' 2014-04-01 14:32:52,095.095 INFO:teuthology.run_tasks:Running task internal.archive... 2014-04-01 14:32:52,095.095 INFO:teuthology.task.internal:Creating archive directory... 2014-04-01 14:32:52,095.095 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'install -d -m0755 -- /home/ubuntu/cephtest/archive' 2014-04-01 14:32:52,416.416 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'install -d -m0755 -- /home/ubuntu/cephtest/archive' 2014-04-01 14:32:52,942.942 INFO:teuthology.run_tasks:Running task internal.coredump... 2014-04-01 14:32:52,943.943 INFO:teuthology.task.internal:Enabling coredump saving... 2014-04-01 14:32:52,943.943 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core' 2014-04-01 14:32:53,269.269 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core' 2014-04-01 14:32:53,277.277 INFO:teuthology.orchestra.run.out:[10.214.133.38]: kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2014-04-01 14:32:53,598.598 INFO:teuthology.orchestra.run.out:[10.214.133.37]: kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2014-04-01 14:32:53,601.601 INFO:teuthology.run_tasks:Running task internal.sudo... 2014-04-01 14:32:53,601.601 INFO:teuthology.task.internal:Configuring sudo... 2014-04-01 14:32:53,601.601 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "sudo sed -i.orig.teuthology -e 's/^\\([^#]*\\) \\(requiretty\\)/\\1 !\\2/g' -e 's/^\\([^#]*\\) !\\(visiblepw\\)/\\1 \\2/g' /etc/sudoers" 2014-04-01 14:32:53,922.922 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "sudo sed -i.orig.teuthology -e 's/^\\([^#]*\\) \\(requiretty\\)/\\1 !\\2/g' -e 's/^\\([^#]*\\) !\\(visiblepw\\)/\\1 \\2/g' /etc/sudoers" 2014-04-01 14:32:54,253.253 INFO:teuthology.run_tasks:Running task internal.syslog... 2014-04-01 14:32:54,254.254 INFO:teuthology.run_tasks:Running task internal.timer... 2014-04-01 14:32:54,254.254 INFO:teuthology.task.internal:Starting timer... 2014-04-01 14:32:54,254.254 INFO:teuthology.run_tasks:Running task chef... 2014-04-01 14:32:54,259.259 INFO:teuthology.task.chef:Running chef-solo... 2014-04-01 14:32:54,259.259 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "wget -O- 'http://ceph.com/git/?p=ceph-qa-chef.git;a=blob_plain;f=solo/solo-from-scratch;hb=HEAD' | sh -x" 2014-04-01 14:32:54,582.582 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "wget -O- 'http://ceph.com/git/?p=ceph-qa-chef.git;a=blob_plain;f=solo/solo-from-scratch;hb=HEAD' | sh -x" 2014-04-01 14:32:54,629.629 INFO:teuthology.orchestra.run.err:[10.214.133.38]: --2014-04-01 05:32:54-- http://ceph.com/git/?p=ceph-qa-chef.git;a=blob_plain;f=solo/solo-from-scratch;hb=HEAD 2014-04-01 14:32:54,630.630 INFO:teuthology.orchestra.run.err:[10.214.133.38]: Resolving ceph.com (ceph.com)... 208.113.241.137, 2607:f298:4:147::b05:fe2a 2014-04-01 14:32:54,630.630 INFO:teuthology.orchestra.run.err:[10.214.133.38]: Connecting to ceph.com (ceph.com)|208.113.241.137|:80... connected. 2014-04-01 14:32:54,801.801 INFO:teuthology.orchestra.run.err:[10.214.133.38]: HTTP request sent, awaiting response... 200 OK 2014-04-01 14:32:54,801.801 INFO:teuthology.orchestra.run.err:[10.214.133.38]: Length: unspecified [text/plain] 2014-04-01 14:32:54,801.801 INFO:teuthology.orchestra.run.err:[10.214.133.38]: Saving to: `STDOUT' 2014-04-01 14:32:54,801.801 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 14:32:54,801.801 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 0K ...+ set -e 2014-04-01 14:32:54,801.801 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + dirname sh 2014-04-01 14:32:54,801.801 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + cd . 2014-04-01 14:32:54,801.801 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + python -c import platform; distro = platform.dist()[0]; print distro.lower().rstrip() 2014-04-01 14:32:54,801.801 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 151M=0s 2014-04-01 14:32:54,801.801 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 14:32:54,842.842 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 05:32:54 (151 MB/s) - written to stdout [3313] 2014-04-01 14:32:54,842.842 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 14:32:54,843.843 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + distro=ubuntu 2014-04-01 14:32:54,843.843 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + python -c import platform; distro = platform.dist()[1]; print distro 2014-04-01 14:32:54,874.874 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + distroversion=12.04 2014-04-01 14:32:54,875.875 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + echo ubuntu 2014-04-01 14:32:54,945.945 INFO:teuthology.orchestra.run.err:[10.214.133.37]: --2014-04-01 05:32:55-- http://ceph.com/git/?p=ceph-qa-chef.git;a=blob_plain;f=solo/solo-from-scratch;hb=HEAD 2014-04-01 14:32:54,945.945 INFO:teuthology.orchestra.run.err:[10.214.133.37]: Resolving ceph.com (ceph.com)... 208.113.241.137, 2607:f298:4:147::b05:fe2a 2014-04-01 14:32:54,945.945 INFO:teuthology.orchestra.run.err:[10.214.133.37]: Connecting to ceph.com (ceph.com)|208.113.241.137|:80... connected. 2014-04-01 14:32:54,964.964 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + grep -ic -e centos -e redhat -e fedora 2014-04-01 14:32:54,964.964 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + true 2014-04-01 14:32:54,965.965 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + centos=0 2014-04-01 14:32:54,965.965 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + echo ubuntu 2014-04-01 14:32:54,965.965 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + grep -ic -e ubuntu -e debian 2014-04-01 14:32:54,965.965 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + ubuntu=1 2014-04-01 14:32:54,965.965 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + echo 12.04 2014-04-01 14:32:54,966.966 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + grep -ic 14\.04 2014-04-01 14:32:54,966.966 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + true 2014-04-01 14:32:54,966.966 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + trusty=0 2014-04-01 14:32:54,966.966 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + [ 1 -gt 0 ] 2014-04-01 14:32:54,967.967 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + [ 0 -gt 0 ] 2014-04-01 14:32:54,967.967 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + nodepends=--no-install-recommends 2014-04-01 14:32:54,967.967 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + dpkg --status -- git 2014-04-01 14:32:54,968.968 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + sed -n s/^Status: //p 2014-04-01 14:32:54,968.968 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + [ install ok installed != install ok installed ] 2014-04-01 14:32:54,968.968 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + [ -n ] 2014-04-01 14:32:54,968.968 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + [ 0 -gt 0 ] 2014-04-01 14:32:54,969.969 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + mktemp -d --tmpdir solo-from-scratch.XXXXXXXXXXXX 2014-04-01 14:32:54,969.969 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + SCRATCH=/tmp/solo-from-scratch.yqo31HbbiuQB 2014-04-01 14:32:54,969.969 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + cd /tmp/solo-from-scratch.yqo31HbbiuQB 2014-04-01 14:32:54,969.969 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + trap cleanup INT TERM EXIT 2014-04-01 14:32:54,969.969 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + git init 2014-04-01 14:32:55,025.025 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Initialized empty Git repository in /tmp/solo-from-scratch.yqo31HbbiuQB/.git/ 2014-04-01 14:32:55,025.025 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + git pull git://ceph.com/ceph-qa-chef.git 2014-04-01 14:32:55,094.094 INFO:teuthology.orchestra.run.err:[10.214.133.37]: HTTP request sent, awaiting response... 200 OK 2014-04-01 14:32:55,095.095 INFO:teuthology.orchestra.run.err:[10.214.133.37]: Length: unspecified [text/plain] 2014-04-01 14:32:55,095.095 INFO:teuthology.orchestra.run.err:[10.214.133.37]: Saving to: `STDOUT' 2014-04-01 14:32:55,095.095 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 14:32:55,095.095 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 0K ...+ set -e 2014-04-01 14:32:55,095.095 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + dirname sh 2014-04-01 14:32:55,095.095 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + cd . 2014-04-01 14:32:55,095.095 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + python -c import platform; distro = platform.dist()[0]; print distro.lower().rstrip() 2014-04-01 14:32:55,096.096 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 152M=0s 2014-04-01 14:32:55,096.096 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 14:32:55,096.096 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 05:32:55 (152 MB/s) - written to stdout [3313] 2014-04-01 14:32:55,096.096 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 14:32:55,137.137 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + distro=ubuntu 2014-04-01 14:32:55,171.171 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + python -c import platform; distro = platform.dist()[1]; print distro 2014-04-01 14:32:55,171.171 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + distroversion=12.04 2014-04-01 14:32:55,171.171 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + echo ubuntu 2014-04-01 14:32:55,172.172 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + grep -ic -e centos -e redhat -e fedora 2014-04-01 14:32:55,172.172 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + true 2014-04-01 14:32:55,172.172 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + centos=0 2014-04-01 14:32:55,246.246 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + echo ubuntu 2014-04-01 14:32:55,246.246 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + grep -ic -e ubuntu -e debian 2014-04-01 14:32:55,246.246 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + ubuntu=1 2014-04-01 14:32:55,246.246 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + echo 12.04 2014-04-01 14:32:55,246.246 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + grep -ic 14\.04 2014-04-01 14:32:55,246.246 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + true 2014-04-01 14:32:55,246.246 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + trusty=0 2014-04-01 14:32:55,246.246 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + [ 1 -gt 0 ] 2014-04-01 14:32:55,247.247 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + [ 0 -gt 0 ] 2014-04-01 14:32:55,247.247 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + nodepends=--no-install-recommends 2014-04-01 14:32:55,247.247 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + dpkg --status -- git 2014-04-01 14:32:55,247.247 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + sed -n s/^Status: //p 2014-04-01 14:32:55,247.247 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + [ install ok installed != install ok installed ] 2014-04-01 14:32:55,247.247 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + [ -n ] 2014-04-01 14:32:55,247.247 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + [ 0 -gt 0 ] 2014-04-01 14:32:55,247.247 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + mktemp -d --tmpdir solo-from-scratch.XXXXXXXXXXXX 2014-04-01 14:32:55,247.247 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + SCRATCH=/tmp/solo-from-scratch.E8QwQUlZYOc5 2014-04-01 14:32:55,247.247 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + cd /tmp/solo-from-scratch.E8QwQUlZYOc5 2014-04-01 14:32:55,247.247 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + trap cleanup INT TERM EXIT 2014-04-01 14:32:55,247.247 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + git init 2014-04-01 14:32:55,296.296 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Initialized empty Git repository in /tmp/solo-from-scratch.E8QwQUlZYOc5/.git/ 2014-04-01 14:32:55,297.297 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + git pull git://ceph.com/ceph-qa-chef.git 2014-04-01 14:32:57,468.468 INFO:teuthology.orchestra.run.err:[10.214.133.38]: From git://ceph.com/ceph-qa-chef 2014-04-01 14:32:57,468.468 INFO:teuthology.orchestra.run.err:[10.214.133.38]: * branch HEAD -> FETCH_HEAD 2014-04-01 14:32:57,533.533 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + ./solo/run 2014-04-01 14:32:57,675.675 INFO:teuthology.orchestra.run.err:[10.214.133.37]: From git://ceph.com/ceph-qa-chef 2014-04-01 14:32:57,675.675 INFO:teuthology.orchestra.run.err:[10.214.133.37]: * branch HEAD -> FETCH_HEAD 2014-04-01 14:32:57,738.738 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + ./solo/run 2014-04-01 14:33:11,446.446 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:32:59-07:00] INFO: Forking chef instance to converge... 2014-04-01 14:33:11,447.447 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:32:59-07:00] INFO: *** Chef 11.8.0 *** 2014-04-01 14:33:11,447.447 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:32:59-07:00] INFO: Chef-client pid: 2046 2014-04-01 14:33:11,447.447 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:00-07:00] INFO: Setting the run_list to ["recipe[ceph-qa::default]"] from JSON 2014-04-01 14:33:11,447.447 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:00-07:00] INFO: Run List is [recipe[ceph-qa::default]] 2014-04-01 14:33:11,447.447 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:00-07:00] INFO: Run List expands to [ceph-qa::default] 2014-04-01 14:33:11,448.448 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:00-07:00] INFO: Starting Chef Run for plana78.front.sepia.ceph.com 2014-04-01 14:33:11,448.448 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:00-07:00] INFO: Running start handlers 2014-04-01 14:33:11,448.448 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:00-07:00] INFO: Start handlers complete. 2014-04-01 14:33:11,448.448 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:00-07:00] INFO: Processing file[/etc/security/limits.d/ubuntu.conf] action create (ceph-qa::default line 8) 2014-04-01 14:33:11,448.448 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:00-07:00] INFO: Processing execute[remove ceph packages] action run (ceph-qa::ubuntu line 9) 2014-04-01 14:33:11,449.449 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:01-07:00] INFO: execute[remove ceph packages] ran successfully 2014-04-01 14:33:11,449.449 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:01-07:00] INFO: Processing execute[remove /etc/ceph] action run (ceph-qa::ubuntu line 12) 2014-04-01 14:33:11,449.449 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:01-07:00] INFO: execute[remove /etc/ceph] ran successfully 2014-04-01 14:33:11,449.449 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:01-07:00] INFO: Processing execute[remove old source entries] action run (ceph-qa::ubuntu line 15) 2014-04-01 14:33:11,449.449 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:01-07:00] INFO: execute[remove old source entries] ran successfully 2014-04-01 14:33:11,450.450 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:01-07:00] INFO: Processing cookbook_file[/etc/apt/sources.list] action create (ceph-qa::ubuntu line 41) 2014-04-01 14:33:11,450.450 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:01-07:00] INFO: Processing cookbook_file[/etc/cron.weekly/kernel-clean] action create (ceph-qa::ubuntu line 66) 2014-04-01 14:33:11,450.450 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:01-07:00] INFO: Processing execute[Restarting Cron] action run (ceph-qa::ubuntu line 72) 2014-04-01 14:33:11,450.450 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:01-07:00] INFO: execute[Restarting Cron] ran successfully 2014-04-01 14:33:11,450.450 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:01-07:00] INFO: Processing execute[add autobuild gpg key to apt] action run (ceph-qa::ubuntu line 87) 2014-04-01 14:33:11,451.451 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:02-07:00] INFO: execute[add autobuild gpg key to apt] ran successfully 2014-04-01 14:33:11,451.451 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:02-07:00] INFO: Processing file[/etc/apt/sources.list.d/ceph-extras.list] action create (ceph-qa::ubuntu line 96) 2014-04-01 14:33:11,451.451 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:02-07:00] INFO: file[/etc/apt/sources.list.d/ceph-extras.list] created file /etc/apt/sources.list.d/ceph-extras.list 2014-04-01 14:33:11,451.451 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:02-07:00] INFO: file[/etc/apt/sources.list.d/ceph-extras.list] updated file contents /etc/apt/sources.list.d/ceph-extras.list 2014-04-01 14:33:11,451.451 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:02-07:00] INFO: file[/etc/apt/sources.list.d/ceph-extras.list] owner changed to 0 2014-04-01 14:33:11,452.452 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:02-07:00] INFO: file[/etc/apt/sources.list.d/ceph-extras.list] group changed to 0 2014-04-01 14:33:11,452.452 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:02-07:00] INFO: file[/etc/apt/sources.list.d/ceph-extras.list] mode changed to 644 2014-04-01 14:33:11,452.452 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:02-07:00] INFO: Processing execute[add release gpg key to apt] action run (ceph-qa::ubuntu line 104) 2014-04-01 14:33:11,452.452 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:03-07:00] INFO: execute[add release gpg key to apt] ran successfully 2014-04-01 14:33:11,452.452 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:03-07:00] INFO: Processing package[python-pip] action install (ceph-qa::ubuntu line 112) 2014-04-01 14:33:11,453.453 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:06-07:00] INFO: Processing package[python-virtualenv] action install (ceph-qa::ubuntu line 113) 2014-04-01 14:33:11,453.453 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:06-07:00] INFO: Processing package[python-dev] action install (ceph-qa::ubuntu line 114) 2014-04-01 14:33:11,453.453 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:06-07:00] INFO: Processing package[libevent-dev] action install (ceph-qa::ubuntu line 115) 2014-04-01 14:33:11,453.453 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:06-07:00] INFO: Processing package[perl] action install (ceph-qa::ubuntu line 118) 2014-04-01 14:33:11,453.453 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:06-07:00] INFO: Processing package[libwww-perl] action install (ceph-qa::ubuntu line 119) 2014-04-01 14:33:11,454.454 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:06-07:00] INFO: Processing directory[/home/ubuntu/.cpan/CPAN/] action create (ceph-qa::ubuntu line 121) 2014-04-01 14:33:11,454.454 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:06-07:00] INFO: Processing cookbook_file[/home/ubuntu/.cpan/CPAN/MyConfig.pm] action create (ceph-qa::ubuntu line 127) 2014-04-01 14:33:11,454.454 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:06-07:00] INFO: Processing directory[/root/.cpan/CPAN/] action create (ceph-qa::ubuntu line 134) 2014-04-01 14:33:11,454.454 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:06-07:00] INFO: Processing cookbook_file[/root/.cpan/CPAN/MyConfig.pm] action create (ceph-qa::ubuntu line 140) 2014-04-01 14:33:11,454.454 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:06-07:00] INFO: Processing file[/etc/apt/preferences.d/ceph.pref] action create (ceph-qa::radosgw line 4) 2014-04-01 14:33:11,455.455 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:06-07:00] INFO: Processing file[/etc/apt/sources.list.d/radosgw.list] action create (ceph-qa::radosgw line 16) 2014-04-01 14:33:11,455.455 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:06-07:00] INFO: file[/etc/apt/sources.list.d/radosgw.list] created file /etc/apt/sources.list.d/radosgw.list 2014-04-01 14:33:11,455.455 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:06-07:00] INFO: file[/etc/apt/sources.list.d/radosgw.list] updated file contents /etc/apt/sources.list.d/radosgw.list 2014-04-01 14:33:11,455.455 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:06-07:00] INFO: file[/etc/apt/sources.list.d/radosgw.list] owner changed to 0 2014-04-01 14:33:11,455.455 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:06-07:00] INFO: file[/etc/apt/sources.list.d/radosgw.list] group changed to 0 2014-04-01 14:33:11,455.455 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:06-07:00] INFO: file[/etc/apt/sources.list.d/radosgw.list] mode changed to 644 2014-04-01 14:33:11,456.456 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:06-07:00] INFO: Processing execute[apt-get update] action run (ceph-qa::radosgw line 54) 2014-04-01 14:33:11,456.456 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:09-07:00] INFO: execute[apt-get update] ran successfully 2014-04-01 14:33:11,456.456 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:09-07:00] INFO: Processing package[apache2] action upgrade (ceph-qa::radosgw line 58) 2014-04-01 14:33:11,456.456 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:09-07:00] INFO: Processing package[libapache2-mod-fastcgi] action upgrade (ceph-qa::radosgw line 61) 2014-04-01 14:33:11,456.456 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:09-07:00] INFO: Processing package[libfcgi0ldbl] action install (ceph-qa::radosgw line 64) 2014-04-01 14:33:11,456.456 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:10-07:00] INFO: Processing service[apache2] action disable (ceph-qa::radosgw line 66) 2014-04-01 14:33:11,457.457 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:10-07:00] INFO: Processing service[apache2] action stop (ceph-qa::radosgw line 66) 2014-04-01 14:33:11,457.457 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:10-07:00] INFO: Processing package[lsb-release] action install (ceph-qa::ubuntu line 176) 2014-04-01 14:33:11,457.457 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:10-07:00] INFO: Processing package[build-essential] action install (ceph-qa::ubuntu line 177) 2014-04-01 14:33:11,457.457 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:10-07:00] INFO: Processing package[sysstat] action install (ceph-qa::ubuntu line 178) 2014-04-01 14:33:11,542.542 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:10-07:00] INFO: Processing package[gdb] action install (ceph-qa::ubuntu line 179) 2014-04-01 14:33:11,542.542 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:10-07:00] INFO: Processing package[python-configobj] action install (ceph-qa::ubuntu line 180) 2014-04-01 14:33:11,542.542 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:10-07:00] INFO: Processing package[python-gevent] action install (ceph-qa::ubuntu line 181) 2014-04-01 14:33:11,542.542 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:10-07:00] INFO: Processing package[libedit2] action install (ceph-qa::ubuntu line 184) 2014-04-01 14:33:11,542.542 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:10-07:00] INFO: Processing package[libssl0.9.8] action install (ceph-qa::ubuntu line 185) 2014-04-01 14:33:11,542.542 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:10-07:00] INFO: Processing package[libgoogle-perftools0] action install (ceph-qa::ubuntu line 189) 2014-04-01 14:33:11,542.542 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:10-07:00] INFO: Processing package[libboost-thread1.46.1] action install (ceph-qa::ubuntu line 199) 2014-04-01 14:33:11,542.542 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:10-07:00] INFO: Processing package[cryptsetup-bin] action install (ceph-qa::ubuntu line 203) 2014-04-01 14:33:11,542.542 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:10-07:00] INFO: Processing package[xfsprogs] action install (ceph-qa::ubuntu line 204) 2014-04-01 14:33:11,543.543 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:10-07:00] INFO: Processing package[gdisk] action install (ceph-qa::ubuntu line 205) 2014-04-01 14:33:11,543.543 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:10-07:00] INFO: Processing package[parted] action install (ceph-qa::ubuntu line 206) 2014-04-01 14:33:11,543.543 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:10-07:00] INFO: Processing package[smbios-utils] action install (ceph-qa::ubuntu line 210) 2014-04-01 14:33:11,543.543 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:10-07:00] INFO: Processing package[libleveldb1] action install (ceph-qa::ubuntu line 211) 2014-04-01 14:33:11,543.543 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:10-07:00] INFO: Processing package[libcrypto++9] action install (ceph-qa::ubuntu line 223) 2014-04-01 14:33:11,543.543 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:11-07:00] INFO: Processing package[openmpi-common] action remove (ceph-qa::ubuntu line 234) 2014-04-01 14:33:11,543.543 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:11-07:00] INFO: Processing package[libuuid1] action install (ceph-qa::ubuntu line 238) 2014-04-01 14:33:11,543.543 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:11-07:00] INFO: Processing package[libfcgi] action install (ceph-qa::ubuntu line 239) 2014-04-01 14:33:11,543.543 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:11-07:00] INFO: package[libfcgi] is a virtual package, actually acting on package[libfcgi0ldbl] 2014-04-01 14:33:11,543.543 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:11-07:00] INFO: Processing package[btrfs-tools] action install (ceph-qa::ubuntu line 240) 2014-04-01 14:33:11,543.543 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:11-07:00] INFO: Processing package[libatomic-ops-dev] action install (ceph-qa::ubuntu line 243) 2014-04-01 14:33:11,543.543 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:11-07:00] INFO: Processing package[git-core] action install (ceph-qa::ubuntu line 246) 2014-04-01 14:33:11,544.544 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:11-07:00] INFO: Processing package[attr] action install (ceph-qa::ubuntu line 247) 2014-04-01 14:33:11,654.654 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:32:59-07:00] INFO: Forking chef instance to converge... 2014-04-01 14:33:11,654.654 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:32:59-07:00] INFO: *** Chef 11.8.0 *** 2014-04-01 14:33:11,654.654 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:32:59-07:00] INFO: Chef-client pid: 2038 2014-04-01 14:33:11,655.655 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:00-07:00] INFO: Setting the run_list to ["recipe[ceph-qa::default]"] from JSON 2014-04-01 14:33:11,655.655 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:00-07:00] INFO: Run List is [recipe[ceph-qa::default]] 2014-04-01 14:33:11,655.655 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:00-07:00] INFO: Run List expands to [ceph-qa::default] 2014-04-01 14:33:11,656.656 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:00-07:00] INFO: Starting Chef Run for plana79.front.sepia.ceph.com 2014-04-01 14:33:11,656.656 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:00-07:00] INFO: Running start handlers 2014-04-01 14:33:11,656.656 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:00-07:00] INFO: Start handlers complete. 2014-04-01 14:33:11,657.657 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:01-07:00] INFO: Processing file[/etc/security/limits.d/ubuntu.conf] action create (ceph-qa::default line 8) 2014-04-01 14:33:11,657.657 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:01-07:00] INFO: Processing execute[remove ceph packages] action run (ceph-qa::ubuntu line 9) 2014-04-01 14:33:11,657.657 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:01-07:00] INFO: execute[remove ceph packages] ran successfully 2014-04-01 14:33:11,658.658 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:01-07:00] INFO: Processing execute[remove /etc/ceph] action run (ceph-qa::ubuntu line 12) 2014-04-01 14:33:11,658.658 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:01-07:00] INFO: execute[remove /etc/ceph] ran successfully 2014-04-01 14:33:11,658.658 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:01-07:00] INFO: Processing execute[remove old source entries] action run (ceph-qa::ubuntu line 15) 2014-04-01 14:33:11,659.659 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:01-07:00] INFO: execute[remove old source entries] ran successfully 2014-04-01 14:33:11,659.659 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:01-07:00] INFO: Processing cookbook_file[/etc/apt/sources.list] action create (ceph-qa::ubuntu line 41) 2014-04-01 14:33:11,659.659 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:01-07:00] INFO: Processing cookbook_file[/etc/cron.weekly/kernel-clean] action create (ceph-qa::ubuntu line 66) 2014-04-01 14:33:11,660.660 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:01-07:00] INFO: Processing execute[Restarting Cron] action run (ceph-qa::ubuntu line 72) 2014-04-01 14:33:11,660.660 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:01-07:00] INFO: execute[Restarting Cron] ran successfully 2014-04-01 14:33:11,660.660 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:01-07:00] INFO: Processing execute[add autobuild gpg key to apt] action run (ceph-qa::ubuntu line 87) 2014-04-01 14:33:11,660.660 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:02-07:00] INFO: execute[add autobuild gpg key to apt] ran successfully 2014-04-01 14:33:11,660.660 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:02-07:00] INFO: Processing file[/etc/apt/sources.list.d/ceph-extras.list] action create (ceph-qa::ubuntu line 96) 2014-04-01 14:33:11,660.660 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:02-07:00] INFO: file[/etc/apt/sources.list.d/ceph-extras.list] created file /etc/apt/sources.list.d/ceph-extras.list 2014-04-01 14:33:11,661.661 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:02-07:00] INFO: file[/etc/apt/sources.list.d/ceph-extras.list] updated file contents /etc/apt/sources.list.d/ceph-extras.list 2014-04-01 14:33:11,661.661 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:02-07:00] INFO: file[/etc/apt/sources.list.d/ceph-extras.list] owner changed to 0 2014-04-01 14:33:11,661.661 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:02-07:00] INFO: file[/etc/apt/sources.list.d/ceph-extras.list] group changed to 0 2014-04-01 14:33:11,661.661 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:02-07:00] INFO: file[/etc/apt/sources.list.d/ceph-extras.list] mode changed to 644 2014-04-01 14:33:11,661.661 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:02-07:00] INFO: Processing execute[add release gpg key to apt] action run (ceph-qa::ubuntu line 104) 2014-04-01 14:33:11,661.661 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:04-07:00] INFO: execute[add release gpg key to apt] ran successfully 2014-04-01 14:33:11,661.661 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:04-07:00] INFO: Processing package[python-pip] action install (ceph-qa::ubuntu line 112) 2014-04-01 14:33:11,661.661 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:06-07:00] INFO: Processing package[python-virtualenv] action install (ceph-qa::ubuntu line 113) 2014-04-01 14:33:11,661.661 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:06-07:00] INFO: Processing package[python-dev] action install (ceph-qa::ubuntu line 114) 2014-04-01 14:33:11,661.661 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:06-07:00] INFO: Processing package[libevent-dev] action install (ceph-qa::ubuntu line 115) 2014-04-01 14:33:11,661.661 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:06-07:00] INFO: Processing package[perl] action install (ceph-qa::ubuntu line 118) 2014-04-01 14:33:11,661.661 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:07-07:00] INFO: Processing package[libwww-perl] action install (ceph-qa::ubuntu line 119) 2014-04-01 14:33:11,661.661 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:07-07:00] INFO: Processing directory[/home/ubuntu/.cpan/CPAN/] action create (ceph-qa::ubuntu line 121) 2014-04-01 14:33:11,661.661 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:07-07:00] INFO: Processing cookbook_file[/home/ubuntu/.cpan/CPAN/MyConfig.pm] action create (ceph-qa::ubuntu line 127) 2014-04-01 14:33:11,661.661 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:07-07:00] INFO: Processing directory[/root/.cpan/CPAN/] action create (ceph-qa::ubuntu line 134) 2014-04-01 14:33:11,661.661 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:07-07:00] INFO: Processing cookbook_file[/root/.cpan/CPAN/MyConfig.pm] action create (ceph-qa::ubuntu line 140) 2014-04-01 14:33:11,662.662 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:07-07:00] INFO: Processing file[/etc/apt/preferences.d/ceph.pref] action create (ceph-qa::radosgw line 4) 2014-04-01 14:33:11,662.662 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:07-07:00] INFO: Processing file[/etc/apt/sources.list.d/radosgw.list] action create (ceph-qa::radosgw line 16) 2014-04-01 14:33:11,662.662 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:07-07:00] INFO: file[/etc/apt/sources.list.d/radosgw.list] created file /etc/apt/sources.list.d/radosgw.list 2014-04-01 14:33:11,662.662 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:07-07:00] INFO: file[/etc/apt/sources.list.d/radosgw.list] updated file contents /etc/apt/sources.list.d/radosgw.list 2014-04-01 14:33:11,662.662 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:07-07:00] INFO: file[/etc/apt/sources.list.d/radosgw.list] owner changed to 0 2014-04-01 14:33:11,662.662 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:07-07:00] INFO: file[/etc/apt/sources.list.d/radosgw.list] group changed to 0 2014-04-01 14:33:11,662.662 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:07-07:00] INFO: file[/etc/apt/sources.list.d/radosgw.list] mode changed to 644 2014-04-01 14:33:11,662.662 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:07-07:00] INFO: Processing execute[apt-get update] action run (ceph-qa::radosgw line 54) 2014-04-01 14:33:11,662.662 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:10-07:00] INFO: execute[apt-get update] ran successfully 2014-04-01 14:33:11,662.662 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:10-07:00] INFO: Processing package[apache2] action upgrade (ceph-qa::radosgw line 58) 2014-04-01 14:33:11,662.662 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:10-07:00] INFO: Processing package[libapache2-mod-fastcgi] action upgrade (ceph-qa::radosgw line 61) 2014-04-01 14:33:11,662.662 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:10-07:00] INFO: Processing package[libfcgi0ldbl] action install (ceph-qa::radosgw line 64) 2014-04-01 14:33:11,662.662 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:10-07:00] INFO: Processing service[apache2] action disable (ceph-qa::radosgw line 66) 2014-04-01 14:33:11,662.662 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:10-07:00] INFO: Processing service[apache2] action stop (ceph-qa::radosgw line 66) 2014-04-01 14:33:11,662.662 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:10-07:00] INFO: Processing package[lsb-release] action install (ceph-qa::ubuntu line 176) 2014-04-01 14:33:11,821.821 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:10-07:00] INFO: Processing package[build-essential] action install (ceph-qa::ubuntu line 177) 2014-04-01 14:33:11,822.822 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:10-07:00] INFO: Processing package[sysstat] action install (ceph-qa::ubuntu line 178) 2014-04-01 14:33:11,822.822 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:10-07:00] INFO: Processing package[gdb] action install (ceph-qa::ubuntu line 179) 2014-04-01 14:33:11,822.822 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:10-07:00] INFO: Processing package[python-configobj] action install (ceph-qa::ubuntu line 180) 2014-04-01 14:33:11,823.823 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:10-07:00] INFO: Processing package[python-gevent] action install (ceph-qa::ubuntu line 181) 2014-04-01 14:33:11,823.823 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:10-07:00] INFO: Processing package[libedit2] action install (ceph-qa::ubuntu line 184) 2014-04-01 14:33:11,823.823 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:11-07:00] INFO: Processing package[libssl0.9.8] action install (ceph-qa::ubuntu line 185) 2014-04-01 14:33:11,823.823 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:11-07:00] INFO: Processing package[libgoogle-perftools0] action install (ceph-qa::ubuntu line 189) 2014-04-01 14:33:11,823.823 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:11-07:00] INFO: Processing package[libboost-thread1.46.1] action install (ceph-qa::ubuntu line 199) 2014-04-01 14:33:11,824.824 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:11-07:00] INFO: Processing package[cryptsetup-bin] action install (ceph-qa::ubuntu line 203) 2014-04-01 14:33:11,824.824 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:11-07:00] INFO: Processing package[xfsprogs] action install (ceph-qa::ubuntu line 204) 2014-04-01 14:33:11,824.824 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:11-07:00] INFO: Processing package[gdisk] action install (ceph-qa::ubuntu line 205) 2014-04-01 14:33:11,824.824 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:11-07:00] INFO: Processing package[parted] action install (ceph-qa::ubuntu line 206) 2014-04-01 14:33:11,824.824 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:11-07:00] INFO: Processing package[smbios-utils] action install (ceph-qa::ubuntu line 210) 2014-04-01 14:33:11,825.825 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:11-07:00] INFO: Processing package[libleveldb1] action install (ceph-qa::ubuntu line 211) 2014-04-01 14:33:11,825.825 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:11-07:00] INFO: Processing package[libcrypto++9] action install (ceph-qa::ubuntu line 223) 2014-04-01 14:33:11,825.825 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:11-07:00] INFO: Processing package[openmpi-common] action remove (ceph-qa::ubuntu line 234) 2014-04-01 14:33:11,825.825 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:11-07:00] INFO: Processing package[libuuid1] action install (ceph-qa::ubuntu line 238) 2014-04-01 14:33:11,826.826 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:11-07:00] INFO: Processing package[libfcgi] action install (ceph-qa::ubuntu line 239) 2014-04-01 14:33:11,826.826 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:11-07:00] INFO: package[libfcgi] is a virtual package, actually acting on package[libfcgi0ldbl] 2014-04-01 14:33:11,826.826 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:11-07:00] INFO: Processing package[btrfs-tools] action install (ceph-qa::ubuntu line 240) 2014-04-01 14:33:11,826.826 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:11-07:00] INFO: Processing package[libatomic-ops-dev] action install (ceph-qa::ubuntu line 243) 2014-04-01 14:33:11,826.826 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:11-07:00] INFO: Processing package[git-core] action install (ceph-qa::ubuntu line 246) 2014-04-01 14:33:11,827.827 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:11-07:00] INFO: Processing package[attr] action install (ceph-qa::ubuntu line 247) 2014-04-01 14:33:18,455.455 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:11-07:00] INFO: Processing package[dbench] action install (ceph-qa::ubuntu line 248) 2014-04-01 14:33:18,455.455 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:11-07:00] INFO: Processing package[bonnie++] action install (ceph-qa::ubuntu line 249) 2014-04-01 14:33:18,455.455 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:11-07:00] INFO: Processing package[iozone3] action install (ceph-qa::ubuntu line 250) 2014-04-01 14:33:18,455.455 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:11-07:00] INFO: Processing package[tiobench] action install (ceph-qa::ubuntu line 252) 2014-04-01 14:33:18,455.455 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:11-07:00] INFO: Processing package[ltp-kernel-test] action install (ceph-qa::ubuntu line 267) 2014-04-01 14:33:18,455.455 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:11-07:00] INFO: Processing package[valgrind] action install (ceph-qa::ubuntu line 269) 2014-04-01 14:33:18,455.455 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:11-07:00] INFO: Processing package[python-nose] action install (ceph-qa::ubuntu line 270) 2014-04-01 14:33:18,455.455 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:12-07:00] INFO: Processing package[mpich2] action install (ceph-qa::ubuntu line 271) 2014-04-01 14:33:18,456.456 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:12-07:00] INFO: Processing package[libmpich2-3] action install (ceph-qa::ubuntu line 272) 2014-04-01 14:33:18,456.456 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:12-07:00] INFO: Processing package[libmpich2-dev] action install (ceph-qa::ubuntu line 273) 2014-04-01 14:33:18,456.456 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:12-07:00] INFO: Processing package[ant] action install (ceph-qa::ubuntu line 274) 2014-04-01 14:33:18,456.456 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:12-07:00] INFO: Processing package[libtool] action install (ceph-qa::ubuntu line 277) 2014-04-01 14:33:18,456.456 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:12-07:00] INFO: Processing package[automake] action install (ceph-qa::ubuntu line 278) 2014-04-01 14:33:18,456.456 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:12-07:00] INFO: Processing package[gettext] action install (ceph-qa::ubuntu line 279) 2014-04-01 14:33:18,456.456 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:12-07:00] INFO: Processing package[uuid-dev] action install (ceph-qa::ubuntu line 280) 2014-04-01 14:33:18,456.456 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:12-07:00] INFO: Processing package[libacl1-dev] action install (ceph-qa::ubuntu line 281) 2014-04-01 14:33:18,456.456 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:12-07:00] INFO: Processing package[bc] action install (ceph-qa::ubuntu line 282) 2014-04-01 14:33:18,456.456 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:12-07:00] INFO: Processing package[xfsdump] action install (ceph-qa::ubuntu line 283) 2014-04-01 14:33:18,456.456 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:12-07:00] INFO: Processing package[dmapi] action install (ceph-qa::ubuntu line 284) 2014-04-01 14:33:18,456.456 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:12-07:00] INFO: package[dmapi] is a virtual package, actually acting on package[libdm0] 2014-04-01 14:33:18,456.456 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:12-07:00] INFO: Processing package[xfslibs-dev] action install (ceph-qa::ubuntu line 285) 2014-04-01 14:33:18,456.456 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:12-07:00] INFO: Processing package[libattr1-dev] action install (ceph-qa::ubuntu line 286) 2014-04-01 14:33:18,456.456 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:12-07:00] INFO: Processing package[sysprof] action install (ceph-qa::ubuntu line 290) 2014-04-01 14:33:18,456.456 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:12-07:00] INFO: Processing package[vim] action install (ceph-qa::ubuntu line 292) 2014-04-01 14:33:18,456.456 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:12-07:00] INFO: Processing package[pdsh] action install (ceph-qa::ubuntu line 293) 2014-04-01 14:33:18,456.456 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:12-07:00] INFO: Processing package[collectl] action install (ceph-qa::ubuntu line 294) 2014-04-01 14:33:18,456.456 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:12-07:00] INFO: Processing service[collectl] action disable (ceph-qa::ubuntu line 295) 2014-04-01 14:33:18,456.456 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:12-07:00] INFO: Processing service[collectl] action stop (ceph-qa::ubuntu line 295) 2014-04-01 14:33:18,457.457 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:12-07:00] INFO: Processing package[blktrace] action install (ceph-qa::ubuntu line 300) 2014-04-01 14:33:18,457.457 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:12-07:00] INFO: Processing package[python-numpy] action install (ceph-qa::ubuntu line 301) 2014-04-01 14:33:18,457.457 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:12-07:00] INFO: Processing package[python-matplotlib] action install (ceph-qa::ubuntu line 302) 2014-04-01 14:33:18,457.457 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:12-07:00] INFO: Processing package[mencoder] action install (ceph-qa::ubuntu line 303) 2014-04-01 14:33:18,457.457 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:13-07:00] INFO: Processing package[kvm] action install (ceph-qa::ubuntu line 309) 2014-04-01 14:33:18,457.457 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:13-07:00] INFO: Processing package[genisoimage] action install (ceph-qa::ubuntu line 311) 2014-04-01 14:33:18,457.457 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:13-07:00] INFO: Processing package[libjson-xs-perl] action install (ceph-qa::ubuntu line 314) 2014-04-01 14:33:18,457.457 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:13-07:00] INFO: Processing package[xml-twig-tools] action install (ceph-qa::ubuntu line 316) 2014-04-01 14:33:18,457.457 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:13-07:00] INFO: Processing package[default-jdk] action install (ceph-qa::ubuntu line 319) 2014-04-01 14:33:18,457.457 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:13-07:00] INFO: Processing package[junit4] action install (ceph-qa::ubuntu line 320) 2014-04-01 14:33:18,457.457 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:13-07:00] INFO: Processing package[tgt] action install (ceph-qa::ubuntu line 323) 2014-04-01 14:33:18,457.457 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:13-07:00] INFO: Processing package[open-iscsi] action install (ceph-qa::ubuntu line 326) 2014-04-01 14:33:18,457.457 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:13-07:00] INFO: Processing package[smartmontools] action install (ceph-qa::ubuntu line 328) 2014-04-01 14:33:18,457.457 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:13-07:00] INFO: Processing package[nagios-nrpe-server] action install (ceph-qa::ubuntu line 329) 2014-04-01 14:33:18,457.457 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:13-07:00] INFO: Processing package[cifs-utils] action install (ceph-qa::ubuntu line 332) 2014-04-01 14:33:18,457.457 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:13-07:00] INFO: Processing package[distcc] action install (ceph-qa::ubuntu line 335) 2014-04-01 14:33:18,457.457 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:13-07:00] INFO: Processing file[/etc/grub.d/02_force_timeout] action create (ceph-qa::ubuntu line 351) 2014-04-01 14:33:18,457.457 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:13-07:00] INFO: Processing package[ntp] action install (ceph-qa::ubuntu line 364) 2014-04-01 14:33:18,457.457 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:13-07:00] INFO: Processing cookbook_file[/etc/ntp.conf] action create (ceph-qa::ubuntu line 366) 2014-04-01 14:33:18,458.458 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:13-07:00] INFO: Processing service[ntp] action enable (ceph-qa::ubuntu line 374) 2014-04-01 14:33:18,458.458 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:13-07:00] INFO: service[ntp] enabled 2014-04-01 14:33:18,458.458 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:13-07:00] INFO: Processing service[ntp] action start (ceph-qa::ubuntu line 374) 2014-04-01 14:33:18,458.458 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:14-07:00] INFO: Processing execute[add user_xattr to root mount options in fstab] action run (ceph-qa::ubuntu line 378) 2014-04-01 14:33:18,458.458 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:14-07:00] INFO: execute[add user_xattr to root mount options in fstab] ran successfully 2014-04-01 14:33:18,458.458 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:14-07:00] INFO: Processing execute[enable xattr for this boot] action run (ceph-qa::ubuntu line 387) 2014-04-01 14:33:18,458.458 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:14-07:00] INFO: execute[enable xattr for this boot] ran successfully 2014-04-01 14:33:18,458.458 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:14-07:00] INFO: Processing execute[allow fuse mounts to be used by non-owners] action run (ceph-qa::ubuntu line 391) 2014-04-01 14:33:18,522.522 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:14-07:00] INFO: execute[allow fuse mounts to be used by non-owners] ran successfully 2014-04-01 14:33:18,522.522 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:14-07:00] INFO: Processing execute[add user ubuntu to group fuse] action run (ceph-qa::ubuntu line 395) 2014-04-01 14:33:18,522.522 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:14-07:00] INFO: execute[add user ubuntu to group fuse] ran successfully 2014-04-01 14:33:18,522.522 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:14-07:00] INFO: Processing file[/etc/fuse.conf] action create (ceph-qa::ubuntu line 399) 2014-04-01 14:33:18,522.522 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:14-07:00] INFO: Processing execute[add user ubuntu to group kvm] action run (ceph-qa::ubuntu line 403) 2014-04-01 14:33:18,522.522 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:14-07:00] INFO: execute[add user ubuntu to group kvm] ran successfully 2014-04-01 14:33:18,522.522 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:14-07:00] INFO: Processing directory[/home/ubuntu/.ssh] action create (ceph-qa::ubuntu line 407) 2014-04-01 14:33:18,522.522 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:14-07:00] INFO: Processing execute[set up ssh keys] action run (ceph-qa::ubuntu line 413) 2014-04-01 14:33:18,522.522 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:15-07:00] INFO: execute[set up ssh keys] ran successfully 2014-04-01 14:33:18,522.522 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:15-07:00] INFO: Processing cookbook_file[/usr/sbin/megacli] action create (ceph-qa::ubuntu line 425) 2014-04-01 14:33:18,522.522 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:15-07:00] INFO: Processing cookbook_file[/usr/sbin/cli64] action create (ceph-qa::ubuntu line 431) 2014-04-01 14:33:18,522.522 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:15-07:00] INFO: Processing directory[/usr/libexec/] action create (ceph-qa::ubuntu line 440) 2014-04-01 14:33:18,522.522 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:15-07:00] INFO: Processing cookbook_file[/usr/libexec/raid.pl] action create (ceph-qa::ubuntu line 446) 2014-04-01 14:33:18,522.522 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:15-07:00] INFO: Processing cookbook_file[/usr/libexec/smart.pl] action create (ceph-qa::ubuntu line 452) 2014-04-01 14:33:18,522.522 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:15-07:00] INFO: Processing cookbook_file[/usr/libexec/diskusage.pl] action create (ceph-qa::ubuntu line 458) 2014-04-01 14:33:18,522.522 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:15-07:00] INFO: Processing cookbook_file[/etc/ssh/ssh_config] action create (ceph-qa::ubuntu line 467) 2014-04-01 14:33:18,522.522 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:15-07:00] INFO: Processing execute[add ubuntu to disk group] action run (ceph-qa::ubuntu line 474) 2014-04-01 14:33:18,522.522 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:15-07:00] INFO: execute[add ubuntu to disk group] ran successfully 2014-04-01 14:33:18,522.522 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:15-07:00] INFO: Processing execute[enable kernel logging to console] action run (ceph-qa::ubuntu line 482) 2014-04-01 14:33:18,523.523 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:15-07:00] INFO: execute[enable kernel logging to console] ran successfully 2014-04-01 14:33:18,523.523 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:15-07:00] INFO: Processing execute[update-grub] action run (ceph-qa::ubuntu line 535) 2014-04-01 14:33:18,523.523 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:18-07:00] INFO: execute[update-grub] ran successfully 2014-04-01 14:33:18,680.680 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:18-07:00] INFO: Processing cookbook_file[/etc/init/ttyS1.conf] action create (ceph-qa::ubuntu line 541) 2014-04-01 14:33:18,681.681 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:11-07:00] INFO: Processing package[dbench] action install (ceph-qa::ubuntu line 248) 2014-04-01 14:33:18,681.681 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:11-07:00] INFO: Processing package[bonnie++] action install (ceph-qa::ubuntu line 249) 2014-04-01 14:33:18,682.682 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:11-07:00] INFO: Processing package[iozone3] action install (ceph-qa::ubuntu line 250) 2014-04-01 14:33:18,682.682 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:11-07:00] INFO: Processing package[tiobench] action install (ceph-qa::ubuntu line 252) 2014-04-01 14:33:18,682.682 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:11-07:00] INFO: Processing package[ltp-kernel-test] action install (ceph-qa::ubuntu line 267) 2014-04-01 14:33:18,682.682 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:11-07:00] INFO: Processing package[valgrind] action install (ceph-qa::ubuntu line 269) 2014-04-01 14:33:18,682.682 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:11-07:00] INFO: Processing package[python-nose] action install (ceph-qa::ubuntu line 270) 2014-04-01 14:33:18,683.683 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:11-07:00] INFO: Processing package[mpich2] action install (ceph-qa::ubuntu line 271) 2014-04-01 14:33:18,683.683 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:11-07:00] INFO: Processing package[libmpich2-3] action install (ceph-qa::ubuntu line 272) 2014-04-01 14:33:18,683.683 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:11-07:00] INFO: Processing package[libmpich2-dev] action install (ceph-qa::ubuntu line 273) 2014-04-01 14:33:18,683.683 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:11-07:00] INFO: Processing package[ant] action install (ceph-qa::ubuntu line 274) 2014-04-01 14:33:18,683.683 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:11-07:00] INFO: Processing package[libtool] action install (ceph-qa::ubuntu line 277) 2014-04-01 14:33:18,683.683 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:11-07:00] INFO: Processing package[automake] action install (ceph-qa::ubuntu line 278) 2014-04-01 14:33:18,684.684 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:11-07:00] INFO: Processing package[gettext] action install (ceph-qa::ubuntu line 279) 2014-04-01 14:33:18,684.684 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:11-07:00] INFO: Processing package[uuid-dev] action install (ceph-qa::ubuntu line 280) 2014-04-01 14:33:18,684.684 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:11-07:00] INFO: Processing package[libacl1-dev] action install (ceph-qa::ubuntu line 281) 2014-04-01 14:33:18,684.684 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:11-07:00] INFO: Processing package[bc] action install (ceph-qa::ubuntu line 282) 2014-04-01 14:33:18,684.684 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:11-07:00] INFO: Processing package[xfsdump] action install (ceph-qa::ubuntu line 283) 2014-04-01 14:33:18,685.685 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:11-07:00] INFO: Processing package[dmapi] action install (ceph-qa::ubuntu line 284) 2014-04-01 14:33:18,685.685 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:12-07:00] INFO: package[dmapi] is a virtual package, actually acting on package[libdm0] 2014-04-01 14:33:18,685.685 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:12-07:00] INFO: Processing package[xfslibs-dev] action install (ceph-qa::ubuntu line 285) 2014-04-01 14:33:18,685.685 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:12-07:00] INFO: Processing package[libattr1-dev] action install (ceph-qa::ubuntu line 286) 2014-04-01 14:33:18,686.686 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:12-07:00] INFO: Processing package[sysprof] action install (ceph-qa::ubuntu line 290) 2014-04-01 14:33:18,686.686 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:12-07:00] INFO: Processing package[vim] action install (ceph-qa::ubuntu line 292) 2014-04-01 14:33:18,686.686 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:12-07:00] INFO: Processing package[pdsh] action install (ceph-qa::ubuntu line 293) 2014-04-01 14:33:18,686.686 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:12-07:00] INFO: Processing package[collectl] action install (ceph-qa::ubuntu line 294) 2014-04-01 14:33:18,686.686 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:12-07:00] INFO: Processing service[collectl] action disable (ceph-qa::ubuntu line 295) 2014-04-01 14:33:18,687.687 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:12-07:00] INFO: Processing service[collectl] action stop (ceph-qa::ubuntu line 295) 2014-04-01 14:33:18,687.687 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:12-07:00] INFO: Processing package[blktrace] action install (ceph-qa::ubuntu line 300) 2014-04-01 14:33:18,687.687 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:12-07:00] INFO: Processing package[python-numpy] action install (ceph-qa::ubuntu line 301) 2014-04-01 14:33:18,687.687 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:12-07:00] INFO: Processing package[python-matplotlib] action install (ceph-qa::ubuntu line 302) 2014-04-01 14:33:18,687.687 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:12-07:00] INFO: Processing package[mencoder] action install (ceph-qa::ubuntu line 303) 2014-04-01 14:33:18,688.688 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:12-07:00] INFO: Processing package[kvm] action install (ceph-qa::ubuntu line 309) 2014-04-01 14:33:18,688.688 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:12-07:00] INFO: Processing package[genisoimage] action install (ceph-qa::ubuntu line 311) 2014-04-01 14:33:18,688.688 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:12-07:00] INFO: Processing package[libjson-xs-perl] action install (ceph-qa::ubuntu line 314) 2014-04-01 14:33:18,688.688 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:12-07:00] INFO: Processing package[xml-twig-tools] action install (ceph-qa::ubuntu line 316) 2014-04-01 14:33:18,688.688 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:12-07:00] INFO: Processing package[default-jdk] action install (ceph-qa::ubuntu line 319) 2014-04-01 14:33:18,689.689 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:12-07:00] INFO: Processing package[junit4] action install (ceph-qa::ubuntu line 320) 2014-04-01 14:33:18,689.689 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:13-07:00] INFO: Processing package[tgt] action install (ceph-qa::ubuntu line 323) 2014-04-01 14:33:18,689.689 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:13-07:00] INFO: Processing package[open-iscsi] action install (ceph-qa::ubuntu line 326) 2014-04-01 14:33:18,689.689 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:13-07:00] INFO: Processing package[smartmontools] action install (ceph-qa::ubuntu line 328) 2014-04-01 14:33:18,689.689 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:13-07:00] INFO: Processing package[nagios-nrpe-server] action install (ceph-qa::ubuntu line 329) 2014-04-01 14:33:18,690.690 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:13-07:00] INFO: Processing package[cifs-utils] action install (ceph-qa::ubuntu line 332) 2014-04-01 14:33:18,690.690 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:13-07:00] INFO: Processing package[distcc] action install (ceph-qa::ubuntu line 335) 2014-04-01 14:33:18,690.690 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:13-07:00] INFO: Processing file[/etc/grub.d/02_force_timeout] action create (ceph-qa::ubuntu line 351) 2014-04-01 14:33:18,690.690 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:13-07:00] INFO: Processing package[ntp] action install (ceph-qa::ubuntu line 364) 2014-04-01 14:33:18,690.690 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:13-07:00] INFO: Processing cookbook_file[/etc/ntp.conf] action create (ceph-qa::ubuntu line 366) 2014-04-01 14:33:18,691.691 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:13-07:00] INFO: Processing service[ntp] action enable (ceph-qa::ubuntu line 374) 2014-04-01 14:33:18,691.691 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:13-07:00] INFO: service[ntp] enabled 2014-04-01 14:33:18,691.691 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:13-07:00] INFO: Processing service[ntp] action start (ceph-qa::ubuntu line 374) 2014-04-01 14:33:18,691.691 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:13-07:00] INFO: Processing execute[add user_xattr to root mount options in fstab] action run (ceph-qa::ubuntu line 378) 2014-04-01 14:33:18,691.691 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:13-07:00] INFO: execute[add user_xattr to root mount options in fstab] ran successfully 2014-04-01 14:33:18,691.691 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:13-07:00] INFO: Processing execute[enable xattr for this boot] action run (ceph-qa::ubuntu line 387) 2014-04-01 14:33:18,914.914 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:14-07:00] INFO: execute[enable xattr for this boot] ran successfully 2014-04-01 14:33:18,914.914 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:14-07:00] INFO: Processing execute[allow fuse mounts to be used by non-owners] action run (ceph-qa::ubuntu line 391) 2014-04-01 14:33:18,914.914 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:14-07:00] INFO: execute[allow fuse mounts to be used by non-owners] ran successfully 2014-04-01 14:33:18,914.914 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:14-07:00] INFO: Processing execute[add user ubuntu to group fuse] action run (ceph-qa::ubuntu line 395) 2014-04-01 14:33:18,914.914 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:14-07:00] INFO: execute[add user ubuntu to group fuse] ran successfully 2014-04-01 14:33:18,914.914 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:14-07:00] INFO: Processing file[/etc/fuse.conf] action create (ceph-qa::ubuntu line 399) 2014-04-01 14:33:18,915.915 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:14-07:00] INFO: Processing execute[add user ubuntu to group kvm] action run (ceph-qa::ubuntu line 403) 2014-04-01 14:33:18,915.915 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:14-07:00] INFO: execute[add user ubuntu to group kvm] ran successfully 2014-04-01 14:33:18,915.915 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:14-07:00] INFO: Processing directory[/home/ubuntu/.ssh] action create (ceph-qa::ubuntu line 407) 2014-04-01 14:33:18,915.915 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:14-07:00] INFO: Processing execute[set up ssh keys] action run (ceph-qa::ubuntu line 413) 2014-04-01 14:33:18,915.915 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:14-07:00] INFO: execute[set up ssh keys] ran successfully 2014-04-01 14:33:18,915.915 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:14-07:00] INFO: Processing cookbook_file[/usr/sbin/megacli] action create (ceph-qa::ubuntu line 425) 2014-04-01 14:33:18,915.915 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:14-07:00] INFO: Processing cookbook_file[/usr/sbin/cli64] action create (ceph-qa::ubuntu line 431) 2014-04-01 14:33:18,915.915 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:14-07:00] INFO: Processing directory[/usr/libexec/] action create (ceph-qa::ubuntu line 440) 2014-04-01 14:33:18,915.915 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:14-07:00] INFO: Processing cookbook_file[/usr/libexec/raid.pl] action create (ceph-qa::ubuntu line 446) 2014-04-01 14:33:18,915.915 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:14-07:00] INFO: Processing cookbook_file[/usr/libexec/smart.pl] action create (ceph-qa::ubuntu line 452) 2014-04-01 14:33:18,916.916 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:14-07:00] INFO: Processing cookbook_file[/usr/libexec/diskusage.pl] action create (ceph-qa::ubuntu line 458) 2014-04-01 14:33:18,916.916 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:14-07:00] INFO: Processing cookbook_file[/etc/ssh/ssh_config] action create (ceph-qa::ubuntu line 467) 2014-04-01 14:33:18,916.916 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:14-07:00] INFO: Processing execute[add ubuntu to disk group] action run (ceph-qa::ubuntu line 474) 2014-04-01 14:33:18,916.916 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:14-07:00] INFO: execute[add ubuntu to disk group] ran successfully 2014-04-01 14:33:18,916.916 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:14-07:00] INFO: Processing execute[enable kernel logging to console] action run (ceph-qa::ubuntu line 482) 2014-04-01 14:33:18,916.916 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:15-07:00] INFO: execute[enable kernel logging to console] ran successfully 2014-04-01 14:33:18,916.916 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:15-07:00] INFO: Processing execute[update-grub] action run (ceph-qa::ubuntu line 535) 2014-04-01 14:33:18,916.916 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:18-07:00] INFO: execute[update-grub] ran successfully 2014-04-01 14:33:18,916.916 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:18-07:00] INFO: Processing cookbook_file[/etc/init/ttyS1.conf] action create (ceph-qa::ubuntu line 541) 2014-04-01 14:33:22,778.778 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:18-07:00] INFO: Processing service[ttyS1] action enable (ceph-qa::ubuntu line 559) 2014-04-01 14:33:22,778.778 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:18-07:00] INFO: Processing service[ttyS1] action start (ceph-qa::ubuntu line 559) 2014-04-01 14:33:22,779.779 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:18-07:00] INFO: Processing package[nfs-kernel-server] action install (ceph-qa::ubuntu line 578) 2014-04-01 14:33:22,779.779 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:18-07:00] INFO: Processing package[ipcalc] action install (ceph-qa::ubuntu line 581) 2014-04-01 14:33:22,779.779 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:18-07:00] INFO: Processing execute[set up static IP in /etc/hosts] action run (ceph-qa::ubuntu line 585) 2014-04-01 14:33:22,779.779 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:18-07:00] INFO: execute[set up static IP in /etc/hosts] ran successfully 2014-04-01 14:33:22,779.779 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:18-07:00] INFO: Processing execute[set up static IP and 10gig interface] action run (ceph-qa::ubuntu line 593) 2014-04-01 14:33:22,780.780 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:18-07:00] INFO: execute[set up static IP and 10gig interface] ran successfully 2014-04-01 14:33:22,780.780 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:18-07:00] INFO: Processing execute[Installing CPAN Amazon::S3] action run (ceph-qa::ubuntu line 645) 2014-04-01 14:33:22,780.780 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:21-07:00] INFO: execute[Installing CPAN Amazon::S3] ran successfully 2014-04-01 14:33:22,780.780 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:21-07:00] INFO: Processing file[/etc/resolvconf/resolv.conf.d/base] action create (ceph-qa::ubuntu line 650) 2014-04-01 14:33:22,780.780 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:21-07:00] INFO: Processing file[/etc/sudoers.d/90-nagios] action create (ceph-qa::ubuntu line 662) 2014-04-01 14:33:22,781.781 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:21-07:00] INFO: Processing cookbook_file[/etc/modules] action create (ceph-qa::ubuntu line 672) 2014-04-01 14:33:22,781.781 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:21-07:00] INFO: Processing cookbook_file[/etc/nagios/nrpe.cfg] action create (ceph-qa::ubuntu line 680) 2014-04-01 14:33:22,781.781 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:21-07:00] INFO: Processing service[nagios-nrpe-server] action enable (ceph-qa::ubuntu line 688) 2014-04-01 14:33:22,781.781 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:21-07:00] INFO: service[nagios-nrpe-server] enabled 2014-04-01 14:33:22,781.781 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:21-07:00] INFO: Processing service[nagios-nrpe-server] action start (ceph-qa::ubuntu line 688) 2014-04-01 14:33:22,782.782 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:21-07:00] INFO: service[nagios-nrpe-server] started 2014-04-01 14:33:22,782.782 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:21-07:00] INFO: Processing file[/etc/default/nagios-nrpe-server] action create (ceph-qa::ubuntu line 693) 2014-04-01 14:33:22,782.782 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:21-07:00] INFO: Processing execute[Restarting resolvdns] action run (ceph-qa::ubuntu line 702) 2014-04-01 14:33:22,782.782 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:21-07:00] INFO: execute[Restarting resolvdns] ran successfully 2014-04-01 14:33:22,782.782 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:21-07:00] INFO: Processing execute[Enabling auto-fsck fix to prevent boot hangup] action run (ceph-qa::ubuntu line 708) 2014-04-01 14:33:22,783.783 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:21-07:00] INFO: execute[Enabling auto-fsck fix to prevent boot hangup] ran successfully 2014-04-01 14:33:22,783.783 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:21-07:00] INFO: Processing bash[ssh_max_sessions] action run (ceph-qa::ubuntu line 714) 2014-04-01 14:33:22,783.783 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:21-07:00] INFO: Processing service[ssh] action restart (ceph-qa::ubuntu line 730) 2014-04-01 14:33:22,783.783 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:22-07:00] INFO: service[ssh] restarted 2014-04-01 14:33:22,783.783 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:22-07:00] INFO: Processing file[/ceph-qa-ready] action create (ceph-qa::ubuntu line 735) 2014-04-01 14:33:22,783.783 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:22-07:00] INFO: Chef Run complete in 21.88476322 seconds 2014-04-01 14:33:22,784.784 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:22-07:00] INFO: Running report handlers 2014-04-01 14:33:22,784.784 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:33:22-07:00] INFO: Report handlers complete 2014-04-01 14:33:22,809.809 INFO:teuthology.orchestra.run.out:[10.214.133.37]: [2014-04-01T05:32:59-07:00] INFO: Forking chef instance to converge... 2014-04-01 14:33:22,834.834 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + cleanup 2014-04-01 14:33:22,835.835 INFO:teuthology.orchestra.run.err:[10.214.133.37]: + rm -rf /tmp/solo-from-scratch.E8QwQUlZYOc5 2014-04-01 14:33:22,880.880 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:18-07:00] INFO: Processing service[ttyS1] action enable (ceph-qa::ubuntu line 559) 2014-04-01 14:33:22,881.881 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:18-07:00] INFO: Processing service[ttyS1] action start (ceph-qa::ubuntu line 559) 2014-04-01 14:33:22,881.881 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:18-07:00] INFO: Processing package[nfs-kernel-server] action install (ceph-qa::ubuntu line 578) 2014-04-01 14:33:22,881.881 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:18-07:00] INFO: Processing package[ipcalc] action install (ceph-qa::ubuntu line 581) 2014-04-01 14:33:22,881.881 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:18-07:00] INFO: Processing execute[set up static IP in /etc/hosts] action run (ceph-qa::ubuntu line 585) 2014-04-01 14:33:22,881.881 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:18-07:00] INFO: execute[set up static IP in /etc/hosts] ran successfully 2014-04-01 14:33:22,882.882 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:18-07:00] INFO: Processing execute[set up static IP and 10gig interface] action run (ceph-qa::ubuntu line 593) 2014-04-01 14:33:22,882.882 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:18-07:00] INFO: execute[set up static IP and 10gig interface] ran successfully 2014-04-01 14:33:22,882.882 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:18-07:00] INFO: Processing execute[Installing CPAN Amazon::S3] action run (ceph-qa::ubuntu line 645) 2014-04-01 14:33:22,882.882 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:21-07:00] INFO: execute[Installing CPAN Amazon::S3] ran successfully 2014-04-01 14:33:22,882.882 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:21-07:00] INFO: Processing file[/etc/resolvconf/resolv.conf.d/base] action create (ceph-qa::ubuntu line 650) 2014-04-01 14:33:22,882.882 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:21-07:00] INFO: Processing file[/etc/sudoers.d/90-nagios] action create (ceph-qa::ubuntu line 662) 2014-04-01 14:33:22,883.883 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:21-07:00] INFO: Processing cookbook_file[/etc/modules] action create (ceph-qa::ubuntu line 672) 2014-04-01 14:33:22,883.883 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:21-07:00] INFO: Processing cookbook_file[/etc/nagios/nrpe.cfg] action create (ceph-qa::ubuntu line 680) 2014-04-01 14:33:22,883.883 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:21-07:00] INFO: Processing service[nagios-nrpe-server] action enable (ceph-qa::ubuntu line 688) 2014-04-01 14:33:22,883.883 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:21-07:00] INFO: service[nagios-nrpe-server] enabled 2014-04-01 14:33:22,883.883 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:21-07:00] INFO: Processing service[nagios-nrpe-server] action start (ceph-qa::ubuntu line 688) 2014-04-01 14:33:22,884.884 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:21-07:00] INFO: service[nagios-nrpe-server] started 2014-04-01 14:33:22,884.884 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:21-07:00] INFO: Processing file[/etc/default/nagios-nrpe-server] action create (ceph-qa::ubuntu line 693) 2014-04-01 14:33:22,884.884 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:21-07:00] INFO: Processing execute[Restarting resolvdns] action run (ceph-qa::ubuntu line 702) 2014-04-01 14:33:22,884.884 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:21-07:00] INFO: execute[Restarting resolvdns] ran successfully 2014-04-01 14:33:22,885.885 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:21-07:00] INFO: Processing execute[Enabling auto-fsck fix to prevent boot hangup] action run (ceph-qa::ubuntu line 708) 2014-04-01 14:33:22,885.885 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:21-07:00] INFO: execute[Enabling auto-fsck fix to prevent boot hangup] ran successfully 2014-04-01 14:33:22,885.885 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:21-07:00] INFO: Processing bash[ssh_max_sessions] action run (ceph-qa::ubuntu line 714) 2014-04-01 14:33:22,885.885 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:21-07:00] INFO: Processing service[ssh] action restart (ceph-qa::ubuntu line 730) 2014-04-01 14:33:22,885.885 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:22-07:00] INFO: service[ssh] restarted 2014-04-01 14:33:22,885.885 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:22-07:00] INFO: Processing file[/ceph-qa-ready] action create (ceph-qa::ubuntu line 735) 2014-04-01 14:33:22,886.886 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:22-07:00] INFO: Chef Run complete in 22.094358704 seconds 2014-04-01 14:33:22,886.886 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:22-07:00] INFO: Running report handlers 2014-04-01 14:33:22,886.886 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:33:22-07:00] INFO: Report handlers complete 2014-04-01 14:33:22,905.905 INFO:teuthology.orchestra.run.out:[10.214.133.38]: [2014-04-01T05:32:59-07:00] INFO: Forking chef instance to converge... 2014-04-01 14:33:22,937.937 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + cleanup 2014-04-01 14:33:22,937.937 INFO:teuthology.orchestra.run.err:[10.214.133.38]: + rm -rf /tmp/solo-from-scratch.yqo31HbbiuQB 2014-04-01 14:33:22,945.945 INFO:teuthology.task.chef:Reconnecting after ceph-qa-chef run 2014-04-01 14:33:22,945.945 INFO:teuthology.misc:Re-opening connections... 2014-04-01 14:33:22,945.945 INFO:teuthology.misc:trying to connect to ubuntu@plana79.front.sepia.ceph.com 2014-04-01 14:33:24,715.715 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'echo online' 2014-04-01 14:33:25,039.039 INFO:teuthology.orchestra.run.out:[10.214.133.37]: online 2014-04-01 14:33:25,191.191 DEBUG:teuthology.misc:waited 2.24612498283 2014-04-01 14:33:26,191.191 INFO:teuthology.misc:trying to connect to ubuntu@plana78.front.sepia.ceph.com 2014-04-01 14:33:27,970.970 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'echo online' 2014-04-01 14:33:28,294.294 INFO:teuthology.orchestra.run.out:[10.214.133.38]: online 2014-04-01 14:33:28,446.446 DEBUG:teuthology.misc:waited 5.50030779839 2014-04-01 14:33:29,445.445 INFO:teuthology.run_tasks:Running task clock.check... 2014-04-01 14:33:29,451.451 INFO:teuthology.task.clock:Checking initial clock skew... 2014-04-01 14:33:29,451.451 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'PATH=/usr/bin:/usr/sbin ntpdc -p' 2014-04-01 14:33:29,913.913 INFO:teuthology.task.clock.ubuntu@plana79.front.sepia.ceph.com.out:[10.214.133.37]: remote local st poll reach delay offset disp 2014-04-01 14:33:29,913.913 INFO:teuthology.task.clock.ubuntu@plana79.front.sepia.ceph.com.out:[10.214.133.37]: ======================================================================= 2014-04-01 14:33:29,914.914 INFO:teuthology.task.clock.ubuntu@plana79.front.sepia.ceph.com.out:[10.214.133.37]: *clock3.dreamhos 10.214.133.37 1 16 377 0.00038 -0.172497 0.02669 2014-04-01 14:33:29,914.914 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'PATH=/usr/bin:/usr/sbin ntpdc -p' 2014-04-01 14:33:30,282.282 INFO:teuthology.task.clock.ubuntu@plana78.front.sepia.ceph.com.out:[10.214.133.38]: remote local st poll reach delay offset disp 2014-04-01 14:33:30,282.282 INFO:teuthology.task.clock.ubuntu@plana78.front.sepia.ceph.com.out:[10.214.133.38]: ======================================================================= 2014-04-01 14:33:30,282.282 INFO:teuthology.task.clock.ubuntu@plana78.front.sepia.ceph.com.out:[10.214.133.38]: *clock3.dreamhos 10.214.133.38 1 32 377 0.00035 -0.000039 0.03558 2014-04-01 14:33:30,284.284 INFO:teuthology.run_tasks:Running task install... 2014-04-01 14:33:30,285.285 DEBUG:teuthology.task.install:project ceph 2014-04-01 14:33:30,285.285 DEBUG:teuthology.task.install:config {'sha1': 'e672c52b4f8b945a516f2eec006e33665a08f045'} 2014-04-01 14:33:30,285.285 INFO:teuthology.task.install:extra packages: [] 2014-04-01 14:33:30,285.285 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo lsb_release -is' 2014-04-01 14:33:30,655.655 DEBUG:teuthology.misc:System to be installed: Ubuntu 2014-04-01 14:33:30,655.655 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo lsb_release -is' 2014-04-01 14:33:30,656.656 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo apt-key list | grep Ceph' 2014-04-01 14:33:31,027.027 DEBUG:teuthology.misc:System to be installed: Ubuntu 2014-04-01 14:33:31,028.028 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo apt-key list | grep Ceph' 2014-04-01 14:33:31,161.161 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'arch' 2014-04-01 14:33:31,537.537 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'arch' 2014-04-01 14:33:31,771.771 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'lsb_release -is' 2014-04-01 14:33:32,146.146 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'lsb_release -is' 2014-04-01 14:33:32,338.338 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'lsb_release -rs' 2014-04-01 14:33:32,708.708 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'lsb_release -rs' 2014-04-01 14:33:32,835.835 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'lsb_release -sc' 2014-04-01 14:33:33,202.202 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'lsb_release -sc' 2014-04-01 14:33:33,336.336 INFO:teuthology.task.install:config is {'sha1': 'e672c52b4f8b945a516f2eec006e33665a08f045', 'wait_for_package': False, 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'flavor': 'basic', 'extras': None} 2014-04-01 14:33:33,336.336 INFO:teuthology.task.install:Installing packages: ceph, ceph-dbg, ceph-mds, ceph-mds-dbg, ceph-common, ceph-common-dbg, ceph-fuse, ceph-fuse-dbg, ceph-test, ceph-test-dbg, radosgw, radosgw-dbg, python-ceph, libcephfs1, libcephfs1-dbg, librados2, librados2-dbg, librbd1, librbd1-dbg on remote deb x86_64 2014-04-01 14:33:33,336.336 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'arch' 2014-04-01 14:33:33,837.837 INFO:teuthology.task.install:config is {'sha1': 'e672c52b4f8b945a516f2eec006e33665a08f045', 'wait_for_package': False, 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'flavor': 'basic', 'extras': None} 2014-04-01 14:33:33,838.838 INFO:teuthology.task.install:Installing packages: ceph, ceph-dbg, ceph-mds, ceph-mds-dbg, ceph-common, ceph-common-dbg, ceph-fuse, ceph-fuse-dbg, ceph-test, ceph-test-dbg, radosgw, radosgw-dbg, python-ceph, libcephfs1, libcephfs1-dbg, librados2, librados2-dbg, librbd1, librbd1-dbg on remote deb x86_64 2014-04-01 14:33:33,838.838 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'arch' 2014-04-01 14:33:34,053.053 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'lsb_release -is' 2014-04-01 14:33:34,473.473 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'lsb_release -rs' 2014-04-01 14:33:34,676.676 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'lsb_release -is' 2014-04-01 14:33:35,044.044 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'lsb_release -sc' 2014-04-01 14:33:35,189.189 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'lsb_release -rs' 2014-04-01 14:33:35,609.609 INFO:teuthology.task.install:config is {'sha1': 'e672c52b4f8b945a516f2eec006e33665a08f045', 'wait_for_package': False, 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'flavor': 'basic', 'extras': None} 2014-04-01 14:33:35,609.609 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo lsb_release -is' 2014-04-01 14:33:35,805.805 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'lsb_release -sc' 2014-04-01 14:33:36,213.213 DEBUG:teuthology.misc:System to be installed: Ubuntu 2014-04-01 14:33:36,214.214 INFO:teuthology.task.install:Pulling from http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-basic/sha1/e672c52b4f8b945a516f2eec006e33665a08f045 2014-04-01 14:33:36,214.214 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'wget -q -O- http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-basic/sha1/e672c52b4f8b945a516f2eec006e33665a08f045/version' 2014-04-01 14:33:36,417.417 INFO:teuthology.task.install:config is {'sha1': 'e672c52b4f8b945a516f2eec006e33665a08f045', 'wait_for_package': False, 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'flavor': 'basic', 'extras': None} 2014-04-01 14:33:36,417.417 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo lsb_release -is' 2014-04-01 14:33:36,830.830 INFO:teuthology.task.install:Package version is 0.78-459-ge672c52-1precise 2014-04-01 14:33:36,831.831 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'echo deb http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-basic/sha1/e672c52b4f8b945a516f2eec006e33665a08f045 precise main | sudo tee /etc/apt/sources.list.d/ceph.list' 2014-04-01 14:33:37,033.033 DEBUG:teuthology.misc:System to be installed: Ubuntu 2014-04-01 14:33:37,034.034 INFO:teuthology.task.install:Pulling from http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-basic/sha1/e672c52b4f8b945a516f2eec006e33665a08f045 2014-04-01 14:33:37,034.034 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'wget -q -O- http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-basic/sha1/e672c52b4f8b945a516f2eec006e33665a08f045/version' 2014-04-01 14:33:37,444.444 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=0.78-459-ge672c52-1precise ceph-dbg=0.78-459-ge672c52-1precise ceph-mds=0.78-459-ge672c52-1precise ceph-mds-dbg=0.78-459-ge672c52-1precise ceph-common=0.78-459-ge672c52-1precise ceph-common-dbg=0.78-459-ge672c52-1precise ceph-fuse=0.78-459-ge672c52-1precise ceph-fuse-dbg=0.78-459-ge672c52-1precise ceph-test=0.78-459-ge672c52-1precise ceph-test-dbg=0.78-459-ge672c52-1precise radosgw=0.78-459-ge672c52-1precise radosgw-dbg=0.78-459-ge672c52-1precise python-ceph=0.78-459-ge672c52-1precise libcephfs1=0.78-459-ge672c52-1precise libcephfs1-dbg=0.78-459-ge672c52-1precise librados2=0.78-459-ge672c52-1precise librados2-dbg=0.78-459-ge672c52-1precise librbd1=0.78-459-ge672c52-1precise librbd1-dbg=0.78-459-ge672c52-1precise' 2014-04-01 14:33:37,646.646 INFO:teuthology.task.install:Package version is 0.78-459-ge672c52-1precise 2014-04-01 14:33:37,647.647 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'echo deb http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-basic/sha1/e672c52b4f8b945a516f2eec006e33665a08f045 precise main | sudo tee /etc/apt/sources.list.d/ceph.list' 2014-04-01 14:33:38,278.278 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=0.78-459-ge672c52-1precise ceph-dbg=0.78-459-ge672c52-1precise ceph-mds=0.78-459-ge672c52-1precise ceph-mds-dbg=0.78-459-ge672c52-1precise ceph-common=0.78-459-ge672c52-1precise ceph-common-dbg=0.78-459-ge672c52-1precise ceph-fuse=0.78-459-ge672c52-1precise ceph-fuse-dbg=0.78-459-ge672c52-1precise ceph-test=0.78-459-ge672c52-1precise ceph-test-dbg=0.78-459-ge672c52-1precise radosgw=0.78-459-ge672c52-1precise radosgw-dbg=0.78-459-ge672c52-1precise python-ceph=0.78-459-ge672c52-1precise libcephfs1=0.78-459-ge672c52-1precise libcephfs1-dbg=0.78-459-ge672c52-1precise librados2=0.78-459-ge672c52-1precise librados2-dbg=0.78-459-ge672c52-1precise librbd1=0.78-459-ge672c52-1precise librbd1-dbg=0.78-459-ge672c52-1precise' 2014-04-01 14:34:45,669.669 DEBUG:teuthology.parallel:result is None 2014-04-01 14:34:46,568.568 DEBUG:teuthology.parallel:result is None 2014-04-01 14:34:46,569.569 INFO:teuthology.run_tasks:Running task ceph... 2014-04-01 14:34:46,593.593 INFO:teuthology.task.ceph:Making ceph log dir writeable by non-root... 2014-04-01 14:34:46,593.593 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo chmod 777 /var/log/ceph' 2014-04-01 14:34:47,052.052 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo chmod 777 /var/log/ceph' 2014-04-01 14:34:47,381.381 INFO:teuthology.task.ceph:Disabling ceph logrotate... 2014-04-01 14:34:47,382.382 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo rm -f -- /etc/logrotate.d/ceph' 2014-04-01 14:34:47,702.702 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo rm -f -- /etc/logrotate.d/ceph' 2014-04-01 14:34:48,067.067 INFO:teuthology.task.ceph:Creating extra log directories... 2014-04-01 14:34:48,068.068 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo install -d -m0755 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger' 2014-04-01 14:34:48,387.387 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo install -d -m0755 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger' 2014-04-01 14:34:48,718.718 INFO:teuthology.task.ceph:Shipping valgrind.supp... 2014-04-01 14:34:48,719.719 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /home/ubuntu/cephtest/valgrind.supp' 2014-04-01 14:34:49,518.518 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /home/ubuntu/cephtest/valgrind.supp' 2014-04-01 14:34:50,179.179 INFO:teuthology.task.ceph:Shipping 'daemon-helper'... 2014-04-01 14:34:50,180.180 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /usr/bin/daemon-helper' 2014-04-01 14:34:50,810.810 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo chmod a=rx -- /usr/bin/daemon-helper' 2014-04-01 14:34:51,309.309 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /usr/bin/daemon-helper' 2014-04-01 14:34:51,980.980 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo chmod a=rx -- /usr/bin/daemon-helper' 2014-04-01 14:34:52,509.509 INFO:teuthology.task.ceph:Shipping 'adjust-ulimits'... 2014-04-01 14:34:52,509.509 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /usr/bin/adjust-ulimits' 2014-04-01 14:34:53,179.179 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo chmod a=rx -- /usr/bin/adjust-ulimits' 2014-04-01 14:34:53,677.677 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /usr/bin/adjust-ulimits' 2014-04-01 14:34:54,342.342 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo chmod a=rx -- /usr/bin/adjust-ulimits' 2014-04-01 14:34:54,878.878 INFO:teuthology.task.ceph:Shipping 'kcon_most'... 2014-04-01 14:34:54,879.879 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /usr/bin/kcon_most' 2014-04-01 14:34:55,564.564 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo chmod a=rx -- /usr/bin/kcon_most' 2014-04-01 14:34:56,106.106 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /usr/bin/kcon_most' 2014-04-01 14:34:56,859.859 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo chmod a=rx -- /usr/bin/kcon_most' 2014-04-01 14:34:57,391.391 INFO:teuthology.task.ceph:Creating ceph cluster... 2014-04-01 14:34:57,391.391 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'install -d -m0755 -- /home/ubuntu/cephtest/data' 2014-04-01 14:34:57,881.881 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'install -d -m0755 -- /home/ubuntu/cephtest/data' 2014-04-01 14:34:58,355.355 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo install -d -m0777 -- /var/run/ceph' 2014-04-01 14:34:58,678.678 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo install -d -m0777 -- /var/run/ceph' 2014-04-01 14:34:59,011.011 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'cat -- /scratch_devs' 2014-04-01 14:34:59,688.688 INFO:teuthology.orchestra.run.err:[10.214.133.37]: cat: /scratch_devs: No such file or directory 2014-04-01 14:34:59,689.689 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'ls /dev/[sv]d?' 2014-04-01 14:35:00,369.369 DEBUG:teuthology.misc:devs=['/dev/sda', '/dev/sdb', '/dev/sdc', '/dev/sdd'] 2014-04-01 14:35:00,370.370 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'stat /dev/sda && sudo dd if=/dev/sda of=/dev/null count=1 && ! mount | grep -q /dev/sda' 2014-04-01 14:35:00,895.895 INFO:teuthology.orchestra.run.out:[10.214.133.37]: File: `/dev/sda' 2014-04-01 14:35:00,896.896 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Size: 0 Blocks: 0 IO Block: 4096 block special file 2014-04-01 14:35:00,896.896 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Device: 4h/4d Inode: 13404 Links: 1 Device type: 8,0 2014-04-01 14:35:00,896.896 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2014-04-01 14:35:00,896.896 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Access: 2014-04-01 05:25:43.364130854 -0700 2014-04-01 14:35:00,897.897 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Modify: 2014-04-01 05:25:43.240130858 -0700 2014-04-01 14:35:00,897.897 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Change: 2014-04-01 05:25:43.240130858 -0700 2014-04-01 14:35:00,898.898 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Birth: - 2014-04-01 14:35:00,903.903 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 1+0 records in 2014-04-01 14:35:00,903.903 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 1+0 records out 2014-04-01 14:35:00,903.903 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 512 bytes (512 B) copied, 3.9669e-05 s, 12.9 MB/s 2014-04-01 14:35:00,905.905 DEBUG:teuthology.misc:get_scratch_devices: /dev/sda is in use 2014-04-01 14:35:00,906.906 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'stat /dev/sdb && sudo dd if=/dev/sdb of=/dev/null count=1 && ! mount | grep -q /dev/sdb' 2014-04-01 14:35:01,400.400 INFO:teuthology.orchestra.run.out:[10.214.133.37]: File: `/dev/sdb' 2014-04-01 14:35:01,401.401 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Size: 0 Blocks: 0 IO Block: 4096 block special file 2014-04-01 14:35:01,401.401 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Device: 4h/4d Inode: 13401 Links: 1 Device type: 8,10 2014-04-01 14:35:01,401.401 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2014-04-01 14:35:01,402.402 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Access: 2014-04-01 05:33:00.915518171 -0700 2014-04-01 14:35:01,402.402 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Modify: 2014-04-01 05:25:41.568130917 -0700 2014-04-01 14:35:01,402.402 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Change: 2014-04-01 05:25:41.568130917 -0700 2014-04-01 14:35:01,402.402 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Birth: - 2014-04-01 14:35:01,405.405 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 1+0 records in 2014-04-01 14:35:01,405.405 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 1+0 records out 2014-04-01 14:35:01,406.406 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 512 bytes (512 B) copied, 0.000604206 s, 847 kB/s 2014-04-01 14:35:01,409.409 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'stat /dev/sdc && sudo dd if=/dev/sdc of=/dev/null count=1 && ! mount | grep -q /dev/sdc' 2014-04-01 14:35:01,903.903 INFO:teuthology.orchestra.run.out:[10.214.133.37]: File: `/dev/sdc' 2014-04-01 14:35:01,903.903 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Size: 0 Blocks: 0 IO Block: 4096 block special file 2014-04-01 14:35:01,903.903 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Device: 4h/4d Inode: 13402 Links: 1 Device type: 8,20 2014-04-01 14:35:01,903.903 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2014-04-01 14:35:01,903.903 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Access: 2014-04-01 05:33:00.927524285 -0700 2014-04-01 14:35:01,903.903 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Modify: 2014-04-01 05:25:41.568130917 -0700 2014-04-01 14:35:01,903.903 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Change: 2014-04-01 05:25:41.568130917 -0700 2014-04-01 14:35:01,903.903 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Birth: - 2014-04-01 14:35:01,907.907 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 1+0 records in 2014-04-01 14:35:01,907.907 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 1+0 records out 2014-04-01 14:35:01,907.907 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 512 bytes (512 B) copied, 0.000471251 s, 1.1 MB/s 2014-04-01 14:35:01,910.910 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'stat /dev/sdd && sudo dd if=/dev/sdd of=/dev/null count=1 && ! mount | grep -q /dev/sdd' 2014-04-01 14:35:02,401.401 INFO:teuthology.orchestra.run.out:[10.214.133.37]: File: `/dev/sdd' 2014-04-01 14:35:02,402.402 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Size: 0 Blocks: 0 IO Block: 4096 block special file 2014-04-01 14:35:02,402.402 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Device: 4h/4d Inode: 13403 Links: 1 Device type: 8,30 2014-04-01 14:35:02,403.403 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2014-04-01 14:35:02,403.403 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Access: 2014-04-01 05:33:00.939530401 -0700 2014-04-01 14:35:02,403.403 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Modify: 2014-04-01 05:25:41.568130917 -0700 2014-04-01 14:35:02,404.404 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Change: 2014-04-01 05:25:41.568130917 -0700 2014-04-01 14:35:02,404.404 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Birth: - 2014-04-01 14:35:02,407.407 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 1+0 records in 2014-04-01 14:35:02,408.408 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 1+0 records out 2014-04-01 14:35:02,408.408 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 512 bytes (512 B) copied, 0.000388339 s, 1.3 MB/s 2014-04-01 14:35:02,412.412 INFO:teuthology.task.ceph:fs option selected, checking for scratch devs 2014-04-01 14:35:02,412.412 INFO:teuthology.task.ceph:found devs: ['/dev/sdb', '/dev/sdc', '/dev/sdd'] 2014-04-01 14:35:02,413.413 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "ls -l '/dev/disk/by-id/wwn-*'" 2014-04-01 14:35:02,912.912 INFO:teuthology.orchestra.run.err:[10.214.133.37]: ls: cannot access /dev/disk/by-id/wwn-*: No such file or directory 2014-04-01 14:35:03,069.069 ERROR:teuthology.misc:Failed to get wwn devices! Using /dev/sd* devices... 2014-04-01 14:35:03,070.070 INFO:teuthology.task.ceph:dev map: {'1': '/dev/sdb', '0': '/dev/sdd', '2': '/dev/sdc'} 2014-04-01 14:35:03,070.070 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'cat -- /scratch_devs' 2014-04-01 14:35:03,402.402 INFO:teuthology.orchestra.run.err:[10.214.133.38]: cat: /scratch_devs: No such file or directory 2014-04-01 14:35:03,403.403 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'ls /dev/[sv]d?' 2014-04-01 14:35:04,106.106 DEBUG:teuthology.misc:devs=['/dev/sda', '/dev/sdb', '/dev/sdc', '/dev/sdd'] 2014-04-01 14:35:04,106.106 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'stat /dev/sda && sudo dd if=/dev/sda of=/dev/null count=1 && ! mount | grep -q /dev/sda' 2014-04-01 14:35:04,649.649 INFO:teuthology.orchestra.run.out:[10.214.133.38]: File: `/dev/sda' 2014-04-01 14:35:04,649.649 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Size: 0 Blocks: 0 IO Block: 4096 block special file 2014-04-01 14:35:04,649.649 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Device: 4h/4d Inode: 13411 Links: 1 Device type: 8,0 2014-04-01 14:35:04,650.650 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2014-04-01 14:35:04,650.650 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Access: 2014-04-01 05:23:44.613983208 -0700 2014-04-01 14:35:04,650.650 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Modify: 2014-04-01 05:23:44.533983205 -0700 2014-04-01 14:35:04,650.650 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Change: 2014-04-01 05:23:44.533983205 -0700 2014-04-01 14:35:04,650.650 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Birth: - 2014-04-01 14:35:04,654.654 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 1+0 records in 2014-04-01 14:35:04,655.655 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 1+0 records out 2014-04-01 14:35:04,655.655 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 512 bytes (512 B) copied, 3.9149e-05 s, 13.1 MB/s 2014-04-01 14:35:04,658.658 DEBUG:teuthology.misc:get_scratch_devices: /dev/sda is in use 2014-04-01 14:35:04,659.659 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'stat /dev/sdb && sudo dd if=/dev/sdb of=/dev/null count=1 && ! mount | grep -q /dev/sdb' 2014-04-01 14:35:05,206.206 INFO:teuthology.orchestra.run.out:[10.214.133.38]: File: `/dev/sdb' 2014-04-01 14:35:05,206.206 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Size: 0 Blocks: 0 IO Block: 4096 block special file 2014-04-01 14:35:05,206.206 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Device: 4h/4d Inode: 13414 Links: 1 Device type: 8,10 2014-04-01 14:35:05,207.207 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2014-04-01 14:35:05,207.207 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Access: 2014-04-01 05:33:00.611584131 -0700 2014-04-01 14:35:05,207.207 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Modify: 2014-04-01 05:23:43.469983153 -0700 2014-04-01 14:35:05,207.207 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Change: 2014-04-01 05:23:43.469983153 -0700 2014-04-01 14:35:05,207.207 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Birth: - 2014-04-01 14:35:05,211.211 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 1+0 records in 2014-04-01 14:35:05,211.211 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 1+0 records out 2014-04-01 14:35:05,212.212 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 512 bytes (512 B) copied, 0.000568712 s, 900 kB/s 2014-04-01 14:35:05,215.215 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'stat /dev/sdc && sudo dd if=/dev/sdc of=/dev/null count=1 && ! mount | grep -q /dev/sdc' 2014-04-01 14:35:05,799.799 INFO:teuthology.orchestra.run.out:[10.214.133.38]: File: `/dev/sdc' 2014-04-01 14:35:05,799.799 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Size: 0 Blocks: 0 IO Block: 4096 block special file 2014-04-01 14:35:05,800.800 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Device: 4h/4d Inode: 11515 Links: 1 Device type: 8,20 2014-04-01 14:35:05,800.800 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2014-04-01 14:35:05,800.800 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Access: 2014-04-01 05:33:00.623584252 -0700 2014-04-01 14:35:05,800.800 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Modify: 2014-04-01 05:23:43.469983153 -0700 2014-04-01 14:35:05,800.800 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Change: 2014-04-01 05:23:43.469983153 -0700 2014-04-01 14:35:05,801.801 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Birth: - 2014-04-01 14:35:05,804.804 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 1+0 records in 2014-04-01 14:35:05,805.805 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 1+0 records out 2014-04-01 14:35:05,805.805 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 512 bytes (512 B) copied, 0.000476604 s, 1.1 MB/s 2014-04-01 14:35:05,808.808 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'stat /dev/sdd && sudo dd if=/dev/sdd of=/dev/null count=1 && ! mount | grep -q /dev/sdd' 2014-04-01 14:35:06,400.400 INFO:teuthology.orchestra.run.out:[10.214.133.38]: File: `/dev/sdd' 2014-04-01 14:35:06,401.401 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Size: 0 Blocks: 0 IO Block: 4096 block special file 2014-04-01 14:35:06,401.401 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Device: 4h/4d Inode: 11516 Links: 1 Device type: 8,30 2014-04-01 14:35:06,401.401 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2014-04-01 14:35:06,401.401 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Access: 2014-04-01 05:33:00.631584334 -0700 2014-04-01 14:35:06,402.402 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Modify: 2014-04-01 05:23:43.469983153 -0700 2014-04-01 14:35:06,402.402 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Change: 2014-04-01 05:23:43.469983153 -0700 2014-04-01 14:35:06,402.402 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Birth: - 2014-04-01 14:35:06,405.405 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 1+0 records in 2014-04-01 14:35:06,405.405 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 1+0 records out 2014-04-01 14:35:06,405.405 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 512 bytes (512 B) copied, 0.000465938 s, 1.1 MB/s 2014-04-01 14:35:06,408.408 INFO:teuthology.task.ceph:fs option selected, checking for scratch devs 2014-04-01 14:35:06,409.409 INFO:teuthology.task.ceph:found devs: ['/dev/sdb', '/dev/sdc', '/dev/sdd'] 2014-04-01 14:35:06,409.409 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "ls -l '/dev/disk/by-id/wwn-*'" 2014-04-01 14:35:07,006.006 INFO:teuthology.orchestra.run.err:[10.214.133.38]: ls: cannot access /dev/disk/by-id/wwn-*: No such file or directory 2014-04-01 14:35:07,007.007 ERROR:teuthology.misc:Failed to get wwn devices! Using /dev/sd* devices... 2014-04-01 14:35:07,007.007 INFO:teuthology.task.ceph:dev map: {'3': '/dev/sdd', '5': '/dev/sdc', '4': '/dev/sdb'} 2014-04-01 14:35:07,007.007 INFO:teuthology.task.ceph:Generating config... 2014-04-01 14:35:07,014.014 INFO:teuthology.task.ceph:[global] ms inject socket failures = 5000 2014-04-01 14:35:07,015.015 INFO:teuthology.task.ceph:[mon] debug mon = 20 2014-04-01 14:35:07,015.015 INFO:teuthology.task.ceph:[mon] debug paxos = 20 2014-04-01 14:35:07,015.015 INFO:teuthology.task.ceph:[mon] debug ms = 1 2014-04-01 14:35:07,015.015 INFO:teuthology.task.ceph:[osd] debug osd = 20 2014-04-01 14:35:07,016.016 INFO:teuthology.task.ceph:[osd] debug filestore = 20 2014-04-01 14:35:07,016.016 INFO:teuthology.task.ceph:[osd] debug journal = 20 2014-04-01 14:35:07,016.016 INFO:teuthology.task.ceph:[osd] debug ms = 1 2014-04-01 14:35:07,016.016 INFO:teuthology.task.ceph:Setting up mon.a... 2014-04-01 14:35:07,017.017 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring' 2014-04-01 14:35:07,441.441 INFO:teuthology.orchestra.run.out:[10.214.133.37]: creating /etc/ceph/ceph.keyring 2014-04-01 14:35:07,452.452 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring' 2014-04-01 14:35:08,013.013 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo chmod 0644 /etc/ceph/ceph.keyring' 2014-04-01 14:35:08,547.547 DEBUG:teuthology.misc:Ceph mon addresses: [('a', '10.214.133.37:6789'), ('c', '10.214.133.37:6790'), ('b', '10.214.133.38:6789')] 2014-04-01 14:35:08,548.548 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --add a 10.214.133.37:6789 --add c 10.214.133.37:6790 --add b 10.214.133.38:6789 --print /home/ubuntu/cephtest/monmap' 2014-04-01 14:35:09,062.062 INFO:teuthology.task.ceph:Writing ceph.conf for FSID 4fc4b977-b888-4422-ba3c-54f89f334abf... 2014-04-01 14:35:09,065.065 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf' 2014-04-01 14:35:09,489.489 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf' 2014-04-01 14:35:10,134.134 INFO:teuthology.task.ceph:Creating admin key on mon.a... 2014-04-01 14:35:10,134.134 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow *' --cap mds allow /etc/ceph/ceph.keyring" 2014-04-01 14:35:10,683.683 INFO:teuthology.task.ceph:Copying monmap to all nodes... 2014-04-01 14:35:10,683.683 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'cat -- /etc/ceph/ceph.keyring' 2014-04-01 14:35:11,301.301 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'cat -- /home/ubuntu/cephtest/monmap' 2014-04-01 14:35:11,973.973 INFO:teuthology.task.ceph:Sending monmap to node ubuntu@plana79.front.sepia.ceph.com 2014-04-01 14:35:11,974.974 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /etc/ceph/ceph.keyring && sudo chmod 0644 /etc/ceph/ceph.keyring' 2014-04-01 14:35:12,844.844 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /home/ubuntu/cephtest/monmap' 2014-04-01 14:35:13,722.722 INFO:teuthology.task.ceph:Sending monmap to node ubuntu@plana78.front.sepia.ceph.com 2014-04-01 14:35:13,722.722 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /etc/ceph/ceph.keyring && sudo chmod 0644 /etc/ceph/ceph.keyring' 2014-04-01 14:35:14,388.388 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /home/ubuntu/cephtest/monmap' 2014-04-01 14:35:15,253.253 INFO:teuthology.task.ceph:Setting up mon nodes... 2014-04-01 14:35:15,253.253 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage osdmaptool -c /etc/ceph/ceph.conf --clobber --createsimple 6 /home/ubuntu/cephtest/osdmap --pg_bits 2 --pgp_bits 4' 2014-04-01 14:35:15,774.774 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage osdmaptool -c /etc/ceph/ceph.conf --clobber --createsimple 6 /home/ubuntu/cephtest/osdmap --pg_bits 2 --pgp_bits 4' 2014-04-01 14:35:15,801.801 INFO:teuthology.orchestra.run.err:[10.214.133.38]: osdmaptool: osdmap file '/home/ubuntu/cephtest/osdmap' 2014-04-01 14:35:15,802.802 INFO:teuthology.orchestra.run.out:[10.214.133.38]: osdmaptool: writing epoch 1 to /home/ubuntu/cephtest/osdmap 2014-04-01 14:35:16,250.250 INFO:teuthology.orchestra.run.err:[10.214.133.37]: osdmaptool: osdmap file '/home/ubuntu/cephtest/osdmap' 2014-04-01 14:35:16,250.250 INFO:teuthology.orchestra.run.out:[10.214.133.37]: osdmaptool: writing epoch 1 to /home/ubuntu/cephtest/osdmap 2014-04-01 14:35:16,252.252 INFO:teuthology.task.ceph:Setting up mds nodes... 2014-04-01 14:35:16,252.252 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring' 2014-04-01 14:35:16,622.622 INFO:teuthology.orchestra.run.out:[10.214.133.38]: creating /var/lib/ceph/mds/ceph-a/keyring 2014-04-01 14:35:16,623.623 INFO:teuthology.task.ceph_client:Setting up client nodes... 2014-04-01 14:35:16,624.624 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring' 2014-04-01 14:35:16,963.963 INFO:teuthology.orchestra.run.out:[10.214.133.37]: creating /etc/ceph/ceph.client.0.keyring 2014-04-01 14:35:16,971.971 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring' 2014-04-01 14:35:17,310.310 INFO:teuthology.orchestra.run.out:[10.214.133.38]: creating /etc/ceph/ceph.client.1.keyring 2014-04-01 14:35:17,316.316 INFO:teuthology.task.ceph:Running mkfs on osd nodes... 2014-04-01 14:35:17,317.317 INFO:teuthology.task.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@plana79.front.sepia.ceph.com'): {'1': '/dev/sdb', '0': '/dev/sdd', '2': '/dev/sdc'}, Remote(name='ubuntu@plana78.front.sepia.ceph.com'): {'3': '/dev/sdd', '5': '/dev/sdc', '4': '/dev/sdb'}} 2014-04-01 14:35:17,317.317 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo mkdir -p /var/lib/ceph/osd/ceph-0' 2014-04-01 14:35:17,646.646 INFO:teuthology.task.ceph:{} 2014-04-01 14:35:17,647.647 INFO:teuthology.task.ceph:0 2014-04-01 14:35:17,647.647 INFO:teuthology.task.ceph:['mkfs.ext4'] on /dev/sdd on ubuntu@plana79.front.sepia.ceph.com 2014-04-01 14:35:17,647.647 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'yes | sudo mkfs.ext4 /dev/sdd' 2014-04-01 14:35:18,177.177 INFO:teuthology.orchestra.run.err:[10.214.133.37]: mke2fs 1.42 (29-Nov-2011) 2014-04-01 14:35:18,190.190 INFO:teuthology.orchestra.run.out:[10.214.133.37]: /dev/sdd is entire device, not just one partition! 2014-04-01 14:35:18,191.191 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Proceed anyway? (y,n) Filesystem label= 2014-04-01 14:35:18,191.191 INFO:teuthology.orchestra.run.out:[10.214.133.37]: OS type: Linux 2014-04-01 14:35:18,191.191 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Block size=4096 (log=2) 2014-04-01 14:35:18,191.191 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Fragment size=4096 (log=2) 2014-04-01 14:35:18,191.191 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Stride=0 blocks, Stripe width=0 blocks 2014-04-01 14:35:18,191.191 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 30531584 inodes, 122096646 blocks 2014-04-01 14:35:18,191.191 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 6104832 blocks (5.00%) reserved for the super user 2014-04-01 14:35:18,191.191 INFO:teuthology.orchestra.run.out:[10.214.133.37]: First data block=0 2014-04-01 14:35:18,191.191 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Maximum filesystem blocks=4294967296 2014-04-01 14:35:18,191.191 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 3727 block groups 2014-04-01 14:35:18,191.191 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 32768 blocks per group, 32768 fragments per group 2014-04-01 14:35:18,191.191 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 8192 inodes per group 2014-04-01 14:35:18,191.191 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Superblock backups stored on blocks: 2014-04-01 14:35:18,191.191 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 2014-04-01 14:35:18,192.192 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 2014-04-01 14:35:18,192.192 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 102400000 2014-04-01 14:35:18,192.192 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 2014-04-01 14:35:19,406.406 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Allocating group tables: done 2014-04-01 14:35:21,904.904 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Writing inode tables: done 2014-04-01 14:35:23,832.832 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Creating journal (32768 blocks): done 2014-04-01 14:35:24,120.120 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Writing superblocks and filesystem accounting information: done 2014-04-01 14:35:26,177.177 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 2014-04-01 14:35:26,406.406 INFO:teuthology.task.ceph:mount /dev/sdd on ubuntu@plana79.front.sepia.ceph.com -o noatime,user_xattr 2014-04-01 14:35:26,406.406 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo mount -t ext4 -o noatime,user_xattr /dev/sdd /var/lib/ceph/osd/ceph-0' 2014-04-01 14:35:26,798.798 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo mkdir -p /var/lib/ceph/osd/ceph-1' 2014-04-01 14:35:27,263.263 INFO:teuthology.task.ceph:{} 2014-04-01 14:35:27,263.263 INFO:teuthology.task.ceph:1 2014-04-01 14:35:27,264.264 INFO:teuthology.task.ceph:['mkfs.ext4'] on /dev/sdb on ubuntu@plana79.front.sepia.ceph.com 2014-04-01 14:35:27,264.264 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'yes | sudo mkfs.ext4 /dev/sdb' 2014-04-01 14:35:27,766.766 INFO:teuthology.orchestra.run.err:[10.214.133.37]: mke2fs 1.42 (29-Nov-2011) 2014-04-01 14:35:27,789.789 INFO:teuthology.orchestra.run.out:[10.214.133.37]: /dev/sdb is entire device, not just one partition! 2014-04-01 14:35:27,789.789 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Proceed anyway? (y,n) Filesystem label= 2014-04-01 14:35:27,789.789 INFO:teuthology.orchestra.run.out:[10.214.133.37]: OS type: Linux 2014-04-01 14:35:27,789.789 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Block size=4096 (log=2) 2014-04-01 14:35:27,789.789 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Fragment size=4096 (log=2) 2014-04-01 14:35:27,790.790 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Stride=0 blocks, Stripe width=0 blocks 2014-04-01 14:35:27,790.790 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 30531584 inodes, 122096646 blocks 2014-04-01 14:35:27,790.790 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 6104832 blocks (5.00%) reserved for the super user 2014-04-01 14:35:27,790.790 INFO:teuthology.orchestra.run.out:[10.214.133.37]: First data block=0 2014-04-01 14:35:27,790.790 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Maximum filesystem blocks=4294967296 2014-04-01 14:35:27,790.790 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 3727 block groups 2014-04-01 14:35:27,790.790 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 32768 blocks per group, 32768 fragments per group 2014-04-01 14:35:27,790.790 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 8192 inodes per group 2014-04-01 14:35:27,790.790 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Superblock backups stored on blocks: 2014-04-01 14:35:27,790.790 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 2014-04-01 14:35:27,790.790 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 2014-04-01 14:35:27,791.791 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 102400000 2014-04-01 14:35:27,791.791 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 2014-04-01 14:35:28,799.799 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Allocating group tables: done 2014-04-01 14:35:30,797.797 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Writing inode tables: done 2014-04-01 14:35:32,917.917 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Creating journal (32768 blocks): done 2014-04-01 14:35:33,453.453 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Writing superblocks and filesystem accounting information: done 2014-04-01 14:35:35,228.228 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 2014-04-01 14:35:35,624.624 INFO:teuthology.task.ceph:mount /dev/sdb on ubuntu@plana79.front.sepia.ceph.com -o noatime,user_xattr 2014-04-01 14:35:35,625.625 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo mount -t ext4 -o noatime,user_xattr /dev/sdb /var/lib/ceph/osd/ceph-1' 2014-04-01 14:35:36,049.049 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo mkdir -p /var/lib/ceph/osd/ceph-2' 2014-04-01 14:35:36,635.635 INFO:teuthology.task.ceph:{} 2014-04-01 14:35:36,635.635 INFO:teuthology.task.ceph:2 2014-04-01 14:35:36,636.636 INFO:teuthology.task.ceph:['mkfs.ext4'] on /dev/sdc on ubuntu@plana79.front.sepia.ceph.com 2014-04-01 14:35:36,636.636 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'yes | sudo mkfs.ext4 /dev/sdc' 2014-04-01 14:35:37,253.253 INFO:teuthology.orchestra.run.err:[10.214.133.37]: mke2fs 1.42 (29-Nov-2011) 2014-04-01 14:35:37,254.254 INFO:teuthology.orchestra.run.out:[10.214.133.37]: /dev/sdc is entire device, not just one partition! 2014-04-01 14:35:37,254.254 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Proceed anyway? (y,n) Filesystem label= 2014-04-01 14:35:37,254.254 INFO:teuthology.orchestra.run.out:[10.214.133.37]: OS type: Linux 2014-04-01 14:35:37,255.255 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Block size=4096 (log=2) 2014-04-01 14:35:37,255.255 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Fragment size=4096 (log=2) 2014-04-01 14:35:37,255.255 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Stride=0 blocks, Stripe width=0 blocks 2014-04-01 14:35:37,256.256 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 30531584 inodes, 122096646 blocks 2014-04-01 14:35:37,256.256 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 6104832 blocks (5.00%) reserved for the super user 2014-04-01 14:35:37,256.256 INFO:teuthology.orchestra.run.out:[10.214.133.37]: First data block=0 2014-04-01 14:35:37,257.257 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Maximum filesystem blocks=4294967296 2014-04-01 14:35:37,257.257 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 3727 block groups 2014-04-01 14:35:37,257.257 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 32768 blocks per group, 32768 fragments per group 2014-04-01 14:35:37,257.257 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 8192 inodes per group 2014-04-01 14:35:37,258.258 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Superblock backups stored on blocks: 2014-04-01 14:35:37,258.258 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 2014-04-01 14:35:37,258.258 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 2014-04-01 14:35:37,259.259 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 102400000 2014-04-01 14:35:37,259.259 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 2014-04-01 14:35:38,318.318 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Allocating group tables: done 2014-04-01 14:35:40,245.245 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Writing inode tables: done 2014-04-01 14:35:42,426.426 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Creating journal (32768 blocks): done 2014-04-01 14:35:42,812.812 INFO:teuthology.orchestra.run.out:[10.214.133.37]: Writing superblocks and filesystem accounting information: done 2014-04-01 14:35:45,032.032 INFO:teuthology.orchestra.run.out:[10.214.133.37]: 2014-04-01 14:35:45,032.032 INFO:teuthology.task.ceph:mount /dev/sdc on ubuntu@plana79.front.sepia.ceph.com -o noatime,user_xattr 2014-04-01 14:35:45,032.032 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo mount -t ext4 -o noatime,user_xattr /dev/sdc /var/lib/ceph/osd/ceph-2' 2014-04-01 14:35:45,420.420 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/monmap' 2014-04-01 14:35:46,310.310 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 05:35:46.498969 7f884aa29780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 14:35:46,804.804 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 05:35:46.993307 7f884aa29780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 14:35:47,835.835 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 05:35:47.008287 7f884aa29780 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory 2014-04-01 14:35:47,839.839 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 05:35:47.505527 7f884aa29780 -1 created object store /var/lib/ceph/osd/ceph-0 journal /var/lib/ceph/osd/ceph-0/journal for osd.0 fsid 4fc4b977-b888-4422-ba3c-54f89f334abf 2014-04-01 14:35:47,840.840 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 05:35:47.505573 7f884aa29780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2014-04-01 14:35:47,840.840 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 05:35:47.505647 7f884aa29780 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2014-04-01 14:35:47,841.841 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/monmap' 2014-04-01 14:35:48,634.634 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 05:35:48.823503 7fa86c8db780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 14:35:49,106.106 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 05:35:49.292807 7fa86c8db780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 14:35:49,119.119 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 05:35:49.308993 7fa86c8db780 -1 filestore(/var/lib/ceph/osd/ceph-1) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory 2014-04-01 14:35:49,643.643 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 05:35:49.831029 7fa86c8db780 -1 created object store /var/lib/ceph/osd/ceph-1 journal /var/lib/ceph/osd/ceph-1/journal for osd.1 fsid 4fc4b977-b888-4422-ba3c-54f89f334abf 2014-04-01 14:35:49,643.643 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 05:35:49.831077 7fa86c8db780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2014-04-01 14:35:49,643.643 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 05:35:49.831152 7fa86c8db780 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2014-04-01 14:35:49,644.644 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/monmap' 2014-04-01 14:35:50,414.414 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 05:35:50.604130 7fcca0cba780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 14:35:50,876.876 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 05:35:51.065066 7fcca0cba780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 14:35:50,893.893 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 05:35:51.082924 7fcca0cba780 -1 filestore(/var/lib/ceph/osd/ceph-2) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory 2014-04-01 14:35:51,396.396 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 05:35:51.585412 7fcca0cba780 -1 created object store /var/lib/ceph/osd/ceph-2 journal /var/lib/ceph/osd/ceph-2/journal for osd.2 fsid 4fc4b977-b888-4422-ba3c-54f89f334abf 2014-04-01 14:35:51,397.397 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 05:35:51.585458 7fcca0cba780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2014-04-01 14:35:51,397.397 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 05:35:51.585531 7fcca0cba780 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2014-04-01 14:35:51,398.398 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo mkdir -p /var/lib/ceph/osd/ceph-3' 2014-04-01 14:35:51,726.726 INFO:teuthology.task.ceph:{} 2014-04-01 14:35:51,727.727 INFO:teuthology.task.ceph:3 2014-04-01 14:35:51,727.727 INFO:teuthology.task.ceph:['mkfs.ext4'] on /dev/sdd on ubuntu@plana78.front.sepia.ceph.com 2014-04-01 14:35:51,728.728 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'yes | sudo mkfs.ext4 /dev/sdd' 2014-04-01 14:35:52,251.251 INFO:teuthology.orchestra.run.err:[10.214.133.38]: mke2fs 1.42 (29-Nov-2011) 2014-04-01 14:35:52,269.269 INFO:teuthology.orchestra.run.out:[10.214.133.38]: /dev/sdd is entire device, not just one partition! 2014-04-01 14:35:52,270.270 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Proceed anyway? (y,n) Filesystem label= 2014-04-01 14:35:52,270.270 INFO:teuthology.orchestra.run.out:[10.214.133.38]: OS type: Linux 2014-04-01 14:35:52,270.270 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Block size=4096 (log=2) 2014-04-01 14:35:52,271.271 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Fragment size=4096 (log=2) 2014-04-01 14:35:52,271.271 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Stride=0 blocks, Stripe width=0 blocks 2014-04-01 14:35:52,271.271 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 30531584 inodes, 122096646 blocks 2014-04-01 14:35:52,271.271 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 6104832 blocks (5.00%) reserved for the super user 2014-04-01 14:35:52,271.271 INFO:teuthology.orchestra.run.out:[10.214.133.38]: First data block=0 2014-04-01 14:35:52,272.272 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Maximum filesystem blocks=4294967296 2014-04-01 14:35:52,272.272 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 3727 block groups 2014-04-01 14:35:52,272.272 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 32768 blocks per group, 32768 fragments per group 2014-04-01 14:35:52,272.272 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 8192 inodes per group 2014-04-01 14:35:52,273.273 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Superblock backups stored on blocks: 2014-04-01 14:35:52,273.273 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 2014-04-01 14:35:52,273.273 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 2014-04-01 14:35:52,273.273 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 102400000 2014-04-01 14:35:52,273.273 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 2014-04-01 14:35:53,462.462 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Allocating group tables: done 2014-04-01 14:35:55,330.330 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Writing inode tables: done 2014-04-01 14:35:57,536.536 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Creating journal (32768 blocks): done 2014-04-01 14:35:57,929.929 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Writing superblocks and filesystem accounting information: done 2014-04-01 14:35:59,957.957 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 2014-04-01 14:36:00,363.363 INFO:teuthology.task.ceph:mount /dev/sdd on ubuntu@plana78.front.sepia.ceph.com -o noatime,user_xattr 2014-04-01 14:36:00,364.364 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo mount -t ext4 -o noatime,user_xattr /dev/sdd /var/lib/ceph/osd/ceph-3' 2014-04-01 14:36:00,810.810 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo mkdir -p /var/lib/ceph/osd/ceph-4' 2014-04-01 14:36:01,321.321 INFO:teuthology.task.ceph:{} 2014-04-01 14:36:01,321.321 INFO:teuthology.task.ceph:4 2014-04-01 14:36:01,321.321 INFO:teuthology.task.ceph:['mkfs.ext4'] on /dev/sdb on ubuntu@plana78.front.sepia.ceph.com 2014-04-01 14:36:01,322.322 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'yes | sudo mkfs.ext4 /dev/sdb' 2014-04-01 14:36:01,920.920 INFO:teuthology.orchestra.run.err:[10.214.133.38]: mke2fs 1.42 (29-Nov-2011) 2014-04-01 14:36:01,935.935 INFO:teuthology.orchestra.run.out:[10.214.133.38]: /dev/sdb is entire device, not just one partition! 2014-04-01 14:36:01,935.935 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Proceed anyway? (y,n) Filesystem label= 2014-04-01 14:36:01,935.935 INFO:teuthology.orchestra.run.out:[10.214.133.38]: OS type: Linux 2014-04-01 14:36:01,935.935 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Block size=4096 (log=2) 2014-04-01 14:36:01,935.935 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Fragment size=4096 (log=2) 2014-04-01 14:36:01,935.935 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Stride=0 blocks, Stripe width=0 blocks 2014-04-01 14:36:01,935.935 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 30531584 inodes, 122096646 blocks 2014-04-01 14:36:01,935.935 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 6104832 blocks (5.00%) reserved for the super user 2014-04-01 14:36:01,935.935 INFO:teuthology.orchestra.run.out:[10.214.133.38]: First data block=0 2014-04-01 14:36:01,936.936 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Maximum filesystem blocks=4294967296 2014-04-01 14:36:01,936.936 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 3727 block groups 2014-04-01 14:36:01,936.936 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 32768 blocks per group, 32768 fragments per group 2014-04-01 14:36:01,936.936 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 8192 inodes per group 2014-04-01 14:36:01,936.936 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Superblock backups stored on blocks: 2014-04-01 14:36:01,936.936 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 2014-04-01 14:36:01,936.936 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 2014-04-01 14:36:01,936.936 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 102400000 2014-04-01 14:36:01,936.936 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 2014-04-01 14:36:02,866.866 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Allocating group tables: done 2014-04-01 14:36:04,734.734 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Writing inode tables: done 2014-04-01 14:36:07,123.123 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Creating journal (32768 blocks): done 2014-04-01 14:36:07,938.938 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Writing superblocks and filesystem accounting information: done 2014-04-01 14:36:09,900.900 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 2014-04-01 14:36:10,056.056 INFO:teuthology.task.ceph:mount /dev/sdb on ubuntu@plana78.front.sepia.ceph.com -o noatime,user_xattr 2014-04-01 14:36:10,056.056 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo mount -t ext4 -o noatime,user_xattr /dev/sdb /var/lib/ceph/osd/ceph-4' 2014-04-01 14:36:10,506.506 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo mkdir -p /var/lib/ceph/osd/ceph-5' 2014-04-01 14:36:11,075.075 INFO:teuthology.task.ceph:{} 2014-04-01 14:36:11,075.075 INFO:teuthology.task.ceph:5 2014-04-01 14:36:11,076.076 INFO:teuthology.task.ceph:['mkfs.ext4'] on /dev/sdc on ubuntu@plana78.front.sepia.ceph.com 2014-04-01 14:36:11,076.076 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'yes | sudo mkfs.ext4 /dev/sdc' 2014-04-01 14:36:11,673.673 INFO:teuthology.orchestra.run.err:[10.214.133.38]: mke2fs 1.42 (29-Nov-2011) 2014-04-01 14:36:11,688.688 INFO:teuthology.orchestra.run.out:[10.214.133.38]: /dev/sdc is entire device, not just one partition! 2014-04-01 14:36:11,688.688 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Proceed anyway? (y,n) Filesystem label= 2014-04-01 14:36:11,688.688 INFO:teuthology.orchestra.run.out:[10.214.133.38]: OS type: Linux 2014-04-01 14:36:11,688.688 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Block size=4096 (log=2) 2014-04-01 14:36:11,688.688 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Fragment size=4096 (log=2) 2014-04-01 14:36:11,688.688 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Stride=0 blocks, Stripe width=0 blocks 2014-04-01 14:36:11,688.688 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 30531584 inodes, 122096646 blocks 2014-04-01 14:36:11,688.688 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 6104832 blocks (5.00%) reserved for the super user 2014-04-01 14:36:11,688.688 INFO:teuthology.orchestra.run.out:[10.214.133.38]: First data block=0 2014-04-01 14:36:11,688.688 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Maximum filesystem blocks=4294967296 2014-04-01 14:36:11,689.689 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 3727 block groups 2014-04-01 14:36:11,689.689 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 32768 blocks per group, 32768 fragments per group 2014-04-01 14:36:11,689.689 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 8192 inodes per group 2014-04-01 14:36:11,689.689 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Superblock backups stored on blocks: 2014-04-01 14:36:11,690.690 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 2014-04-01 14:36:11,690.690 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 2014-04-01 14:36:11,690.690 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 102400000 2014-04-01 14:36:11,690.690 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 2014-04-01 14:36:12,604.604 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Allocating group tables: done 2014-04-01 14:36:14,790.790 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Writing inode tables: done 2014-04-01 14:36:16,810.810 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Creating journal (32768 blocks): done 2014-04-01 14:36:17,236.236 INFO:teuthology.orchestra.run.out:[10.214.133.38]: Writing superblocks and filesystem accounting information: done 2014-04-01 14:36:19,128.128 INFO:teuthology.orchestra.run.out:[10.214.133.38]: 2014-04-01 14:36:19,535.535 INFO:teuthology.task.ceph:mount /dev/sdc on ubuntu@plana78.front.sepia.ceph.com -o noatime,user_xattr 2014-04-01 14:36:19,536.536 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo mount -t ext4 -o noatime,user_xattr /dev/sdc /var/lib/ceph/osd/ceph-5' 2014-04-01 14:36:19,910.910 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/monmap' 2014-04-01 14:36:20,778.778 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 05:36:20.695285 7f169c2dd780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 14:36:21,214.214 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 05:36:21.130980 7f169c2dd780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 14:36:21,230.230 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 05:36:21.148796 7f169c2dd780 -1 filestore(/var/lib/ceph/osd/ceph-3) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory 2014-04-01 14:36:21,737.737 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 05:36:21.651228 7f169c2dd780 -1 created object store /var/lib/ceph/osd/ceph-3 journal /var/lib/ceph/osd/ceph-3/journal for osd.3 fsid 4fc4b977-b888-4422-ba3c-54f89f334abf 2014-04-01 14:36:21,737.737 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 05:36:21.651274 7f169c2dd780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2014-04-01 14:36:21,737.737 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 05:36:21.651348 7f169c2dd780 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2014-04-01 14:36:21,738.738 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/monmap' 2014-04-01 14:36:22,619.619 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 05:36:22.534578 7f5d2400a780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 14:36:23,046.046 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 05:36:22.962027 7f5d2400a780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 14:36:23,062.062 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 05:36:22.980046 7f5d2400a780 -1 filestore(/var/lib/ceph/osd/ceph-4) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory 2014-04-01 14:36:23,551.551 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 05:36:23.465859 7f5d2400a780 -1 created object store /var/lib/ceph/osd/ceph-4 journal /var/lib/ceph/osd/ceph-4/journal for osd.4 fsid 4fc4b977-b888-4422-ba3c-54f89f334abf 2014-04-01 14:36:23,551.551 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 05:36:23.465908 7f5d2400a780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2014-04-01 14:36:23,552.552 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 05:36:23.465983 7f5d2400a780 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2014-04-01 14:36:23,553.553 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/monmap' 2014-04-01 14:36:24,355.355 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 05:36:24.271487 7f8ea6193780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 14:36:24,800.800 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 05:36:24.715590 7f8ea6193780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 14:36:24,815.815 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 05:36:24.732481 7f8ea6193780 -1 filestore(/var/lib/ceph/osd/ceph-5) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory 2014-04-01 14:36:25,321.321 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 05:36:25.235773 7f8ea6193780 -1 created object store /var/lib/ceph/osd/ceph-5 journal /var/lib/ceph/osd/ceph-5/journal for osd.5 fsid 4fc4b977-b888-4422-ba3c-54f89f334abf 2014-04-01 14:36:25,321.321 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 05:36:25.235819 7f8ea6193780 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2014-04-01 14:36:25,322.322 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2014-04-01 05:36:25.235891 7f8ea6193780 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2014-04-01 14:36:25,322.322 INFO:teuthology.task.ceph:Reading keys from all nodes... 2014-04-01 14:36:25,323.323 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo cat -- /var/lib/ceph/osd/ceph-0/keyring' 2014-04-01 14:36:25,718.718 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo cat -- /var/lib/ceph/osd/ceph-1/keyring' 2014-04-01 14:36:26,333.333 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo cat -- /var/lib/ceph/osd/ceph-2/keyring' 2014-04-01 14:36:26,929.929 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo cat -- /var/lib/ceph/mds/ceph-a/keyring' 2014-04-01 14:36:27,326.326 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo cat -- /var/lib/ceph/osd/ceph-3/keyring' 2014-04-01 14:36:27,911.911 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo cat -- /var/lib/ceph/osd/ceph-4/keyring' 2014-04-01 14:36:28,411.411 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo cat -- /var/lib/ceph/osd/ceph-5/keyring' 2014-04-01 14:36:28,907.907 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'cat -- /etc/ceph/ceph.client.0.keyring' 2014-04-01 14:36:29,379.379 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'cat -- /etc/ceph/ceph.client.1.keyring' 2014-04-01 14:36:29,850.850 INFO:teuthology.task.ceph:Adding keys to all mons... 2014-04-01 14:36:29,850.850 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo tee -a /etc/ceph/ceph.keyring' 2014-04-01 14:36:30,341.341 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo tee -a /etc/ceph/ceph.keyring' 2014-04-01 14:36:30,971.971 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap osd 'allow *' --cap mon 'allow *'" 2014-04-01 14:36:31,506.506 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap osd 'allow *' --cap mon 'allow *'" 2014-04-01 14:36:31,851.851 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap osd 'allow *' --cap mon 'allow *'" 2014-04-01 14:36:32,172.172 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap osd 'allow *' --cap mon 'allow *'" 2014-04-01 14:36:32,517.517 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap osd 'allow *' --cap mon 'allow *'" 2014-04-01 14:36:32,837.837 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap osd 'allow *' --cap mon 'allow *'" 2014-04-01 14:36:33,181.181 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mds allow --cap osd 'allow *' --cap mon 'allow *'" 2014-04-01 14:36:33,502.502 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mds allow --cap osd 'allow *' --cap mon 'allow *'" 2014-04-01 14:36:33,847.847 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap osd 'allow *' --cap mon 'allow *'" 2014-04-01 14:36:34,167.167 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap osd 'allow *' --cap mon 'allow *'" 2014-04-01 14:36:34,512.512 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap osd 'allow *' --cap mon 'allow *'" 2014-04-01 14:36:34,832.832 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap osd 'allow *' --cap mon 'allow *'" 2014-04-01 14:36:35,176.176 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap osd 'allow *' --cap mon 'allow *'" 2014-04-01 14:36:35,499.499 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap osd 'allow *' --cap mon 'allow *'" 2014-04-01 14:36:35,839.839 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mon 'allow rw'" 2014-04-01 14:36:36,164.164 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mon 'allow rw'" 2014-04-01 14:36:36,510.510 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mds allow --cap osd 'allow rwx' --cap mon 'allow rw'" 2014-04-01 14:36:36,871.871 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mds allow --cap osd 'allow rwx' --cap mon 'allow rw'" 2014-04-01 14:36:37,265.265 INFO:teuthology.task.ceph:Running mkfs on mon nodes... 2014-04-01 14:36:37,265.265 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo mkdir -p /var/lib/ceph/mon/ceph-a' 2014-04-01 14:36:37,768.768 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --mkfs -i a --monmap=/home/ubuntu/cephtest/monmap --osdmap=/home/ubuntu/cephtest/osdmap --keyring=/etc/ceph/ceph.keyring' 2014-04-01 14:36:38,331.331 INFO:teuthology.orchestra.run.out:[10.214.133.37]: ceph-mon: set fsid to 4fc4b977-b888-4422-ba3c-54f89f334abf 2014-04-01 14:36:38,469.469 INFO:teuthology.orchestra.run.out:[10.214.133.37]: ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2014-04-01 14:36:38,473.473 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo mkdir -p /var/lib/ceph/mon/ceph-c' 2014-04-01 14:36:39,012.012 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --mkfs -i c --monmap=/home/ubuntu/cephtest/monmap --osdmap=/home/ubuntu/cephtest/osdmap --keyring=/etc/ceph/ceph.keyring' 2014-04-01 14:36:39,570.570 INFO:teuthology.orchestra.run.out:[10.214.133.37]: ceph-mon: set fsid to 4fc4b977-b888-4422-ba3c-54f89f334abf 2014-04-01 14:36:39,762.762 INFO:teuthology.orchestra.run.out:[10.214.133.37]: ceph-mon: created monfs at /var/lib/ceph/mon/ceph-c for mon.c 2014-04-01 14:36:39,763.763 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo mkdir -p /var/lib/ceph/mon/ceph-b' 2014-04-01 14:36:40,100.100 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --mkfs -i b --monmap=/home/ubuntu/cephtest/monmap --osdmap=/home/ubuntu/cephtest/osdmap --keyring=/etc/ceph/ceph.keyring' 2014-04-01 14:36:40,622.622 INFO:teuthology.orchestra.run.out:[10.214.133.38]: ceph-mon: set fsid to 4fc4b977-b888-4422-ba3c-54f89f334abf 2014-04-01 14:36:40,765.765 INFO:teuthology.orchestra.run.out:[10.214.133.38]: ceph-mon: created monfs at /var/lib/ceph/mon/ceph-b for mon.b 2014-04-01 14:36:40,766.766 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'rm -- /home/ubuntu/cephtest/monmap /home/ubuntu/cephtest/osdmap' 2014-04-01 14:36:41,306.306 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'rm -- /home/ubuntu/cephtest/monmap /home/ubuntu/cephtest/osdmap' 2014-04-01 14:36:41,694.694 INFO:teuthology.task.ceph:Starting mon daemons... 2014-04-01 14:36:41,695.695 INFO:teuthology.task.ceph.mon.a:Restarting 2014-04-01 14:36:41,695.695 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f -i a' 2014-04-01 14:36:42,231.231 INFO:teuthology.task.ceph.mon.a:Started 2014-04-01 14:36:42,232.232 INFO:teuthology.task.ceph.mon.c:Restarting 2014-04-01 14:36:42,232.232 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f -i c' 2014-04-01 14:36:42,595.595 INFO:teuthology.task.ceph.mon.c:Started 2014-04-01 14:36:42,595.595 INFO:teuthology.task.ceph.mon.b:Restarting 2014-04-01 14:36:42,596.596 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f -i b' 2014-04-01 14:36:42,917.917 INFO:teuthology.task.ceph.mon.b:Started 2014-04-01 14:36:42,917.917 INFO:teuthology.task.ceph:Starting osd daemons... 2014-04-01 14:36:42,918.918 INFO:teuthology.task.ceph.osd.0:Restarting 2014-04-01 14:36:42,918.918 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 0' 2014-04-01 14:36:43,245.245 INFO:teuthology.task.ceph.osd.0:Started 2014-04-01 14:36:43,246.246 INFO:teuthology.task.ceph.osd.1:Restarting 2014-04-01 14:36:43,247.247 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 1' 2014-04-01 14:36:43,306.306 INFO:teuthology.task.ceph.osd.0.out:[10.214.133.37]: starting osd.0 at :/0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal 2014-04-01 14:36:43,578.578 INFO:teuthology.task.ceph.osd.1:Started 2014-04-01 14:36:43,578.578 INFO:teuthology.task.ceph.osd.2:Restarting 2014-04-01 14:36:43,579.579 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 2' 2014-04-01 14:36:43,623.623 INFO:teuthology.task.ceph.osd.1.out:[10.214.133.37]: starting osd.1 at :/0 osd_data /var/lib/ceph/osd/ceph-1 /var/lib/ceph/osd/ceph-1/journal 2014-04-01 14:36:43,665.665 INFO:teuthology.task.ceph.osd.0.err:[10.214.133.37]: 2014-04-01 05:36:43.877120 7f49b4d77780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 14:36:43,945.945 INFO:teuthology.task.ceph.osd.2:Started 2014-04-01 14:36:43,945.945 INFO:teuthology.task.ceph.osd.3:Restarting 2014-04-01 14:36:43,946.946 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 3' 2014-04-01 14:36:43,976.976 INFO:teuthology.task.ceph.osd.1.err:[10.214.133.37]: 2014-04-01 05:36:44.183640 7f330528d780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 14:36:43,988.988 INFO:teuthology.task.ceph.osd.2.out:[10.214.133.37]: starting osd.2 at :/0 osd_data /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal 2014-04-01 14:36:44,337.337 INFO:teuthology.task.ceph.osd.3:Started 2014-04-01 14:36:44,337.337 INFO:teuthology.task.ceph.osd.4:Restarting 2014-04-01 14:36:44,337.337 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 4' 2014-04-01 14:36:44,351.351 INFO:teuthology.task.ceph.osd.2.err:[10.214.133.37]: 2014-04-01 05:36:44.563460 7f32455ab780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 14:36:44,387.387 INFO:teuthology.task.ceph.osd.3.out:[10.214.133.38]: starting osd.3 at :/0 osd_data /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal 2014-04-01 14:36:44,660.660 INFO:teuthology.task.ceph.osd.4:Started 2014-04-01 14:36:44,661.661 INFO:teuthology.task.ceph.osd.5:Restarting 2014-04-01 14:36:44,661.661 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 5' 2014-04-01 14:36:44,712.712 INFO:teuthology.task.ceph.osd.4.out:[10.214.133.38]: starting osd.4 at :/0 osd_data /var/lib/ceph/osd/ceph-4 /var/lib/ceph/osd/ceph-4/journal 2014-04-01 14:36:44,762.762 INFO:teuthology.task.ceph.osd.3.err:[10.214.133.38]: 2014-04-01 05:36:44.678756 7f269d33c780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 14:36:44,980.980 INFO:teuthology.task.ceph.osd.5:Started 2014-04-01 14:36:44,981.981 INFO:teuthology.task.ceph:Starting mds daemons... 2014-04-01 14:36:44,981.981 INFO:teuthology.task.ceph.mds.a:Restarting 2014-04-01 14:36:44,982.982 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f -i a' 2014-04-01 14:36:45,033.033 INFO:teuthology.task.ceph.osd.5.out:[10.214.133.38]: starting osd.5 at :/0 osd_data /var/lib/ceph/osd/ceph-5 /var/lib/ceph/osd/ceph-5/journal 2014-04-01 14:36:45,049.049 INFO:teuthology.task.ceph.osd.4.err:[10.214.133.38]: 2014-04-01 05:36:44.959648 7ffdd3008780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 14:36:45,302.302 INFO:teuthology.task.ceph.mds.a:Started 2014-04-01 14:36:45,302.302 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph mds set_max_mds 1' 2014-04-01 14:36:45,363.363 INFO:teuthology.task.ceph.mds.a.out:[10.214.133.38]: starting mds.a at :/0 2014-04-01 14:36:45,364.364 INFO:teuthology.task.ceph.osd.5.err:[10.214.133.38]: 2014-04-01 05:36:45.279860 7f91278fb780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 14:36:53,349.349 INFO:teuthology.orchestra.run.err:[10.214.133.37]: max_mds = 1 2014-04-01 14:36:53,361.361 INFO:teuthology.task.ceph:Waiting until ceph is healthy... 2014-04-01 14:36:53,361.361 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd dump --format=json' 2014-04-01 14:36:54,064.064 DEBUG:teuthology.misc:5 of 6 OSDs are up 2014-04-01 14:36:55,062.062 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd dump --format=json' 2014-04-01 14:36:55,626.626 DEBUG:teuthology.misc:5 of 6 OSDs are up 2014-04-01 14:36:56,624.624 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd dump --format=json' 2014-04-01 14:36:57,177.177 DEBUG:teuthology.misc:6 of 6 OSDs are up 2014-04-01 14:37:03,174.174 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health' 2014-04-01 14:37:03,726.726 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN 38 pgs peering; 38 pgs stuck inactive; 38 pgs stuck unclean 2014-04-01 14:37:10,726.726 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health' 2014-04-01 14:37:11,274.274 DEBUG:teuthology.misc:Ceph health: HEALTH_OK 2014-04-01 14:37:11,275.275 INFO:teuthology.run_tasks:Running task thrashosds... 2014-04-01 14:37:11,298.298 INFO:teuthology.task.thrashosds:Beginning thrashosds... 2014-04-01 14:37:11,298.298 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd dump --format=json' 2014-04-01 14:37:11,991.991 INFO:teuthology.task.thrashosds.ceph_manager:[{u'cache_target_full_ratio_micro': 0, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'pg_placement_num': 24, u'quota_max_bytes': 0, u'erasure_code_profile': u'', u'size': 2, u'snap_seq': 0, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'target_max_objects': 0, u'pg_num': 24, u'type': 1, u'crush_ruleset': 0, u'pool_name': u'data', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'cache_mode': u'none', u'min_size': 1, u'crash_replay_interval': 45, u'object_hash': 2, u'write_tier': -1, u'cache_target_dirty_ratio_micro': 0, u'pool': 0, u'removed_snaps': u'[]', u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'pool_snaps': [], u'quota_max_objects': 0, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'snap_epoch': 0, u'last_change': u'1', u'read_tier': -1}, {u'cache_target_full_ratio_micro': 0, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'pg_placement_num': 24, u'quota_max_bytes': 0, u'erasure_code_profile': u'', u'size': 2, u'snap_seq': 0, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'target_max_objects': 0, u'pg_num': 24, u'type': 1, u'crush_ruleset': 0, u'pool_name': u'metadata', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'cache_mode': u'none', u'min_size': 1, u'crash_replay_interval': 0, u'object_hash': 2, u'write_tier': -1, u'cache_target_dirty_ratio_micro': 0, u'pool': 1, u'removed_snaps': u'[]', u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'pool_snaps': [], u'quota_max_objects': 0, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'snap_epoch': 0, u'last_change': u'1', u'read_tier': -1}, {u'cache_target_full_ratio_micro': 0, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'pg_placement_num': 24, u'quota_max_bytes': 0, u'erasure_code_profile': u'', u'size': 2, u'snap_seq': 0, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'target_max_objects': 0, u'pg_num': 24, u'type': 1, u'crush_ruleset': 0, u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'cache_mode': u'none', u'min_size': 1, u'crash_replay_interval': 0, u'object_hash': 2, u'write_tier': -1, u'cache_target_dirty_ratio_micro': 0, u'pool': 2, u'removed_snaps': u'[]', u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'pool_snaps': [], u'quota_max_objects': 0, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'snap_epoch': 0, u'last_change': u'1', u'read_tier': -1}] 2014-04-01 14:37:11,992.992 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get data pg_num' 2014-04-01 14:37:12,683.683 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get metadata pg_num' 2014-04-01 14:37:13,495.495 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get rbd pg_num' 2014-04-01 14:37:14,183.183 INFO:teuthology.task.thrashosds.ceph_manager:waiting for clean 2014-04-01 14:37:14,184.184 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:37:15,092.092 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:37:15,562.562 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:37:16,345.345 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:37:16,660.660 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:37:17,439.439 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.160844', u'skew': u'-0.277086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.160744', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': []}]}, u'overall_status': u'HEALTH_OK', u'summary': []}, u'pgmap': {u'bytes_total': 2952748449792, u'num_pgs': 72, u'data_bytes': 1884, u'bytes_used': 1071902720, u'op_per_sec': 3, u'version': 14, u'pgs_by_state': [{u'count': 72, u'state_name': u'active+clean'}], u'write_bytes_sec': 1289, u'bytes_avail': 2801543532544}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 7, u'num_in_osds': 6}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:37:17,440.440 INFO:teuthology.task.thrashosds.ceph_manager:clean! 2014-04-01 14:37:17,440.440 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd dump' 2014-04-01 14:37:18,189.189 INFO:teuthology.task.thrashosds.ceph_manager:['osd.0 up in weight 1 up_from 5 up_thru 5 down_at 0 last_clean_interval [0,0) 10.214.133.37:6800/4405 10.214.133.37:6801/4405 10.214.133.37:6802/4405 10.214.133.37:6803/4405 exists,up 1137bb41-c4a5-4adb-876f-db2d79102b65', 'osd.1 up in weight 1 up_from 2 up_thru 6 down_at 0 last_clean_interval [0,0) 10.214.133.37:6805/4412 10.214.133.37:6806/4412 10.214.133.37:6807/4412 10.214.133.37:6808/4412 exists,up 5371287f-e9e4-4cbd-94c6-5c5f23fb6a14', 'osd.2 up in weight 1 up_from 3 up_thru 5 down_at 0 last_clean_interval [0,0) 10.214.133.37:6810/4439 10.214.133.37:6811/4439 10.214.133.37:6812/4439 10.214.133.37:6813/4439 exists,up 1f91513f-cbd4-4a9c-828e-d5406fb58131', 'osd.3 up in weight 1 up_from 3 up_thru 5 down_at 0 last_clean_interval [0,0) 10.214.133.38:6800/4373 10.214.133.38:6801/4373 10.214.133.38:6802/4373 10.214.133.38:6803/4373 exists,up d27b3cab-5475-40bb-8a03-0756a73096f3', 'osd.4 up in weight 1 up_from 3 up_thru 6 down_at 0 last_clean_interval [0,0) 10.214.133.38:6805/4380 10.214.133.38:6806/4380 10.214.133.38:6807/4380 10.214.133.38:6808/4380 exists,up 22e3726b-b862-4d09-b283-9f3ba929eab7', 'osd.5 up in weight 1 up_from 3 up_thru 5 down_at 0 last_clean_interval [0,0) 10.214.133.38:6810/4408 10.214.133.38:6811/4408 10.214.133.38:6812/4408 10.214.133.38:6813/4408 exists,up 4b9e0851-9922-4298-98dc-976125b37535'] 2014-04-01 14:37:18,190.190 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph -- tell 'mon.*' injectargs '--mon-osd-down-out-interval 0'" 2014-04-01 14:37:18,870.870 INFO:teuthology.orchestra.run.err:[10.214.133.37]: mon.a: injectargs:mon_osd_down_out_interval = '0' 2014-04-01 14:37:18,977.977 INFO:teuthology.orchestra.run.err:[10.214.133.37]: mon.b: injectargs:mon_osd_down_out_interval = '0' 2014-04-01 14:37:19,082.082 INFO:teuthology.orchestra.run.err:[10.214.133.37]: mon.c: injectargs:mon_osd_down_out_interval = '0' 2014-04-01 14:37:19,093.093 INFO:teuthology.run_tasks:Running task radosbench... 2014-04-01 14:37:19,097.097 INFO:teuthology.task.radosbench:Beginning radosbench... 2014-04-01 14:37:19,097.097 INFO:teuthology.task.thrashosds.ceph_manager:creating pool_name unique_pool_0 2014-04-01 14:37:19,097.097 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd erasure-code-profile set teuthologyprofile ruleset-failure-domain=osd m=1 k=2' 2014-04-01 14:37:19,098.098 INFO:teuthology.task.thrashosds.thrasher:starting do_thrash 2014-04-01 14:37:19,098.098 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-04-01 14:37:19,098.098 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:37:20,051.051 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool create unique_pool_0 16 16 erasure teuthologyprofile' 2014-04-01 14:37:22,458.458 INFO:teuthology.orchestra.run.err:[10.214.133.37]: pool 'unique_pool_0' created 2014-04-01 14:37:22,467.467 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: "/bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --name client.0 -p unique_pool_0 bench 1800 write'" 2014-04-01 14:37:22,467.467 INFO:teuthology.task.thrashosds.thrasher:Growing pool rbd 2014-04-01 14:37:22,467.467 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:37:23,073.073 DEBUG:teuthology.run_tasks:Unwinding manager radosbench 2014-04-01 14:37:23,073.073 INFO:teuthology.task.radosbench:joining radosbench (timing out after 9000s) 2014-04-01 14:37:23,100.100 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: Maintaining 16 concurrent writes of 4194304 bytes for up to 1800 seconds or 0 objects 2014-04-01 14:37:23,146.146 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: Object prefix: benchmark_data_plana79_5437 2014-04-01 14:37:23,146.146 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:37:23,164.164 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 0 0 0 0 0 0 - 0 2014-04-01 14:37:23,299.299 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:37:24,145.145 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1 16 16 0 0 0 - 0 2014-04-01 14:37:25,146.146 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2 16 38 22 43.9909 44 0.220457 1.0712 2014-04-01 14:37:26,148.148 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 3 16 63 47 62.6547 100 0.910942 0.901007 2014-04-01 14:37:27,143.143 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 4 16 79 63 62.9879 64 0.370548 0.804046 2014-04-01 14:37:28,144.144 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 5 16 79 63 50.3905 0 - 0.804046 2014-04-01 14:37:28,758.758 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-04-01 14:37:28,759.759 INFO:teuthology.task.thrashosds.ceph_manager:waiting for recovery to complete 2014-04-01 14:37:28,759.759 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:37:29,143.143 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 6 16 79 63 41.9919 0 - 0.804046 2014-04-01 14:37:29,418.418 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:37:30,154.154 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:37:30,154.154 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 7 16 79 63 35.9933 0 - 0.804046 2014-04-01 14:37:31,045.045 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:37:31,396.396 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 8 16 79 63 31.4942 0 - 0.804046 2014-04-01 14:37:31,630.630 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:37:32,143.143 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 9 16 79 63 27.9947 0 - 0.804046 2014-04-01 14:37:32,469.469 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.160844', u'skew': u'-0.277086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.160744', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': []}]}, u'overall_status': u'HEALTH_OK', u'summary': []}, u'pgmap': {u'bytes_total': 2952748449792, u'num_pgs': 88, u'data_bytes': 272631644, u'bytes_used': 1334517760, u'op_per_sec': 46, u'version': 21, u'pgs_by_state': [{u'count': 88, u'state_name': u'active+clean'}], u'write_bytes_sec': 193207163, u'bytes_avail': 2801280917504}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 11, u'num_in_osds': 6}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:37:32,470.470 INFO:teuthology.task.thrashosds.ceph_manager:recovered! 2014-04-01 14:37:32,470.470 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:37:32,470.470 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool rbd 2014-04-01 14:37:32,471.471 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:37:33,143.143 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 10 16 79 63 25.1954 0 - 0.804046 2014-04-01 14:37:33,260.260 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:37:33,789.789 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set rbd pgp_num 24' 2014-04-01 14:37:34,180.180 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 11 16 79 63 22.9049 0 - 0.804046 2014-04-01 14:37:34,907.907 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 2 pgp_num to 24 2014-04-01 14:37:35,143.143 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 12 16 79 63 20.9962 0 - 0.804046 2014-04-01 14:37:36,144.144 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 13 16 89 73 22.4576 4.44444 9.93861 2.03512 2014-04-01 14:37:37,159.159 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 14 16 117 101 28.8521 112 0.765679 2.16161 2014-04-01 14:37:37,774.774 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 05:37:38.004339 7fe3a3add700 0 -- 10.214.133.37:0/1005437 >> 10.214.133.38:6805/4380 pipe(0x1576750 sd=9 :46821 s=2 pgs=5 cs=1 l=1 c=0x15769b0).injecting socket failure 2014-04-01 14:37:38,191.191 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 15 16 143 127 33.8608 104 0.16111 1.83677 2014-04-01 14:37:38,596.596 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 05:37:38.837447 7fe3a3bde700 0 -- 10.214.133.37:0/1005437 >> 10.214.133.38:6800/4373 pipe(0x15734c0 sd=8 :46118 s=2 pgs=5 cs=1 l=1 c=0x1573720).injecting socket failure 2014-04-01 14:37:39,206.206 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 16 16 169 153 38.2434 104 0.541009 1.64327 2014-04-01 14:37:39,906.906 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-04-01 14:37:39,906.906 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:37:39,907.907 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool metadata 2014-04-01 14:37:39,907.907 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:37:40,141.141 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 17 16 192 176 41.4047 92 0.408915 1.49673 2014-04-01 14:37:40,885.885 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:37:41,198.198 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 18 16 220 204 45.3257 112 0.592038 1.37693 2014-04-01 14:37:41,362.362 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set metadata pgp_num 24' 2014-04-01 14:37:42,144.144 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 19 16 256 240 50.5179 144 0.487524 1.24819 2014-04-01 14:37:42,521.521 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 1 pgp_num to 24 2014-04-01 14:37:43,146.146 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:37:43.384882min lat: 0.125361 max lat: 9.93861 avg lat: 1.21502 2014-04-01 14:37:43,146.146 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:37:43,147.147 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 20 16 276 260 51.9914 80 0.475816 1.21502 2014-04-01 14:37:44,147.147 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 21 16 307 291 55.4194 124 0.209032 1.14025 2014-04-01 14:37:45,143.143 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 22 16 333 317 57.6269 104 0.719354 1.09354 2014-04-01 14:37:46,151.151 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 23 16 343 327 56.8602 40 0.684969 1.07752 2014-04-01 14:37:47,144.144 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 24 16 343 327 54.491 0 - 1.07752 2014-04-01 14:37:47,529.529 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-04-01 14:37:47,530.530 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:37:47,530.530 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool metadata 2014-04-01 14:37:47,531.531 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:37:48,265.265 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:37:48,451.451 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 25 16 343 327 52.3113 0 - 1.07752 2014-04-01 14:37:48,779.779 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set metadata pgp_num 24' 2014-04-01 14:37:49,144.144 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 26 16 343 327 50.2993 0 - 1.07752 2014-04-01 14:37:50,154.154 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 27 16 348 332 49.1771 5 5.2945 1.14074 2014-04-01 14:37:50,332.332 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 1 pgp_num to 24 2014-04-01 14:37:51,145.145 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 28 16 374 358 51.1345 104 0.512368 1.22986 2014-04-01 14:37:52,149.149 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 29 16 398 382 52.6811 96 0.267373 1.19564 2014-04-01 14:37:53,145.145 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 30 16 433 417 55.591 140 0.150445 1.13988 2014-04-01 14:37:54,142.142 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 31 16 452 436 56.2491 76 0.557883 1.11786 2014-04-01 14:37:55,143.143 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 32 16 452 436 54.4914 0 - 1.11786 2014-04-01 14:37:55,330.330 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-04-01 14:37:55,331.331 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:37:55,331.331 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 2 2014-04-01 14:37:55,331.331 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 14:37:55,332.332 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 14:37:55,332.332 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3' 2014-04-01 14:37:56,143.143 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 33 16 452 436 52.84 0 - 1.11786 2014-04-01 14:37:57,143.143 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 34 16 452 436 51.2859 0 - 1.11786 2014-04-01 14:37:58,152.152 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 35 16 452 436 49.8206 0 - 1.11786 2014-04-01 14:37:59,146.146 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 36 16 473 457 50.7697 16.8 0.467998 1.25008 2014-04-01 14:38:00,149.149 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 37 16 506 490 52.9646 132 0.313314 1.20279 2014-04-01 14:38:00,762.762 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-04-01 14:38:00,763.763 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:38:00,763.763 INFO:teuthology.task.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.963696 2014-04-01 14:38:00,764.764 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 5 0.963695840769' 2014-04-01 14:38:01,153.153 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 38 16 528 512 53.8863 88 0.416547 1.18069 2014-04-01 14:38:01,696.696 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set osd.5 primary-affinity to 0.963696 (8631562) 2014-04-01 14:38:02,149.149 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 39 16 555 539 55.2735 108 0.732869 1.15107 2014-04-01 14:38:03,149.149 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:38:03.387750min lat: 0.121441 max lat: 9.93861 avg lat: 1.12925 2014-04-01 14:38:03,149.149 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:38:03,150.150 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 40 16 580 564 56.3913 100 0.555246 1.12925 2014-04-01 14:38:04,144.144 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 41 16 604 588 57.357 96 0.133893 1.10566 2014-04-01 14:38:05,145.145 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 42 16 617 601 57.2293 52 0.13427 1.09664 2014-04-01 14:38:06,147.147 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 43 16 623 607 56.4564 24 0.443907 1.09138 2014-04-01 14:38:06,707.707 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-04-01 14:38:06,708.708 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:38:06,708.708 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 0 2014-04-01 14:38:06,708.708 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 14:38:06,709.709 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 14:38:06,709.709 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3' 2014-04-01 14:38:07,145.145 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 44 16 623 607 55.1733 0 - 1.09138 2014-04-01 14:38:08,146.146 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 45 16 642 626 55.6359 38 0.442508 1.1437 2014-04-01 14:38:09,151.151 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 46 16 674 658 57.2086 128 0.384482 1.11464 2014-04-01 14:38:10,148.148 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 47 16 696 680 57.8635 88 0.211028 1.09966 2014-04-01 14:38:11,149.149 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 48 16 722 706 58.8244 104 0.235555 1.07966 2014-04-01 14:38:12,134.134 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-04-01 14:38:12,135.135 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:38:12,136.136 INFO:teuthology.task.thrashosds.thrasher:Growing pool data 2014-04-01 14:38:12,136.136 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:38:12,150.150 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 49 16 752 736 60.0725 120 0.413116 1.06053 2014-04-01 14:38:12,898.898 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:38:13,148.148 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 50 16 774 758 60.6308 88 1.05101 1.05017 2014-04-01 14:38:13,249.249 INFO:teuthology.task.thrashosds.ceph_manager:increase pool size by 10 2014-04-01 14:38:13,249.249 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set data pg_num 34' 2014-04-01 14:38:14,148.148 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 51 16 800 784 61.4809 104 0.715501 1.03385 2014-04-01 14:38:14,636.636 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 0 pg_num to 34 2014-04-01 14:38:15,149.149 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 52 16 834 818 62.9135 136 0.749181 1.01207 2014-04-01 14:38:16,147.147 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 53 16 842 826 62.3302 32 0.428914 1.00767 2014-04-01 14:38:17,146.146 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 54 16 842 826 61.1759 0 - 1.00767 2014-04-01 14:38:18,147.147 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 55 16 842 826 60.0636 0 - 1.00767 2014-04-01 14:38:19,146.146 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 56 16 842 826 58.9911 0 - 1.00767 2014-04-01 14:38:19,646.646 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-04-01 14:38:19,647.647 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:38:19,647.647 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 1 2014-04-01 14:38:19,647.647 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 14:38:19,648.648 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 14:38:19,648.648 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3' 2014-04-01 14:38:20,152.152 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 57 16 864 848 59.4998 22 0.426184 1.07231 2014-04-01 14:38:21,151.151 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 58 16 887 871 60.0599 92 0.751515 1.06193 2014-04-01 14:38:22,149.149 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 59 16 917 901 61.0755 120 0.372116 1.04083 2014-04-01 14:38:23,147.147 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:38:23.390658min lat: 0.121441 max lat: 9.93861 avg lat: 1.03955 2014-04-01 14:38:23,148.148 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:38:23,148.148 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 60 16 920 904 60.2576 12 0.508666 1.03955 2014-04-01 14:38:24,147.147 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 61 16 920 904 59.2697 0 - 1.03955 2014-04-01 14:38:25,111.111 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-04-01 14:38:25,111.111 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:38:25,112.112 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool unique_pool_0 2014-04-01 14:38:25,112.112 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:38:25,147.147 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 62 16 920 904 58.3137 0 - 1.03955 2014-04-01 14:38:25,815.815 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:38:26,150.150 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 63 16 920 904 57.3881 0 - 1.03955 2014-04-01 14:38:26,290.290 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set unique_pool_0 pgp_num 16' 2014-04-01 14:38:27,150.150 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 64 16 930 914 57.1164 10 5.27112 1.08727 2014-04-01 14:38:27,426.426 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 3 pgp_num to 16 2014-04-01 14:38:28,148.148 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 65 16 953 937 57.6528 92 0.79315 1.10265 2014-04-01 14:38:29,148.148 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 66 16 975 959 58.1124 88 0.468836 1.08921 2014-04-01 14:38:30,151.151 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 67 16 1006 990 59.0956 124 0.569747 1.07846 2014-04-01 14:38:31,150.150 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 68 16 1027 1011 59.4617 84 0.210421 1.07163 2014-04-01 14:38:32,152.152 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 69 16 1048 1032 59.8172 84 0.988456 1.06179 2014-04-01 14:38:32,439.439 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-04-01 14:38:32,440.440 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:38:32,440.440 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 1 2014-04-01 14:38:32,441.441 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 14:38:32,441.441 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 14:38:32,441.441 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3' 2014-04-01 14:38:33,151.151 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 70 16 1075 1059 60.5053 108 0.791405 1.05236 2014-04-01 14:38:34,152.152 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 71 16 1114 1098 61.85 156 0.495011 1.03223 2014-04-01 14:38:35,151.151 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 72 16 1126 1110 61.657 48 0.985986 1.02631 2014-04-01 14:38:36,150.150 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 73 16 1126 1110 60.8124 0 - 1.02631 2014-04-01 14:38:37,149.149 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 74 16 1126 1110 59.9906 0 - 1.02631 2014-04-01 14:38:37,869.869 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-04-01 14:38:37,870.870 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:38:37,870.870 INFO:teuthology.task.thrashosds.thrasher:Growing pool rbd 2014-04-01 14:38:37,870.870 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:38:38,150.150 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 75 16 1126 1110 59.1908 0 - 1.02631 2014-04-01 14:38:38,577.577 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:38:39,059.059 INFO:teuthology.task.thrashosds.ceph_manager:increase pool size by 10 2014-04-01 14:38:39,060.060 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set rbd pg_num 34' 2014-04-01 14:38:39,150.150 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 76 16 1126 1110 58.4119 0 - 1.02631 2014-04-01 14:38:40,153.153 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 77 16 1158 1142 59.3155 25.6 0.273714 1.07309 2014-04-01 14:38:41,158.158 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 78 16 1182 1166 59.7856 96 0.597284 1.06747 2014-04-01 14:38:41,262.262 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 2 pg_num to 34 2014-04-01 14:38:42,153.153 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 79 16 1197 1181 59.7882 60 0.703621 1.06177 2014-04-01 14:38:43,168.168 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:38:43.393871min lat: 0.11257 max lat: 9.93861 avg lat: 1.05816 2014-04-01 14:38:43,169.169 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:38:43,169.169 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 80 16 1223 1207 60.3407 104 0.242688 1.05816 2014-04-01 14:38:44,154.154 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 81 16 1255 1239 61.1758 128 0.262377 1.04267 2014-04-01 14:38:45,151.151 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 82 16 1258 1242 60.5761 12 0.501932 1.04127 2014-04-01 14:38:46,153.153 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 83 16 1258 1242 59.8462 0 - 1.04127 2014-04-01 14:38:46,272.272 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-04-01 14:38:46,272.272 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:38:46,273.273 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 3 2014-04-01 14:38:46,273.273 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 14:38:46,273.273 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 14:38:46,274.274 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3' 2014-04-01 14:38:47,151.151 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 84 16 1258 1242 59.1338 0 - 1.04127 2014-04-01 14:38:48,153.153 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 85 16 1258 1242 58.4381 0 - 1.04127 2014-04-01 14:38:49,154.154 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 86 16 1258 1242 57.7585 0 - 1.04127 2014-04-01 14:38:50,153.153 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 87 16 1276 1260 57.9221 14.4 0.411085 1.10125 2014-04-01 14:38:51,156.156 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 88 16 1309 1293 58.7637 132 0.249896 1.08701 2014-04-01 14:38:52,153.153 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 89 16 1335 1319 59.2719 104 0.622517 1.07543 2014-04-01 14:38:52,232.232 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-04-01 14:38:52,232.232 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:38:52,233.233 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 2 2014-04-01 14:38:52,233.233 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 14:38:52,233.233 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 14:38:52,234.234 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3' 2014-04-01 14:38:53,152.152 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 90 16 1337 1321 58.7022 8 0.630188 1.07462 2014-04-01 14:38:54,152.152 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 91 16 1337 1321 58.0571 0 - 1.07462 2014-04-01 14:38:55,170.170 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 92 16 1337 1321 57.426 0 - 1.07462 2014-04-01 14:38:56,159.159 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 93 16 1337 1321 56.8085 0 - 1.07462 2014-04-01 14:38:57,159.159 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 94 16 1342 1326 56.4169 5 5.40833 1.09112 2014-04-01 14:38:57,668.668 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-04-01 14:38:57,669.669 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:38:57,669.669 INFO:teuthology.task.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.327391 2014-04-01 14:38:57,669.669 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 4 0.327390637703' 2014-04-01 14:38:58,154.154 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 95 16 1369 1353 56.9598 108 0.937881 1.11849 2014-04-01 14:38:58,762.762 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set osd.4 primary-affinity to 0.327391 (8214552) 2014-04-01 14:38:59,158.158 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 96 16 1385 1369 57.033 64 0.738484 1.11269 2014-04-01 14:39:00,161.161 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 97 16 1410 1394 57.4759 100 0.415092 1.10421 2014-04-01 14:39:01,165.165 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 98 16 1433 1417 57.828 92 0.458253 1.10354 2014-04-01 14:39:02,158.158 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 99 16 1461 1445 58.3751 112 0.485882 1.09376 2014-04-01 14:39:03,155.155 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:39:03.396606min lat: 0.111203 max lat: 9.93861 avg lat: 1.08976 2014-04-01 14:39:03,156.156 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:39:03,156.156 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 100 16 1479 1463 58.5112 72 0.450674 1.08976 2014-04-01 14:39:03,773.773 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-04-01 14:39:03,773.773 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:39:03,774.774 INFO:teuthology.task.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.930845 2014-04-01 14:39:03,774.774 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 4 0.930845098062' 2014-04-01 14:39:04,155.155 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 101 16 1484 1468 58.1299 20 0.114237 1.08776 2014-04-01 14:39:04,823.823 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set osd.4 primary-affinity to 0.930845 (8610032) 2014-04-01 14:39:05,154.154 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 102 16 1489 1473 57.756 20 0.110174 1.08482 2014-04-01 14:39:06,156.156 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 103 16 1490 1474 57.2341 4 0.919119 1.08471 2014-04-01 14:39:07,154.154 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 104 16 1509 1493 57.4145 76 0.362106 1.10814 2014-04-01 14:39:08,158.158 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 105 16 1538 1522 57.9723 116 0.576452 1.09999 2014-04-01 14:39:09,155.155 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 106 16 1564 1548 58.4064 104 0.843298 1.09011 2014-04-01 14:39:09,833.833 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4] 2014-04-01 14:39:09,834.834 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:39:09,834.834 INFO:teuthology.task.thrashosds.thrasher:Killing osd 2, live_osds are [1, 0, 3, 2, 5, 4] 2014-04-01 14:39:09,835.835 DEBUG:teuthology.task.ceph.osd.2:waiting for process to exit 2014-04-01 14:39:10,119.119 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 05:39:10.360193 7fe3a35da700 0 -- 10.214.133.37:0/1005437 >> 10.214.133.37:6810/4439 pipe(0x7fe3ac00a7b0 sd=12 :0 s=1 pgs=0 cs=0 l=1 c=0x7fe3ac014150).fault 2014-04-01 14:39:10,125.125 INFO:teuthology.task.ceph.osd.2:Stopped 2014-04-01 14:39:10,155.155 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 107 16 1595 1579 59.0193 124 0.372339 1.0793 2014-04-01 14:39:11,155.155 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 108 16 1615 1599 59.2135 80 0.369735 1.07512 2014-04-01 14:39:12,154.154 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 109 16 1625 1609 59.0371 40 0.101482 1.07079 2014-04-01 14:39:13,155.155 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 110 16 1635 1619 58.864 40 0.107381 1.06498 2014-04-01 14:39:14,155.155 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 111 16 1641 1625 58.5499 24 0.345919 1.06155 2014-04-01 14:39:15,123.123 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [2] live_osds: [1, 0, 3, 5, 4] 2014-04-01 14:39:15,123.123 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:39:15,124.124 INFO:teuthology.task.thrashosds.thrasher:Growing pool metadata 2014-04-01 14:39:15,124.124 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:39:15,154.154 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 112 16 1641 1625 58.0271 0 - 1.06155 2014-04-01 14:39:16,130.130 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:39:16,297.297 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 113 16 1641 1625 57.5135 0 - 1.06155 2014-04-01 14:39:16,606.606 INFO:teuthology.task.thrashosds.ceph_manager:increase pool size by 10 2014-04-01 14:39:16,606.606 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set metadata pg_num 34' 2014-04-01 14:39:17,155.155 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 114 16 1641 1625 57.009 0 - 1.06155 2014-04-01 14:39:17,860.860 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 1 pg_num to 34 2014-04-01 14:39:18,157.157 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 115 16 1641 1625 56.5133 0 - 1.06155 2014-04-01 14:39:19,158.158 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 116 16 1641 1625 56.0261 0 - 1.06155 2014-04-01 14:39:20,156.156 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 117 16 1641 1625 55.5472 0 - 1.06155 2014-04-01 14:39:21,156.156 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 118 16 1641 1625 55.0765 0 - 1.06155 2014-04-01 14:39:22,156.156 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 119 16 1641 1625 54.6137 0 - 1.06155 2014-04-01 14:39:22,871.871 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [2] live_osds: [1, 0, 3, 5, 4] 2014-04-01 14:39:22,872.872 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:39:22,872.872 INFO:teuthology.task.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.000000 2014-04-01 14:39:22,873.873 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 3 0' 2014-04-01 14:39:23,156.156 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:39:23.399586min lat: 0.069254 max lat: 9.93861 avg lat: 1.06155 2014-04-01 14:39:23,157.157 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:39:23,157.157 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 120 16 1641 1625 54.1585 0 - 1.06155 2014-04-01 14:39:24,157.157 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 121 16 1641 1625 53.7109 0 - 1.06155 2014-04-01 14:39:24,631.631 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set osd.3 primary-affinity to 0 (802) 2014-04-01 14:39:25,157.157 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 122 16 1641 1625 53.2707 0 - 1.06155 2014-04-01 14:39:26,157.157 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 123 16 1641 1625 52.8376 0 - 1.06155 2014-04-01 14:39:27,157.157 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 124 16 1641 1625 52.4115 0 - 1.06155 2014-04-01 14:39:28,157.157 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 125 16 1641 1625 51.9922 0 - 1.06155 2014-04-01 14:39:29,645.645 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [2] live_osds: [1, 0, 3, 5, 4] 2014-04-01 14:39:29,646.646 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:39:29,646.646 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool rbd 2014-04-01 14:39:29,646.646 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:39:30,176.176 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 126 16 1641 1625 51.5795 0 - 1.06155 2014-04-01 14:39:30,179.179 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 127 16 1641 1625 51.1734 0 - 1.06155 2014-04-01 14:39:30,816.816 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:39:31,158.158 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 128 16 1641 1625 50.7736 0 - 1.06155 2014-04-01 14:39:32,158.158 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 129 16 1641 1625 50.38 0 - 1.06155 2014-04-01 14:39:33,160.160 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 130 16 1682 1666 51.2539 8.63158 0.190124 1.24438 2014-04-01 14:39:34,168.168 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 131 16 1698 1682 51.3511 64 1.21485 1.2428 2014-04-01 14:39:35,163.163 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 132 16 1730 1714 51.9316 128 0.377281 1.22871 2014-04-01 14:39:36,158.158 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 133 16 1740 1724 51.8419 40 0.491054 1.22539 2014-04-01 14:39:36,435.435 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [2] live_osds: [1, 0, 3, 5, 4] 2014-04-01 14:39:36,436.436 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:39:36,436.436 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 0 2014-04-01 14:39:36,437.437 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 14:39:36,437.437 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 14:39:36,438.438 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3' 2014-04-01 14:39:37,173.173 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 134 16 1740 1724 51.455 0 - 1.22539 2014-04-01 14:39:38,197.197 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 135 16 1740 1724 51.0738 0 - 1.22539 2014-04-01 14:39:39,220.220 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 136 16 1740 1724 50.6983 0 - 1.22539 2014-04-01 14:39:40,159.159 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 137 16 1740 1724 50.3282 0 - 1.22539 2014-04-01 14:39:41,160.160 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 138 16 1766 1750 50.717 20.8 0.538413 1.25704 2014-04-01 14:39:42,013.013 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [2] live_osds: [1, 0, 3, 5, 4] 2014-04-01 14:39:42,014.014 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:39:42,015.015 INFO:teuthology.task.thrashosds.thrasher:Reviving osd 2 2014-04-01 14:39:42,015.015 INFO:teuthology.task.ceph.osd.2:Restarting 2014-04-01 14:39:42,015.015 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 2' 2014-04-01 14:39:42,161.161 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 139 16 1795 1779 51.1866 116 0.354454 1.2467 2014-04-01 14:39:42,334.334 INFO:teuthology.task.ceph.osd.2:Started 2014-04-01 14:39:42,335.335 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight' 2014-04-01 14:39:42,433.433 INFO:teuthology.task.ceph.osd.2.out:[10.214.133.37]: starting osd.2 at :/0 osd_data /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal 2014-04-01 14:39:42,764.764 INFO:teuthology.orchestra.run.err:[10.214.133.37]: no valid command found; 10 closest matches: 2014-04-01 14:39:42,764.764 INFO:teuthology.orchestra.run.err:[10.214.133.37]: config show 2014-04-01 14:39:42,765.765 INFO:teuthology.orchestra.run.err:[10.214.133.37]: help 2014-04-01 14:39:42,765.765 INFO:teuthology.orchestra.run.err:[10.214.133.37]: log dump 2014-04-01 14:39:42,772.772 INFO:teuthology.orchestra.run.err:[10.214.133.37]: get_command_descriptions 2014-04-01 14:39:42,772.772 INFO:teuthology.orchestra.run.err:[10.214.133.37]: git_version 2014-04-01 14:39:42,773.773 INFO:teuthology.orchestra.run.err:[10.214.133.37]: config set [...] 2014-04-01 14:39:42,773.773 INFO:teuthology.orchestra.run.err:[10.214.133.37]: version 2014-04-01 14:39:42,773.773 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2 2014-04-01 14:39:42,774.774 INFO:teuthology.orchestra.run.err:[10.214.133.37]: config get 2014-04-01 14:39:42,774.774 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 0 2014-04-01 14:39:42,774.774 INFO:teuthology.orchestra.run.err:[10.214.133.37]: admin_socket: invalid command 2014-04-01 14:39:42,775.775 INFO:teuthology.task.thrashosds.ceph_manager:waiting on admin_socket for 2, ['dump_ops_in_flight'] 2014-04-01 14:39:42,854.854 INFO:teuthology.task.ceph.osd.2.err:[10.214.133.37]: 2014-04-01 05:39:43.090476 7f7c948cc780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 14:39:43,159.159 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:39:43.402495min lat: 0.069254 max lat: 22.6432 avg lat: 1.23944 2014-04-01 14:39:43,160.160 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:39:43,160.160 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 140 16 1816 1800 51.4209 84 0.596875 1.23944 2014-04-01 14:39:44,166.166 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 141 16 1844 1828 51.8504 112 0.253457 1.23112 2014-04-01 14:39:45,161.161 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 142 16 1867 1851 52.1331 92 0.398165 1.22418 2014-04-01 14:39:46,164.164 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 143 16 1883 1867 52.216 64 0.184982 1.21735 2014-04-01 14:39:47,160.160 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 144 16 1898 1882 52.27 60 0.670766 1.2149 2014-04-01 14:39:47,770.770 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight' 2014-04-01 14:39:48,160.160 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 145 16 1898 1882 51.9095 0 - 1.2149 2014-04-01 14:39:49,163.163 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 146 16 1898 1882 51.554 0 - 1.2149 2014-04-01 14:39:50,161.161 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 147 16 1898 1882 51.2033 0 - 1.2149 2014-04-01 14:39:51,165.165 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 148 15 1912 1897 51.2627 15 4.27916 1.24258 2014-04-01 14:39:52,165.165 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 149 16 1938 1922 51.5897 100 0.811994 1.23799 2014-04-01 14:39:53,162.162 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 150 16 1962 1946 51.8856 96 0.416219 1.22927 2014-04-01 14:39:53,196.196 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 5, 4, 2] 2014-04-01 14:39:53,196.196 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:39:53,197.197 INFO:teuthology.task.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.000000 2014-04-01 14:39:53,197.197 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 5 0' 2014-04-01 14:39:54,167.167 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 151 16 1992 1976 52.3366 120 0.448335 1.22113 2014-04-01 14:39:54,180.180 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set osd.5 primary-affinity to 0 (802) 2014-04-01 14:39:55,161.161 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 152 16 2012 1996 52.5185 80 0.512904 1.21409 2014-04-01 14:39:56,162.162 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 153 16 2025 2009 52.5151 52 0.116806 1.2099 2014-04-01 14:39:57,161.161 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 154 16 2026 2010 52.2 4 0.193463 1.20939 2014-04-01 14:39:58,162.162 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 155 16 2031 2015 51.9923 20 0.124409 1.2129 2014-04-01 14:39:59,162.162 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 156 16 2031 2015 51.659 0 - 1.2129 2014-04-01 14:39:59,188.188 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 5, 4, 2] 2014-04-01 14:39:59,188.188 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:39:59,188.188 INFO:teuthology.task.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.000000 2014-04-01 14:39:59,188.188 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 1 0' 2014-04-01 14:40:00,162.162 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 157 16 2031 2015 51.3299 0 - 1.2129 2014-04-01 14:40:00,737.737 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set osd.1 primary-affinity to 0 (802) 2014-04-01 14:40:01,166.166 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 158 16 2056 2040 51.6379 33.3333 0.423308 1.23662 2014-04-01 14:40:02,167.167 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 159 16 2078 2062 51.8665 88 0.332472 1.22954 2014-04-01 14:40:03,163.163 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:40:03.405296min lat: 0.069254 max lat: 22.6432 avg lat: 1.22159 2014-04-01 14:40:03,163.163 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:40:03,164.164 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 160 16 2095 2079 51.9673 68 0.397453 1.22159 2014-04-01 14:40:04,164.164 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 161 16 2110 2094 52.0171 60 0.322465 1.21657 2014-04-01 14:40:05,168.168 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 162 16 2125 2109 52.0664 60 0.377082 1.2092 2014-04-01 14:40:05,746.746 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 5, 4, 2] 2014-04-01 14:40:05,746.746 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:40:05,746.746 INFO:teuthology.task.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.030904 2014-04-01 14:40:05,746.746 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 4 0.0309041946371' 2014-04-01 14:40:06,170.170 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 163 16 2135 2119 51.9923 40 0.22462 1.20445 2014-04-01 14:40:07,068.068 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set osd.4 primary-affinity to 0.0309042 (820252) 2014-04-01 14:40:07,163.163 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 164 16 2146 2130 51.9435 44 0.438889 1.21225 2014-04-01 14:40:08,171.171 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 165 16 2167 2151 52.1378 84 0.447239 1.22267 2014-04-01 14:40:09,168.168 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 166 16 2183 2167 52.2092 64 0.570955 1.21793 2014-04-01 14:40:10,169.169 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 167 16 2198 2182 52.2558 60 0.23206 1.21407 2014-04-01 14:40:11,276.276 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 168 16 2202 2186 52.0399 16 0.160535 1.21236 2014-04-01 14:40:12,072.072 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 5, 4, 2] 2014-04-01 14:40:12,073.073 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:40:12,073.073 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 2 2014-04-01 14:40:12,074.074 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 14:40:12,074.074 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 14:40:12,075.075 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3' 2014-04-01 14:40:12,164.164 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 169 16 2203 2187 51.7557 4 0.111125 1.21186 2014-04-01 14:40:13,164.164 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 170 16 2203 2187 51.4512 0 - 1.21186 2014-04-01 14:40:14,167.167 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 171 16 2223 2207 51.6181 40 0.354031 1.23784 2014-04-01 14:40:15,165.165 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 172 16 2243 2227 51.783 80 0.381424 1.23144 2014-04-01 14:40:16,165.165 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 173 16 2254 2238 51.738 44 0.089318 1.22664 2014-04-01 14:40:17,167.167 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 174 15 2266 2251 51.7395 52 2.17303 1.23182 2014-04-01 14:40:17,500.500 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 5, 4, 2] 2014-04-01 14:40:17,501.501 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:40:17,501.501 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 3 2014-04-01 14:40:17,502.502 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 14:40:17,502.502 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 14:40:17,502.502 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3' 2014-04-01 14:40:18,165.165 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 175 16 2287 2271 51.9009 80 0.407092 1.22977 2014-04-01 14:40:19,167.167 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 176 16 2309 2293 52.1059 88 0.119005 1.22304 2014-04-01 14:40:20,165.165 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 177 16 2315 2299 51.9471 24 0.108013 1.22117 2014-04-01 14:40:21,166.166 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 178 16 2315 2299 51.6553 0 - 1.22117 2014-04-01 14:40:22,165.165 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 179 16 2315 2299 51.3667 0 - 1.22117 2014-04-01 14:40:23,103.103 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 5, 4, 2] 2014-04-01 14:40:23,103.103 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:40:23,104.104 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 5 2014-04-01 14:40:23,104.104 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 14:40:23,105.105 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 14:40:23,105.105 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3' 2014-04-01 14:40:23,167.167 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:40:23.408163min lat: 0.069254 max lat: 22.6432 avg lat: 1.24085 2014-04-01 14:40:23,167.167 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:40:23,167.167 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 180 16 2335 2319 51.5257 26.6667 0.41922 1.24085 2014-04-01 14:40:24,174.174 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 181 16 2363 2347 51.8598 112 0.576267 1.23254 2014-04-01 14:40:25,170.170 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 182 16 2385 2369 52.0583 88 0.781146 1.22751 2014-04-01 14:40:26,167.167 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 183 16 2413 2397 52.3857 112 0.767223 1.22023 2014-04-01 14:40:27,169.169 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 184 16 2434 2418 52.5575 84 0.698728 1.21526 2014-04-01 14:40:28,169.169 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 185 15 2455 2440 52.749 88 0.7945 1.20987 2014-04-01 14:40:28,544.544 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 5, 4, 2] 2014-04-01 14:40:28,544.544 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:40:28,545.545 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool metadata 2014-04-01 14:40:28,546.546 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:40:29,167.167 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 186 16 2472 2456 52.8094 64 0.357345 1.20499 2014-04-01 14:40:29,363.363 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:40:29,999.999 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set metadata pgp_num 34' 2014-04-01 14:40:30,167.167 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 187 16 2472 2456 52.527 0 - 1.20499 2014-04-01 14:40:31,171.171 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 188 16 2485 2469 52.5242 26 0.260262 1.2086 2014-04-01 14:40:31,410.410 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 1 pgp_num to 34 2014-04-01 14:40:32,170.170 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 189 16 2488 2472 52.3097 12 0.468559 1.20756 2014-04-01 14:40:33,173.173 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 190 16 2509 2493 52.4765 84 0.262589 1.21802 2014-04-01 14:40:34,167.167 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 191 16 2512 2496 52.2646 12 0.242719 1.21705 2014-04-01 14:40:35,171.171 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 192 16 2536 2520 52.4923 96 0.381572 1.21744 2014-04-01 14:40:36,168.168 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 193 15 2561 2546 52.7591 104 0.763512 1.2115 2014-04-01 14:40:36,419.419 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 5, 4, 2] 2014-04-01 14:40:36,419.419 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:40:36,420.420 INFO:teuthology.task.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 4, 5] 2014-04-01 14:40:36,420.420 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 4' 2014-04-01 14:40:37,173.173 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 194 16 2587 2571 53.0025 100 0.383384 1.20595 2014-04-01 14:40:38,174.174 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 195 16 2613 2597 53.264 104 0.392461 1.19987 2014-04-01 14:40:38,390.390 INFO:teuthology.orchestra.run.err:[10.214.133.37]: marked out osd.4. 2014-04-01 14:40:39,218.218 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 196 16 2620 2604 53.1351 28 0.585473 1.19811 2014-04-01 14:40:40,168.168 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 197 16 2621 2605 52.8857 4 1.62339 1.19827 2014-04-01 14:40:41,175.175 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 198 16 2626 2610 52.7196 20 0.497382 1.19888 2014-04-01 14:40:42,196.196 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 199 16 2638 2622 52.6958 48 0.691183 1.20001 2014-04-01 14:40:43,220.220 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:40:43.410823min lat: 0.064365 max lat: 22.6432 avg lat: 1.20041 2014-04-01 14:40:43,220.220 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:40:43,220.220 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 200 16 2656 2640 52.7923 72 0.538411 1.20041 2014-04-01 14:40:43,399.399 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5] out_osds: [4] dead_osds: [] live_osds: [1, 0, 3, 5, 4, 2] 2014-04-01 14:40:43,399.399 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:40:43,400.400 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 2 2014-04-01 14:40:43,400.400 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 14:40:43,400.400 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 14:40:43,401.401 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3' 2014-04-01 14:40:44,186.186 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 201 16 2669 2653 52.7883 52 0.094878 1.20162 2014-04-01 14:40:45,170.170 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 202 16 2682 2666 52.7844 52 1.44289 1.2062 2014-04-01 14:40:46,170.170 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 203 16 2684 2668 52.5637 8 0.410208 1.20572 2014-04-01 14:40:47,170.170 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 204 16 2684 2668 52.3061 0 - 1.20572 2014-04-01 14:40:48,169.169 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 205 16 2684 2668 52.0509 0 - 1.20572 2014-04-01 14:40:49,040.040 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5] out_osds: [4] dead_osds: [] live_osds: [1, 0, 3, 5, 4, 2] 2014-04-01 14:40:49,041.041 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:40:49,041.041 INFO:teuthology.task.thrashosds.thrasher:Adding osd 4 2014-04-01 14:40:49,042.042 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 4' 2014-04-01 14:40:49,168.168 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 206 16 2696 2680 52.0312 16 4.65865 1.21729 2014-04-01 14:40:50,170.170 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 207 16 2704 2688 51.9344 32 0.918561 1.22241 2014-04-01 14:40:50,560.560 INFO:teuthology.orchestra.run.err:[10.214.133.37]: marked in osd.4. 2014-04-01 14:40:50,568.568 INFO:teuthology.task.thrashosds.thrasher:Added osd 4 2014-04-01 14:40:51,170.170 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 208 16 2711 2695 51.8193 28 0.158132 1.22166 2014-04-01 14:40:52,170.170 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 209 16 2711 2695 51.5714 0 - 1.22166 2014-04-01 14:40:53,170.170 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 210 16 2713 2697 51.3639 4 3.88925 1.22757 2014-04-01 14:40:54,174.174 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 211 16 2718 2702 51.2153 20 3.71813 1.23273 2014-04-01 14:40:55,170.170 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 212 16 2730 2714 51.2001 48 0.335939 1.24231 2014-04-01 14:40:55,569.569 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 4] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 5, 4, 2] 2014-04-01 14:40:55,569.569 INFO:teuthology.task.thrashosds.ceph_manager:waiting for recovery to complete 2014-04-01 14:40:55,570.570 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:40:56,170.170 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 213 16 2746 2730 51.2601 64 0.532024 1.24102 2014-04-01 14:40:56,240.240 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:40:57,022.022 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:40:57,170.170 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 214 16 2755 2739 51.1888 36 0.983497 1.23897 2014-04-01 14:40:57,712.712 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:40:58,171.171 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 215 16 2755 2739 50.9507 0 - 1.23897 2014-04-01 14:40:58,332.332 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:40:59,054.054 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.160844', u'skew': u'-0.277086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.160744', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:40:42.636613', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439084248, u'health': u'HEALTH_OK', u'kb_used': 9244736, u'store_stats': {u'bytes_total': 6293402, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:40:42.975599', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438954800, u'health': u'HEALTH_OK', u'kb_used': 9374184, u'store_stats': {u'bytes_total': 9439130, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:40:42.991938', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439083268, u'health': u'HEALTH_OK', u'kb_used': 9245716, u'store_stats': {u'bytes_total': 8390554, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 207/8268 objects degraded (2.504%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'degraded_objects': 207, u'num_pgs': 118, u'data_bytes': 11484006756, u'degraded_total': 8268, u'bytes_used': 18357932032, u'op_per_sec': 10, u'write_bytes_sec': 43522152, u'version': 137, u'pgs_by_state': [{u'count': 3, u'state_name': u'active'}, {u'count': 2, u'state_name': u'active+recovery_wait'}, {u'count': 102, u'state_name': u'active+clean'}, {u'count': 11, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.504', u'bytes_avail': 2784257503232}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 45, u'num_in_osds': 6}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:40:59,055.055 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:40:59,170.170 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 216 16 2755 2739 50.7148 0 - 1.23897 2014-04-01 14:40:59,815.815 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:41:00,171.171 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 217 16 2755 2739 50.4811 0 - 1.23897 2014-04-01 14:41:01,174.174 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 218 16 2755 2739 50.2495 0 - 1.23897 2014-04-01 14:41:02,175.175 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 219 16 2771 2755 50.3123 12.8 5.52832 1.2599 2014-04-01 14:41:03,172.172 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:41:03.413843min lat: 0.064365 max lat: 22.6432 avg lat: 1.25816 2014-04-01 14:41:03,172.172 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:41:03,172.172 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 220 16 2786 2770 50.3563 60 0.114573 1.25816 2014-04-01 14:41:03,419.419 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:41:04,123.123 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:41:04,281.281 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 221 15 2795 2780 50.3094 40 14.9686 1.26649 2014-04-01 14:41:04,744.744 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:41:05,171.171 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 222 16 2796 2780 50.0827 0 - 1.26649 2014-04-01 14:41:05,526.526 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.160844', u'skew': u'-0.277086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.160744', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:40:42.636613', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439084248, u'health': u'HEALTH_OK', u'kb_used': 9244736, u'store_stats': {u'bytes_total': 6293402, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:40:42.975599', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438954800, u'health': u'HEALTH_OK', u'kb_used': 9374184, u'store_stats': {u'bytes_total': 9439130, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:40:42.991938', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439083268, u'health': u'HEALTH_OK', u'kb_used': 9245716, u'store_stats': {u'bytes_total': 8390554, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 230/8316 objects degraded (2.766%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 9, u'degraded_objects': 230, u'num_pgs': 118, u'recovering_keys_per_sec': 0, u'data_bytes': 11551115620, u'degraded_total': 8316, u'bytes_used': 18677936128, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 13410387, u'op_per_sec': 3, u'version': 139, u'pgs_by_state': [{u'count': 105, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.766', u'bytes_avail': 2783937499136}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 45, u'num_in_osds': 6}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:41:05,527.527 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:41:06,172.172 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 223 16 2796 2780 49.8582 0 - 1.26649 2014-04-01 14:41:06,220.220 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:41:07,172.172 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 224 16 2796 2780 49.6356 0 - 1.26649 2014-04-01 14:41:08,173.173 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 225 16 2796 2780 49.415 0 - 1.26649 2014-04-01 14:41:09,178.178 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 226 16 2805 2789 49.3556 7.2 0.194212 1.28006 2014-04-01 14:41:09,831.831 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:41:10,173.173 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 227 16 2818 2802 49.3672 52 0.20925 1.29122 2014-04-01 14:41:10,596.596 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:41:11,174.174 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 228 16 2837 2821 49.484 76 0.208248 1.29065 2014-04-01 14:41:11,239.239 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:41:11,982.982 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.160844', u'skew': u'-0.277086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.160744', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:40:42.636613', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439084248, u'health': u'HEALTH_OK', u'kb_used': 9244736, u'store_stats': {u'bytes_total': 6293402, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:40:42.975599', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438954800, u'health': u'HEALTH_OK', u'kb_used': 9374184, u'store_stats': {u'bytes_total': 9439130, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:40:42.991938', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439083268, u'health': u'HEALTH_OK', u'kb_used': 9245716, u'store_stats': {u'bytes_total': 8390554, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'7 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 200/8517 objects degraded (2.348%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'degraded_objects': 200, u'num_pgs': 118, u'data_bytes': 11832133988, u'degraded_total': 8517, u'bytes_used': 18871242752, u'op_per_sec': 6, u'write_bytes_sec': 25897861, u'version': 143, u'pgs_by_state': [{u'count': 111, u'state_name': u'active+clean'}, {u'count': 7, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.348', u'bytes_avail': 2783744192512}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 45, u'num_in_osds': 6}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:41:11,983.983 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:41:12,172.172 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 229 16 2840 2824 49.3203 12 0.547537 1.29026 2014-04-01 14:41:12,668.668 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:41:13,173.173 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 230 16 2848 2832 49.245 32 0.170624 1.29119 2014-04-01 14:41:14,180.180 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 231 16 2849 2833 49.0491 4 3.50962 1.29197 2014-04-01 14:41:15,181.181 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 232 16 2849 2833 48.8377 0 - 1.29197 2014-04-01 14:41:16,180.180 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 233 16 2849 2833 48.6281 0 - 1.29197 2014-04-01 14:41:16,292.292 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:41:17,208.208 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:41:17,410.410 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 234 16 2854 2838 48.5057 6.66667 5.98717 1.3005 2014-04-01 14:41:17,939.939 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:41:18,179.179 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 235 15 2882 2867 48.7929 116 0.762549 1.30963 2014-04-01 14:41:18,880.880 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.160844', u'skew': u'-0.277086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.160744', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:40:42.636613', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439084248, u'health': u'HEALTH_OK', u'kb_used': 9244736, u'store_stats': {u'bytes_total': 6293402, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:40:42.975599', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438954800, u'health': u'HEALTH_OK', u'kb_used': 9374184, u'store_stats': {u'bytes_total': 9439130, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:40:42.991938', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439083268, u'health': u'HEALTH_OK', u'kb_used': 9245716, u'store_stats': {u'bytes_total': 8390554, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 167/8598 objects degraded (1.942%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'degraded_objects': 167, u'num_pgs': 118, u'data_bytes': 11945380196, u'degraded_total': 8598, u'bytes_used': 19078201344, u'op_per_sec': 5, u'write_bytes_sec': 22700057, u'version': 145, u'pgs_by_state': [{u'count': 113, u'state_name': u'active+clean'}, {u'count': 5, u'state_name': u'active+recovering'}], u'degraded_ratio': u'1.942', u'bytes_avail': 2783537233920}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 45, u'num_in_osds': 6}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:41:18,881.881 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:41:19,179.179 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 236 16 2897 2881 48.8234 56 0.315812 1.30664 2014-04-01 14:41:19,684.684 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:41:20,180.180 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 237 16 2897 2881 48.6174 0 - 1.30664 2014-04-01 14:41:21,174.174 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 238 16 2897 2881 48.4131 0 - 1.30664 2014-04-01 14:41:22,174.174 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 239 16 2897 2881 48.2105 0 - 1.30664 2014-04-01 14:41:23,178.178 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:41:23.416622min lat: 0.064365 max lat: 22.6432 avg lat: 1.32285 2014-04-01 14:41:23,178.178 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:41:23,179.179 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 240 16 2917 2901 48.3429 20 0.145 1.32285 2014-04-01 14:41:23,345.345 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:41:24,174.174 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 241 16 2945 2929 48.607 112 0.388753 1.31467 2014-04-01 14:41:25,176.176 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 242 16 2956 2940 48.588 44 1.29038 1.31316 2014-04-01 14:41:26,175.175 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 243 16 2967 2951 48.569 44 0.095785 1.31159 2014-04-01 14:41:27,315.315 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:41:27,467.467 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 244 16 2967 2951 48.37 0 - 1.31159 2014-04-01 14:41:27,945.945 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:41:28,175.175 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 245 16 2967 2951 48.1726 0 - 1.31159 2014-04-01 14:41:28,640.640 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.160844', u'skew': u'-0.277086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.160744', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:40:42.636613', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439084248, u'health': u'HEALTH_OK', u'kb_used': 9244736, u'store_stats': {u'bytes_total': 6293402, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:40:42.975599', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438954800, u'health': u'HEALTH_OK', u'kb_used': 9374184, u'store_stats': {u'bytes_total': 9439130, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:40:42.991938', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 439083268, u'health': u'HEALTH_OK', u'kb_used': 9245716, u'store_stats': {u'bytes_total': 8390554, u'bytes_log': 8323072, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'num_pgs': 118, u'data_bytes': 12431919460, u'bytes_used': 19478585344, u'op_per_sec': 14, u'version': 149, u'pgs_by_state': [{u'count': 118, u'state_name': u'active+clean'}], u'write_bytes_sec': 60223077, u'bytes_avail': 2783136849920}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 45, u'num_in_osds': 6}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:41:28,641.641 INFO:teuthology.task.thrashosds.ceph_manager:recovered! 2014-04-01 14:41:28,641.641 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:41:28,641.641 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 0 2014-04-01 14:41:28,641.641 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 14:41:28,642.642 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 14:41:28,642.642 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3' 2014-04-01 14:41:29,175.175 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 246 16 2967 2951 47.9767 0 - 1.31159 2014-04-01 14:41:30,176.176 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 247 16 2967 2951 47.7825 0 - 1.31159 2014-04-01 14:41:31,180.180 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 248 16 2982 2966 47.8317 12 5.77577 1.3353 2014-04-01 14:41:32,181.181 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 249 16 3005 2989 48.009 92 0.922209 1.33059 2014-04-01 14:41:33,177.177 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 250 16 3034 3018 48.2809 116 0.671038 1.3234 2014-04-01 14:41:34,196.196 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 251 16 3050 3034 48.3435 64 0.226926 1.3195 2014-04-01 14:41:34,202.202 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 4] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 5, 4, 2] 2014-04-01 14:41:34,203.203 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:41:34,203.203 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 4 2014-04-01 14:41:34,203.203 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 14:41:34,203.203 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 14:41:34,204.204 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3' 2014-04-01 14:41:35,178.178 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 252 16 3050 3034 48.1517 0 - 1.3195 2014-04-01 14:41:36,177.177 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 253 16 3050 3034 47.9613 0 - 1.3195 2014-04-01 14:41:37,184.184 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 254 16 3050 3034 47.7725 0 - 1.3195 2014-04-01 14:41:38,208.208 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 255 16 3050 3034 47.5852 0 - 1.3195 2014-04-01 14:41:39,233.233 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 256 16 3068 3052 47.6805 14.4 0.524638 1.34011 2014-04-01 14:41:39,727.727 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 4] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 5, 4, 2] 2014-04-01 14:41:39,727.727 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:41:39,728.728 INFO:teuthology.task.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 3, 5, 4] 2014-04-01 14:41:39,728.728 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 1' 2014-04-01 14:41:40,183.183 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 257 16 3109 3093 48.133 164 0.603121 1.32814 2014-04-01 14:41:40,613.613 INFO:teuthology.orchestra.run.err:[10.214.133.37]: marked out osd.1. 2014-04-01 14:41:41,177.177 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 258 16 3109 3093 47.9464 0 - 1.32814 2014-04-01 14:41:42,179.179 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 259 16 3109 3093 47.7613 0 - 1.32814 2014-04-01 14:41:43,178.178 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:41:43.419847min lat: 0.064365 max lat: 22.6432 avg lat: 1.32814 2014-04-01 14:41:43,178.178 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:41:43,178.178 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 260 16 3109 3093 47.5776 0 - 1.32814 2014-04-01 14:41:44,178.178 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 261 16 3109 3093 47.3953 0 - 1.32814 2014-04-01 14:41:45,181.181 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 262 16 3109 3093 47.2144 0 - 1.32814 2014-04-01 14:41:45,627.627 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 4] out_osds: [1] dead_osds: [] live_osds: [1, 0, 3, 5, 4, 2] 2014-04-01 14:41:45,627.627 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:41:45,628.628 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 3 2014-04-01 14:41:45,628.628 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 14:41:45,628.628 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 14:41:45,629.629 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3' 2014-04-01 14:41:46,178.178 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 263 16 3109 3093 47.0349 0 - 1.32814 2014-04-01 14:41:47,178.178 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 264 16 3109 3093 46.8567 0 - 1.32814 2014-04-01 14:41:48,180.180 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 265 16 3112 3096 46.7252 1.5 8.16934 1.3349 2014-04-01 14:41:49,179.179 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 266 16 3116 3100 46.6097 16 0.896741 1.33662 2014-04-01 14:41:50,181.181 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 267 16 3119 3103 46.48 12 0.272474 1.33607 2014-04-01 14:41:51,179.179 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 268 16 3122 3106 46.3514 12 0.691251 1.34164 2014-04-01 14:41:51,463.463 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 4] out_osds: [1] dead_osds: [] live_osds: [1, 0, 3, 5, 4, 2] 2014-04-01 14:41:51,464.464 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:41:51,464.464 INFO:teuthology.task.thrashosds.thrasher:Adding osd 1 2014-04-01 14:41:51,465.465 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 1' 2014-04-01 14:41:52,183.183 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 269 16 3137 3121 46.4021 60 0.5087 1.35113 2014-04-01 14:41:52,400.400 INFO:teuthology.orchestra.run.err:[10.214.133.37]: marked in osd.1. 2014-04-01 14:41:52,553.553 INFO:teuthology.task.thrashosds.thrasher:Added osd 1 2014-04-01 14:41:53,180.180 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 270 16 3145 3129 46.3487 32 0.748324 1.34867 2014-04-01 14:41:54,180.180 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 271 16 3147 3131 46.2072 8 0.426233 1.34817 2014-04-01 14:41:55,182.182 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 272 16 3148 3132 46.052 4 15.1821 1.35258 2014-04-01 14:41:56,181.181 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 273 16 3160 3144 46.0592 48 0.509996 1.37993 2014-04-01 14:41:57,186.186 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 274 16 3182 3166 46.2122 88 0.320583 1.37478 2014-04-01 14:41:57,554.554 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 4, 1] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 5, 4, 2] 2014-04-01 14:41:57,555.555 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:41:57,555.555 INFO:teuthology.task.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 2, 3, 5, 4, 1] 2014-04-01 14:41:57,555.555 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 5' 2014-04-01 14:41:58,182.182 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 275 16 3204 3188 46.3641 88 0.47951 1.3686 2014-04-01 14:41:59,183.183 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 276 16 3218 3202 46.399 56 0.684569 1.36485 2014-04-01 14:41:59,329.329 INFO:teuthology.orchestra.run.err:[10.214.133.37]: marked out osd.5. 2014-04-01 14:42:00,182.182 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 277 16 3223 3207 46.3037 20 0.473121 1.36369 2014-04-01 14:42:01,183.183 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 278 16 3223 3207 46.1371 0 - 1.36369 2014-04-01 14:42:02,183.183 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 279 16 3225 3209 46.0004 4 2.00829 1.36409 2014-04-01 14:42:03,181.181 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:42:03.422634min lat: 0.064365 max lat: 22.6432 avg lat: 1.36409 2014-04-01 14:42:03,181.181 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:42:03,182.182 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 280 16 3225 3209 45.8361 0 - 1.36409 2014-04-01 14:42:04,181.181 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 281 16 3241 3225 45.9008 32 0.503913 1.36942 2014-04-01 14:42:04,339.339 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 1] out_osds: [5] dead_osds: [] live_osds: [1, 0, 3, 5, 4, 2] 2014-04-01 14:42:04,339.339 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:42:04,339.339 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool rbd 2014-04-01 14:42:04,339.339 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:42:04,945.945 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:42:05,257.257 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 282 15 3248 3233 45.8514 32 0.311234 1.36802 2014-04-01 14:42:05,504.504 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 05:42:05.745527 7fe3a7fff700 0 -- 10.214.133.37:0/1005437 >> 10.214.133.37:6814/6341 pipe(0x7fe3ac018ad0 sd=12 :56755 s=2 pgs=2 cs=1 l=1 c=0x7fe3ac01c7d0).injecting socket failure 2014-04-01 14:42:05,880.880 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set rbd pgp_num 34' 2014-04-01 14:42:06,182.182 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 283 16 3264 3248 45.9014 60 13.919 1.37241 2014-04-01 14:42:07,181.181 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 284 16 3273 3257 45.8665 36 0.212056 1.38089 2014-04-01 14:42:07,889.889 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 2 pgp_num to 34 2014-04-01 14:42:08,180.180 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 285 16 3276 3260 45.7477 12 0.261949 1.38532 2014-04-01 14:42:08,811.811 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 05:42:09.051039 7fe3a35da700 0 -- 10.214.133.37:0/1005437 >> 10.214.133.37:6814/6341 pipe(0x7fe3ac018ad0 sd=12 :56864 s=2 pgs=3 cs=1 l=1 c=0x7fe3ac00c980).injecting socket failure 2014-04-01 14:42:09,182.182 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 286 16 3283 3267 45.6856 28 1.23493 1.38914 2014-04-01 14:42:09,397.397 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 05:42:09.637307 7fe3a2fd6700 0 -- 10.214.133.37:0/1005437 >> 10.214.133.37:6814/6341 pipe(0x7fe3ac022670 sd=13 :56869 s=2 pgs=4 cs=1 l=1 c=0x7fe3ac01b940).injecting socket failure 2014-04-01 14:42:10,187.187 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 287 16 3301 3285 45.7773 72 0.612895 1.39016 2014-04-01 14:42:11,211.211 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 288 16 3318 3302 45.8544 68 0.720474 1.38749 2014-04-01 14:42:12,192.192 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 289 16 3326 3310 45.8065 32 0.764033 1.38627 2014-04-01 14:42:12,887.887 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 1] out_osds: [5] dead_osds: [] live_osds: [1, 0, 3, 5, 4, 2] 2014-04-01 14:42:12,888.888 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:42:12,888.888 INFO:teuthology.task.thrashosds.thrasher:Killing osd 4, live_osds are [1, 0, 3, 5, 4, 2] 2014-04-01 14:42:12,889.889 DEBUG:teuthology.task.ceph.osd.4:waiting for process to exit 2014-04-01 14:42:13,086.086 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 05:42:13.328509 7fe3a3add700 0 -- 10.214.133.37:0/1005437 >> 10.214.133.38:6805/4380 pipe(0x7fe3ac012e70 sd=13 :0 s=1 pgs=0 cs=0 l=1 c=0x7fe3ac00a0c0).fault 2014-04-01 14:42:13,090.090 INFO:teuthology.task.ceph.osd.4:Stopped 2014-04-01 14:42:13,181.181 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 290 16 3333 3317 45.745 28 0.204364 1.38452 2014-04-01 14:42:14,184.184 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 291 16 3333 3317 45.5878 0 - 1.38452 2014-04-01 14:42:15,184.184 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 292 16 3333 3317 45.4317 0 - 1.38452 2014-04-01 14:42:16,183.183 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 293 16 3341 3325 45.3859 10.6667 4.70554 1.39297 2014-04-01 14:42:17,183.183 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 294 16 3345 3329 45.2859 16 0.876253 1.39528 2014-04-01 14:42:18,091.091 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 1] out_osds: [5] dead_osds: [4] live_osds: [1, 0, 3, 5, 2] 2014-04-01 14:42:18,091.091 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:42:18,092.092 INFO:teuthology.task.thrashosds.thrasher:Killing osd 1, live_osds are [1, 0, 3, 5, 2] 2014-04-01 14:42:18,092.092 DEBUG:teuthology.task.ceph.osd.1:waiting for process to exit 2014-04-01 14:42:18,263.263 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 05:42:18.180666 7fe3a3cdf700 0 -- 10.214.133.37:0/1005437 >> 10.214.133.37:6805/4412 pipe(0x7fe3ac016e30 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7fe3ac003180).fault 2014-04-01 14:42:18,267.267 INFO:teuthology.task.ceph.osd.1:Stopped 2014-04-01 14:42:18,514.514 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 295 16 3345 3329 45.1324 0 - 1.39528 2014-04-01 14:42:19,513.513 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 296 16 3345 3329 44.9799 0 - 1.39528 2014-04-01 14:42:20,509.509 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 297 16 3345 3329 44.8285 0 - 1.39528 2014-04-01 14:42:21,507.507 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 298 16 3345 3329 44.678 0 - 1.39528 2014-04-01 14:42:22,509.509 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 299 16 3345 3329 44.5286 0 - 1.39528 2014-04-01 14:42:23,268.268 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 1] out_osds: [5] dead_osds: [4, 1] live_osds: [0, 3, 5, 2] 2014-04-01 14:42:23,269.269 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:42:23,269.269 INFO:teuthology.task.thrashosds.thrasher:Growing pool unique_pool_0 2014-04-01 14:42:23,270.270 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:42:23,513.513 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:42:23.425392min lat: 0.064365 max lat: 22.6432 avg lat: 1.39528 2014-04-01 14:42:23,513.513 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:42:23,590.590 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 300 16 3345 3329 44.3802 0 - 1.39528 2014-04-01 14:42:23,971.971 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:42:24,515.515 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 301 16 3345 3329 44.2327 0 - 1.39528 2014-04-01 14:42:24,634.634 INFO:teuthology.task.thrashosds.ceph_manager:increase pool size by 10 2014-04-01 14:42:24,634.634 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set unique_pool_0 pg_num 26' 2014-04-01 14:42:25,513.513 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 302 16 3345 3329 44.0863 0 - 1.39528 2014-04-01 14:42:25,608.608 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 3 pg_num to 26 2014-04-01 14:42:26,514.514 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 303 16 3345 3329 43.9408 0 - 1.39528 2014-04-01 14:42:27,509.509 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 304 16 3345 3329 43.7962 0 - 1.39528 2014-04-01 14:42:28,514.514 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 305 16 3345 3329 43.6526 0 - 1.39528 2014-04-01 14:42:29,514.514 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 306 16 3345 3329 43.51 0 - 1.39528 2014-04-01 14:42:30,536.536 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 307 16 3345 3329 43.3683 0 - 1.39528 2014-04-01 14:42:30,618.618 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 1] out_osds: [5] dead_osds: [4, 1] live_osds: [0, 3, 5, 2] 2014-04-01 14:42:30,618.618 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:42:30,618.618 INFO:teuthology.task.thrashosds.thrasher:Adding osd 5 2014-04-01 14:42:30,618.618 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 5' 2014-04-01 14:42:31,510.510 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 308 16 3345 3329 43.2275 0 - 1.39528 2014-04-01 14:42:31,677.677 INFO:teuthology.orchestra.run.err:[10.214.133.37]: marked in osd.5. 2014-04-01 14:42:31,686.686 INFO:teuthology.task.thrashosds.thrasher:Added osd 5 2014-04-01 14:42:32,510.510 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 309 16 3345 3329 43.0876 0 - 1.39528 2014-04-01 14:42:33,510.510 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 310 16 3345 3329 42.9486 0 - 1.39528 2014-04-01 14:42:34,512.512 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 311 16 3345 3329 42.8105 0 - 1.39528 2014-04-01 14:42:35,510.510 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 312 16 3345 3329 42.6733 0 - 1.39528 2014-04-01 14:42:36,510.510 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 313 16 3345 3329 42.5369 0 - 1.39528 2014-04-01 14:42:36,686.686 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 1, 5] out_osds: [] dead_osds: [4, 1] live_osds: [0, 3, 5, 2] 2014-04-01 14:42:36,687.687 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:42:36,687.687 INFO:teuthology.task.thrashosds.thrasher:Setting osd 3 primary_affinity to 1.000000 2014-04-01 14:42:36,687.687 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 3 1' 2014-04-01 14:42:37,511.511 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 314 16 3345 3329 42.4015 0 - 1.39528 2014-04-01 14:42:38,512.512 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 315 16 3345 3329 42.2669 0 - 1.39528 2014-04-01 14:42:38,587.587 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set osd.3 primary-affinity to 1 (8655362) 2014-04-01 14:42:39,525.525 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 316 16 3345 3329 42.1331 0 - 1.39528 2014-04-01 14:42:40,510.510 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 317 16 3345 3329 42.0002 0 - 1.39528 2014-04-01 14:42:41,510.510 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 318 16 3345 3329 41.8681 0 - 1.39528 2014-04-01 14:42:42,510.510 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 319 16 3345 3329 41.7369 0 - 1.39528 2014-04-01 14:42:43,512.512 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:42:43.428125min lat: 0.064365 max lat: 22.6432 avg lat: 1.39528 2014-04-01 14:42:43,512.512 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:42:43,513.513 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 320 16 3345 3329 41.6064 0 - 1.39528 2014-04-01 14:42:43,597.597 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 1, 5] out_osds: [] dead_osds: [4, 1] live_osds: [0, 3, 5, 2] 2014-04-01 14:42:43,597.597 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:42:43,597.597 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 5 2014-04-01 14:42:43,598.598 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 14:42:43,598.598 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 14:42:43,598.598 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3' 2014-04-01 14:42:44,512.512 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 321 16 3345 3329 41.4768 0 - 1.39528 2014-04-01 14:42:45,512.512 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 322 16 3345 3329 41.348 0 - 1.39528 2014-04-01 14:42:46,512.512 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 323 16 3345 3329 41.22 0 - 1.39528 2014-04-01 14:42:47,512.512 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 324 16 3345 3329 41.0928 0 - 1.39528 2014-04-01 14:42:48,512.512 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 325 16 3345 3329 40.9664 0 - 1.39528 2014-04-01 14:42:49,028.028 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 1, 5] out_osds: [] dead_osds: [4, 1] live_osds: [0, 3, 5, 2] 2014-04-01 14:42:49,028.028 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:42:49,029.029 INFO:teuthology.task.thrashosds.thrasher:Reviving osd 4 2014-04-01 14:42:49,029.029 INFO:teuthology.task.ceph.osd.4:Restarting 2014-04-01 14:42:49,029.029 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 4' 2014-04-01 14:42:49,347.347 INFO:teuthology.task.ceph.osd.4:Started 2014-04-01 14:42:49,347.347 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight' 2014-04-01 14:42:49,506.506 INFO:teuthology.task.ceph.osd.4.out:[10.214.133.38]: starting osd.4 at :/0 osd_data /var/lib/ceph/osd/ceph-4 /var/lib/ceph/osd/ceph-4/journal 2014-04-01 14:42:49,513.513 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 326 16 3345 3329 40.8407 0 - 1.39528 2014-04-01 14:42:49,767.767 INFO:teuthology.orchestra.run.err:[10.214.133.38]: no valid command found; 10 closest matches: 2014-04-01 14:42:49,768.768 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config show 2014-04-01 14:42:49,768.768 INFO:teuthology.orchestra.run.err:[10.214.133.38]: help 2014-04-01 14:42:49,768.768 INFO:teuthology.orchestra.run.err:[10.214.133.38]: log dump 2014-04-01 14:42:49,768.768 INFO:teuthology.orchestra.run.err:[10.214.133.38]: get_command_descriptions 2014-04-01 14:42:49,768.768 INFO:teuthology.orchestra.run.err:[10.214.133.38]: git_version 2014-04-01 14:42:49,768.768 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config set [...] 2014-04-01 14:42:49,769.769 INFO:teuthology.orchestra.run.err:[10.214.133.38]: version 2014-04-01 14:42:49,769.769 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2 2014-04-01 14:42:49,769.769 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config get 2014-04-01 14:42:49,772.772 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 0 2014-04-01 14:42:49,772.772 INFO:teuthology.orchestra.run.err:[10.214.133.38]: admin_socket: invalid command 2014-04-01 14:42:49,772.772 INFO:teuthology.task.thrashosds.ceph_manager:waiting on admin_socket for 4, ['dump_ops_in_flight'] 2014-04-01 14:42:49,978.978 INFO:teuthology.task.ceph.osd.4.err:[10.214.133.38]: 2014-04-01 05:42:49.894690 7f7f39ae3780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 14:42:50,513.513 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 327 16 3346 3330 40.728 0.121212 34.0269 1.40507 2014-04-01 14:42:51,512.512 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 328 16 3346 3330 40.6039 0 - 1.40507 2014-04-01 14:42:52,512.512 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 329 16 3346 3330 40.4804 0 - 1.40507 2014-04-01 14:42:53,513.513 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 330 16 3346 3330 40.3578 0 - 1.40507 2014-04-01 14:42:54,513.513 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 331 16 3346 3330 40.2359 0 - 1.40507 2014-04-01 14:42:54,773.773 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight' 2014-04-01 14:42:55,514.514 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 332 16 3346 3330 40.1147 0 - 1.40507 2014-04-01 14:42:56,514.514 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 333 16 3346 3330 39.9942 0 - 1.40507 2014-04-01 14:42:57,514.514 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 334 16 3346 3330 39.8744 0 - 1.40507 2014-04-01 14:42:58,514.514 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 335 16 3346 3330 39.7554 0 - 1.40507 2014-04-01 14:42:59,514.514 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 336 16 3346 3330 39.6371 0 - 1.40507 2014-04-01 14:43:00,198.198 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 1, 5] out_osds: [] dead_osds: [1] live_osds: [0, 3, 5, 2, 4] 2014-04-01 14:43:00,199.199 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:43:00,199.199 INFO:teuthology.task.thrashosds.thrasher:Growing pool data 2014-04-01 14:43:00,200.200 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:43:00,514.514 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 337 16 3346 3330 39.5195 0 - 1.40507 2014-04-01 14:43:01,008.008 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:43:01,514.514 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 338 16 3346 3330 39.4026 0 - 1.40507 2014-04-01 14:43:01,644.644 INFO:teuthology.task.thrashosds.ceph_manager:increase pool size by 10 2014-04-01 14:43:01,644.644 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set data pg_num 44' 2014-04-01 14:43:02,514.514 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 339 16 3346 3330 39.2863 0 - 1.40507 2014-04-01 14:43:02,660.660 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 0 pg_num to 44 2014-04-01 14:43:03,519.519 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:43:03.431059min lat: 0.064365 max lat: 34.0269 avg lat: 1.40507 2014-04-01 14:43:03,519.519 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:43:03,519.519 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 340 16 3346 3330 39.1708 0 - 1.40507 2014-04-01 14:43:04,515.515 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 341 16 3346 3330 39.0559 0 - 1.40507 2014-04-01 14:43:05,515.515 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 342 16 3346 3330 38.9417 0 - 1.40507 2014-04-01 14:43:06,514.514 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 343 16 3346 3330 38.8282 0 - 1.40507 2014-04-01 14:43:07,514.514 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 344 16 3346 3330 38.7153 0 - 1.40507 2014-04-01 14:43:07,671.671 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 1, 5] out_osds: [] dead_osds: [1] live_osds: [0, 3, 5, 2, 4] 2014-04-01 14:43:07,672.672 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:43:07,672.672 INFO:teuthology.task.thrashosds.thrasher:Growing pool unique_pool_0 2014-04-01 14:43:07,672.672 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:43:08,391.391 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:43:08,548.548 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 345 16 3346 3330 38.6031 0 - 1.40507 2014-04-01 14:43:09,516.516 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 346 16 3346 3330 38.4915 0 - 1.40507 2014-04-01 14:43:10,516.516 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 347 16 3346 3330 38.3806 0 - 1.40507 2014-04-01 14:43:11,517.517 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 348 16 3346 3330 38.2703 0 - 1.40507 2014-04-01 14:43:12,516.516 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 349 16 3346 3330 38.1606 0 - 1.40507 2014-04-01 14:43:13,516.516 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 350 16 3346 3330 38.0516 0 - 1.40507 2014-04-01 14:43:14,158.158 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 1, 5] out_osds: [] dead_osds: [1] live_osds: [0, 3, 5, 2, 4] 2014-04-01 14:43:14,160.160 INFO:teuthology.task.thrashosds.thrasher:Reviving osd 1 2014-04-01 14:43:14,160.160 INFO:teuthology.task.ceph.osd.1:Restarting 2014-04-01 14:43:14,161.161 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 1' 2014-04-01 14:43:14,480.480 INFO:teuthology.task.ceph.osd.1:Started 2014-04-01 14:43:14,480.480 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight' 2014-04-01 14:43:14,515.515 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 351 16 3346 3330 37.9432 0 - 1.40507 2014-04-01 14:43:14,587.587 INFO:teuthology.task.ceph.osd.1.out:[10.214.133.37]: starting osd.1 at :/0 osd_data /var/lib/ceph/osd/ceph-1 /var/lib/ceph/osd/ceph-1/journal 2014-04-01 14:43:14,895.895 INFO:teuthology.orchestra.run.err:[10.214.133.37]: no valid command found; 10 closest matches: 2014-04-01 14:43:14,896.896 INFO:teuthology.orchestra.run.err:[10.214.133.37]: config show 2014-04-01 14:43:14,896.896 INFO:teuthology.orchestra.run.err:[10.214.133.37]: help 2014-04-01 14:43:14,896.896 INFO:teuthology.orchestra.run.err:[10.214.133.37]: log dump 2014-04-01 14:43:14,897.897 INFO:teuthology.orchestra.run.err:[10.214.133.37]: get_command_descriptions 2014-04-01 14:43:14,897.897 INFO:teuthology.orchestra.run.err:[10.214.133.37]: git_version 2014-04-01 14:43:14,897.897 INFO:teuthology.orchestra.run.err:[10.214.133.37]: config set [...] 2014-04-01 14:43:14,898.898 INFO:teuthology.orchestra.run.err:[10.214.133.37]: version 2014-04-01 14:43:14,898.898 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2 2014-04-01 14:43:14,899.899 INFO:teuthology.orchestra.run.err:[10.214.133.37]: config get 2014-04-01 14:43:14,899.899 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 0 2014-04-01 14:43:14,899.899 INFO:teuthology.orchestra.run.err:[10.214.133.37]: admin_socket: invalid command 2014-04-01 14:43:14,901.901 INFO:teuthology.task.thrashosds.ceph_manager:waiting on admin_socket for 1, ['dump_ops_in_flight'] 2014-04-01 14:43:15,228.228 INFO:teuthology.task.ceph.osd.1.err:[10.214.133.37]: 2014-04-01 05:43:15.143876 7fc5ce330780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 14:43:15,517.517 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 352 16 3346 3330 37.8354 0 - 1.40507 2014-04-01 14:43:16,517.517 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 353 16 3346 3330 37.7282 0 - 1.40507 2014-04-01 14:43:17,519.519 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 354 16 3346 3330 37.6216 0 - 1.40507 2014-04-01 14:43:18,518.518 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 355 16 3346 3330 37.5156 0 - 1.40507 2014-04-01 14:43:19,517.517 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 356 16 3346 3330 37.4103 0 - 1.40507 2014-04-01 14:43:19,901.901 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight' 2014-04-01 14:43:20,327.327 INFO:teuthology.task.thrashosds.ceph_manager:waiting for recovery to complete 2014-04-01 14:43:20,327.327 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:43:20,518.518 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 357 16 3346 3330 37.3055 0 - 1.40507 2014-04-01 14:43:21,105.105 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:43:21,518.518 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 358 16 3346 3330 37.2013 0 - 1.40507 2014-04-01 14:43:21,881.881 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:43:22,517.517 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 359 16 3346 3330 37.0977 0 - 1.40507 2014-04-01 14:43:22,703.703 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:43:23,327.327 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:43:23,518.518 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:43:23.434091min lat: 0.064365 max lat: 34.0269 avg lat: 1.40507 2014-04-01 14:43:23,519.519 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:43:23,519.519 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 360 16 3346 3330 36.9946 0 - 1.40507 2014-04-01 14:43:24,038.038 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.128820', u'skew': u'-0.286603', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.128723', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:42:42.637108', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438934428, u'health': u'HEALTH_OK', u'kb_used': 9394556, u'store_stats': {u'bytes_total': 9439130, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:42:42.976096', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438876476, u'health': u'HEALTH_OK', u'kb_used': 9452508, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:42:43.036177', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438934352, u'health': u'HEALTH_OK', u'kb_used': 9394632, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs down'}, {u'severity': u'HEALTH_WARN', u'summary': u'29 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'7 pgs stale'}, {u'severity': u'HEALTH_WARN', u'summary': u'7 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'22 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'15 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 44 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'num_pgs': 138, u'data_bytes': 22674409794, u'bytes_used': 21918834688, u'version': 210, u'pgs_by_state': [{u'count': 48, u'state_name': u'inactive'}, {u'count': 1, u'state_name': u'creating'}, {u'count': 7, u'state_name': u'stale+active+clean'}, {u'count': 24, u'state_name': u'peering'}, {u'count': 6, u'state_name': u'active+remapped'}, {u'count': 47, u'state_name': u'active+clean'}, {u'count': 4, u'state_name': u'remapped+peering'}, {u'count': 1, u'state_name': u'down+peering'}], u'bytes_avail': 2780696600576}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 77, u'num_in_osds': 6}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:43:24,038.038 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:43:24,518.518 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 361 16 3346 3330 36.8921 0 - 1.40507 2014-04-01 14:43:24,735.735 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:43:25,518.518 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 362 16 3346 3330 36.7902 0 - 1.40507 2014-04-01 14:43:26,519.519 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 363 16 3346 3330 36.6889 0 - 1.40507 2014-04-01 14:43:27,519.519 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 364 16 3346 3330 36.5881 0 - 1.40507 2014-04-01 14:43:28,519.519 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:43:28,520.520 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 365 16 3346 3330 36.4878 0 - 1.40507 2014-04-01 14:43:29,326.326 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:43:29,521.521 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 366 16 3346 3330 36.3881 0 - 1.40507 2014-04-01 14:43:29,953.953 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:43:30,519.519 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 367 16 3346 3330 36.289 0 - 1.40507 2014-04-01 14:43:30,829.829 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.128820', u'skew': u'-0.286603', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.128723', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:42:42.637108', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438934428, u'health': u'HEALTH_OK', u'kb_used': 9394556, u'store_stats': {u'bytes_total': 9439130, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:42:42.976096', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438876476, u'health': u'HEALTH_OK', u'kb_used': 9452508, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:42:43.036177', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438934352, u'health': u'HEALTH_OK', u'kb_used': 9394632, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'4 pgs degraded'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs down'}, {u'severity': u'HEALTH_WARN', u'summary': u'39 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'22 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'15 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 44 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'num_pgs': 138, u'data_bytes': 22674409794, u'bytes_used': 22299885568, u'version': 214, u'pgs_by_state': [{u'count': 1, u'state_name': u'replay+degraded'}, {u'count': 48, u'state_name': u'inactive'}, {u'count': 3, u'state_name': u'degraded'}, {u'count': 1, u'state_name': u'creating'}, {u'count': 34, u'state_name': u'peering'}, {u'count': 46, u'state_name': u'active+clean'}, {u'count': 4, u'state_name': u'remapped+peering'}, {u'count': 1, u'state_name': u'down+peering'}], u'bytes_avail': 2780315549696}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 78, u'num_in_osds': 6}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:43:30,830.830 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:43:31,525.525 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 368 16 3352 3336 36.2556 0.585366 0.146284 1.48199 2014-04-01 14:43:31,762.762 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:43:32,565.565 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 369 16 3359 3343 36.2332 28 81.6911 1.50368 2014-04-01 14:43:33,596.596 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 370 16 3360 3344 36.1461 4 80.97 1.52744 2014-04-01 14:43:34,529.529 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 371 16 3360 3344 36.0487 0 - 1.52744 2014-04-01 14:43:35,564.564 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 372 16 3360 3344 35.9518 0 - 1.52744 2014-04-01 14:43:35,601.601 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:43:36,451.451 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:43:36,614.614 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 373 16 3360 3344 35.8554 0 - 1.52744 2014-04-01 14:43:37,529.529 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 374 16 3361 3345 35.7702 1 84.0831 1.55212 2014-04-01 14:43:37,608.608 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:43:38,418.418 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.128820', u'skew': u'-0.286603', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.128723', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:42:42.637108', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438934428, u'health': u'HEALTH_OK', u'kb_used': 9394556, u'store_stats': {u'bytes_total': 9439130, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:42:42.976096', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438876476, u'health': u'HEALTH_OK', u'kb_used': 9452508, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:42:43.036177', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438934352, u'health': u'HEALTH_OK', u'kb_used': 9394632, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'21 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 pgs stuck inactive'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 87/16338 objects degraded (0.533%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 44 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'recovering_bytes_per_sec': 28, u'degraded_objects': 87, u'num_pgs': 138, u'recovering_keys_per_sec': 9, u'data_bytes': 22766684516, u'degraded_total': 16338, u'bytes_used': 22345748480, u'recovering_objects_per_sec': 1, u'version': 219, u'pgs_by_state': [{u'count': 33, u'state_name': u'inactive'}, {u'count': 21, u'state_name': u'peering'}, {u'count': 76, u'state_name': u'active+clean'}, {u'count': 6, u'state_name': u'active+recovering'}, {u'count': 2, u'state_name': u'replay'}], u'degraded_ratio': u'0.533', u'bytes_avail': 2780269686784}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 6}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:43:38,419.419 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:43:38,653.653 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 375 16 3362 3346 35.6855 4 86.717 1.57758 2014-04-01 14:43:39,352.352 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:43:39,764.764 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 376 16 3362 3346 35.5906 0 - 1.57758 2014-04-01 14:43:40,578.578 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 377 16 3362 3346 35.4962 0 - 1.57758 2014-04-01 14:43:41,522.522 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 378 16 3380 3364 35.5927 24 0.211381 1.71436 2014-04-01 14:43:42,522.522 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 379 16 3403 3387 35.7415 92 0.935246 1.75638 2014-04-01 14:43:43,221.221 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:43:43,525.525 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:43:43.438961min lat: 0.064365 max lat: 110.858 avg lat: 1.78089 2014-04-01 14:43:43,526.526 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:43:43,526.526 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 380 16 3427 3411 35.8998 96 0.889275 1.78089 2014-04-01 14:43:43,799.799 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:43:44,577.577 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 381 16 3454 3438 36.089 108 0.658598 1.77166 2014-04-01 14:43:44,735.735 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:43:45,502.502 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.128820', u'skew': u'-0.286603', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.128723', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:43:42.637341', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438904028, u'health': u'HEALTH_OK', u'kb_used': 9424956, u'store_stats': {u'bytes_total': 11536282, u'bytes_log': 11468800, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:43:42.976330', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438838048, u'health': u'HEALTH_OK', u'kb_used': 9490936, u'store_stats': {u'bytes_total': 16779162, u'bytes_log': 16711680, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:43:43.036437', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438902888, u'health': u'HEALTH_OK', u'kb_used': 9426096, u'store_stats': {u'bytes_total': 16779162, u'bytes_log': 16711680, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'6 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 9/16467 objects degraded (0.055%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 44 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2952748449792, u'degraded_objects': 9, u'num_pgs': 138, u'data_bytes': 22947039588, u'degraded_total': 16467, u'bytes_used': 22310125568, u'op_per_sec': 17, u'write_bytes_sec': 74675786, u'version': 223, u'pgs_by_state': [{u'count': 2, u'state_name': u'active'}, {u'count': 6, u'state_name': u'active+clean+replay'}, {u'count': 130, u'state_name': u'active+clean'}], u'degraded_ratio': u'0.055', u'bytes_avail': 2780305309696}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 80, u'num_in_osds': 6}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:43:45,503.503 INFO:teuthology.task.thrashosds.ceph_manager:recovered! 2014-04-01 14:43:45,503.503 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:43:45,503.503 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 2 2014-04-01 14:43:45,503.503 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 14:43:45,503.503 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 14:43:45,504.504 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3' 2014-04-01 14:43:45,525.525 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 382 16 3467 3451 36.1306 52 0.576946 1.76722 2014-04-01 14:43:46,525.525 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 383 16 3469 3453 36.0572 8 0.215549 1.76645 2014-04-01 14:43:47,524.524 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 384 16 3469 3453 35.9633 0 - 1.76645 2014-04-01 14:43:48,524.524 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 385 16 3469 3453 35.8699 0 - 1.76645 2014-04-01 14:43:49,527.527 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 386 16 3497 3481 36.0671 37.3333 0.373675 1.77323 2014-04-01 14:43:50,527.527 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 387 16 3525 3509 36.2632 112 0.284409 1.76424 2014-04-01 14:43:51,065.065 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 1, 5] out_osds: [] dead_osds: [] live_osds: [0, 3, 5, 2, 4, 1] 2014-04-01 14:43:51,066.066 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:43:51,067.067 INFO:teuthology.task.thrashosds.thrasher:Killing osd 5, live_osds are [0, 3, 5, 2, 4, 1] 2014-04-01 14:43:51,067.067 DEBUG:teuthology.task.ceph.osd.5:waiting for process to exit 2014-04-01 14:43:51,524.524 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 388 16 3546 3530 36.3862 84 0.34316 1.75747 2014-04-01 14:43:51,646.646 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 05:43:51.561852 7fe3a3ee1700 0 -- 10.214.133.37:0/1005437 >> 10.214.133.38:6810/4408 pipe(0x7fe3ac017620 sd=13 :0 s=1 pgs=0 cs=0 l=1 c=0x7fe3ac0071a0).fault 2014-04-01 14:43:51,651.651 INFO:teuthology.task.ceph.osd.5:Stopped 2014-04-01 14:43:52,526.526 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 389 16 3554 3538 36.3749 32 0.166474 1.75458 2014-04-01 14:43:53,525.525 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 390 16 3558 3542 36.3227 16 0.132509 1.75291 2014-04-01 14:43:54,524.524 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 391 16 3558 3542 36.2298 0 - 1.75291 2014-04-01 14:43:55,525.525 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 392 16 3558 3542 36.1374 0 - 1.75291 2014-04-01 14:43:56,525.525 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 393 16 3558 3542 36.0454 0 - 1.75291 2014-04-01 14:43:56,651.651 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 1, 5] out_osds: [] dead_osds: [5] live_osds: [0, 3, 2, 4, 1] 2014-04-01 14:43:56,651.651 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:43:56,652.652 INFO:teuthology.task.thrashosds.thrasher:Growing pool data 2014-04-01 14:43:56,652.652 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:43:57,563.563 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:43:57,723.723 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 394 16 3558 3542 35.9539 0 - 1.75291 2014-04-01 14:43:58,345.345 INFO:teuthology.task.thrashosds.ceph_manager:increase pool size by 10 2014-04-01 14:43:58,345.345 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set data pg_num 54' 2014-04-01 14:43:58,525.525 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 395 16 3558 3542 35.8629 0 - 1.75291 2014-04-01 14:43:59,526.526 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 396 16 3558 3542 35.7723 0 - 1.75291 2014-04-01 14:44:00,514.514 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 0 pg_num to 54 2014-04-01 14:44:00,525.525 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 397 16 3558 3542 35.6822 0 - 1.75291 2014-04-01 14:44:01,527.527 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 398 16 3558 3542 35.5926 0 - 1.75291 2014-04-01 14:44:02,526.526 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 399 16 3558 3542 35.5034 0 - 1.75291 2014-04-01 14:44:03,527.527 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:44:03.442348min lat: 0.064365 max lat: 110.858 avg lat: 1.75291 2014-04-01 14:44:03,527.527 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:44:03,527.527 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 400 16 3558 3542 35.4146 0 - 1.75291 2014-04-01 14:44:04,527.527 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 401 16 3558 3542 35.3263 0 - 1.75291 2014-04-01 14:44:05,522.522 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 1, 5] out_osds: [] dead_osds: [5] live_osds: [0, 3, 2, 4, 1] 2014-04-01 14:44:05,523.523 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:44:05,524.524 INFO:teuthology.task.thrashosds.thrasher:Growing pool metadata 2014-04-01 14:44:05,525.525 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:44:05,527.527 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 402 16 3558 3542 35.2384 0 - 1.75291 2014-04-01 14:44:06,144.144 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:44:06,604.604 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 403 16 3558 3542 35.151 0 - 1.75291 2014-04-01 14:44:07,533.533 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 404 16 3558 3542 35.064 0 - 1.75291 2014-04-01 14:44:08,527.527 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 405 16 3558 3542 34.9774 0 - 1.75291 2014-04-01 14:44:09,527.527 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 406 16 3558 3542 34.8912 0 - 1.75291 2014-04-01 14:44:10,528.528 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 407 16 3558 3542 34.8055 0 - 1.75291 2014-04-01 14:44:11,528.528 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 408 16 3558 3542 34.7202 0 - 1.75291 2014-04-01 14:44:12,073.073 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 1, 5] out_osds: [] dead_osds: [5] live_osds: [0, 3, 2, 4, 1] 2014-04-01 14:44:12,073.073 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:44:12,073.073 INFO:teuthology.task.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 2, 3, 4, 1, 5] 2014-04-01 14:44:12,074.074 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 1' 2014-04-01 14:44:12,528.528 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 409 16 3558 3542 34.6353 0 - 1.75291 2014-04-01 14:44:13,001.001 INFO:teuthology.orchestra.run.err:[10.214.133.37]: marked out osd.1. 2014-04-01 14:44:13,198.198 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 05:44:13.112433 7fe3a7efe700 0 -- 10.214.133.37:0/1005437 >> 10.214.133.38:6800/4373 pipe(0x7fe3ac006de0 sd=9 :46159 s=2 pgs=8 cs=1 l=1 c=0x7fe3ac007040).injecting socket failure 2014-04-01 14:44:13,529.529 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 410 16 3572 3556 34.6874 2.8 21.1878 1.82497 2014-04-01 14:44:14,529.529 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 411 16 3584 3568 34.7198 48 0.122858 1.82024 2014-04-01 14:44:15,530.530 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 412 16 3597 3581 34.7617 52 2.21381 1.82018 2014-04-01 14:44:16,532.532 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 413 16 3612 3596 34.8228 60 0.462964 1.8158 2014-04-01 14:44:17,530.530 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 414 16 3624 3608 34.8546 48 1.05312 1.81326 2014-04-01 14:44:18,016.016 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5] out_osds: [1] dead_osds: [5] live_osds: [0, 3, 2, 4, 1] 2014-04-01 14:44:18,017.017 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:44:18,018.018 INFO:teuthology.task.thrashosds.thrasher:Growing pool data 2014-04-01 14:44:18,018.018 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:44:18,530.530 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 415 16 3657 3641 35.0887 132 0.418355 1.80815 2014-04-01 14:44:18,680.680 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:44:19,588.588 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 416 16 3664 3648 35.0716 28 1.09183 1.81316 2014-04-01 14:44:19,777.777 INFO:teuthology.task.thrashosds.ceph_manager:increase pool size by 10 2014-04-01 14:44:19,777.777 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set data pg_num 64' 2014-04-01 14:44:20,528.528 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 417 16 3675 3659 35.093 44 0.542984 1.819 2014-04-01 14:44:21,399.399 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 0 pg_num to 64 2014-04-01 14:44:21,529.529 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 418 16 3675 3659 35.009 0 - 1.819 2014-04-01 14:44:22,529.529 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 419 16 3675 3659 34.9255 0 - 1.819 2014-04-01 14:44:23,530.530 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:44:23.445263min lat: 0.064365 max lat: 110.858 avg lat: 1.819 2014-04-01 14:44:23,530.530 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:44:23,530.530 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 420 16 3675 3659 34.8423 0 - 1.819 2014-04-01 14:44:24,530.530 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 421 16 3675 3659 34.7596 0 - 1.819 2014-04-01 14:44:25,530.530 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 422 16 3678 3662 34.7056 2.4 0.287173 1.82063 2014-04-01 14:44:26,408.408 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5] out_osds: [1] dead_osds: [5] live_osds: [0, 3, 2, 4, 1] 2014-04-01 14:44:26,409.409 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:44:26,409.409 INFO:teuthology.task.thrashosds.thrasher:Growing pool rbd 2014-04-01 14:44:26,410.410 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:44:26,530.530 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 423 16 3702 3686 34.8505 96 0.25013 1.8337 2014-04-01 14:44:27,292.292 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:44:27,551.551 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 424 16 3702 3686 34.7683 0 - 1.8337 2014-04-01 14:44:28,532.532 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 425 16 3712 3696 34.7806 20 2.13389 1.83397 2014-04-01 14:44:29,533.533 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 426 16 3736 3720 34.9243 96 0.750036 1.83144 2014-04-01 14:44:30,530.530 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 427 16 3749 3733 34.9643 52 0.745899 1.82701 2014-04-01 14:44:31,531.531 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 428 16 3771 3755 35.0881 88 0.137964 1.82069 2014-04-01 14:44:32,530.530 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 429 16 3771 3755 35.0063 0 - 1.82069 2014-04-01 14:44:33,061.061 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5] out_osds: [1] dead_osds: [5] live_osds: [0, 3, 2, 4, 1] 2014-04-01 14:44:33,061.061 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:44:33,062.062 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 1 2014-04-01 14:44:33,062.062 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 14:44:33,062.062 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 14:44:33,063.063 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3' 2014-04-01 14:44:33,531.531 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 430 16 3771 3755 34.9249 0 - 1.82069 2014-04-01 14:44:34,531.531 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 431 16 3771 3755 34.8439 0 - 1.82069 2014-04-01 14:44:35,531.531 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 432 16 3777 3761 34.8188 6 4.65308 1.82511 2014-04-01 14:44:36,531.531 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 433 16 3777 3761 34.7384 0 - 1.82511 2014-04-01 14:44:37,532.532 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 434 16 3798 3782 34.8519 42 0.615195 1.83417 2014-04-01 14:44:38,535.535 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 435 16 3815 3799 34.928 68 1.13433 1.83081 2014-04-01 14:44:38,759.759 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5] out_osds: [1] dead_osds: [5] live_osds: [0, 3, 2, 4, 1] 2014-04-01 14:44:38,759.759 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:44:38,759.759 INFO:teuthology.task.thrashosds.thrasher:Setting osd 5 primary_affinity to 1.000000 2014-04-01 14:44:38,760.760 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 5 1' 2014-04-01 14:44:39,532.532 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 436 16 3824 3808 34.9305 36 1.4058 1.82821 2014-04-01 14:44:39,725.725 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set osd.5 primary-affinity to 1 (8655362) 2014-04-01 14:44:40,536.536 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 437 16 3826 3810 34.8689 8 1.6261 1.82801 2014-04-01 14:44:41,533.533 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 438 16 3826 3810 34.7892 0 - 1.82801 2014-04-01 14:44:42,540.540 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 439 16 3836 3820 34.8011 20 4.04751 1.83343 2014-04-01 14:44:43,541.541 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:44:43.448319min lat: 0.064365 max lat: 110.858 avg lat: 1.8345 2014-04-01 14:44:43,541.541 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:44:43,542.542 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 440 16 3846 3830 34.8129 40 4.0928 1.8345 2014-04-01 14:44:44,534.534 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 441 16 3848 3832 34.7521 8 0.925179 1.83403 2014-04-01 14:44:44,735.735 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5] out_osds: [1] dead_osds: [5] live_osds: [0, 3, 2, 4, 1] 2014-04-01 14:44:44,736.736 INFO:teuthology.task.thrashosds.thrasher:Reviving osd 5 2014-04-01 14:44:44,736.736 INFO:teuthology.task.ceph.osd.5:Restarting 2014-04-01 14:44:44,737.737 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 5' 2014-04-01 14:44:45,106.106 INFO:teuthology.task.ceph.osd.5:Started 2014-04-01 14:44:45,107.107 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight' 2014-04-01 14:44:45,163.163 INFO:teuthology.task.ceph.osd.5.out:[10.214.133.38]: starting osd.5 at :/0 osd_data /var/lib/ceph/osd/ceph-5 /var/lib/ceph/osd/ceph-5/journal 2014-04-01 14:44:45,533.533 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 442 16 3848 3832 34.6735 0 - 1.83403 2014-04-01 14:44:45,538.538 INFO:teuthology.orchestra.run.err:[10.214.133.38]: no valid command found; 10 closest matches: 2014-04-01 14:44:45,539.539 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config show 2014-04-01 14:44:45,539.539 INFO:teuthology.orchestra.run.err:[10.214.133.38]: help 2014-04-01 14:44:45,539.539 INFO:teuthology.orchestra.run.err:[10.214.133.38]: log dump 2014-04-01 14:44:45,539.539 INFO:teuthology.orchestra.run.err:[10.214.133.38]: get_command_descriptions 2014-04-01 14:44:45,539.539 INFO:teuthology.orchestra.run.err:[10.214.133.38]: git_version 2014-04-01 14:44:45,540.540 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config set [...] 2014-04-01 14:44:45,540.540 INFO:teuthology.orchestra.run.err:[10.214.133.38]: version 2014-04-01 14:44:45,540.540 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 2 2014-04-01 14:44:45,540.540 INFO:teuthology.orchestra.run.err:[10.214.133.38]: config get 2014-04-01 14:44:45,545.545 INFO:teuthology.orchestra.run.err:[10.214.133.38]: 0 2014-04-01 14:44:45,545.545 INFO:teuthology.orchestra.run.err:[10.214.133.38]: admin_socket: invalid command 2014-04-01 14:44:45,546.546 INFO:teuthology.task.thrashosds.ceph_manager:waiting on admin_socket for 5, ['dump_ops_in_flight'] 2014-04-01 14:44:45,725.725 INFO:teuthology.task.ceph.osd.5.err:[10.214.133.38]: 2014-04-01 05:44:45.634435 7f5a6fb59780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 14:44:46,543.543 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 443 16 3848 3832 34.5952 0 - 1.83403 2014-04-01 14:44:47,540.540 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 444 16 3872 3856 34.7335 32 0.355554 1.83983 2014-04-01 14:44:48,533.533 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 445 16 3875 3859 34.6824 12 1.57985 1.83944 2014-04-01 14:44:49,534.534 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 446 16 3875 3859 34.6046 0 - 1.83944 2014-04-01 14:44:50,534.534 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 447 16 3881 3865 34.5809 12 0.190253 1.8389 2014-04-01 14:44:50,546.546 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight' 2014-04-01 14:44:51,163.163 INFO:teuthology.task.thrashosds.ceph_manager:waiting for recovery to complete 2014-04-01 14:44:51,164.164 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:44:51,534.534 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 448 16 3888 3872 34.5662 28 4.10952 1.84111 2014-04-01 14:44:52,023.023 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 05:44:51.938516 7f451a7fc700 0 monclient: hunting for new mon 2014-04-01 14:44:52,342.342 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:44:52,538.538 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 449 16 3896 3880 34.5605 32 5.76373 1.84395 2014-04-01 14:44:53,137.137 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:44:53,535.535 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 450 16 3902 3886 34.537 24 0.133343 1.84191 2014-04-01 14:44:53,913.913 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:44:54,538.538 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 451 16 3904 3888 34.4781 8 6.94063 1.84453 2014-04-01 14:44:54,709.709 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:44:55,489.489 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.128820', u'skew': u'-0.286603', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.128723', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:44:42.666067', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438846052, u'health': u'HEALTH_OK', u'kb_used': 9482932, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:44:42.976579', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438772636, u'health': u'HEALTH_OK', u'kb_used': 9556348, u'store_stats': {u'bytes_total': 19924890, u'bytes_log': 19857408, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:44:43.036710', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438844996, u'health': u'HEALTH_OK', u'kb_used': 9483988, u'store_stats': {u'bytes_total': 19924890, u'bytes_log': 19857408, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'8 pgs degraded'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2391/17856 objects degraded (13.390%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 2391, u'num_pgs': 158, u'data_bytes': 24889002340, u'degraded_total': 17856, u'bytes_used': 20926373888, u'op_per_sec': 8, u'write_bytes_sec': 35626572, u'version': 266, u'pgs_by_state': [{u'count': 8, u'state_name': u'active+degraded'}, {u'count': 2, u'state_name': u'peering'}, {u'count': 7, u'state_name': u'active+remapped'}, {u'count': 4, u'state_name': u'active+recovery_wait'}, {u'count': 126, u'state_name': u'active+clean'}, {u'count': 11, u'state_name': u'active+recovering'}], u'degraded_ratio': u'13.390', u'bytes_avail': 2314586488832}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:44:55,490.490 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:44:55,536.536 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 452 16 3911 3895 34.4638 28 2.35341 1.84939 2014-04-01 14:44:56,200.200 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:44:56,535.535 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 453 16 3915 3899 34.423 16 0.238916 1.84896 2014-04-01 14:44:57,535.535 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 454 16 3917 3901 34.3648 8 1.44587 1.8489 2014-04-01 14:44:58,538.538 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 455 16 3936 3920 34.4563 76 2.9093 1.85678 2014-04-01 14:44:59,539.539 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 456 16 3947 3931 34.4772 44 1.02966 1.85395 2014-04-01 14:44:59,955.955 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:45:00,537.537 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 457 16 3958 3942 34.4981 44 1.1847 1.85188 2014-04-01 14:45:00,540.540 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:45:01,475.475 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:45:01,535.535 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 458 16 3966 3950 34.4926 32 1.9393 1.85128 2014-04-01 14:45:02,209.209 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.128820', u'skew': u'-0.286603', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.128723', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:44:42.666067', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438846052, u'health': u'HEALTH_OK', u'kb_used': 9482932, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:44:42.976579', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438772636, u'health': u'HEALTH_OK', u'kb_used': 9556348, u'store_stats': {u'bytes_total': 19924890, u'bytes_log': 19857408, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:44:43.036710', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438844996, u'health': u'HEALTH_OK', u'kb_used': 9483988, u'store_stats': {u'bytes_total': 19924890, u'bytes_log': 19857408, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4167/18099 objects degraded (23.023%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 18283968, u'degraded_objects': 4167, u'num_pgs': 158, u'recovering_keys_per_sec': 0, u'data_bytes': 25228740964, u'degraded_total': 18099, u'bytes_used': 21289369600, u'recovering_objects_per_sec': 4, u'write_bytes_sec': 69479080, u'op_per_sec': 16, u'version': 271, u'pgs_by_state': [{u'count': 12, u'state_name': u'active+recovery_wait'}, {u'count': 133, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'23.023', u'bytes_avail': 2314223493120}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:45:02,211.211 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:45:02,535.535 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 459 16 3966 3950 34.4174 0 - 1.85128 2014-04-01 14:45:02,906.906 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:45:03,536.536 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:45:03.451280min lat: 0.064365 max lat: 110.858 avg lat: 1.85128 2014-04-01 14:45:03,536.536 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:45:03,536.536 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 460 16 3966 3950 34.3426 0 - 1.85128 2014-04-01 14:45:04,537.537 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 461 16 3966 3950 34.2681 0 - 1.85128 2014-04-01 14:45:05,540.540 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 462 16 3971 3955 34.2372 5 4.10499 1.85549 2014-04-01 14:45:06,537.537 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 463 16 3987 3971 34.3015 64 0.649503 1.86314 2014-04-01 14:45:06,819.819 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:45:07,533.533 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:45:07,689.689 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 464 16 3998 3982 34.3224 44 0.625092 1.86013 2014-04-01 14:45:08,318.318 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:45:08,537.537 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 465 16 4005 3989 34.3088 28 2.45852 1.85965 2014-04-01 14:45:09,051.051 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.128820', u'skew': u'-0.286603', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.128723', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:44:42.666067', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438846052, u'health': u'HEALTH_OK', u'kb_used': 9482932, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:44:42.976579', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438772636, u'health': u'HEALTH_OK', u'kb_used': 9556348, u'store_stats': {u'bytes_total': 19924890, u'bytes_log': 19857408, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:44:43.036710', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438844996, u'health': u'HEALTH_OK', u'kb_used': 9483988, u'store_stats': {u'bytes_total': 19924890, u'bytes_log': 19857408, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4168/18186 objects degraded (22.919%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 15290902, u'degraded_objects': 4168, u'num_pgs': 158, u'recovering_keys_per_sec': 0, u'data_bytes': 25350375780, u'degraded_total': 18186, u'bytes_used': 21528367104, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 18819572, u'op_per_sec': 4, u'version': 275, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 135, u'state_name': u'active+clean'}, {u'count': 12, u'state_name': u'active+recovering'}], u'degraded_ratio': u'22.919', u'bytes_avail': 2313984495616}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:45:09,052.052 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:45:09,536.536 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 466 16 4010 3994 34.2781 20 3.27781 1.85976 2014-04-01 14:45:09,931.931 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:45:10,540.540 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 467 16 4010 3994 34.2047 0 - 1.85976 2014-04-01 14:45:11,538.538 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 468 16 4021 4005 34.2256 22 0.491606 1.86425 2014-04-01 14:45:12,537.537 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 469 16 4022 4006 34.1611 4 0.582813 1.86393 2014-04-01 14:45:13,539.539 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 470 16 4026 4010 34.1225 16 5.35146 1.86539 2014-04-01 14:45:13,702.702 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:45:14,426.426 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:45:14,577.577 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 471 16 4029 4013 34.0755 12 6.042 1.86562 2014-04-01 14:45:15,196.196 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:45:15,538.538 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 472 16 4032 4016 34.0288 12 6.8517 1.86855 2014-04-01 14:45:15,990.990 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.128820', u'skew': u'-0.286603', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.128723', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:44:42.666067', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438846052, u'health': u'HEALTH_OK', u'kb_used': 9482932, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:44:42.976579', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438772636, u'health': u'HEALTH_OK', u'kb_used': 9556348, u'store_stats': {u'bytes_total': 19924890, u'bytes_log': 19857408, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:44:43.036710', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438844996, u'health': u'HEALTH_OK', u'kb_used': 9483988, u'store_stats': {u'bytes_total': 19924890, u'bytes_log': 19857408, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4163/18303 objects degraded (22.745%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 13928414, u'degraded_objects': 4163, u'num_pgs': 158, u'recovering_keys_per_sec': 0, u'data_bytes': 25513953636, u'degraded_total': 18303, u'bytes_used': 21836230656, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 16460853, u'op_per_sec': 3, u'version': 280, u'pgs_by_state': [{u'count': 10, u'state_name': u'active+recovery_wait'}, {u'count': 135, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'22.745', u'bytes_avail': 2313676632064}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:45:15,991.991 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:45:16,538.538 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 473 16 4039 4023 34.016 28 1.57579 1.87139 2014-04-01 14:45:16,801.801 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:45:17,716.716 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 474 16 4050 4034 34.0371 44 0.352105 1.87399 2014-04-01 14:45:18,538.538 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 475 16 4062 4046 34.0664 48 0.318707 1.87447 2014-04-01 14:45:19,538.538 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 476 16 4072 4056 34.0789 40 0.805724 1.87277 2014-04-01 14:45:20,539.539 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 477 16 4081 4065 34.0829 36 3.18726 1.87232 2014-04-01 14:45:20,713.713 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:45:21,385.385 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:45:21,693.693 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 478 16 4085 4069 34.0451 16 1.6319 1.87186 2014-04-01 14:45:22,319.319 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:45:22,538.538 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 479 16 4086 4070 33.9823 4 0.723433 1.87158 2014-04-01 14:45:23,022.022 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.128820', u'skew': u'-0.286603', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.128723', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:44:42.666067', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438846052, u'health': u'HEALTH_OK', u'kb_used': 9482932, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:44:42.976579', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438772636, u'health': u'HEALTH_OK', u'kb_used': 9556348, u'store_stats': {u'bytes_total': 19924890, u'bytes_log': 19857408, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:44:43.036710', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438844996, u'health': u'HEALTH_OK', u'kb_used': 9483988, u'store_stats': {u'bytes_total': 19924890, u'bytes_log': 19857408, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4201/18447 objects degraded (22.773%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 16736288, u'degraded_objects': 4201, u'num_pgs': 158, u'recovering_keys_per_sec': 0, u'data_bytes': 25715280228, u'degraded_total': 18447, u'bytes_used': 22144376832, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 36515537, u'op_per_sec': 8, u'version': 284, u'pgs_by_state': [{u'count': 10, u'state_name': u'active+recovery_wait'}, {u'count': 135, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'22.773', u'bytes_avail': 2313368485888}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:45:23,022.022 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:45:23,539.539 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:45:23.454135min lat: 0.064365 max lat: 110.858 avg lat: 1.8741 2014-04-01 14:45:23,540.540 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:45:23,540.540 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 480 16 4090 4074 33.9449 16 1.73218 1.8741 2014-04-01 14:45:23,795.795 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:45:24,539.539 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 481 16 4096 4080 33.9242 24 0.948675 1.87542 2014-04-01 14:45:25,539.539 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 482 16 4098 4082 33.8704 8 5.72145 1.87604 2014-04-01 14:45:26,539.539 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 483 16 4109 4093 33.8914 44 5.61417 1.88184 2014-04-01 14:45:27,539.539 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 484 16 4116 4100 33.8792 28 1.20663 1.88229 2014-04-01 14:45:27,557.557 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:45:28,265.265 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:45:28,541.541 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 485 16 4118 4102 33.8258 8 4.7062 1.88372 2014-04-01 14:45:29,043.043 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:45:29,541.541 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 486 16 4123 4107 33.7974 20 5.17787 1.88529 2014-04-01 14:45:29,979.979 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.128820', u'skew': u'-0.286603', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.128723', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:44:42.666067', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438846052, u'health': u'HEALTH_OK', u'kb_used': 9482932, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:44:42.976579', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438772636, u'health': u'HEALTH_OK', u'kb_used': 9556348, u'store_stats': {u'bytes_total': 19924890, u'bytes_log': 19857408, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:44:43.036710', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438844996, u'health': u'HEALTH_OK', u'kb_used': 9483988, u'store_stats': {u'bytes_total': 19924890, u'bytes_log': 19857408, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4130/18555 objects degraded (22.258%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 17536128, u'degraded_objects': 4130, u'num_pgs': 158, u'recovering_keys_per_sec': 0, u'data_bytes': 25866275172, u'degraded_total': 18555, u'bytes_used': 22395289600, u'recovering_objects_per_sec': 4, u'write_bytes_sec': 15197977, u'op_per_sec': 3, u'version': 289, u'pgs_by_state': [{u'count': 8, u'state_name': u'active+recovery_wait'}, {u'count': 138, u'state_name': u'active+clean'}, {u'count': 12, u'state_name': u'active+recovering'}], u'degraded_ratio': u'22.258', u'bytes_avail': 2313117573120}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:45:29,980.980 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:45:30,543.543 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 487 16 4134 4118 33.8183 44 0.325097 1.88705 2014-04-01 14:45:30,793.793 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:45:31,540.540 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 488 16 4138 4122 33.7818 16 5.19637 1.88671 2014-04-01 14:45:32,540.540 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 489 16 4143 4127 33.7536 20 2.13148 1.88746 2014-04-01 14:45:33,605.605 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 490 16 4154 4138 33.7745 44 2.80249 1.8921 2014-04-01 14:45:34,544.544 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 491 16 4163 4147 33.779 36 3.6483 1.89028 2014-04-01 14:45:34,562.562 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:45:35,581.581 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:45:35,785.785 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 492 16 4169 4153 33.7591 24 3.44485 1.88967 2014-04-01 14:45:36,608.608 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 493 16 4172 4156 33.715 12 2.21087 1.89058 2014-04-01 14:45:36,819.819 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:45:37,628.628 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 494 16 4174 4158 33.6629 8 0.303456 1.89058 2014-04-01 14:45:37,630.630 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.128820', u'skew': u'-0.286603', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.128723', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:44:42.666067', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438846052, u'health': u'HEALTH_OK', u'kb_used': 9482932, u'store_stats': {u'bytes_total': 13633434, u'bytes_log': 13565952, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:44:42.976579', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438772636, u'health': u'HEALTH_OK', u'kb_used': 9556348, u'store_stats': {u'bytes_total': 19924890, u'bytes_log': 19857408, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:44:43.036710', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438844996, u'health': u'HEALTH_OK', u'kb_used': 9483988, u'store_stats': {u'bytes_total': 19924890, u'bytes_log': 19857408, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4140/18705 objects degraded (22.133%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 22026719, u'degraded_objects': 4140, u'num_pgs': 158, u'recovering_keys_per_sec': 0, u'data_bytes': 26075990372, u'degraded_total': 18705, u'bytes_used': 22747111424, u'recovering_objects_per_sec': 5, u'write_bytes_sec': 36711198, u'op_per_sec': 8, u'version': 294, u'pgs_by_state': [{u'count': 8, u'state_name': u'active+recovery_wait'}, {u'count': 138, u'state_name': u'active+clean'}, {u'count': 12, u'state_name': u'active+recovering'}], u'degraded_ratio': u'22.133', u'bytes_avail': 2312765751296}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:45:37,631.631 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:45:38,510.510 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:45:38,669.669 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 495 16 4177 4161 33.6192 12 0.322693 1.89073 2014-04-01 14:45:39,541.541 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 496 16 4185 4169 33.6159 32 5.77691 1.89414 2014-04-01 14:45:40,544.544 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 497 16 4196 4180 33.6368 44 3.49326 1.90053 2014-04-01 14:45:41,541.541 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 498 16 4201 4185 33.6094 20 0.282968 1.89904 2014-04-01 14:45:42,314.314 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:45:42,544.544 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 499 16 4209 4193 33.6062 32 0.729603 1.89797 2014-04-01 14:45:43,026.026 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:45:43,543.543 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:45:43.457056min lat: 0.064365 max lat: 110.858 avg lat: 1.89935 2014-04-01 14:45:43,543.543 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:45:43,543.543 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 500 16 4219 4203 33.6189 40 0.225864 1.89935 2014-04-01 14:45:43,815.815 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:45:44,542.542 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 501 16 4224 4208 33.5917 20 2.62725 1.90014 2014-04-01 14:45:44,594.594 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.128820', u'skew': u'-0.286603', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.128723', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:45:42.666352', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438777628, u'health': u'HEALTH_OK', u'kb_used': 9551356, u'store_stats': {u'bytes_total': 14682010, u'bytes_log': 14614528, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:45:42.976872', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438677272, u'health': u'HEALTH_OK', u'kb_used': 9651712, u'store_stats': {u'bytes_total': 22022042, u'bytes_log': 21954560, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:45:43.036965', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438777428, u'health': u'HEALTH_OK', u'kb_used': 9551556, u'store_stats': {u'bytes_total': 22022042, u'bytes_log': 21954560, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4146/18837 objects degraded (22.010%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 18153881, u'degraded_objects': 4146, u'num_pgs': 158, u'recovering_keys_per_sec': 0, u'data_bytes': 26260539748, u'degraded_total': 18837, u'bytes_used': 23051812864, u'recovering_objects_per_sec': 4, u'write_bytes_sec': 22476234, u'op_per_sec': 5, u'version': 297, u'pgs_by_state': [{u'count': 8, u'state_name': u'active+recovery_wait'}, {u'count': 138, u'state_name': u'active+clean'}, {u'count': 12, u'state_name': u'active+recovering'}], u'degraded_ratio': u'22.010', u'bytes_avail': 2312461049856}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:45:44,595.595 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:45:45,457.457 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:45:45,617.617 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 502 16 4229 4213 33.5647 20 2.30594 1.90059 2014-04-01 14:45:46,542.542 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 503 16 4237 4221 33.5615 32 0.278618 1.89993 2014-04-01 14:45:47,542.542 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 504 16 4243 4227 33.5426 24 3.28308 1.90101 2014-04-01 14:45:48,542.542 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 505 16 4246 4230 33.4999 12 3.05245 1.90163 2014-04-01 14:45:49,234.234 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:45:49,552.552 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 506 16 4258 4242 33.5285 48 2.22711 1.90568 2014-04-01 14:45:49,729.729 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 05:45:49.632475 7f386bfff700 0 monclient: hunting for new mon 2014-04-01 14:45:50,012.012 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:45:50,549.549 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 507 16 4260 4244 33.4782 8 0.964322 1.90516 2014-04-01 14:45:50,869.869 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:45:51,549.549 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 508 16 4267 4251 33.4674 28 0.441991 1.90399 2014-04-01 14:45:51,765.765 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.128820', u'skew': u'-0.286603', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.128723', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:45:42.666352', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438777628, u'health': u'HEALTH_OK', u'kb_used': 9551356, u'store_stats': {u'bytes_total': 14682010, u'bytes_log': 14614528, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:45:42.976872', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438677272, u'health': u'HEALTH_OK', u'kb_used': 9651712, u'store_stats': {u'bytes_total': 22022042, u'bytes_log': 21954560, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:45:43.036965', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438777428, u'health': u'HEALTH_OK', u'kb_used': 9551556, u'store_stats': {u'bytes_total': 22022042, u'bytes_log': 21954560, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4147/18969 objects degraded (21.862%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 17903544, u'degraded_objects': 4147, u'num_pgs': 158, u'recovering_keys_per_sec': 0, u'data_bytes': 26445089124, u'degraded_total': 18969, u'bytes_used': 23391092736, u'recovering_objects_per_sec': 4, u'write_bytes_sec': 22677823, u'op_per_sec': 5, u'version': 302, u'pgs_by_state': [{u'count': 8, u'state_name': u'active+recovery_wait'}, {u'count': 138, u'state_name': u'active+clean'}, {u'count': 12, u'state_name': u'active+recovering'}], u'degraded_ratio': u'21.862', u'bytes_avail': 2312121769984}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:45:51,766.766 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:45:52,544.544 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:45:53,187.187 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 509 16 4271 4255 33.4331 16 3.17754 1.90437 2014-04-01 14:45:53,727.727 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 510 16 4277 4261 33.4146 24 0.255828 1.90513 2014-04-01 14:45:54,552.552 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 511 16 4286 4270 33.4196 36 0.583384 1.90858 2014-04-01 14:45:55,553.553 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 512 16 4290 4274 33.3856 16 1.26752 1.90813 2014-04-01 14:45:56,547.547 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 513 16 4302 4286 33.4141 48 0.37462 1.91083 2014-04-01 14:45:56,917.917 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:45:57,708.708 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:45:57,883.883 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 514 16 4307 4291 33.388 20 3.01911 1.91145 2014-04-01 14:45:58,651.651 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 515 16 4313 4297 33.3697 24 0.25419 1.911 2014-04-01 14:45:58,665.665 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:45:59,377.377 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.128820', u'skew': u'-0.286603', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.128723', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:45:42.666352', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438777628, u'health': u'HEALTH_OK', u'kb_used': 9551356, u'store_stats': {u'bytes_total': 14682010, u'bytes_log': 14614528, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:45:42.976872', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438677272, u'health': u'HEALTH_OK', u'kb_used': 9651712, u'store_stats': {u'bytes_total': 22022042, u'bytes_log': 21954560, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:45:43.036965', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438777428, u'health': u'HEALTH_OK', u'kb_used': 9551556, u'store_stats': {u'bytes_total': 22022042, u'bytes_log': 21954560, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4141/19116 objects degraded (21.662%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 23700482, u'degraded_objects': 4141, u'num_pgs': 158, u'recovering_keys_per_sec': 0, u'data_bytes': 26650610020, u'degraded_total': 19116, u'bytes_used': 23789973504, u'recovering_objects_per_sec': 5, u'write_bytes_sec': 22515458, u'op_per_sec': 5, u'version': 307, u'pgs_by_state': [{u'count': 8, u'state_name': u'active+recovery_wait'}, {u'count': 138, u'state_name': u'active+clean'}, {u'count': 12, u'state_name': u'active+recovering'}], u'degraded_ratio': u'21.662', u'bytes_avail': 2311722889216}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:45:59,378.378 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:45:59,544.544 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 516 16 4313 4297 33.3051 0 - 1.911 2014-04-01 14:46:00,212.212 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:46:00,548.548 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 517 16 4321 4305 33.3025 16 0.699987 1.91331 2014-04-01 14:46:01,544.544 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 518 16 4330 4314 33.3077 36 4.57041 1.91824 2014-04-01 14:46:02,545.545 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 519 16 4335 4319 33.2821 20 1.96734 1.91819 2014-04-01 14:46:03,545.545 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:46:03.459608min lat: 0.064365 max lat: 110.858 avg lat: 1.91729 2014-04-01 14:46:03,545.545 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:46:03,545.545 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 520 16 4342 4326 33.2719 28 0.735761 1.91729 2014-04-01 14:46:04,143.143 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:46:04,547.547 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 521 16 4355 4339 33.3079 52 0.180978 1.91777 2014-04-01 14:46:04,912.912 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:46:05,545.545 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 522 16 4367 4351 33.336 48 0.624515 1.91617 2014-04-01 14:46:05,846.846 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:46:06,546.546 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 523 16 4379 4363 33.364 48 1.65046 1.91512 2014-04-01 14:46:06,611.611 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.128820', u'skew': u'-0.286603', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.128723', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:45:42.666352', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438777628, u'health': u'HEALTH_OK', u'kb_used': 9551356, u'store_stats': {u'bytes_total': 14682010, u'bytes_log': 14614528, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:45:42.976872', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438677272, u'health': u'HEALTH_OK', u'kb_used': 9651712, u'store_stats': {u'bytes_total': 22022042, u'bytes_log': 21954560, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:45:43.036965', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438777428, u'health': u'HEALTH_OK', u'kb_used': 9551556, u'store_stats': {u'bytes_total': 22022042, u'bytes_log': 21954560, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3879/19227 objects degraded (20.175%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 9855923, u'degraded_objects': 3879, u'num_pgs': 158, u'recovering_keys_per_sec': 0, u'data_bytes': 26805799268, u'degraded_total': 19227, u'bytes_used': 24072032256, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 16015875, u'op_per_sec': 3, u'version': 312, u'pgs_by_state': [{u'count': 8, u'state_name': u'active+recovery_wait'}, {u'count': 139, u'state_name': u'active+clean'}, {u'count': 11, u'state_name': u'active+recovering'}], u'degraded_ratio': u'20.175', u'bytes_avail': 2311440830464}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:46:06,611.611 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:46:07,324.324 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:46:07,547.547 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 524 16 4384 4368 33.3385 20 1.57001 1.91496 2014-04-01 14:46:08,545.545 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 525 16 4392 4376 33.336 32 0.164512 1.91404 2014-04-01 14:46:09,545.545 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 526 16 4392 4376 33.2726 0 - 1.91404 2014-04-01 14:46:10,547.547 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 527 16 4402 4386 33.2853 20 2.07102 1.91776 2014-04-01 14:46:11,082.082 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:46:11,546.546 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 528 16 4410 4394 33.2829 32 0.236398 1.91814 2014-04-01 14:46:11,808.808 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:46:12,549.549 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 529 16 4417 4401 33.2729 28 0.991874 1.91834 2014-04-01 14:46:12,578.578 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:46:13,327.327 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.128820', u'skew': u'-0.286603', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.128723', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:45:42.666352', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438777628, u'health': u'HEALTH_OK', u'kb_used': 9551356, u'store_stats': {u'bytes_total': 14682010, u'bytes_log': 14614528, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:45:42.976872', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438677272, u'health': u'HEALTH_OK', u'kb_used': 9651712, u'store_stats': {u'bytes_total': 22022042, u'bytes_log': 21954560, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:45:43.036965', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438777428, u'health': u'HEALTH_OK', u'kb_used': 9551556, u'store_stats': {u'bytes_total': 22022042, u'bytes_log': 21954560, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3896/19428 objects degraded (20.054%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 14565635, u'degraded_objects': 3896, u'num_pgs': 158, u'recovering_keys_per_sec': 0, u'data_bytes': 27086817636, u'degraded_total': 19428, u'bytes_used': 24473124864, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 21848453, u'op_per_sec': 5, u'version': 316, u'pgs_by_state': [{u'count': 6, u'state_name': u'active+recovery_wait'}, {u'count': 140, u'state_name': u'active+clean'}, {u'count': 12, u'state_name': u'active+recovering'}], u'degraded_ratio': u'20.054', u'bytes_avail': 2311039737856}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:46:13,328.328 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:46:13,546.546 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 530 16 4426 4410 33.278 36 0.476732 1.91806 2014-04-01 14:46:14,075.075 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:46:14,548.548 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 531 16 4438 4422 33.3057 48 1.1834 1.91821 2014-04-01 14:46:15,546.546 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 532 16 4447 4431 33.3108 36 2.05754 1.91781 2014-04-01 14:46:16,546.546 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 533 16 4455 4439 33.3083 32 0.923403 1.91756 2014-04-01 14:46:17,546.546 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 534 16 4463 4447 33.3059 32 1.95036 1.91694 2014-04-01 14:46:18,002.002 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:46:18,550.550 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 535 16 4477 4461 33.3483 56 0.574051 1.91619 2014-04-01 14:46:18,910.910 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:46:19,547.547 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 536 16 4479 4463 33.301 8 0.540685 1.91553 2014-04-01 14:46:19,680.680 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:46:20,422.422 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.128820', u'skew': u'-0.286603', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.128723', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:45:42.666352', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438777628, u'health': u'HEALTH_OK', u'kb_used': 9551356, u'store_stats': {u'bytes_total': 14682010, u'bytes_log': 14614528, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:45:42.976872', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438677272, u'health': u'HEALTH_OK', u'kb_used': 9651712, u'store_stats': {u'bytes_total': 22022042, u'bytes_log': 21954560, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:45:43.036965', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438777428, u'health': u'HEALTH_OK', u'kb_used': 9551556, u'store_stats': {u'bytes_total': 22022042, u'bytes_log': 21954560, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'6 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3929/19635 objects degraded (20.010%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 3929, u'num_pgs': 158, u'data_bytes': 27376224612, u'degraded_total': 19635, u'bytes_used': 24762507264, u'op_per_sec': 6, u'write_bytes_sec': 27685780, u'version': 322, u'pgs_by_state': [{u'count': 6, u'state_name': u'active+recovery_wait'}, {u'count': 140, u'state_name': u'active+clean'}, {u'count': 12, u'state_name': u'active+recovering'}], u'degraded_ratio': u'20.010', u'bytes_avail': 2310750355456}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:46:20,423.423 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:46:20,547.547 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 537 16 4491 4475 33.3283 48 1.68341 1.91713 2014-04-01 14:46:21,013.013 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 05:46:20.925282 7f19ec75f700 0 -- :/1010245 >> 10.214.133.37:6790/0 pipe(0x7f19e80193f0 sd=7 :0 s=1 pgs=0 cs=0 l=1 c=0x7f19e8019650).fault 2014-04-01 14:46:21,294.294 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:46:21,609.609 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 538 16 4497 4481 33.311 24 2.46415 1.91746 2014-04-01 14:46:22,548.548 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 539 16 4511 4495 33.3531 56 0.550129 1.91593 2014-04-01 14:46:23,550.550 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:46:23.462415min lat: 0.064365 max lat: 110.858 avg lat: 1.91305 2014-04-01 14:46:23,550.550 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:46:23,550.550 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 540 16 4526 4510 33.4024 60 0.644146 1.91305 2014-04-01 14:46:24,547.547 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 541 16 4537 4521 33.422 44 1.0766 1.91163 2014-04-01 14:46:25,250.250 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:46:25,549.549 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 542 16 4544 4528 33.412 28 2.56015 1.91114 2014-04-01 14:46:25,939.939 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:46:26,548.548 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 543 16 4553 4537 33.4167 36 2.31515 1.9119 2014-04-01 14:46:26,932.932 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:46:27,553.553 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 544 16 4557 4541 33.3847 16 1.99106 1.9117 2014-04-01 14:46:27,707.707 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.128820', u'skew': u'-0.286603', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.128723', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:45:42.666352', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438777628, u'health': u'HEALTH_OK', u'kb_used': 9551356, u'store_stats': {u'bytes_total': 14682010, u'bytes_log': 14614528, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:45:42.976872', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438677272, u'health': u'HEALTH_OK', u'kb_used': 9651712, u'store_stats': {u'bytes_total': 22022042, u'bytes_log': 21954560, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:45:43.036965', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438777428, u'health': u'HEALTH_OK', u'kb_used': 9551556, u'store_stats': {u'bytes_total': 22022042, u'bytes_log': 21954560, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3945/19860 objects degraded (19.864%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 10898056, u'degraded_objects': 3945, u'num_pgs': 158, u'recovering_keys_per_sec': 0, u'data_bytes': 27690797412, u'degraded_total': 19860, u'bytes_used': 25178468352, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 63571994, u'op_per_sec': 15, u'version': 327, u'pgs_by_state': [{u'count': 5, u'state_name': u'active+recovery_wait'}, {u'count': 141, u'state_name': u'active+clean'}, {u'count': 12, u'state_name': u'active+recovering'}], u'degraded_ratio': u'19.864', u'bytes_avail': 2310334394368}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:46:27,707.707 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:46:28,278.278 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 05:46:28.193270 7fc25effd700 0 monclient: hunting for new mon 2014-04-01 14:46:28,433.433 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:46:28,577.577 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 545 16 4564 4548 33.3748 28 0.748106 1.91042 2014-04-01 14:46:29,549.549 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 546 16 4564 4548 33.3137 0 - 1.91042 2014-04-01 14:46:30,550.550 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 547 16 4584 4568 33.399 40 0.626417 1.91513 2014-04-01 14:46:31,548.548 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 548 16 4601 4585 33.4621 68 0.802128 1.91061 2014-04-01 14:46:32,194.194 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:46:32,549.549 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 549 16 4615 4599 33.5032 56 0.971654 1.90772 2014-04-01 14:46:33,550.550 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 550 16 4626 4610 33.5223 44 1.16524 1.90626 2014-04-01 14:46:34,549.549 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 551 16 4629 4613 33.4832 12 1.73085 1.90609 2014-04-01 14:46:35,550.550 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 552 16 4632 4616 33.4443 12 0.494137 1.90529 2014-04-01 14:46:35,757.757 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:46:36,549.549 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 553 16 4632 4616 33.3838 0 - 1.90529 2014-04-01 14:46:36,695.695 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:46:37,550.550 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 554 16 4633 4617 33.3308 2 1.62255 1.90523 2014-04-01 14:46:37,558.558 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.128820', u'skew': u'-0.286603', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.128723', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:45:42.666352', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438777628, u'health': u'HEALTH_OK', u'kb_used': 9551356, u'store_stats': {u'bytes_total': 14682010, u'bytes_log': 14614528, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:45:42.976872', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438677272, u'health': u'HEALTH_OK', u'kb_used': 9651712, u'store_stats': {u'bytes_total': 22022042, u'bytes_log': 21954560, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:45:43.036965', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438777428, u'health': u'HEALTH_OK', u'kb_used': 9551556, u'store_stats': {u'bytes_total': 22022042, u'bytes_log': 21954560, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3949/20136 objects degraded (19.612%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 3949, u'num_pgs': 158, u'data_bytes': 28076673380, u'degraded_total': 20136, u'bytes_used': 25822593024, u'op_per_sec': 14, u'write_bytes_sec': 61917658, u'version': 333, u'pgs_by_state': [{u'count': 4, u'state_name': u'active+recovery_wait'}, {u'count': 142, u'state_name': u'active+clean'}, {u'count': 12, u'state_name': u'active+recovering'}], u'degraded_ratio': u'19.612', u'bytes_avail': 2309690269696}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:46:37,558.558 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:46:38,267.267 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:46:38,552.552 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 555 16 4645 4629 33.3572 48 5.37245 1.91362 2014-04-01 14:46:39,552.552 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 556 16 4663 4647 33.4267 72 0.802057 1.91294 2014-04-01 14:46:40,553.553 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 557 16 4681 4665 33.4959 72 0.51174 1.90956 2014-04-01 14:46:41,553.553 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 558 16 4692 4676 33.5147 44 1.06215 1.90705 2014-04-01 14:46:42,039.039 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:46:42,551.551 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 559 16 4696 4680 33.4834 16 1.71003 1.90657 2014-04-01 14:46:42,751.751 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:46:43,552.552 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:46:43.465270min lat: 0.064365 max lat: 110.858 avg lat: 1.90621 2014-04-01 14:46:43,552.552 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:46:43,552.552 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 560 16 4698 4682 33.4379 8 0.954837 1.90621 2014-04-01 14:46:43,682.682 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:46:44,406.406 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 4, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.128820', u'skew': u'-0.286603', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.128723', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:46:42.666637', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438703548, u'health': u'HEALTH_OK', u'kb_used': 9625436, u'store_stats': {u'bytes_total': 16779162, u'bytes_log': 16711680, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:46:42.977138', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438577788, u'health': u'HEALTH_OK', u'kb_used': 9751196, u'store_stats': {u'bytes_total': 25167770, u'bytes_log': 25100288, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:46:43.037240', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438703352, u'health': u'HEALTH_OK', u'kb_used': 9625632, u'store_stats': {u'bytes_total': 25167770, u'bytes_log': 25100288, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3961/20280 objects degraded (19.532%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 3961, u'num_pgs': 158, u'data_bytes': 28277999972, u'degraded_total': 20280, u'bytes_used': 26082881536, u'op_per_sec': 10, u'write_bytes_sec': 42014486, u'version': 337, u'pgs_by_state': [{u'count': 4, u'state_name': u'active+recovery_wait'}, {u'count': 142, u'state_name': u'active+clean'}, {u'count': 12, u'state_name': u'active+recovering'}], u'degraded_ratio': u'19.532', u'bytes_avail': 2309429981184}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:46:44,406.406 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:46:44,550.550 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 561 16 4699 4683 33.3854 4 1.24288 1.90607 2014-04-01 14:46:45,209.209 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:46:45,551.551 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 562 16 4699 4683 33.326 0 - 1.90607 2014-04-01 14:46:46,552.552 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 563 16 4720 4704 33.416 42 0.546954 1.91373 2014-04-01 14:46:47,551.551 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 564 16 4734 4718 33.456 56 1.32634 1.91108 2014-04-01 14:46:48,551.551 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 565 16 4751 4735 33.5171 68 0.573728 1.90785 2014-04-01 14:46:48,973.973 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:46:49,552.552 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 566 16 4759 4743 33.5144 32 1.07815 1.90671 2014-04-01 14:46:49,714.714 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:46:50,554.554 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 567 16 4761 4745 33.4694 8 1.87712 1.9067 2014-04-01 14:46:50,637.637 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:46:51,345.345 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:46:42.666637', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438703548, u'health': u'HEALTH_OK', u'kb_used': 9625436, u'store_stats': {u'bytes_total': 16779162, u'bytes_log': 16711680, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:46:42.977138', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438577788, u'health': u'HEALTH_OK', u'kb_used': 9751196, u'store_stats': {u'bytes_total': 25167770, u'bytes_log': 25100288, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:46:43.037240', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438703352, u'health': u'HEALTH_OK', u'kb_used': 9625632, u'store_stats': {u'bytes_total': 25167770, u'bytes_log': 25100288, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3967/20448 objects degraded (19.400%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 12361562, u'degraded_objects': 3967, u'num_pgs': 158, u'recovering_keys_per_sec': 0, u'data_bytes': 28512880996, u'degraded_total': 20448, u'bytes_used': 26412138496, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 39556998, u'op_per_sec': 9, u'version': 342, u'pgs_by_state': [{u'count': 4, u'state_name': u'active+recovery_wait'}, {u'count': 142, u'state_name': u'active+clean'}, {u'count': 12, u'state_name': u'active+recovering'}], u'degraded_ratio': u'19.400', u'bytes_avail': 2309100724224}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:46:51,346.346 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:46:51,550.550 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 568 16 4763 4747 33.4246 8 1.70635 1.90677 2014-04-01 14:46:52,050.050 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:46:52,552.552 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 569 16 4766 4750 33.3869 12 1.73566 1.90698 2014-04-01 14:46:53,553.553 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 570 16 4771 4755 33.3634 20 4.30163 1.9102 2014-04-01 14:46:54,557.557 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 571 16 4793 4777 33.4591 88 0.585919 1.91159 2014-04-01 14:46:55,552.552 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 572 16 4797 4781 33.4286 16 1.07673 1.91074 2014-04-01 14:46:55,807.807 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:46:56,525.525 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:46:56,679.679 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 573 16 4813 4797 33.4819 64 1.14427 1.90876 2014-04-01 14:46:57,301.301 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:46:57,553.553 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 574 16 4817 4801 33.4515 16 0.614511 1.90785 2014-04-01 14:46:58,062.062 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:46:42.666637', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438703548, u'health': u'HEALTH_OK', u'kb_used': 9625436, u'store_stats': {u'bytes_total': 16779162, u'bytes_log': 16711680, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:46:42.977138', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438577788, u'health': u'HEALTH_OK', u'kb_used': 9751196, u'store_stats': {u'bytes_total': 25167770, u'bytes_log': 25100288, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:46:43.037240', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438703352, u'health': u'HEALTH_OK', u'kb_used': 9625632, u'store_stats': {u'bytes_total': 25167770, u'bytes_log': 25100288, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3644/20613 objects degraded (17.678%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 3644, u'num_pgs': 158, u'data_bytes': 28743567716, u'degraded_total': 20613, u'bytes_used': 28627623936, u'op_per_sec': 10, u'write_bytes_sec': 44419309, u'version': 346, u'pgs_by_state': [{u'count': 3, u'state_name': u'active+recovery_wait'}, {u'count': 143, u'state_name': u'active+clean'}, {u'count': 12, u'state_name': u'active+recovering'}], u'degraded_ratio': u'17.678', u'bytes_avail': 2306885238784}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:46:58,063.063 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:46:58,559.559 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 575 16 4817 4801 33.3933 0 - 1.90785 2014-04-01 14:46:58,787.787 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:46:59,566.566 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 576 16 4817 4801 33.3353 0 - 1.90785 2014-04-01 14:47:00,557.557 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 577 16 4817 4801 33.2775 0 - 1.90785 2014-04-01 14:47:01,558.558 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 578 16 4841 4825 33.386 24 0.587777 1.9159 2014-04-01 14:47:02,554.554 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 579 16 4858 4842 33.4458 68 0.650154 1.91194 2014-04-01 14:47:02,593.593 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:47:03,373.373 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:47:03,703.703 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:47:03.467847min lat: 0.064365 max lat: 110.858 avg lat: 1.91153 2014-04-01 14:47:03,703.703 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:47:03,704.704 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 580 16 4861 4845 33.4088 12 1.46833 1.91153 2014-04-01 14:47:04,578.578 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 581 16 4869 4853 33.4064 32 0.190905 1.90917 2014-04-01 14:47:04,588.588 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:47:05,357.357 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:46:42.666637', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438703548, u'health': u'HEALTH_OK', u'kb_used': 9625436, u'store_stats': {u'bytes_total': 16779162, u'bytes_log': 16711680, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:46:42.977138', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438577788, u'health': u'HEALTH_OK', u'kb_used': 9751196, u'store_stats': {u'bytes_total': 25167770, u'bytes_log': 25100288, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:46:43.037240', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438703352, u'health': u'HEALTH_OK', u'kb_used': 9625632, u'store_stats': {u'bytes_total': 25167770, u'bytes_log': 25100288, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3638/20775 objects degraded (17.511%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 18898936, u'degraded_objects': 3638, u'num_pgs': 158, u'recovering_keys_per_sec': 0, u'data_bytes': 28970060132, u'degraded_total': 20775, u'bytes_used': 29095706624, u'recovering_objects_per_sec': 4, u'write_bytes_sec': 28978369, u'op_per_sec': 6, u'version': 352, u'pgs_by_state': [{u'count': 3, u'state_name': u'active+recovery_wait'}, {u'count': 143, u'state_name': u'active+clean'}, {u'count': 12, u'state_name': u'active+recovering'}], u'degraded_ratio': u'17.511', u'bytes_avail': 2306417156096}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:47:05,358.358 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:47:05,567.567 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 582 16 4870 4854 33.3559 4 0.600136 1.9089 2014-04-01 14:47:06,135.135 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:47:06,554.554 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 583 16 4871 4855 33.3055 4 0.928066 1.9087 2014-04-01 14:47:07,554.554 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 584 16 4887 4871 33.358 64 3.1666 1.91586 2014-04-01 14:47:08,555.555 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 585 16 4897 4881 33.3694 40 0.639706 1.91433 2014-04-01 14:47:09,895.895 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:47:09,906.906 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 586 16 4909 4893 33.3943 48 1.88832 1.91471 2014-04-01 14:47:10,612.612 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:47:11,380.380 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 587 16 4917 4901 33.392 32 2.62348 1.9135 2014-04-01 14:47:12,150.150 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 588 16 4928 4912 33.41 44 1.17828 1.91261 2014-04-01 14:47:12,814.814 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 589 16 4939 4923 33.428 44 0.910388 1.91113 2014-04-01 14:47:12,822.822 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:47:13,408.408 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:46:42.666637', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438703548, u'health': u'HEALTH_OK', u'kb_used': 9625436, u'store_stats': {u'bytes_total': 16779162, u'bytes_log': 16711680, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:46:42.977138', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438577788, u'health': u'HEALTH_OK', u'kb_used': 9751196, u'store_stats': {u'bytes_total': 25167770, u'bytes_log': 25100288, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:46:43.037240', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438703352, u'health': u'HEALTH_OK', u'kb_used': 9625632, u'store_stats': {u'bytes_total': 25167770, u'bytes_log': 25100288, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3628/20967 objects degraded (17.303%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 3628, u'num_pgs': 158, u'data_bytes': 29238495588, u'degraded_total': 20967, u'bytes_used': 29407928320, u'op_per_sec': 10, u'write_bytes_sec': 42025585, u'version': 357, u'pgs_by_state': [{u'count': 3, u'state_name': u'active+recovery_wait'}, {u'count': 144, u'state_name': u'active+clean'}, {u'count': 11, u'state_name': u'active+recovering'}], u'degraded_ratio': u'17.303', u'bytes_avail': 2306104934400}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:47:13,409.409 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:47:13,555.555 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 590 16 4947 4931 33.4255 32 1.80546 1.91095 2014-04-01 14:47:14,112.112 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:47:14,721.721 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 591 16 4950 4934 33.3893 12 2.23303 1.91086 2014-04-01 14:47:15,556.556 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 592 16 4950 4934 33.3329 0 - 1.91086 2014-04-01 14:47:16,556.556 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 593 16 4950 4934 33.2767 0 - 1.91086 2014-04-01 14:47:17,556.556 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 594 16 4957 4941 33.2678 9.33333 5.58728 1.9145 2014-04-01 14:47:18,066.066 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:47:18,558.558 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 595 16 4977 4961 33.3463 80 0.829557 1.91768 2014-04-01 14:47:18,972.972 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:47:19,674.674 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 596 16 4987 4971 33.3575 40 0.989575 1.91623 2014-04-01 14:47:19,895.895 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:47:21,043.043 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 597 16 4992 4976 33.3351 20 1.38765 1.9156 2014-04-01 14:47:21,557.557 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 598 16 4996 4980 33.3061 16 0.601991 1.91531 2014-04-01 14:47:21,632.632 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:46:42.666637', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438703548, u'health': u'HEALTH_OK', u'kb_used': 9625436, u'store_stats': {u'bytes_total': 16779162, u'bytes_log': 16711680, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:46:42.977138', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438577788, u'health': u'HEALTH_OK', u'kb_used': 9751196, u'store_stats': {u'bytes_total': 25167770, u'bytes_log': 25100288, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:46:43.037240', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438703352, u'health': u'HEALTH_OK', u'kb_used': 9625632, u'store_stats': {u'bytes_total': 25167770, u'bytes_log': 25100288, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3629/21189 objects degraded (17.127%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 3629, u'num_pgs': 158, u'data_bytes': 29548874084, u'degraded_total': 21189, u'bytes_used': 29914771456, u'op_per_sec': 9, u'write_bytes_sec': 40489095, u'version': 363, u'pgs_by_state': [{u'count': 3, u'state_name': u'active+recovery_wait'}, {u'count': 144, u'state_name': u'active+clean'}, {u'count': 11, u'state_name': u'active+recovering'}], u'degraded_ratio': u'17.127', u'bytes_avail': 2305598091264}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:47:21,633.633 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:47:22,446.446 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:47:22,611.611 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 599 16 4998 4982 33.2638 8 0.154851 1.91474 2014-04-01 14:47:23,563.563 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:47:23.470893min lat: 0.064365 max lat: 110.858 avg lat: 1.91835 2014-04-01 14:47:23,564.564 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:47:23,564.564 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 600 16 5006 4990 33.2617 32 4.50315 1.91835 2014-04-01 14:47:24,557.557 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 601 16 5011 4995 33.2396 20 0.094453 1.91787 2014-04-01 14:47:25,557.557 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 602 16 5029 5013 33.304 72 0.617541 1.91899 2014-04-01 14:47:26,378.378 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:47:26,557.557 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 603 16 5038 5022 33.3085 36 2.81785 1.91795 2014-04-01 14:47:27,088.088 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:47:27,582.582 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 604 16 5041 5025 33.2732 12 0.307866 1.91746 2014-04-01 14:47:27,876.876 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:47:28,558.558 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 605 16 5045 5029 33.2446 16 4.29213 1.91787 2014-04-01 14:47:28,646.646 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:46:42.666637', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438703548, u'health': u'HEALTH_OK', u'kb_used': 9625436, u'store_stats': {u'bytes_total': 16779162, u'bytes_log': 16711680, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:46:42.977138', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438577788, u'health': u'HEALTH_OK', u'kb_used': 9751196, u'store_stats': {u'bytes_total': 25167770, u'bytes_log': 25100288, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:46:43.037240', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438703352, u'health': u'HEALTH_OK', u'kb_used': 9625632, u'store_stats': {u'bytes_total': 25167770, u'bytes_log': 25100288, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 3231/21291 objects degraded (15.175%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 3231, u'num_pgs': 158, u'data_bytes': 29691480420, u'degraded_total': 21291, u'bytes_used': 30159163392, u'op_per_sec': 7, u'write_bytes_sec': 29850597, u'version': 367, u'pgs_by_state': [{u'count': 3, u'state_name': u'active+recovery_wait'}, {u'count': 145, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'15.175', u'bytes_avail': 2305353699328}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:47:28,647.647 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:47:29,509.509 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:47:29,664.664 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 606 16 5054 5038 33.2492 36 0.080594 1.91737 2014-04-01 14:47:30,557.557 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 607 16 5062 5046 33.2471 32 0.195049 1.91952 2014-04-01 14:47:31,557.557 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 608 16 5074 5058 33.2714 48 3.46046 1.91952 2014-04-01 14:47:32,558.558 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 609 16 5080 5064 33.2561 24 2.89616 1.92009 2014-04-01 14:47:33,327.327 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:47:33,557.557 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 610 16 5086 5070 33.2409 24 2.28074 1.91971 2014-04-01 14:47:34,124.124 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:47:34,561.561 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 611 16 5087 5071 33.1931 4 0.109495 1.91936 2014-04-01 14:47:35,085.085 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:47:35,557.557 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 612 16 5087 5071 33.1389 0 - 1.91936 2014-04-01 14:47:35,944.944 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:46:42.666637', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438703548, u'health': u'HEALTH_OK', u'kb_used': 9625436, u'store_stats': {u'bytes_total': 16779162, u'bytes_log': 16711680, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:46:42.977138', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438577788, u'health': u'HEALTH_OK', u'kb_used': 9751196, u'store_stats': {u'bytes_total': 25167770, u'bytes_log': 25100288, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}, {u'last_updated': u'2014-04-01 05:46:43.037240', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438703352, u'health': u'HEALTH_OK', u'kb_used': 9625632, u'store_stats': {u'bytes_total': 25167770, u'bytes_log': 25100288, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1930}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2836/21468 objects degraded (13.210%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 18848771, u'degraded_objects': 2836, u'num_pgs': 158, u'recovering_keys_per_sec': 0, u'data_bytes': 29938944356, u'degraded_total': 21468, u'bytes_used': 30535200768, u'recovering_objects_per_sec': 4, u'write_bytes_sec': 28901449, u'op_per_sec': 6, u'version': 372, u'pgs_by_state': [{u'count': 2, u'state_name': u'active+recovery_wait'}, {u'count': 147, u'state_name': u'active+clean'}, {u'count': 9, u'state_name': u'active+recovering'}], u'degraded_ratio': u'13.210', u'bytes_avail': 2304977661952}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:47:35,944.944 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:47:36,558.558 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 613 16 5087 5071 33.0848 0 - 1.91936 2014-04-01 14:47:36,821.821 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:47:37,740.740 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 614 16 5094 5078 33.0765 9.33333 0.249914 1.92373 2014-04-01 14:47:38,564.564 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 615 16 5113 5097 33.1463 76 0.35018 1.92915 2014-04-01 14:47:39,579.579 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 616 16 5117 5101 33.1184 16 0.110892 1.92816 2014-04-01 14:47:40,560.560 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 617 16 5121 5105 33.0907 16 0.463663 1.9278 2014-04-01 14:47:41,144.144 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:47:41,559.559 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 618 16 5121 5105 33.0371 0 - 1.9278 2014-04-01 14:47:41,853.853 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:47:42,560.560 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 619 16 5128 5112 33.029 14 0.230733 1.92934 2014-04-01 14:47:42,806.806 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:47:43,523.523 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:47:42.666945', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438682916, u'health': u'HEALTH_OK', u'kb_used': 9646068, u'store_stats': {u'bytes_total': 4090746, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993578}}, {u'last_updated': u'2014-04-01 05:47:42.977394', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438514140, u'health': u'HEALTH_OK', u'kb_used': 9814844, u'store_stats': {u'bytes_total': 5139341, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993597}}, {u'last_updated': u'2014-04-01 05:47:43.037521', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438682768, u'health': u'HEALTH_OK', u'kb_used': 9646216, u'store_stats': {u'bytes_total': 5139357, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993613}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2780/21606 objects degraded (12.867%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 2780, u'num_pgs': 158, u'data_bytes': 30131882340, u'degraded_total': 21606, u'bytes_used': 30745882624, u'op_per_sec': 9, u'write_bytes_sec': 38264414, u'version': 377, u'pgs_by_state': [{u'count': 148, u'state_name': u'active+clean'}, {u'count': 10, u'state_name': u'active+recovering'}], u'degraded_ratio': u'12.867', u'bytes_avail': 2304766980096}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:47:43,524.524 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:47:43,559.559 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:47:43.474136min lat: 0.064365 max lat: 110.858 avg lat: 1.92934 2014-04-01 14:47:43,559.559 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:47:43,559.559 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 620 16 5128 5112 32.9757 0 - 1.92934 2014-04-01 14:47:44,384.384 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:47:44,690.690 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 621 16 5141 5125 33.0063 26 0.261159 1.9356 2014-04-01 14:47:45,560.560 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 622 16 5147 5131 32.9919 24 3.23631 1.93611 2014-04-01 14:47:46,560.560 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 623 16 5153 5137 32.9774 24 0.120768 1.9356 2014-04-01 14:47:47,560.560 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 624 16 5161 5145 32.9758 32 0.118914 1.93489 2014-04-01 14:47:48,297.297 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:47:48,561.561 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 625 16 5163 5147 32.9359 8 3.36745 1.93565 2014-04-01 14:47:49,172.172 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:47:49,560.560 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 626 16 5171 5155 32.9344 32 4.60147 1.93663 2014-04-01 14:47:50,100.100 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:47:50,561.561 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 627 16 5175 5159 32.9074 16 4.29178 1.93785 2014-04-01 14:47:50,823.823 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:47:42.666945', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438682916, u'health': u'HEALTH_OK', u'kb_used': 9646068, u'store_stats': {u'bytes_total': 4090746, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993578}}, {u'last_updated': u'2014-04-01 05:47:42.977394', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438514140, u'health': u'HEALTH_OK', u'kb_used': 9814844, u'store_stats': {u'bytes_total': 5139341, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993597}}, {u'last_updated': u'2014-04-01 05:47:43.037521', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438682768, u'health': u'HEALTH_OK', u'kb_used': 9646216, u'store_stats': {u'bytes_total': 5139357, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993613}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2684/21711 objects degraded (12.362%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 19288655, u'degraded_objects': 2684, u'num_pgs': 158, u'recovering_keys_per_sec': 0, u'data_bytes': 30278682980, u'degraded_total': 21711, u'bytes_used': 31017840640, u'recovering_objects_per_sec': 4, u'write_bytes_sec': 19288655, u'op_per_sec': 4, u'version': 382, u'pgs_by_state': [{u'count': 149, u'state_name': u'active+clean'}, {u'count': 9, u'state_name': u'active+recovering'}], u'degraded_ratio': u'12.362', u'bytes_avail': 2304495022080}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:47:50,823.823 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:47:51,561.561 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 628 16 5183 5167 32.9059 32 0.064627 1.93888 2014-04-01 14:47:51,583.583 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:47:52,561.561 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 629 16 5185 5169 32.8663 8 0.149217 1.93827 2014-04-01 14:47:53,562.562 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 630 16 5186 5170 32.8205 4 7.54517 1.93936 2014-04-01 14:47:54,562.562 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 631 16 5190 5174 32.7938 16 3.43807 1.94126 2014-04-01 14:47:55,341.341 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:47:55,563.563 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 632 16 5192 5176 32.7546 8 6.29995 1.9431 2014-04-01 14:47:56,051.051 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:47:56,562.562 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 633 16 5196 5180 32.7281 16 5.90238 1.94608 2014-04-01 14:47:56,843.843 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:47:57,563.563 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 634 16 5202 5186 32.7144 24 6.85785 1.94822 2014-04-01 14:47:57,586.586 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:47:42.666945', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438682916, u'health': u'HEALTH_OK', u'kb_used': 9646068, u'store_stats': {u'bytes_total': 4090746, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993578}}, {u'last_updated': u'2014-04-01 05:47:42.977394', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438514140, u'health': u'HEALTH_OK', u'kb_used': 9814844, u'store_stats': {u'bytes_total': 5139341, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993597}}, {u'last_updated': u'2014-04-01 05:47:43.037521', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438682768, u'health': u'HEALTH_OK', u'kb_used': 9646216, u'store_stats': {u'bytes_total': 5139357, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993613}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'9 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2690/21807 objects degraded (12.335%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 2690, u'num_pgs': 158, u'data_bytes': 30412900708, u'degraded_total': 21807, u'bytes_used': 31217086464, u'op_per_sec': 4, u'write_bytes_sec': 19182953, u'version': 387, u'pgs_by_state': [{u'count': 149, u'state_name': u'active+clean'}, {u'count': 9, u'state_name': u'active+recovering'}], u'degraded_ratio': u'12.335', u'bytes_avail': 2304295776256}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:47:57,586.586 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:47:58,449.449 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:47:58,611.611 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 635 16 5205 5189 32.6817 12 0.556547 1.94821 2014-04-01 14:47:59,688.688 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 636 16 5208 5192 32.6492 12 4.93618 1.94931 2014-04-01 14:48:00,562.562 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 637 16 5216 5200 32.6482 32 0.112959 1.95293 2014-04-01 14:48:01,563.563 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 638 16 5219 5203 32.6158 12 6.5975 1.95386 2014-04-01 14:48:02,563.563 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 639 16 5222 5206 32.5836 12 5.51248 1.95547 2014-04-01 14:48:02,685.685 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:48:03,354.354 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:48:03,669.669 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:48:03.477274min lat: 0.064365 max lat: 110.858 avg lat: 1.95814 2014-04-01 14:48:03,669.669 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:48:03,670.670 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 640 16 5226 5210 32.5576 16 5.71438 1.95814 2014-04-01 14:48:04,296.296 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:48:04,563.563 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 641 16 5227 5211 32.5131 4 7.98842 1.95929 2014-04-01 14:48:05,054.054 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:47:42.666945', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438682916, u'health': u'HEALTH_OK', u'kb_used': 9646068, u'store_stats': {u'bytes_total': 4090746, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993578}}, {u'last_updated': u'2014-04-01 05:47:42.977394', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438514140, u'health': u'HEALTH_OK', u'kb_used': 9814844, u'store_stats': {u'bytes_total': 5139341, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993597}}, {u'last_updated': u'2014-04-01 05:47:43.037521', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438682768, u'health': u'HEALTH_OK', u'kb_used': 9646216, u'store_stats': {u'bytes_total': 5139357, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993613}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2544/21903 objects degraded (11.615%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 6109496, u'degraded_objects': 2544, u'num_pgs': 158, u'recovering_keys_per_sec': 0, u'data_bytes': 30547118436, u'degraded_total': 21903, u'bytes_used': 31410876416, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 25659886, u'op_per_sec': 6, u'version': 391, u'pgs_by_state': [{u'count': 150, u'state_name': u'active+clean'}, {u'count': 8, u'state_name': u'active+recovering'}], u'degraded_ratio': u'11.615', u'bytes_avail': 2304101986304}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:48:05,054.054 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:48:05,563.563 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 642 16 5228 5212 32.4687 4 6.74937 1.96021 2014-04-01 14:48:05,811.811 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:48:06,573.573 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 643 16 5233 5217 32.4493 20 0.106484 1.96154 2014-04-01 14:48:07,574.574 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 644 16 5237 5221 32.4237 16 0.979683 1.96204 2014-04-01 14:48:08,563.563 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 645 16 5245 5229 32.4231 32 7.15538 1.96638 2014-04-01 14:48:09,571.571 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 646 16 5254 5238 32.4286 36 0.288107 1.96682 2014-04-01 14:48:09,727.727 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:48:10,393.393 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:48:10,739.739 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 647 16 5268 5252 32.465 56 1.19801 1.96858 2014-04-01 14:48:11,397.397 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:48:11,571.571 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 648 16 5273 5257 32.4458 20 1.5976 1.96808 2014-04-01 14:48:12,163.163 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:47:42.666945', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438682916, u'health': u'HEALTH_OK', u'kb_used': 9646068, u'store_stats': {u'bytes_total': 4090746, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993578}}, {u'last_updated': u'2014-04-01 05:47:42.977394', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438514140, u'health': u'HEALTH_OK', u'kb_used': 9814844, u'store_stats': {u'bytes_total': 5139341, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993597}}, {u'last_updated': u'2014-04-01 05:47:43.037521', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438682768, u'health': u'HEALTH_OK', u'kb_used': 9646216, u'store_stats': {u'bytes_total': 5139357, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993613}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'8 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2546/21957 objects degraded (11.595%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 2546, u'num_pgs': 158, u'data_bytes': 30622615908, u'degraded_total': 21957, u'bytes_used': 31537197056, u'op_per_sec': 3, u'write_bytes_sec': 12893745, u'version': 396, u'pgs_by_state': [{u'count': 150, u'state_name': u'active+clean'}, {u'count': 8, u'state_name': u'active+recovering'}], u'degraded_ratio': u'11.595', u'bytes_avail': 2303975665664}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:48:12,164.164 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:48:12,566.566 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 649 16 5279 5263 32.4327 24 2.93072 1.96904 2014-04-01 14:48:12,882.882 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:48:13,568.568 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 650 16 5280 5264 32.389 4 0.220231 1.96871 2014-04-01 14:48:14,572.572 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 651 16 5283 5267 32.3577 12 0.234684 1.9691 2014-04-01 14:48:15,572.572 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 652 16 5285 5269 32.3203 8 0.167761 1.96934 2014-04-01 14:48:16,566.566 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 653 16 5290 5274 32.3014 20 0.115822 1.96987 2014-04-01 14:48:16,725.725 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:48:17,560.560 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:48:17,721.721 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 654 16 5294 5278 32.2765 16 7.44423 1.97216 2014-04-01 14:48:18,506.506 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:48:18,565.565 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 655 16 5297 5281 32.2456 12 0.857315 1.97259 2014-04-01 14:48:19,279.279 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:47:42.666945', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438682916, u'health': u'HEALTH_OK', u'kb_used': 9646068, u'store_stats': {u'bytes_total': 4090746, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993578}}, {u'last_updated': u'2014-04-01 05:47:42.977394', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438514140, u'health': u'HEALTH_OK', u'kb_used': 9814844, u'store_stats': {u'bytes_total': 5139341, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993597}}, {u'last_updated': u'2014-04-01 05:47:43.037521', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438682768, u'health': u'HEALTH_OK', u'kb_used': 9646216, u'store_stats': {u'bytes_total': 5139357, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993613}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'7 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 2427/22110 objects degraded (10.977%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 2427, u'num_pgs': 158, u'data_bytes': 30836525412, u'degraded_total': 22110, u'bytes_used': 31848304640, u'op_per_sec': 4, u'write_bytes_sec': 19940301, u'version': 401, u'pgs_by_state': [{u'count': 151, u'state_name': u'active+clean'}, {u'count': 7, u'state_name': u'active+recovering'}], u'degraded_ratio': u'10.977', u'bytes_avail': 2303664558080}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:48:19,280.280 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:48:19,566.566 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 656 16 5314 5298 32.3 68 0.436751 1.9791 2014-04-01 14:48:20,320.320 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:48:20,631.631 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 657 16 5328 5312 32.3361 56 1.33676 1.97692 2014-04-01 14:48:21,567.567 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 658 16 5343 5327 32.3781 60 1.7431 1.97413 2014-04-01 14:48:22,566.566 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 659 16 5354 5338 32.3958 44 0.18893 1.97231 2014-04-01 14:48:23,567.567 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:48:23.480350min lat: 0.064365 max lat: 110.858 avg lat: 1.97202 2014-04-01 14:48:23,567.567 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:48:23,567.567 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 660 16 5356 5340 32.3588 8 2.0495 1.97202 2014-04-01 14:48:24,247.247 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:48:24,569.569 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 661 16 5367 5351 32.3764 44 2.45584 1.97274 2014-04-01 14:48:25,194.194 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:48:25,566.566 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 662 16 5379 5363 32.4 48 0.105644 1.97161 2014-04-01 14:48:25,970.970 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:48:26,567.567 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 663 16 5382 5366 32.3692 12 1.35299 1.97199 2014-04-01 14:48:26,676.676 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:47:42.666945', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438682916, u'health': u'HEALTH_OK', u'kb_used': 9646068, u'store_stats': {u'bytes_total': 4090746, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993578}}, {u'last_updated': u'2014-04-01 05:47:42.977394', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438514140, u'health': u'HEALTH_OK', u'kb_used': 9814844, u'store_stats': {u'bytes_total': 5139341, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993597}}, {u'last_updated': u'2014-04-01 05:47:43.037521', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438682768, u'health': u'HEALTH_OK', u'kb_used': 9646216, u'store_stats': {u'bytes_total': 5139357, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993613}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'4 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1888/22353 objects degraded (8.446%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1888, u'num_pgs': 158, u'data_bytes': 31176264036, u'degraded_total': 22353, u'bytes_used': 32183033856, u'op_per_sec': 17, u'write_bytes_sec': 73051849, u'version': 406, u'pgs_by_state': [{u'count': 154, u'state_name': u'active+clean'}, {u'count': 4, u'state_name': u'active+recovering'}], u'degraded_ratio': u'8.446', u'bytes_avail': 2303329828864}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:48:26,677.677 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:48:27,395.395 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:48:27,567.567 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 664 16 5382 5366 32.3205 0 - 1.97199 2014-04-01 14:48:28,567.567 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 665 16 5385 5369 32.2899 6 3.5933 1.9731 2014-04-01 14:48:29,567.567 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 666 16 5389 5373 32.2654 16 0.115424 1.9727 2014-04-01 14:48:30,568.568 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 667 16 5391 5375 32.2291 8 0.139852 1.97202 2014-04-01 14:48:31,156.156 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:48:31,571.571 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 668 16 5408 5392 32.2826 68 1.97764 1.98135 2014-04-01 14:48:31,850.850 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:48:32,568.568 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 669 16 5420 5404 32.3061 48 0.689277 1.97857 2014-04-01 14:48:32,774.774 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:48:33,479.479 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:47:42.666945', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438682916, u'health': u'HEALTH_OK', u'kb_used': 9646068, u'store_stats': {u'bytes_total': 4090746, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993578}}, {u'last_updated': u'2014-04-01 05:47:42.977394', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438514140, u'health': u'HEALTH_OK', u'kb_used': 9814844, u'store_stats': {u'bytes_total': 5139341, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993597}}, {u'last_updated': u'2014-04-01 05:47:43.037521', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438682768, u'health': u'HEALTH_OK', u'kb_used': 9646216, u'store_stats': {u'bytes_total': 5139357, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993613}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'4 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1877/22413 objects degraded (8.375%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1877, u'num_pgs': 158, u'data_bytes': 31260150116, u'degraded_total': 22413, u'bytes_used': 32385269760, u'op_per_sec': 2, u'write_bytes_sec': 9641399, u'version': 410, u'pgs_by_state': [{u'count': 154, u'state_name': u'active+clean'}, {u'count': 4, u'state_name': u'active+recovering'}], u'degraded_ratio': u'8.375', u'bytes_avail': 2303127592960}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:48:33,480.480 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:48:33,567.567 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 670 16 5422 5406 32.2698 8 0.113734 1.97793 2014-04-01 14:48:34,223.223 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:48:34,567.567 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 671 16 5422 5406 32.2217 0 - 1.97793 2014-04-01 14:48:35,568.568 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 672 16 5422 5406 32.1738 0 - 1.97793 2014-04-01 14:48:36,568.568 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 673 16 5422 5406 32.1259 0 - 1.97793 2014-04-01 14:48:37,574.574 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 674 16 5437 5421 32.1673 15 0.35923 1.98584 2014-04-01 14:48:37,989.989 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:48:38,572.572 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 675 16 5444 5428 32.1611 28 6.17337 1.98676 2014-04-01 14:48:38,773.773 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:48:39,555.555 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:48:39,569.569 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 676 16 5453 5437 32.1668 36 0.286378 1.9861 2014-04-01 14:48:39,832.832 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 05:48:39.736289 7fe3a7efe700 0 -- 10.214.133.37:0/1005437 >> 10.214.133.38:6800/4373 pipe(0x7fe3ac007a10 sd=13 :46848 s=2 pgs=11 cs=1 l=1 c=0x7fe3ac009070).injecting socket failure 2014-04-01 14:48:40,293.293 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:47:42.666945', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438682916, u'health': u'HEALTH_OK', u'kb_used': 9646068, u'store_stats': {u'bytes_total': 4090746, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993578}}, {u'last_updated': u'2014-04-01 05:47:42.977394', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438514140, u'health': u'HEALTH_OK', u'kb_used': 9814844, u'store_stats': {u'bytes_total': 5139341, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993597}}, {u'last_updated': u'2014-04-01 05:47:43.037521', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438682768, u'health': u'HEALTH_OK', u'kb_used': 9646216, u'store_stats': {u'bytes_total': 5139357, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993613}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'4 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1871/22530 objects degraded (8.304%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 5040160, u'degraded_objects': 1871, u'num_pgs': 158, u'recovering_keys_per_sec': 0, u'data_bytes': 31423727972, u'degraded_total': 22530, u'bytes_used': 32618606592, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 13860440, u'op_per_sec': 3, u'version': 415, u'pgs_by_state': [{u'count': 154, u'state_name': u'active+clean'}, {u'count': 4, u'state_name': u'active+recovering'}], u'degraded_ratio': u'8.304', u'bytes_avail': 2302894256128}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:48:40,294.294 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:48:40,570.570 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 677 16 5458 5442 32.1488 20 0.199454 1.9854 2014-04-01 14:48:41,110.110 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:48:41,570.570 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 678 16 5472 5456 32.184 56 0.461801 1.98556 2014-04-01 14:48:42,570.570 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 679 16 5483 5467 32.2014 44 0.147661 1.98355 2014-04-01 14:48:43,570.570 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:48:43.483365min lat: 0.064365 max lat: 110.858 avg lat: 1.98273 2014-04-01 14:48:43,570.570 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:48:43,571.571 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 680 16 5486 5470 32.1717 12 0.200364 1.98273 2014-04-01 14:48:44,570.570 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 681 16 5492 5476 32.1597 24 3.70681 1.98457 2014-04-01 14:48:44,909.909 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:48:45,701.701 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:48:45,872.872 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 682 16 5495 5479 32.1301 12 3.62328 1.98436 2014-04-01 14:48:46,560.560 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:48:46,569.569 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 683 16 5495 5479 32.083 0 - 1.98436 2014-04-01 14:48:47,445.445 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:48:42.667217', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438649828, u'health': u'HEALTH_OK', u'kb_used': 9679156, u'store_stats': {u'bytes_total': 6187898, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993578}}, {u'last_updated': u'2014-04-01 05:48:42.977648', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438449528, u'health': u'HEALTH_OK', u'kb_used': 9879456, u'store_stats': {u'bytes_total': 7236493, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993597}}, {u'last_updated': u'2014-04-01 05:48:43.037762', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438649640, u'health': u'HEALTH_OK', u'kb_used': 9679344, u'store_stats': {u'bytes_total': 7236509, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993613}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'4 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1871/22713 objects degraded (8.238%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1871, u'num_pgs': 158, u'data_bytes': 31679580516, u'degraded_total': 22713, u'bytes_used': 32824455168, u'op_per_sec': 9, u'write_bytes_sec': 41315599, u'version': 420, u'pgs_by_state': [{u'count': 154, u'state_name': u'active+clean'}, {u'count': 4, u'state_name': u'active+recovering'}], u'degraded_ratio': u'8.238', u'bytes_avail': 2302688407552}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:48:47,446.446 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:48:47,570.570 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 684 16 5495 5479 32.0361 0 - 1.98436 2014-04-01 14:48:48,219.219 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:48:48,571.571 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 685 16 5505 5489 32.0478 13.3333 0.510357 1.98818 2014-04-01 14:48:49,574.574 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 686 16 5529 5513 32.141 96 0.649526 1.99032 2014-04-01 14:48:50,571.571 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 687 16 5537 5521 32.1407 32 0.746778 1.98876 2014-04-01 14:48:51,571.571 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 688 16 5557 5541 32.2103 80 0.634493 1.98511 2014-04-01 14:48:52,176.176 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:48:52,571.571 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 689 16 5570 5554 32.239 52 0.161928 1.98252 2014-04-01 14:48:52,952.952 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:48:53,572.572 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 690 16 5570 5554 32.1923 0 - 1.98252 2014-04-01 14:48:53,720.720 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:48:54,502.502 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:48:42.667217', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438649828, u'health': u'HEALTH_OK', u'kb_used': 9679156, u'store_stats': {u'bytes_total': 6187898, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993578}}, {u'last_updated': u'2014-04-01 05:48:42.977648', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438449528, u'health': u'HEALTH_OK', u'kb_used': 9879456, u'store_stats': {u'bytes_total': 7236493, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993597}}, {u'last_updated': u'2014-04-01 05:48:43.037762', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438649640, u'health': u'HEALTH_OK', u'kb_used': 9679344, u'store_stats': {u'bytes_total': 7236509, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993613}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1781/22890 objects degraded (7.781%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 10994762, u'degraded_objects': 1781, u'num_pgs': 158, u'recovering_keys_per_sec': 0, u'data_bytes': 31927044452, u'degraded_total': 22890, u'bytes_used': 33134874624, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 40680621, u'op_per_sec': 9, u'version': 424, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.781', u'bytes_avail': 2302377988096}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:48:54,503.503 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:48:54,570.570 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 691 16 5570 5554 32.1457 0 - 1.98252 2014-04-01 14:48:55,266.266 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:48:55,579.579 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 692 16 5570 5554 32.0992 0 - 1.98252 2014-04-01 14:48:56,572.572 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 693 16 5570 5554 32.0529 0 - 1.98252 2014-04-01 14:48:57,577.577 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 694 16 5598 5582 32.1681 22.4 0.453245 1.98857 2014-04-01 14:48:58,575.575 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 695 16 5618 5602 32.2369 80 0.730202 1.98412 2014-04-01 14:48:59,173.173 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:48:59,572.572 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 696 16 5631 5615 32.2653 52 0.831043 1.98109 2014-04-01 14:48:59,978.978 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:49:00,572.572 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 697 16 5642 5626 32.2821 44 0.360229 1.97885 2014-04-01 14:49:00,904.904 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:49:01,573.573 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 698 16 5643 5627 32.2416 4 2.13224 1.97888 2014-04-01 14:49:01,697.697 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:48:42.667217', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438649828, u'health': u'HEALTH_OK', u'kb_used': 9679156, u'store_stats': {u'bytes_total': 6187898, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993578}}, {u'last_updated': u'2014-04-01 05:48:42.977648', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438449528, u'health': u'HEALTH_OK', u'kb_used': 9879456, u'store_stats': {u'bytes_total': 7236493, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993597}}, {u'last_updated': u'2014-04-01 05:48:43.037762', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438649640, u'health': u'HEALTH_OK', u'kb_used': 9679344, u'store_stats': {u'bytes_total': 7236509, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993613}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1795/23046 objects degraded (7.789%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1795, u'num_pgs': 158, u'data_bytes': 32145148260, u'degraded_total': 23046, u'bytes_used': 34953306112, u'op_per_sec': 7, u'write_bytes_sec': 32501193, u'version': 429, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.789', u'bytes_avail': 2300559556608}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:49:01,697.697 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:49:02,450.450 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:49:02,606.606 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 699 16 5643 5627 32.1955 0 - 1.97888 2014-04-01 14:49:03,573.573 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:49:03.486503min lat: 0.064365 max lat: 110.858 avg lat: 1.97888 2014-04-01 14:49:03,573.573 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:49:03,573.573 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 700 16 5643 5627 32.1495 0 - 1.97888 2014-04-01 14:49:04,582.582 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 701 16 5659 5643 32.1949 21.3333 5.43944 1.98658 2014-04-01 14:49:05,573.573 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 702 16 5671 5655 32.2174 48 0.82128 1.98427 2014-04-01 14:49:06,217.217 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:49:06,573.573 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 703 16 5690 5674 32.2797 76 0.316716 1.98114 2014-04-01 14:49:06,920.920 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:49:07,573.573 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 704 16 5699 5683 32.2849 36 0.19089 1.97902 2014-04-01 14:49:07,692.692 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:49:08,404.404 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:48:42.667217', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438649828, u'health': u'HEALTH_OK', u'kb_used': 9679156, u'store_stats': {u'bytes_total': 6187898, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993578}}, {u'last_updated': u'2014-04-01 05:48:42.977648', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438449528, u'health': u'HEALTH_OK', u'kb_used': 9879456, u'store_stats': {u'bytes_total': 7236493, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993597}}, {u'last_updated': u'2014-04-01 05:48:43.037762', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438649640, u'health': u'HEALTH_OK', u'kb_used': 9679344, u'store_stats': {u'bytes_total': 7236509, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993613}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1787/23220 objects degraded (7.696%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1787, u'num_pgs': 158, u'data_bytes': 32388417892, u'degraded_total': 23220, u'bytes_used': 35175534592, u'op_per_sec': 6, u'write_bytes_sec': 27300682, u'version': 433, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.696', u'bytes_avail': 2300337328128}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:49:08,405.405 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:49:08,573.573 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 705 16 5700 5684 32.2448 4 0.461948 1.97875 2014-04-01 14:49:09,155.155 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:49:09,575.575 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 706 16 5700 5684 32.1991 0 - 1.97875 2014-04-01 14:49:10,576.576 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 707 16 5719 5703 32.2611 38 2.80343 1.98293 2014-04-01 14:49:11,580.580 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 708 16 5741 5725 32.3398 88 0.626178 1.97818 2014-04-01 14:49:12,574.574 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 709 16 5753 5737 32.3619 48 1.4843 1.97564 2014-04-01 14:49:12,915.915 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:49:14,355.355 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:49:14,667.667 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 710 16 5760 5744 32.3557 28 0.385051 1.97453 2014-04-01 14:49:14,667.667 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 711 16 5760 5744 32.3102 0 - 1.97453 2014-04-01 14:49:15,288.288 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:49:15,575.575 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 712 16 5760 5744 32.2648 0 - 1.97453 2014-04-01 14:49:16,004.004 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:48:42.667217', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438649828, u'health': u'HEALTH_OK', u'kb_used': 9679156, u'store_stats': {u'bytes_total': 6187898, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993578}}, {u'last_updated': u'2014-04-01 05:48:42.977648', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438449528, u'health': u'HEALTH_OK', u'kb_used': 9879456, u'store_stats': {u'bytes_total': 7236493, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993597}}, {u'last_updated': u'2014-04-01 05:48:43.037762', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438649640, u'health': u'HEALTH_OK', u'kb_used': 9679344, u'store_stats': {u'bytes_total': 7236509, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993613}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1795/23454 objects degraded (7.653%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1795, u'num_pgs': 158, u'data_bytes': 32715573604, u'degraded_total': 23454, u'bytes_used': 35607363584, u'op_per_sec': 8, u'write_bytes_sec': 37515420, u'version': 439, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.653', u'bytes_avail': 2299905499136}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:49:16,005.005 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:49:16,575.575 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 713 16 5760 5744 32.2196 0 - 1.97453 2014-04-01 14:49:17,583.583 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 714 16 5760 5744 32.1745 0 - 1.97453 2014-04-01 14:49:18,588.588 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 715 16 5785 5769 32.2693 20 0.209944 1.98257 2014-04-01 14:49:19,574.574 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 716 16 5802 5786 32.3192 68 1.11036 1.9787 2014-04-01 14:49:19,876.876 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:49:20,721.721 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 717 16 5807 5791 32.302 20 1.68566 1.9779 2014-04-01 14:49:21,584.584 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 718 16 5809 5793 32.2682 8 0.837054 1.97756 2014-04-01 14:49:22,583.583 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 719 16 5809 5793 32.2233 0 - 1.97756 2014-04-01 14:49:23,583.583 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:49:23.489185min lat: 0.064365 max lat: 110.858 avg lat: 1.98101 2014-04-01 14:49:23,745.745 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:49:23,745.745 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 720 16 5817 5801 32.223 16 4.14426 1.98101 2014-04-01 14:49:23,874.874 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:49:24,512.512 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:49:24,839.839 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 721 16 5839 5823 32.3003 88 0.666227 1.98031 2014-04-01 14:49:25,576.576 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 722 16 5846 5830 32.2943 28 0.883992 1.97885 2014-04-01 14:49:25,606.606 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:49:26,509.509 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:48:42.667217', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438649828, u'health': u'HEALTH_OK', u'kb_used': 9679156, u'store_stats': {u'bytes_total': 6187898, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993578}}, {u'last_updated': u'2014-04-01 05:48:42.977648', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438449528, u'health': u'HEALTH_OK', u'kb_used': 9879456, u'store_stats': {u'bytes_total': 7236493, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993597}}, {u'last_updated': u'2014-04-01 05:48:43.037762', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438649640, u'health': u'HEALTH_OK', u'kb_used': 9679344, u'store_stats': {u'bytes_total': 7236509, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993613}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1797/23721 objects degraded (7.576%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1797, u'num_pgs': 158, u'data_bytes': 33088866660, u'degraded_total': 23721, u'bytes_used': 36108460032, u'op_per_sec': 7, u'write_bytes_sec': 29867507, u'version': 446, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.576', u'bytes_avail': 2299404402688}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:49:26,510.510 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:49:26,576.576 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 723 16 5846 5830 32.2497 0 - 1.97885 2014-04-01 14:49:27,265.265 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:49:27,581.581 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 724 16 5846 5830 32.2051 0 - 1.97885 2014-04-01 14:49:28,576.576 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 725 16 5846 5830 32.1607 0 - 1.97885 2014-04-01 14:49:29,580.580 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 726 16 5850 5834 32.1384 4 5.23461 1.98118 2014-04-01 14:49:30,577.577 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 727 16 5866 5850 32.1823 64 0.314955 1.98653 2014-04-01 14:49:31,188.188 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:49:31,581.581 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 728 16 5889 5873 32.2644 92 0.175975 1.98203 2014-04-01 14:49:31,789.789 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:49:32,719.719 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 729 16 5905 5889 32.3079 64 1.073 1.97909 2014-04-01 14:49:32,882.882 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:49:33,577.577 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 730 16 5906 5890 32.2691 4 0.877998 1.9789 2014-04-01 14:49:33,662.662 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:48:42.667217', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438649828, u'health': u'HEALTH_OK', u'kb_used': 9679156, u'store_stats': {u'bytes_total': 6187898, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993578}}, {u'last_updated': u'2014-04-01 05:48:42.977648', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438449528, u'health': u'HEALTH_OK', u'kb_used': 9879456, u'store_stats': {u'bytes_total': 7236493, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993597}}, {u'last_updated': u'2014-04-01 05:48:43.037762', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438649640, u'health': u'HEALTH_OK', u'kb_used': 9679344, u'store_stats': {u'bytes_total': 7236509, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993613}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1795/23823 objects degraded (7.535%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1795, u'num_pgs': 158, u'data_bytes': 33231472996, u'degraded_total': 23823, u'bytes_used': 36224679936, u'op_per_sec': 7, u'write_bytes_sec': 31947230, u'version': 450, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.535', u'bytes_avail': 2299288182784}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:49:33,663.663 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:49:34,423.423 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:49:34,893.893 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 731 16 5906 5890 32.225 0 - 1.9789 2014-04-01 14:49:35,580.580 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 732 16 5906 5890 32.181 0 - 1.9789 2014-04-01 14:49:36,581.581 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 733 16 5921 5905 32.2189 20 3.54236 1.9844 2014-04-01 14:49:37,581.581 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 734 16 5938 5922 32.2677 68 0.752248 1.98164 2014-04-01 14:49:38,578.578 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:49:38,581.581 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 735 16 5959 5943 32.338 84 0.805765 1.97794 2014-04-01 14:49:39,389.389 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:49:39,709.709 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 736 16 5973 5957 32.3702 56 0.712496 1.97542 2014-04-01 14:49:40,490.490 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:49:40,578.578 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 737 16 5975 5959 32.3371 8 0.31035 1.97492 2014-04-01 14:49:41,195.195 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:48:42.667217', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438649828, u'health': u'HEALTH_OK', u'kb_used': 9679156, u'store_stats': {u'bytes_total': 6187898, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993578}}, {u'last_updated': u'2014-04-01 05:48:42.977648', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438449528, u'health': u'HEALTH_OK', u'kb_used': 9879456, u'store_stats': {u'bytes_total': 7236493, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993597}}, {u'last_updated': u'2014-04-01 05:48:43.037762', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438649640, u'health': u'HEALTH_OK', u'kb_used': 9679344, u'store_stats': {u'bytes_total': 7236509, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993613}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1798/24069 objects degraded (7.470%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1798, u'num_pgs': 158, u'data_bytes': 33575405924, u'degraded_total': 24069, u'bytes_used': 36525449216, u'op_per_sec': 9, u'write_bytes_sec': 39556422, u'version': 455, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.470', u'bytes_avail': 2298987413504}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:49:41,196.196 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:49:41,579.579 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 738 16 5976 5960 32.2987 4 0.134405 1.97461 2014-04-01 14:49:41,891.891 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:49:42,579.579 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 739 16 5976 5960 32.255 0 - 1.97461 2014-04-01 14:49:43,579.579 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:49:43.492292min lat: 0.064365 max lat: 110.858 avg lat: 1.97461 2014-04-01 14:49:43,579.579 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:49:43,580.580 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 740 16 5976 5960 32.2114 0 - 1.97461 2014-04-01 14:49:44,590.590 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 741 16 5989 5973 32.2381 17.3333 5.23116 1.98168 2014-04-01 14:49:45,579.579 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 742 16 6008 5992 32.2971 76 0.759589 1.9798 2014-04-01 14:49:45,657.657 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:49:46,430.430 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:49:46,742.742 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 743 16 6018 6002 32.3074 40 0.151464 1.97768 2014-04-01 14:49:47,376.376 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:49:47,580.580 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 744 16 6031 6015 32.3339 52 0.324206 1.97637 2014-04-01 14:49:48,141.141 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:49:42.667501', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438611604, u'health': u'HEALTH_OK', u'kb_used': 9717380, u'store_stats': {u'bytes_total': 7236474, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993578}}, {u'last_updated': u'2014-04-01 05:49:42.977895', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438369892, u'health': u'HEALTH_OK', u'kb_used': 9959092, u'store_stats': {u'bytes_total': 9333645, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993597}}, {u'last_updated': u'2014-04-01 05:49:43.037992', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438611604, u'health': u'HEALTH_OK', u'kb_used': 9717380, u'store_stats': {u'bytes_total': 9333661, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993613}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1803/24228 objects degraded (7.442%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1803, u'num_pgs': 158, u'data_bytes': 33797704036, u'degraded_total': 24228, u'bytes_used': 36905644032, u'op_per_sec': 9, u'write_bytes_sec': 40854844, u'version': 459, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.442', u'bytes_avail': 2298607218688}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:49:48,142.142 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:49:48,580.580 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 745 16 6042 6026 32.3495 44 0.146972 1.97538 2014-04-01 14:49:48,952.952 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:49:49,728.728 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 746 16 6044 6028 32.3169 8 0.51075 1.97484 2014-04-01 14:49:50,580.580 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 747 16 6045 6029 32.279 4 0.517982 1.97459 2014-04-01 14:49:51,588.588 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 748 16 6051 6035 32.2679 24 0.14175 1.97653 2014-04-01 14:49:52,581.581 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 749 16 6075 6059 32.353 96 0.528462 1.97687 2014-04-01 14:49:52,871.871 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:49:53,581.581 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 750 16 6094 6078 32.4112 76 0.77616 1.97313 2014-04-01 14:49:53,641.641 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:49:54,585.585 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 751 16 6108 6092 32.4426 56 1.02296 1.97084 2014-04-01 14:49:54,602.602 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:49:55,365.365 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:49:42.667501', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438611604, u'health': u'HEALTH_OK', u'kb_used': 9717380, u'store_stats': {u'bytes_total': 7236474, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993578}}, {u'last_updated': u'2014-04-01 05:49:42.977895', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438369892, u'health': u'HEALTH_OK', u'kb_used': 9959092, u'store_stats': {u'bytes_total': 9333645, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993597}}, {u'last_updated': u'2014-04-01 05:49:43.037992', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438611604, u'health': u'HEALTH_OK', u'kb_used': 9717380, u'store_stats': {u'bytes_total': 9333661, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993613}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1798/24474 objects degraded (7.347%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1798, u'num_pgs': 158, u'data_bytes': 34141636964, u'degraded_total': 24474, u'bytes_used': 37203947520, u'op_per_sec': 13, u'write_bytes_sec': 58102667, u'version': 464, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.347', u'bytes_avail': 2298308915200}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:49:55,366.366 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:49:55,581.581 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 752 16 6114 6098 32.4313 24 0.740739 1.96975 2014-04-01 14:49:56,139.139 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:49:56,599.599 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 753 16 6114 6098 32.3883 0 - 1.96975 2014-04-01 14:49:57,581.581 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 754 16 6114 6098 32.3453 0 - 1.96975 2014-04-01 14:49:58,581.581 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 755 16 6114 6098 32.3025 0 - 1.96975 2014-04-01 14:49:59,582.582 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 756 16 6132 6116 32.355 18 0.251881 1.97514 2014-04-01 14:50:00,065.065 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:50:00,581.581 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 757 16 6147 6131 32.3915 60 0.38664 1.97439 2014-04-01 14:50:00,836.836 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:50:01,582.582 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 758 16 6161 6145 32.4226 56 0.141364 1.97158 2014-04-01 14:50:01,609.609 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:50:02,332.332 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:49:42.667501', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438611604, u'health': u'HEALTH_OK', u'kb_used': 9717380, u'store_stats': {u'bytes_total': 7236474, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993578}}, {u'last_updated': u'2014-04-01 05:49:42.977895', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438369892, u'health': u'HEALTH_OK', u'kb_used': 9959092, u'store_stats': {u'bytes_total': 9333645, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993597}}, {u'last_updated': u'2014-04-01 05:49:43.037992', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438611604, u'health': u'HEALTH_OK', u'kb_used': 9717380, u'store_stats': {u'bytes_total': 9333661, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993613}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1831/24654 objects degraded (7.427%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1831, u'num_pgs': 158, u'data_bytes': 34393295204, u'degraded_total': 24654, u'bytes_used': 37606961152, u'op_per_sec': 11, u'write_bytes_sec': 50039394, u'version': 470, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.427', u'bytes_avail': 2297905901568}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:50:02,333.333 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:50:02,581.581 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 759 16 6162 6146 32.3851 4 2.43862 1.97165 2014-04-01 14:50:03,184.184 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:50:03,583.583 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:50:03.495288min lat: 0.064365 max lat: 110.858 avg lat: 1.97188 2014-04-01 14:50:03,584.584 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:50:03,584.584 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 760 16 6165 6149 32.3583 12 1.70734 1.97188 2014-04-01 14:50:04,582.582 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 761 16 6165 6149 32.3158 0 - 1.97188 2014-04-01 14:50:05,583.583 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 762 16 6165 6149 32.2734 0 - 1.97188 2014-04-01 14:50:06,584.584 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 763 16 6165 6149 32.2311 0 - 1.97188 2014-04-01 14:50:06,952.952 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:50:07,583.583 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 764 16 6192 6176 32.3302 27 0.298339 1.97825 2014-04-01 14:50:07,870.870 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:50:08,583.583 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 765 16 6210 6194 32.3821 72 0.704699 1.97504 2014-04-01 14:50:08,820.820 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:50:09,544.544 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:49:42.667501', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438611604, u'health': u'HEALTH_OK', u'kb_used': 9717380, u'store_stats': {u'bytes_total': 7236474, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993578}}, {u'last_updated': u'2014-04-01 05:49:42.977895', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438369892, u'health': u'HEALTH_OK', u'kb_used': 9959092, u'store_stats': {u'bytes_total': 9333645, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993597}}, {u'last_updated': u'2014-04-01 05:49:43.037992', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438611604, u'health': u'HEALTH_OK', u'kb_used': 9717380, u'store_stats': {u'bytes_total': 9333661, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993613}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1815/24783 objects degraded (7.324%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1815, u'num_pgs': 158, u'data_bytes': 34573650276, u'degraded_total': 24783, u'bytes_used': 37849272320, u'op_per_sec': 6, u'write_bytes_sec': 26413574, u'version': 474, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.324', u'bytes_avail': 2297663590400}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:50:09,545.545 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:50:09,585.585 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 766 16 6229 6213 32.439 76 0.388877 1.97127 2014-04-01 14:50:10,312.312 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:50:10,585.585 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 767 16 6234 6218 32.4228 20 0.421478 1.9705 2014-04-01 14:50:11,583.583 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 768 16 6234 6218 32.3806 0 - 1.9705 2014-04-01 14:50:12,584.584 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 769 16 6234 6218 32.3385 0 - 1.9705 2014-04-01 14:50:13,584.584 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 770 16 6234 6218 32.2965 0 - 1.9705 2014-04-01 14:50:14,227.227 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:50:14,589.589 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 771 16 6250 6234 32.3376 16 5.84263 1.9784 2014-04-01 14:50:15,273.273 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:50:15,583.583 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 772 16 6266 6250 32.3786 64 0.637009 1.97515 2014-04-01 14:50:16,050.050 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:50:16,584.584 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 773 16 6284 6268 32.4298 72 0.756538 1.97181 2014-04-01 14:50:16,756.756 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:49:42.667501', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438611604, u'health': u'HEALTH_OK', u'kb_used': 9717380, u'store_stats': {u'bytes_total': 7236474, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993578}}, {u'last_updated': u'2014-04-01 05:49:42.977895', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438369892, u'health': u'HEALTH_OK', u'kb_used': 9959092, u'store_stats': {u'bytes_total': 9333645, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993597}}, {u'last_updated': u'2014-04-01 05:49:43.037992', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438611604, u'health': u'HEALTH_OK', u'kb_used': 9717380, u'store_stats': {u'bytes_total': 9333661, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993613}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1836/25011 objects degraded (7.341%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1836, u'num_pgs': 158, u'data_bytes': 34892417380, u'degraded_total': 25011, u'bytes_used': 38180012032, u'op_per_sec': 10, u'write_bytes_sec': 43179640, u'version': 479, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.341', u'bytes_avail': 2297332850688}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:50:16,757.757 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:50:17,464.464 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:50:17,616.616 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 774 16 6288 6272 32.4086 16 0.148955 1.97096 2014-04-01 14:50:18,583.583 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 775 16 6288 6272 32.3668 0 - 1.97096 2014-04-01 14:50:19,584.584 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 776 16 6288 6272 32.3251 0 - 1.97096 2014-04-01 14:50:20,584.584 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 777 16 6288 6272 32.2835 0 - 1.97096 2014-04-01 14:50:21,259.259 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:50:21,588.588 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 778 16 6288 6272 32.242 0 - 1.97096 2014-04-01 14:50:22,086.086 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:50:22,589.589 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 779 16 6312 6296 32.3238 19.2 0.516588 1.97902 2014-04-01 14:50:23,173.173 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:50:23,589.589 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:50:23.498004min lat: 0.064365 max lat: 110.858 avg lat: 1.97508 2014-04-01 14:50:23,589.589 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:50:23,589.589 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 780 16 6333 6317 32.39 84 0.353002 1.97508 2014-04-01 14:50:23,999.999 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:49:42.667501', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438611604, u'health': u'HEALTH_OK', u'kb_used': 9717380, u'store_stats': {u'bytes_total': 7236474, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993578}}, {u'last_updated': u'2014-04-01 05:49:42.977895', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438369892, u'health': u'HEALTH_OK', u'kb_used': 9959092, u'store_stats': {u'bytes_total': 9333645, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993597}}, {u'last_updated': u'2014-04-01 05:49:43.037992', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438611604, u'health': u'HEALTH_OK', u'kb_used': 9717380, u'store_stats': {u'bytes_total': 9333661, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993613}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1832/25134 objects degraded (7.289%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1832, u'num_pgs': 158, u'data_bytes': 35064383844, u'degraded_total': 25134, u'bytes_used': 38324162560, u'op_per_sec': 8, u'write_bytes_sec': 34663255, u'version': 483, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.289', u'bytes_avail': 2297188700160}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:50:24,000.000 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:50:24,585.585 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 781 16 6345 6329 32.41 48 1.05154 1.97317 2014-04-01 14:50:24,905.905 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:50:25,585.585 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 782 16 6353 6337 32.4095 32 0.42139 1.97152 2014-04-01 14:50:26,600.600 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 783 16 6355 6339 32.3783 8 0.186733 1.97098 2014-04-01 14:50:27,600.600 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 784 16 6355 6339 32.337 0 - 1.97098 2014-04-01 14:50:28,589.589 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 785 16 6355 6339 32.2958 0 - 1.97098 2014-04-01 14:50:28,889.889 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:50:29,578.578 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:50:29,970.970 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 786 16 6367 6351 32.3158 16 0.136929 1.97558 2014-04-01 14:50:30,586.586 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 787 16 6385 6369 32.3662 72 0.526954 1.97599 2014-04-01 14:50:30,604.604 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:50:31,478.478 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:49:42.667501', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438611604, u'health': u'HEALTH_OK', u'kb_used': 9717380, u'store_stats': {u'bytes_total': 7236474, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993578}}, {u'last_updated': u'2014-04-01 05:49:42.977895', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438369892, u'health': u'HEALTH_OK', u'kb_used': 9959092, u'store_stats': {u'bytes_total': 9333645, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993597}}, {u'last_updated': u'2014-04-01 05:49:43.037992', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438611604, u'health': u'HEALTH_OK', u'kb_used': 9717380, u'store_stats': {u'bytes_total': 9333661, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993613}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1837/25317 objects degraded (7.256%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1837, u'num_pgs': 158, u'data_bytes': 35320236388, u'degraded_total': 25317, u'bytes_used': 38690521088, u'op_per_sec': 4, u'write_bytes_sec': 18287524, u'version': 489, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.256', u'bytes_avail': 2296822341632}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:50:31,479.479 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:50:31,590.590 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 788 16 6394 6378 32.3708 36 1.27189 1.97485 2014-04-01 14:50:32,273.273 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:50:32,592.592 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 789 16 6407 6391 32.3957 52 0.460591 1.97292 2014-04-01 14:50:33,586.586 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 790 16 6407 6391 32.3547 0 - 1.97292 2014-04-01 14:50:34,587.587 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 791 16 6407 6391 32.3138 0 - 1.97292 2014-04-01 14:50:35,587.587 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 792 16 6407 6391 32.273 0 - 1.97292 2014-04-01 14:50:36,244.244 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:50:36,592.592 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 793 16 6422 6406 32.3079 15 0.306448 1.97773 2014-04-01 14:50:36,870.870 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:50:37,588.588 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 794 16 6443 6427 32.373 84 0.845717 1.97557 2014-04-01 14:50:37,824.824 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:50:38,587.587 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 795 16 6452 6436 32.3776 36 0.738583 1.97424 2014-04-01 14:50:38,642.642 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:49:42.667501', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438611604, u'health': u'HEALTH_OK', u'kb_used': 9717380, u'store_stats': {u'bytes_total': 7236474, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993578}}, {u'last_updated': u'2014-04-01 05:49:42.977895', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438369892, u'health': u'HEALTH_OK', u'kb_used': 9959092, u'store_stats': {u'bytes_total': 9333645, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993597}}, {u'last_updated': u'2014-04-01 05:49:43.037992', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438611604, u'health': u'HEALTH_OK', u'kb_used': 9717380, u'store_stats': {u'bytes_total': 9333661, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1993613}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1839/25485 objects degraded (7.216%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1839, u'num_pgs': 158, u'data_bytes': 35555117412, u'degraded_total': 25485, u'bytes_used': 38901456896, u'op_per_sec': 7, u'write_bytes_sec': 31300734, u'version': 493, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.216', u'bytes_avail': 2296611405824}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:50:38,642.642 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:50:39,471.471 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:50:39,625.625 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 796 16 6466 6450 32.4072 56 1.66734 1.97323 2014-04-01 14:50:40,587.587 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 797 16 6469 6453 32.3816 12 0.826317 1.97287 2014-04-01 14:50:41,589.589 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 798 16 6475 6459 32.3711 24 0.27362 1.9715 2014-04-01 14:50:42,588.588 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 799 16 6475 6459 32.3306 0 - 1.9715 2014-04-01 14:50:43,238.238 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:50:43,588.588 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:50:43.501076min lat: 0.064365 max lat: 110.858 avg lat: 1.9715 2014-04-01 14:50:43,589.589 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:50:43,589.589 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 800 16 6475 6459 32.2902 0 - 1.9715 2014-04-01 14:50:43,921.921 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:50:44,589.589 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 801 16 6502 6486 32.3847 36 0.770031 1.97513 2014-04-01 14:50:44,881.881 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:50:45,594.594 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 802 16 6516 6500 32.4141 56 1.76831 1.97297 2014-04-01 14:50:45,609.609 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:50:42.667757', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438583328, u'health': u'HEALTH_OK', u'kb_used': 9745656, u'store_stats': {u'bytes_total': 3422377, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373785}}, {u'last_updated': u'2014-04-01 05:50:42.978092', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438301656, u'health': u'HEALTH_OK', u'kb_used': 10027328, u'store_stats': {u'bytes_total': 4470902, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373734}}, {u'last_updated': u'2014-04-01 05:50:43.038354', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438583320, u'health': u'HEALTH_OK', u'kb_used': 9745664, u'store_stats': {u'bytes_total': 4470939, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373771}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1842/25677 objects degraded (7.174%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1842, u'num_pgs': 158, u'data_bytes': 35823552868, u'degraded_total': 25677, u'bytes_used': 39290118144, u'op_per_sec': 3, u'write_bytes_sec': 16062963, u'version': 499, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.174', u'bytes_avail': 2296222744576}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:50:45,609.609 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:50:46,378.378 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:50:46,681.681 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 803 16 6529 6513 32.4385 52 0.616316 1.9708 2014-04-01 14:50:47,590.590 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 804 16 6531 6515 32.4081 8 1.22867 1.97045 2014-04-01 14:50:48,588.588 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 805 16 6531 6515 32.3678 0 - 1.97045 2014-04-01 14:50:49,589.589 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 806 16 6531 6515 32.3277 0 - 1.97045 2014-04-01 14:50:50,323.323 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:50:50,591.591 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 807 16 6531 6515 32.2876 0 - 1.97045 2014-04-01 14:50:51,061.061 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:50:51,594.594 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 808 16 6545 6529 32.317 14 5.80225 1.97841 2014-04-01 14:50:52,010.010 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:50:52,591.591 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 809 16 6561 6545 32.3561 64 1.05194 1.97664 2014-04-01 14:50:52,745.745 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:50:42.667757', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438583328, u'health': u'HEALTH_OK', u'kb_used': 9745656, u'store_stats': {u'bytes_total': 3422377, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373785}}, {u'last_updated': u'2014-04-01 05:50:42.978092', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438301656, u'health': u'HEALTH_OK', u'kb_used': 10027328, u'store_stats': {u'bytes_total': 4470902, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373734}}, {u'last_updated': u'2014-04-01 05:50:43.038354', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438583320, u'health': u'HEALTH_OK', u'kb_used': 9745664, u'store_stats': {u'bytes_total': 4470939, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373771}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1849/25851 objects degraded (7.153%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1849, u'num_pgs': 158, u'data_bytes': 36066822500, u'degraded_total': 25851, u'bytes_used': 39642312704, u'op_per_sec': 4, u'write_bytes_sec': 20092273, u'version': 504, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.153', u'bytes_avail': 2295870550016}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:50:52,747.747 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:50:53,452.452 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:50:53,607.607 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 810 16 6577 6561 32.3952 64 0.434636 1.97414 2014-04-01 14:50:54,590.590 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 811 16 6587 6571 32.4045 40 0.32436 1.97217 2014-04-01 14:50:55,590.590 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 812 16 6587 6571 32.3646 0 - 1.97217 2014-04-01 14:50:56,590.590 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 813 16 6587 6571 32.3248 0 - 1.97217 2014-04-01 14:50:57,380.380 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:50:57,590.590 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 814 16 6587 6571 32.2851 0 - 1.97217 2014-04-01 14:50:58,150.150 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:50:58,591.591 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 815 16 6587 6571 32.2455 0 - 1.97217 2014-04-01 14:50:58,942.942 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:50:59,591.591 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 816 16 6609 6593 32.3138 17.6 0.509841 1.97901 2014-04-01 14:50:59,669.669 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:50:42.667757', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438583328, u'health': u'HEALTH_OK', u'kb_used': 9745656, u'store_stats': {u'bytes_total': 3422377, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373785}}, {u'last_updated': u'2014-04-01 05:50:42.978092', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438301656, u'health': u'HEALTH_OK', u'kb_used': 10027328, u'store_stats': {u'bytes_total': 4470902, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373734}}, {u'last_updated': u'2014-04-01 05:50:43.038354', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438583320, u'health': u'HEALTH_OK', u'kb_used': 9745664, u'store_stats': {u'bytes_total': 4470939, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373771}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1847/26001 objects degraded (7.104%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1847, u'num_pgs': 158, u'data_bytes': 36276537700, u'degraded_total': 26001, u'bytes_used': 41509199872, u'op_per_sec': 8, u'write_bytes_sec': 37433423, u'version': 508, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.104', u'bytes_avail': 2294003662848}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:50:59,670.670 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:51:00,593.593 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 817 16 6625 6609 32.3526 64 0.462254 1.97721 2014-04-01 14:51:00,622.622 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:51:01,594.594 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 818 16 6648 6632 32.4255 92 0.944754 1.97266 2014-04-01 14:51:02,591.591 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 819 16 6665 6649 32.4689 68 1 1.96991 2014-04-01 14:51:03,591.591 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:51:03.503996min lat: 0.064365 max lat: 110.858 avg lat: 1.96805 2014-04-01 14:51:03,592.592 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:51:03,592.592 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 820 16 6674 6658 32.4732 36 0.179176 1.96805 2014-04-01 14:51:04,406.406 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:51:04,598.598 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 821 16 6680 6664 32.4629 24 0.128694 1.96658 2014-04-01 14:51:05,115.115 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:51:05,593.593 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 822 16 6680 6664 32.4234 0 - 1.96658 2014-04-01 14:51:05,888.888 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:51:06,592.592 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 823 16 6680 6664 32.384 0 - 1.96658 2014-04-01 14:51:06,818.818 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:50:42.667757', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438583328, u'health': u'HEALTH_OK', u'kb_used': 9745656, u'store_stats': {u'bytes_total': 3422377, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373785}}, {u'last_updated': u'2014-04-01 05:50:42.978092', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438301656, u'health': u'HEALTH_OK', u'kb_used': 10027328, u'store_stats': {u'bytes_total': 4470902, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373734}}, {u'last_updated': u'2014-04-01 05:50:43.038354', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438583320, u'health': u'HEALTH_OK', u'kb_used': 9745664, u'store_stats': {u'bytes_total': 4470939, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373771}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1853/26280 objects degraded (7.051%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1853, u'num_pgs': 158, u'data_bytes': 36666607972, u'degraded_total': 26280, u'bytes_used': 42009931776, u'op_per_sec': 14, u'write_bytes_sec': 59130399, u'version': 514, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.051', u'bytes_avail': 2293502930944}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:51:06,818.818 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:51:07,596.596 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 824 16 6692 6676 32.4029 16 5.00629 1.97195 2014-04-01 14:51:07,806.806 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:51:08,592.592 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 825 16 6705 6689 32.4267 52 0.743611 1.97221 2014-04-01 14:51:09,592.592 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 826 16 6722 6706 32.4697 68 1.3501 1.97015 2014-04-01 14:51:10,592.592 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 827 16 6731 6715 32.474 36 0.725174 1.96855 2014-04-01 14:51:11,593.593 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 828 16 6732 6716 32.4396 4 0.289838 1.9683 2014-04-01 14:51:11,723.723 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:51:12,596.596 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:51:12,755.755 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 829 16 6732 6716 32.4005 0 - 1.9683 2014-04-01 14:51:13,381.381 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:51:13,593.593 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 830 16 6732 6716 32.3614 0 - 1.9683 2014-04-01 14:51:14,148.148 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:50:42.667757', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438583328, u'health': u'HEALTH_OK', u'kb_used': 9745656, u'store_stats': {u'bytes_total': 3422377, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373785}}, {u'last_updated': u'2014-04-01 05:50:42.978092', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438301656, u'health': u'HEALTH_OK', u'kb_used': 10027328, u'store_stats': {u'bytes_total': 4470902, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373734}}, {u'last_updated': u'2014-04-01 05:50:43.038354', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438583320, u'health': u'HEALTH_OK', u'kb_used': 9745664, u'store_stats': {u'bytes_total': 4470939, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373771}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1859/26442 objects degraded (7.030%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1859, u'num_pgs': 158, u'data_bytes': 36893100388, u'degraded_total': 26442, u'bytes_used': 42295582720, u'op_per_sec': 9, u'write_bytes_sec': 38560562, u'version': 518, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.030', u'bytes_avail': 2293217280000}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:51:14,149.149 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:51:14,593.593 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 831 16 6732 6716 32.3225 0 - 1.9683 2014-04-01 14:51:15,161.161 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:51:15,594.594 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 832 16 6757 6741 32.4038 25 0.699773 1.97413 2014-04-01 14:51:16,594.594 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 833 16 6773 6757 32.4417 64 1.18541 1.97131 2014-04-01 14:51:17,593.593 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 834 16 6784 6768 32.4556 44 1.44353 1.96967 2014-04-01 14:51:18,594.594 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 835 16 6787 6771 32.4311 12 0.195145 1.96913 2014-04-01 14:51:19,255.255 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:51:19,595.595 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 836 16 6787 6771 32.3923 0 - 1.96913 2014-04-01 14:51:20,235.235 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:51:20,597.597 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 837 16 6787 6771 32.3536 0 - 1.96913 2014-04-01 14:51:21,015.015 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:51:21,594.594 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 838 16 6790 6774 32.3293 4 5.30837 1.97063 2014-04-01 14:51:21,749.749 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:50:42.667757', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438583328, u'health': u'HEALTH_OK', u'kb_used': 9745656, u'store_stats': {u'bytes_total': 3422377, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373785}}, {u'last_updated': u'2014-04-01 05:50:42.978092', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438301656, u'health': u'HEALTH_OK', u'kb_used': 10027328, u'store_stats': {u'bytes_total': 4470902, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373734}}, {u'last_updated': u'2014-04-01 05:50:43.038354', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438583320, u'health': u'HEALTH_OK', u'kb_used': 9745664, u'store_stats': {u'bytes_total': 4470939, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373771}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1864/26607 objects degraded (7.006%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1864, u'num_pgs': 158, u'data_bytes': 37123787108, u'degraded_total': 26607, u'bytes_used': 42677686272, u'op_per_sec': 10, u'write_bytes_sec': 43536131, u'version': 523, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'7.006', u'bytes_avail': 2292835176448}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:51:21,750.750 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:51:22,595.595 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 839 16 6812 6796 32.3956 88 0.953809 1.97424 2014-04-01 14:51:22,702.702 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:51:23,597.597 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:51:23.506953min lat: 0.064365 max lat: 110.858 avg lat: 1.97062 2014-04-01 14:51:23,597.597 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:51:23,615.615 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 840 16 6833 6817 32.4571 84 0.972577 1.97062 2014-04-01 14:51:24,595.595 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 841 16 6845 6829 32.4755 48 0.360879 1.96852 2014-04-01 14:51:25,595.595 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 842 16 6848 6832 32.4512 12 0.39784 1.96787 2014-04-01 14:51:26,595.595 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 843 16 6848 6832 32.4127 0 - 1.96787 2014-04-01 14:51:26,618.618 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:51:27,436.436 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:51:27,743.743 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 844 16 6862 6846 32.4407 28 0.145859 1.97048 2014-04-01 14:51:28,226.226 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:51:28,595.595 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 845 16 6880 6864 32.4875 72 0.744325 1.96848 2014-04-01 14:51:29,047.047 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:50:42.667757', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438583328, u'health': u'HEALTH_OK', u'kb_used': 9745656, u'store_stats': {u'bytes_total': 3422377, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373785}}, {u'last_updated': u'2014-04-01 05:50:42.978092', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438301656, u'health': u'HEALTH_OK', u'kb_used': 10027328, u'store_stats': {u'bytes_total': 4470902, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373734}}, {u'last_updated': u'2014-04-01 05:50:43.038354', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438583320, u'health': u'HEALTH_OK', u'kb_used': 9745664, u'store_stats': {u'bytes_total': 4470939, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373771}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1872/26772 objects degraded (6.992%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1872, u'num_pgs': 158, u'data_bytes': 37354473828, u'degraded_total': 26772, u'bytes_used': 42818478080, u'op_per_sec': 12, u'write_bytes_sec': 53874772, u'version': 527, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.992', u'bytes_avail': 2292694384640}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:51:29,049.049 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:51:29,600.600 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 846 16 6901 6885 32.5483 84 0.486216 1.96512 2014-04-01 14:51:29,938.938 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:51:30,598.598 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 847 16 6914 6898 32.5713 52 0.850611 1.96306 2014-04-01 14:51:31,596.596 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 848 16 6919 6903 32.5565 20 2.14793 1.96231 2014-04-01 14:51:32,596.596 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 849 16 6920 6904 32.5228 4 0.600983 1.96211 2014-04-01 14:51:33,596.596 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 850 16 6920 6904 32.4846 0 - 1.96211 2014-04-01 14:51:33,726.726 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:51:34,597.597 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:51:34,760.760 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 851 16 6920 6904 32.4464 0 - 1.96211 2014-04-01 14:51:35,851.851 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 852 16 6940 6924 32.5022 26.6667 0.854224 1.96812 2014-04-01 14:51:35,869.869 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:51:36,639.639 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 853 16 6962 6946 32.5672 88 0.693463 1.96363 2014-04-01 14:51:36,697.697 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:50:42.667757', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438583328, u'health': u'HEALTH_OK', u'kb_used': 9745656, u'store_stats': {u'bytes_total': 3422377, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373785}}, {u'last_updated': u'2014-04-01 05:50:42.978092', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438301656, u'health': u'HEALTH_OK', u'kb_used': 10027328, u'store_stats': {u'bytes_total': 4470902, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373734}}, {u'last_updated': u'2014-04-01 05:50:43.038354', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438583320, u'health': u'HEALTH_OK', u'kb_used': 9745664, u'store_stats': {u'bytes_total': 4470939, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373771}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1872/27015 objects degraded (6.929%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1872, u'num_pgs': 158, u'data_bytes': 37694212452, u'degraded_total': 27015, u'bytes_used': 43190853632, u'op_per_sec': 8, u'write_bytes_sec': 35373868, u'version': 532, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.929', u'bytes_avail': 2292322009088}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:51:36,698.698 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:51:37,684.684 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:51:38,097.097 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 854 16 6970 6954 32.5666 32 0.575743 1.96239 2014-04-01 14:51:38,699.699 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 855 16 6971 6955 32.5331 4 0.277891 1.96215 2014-04-01 14:51:39,618.618 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 856 16 6971 6955 32.4951 0 - 1.96215 2014-04-01 14:51:40,606.606 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 857 16 6985 6969 32.5226 28 3.86751 1.96616 2014-04-01 14:51:41,604.604 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 858 16 7007 6991 32.5872 88 0.457383 1.96329 2014-04-01 14:51:42,089.089 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:51:42,597.597 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 859 16 7018 7002 32.6005 44 0.919325 1.96145 2014-04-01 14:51:42,803.803 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:51:43,586.586 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:51:43,596.596 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:51:43.510077min lat: 0.064365 max lat: 110.858 avg lat: 1.95857 2014-04-01 14:51:43,597.597 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:51:43,597.597 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 860 16 7037 7021 32.6509 76 0.740587 1.95857 2014-04-01 14:51:44,355.355 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 6, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.103187', u'skew': u'-0.229086', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.103115', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:51:42.668056', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438541228, u'health': u'HEALTH_OK', u'kb_used': 9787756, u'store_stats': {u'bytes_total': 5519529, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373785}}, {u'last_updated': u'2014-04-01 05:51:42.978314', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438219580, u'health': u'HEALTH_OK', u'kb_used': 10109404, u'store_stats': {u'bytes_total': 6568054, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373734}}, {u'last_updated': u'2014-04-01 05:51:43.038598', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438540896, u'health': u'HEALTH_OK', u'kb_used': 9788088, u'store_stats': {u'bytes_total': 6568091, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373771}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1880/27222 objects degraded (6.906%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1880, u'num_pgs': 158, u'data_bytes': 37983619428, u'degraded_total': 27222, u'bytes_used': 43577520128, u'op_per_sec': 7, u'write_bytes_sec': 32349633, u'version': 537, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.906', u'bytes_avail': 2291935342592}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:51:44,356.356 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:51:44,597.597 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 861 16 7041 7025 32.6316 16 0.145779 1.95789 2014-04-01 14:51:45,164.164 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:51:45,598.598 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 862 16 7041 7025 32.5937 0 - 1.95789 2014-04-01 14:51:46,599.599 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 863 16 7041 7025 32.556 0 - 1.95789 2014-04-01 14:51:47,598.598 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 864 16 7041 7025 32.5183 0 - 1.95789 2014-04-01 14:51:48,601.601 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 865 16 7060 7044 32.5685 19 0.336819 1.96402 2014-04-01 14:51:48,927.927 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:51:49,646.646 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:51:49,810.810 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 05:51:49.476235 7fe3a7efe700 0 -- 10.214.133.37:0/1005437 >> 10.214.133.38:6800/4373 pipe(0x7fe3ac00ade0 sd=12 :47050 s=2 pgs=12 cs=1 l=1 c=0x7fe3ac00fa60).injecting socket failure 2014-04-01 14:51:49,810.810 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 866 16 7076 7060 32.6048 64 1.30772 1.96191 2014-04-01 14:51:50,448.448 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:51:50,601.601 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 867 16 7098 7082 32.6687 88 0.570157 1.95818 2014-04-01 14:51:51,308.308 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.082877', u'skew': u'-0.182834', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.082621', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:51:42.668056', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438541228, u'health': u'HEALTH_OK', u'kb_used': 9787756, u'store_stats': {u'bytes_total': 5519529, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373785}}, {u'last_updated': u'2014-04-01 05:51:42.978314', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438219580, u'health': u'HEALTH_OK', u'kb_used': 10109404, u'store_stats': {u'bytes_total': 6568054, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373734}}, {u'last_updated': u'2014-04-01 05:51:43.038598', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438540896, u'health': u'HEALTH_OK', u'kb_used': 9788088, u'store_stats': {u'bytes_total': 6568091, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373771}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1891/27417 objects degraded (6.897%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1891, u'num_pgs': 158, u'data_bytes': 38256249188, u'degraded_total': 27417, u'bytes_used': 43788161024, u'op_per_sec': 4, u'write_bytes_sec': 19118755, u'version': 542, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.897', u'bytes_avail': 2291724701696}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:51:51,309.309 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:51:51,598.598 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 868 16 7112 7096 32.6956 56 1.27178 1.95631 2014-04-01 14:51:52,350.350 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:51:52,601.601 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 869 16 7121 7105 32.6994 36 0.183786 1.95501 2014-04-01 14:51:53,598.598 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 870 16 7121 7105 32.6618 0 - 1.95501 2014-04-01 14:51:54,599.599 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 871 16 7121 7105 32.6243 0 - 1.95501 2014-04-01 14:51:55,599.599 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 872 16 7121 7105 32.5869 0 - 1.95501 2014-04-01 14:51:56,114.114 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:51:56,602.602 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 873 16 7124 7108 32.5633 3 5.71711 1.95657 2014-04-01 14:51:56,909.909 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:51:57,602.602 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 874 16 7151 7135 32.6496 108 0.394217 1.95931 2014-04-01 14:51:57,889.889 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:51:58,603.603 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 875 16 7167 7151 32.6854 64 1.35641 1.95669 2014-04-01 14:51:58,642.642 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.082877', u'skew': u'-0.182834', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.082621', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:51:42.668056', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438541228, u'health': u'HEALTH_OK', u'kb_used': 9787756, u'store_stats': {u'bytes_total': 5519529, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373785}}, {u'last_updated': u'2014-04-01 05:51:42.978314', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438219580, u'health': u'HEALTH_OK', u'kb_used': 10109404, u'store_stats': {u'bytes_total': 6568054, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373734}}, {u'last_updated': u'2014-04-01 05:51:43.038598', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438540896, u'health': u'HEALTH_OK', u'kb_used': 9788088, u'store_stats': {u'bytes_total': 6568091, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373771}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1902/27597 objects degraded (6.892%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1902, u'num_pgs': 158, u'data_bytes': 38507907428, u'degraded_total': 27597, u'bytes_used': 44269518848, u'op_per_sec': 7, u'write_bytes_sec': 29667722, u'version': 546, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.892', u'bytes_avail': 2291243343872}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:51:58,643.643 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:51:59,389.389 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:51:59,698.698 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 876 16 7179 7163 32.7029 48 2.05401 1.95504 2014-04-01 14:52:00,601.601 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 877 16 7186 7170 32.6975 28 0.266489 1.95383 2014-04-01 14:52:01,602.602 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 878 16 7186 7170 32.6603 0 - 1.95383 2014-04-01 14:52:02,605.605 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 879 16 7196 7180 32.6686 20 3.67464 1.9567 2014-04-01 14:52:03,304.304 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:52:03,607.607 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:52:03.513078min lat: 0.064365 max lat: 110.858 avg lat: 1.95484 2014-04-01 14:52:03,607.607 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:52:03,607.607 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 880 16 7217 7201 32.7269 84 0.325499 1.95484 2014-04-01 14:52:04,158.158 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:52:04,601.601 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 881 16 7232 7216 32.7579 60 0.510716 1.95207 2014-04-01 14:52:04,934.934 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:52:05,601.601 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 882 16 7244 7228 32.7751 48 2.01118 1.9503 2014-04-01 14:52:05,653.653 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.082877', u'skew': u'-0.182834', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.082621', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:51:42.668056', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438541228, u'health': u'HEALTH_OK', u'kb_used': 9787756, u'store_stats': {u'bytes_total': 5519529, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373785}}, {u'last_updated': u'2014-04-01 05:51:42.978314', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438219580, u'health': u'HEALTH_OK', u'kb_used': 10109404, u'store_stats': {u'bytes_total': 6568054, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373734}}, {u'last_updated': u'2014-04-01 05:51:43.038598', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438540896, u'health': u'HEALTH_OK', u'kb_used': 9788088, u'store_stats': {u'bytes_total': 6568091, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373771}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1910/27861 objects degraded (6.855%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1910, u'num_pgs': 158, u'data_bytes': 38877006180, u'degraded_total': 27861, u'bytes_used': 44626042880, u'op_per_sec': 9, u'write_bytes_sec': 41253149, u'version': 552, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.855', u'bytes_avail': 2290886819840}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:52:05,653.653 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:52:06,601.601 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 883 16 7260 7244 32.8105 64 0.185219 1.94905 2014-04-01 14:52:06,620.620 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:52:07,601.601 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 884 16 7261 7245 32.7779 4 0.461069 1.94885 2014-04-01 14:52:08,602.602 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 885 16 7261 7245 32.7409 0 - 1.94885 2014-04-01 14:52:09,602.602 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 886 16 7261 7245 32.7039 0 - 1.94885 2014-04-01 14:52:10,387.387 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:52:10,605.605 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 887 15 7277 7262 32.7437 22.6667 0.580002 1.95288 2014-04-01 14:52:11,110.110 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:52:11,602.602 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 888 16 7301 7285 32.8104 92 0.815393 1.94965 2014-04-01 14:52:12,074.074 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:52:12,605.605 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 889 16 7312 7296 32.823 44 1.39206 1.94804 2014-04-01 14:52:12,657.657 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 05:52:12.569520 7fbd16ffd700 0 monclient: hunting for new mon 2014-04-01 14:52:12,777.777 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.082877', u'skew': u'-0.182834', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.082621', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:51:42.668056', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438541228, u'health': u'HEALTH_OK', u'kb_used': 9787756, u'store_stats': {u'bytes_total': 5519529, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373785}}, {u'last_updated': u'2014-04-01 05:51:42.978314', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438219580, u'health': u'HEALTH_OK', u'kb_used': 10109404, u'store_stats': {u'bytes_total': 6568054, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373734}}, {u'last_updated': u'2014-04-01 05:51:43.038598', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438540896, u'health': u'HEALTH_OK', u'kb_used': 9788088, u'store_stats': {u'bytes_total': 6568091, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373771}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1903/28059 objects degraded (6.782%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1903, u'num_pgs': 158, u'data_bytes': 39153830244, u'degraded_total': 28059, u'bytes_used': 44994838528, u'op_per_sec': 7, u'write_bytes_sec': 31045124, u'version': 557, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.782', u'bytes_avail': 2290518024192}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:52:12,777.777 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:52:13,519.519 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:52:13,679.679 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 890 16 7319 7303 32.8176 28 0.226906 1.94685 2014-04-01 14:52:14,603.603 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 891 16 7319 7303 32.7807 0 - 1.94685 2014-04-01 14:52:15,603.603 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 892 16 7319 7303 32.744 0 - 1.94685 2014-04-01 14:52:16,614.614 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 893 16 7319 7303 32.7073 0 - 1.94685 2014-04-01 14:52:17,443.443 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:52:17,606.606 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 894 16 7339 7323 32.7602 20 0.432361 1.95223 2014-04-01 14:52:18,067.067 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:52:18,605.605 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 895 16 7356 7340 32.7996 68 0.185025 1.94941 2014-04-01 14:52:19,010.010 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:52:19,604.604 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 896 16 7370 7354 32.8254 56 0.757061 1.94798 2014-04-01 14:52:19,717.717 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.082877', u'skew': u'-0.182834', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.082621', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:51:42.668056', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438541228, u'health': u'HEALTH_OK', u'kb_used': 9787756, u'store_stats': {u'bytes_total': 5519529, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373785}}, {u'last_updated': u'2014-04-01 05:51:42.978314', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438219580, u'health': u'HEALTH_OK', u'kb_used': 10109404, u'store_stats': {u'bytes_total': 6568054, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373734}}, {u'last_updated': u'2014-04-01 05:51:43.038598', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438540896, u'health': u'HEALTH_OK', u'kb_used': 9788088, u'store_stats': {u'bytes_total': 6568091, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373771}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1906/28257 objects degraded (6.745%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1906, u'num_pgs': 158, u'data_bytes': 39430654308, u'degraded_total': 28257, u'bytes_used': 45286027264, u'op_per_sec': 11, u'write_bytes_sec': 50300388, u'version': 561, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.745', u'bytes_avail': 2290226835456}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:52:19,717.717 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:52:20,435.435 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:52:20,732.732 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 897 16 7381 7365 32.8379 44 0.167687 1.94605 2014-04-01 14:52:21,604.604 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 898 16 7381 7365 32.8013 0 - 1.94605 2014-04-01 14:52:22,604.604 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 899 16 7381 7365 32.7648 0 - 1.94605 2014-04-01 14:52:23,604.604 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:52:23.516249min lat: 0.064365 max lat: 110.858 avg lat: 1.94605 2014-04-01 14:52:23,604.604 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:52:23,605.605 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 900 16 7381 7365 32.7284 0 - 1.94605 2014-04-01 14:52:24,195.195 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:52:24,606.606 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 901 16 7389 7373 32.7276 8 0.164688 1.94905 2014-04-01 14:52:25,607.607 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 902 16 7411 7395 32.7889 88 0.883085 1.95091 2014-04-01 14:52:26,607.607 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 903 16 7429 7413 32.8323 72 1.0333 1.94819 2014-04-01 14:52:27,605.605 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 904 16 7443 7427 32.8579 56 0.832954 1.94624 2014-04-01 14:52:27,993.993 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:52:28,612.612 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 905 16 7449 7433 32.8481 24 0.534025 1.9454 2014-04-01 14:52:28,765.765 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:52:29,486.486 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.082877', u'skew': u'-0.182834', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.082621', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:51:42.668056', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438541228, u'health': u'HEALTH_OK', u'kb_used': 9787756, u'store_stats': {u'bytes_total': 5519529, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373785}}, {u'last_updated': u'2014-04-01 05:51:42.978314', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438219580, u'health': u'HEALTH_OK', u'kb_used': 10109404, u'store_stats': {u'bytes_total': 6568054, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373734}}, {u'last_updated': u'2014-04-01 05:51:43.038598', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438540896, u'health': u'HEALTH_OK', u'kb_used': 9788088, u'store_stats': {u'bytes_total': 6568091, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373771}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1906/28473 objects degraded (6.694%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1906, u'num_pgs': 158, u'data_bytes': 39732644196, u'degraded_total': 28473, u'bytes_used': 45587070976, u'op_per_sec': 5, u'write_bytes_sec': 23592441, u'version': 567, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.694', u'bytes_avail': 2289925791744}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:52:29,487.487 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:52:29,605.605 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 906 16 7449 7433 32.8119 0 - 1.9454 2014-04-01 14:52:30,255.255 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:52:30,607.607 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 907 16 7449 7433 32.7757 0 - 1.9454 2014-04-01 14:52:31,606.606 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 908 16 7449 7433 32.7396 0 - 1.9454 2014-04-01 14:52:32,610.610 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 909 16 7468 7452 32.7872 19 0.589024 1.95145 2014-04-01 14:52:33,608.608 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 910 16 7481 7465 32.8083 52 1.07044 1.94936 2014-04-01 14:52:34,027.027 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:52:34,610.610 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 911 16 7498 7482 32.8469 68 0.639721 1.94713 2014-04-01 14:52:34,814.814 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:52:35,593.593 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:52:35,605.605 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 912 16 7511 7495 32.8679 52 0.532256 1.94524 2014-04-01 14:52:36,311.311 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.082877', u'skew': u'-0.182834', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.082621', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:51:42.668056', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438541228, u'health': u'HEALTH_OK', u'kb_used': 9787756, u'store_stats': {u'bytes_total': 5519529, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373785}}, {u'last_updated': u'2014-04-01 05:51:42.978314', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438219580, u'health': u'HEALTH_OK', u'kb_used': 10109404, u'store_stats': {u'bytes_total': 6568054, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373734}}, {u'last_updated': u'2014-04-01 05:51:43.038598', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438540896, u'health': u'HEALTH_OK', u'kb_used': 9788088, u'store_stats': {u'bytes_total': 6568091, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373771}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1914/28656 objects degraded (6.679%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1914, u'num_pgs': 158, u'data_bytes': 39988496740, u'degraded_total': 28656, u'bytes_used': 45915971584, u'op_per_sec': 7, u'write_bytes_sec': 30236427, u'version': 572, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.679', u'bytes_avail': 2289596891136}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:52:36,312.312 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:52:36,608.608 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 913 16 7513 7497 32.8406 8 0.404333 1.94485 2014-04-01 14:52:37,016.016 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:52:37,607.607 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 914 16 7513 7497 32.8047 0 - 1.94485 2014-04-01 14:52:38,607.607 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 915 16 7513 7497 32.7689 0 - 1.94485 2014-04-01 14:52:39,608.608 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 916 16 7529 7513 32.8029 21.3333 0.260676 1.94909 2014-04-01 14:52:40,045.045 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 05:52:39.952638 7fe3a7efe700 0 -- 10.214.133.37:0/1005437 >> 10.214.133.38:6800/4373 pipe(0x7fe3ac017a10 sd=13 :47202 s=2 pgs=13 cs=1 l=1 c=0x7fe3ac00b4e0).injecting socket failure 2014-04-01 14:52:40,607.607 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 917 16 7547 7531 32.8457 72 0.16475 1.94714 2014-04-01 14:52:40,787.787 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:52:41,676.676 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:52:41,827.827 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 918 16 7559 7543 32.8622 48 0.625342 1.94557 2014-04-01 14:52:42,457.457 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:52:42,608.608 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 919 16 7571 7555 32.8786 48 0.651814 1.94449 2014-04-01 14:52:43,140.140 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.082877', u'skew': u'-0.182834', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.082621', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:52:42.668361', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438498324, u'health': u'HEALTH_OK', u'kb_used': 9830660, u'store_stats': {u'bytes_total': 6568105, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373785}}, {u'last_updated': u'2014-04-01 05:52:42.978588', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438137764, u'health': u'HEALTH_OK', u'kb_used': 10191220, u'store_stats': {u'bytes_total': 8665206, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373734}}, {u'last_updated': u'2014-04-01 05:52:43.038899', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438498272, u'health': u'HEALTH_OK', u'kb_used': 9830712, u'store_stats': {u'bytes_total': 8665243, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373771}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1924/28848 objects degraded (6.669%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1924, u'num_pgs': 158, u'data_bytes': 40256932196, u'degraded_total': 28848, u'bytes_used': 46134816768, u'op_per_sec': 8, u'write_bytes_sec': 35966103, u'version': 576, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.669', u'bytes_avail': 2289378045952}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:52:43,140.140 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:52:43,625.625 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:52:43.519265min lat: 0.064365 max lat: 110.858 avg lat: 1.94432 2014-04-01 14:52:43,626.626 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:52:43,626.626 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 920 16 7572 7556 32.8473 4 0.615355 1.94432 2014-04-01 14:52:43,864.864 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:52:44,616.616 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 921 16 7572 7556 32.8116 0 - 1.94432 2014-04-01 14:52:45,615.615 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 922 16 7572 7556 32.776 0 - 1.94432 2014-04-01 14:52:46,614.614 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 923 16 7579 7563 32.7708 9.33333 0.258956 1.94675 2014-04-01 14:52:47,618.618 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 924 16 7602 7586 32.8349 92 0.776836 1.94848 2014-04-01 14:52:47,850.850 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:52:48,438.438 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:52:48,767.767 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 925 16 7620 7604 32.8772 72 1.01798 1.94595 2014-04-01 14:52:49,512.512 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:52:49,613.613 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 926 16 7630 7614 32.8849 40 0.200593 1.94432 2014-04-01 14:52:50,275.275 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.082877', u'skew': u'-0.182834', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.082621', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:52:42.668361', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438498324, u'health': u'HEALTH_OK', u'kb_used': 9830660, u'store_stats': {u'bytes_total': 6568105, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373785}}, {u'last_updated': u'2014-04-01 05:52:42.978588', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438137764, u'health': u'HEALTH_OK', u'kb_used': 10191220, u'store_stats': {u'bytes_total': 8665206, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373734}}, {u'last_updated': u'2014-04-01 05:52:43.038899', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438498272, u'health': u'HEALTH_OK', u'kb_used': 9830712, u'store_stats': {u'bytes_total': 8665243, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373771}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1927/29022 objects degraded (6.640%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1927, u'num_pgs': 158, u'data_bytes': 40500201828, u'degraded_total': 29022, u'bytes_used': 46543339520, u'op_per_sec': 7, u'write_bytes_sec': 31867820, u'version': 581, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.640', u'bytes_avail': 2288969523200}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:52:50,276.276 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:52:50,613.613 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 927 16 7632 7616 32.8581 8 0.265449 1.94393 2014-04-01 14:52:50,997.997 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:52:51,609.609 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 928 16 7632 7616 32.8227 0 - 1.94393 2014-04-01 14:52:52,609.609 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 929 16 7632 7616 32.7873 0 - 1.94393 2014-04-01 14:52:53,610.610 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 930 16 7632 7616 32.7521 0 - 1.94393 2014-04-01 14:52:54,612.612 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 931 16 7646 7630 32.777 14 5.61379 1.95014 2014-04-01 14:52:54,915.915 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:52:55,697.697 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:52:55,856.856 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 932 16 7669 7653 32.8406 92 0.444142 1.94779 2014-04-01 14:52:56,486.486 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:52:56,610.610 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 933 16 7691 7675 32.8997 88 0.319916 1.94452 2014-04-01 14:52:57,316.316 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.082877', u'skew': u'-0.182834', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.082621', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:52:42.668361', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438498324, u'health': u'HEALTH_OK', u'kb_used': 9830660, u'store_stats': {u'bytes_total': 6568105, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373785}}, {u'last_updated': u'2014-04-01 05:52:42.978588', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438137764, u'health': u'HEALTH_OK', u'kb_used': 10191220, u'store_stats': {u'bytes_total': 8665206, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373734}}, {u'last_updated': u'2014-04-01 05:52:43.038899', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438498272, u'health': u'HEALTH_OK', u'kb_used': 9830712, u'store_stats': {u'bytes_total': 8665243, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373771}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1942/29214 objects degraded (6.647%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1942, u'num_pgs': 158, u'data_bytes': 40768637284, u'degraded_total': 29214, u'bytes_used': 48778903552, u'op_per_sec': 8, u'write_bytes_sec': 35013982, u'version': 586, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.647', u'bytes_avail': 2286733959168}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:52:57,316.316 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:52:57,609.609 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 934 16 7699 7683 32.8987 32 0.947791 1.94352 2014-04-01 14:52:58,022.022 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:52:58,662.662 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 935 16 7699 7683 32.8635 0 - 1.94352 2014-04-01 14:52:59,610.610 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 936 16 7699 7683 32.8284 0 - 1.94352 2014-04-01 14:53:00,610.610 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 937 16 7699 7683 32.7934 0 - 1.94352 2014-04-01 14:53:01,614.614 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 938 16 7703 7687 32.7755 4 5.32348 1.94537 2014-04-01 14:53:01,934.934 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:53:02,670.670 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:53:02,831.831 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 939 16 7728 7712 32.8471 100 0.62592 1.94737 2014-04-01 14:53:03,468.468 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:53:03,611.611 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:53:03.522298min lat: 0.064365 max lat: 110.858 avg lat: 1.9439 2014-04-01 14:53:03,611.611 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:53:03,611.611 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 940 16 7751 7735 32.91 92 0.568297 1.9439 2014-04-01 14:53:04,189.189 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.082877', u'skew': u'-0.182834', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.082621', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:52:42.668361', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438498324, u'health': u'HEALTH_OK', u'kb_used': 9830660, u'store_stats': {u'bytes_total': 6568105, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373785}}, {u'last_updated': u'2014-04-01 05:52:42.978588', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438137764, u'health': u'HEALTH_OK', u'kb_used': 10191220, u'store_stats': {u'bytes_total': 8665206, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373734}}, {u'last_updated': u'2014-04-01 05:52:43.038899', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438498272, u'health': u'HEALTH_OK', u'kb_used': 9830712, u'store_stats': {u'bytes_total': 8665243, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373771}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'3 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1936/29364 objects degraded (6.593%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1936, u'num_pgs': 158, u'data_bytes': 40978352484, u'degraded_total': 29364, u'bytes_used': 48945385472, u'op_per_sec': 8, u'write_bytes_sec': 37487902, u'version': 590, u'pgs_by_state': [{u'count': 155, u'state_name': u'active+clean'}, {u'count': 3, u'state_name': u'active+recovering'}], u'degraded_ratio': u'6.593', u'bytes_avail': 2286567477248}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:53:04,190.190 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:53:04,610.610 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 941 16 7764 7748 32.9302 52 0.503066 1.94188 2014-04-01 14:53:04,898.898 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:53:05,724.724 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 942 16 7769 7753 32.9165 20 0.529879 1.94093 2014-04-01 14:53:06,613.613 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 943 16 7769 7753 32.8816 0 - 1.94093 2014-04-01 14:53:07,613.613 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 944 16 7769 7753 32.8468 0 - 1.94093 2014-04-01 14:53:08,611.611 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 945 16 7769 7753 32.812 0 - 1.94093 2014-04-01 14:53:08,882.882 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:53:09,710.710 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:53:09,862.862 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 946 16 7769 7753 32.7773 0 - 1.94093 2014-04-01 14:53:10,490.490 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:53:10,616.616 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 947 16 7797 7781 32.861 22.4 0.242371 1.94691 2014-04-01 14:53:11,535.535 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.082877', u'skew': u'-0.182834', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.082621', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:52:42.668361', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438498324, u'health': u'HEALTH_OK', u'kb_used': 9830660, u'store_stats': {u'bytes_total': 6568105, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373785}}, {u'last_updated': u'2014-04-01 05:52:42.978588', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438137764, u'health': u'HEALTH_OK', u'kb_used': 10191220, u'store_stats': {u'bytes_total': 8665206, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373734}}, {u'last_updated': u'2014-04-01 05:52:43.038899', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438498272, u'health': u'HEALTH_OK', u'kb_used': 9830712, u'store_stats': {u'bytes_total': 8665243, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373771}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1204/29598 objects degraded (4.068%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1204, u'num_pgs': 158, u'data_bytes': 41305508196, u'degraded_total': 29598, u'bytes_used': 49414221824, u'op_per_sec': 5, u'write_bytes_sec': 21829672, u'version': 595, u'pgs_by_state': [{u'count': 156, u'state_name': u'active+clean'}, {u'count': 2, u'state_name': u'active+recovering'}], u'degraded_ratio': u'4.068', u'bytes_avail': 2286098640896}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:53:11,535.535 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:53:11,611.611 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 948 16 7819 7803 32.9191 88 0.831776 1.94336 2014-04-01 14:53:12,395.395 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:53:12,729.729 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 949 16 7835 7819 32.9519 64 1.06734 1.94064 2014-04-01 14:53:13,614.614 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 950 16 7846 7830 32.9635 44 0.406593 1.93885 2014-04-01 14:53:14,612.612 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 951 16 7846 7830 32.9288 0 - 1.93885 2014-04-01 14:53:15,613.613 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 952 16 7846 7830 32.8942 0 - 1.93885 2014-04-01 14:53:16,264.264 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:53:16,613.613 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 953 16 7846 7830 32.8597 0 - 1.93885 2014-04-01 14:53:16,972.972 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:53:17,615.615 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 954 16 7860 7844 32.884 14 5.33965 1.94475 2014-04-01 14:53:17,755.755 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:53:18,506.506 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.082877', u'skew': u'-0.182834', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.082621', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:52:42.668361', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438498324, u'health': u'HEALTH_OK', u'kb_used': 9830660, u'store_stats': {u'bytes_total': 6568105, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373785}}, {u'last_updated': u'2014-04-01 05:52:42.978588', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438137764, u'health': u'HEALTH_OK', u'kb_used': 10191220, u'store_stats': {u'bytes_total': 8665206, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373734}}, {u'last_updated': u'2014-04-01 05:52:43.038899', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438498272, u'health': u'HEALTH_OK', u'kb_used': 9830712, u'store_stats': {u'bytes_total': 8665243, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373771}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1201/29805 objects degraded (4.030%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1201, u'num_pgs': 158, u'data_bytes': 41594915172, u'degraded_total': 29805, u'bytes_used': 49578520576, u'op_per_sec': 11, u'write_bytes_sec': 46278379, u'version': 599, u'pgs_by_state': [{u'count': 156, u'state_name': u'active+clean'}, {u'count': 2, u'state_name': u'active+recovering'}], u'degraded_ratio': u'4.030', u'bytes_avail': 2285934342144}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:53:18,507.507 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:53:18,616.616 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 955 16 7873 7857 32.904 52 1.32 1.94363 2014-04-01 14:53:19,440.440 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:53:19,618.618 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 956 16 7896 7880 32.9658 92 0.582501 1.94053 2014-04-01 14:53:20,614.614 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 957 16 7913 7897 33.0024 68 0.60919 1.93805 2014-04-01 14:53:21,612.612 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 958 16 7918 7902 32.9888 20 0.398548 1.93723 2014-04-01 14:53:22,613.613 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 959 16 7919 7903 32.9586 4 0.394433 1.93704 2014-04-01 14:53:23,230.230 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:53:23,616.616 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:53:23.525398min lat: 0.064365 max lat: 110.858 avg lat: 1.93704 2014-04-01 14:53:23,616.616 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:53:23,617.617 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 960 16 7919 7903 32.9242 0 - 1.93704 2014-04-01 14:53:23,929.929 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:53:24,614.614 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 961 16 7919 7903 32.89 0 - 1.93704 2014-04-01 14:53:24,706.706 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:53:25,463.463 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.082877', u'skew': u'-0.182834', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.082621', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:52:42.668361', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438498324, u'health': u'HEALTH_OK', u'kb_used': 9830660, u'store_stats': {u'bytes_total': 6568105, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373785}}, {u'last_updated': u'2014-04-01 05:52:42.978588', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438137764, u'health': u'HEALTH_OK', u'kb_used': 10191220, u'store_stats': {u'bytes_total': 8665206, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373734}}, {u'last_updated': u'2014-04-01 05:52:43.038899', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438498272, u'health': u'HEALTH_OK', u'kb_used': 9830712, u'store_stats': {u'bytes_total': 8665243, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373771}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1204/30003 objects degraded (4.013%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1204, u'num_pgs': 158, u'data_bytes': 41871739236, u'degraded_total': 30003, u'bytes_used': 50013868032, u'op_per_sec': 7, u'write_bytes_sec': 31856118, u'version': 604, u'pgs_by_state': [{u'count': 156, u'state_name': u'active+clean'}, {u'count': 2, u'state_name': u'active+recovering'}], u'degraded_ratio': u'4.013', u'bytes_avail': 2285498994688}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:53:25,464.464 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:53:25,617.617 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 962 16 7935 7919 32.9223 21.3333 4.55954 1.94203 2014-04-01 14:53:26,541.541 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:53:26,694.694 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 963 16 7949 7933 32.9463 56 0.795883 1.94121 2014-04-01 14:53:27,617.617 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 964 16 7973 7957 33.0117 96 0.866576 1.93779 2014-04-01 14:53:28,615.615 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 965 16 7983 7967 33.0189 40 0.754272 1.93644 2014-04-01 14:53:29,615.615 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 966 16 7984 7968 32.9888 4 0.862782 1.93631 2014-04-01 14:53:30,465.465 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:53:30,616.616 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 967 16 7984 7968 32.9547 0 - 1.93631 2014-04-01 14:53:31,219.219 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:53:31,617.617 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 968 16 7984 7968 32.9207 0 - 1.93631 2014-04-01 14:53:31,993.993 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:53:32,615.615 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 969 16 7984 7968 32.8867 0 - 1.93631 2014-04-01 14:53:32,769.769 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.082877', u'skew': u'-0.182834', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.082621', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:52:42.668361', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438498324, u'health': u'HEALTH_OK', u'kb_used': 9830660, u'store_stats': {u'bytes_total': 6568105, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373785}}, {u'last_updated': u'2014-04-01 05:52:42.978588', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438137764, u'health': u'HEALTH_OK', u'kb_used': 10191220, u'store_stats': {u'bytes_total': 8665206, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373734}}, {u'last_updated': u'2014-04-01 05:52:43.038899', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438498272, u'health': u'HEALTH_OK', u'kb_used': 9830712, u'store_stats': {u'bytes_total': 8665243, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373771}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1206/30210 objects degraded (3.992%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1206, u'num_pgs': 158, u'data_bytes': 42161146212, u'degraded_total': 30210, u'bytes_used': 50256027648, u'op_per_sec': 11, u'write_bytes_sec': 49196213, u'version': 609, u'pgs_by_state': [{u'count': 156, u'state_name': u'active+clean'}, {u'count': 2, u'state_name': u'active+recovering'}], u'degraded_ratio': u'3.992', u'bytes_avail': 2285256835072}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:53:32,769.769 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:53:33,617.617 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 970 16 8013 7997 32.9724 29 0.576624 1.94068 2014-04-01 14:53:33,657.657 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:53:34,704.704 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 971 16 8029 8013 33.0043 64 1.07805 1.93816 2014-04-01 14:53:35,619.619 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 972 16 8049 8033 33.0527 80 0.845584 1.93549 2014-04-01 14:53:36,616.616 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 973 16 8051 8035 33.0269 8 0.729138 1.9352 2014-04-01 14:53:37,616.616 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 974 16 8051 8035 32.993 0 - 1.9352 2014-04-01 14:53:37,704.704 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:53:38,600.600 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:53:38,803.803 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 975 16 8051 8035 32.9592 0 - 1.9352 2014-04-01 14:53:39,616.616 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 976 16 8051 8035 32.9254 0 - 1.9352 2014-04-01 14:53:39,684.684 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:53:40,616.616 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 977 16 8051 8035 32.8917 0 - 1.9352 2014-04-01 14:53:40,660.660 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.082877', u'skew': u'-0.182834', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.082621', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:52:42.668361', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438498324, u'health': u'HEALTH_OK', u'kb_used': 9830660, u'store_stats': {u'bytes_total': 6568105, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373785}}, {u'last_updated': u'2014-04-01 05:52:42.978588', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438137764, u'health': u'HEALTH_OK', u'kb_used': 10191220, u'store_stats': {u'bytes_total': 8665206, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373734}}, {u'last_updated': u'2014-04-01 05:52:43.038899', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438498272, u'health': u'HEALTH_OK', u'kb_used': 9830712, u'store_stats': {u'bytes_total': 8665243, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2373771}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'2 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'2 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 1207/30396 objects degraded (3.971%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 1207, u'num_pgs': 158, u'data_bytes': 42421193060, u'degraded_total': 30396, u'bytes_used': 50691715072, u'op_per_sec': 5, u'write_bytes_sec': 22680759, u'version': 614, u'pgs_by_state': [{u'count': 156, u'state_name': u'active+clean'}, {u'count': 2, u'state_name': u'active+recovering'}], u'degraded_ratio': u'3.971', u'bytes_avail': 2284821147648}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:53:40,660.660 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:53:41,372.372 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:53:41,692.692 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 978 16 8068 8052 32.9276 13.6 5.31094 1.94301 2014-04-01 14:53:42,618.618 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 979 16 8093 8077 32.9961 100 0.547902 1.93844 2014-04-01 14:53:43,623.623 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:53:43.528338min lat: 0.064365 max lat: 110.858 avg lat: 1.93557 2014-04-01 14:53:43,623.623 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:53:43,623.623 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 980 16 8116 8100 33.0563 92 0.514157 1.93557 2014-04-01 14:53:44,621.621 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 981 16 8127 8111 33.0674 44 0.695863 1.93413 2014-04-01 14:53:45,156.156 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:53:45,619.619 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 982 16 8127 8111 33.0337 0 - 1.93413 2014-04-01 14:53:45,914.914 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:53:46,619.619 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 983 16 8127 8111 33.0001 0 - 1.93413 2014-04-01 14:53:46,680.680 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:53:47,456.456 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.082877', u'skew': u'-0.182834', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.082621', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:53:42.668594', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438471716, u'health': u'HEALTH_OK', u'kb_used': 9857268, u'store_stats': {u'bytes_total': 3096379, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572075}}, {u'last_updated': u'2014-04-01 05:53:42.978822', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438071564, u'health': u'HEALTH_OK', u'kb_used': 10257420, u'store_stats': {u'bytes_total': 3620610, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572018}}, {u'last_updated': u'2014-04-01 05:53:43.039127', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438471316, u'health': u'HEALTH_OK', u'kb_used': 9857668, u'store_stats': {u'bytes_total': 3620592, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572000}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 753/30606 objects degraded (2.460%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 753, u'num_pgs': 158, u'data_bytes': 42714794340, u'degraded_total': 30606, u'bytes_used': 50900623360, u'op_per_sec': 15, u'write_bytes_sec': 66966578, u'version': 618, u'pgs_by_state': [{u'count': 157, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.460', u'bytes_avail': 2284612239360}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:53:47,457.457 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:53:47,616.616 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 984 16 8127 8111 32.9666 0 - 1.93413 2014-04-01 14:53:48,147.147 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:53:48,764.764 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 985 16 8127 8111 32.9331 0 - 1.93413 2014-04-01 14:53:49,620.620 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 986 16 8149 8133 32.989 17.6 0.415346 1.93932 2014-04-01 14:53:50,619.619 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 987 16 8167 8151 33.0285 72 0.346604 1.93714 2014-04-01 14:53:51,622.622 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 988 16 8189 8173 33.0841 88 0.860294 1.93381 2014-04-01 14:53:52,066.066 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:53:52,624.624 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 989 16 8207 8191 33.1235 72 0.741905 1.93119 2014-04-01 14:53:53,026.026 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:53:53,623.623 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 990 16 8221 8205 33.1466 56 1.06608 1.92933 2014-04-01 14:53:53,860.860 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:53:54,611.611 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.082877', u'skew': u'-0.182834', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.082621', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:53:42.668594', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438471716, u'health': u'HEALTH_OK', u'kb_used': 9857268, u'store_stats': {u'bytes_total': 3096379, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572075}}, {u'last_updated': u'2014-04-01 05:53:42.978822', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438071564, u'health': u'HEALTH_OK', u'kb_used': 10257420, u'store_stats': {u'bytes_total': 3620610, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572018}}, {u'last_updated': u'2014-04-01 05:53:43.039127', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438471316, u'health': u'HEALTH_OK', u'kb_used': 9857668, u'store_stats': {u'bytes_total': 3620592, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572000}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 755/30750 objects degraded (2.455%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 755, u'num_pgs': 158, u'data_bytes': 42916120932, u'degraded_total': 30750, u'bytes_used': 51207798784, u'op_per_sec': 13, u'write_bytes_sec': 58357859, u'version': 623, u'pgs_by_state': [{u'count': 157, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.455', u'bytes_avail': 2284305063936}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:53:54,612.612 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:53:54,623.623 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 991 16 8223 8207 33.1212 8 0.788217 1.92908 2014-04-01 14:53:55,381.381 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:53:55,621.621 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 992 16 8223 8207 33.0878 0 - 1.92908 2014-04-01 14:53:56,619.619 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 993 16 8223 8207 33.0545 0 - 1.92908 2014-04-01 14:53:57,621.621 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 994 16 8244 8228 33.1057 28 0.521509 1.93212 2014-04-01 14:53:58,629.629 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 995 16 8267 8251 33.1649 92 0.844827 1.92885 2014-04-01 14:53:59,179.179 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:53:59,215.215 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 05:53:59.121681 7fe3a2fd6700 0 -- 10.214.133.37:0/1005437 >> 10.214.133.37:6814/6341 pipe(0x7fe3ac00dff0 sd=14 :57470 s=2 pgs=6 cs=1 l=1 c=0x7fe3ac01f700).injecting socket failure 2014-04-01 14:53:59,628.628 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 996 16 8291 8275 33.228 96 0.596532 1.92538 2014-04-01 14:53:59,963.963 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:54:00,619.619 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 997 16 8302 8286 33.2388 44 0.192137 1.92355 2014-04-01 14:54:00,760.760 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:54:01,536.536 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.082877', u'skew': u'-0.182834', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.082621', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:53:42.668594', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438471716, u'health': u'HEALTH_OK', u'kb_used': 9857268, u'store_stats': {u'bytes_total': 3096379, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572075}}, {u'last_updated': u'2014-04-01 05:53:42.978822', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438071564, u'health': u'HEALTH_OK', u'kb_used': 10257420, u'store_stats': {u'bytes_total': 3620610, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572018}}, {u'last_updated': u'2014-04-01 05:53:43.039127', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438471316, u'health': u'HEALTH_OK', u'kb_used': 9857668, u'store_stats': {u'bytes_total': 3620592, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572000}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 757/31011 objects degraded (2.441%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 757, u'num_pgs': 158, u'data_bytes': 43281025380, u'degraded_total': 31011, u'bytes_used': 51600494592, u'op_per_sec': 9, u'write_bytes_sec': 39518773, u'version': 628, u'pgs_by_state': [{u'count': 157, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.441', u'bytes_avail': 2283912368128}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:54:01,537.537 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:54:01,619.619 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 998 16 8302 8286 33.2055 0 - 1.92355 2014-04-01 14:54:02,294.294 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:54:02,749.749 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 999 16 8302 8286 33.1722 0 - 1.92355 2014-04-01 14:54:03,620.620 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:54:03.531222min lat: 0.064365 max lat: 110.858 avg lat: 1.92355 2014-04-01 14:54:03,620.620 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:54:03,621.621 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1000 16 8302 8286 33.139 0 - 1.92355 2014-04-01 14:54:04,620.620 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1001 16 8302 8286 33.1059 0 - 1.92355 2014-04-01 14:54:05,620.620 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1002 16 8319 8303 33.1407 13.6 0.853493 1.9298 2014-04-01 14:54:06,362.362 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:54:06,620.620 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1003 16 8346 8330 33.2154 108 0.842583 1.92607 2014-04-01 14:54:06,928.928 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:54:07,621.621 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1004 16 8370 8354 33.2779 96 0.544542 1.92243 2014-04-01 14:54:07,879.879 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:54:08,621.621 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1005 16 8383 8367 33.2965 52 0.788787 1.92066 2014-04-01 14:54:08,668.668 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.082877', u'skew': u'-0.182834', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.082621', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:53:42.668594', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438471716, u'health': u'HEALTH_OK', u'kb_used': 9857268, u'store_stats': {u'bytes_total': 3096379, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572075}}, {u'last_updated': u'2014-04-01 05:53:42.978822', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438071564, u'health': u'HEALTH_OK', u'kb_used': 10257420, u'store_stats': {u'bytes_total': 3620610, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572018}}, {u'last_updated': u'2014-04-01 05:53:43.039127', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438471316, u'health': u'HEALTH_OK', u'kb_used': 9857668, u'store_stats': {u'bytes_total': 3620592, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572000}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 757/31188 objects degraded (2.427%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 757, u'num_pgs': 158, u'data_bytes': 43528489316, u'degraded_total': 31188, u'bytes_used': 51934736384, u'op_per_sec': 8, u'write_bytes_sec': 35471515, u'version': 632, u'pgs_by_state': [{u'count': 157, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.427', u'bytes_avail': 2283578126336}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:54:08,668.668 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:54:09,442.442 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:54:09,792.792 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1006 16 8391 8375 33.2952 32 1.26782 1.92025 2014-04-01 14:54:10,621.621 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1007 16 8394 8378 33.2741 12 0.109922 1.91977 2014-04-01 14:54:11,622.622 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1008 16 8394 8378 33.2411 0 - 1.91977 2014-04-01 14:54:12,624.624 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1009 16 8408 8392 33.2636 28 3.12351 1.92254 2014-04-01 14:54:13,330.330 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:54:13,620.620 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1010 16 8421 8405 33.2821 52 0.57265 1.92149 2014-04-01 14:54:13,980.980 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:54:14,763.763 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1011 16 8449 8433 33.36 112 0.610709 1.91761 2014-04-01 14:54:15,109.109 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:54:15,623.623 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1012 16 8465 8449 33.3903 64 0.353553 1.91604 2014-04-01 14:54:15,942.942 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.082877', u'skew': u'-0.182834', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.082621', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:53:42.668594', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438471716, u'health': u'HEALTH_OK', u'kb_used': 9857268, u'store_stats': {u'bytes_total': 3096379, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572075}}, {u'last_updated': u'2014-04-01 05:53:42.978822', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438071564, u'health': u'HEALTH_OK', u'kb_used': 10257420, u'store_stats': {u'bytes_total': 3620610, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572018}}, {u'last_updated': u'2014-04-01 05:53:43.039127', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438471316, u'health': u'HEALTH_OK', u'kb_used': 9857668, u'store_stats': {u'bytes_total': 3620592, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572000}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 765/31461 objects degraded (2.432%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 765, u'num_pgs': 158, u'data_bytes': 43910170980, u'degraded_total': 31461, u'bytes_used': 52286328832, u'op_per_sec': 4, u'write_bytes_sec': 18883910, u'version': 638, u'pgs_by_state': [{u'count': 157, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.432', u'bytes_avail': 2283226533888}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:54:15,942.942 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:54:16,623.623 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1013 16 8468 8452 33.3691 12 0.232962 1.91564 2014-04-01 14:54:16,738.738 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:54:17,622.622 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1014 16 8468 8452 33.3362 0 - 1.91564 2014-04-01 14:54:18,622.622 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1015 16 8468 8452 33.3034 0 - 1.91564 2014-04-01 14:54:19,622.622 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1016 16 8468 8452 33.2706 0 - 1.91564 2014-04-01 14:54:20,496.496 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:54:20,626.626 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1017 16 8490 8474 33.3244 22 0.615532 1.91985 2014-04-01 14:54:21,240.240 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:54:21,624.624 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1018 16 8509 8493 33.3663 76 0.390876 1.91743 2014-04-01 14:54:22,483.483 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:54:22,623.623 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1019 16 8525 8509 33.3964 64 0.747968 1.91533 2014-04-01 14:54:23,343.343 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.082877', u'skew': u'-0.182834', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.082621', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:53:42.668594', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438471716, u'health': u'HEALTH_OK', u'kb_used': 9857268, u'store_stats': {u'bytes_total': 3096379, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572075}}, {u'last_updated': u'2014-04-01 05:53:42.978822', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438071564, u'health': u'HEALTH_OK', u'kb_used': 10257420, u'store_stats': {u'bytes_total': 3620610, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572018}}, {u'last_updated': u'2014-04-01 05:53:43.039127', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438471316, u'health': u'HEALTH_OK', u'kb_used': 9857668, u'store_stats': {u'bytes_total': 3620592, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572000}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 776/31665 objects degraded (2.451%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 776, u'num_pgs': 158, u'data_bytes': 44195383652, u'degraded_total': 31665, u'bytes_used': 52596215808, u'op_per_sec': 8, u'write_bytes_sec': 36827260, u'version': 642, u'pgs_by_state': [{u'count': 157, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.451', u'bytes_avail': 2282916646912}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:54:23,343.343 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:54:23,630.630 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:54:23.534122min lat: 0.064365 max lat: 110.858 avg lat: 1.91249 2014-04-01 14:54:23,630.630 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:54:23,630.630 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1020 16 8547 8531 33.4499 88 0.859152 1.91249 2014-04-01 14:54:24,050.050 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:54:24,622.622 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1021 16 8559 8543 33.4641 48 1.20138 1.91106 2014-04-01 14:54:25,623.623 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1022 16 8560 8544 33.4353 4 0.641594 1.91091 2014-04-01 14:54:26,624.624 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1023 16 8560 8544 33.4026 0 - 1.91091 2014-04-01 14:54:27,623.623 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1024 16 8560 8544 33.37 0 - 1.91091 2014-04-01 14:54:27,816.816 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:54:28,600.600 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:54:28,756.756 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1025 16 8560 8544 33.3375 0 - 1.91091 2014-04-01 14:54:29,375.375 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:54:29,624.624 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1026 16 8576 8560 33.3673 16 5.39938 1.91755 2014-04-01 14:54:30,159.159 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.082877', u'skew': u'-0.182834', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.082621', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:53:42.668594', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438471716, u'health': u'HEALTH_OK', u'kb_used': 9857268, u'store_stats': {u'bytes_total': 3096379, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572075}}, {u'last_updated': u'2014-04-01 05:53:42.978822', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438071564, u'health': u'HEALTH_OK', u'kb_used': 10257420, u'store_stats': {u'bytes_total': 3620610, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572018}}, {u'last_updated': u'2014-04-01 05:53:43.039127', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438471316, u'health': u'HEALTH_OK', u'kb_used': 9857668, u'store_stats': {u'bytes_total': 3620592, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572000}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 780/31947 objects degraded (2.442%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 780, u'num_pgs': 158, u'data_bytes': 44589648228, u'degraded_total': 31947, u'bytes_used': 52973326336, u'op_per_sec': 2, u'write_bytes_sec': 11075122, u'version': 647, u'pgs_by_state': [{u'count': 157, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.442', u'bytes_avail': 2282539536384}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:54:30,160.160 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:54:30,623.623 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1027 16 8599 8583 33.4244 92 0.613918 1.91371 2014-04-01 14:54:30,890.890 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:54:31,626.626 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1028 16 8624 8608 33.4892 100 0.555332 1.9104 2014-04-01 14:54:32,624.624 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1029 16 8633 8617 33.4916 36 1.02405 1.90926 2014-04-01 14:54:33,624.624 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1030 16 8641 8625 33.4901 32 0.197607 1.90815 2014-04-01 14:54:34,624.624 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1031 16 8643 8627 33.4654 8 0.093597 1.90775 2014-04-01 14:54:34,674.674 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:54:35,439.439 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:54:35,901.901 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1032 16 8643 8627 33.433 0 - 1.90775 2014-04-01 14:54:36,366.366 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:54:36,625.625 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1033 16 8643 8627 33.4006 0 - 1.90775 2014-04-01 14:54:37,344.344 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.082877', u'skew': u'-0.182834', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.082621', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:53:42.668594', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438471716, u'health': u'HEALTH_OK', u'kb_used': 9857268, u'store_stats': {u'bytes_total': 3096379, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572075}}, {u'last_updated': u'2014-04-01 05:53:42.978822', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438071564, u'health': u'HEALTH_OK', u'kb_used': 10257420, u'store_stats': {u'bytes_total': 3620610, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572018}}, {u'last_updated': u'2014-04-01 05:53:43.039127', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438471316, u'health': u'HEALTH_OK', u'kb_used': 9857668, u'store_stats': {u'bytes_total': 3620592, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572000}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 773/32169 objects degraded (2.403%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 773, u'num_pgs': 158, u'data_bytes': 44900026724, u'degraded_total': 32169, u'bytes_used': 53364117504, u'op_per_sec': 7, u'write_bytes_sec': 33218277, u'version': 652, u'pgs_by_state': [{u'count': 157, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.403', u'bytes_avail': 2282148745216}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:54:37,345.345 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:54:37,624.624 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1034 16 8643 8627 33.3683 0 - 1.90775 2014-04-01 14:54:38,409.409 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:54:38,723.723 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1035 16 8669 8653 33.4365 26 0.765699 1.91343 2014-04-01 14:54:39,626.626 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1036 16 8690 8674 33.4853 84 0.770311 1.91052 2014-04-01 14:54:40,625.625 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1037 16 8710 8694 33.5302 80 0.842174 1.90797 2014-04-01 14:54:41,625.625 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1038 16 8724 8708 33.5518 56 1.17837 1.9064 2014-04-01 14:54:42,497.497 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:54:42,624.624 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1039 16 8729 8713 33.5388 20 0.599282 1.9057 2014-04-01 14:54:43,048.048 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:54:43,626.626 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:54:43.536979min lat: 0.064365 max lat: 110.858 avg lat: 1.9057 2014-04-01 14:54:43,626.626 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:54:43,626.626 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1040 16 8729 8713 33.5065 0 - 1.9057 2014-04-01 14:54:43,970.970 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:54:45,499.499 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1041 16 8729 8713 33.4743 0 - 1.9057 2014-04-01 14:54:45,505.505 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.082877', u'skew': u'-0.182834', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.082621', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:54:42.668878', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438423388, u'health': u'HEALTH_OK', u'kb_used': 9905596, u'store_stats': {u'bytes_total': 4669243, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572075}}, {u'last_updated': u'2014-04-01 05:54:42.979137', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437989972, u'health': u'HEALTH_OK', u'kb_used': 10339012, u'store_stats': {u'bytes_total': 5717762, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572018}}, {u'last_updated': u'2014-04-01 05:54:43.039390', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438423368, u'health': u'HEALTH_OK', u'kb_used': 9905616, u'store_stats': {u'bytes_total': 5717744, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572000}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 787/32370 objects degraded (2.431%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 787, u'num_pgs': 158, u'data_bytes': 45181045092, u'degraded_total': 32370, u'bytes_used': 53659205632, u'op_per_sec': 12, u'write_bytes_sec': 52281696, u'version': 656, u'pgs_by_state': [{u'count': 157, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+recovering'}], u'degraded_ratio': u'2.431', u'bytes_avail': 2281853657088}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:54:45,506.506 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:54:45,626.626 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1042 16 8729 8713 33.4422 0 - 1.9057 2014-04-01 14:54:46,630.630 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1043 16 8762 8746 33.5367 33 0.579856 1.90747 2014-04-01 14:54:47,001.001 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:54:47,628.628 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1044 16 8783 8767 33.585 84 0.61502 1.90514 2014-04-01 14:54:48,630.630 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1045 16 8800 8784 33.6179 68 1.04528 1.90304 2014-04-01 14:54:49,633.633 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1046 16 8822 8806 33.6698 88 0.806081 1.9001 2014-04-01 14:54:50,631.631 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1047 16 8848 8832 33.737 104 0.211111 1.89643 2014-04-01 14:54:50,771.771 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:54:51,629.629 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1048 16 8864 8848 33.7659 64 0.722384 1.89414 2014-04-01 14:54:51,676.676 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:54:52,609.609 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:54:52,628.628 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1049 16 8864 8848 33.7337 0 - 1.89414 2014-04-01 14:54:53,309.309 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.082877', u'skew': u'-0.182834', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.082621', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:54:42.668878', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438423388, u'health': u'HEALTH_OK', u'kb_used': 9905596, u'store_stats': {u'bytes_total': 4669243, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572075}}, {u'last_updated': u'2014-04-01 05:54:42.979137', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437989972, u'health': u'HEALTH_OK', u'kb_used': 10339012, u'store_stats': {u'bytes_total': 5717762, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572018}}, {u'last_updated': u'2014-04-01 05:54:43.039390', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438423368, u'health': u'HEALTH_OK', u'kb_used': 9905616, u'store_stats': {u'bytes_total': 5717744, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572000}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'num_pgs': 158, u'data_bytes': 45663390052, u'bytes_used': 54086238208, u'op_per_sec': 28, u'version': 661, u'pgs_by_state': [{u'count': 158, u'state_name': u'active+clean'}], u'write_bytes_sec': 118679478, u'bytes_avail': 2281426624512}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 95, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:54:53,310.310 INFO:teuthology.task.thrashosds.ceph_manager:recovered! 2014-04-01 14:54:53,310.310 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:54:53,310.310 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 5 2014-04-01 14:54:53,310.310 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 14:54:53,310.310 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 14:54:53,310.310 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3' 2014-04-01 14:54:53,629.629 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1050 16 8864 8848 33.7016 0 - 1.89414 2014-04-01 14:54:54,630.630 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1051 16 8864 8848 33.6695 0 - 1.89414 2014-04-01 14:54:55,630.630 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1052 16 8864 8848 33.6375 0 - 1.89414 2014-04-01 14:54:56,635.635 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1053 16 8881 8865 33.6701 13.6 0.344614 1.90043 2014-04-01 14:54:57,631.631 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1054 16 8896 8880 33.6951 60 0.849143 1.89856 2014-04-01 14:54:58,631.631 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1055 16 8917 8901 33.7427 84 0.588303 1.8957 2014-04-01 14:54:59,502.502 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5] out_osds: [1] dead_osds: [] live_osds: [0, 3, 2, 4, 1, 5] 2014-04-01 14:54:59,503.503 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:54:59,503.503 INFO:teuthology.task.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.909540 2014-04-01 14:54:59,505.505 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 0 0.909539906098' 2014-04-01 14:54:59,632.632 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1056 16 8942 8926 33.8055 100 0.532718 1.89253 2014-04-01 14:55:00,631.631 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1057 16 8955 8939 33.8227 52 1.42345 1.89086 2014-04-01 14:55:01,631.631 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1058 16 8958 8942 33.8021 12 0.293596 1.89037 2014-04-01 14:55:01,703.703 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set osd.0 primary-affinity to 0.90954 (8596072) 2014-04-01 14:55:02,679.679 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1059 16 8958 8942 33.7701 0 - 1.89037 2014-04-01 14:55:03,701.701 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:55:03.542318min lat: 0.064365 max lat: 110.858 avg lat: 1.89037 2014-04-01 14:55:03,701.701 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:55:03,900.900 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1060 16 8958 8942 33.7383 0 - 1.89037 2014-04-01 14:55:04,715.715 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1061 16 8958 8942 33.7065 0 - 1.89037 2014-04-01 14:55:05,636.636 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1062 16 8959 8943 33.6785 1 4.65512 1.89068 2014-04-01 14:55:06,670.670 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1063 16 8987 8971 33.7522 112 0.343561 1.89579 2014-04-01 14:55:06,716.716 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5] out_osds: [1] dead_osds: [] live_osds: [0, 3, 2, 4, 1, 5] 2014-04-01 14:55:06,717.717 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:55:06,717.717 INFO:teuthology.task.thrashosds.thrasher:Growing pool rbd 2014-04-01 14:55:06,717.717 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:55:07,483.483 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:55:08,006.006 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1064 16 9003 8987 33.7806 64 1.0546 1.89356 2014-04-01 14:55:08,642.642 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1065 16 9026 9010 33.8352 92 0.470785 1.89061 2014-04-01 14:55:08,827.827 INFO:teuthology.task.thrashosds.ceph_manager:increase pool size by 10 2014-04-01 14:55:08,828.828 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set rbd pg_num 44' 2014-04-01 14:55:09,634.634 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1066 16 9049 9033 33.8898 92 0.27191 1.88792 2014-04-01 14:55:10,131.131 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 2 pg_num to 44 2014-04-01 14:55:10,632.632 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1067 16 9069 9053 33.933 80 0.885067 1.885 2014-04-01 14:55:11,634.634 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1068 16 9075 9059 33.9237 24 0.629683 1.88426 2014-04-01 14:55:12,635.635 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1069 16 9075 9059 33.892 0 - 1.88426 2014-04-01 14:55:13,633.633 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1070 16 9075 9059 33.8603 0 - 1.88426 2014-04-01 14:55:14,633.633 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1071 16 9075 9059 33.8287 0 - 1.88426 2014-04-01 14:55:15,140.140 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5] out_osds: [1] dead_osds: [] live_osds: [0, 3, 2, 4, 1, 5] 2014-04-01 14:55:15,141.141 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:55:15,141.141 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool rbd 2014-04-01 14:55:15,141.141 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:55:15,633.633 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1072 16 9102 9086 33.8978 27 0.512747 1.88703 2014-04-01 14:55:15,898.898 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:55:16,634.634 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1073 16 9110 9094 33.8961 32 1.64934 1.8865 2014-04-01 14:55:17,637.637 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1074 16 9133 9117 33.9502 92 0.878827 1.88418 2014-04-01 14:55:18,633.633 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1075 16 9149 9133 33.9781 64 1.10931 1.88245 2014-04-01 14:55:19,634.634 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1076 16 9158 9142 33.98 36 0.697243 1.88142 2014-04-01 14:55:20,634.634 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1077 16 9159 9143 33.9521 4 0.323099 1.88125 2014-04-01 14:55:21,634.634 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1078 16 9159 9143 33.9206 0 - 1.88125 2014-04-01 14:55:21,819.819 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5] out_osds: [1] dead_osds: [] live_osds: [0, 3, 2, 4, 1, 5] 2014-04-01 14:55:21,819.819 INFO:teuthology.task.thrashosds.ceph_manager:waiting for recovery to complete 2014-04-01 14:55:21,819.819 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:55:22,530.530 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:55:22,684.684 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1079 16 9174 9158 33.9448 30 3.63348 1.88479 2014-04-01 14:55:23,317.317 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:55:23,633.633 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:55:23.545456min lat: 0.064365 max lat: 110.858 avg lat: 1.88205 2014-04-01 14:55:23,634.634 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:55:23,634.634 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1080 16 9192 9176 33.98 72 0.450791 1.88205 2014-04-01 14:55:24,060.060 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:55:24,637.637 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1081 16 9219 9203 34.0485 108 0.63668 1.8791 2014-04-01 14:55:24,846.846 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:55:25,640.640 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1082 16 9246 9230 34.1168 108 0.432798 1.8755 2014-04-01 14:55:25,831.831 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 8, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.082877', u'skew': u'-0.182834', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.082621', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:54:42.668878', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438423388, u'health': u'HEALTH_OK', u'kb_used': 9905596, u'store_stats': {u'bytes_total': 4669243, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572075}}, {u'last_updated': u'2014-04-01 05:54:42.979137', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437989972, u'health': u'HEALTH_OK', u'kb_used': 10339012, u'store_stats': {u'bytes_total': 5717762, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572018}}, {u'last_updated': u'2014-04-01 05:54:43.039390', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438423368, u'health': u'HEALTH_OK', u'kb_used': 9905616, u'store_stats': {u'bytes_total': 5717744, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2572000}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 64 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool unique_pool_0 pg_num 26 > pgp_num 16'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'num_pgs': 168, u'data_bytes': 47060093284, u'bytes_used': 57975853056, u'op_per_sec': 11, u'version': 680, u'pgs_by_state': [{u'count': 168, u'state_name': u'active+clean'}], u'write_bytes_sec': 47481282, u'bytes_avail': 2277537009664}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 99, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:55:25,831.831 INFO:teuthology.task.thrashosds.ceph_manager:recovered! 2014-04-01 14:55:25,832.832 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:55:25,832.832 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 3 2014-04-01 14:55:25,832.832 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 14:55:25,832.832 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 14:55:25,833.833 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3' 2014-04-01 14:55:26,638.638 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1083 16 9253 9237 34.1112 28 0.908804 1.87462 2014-04-01 14:55:27,639.639 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1084 16 9275 9259 34.1609 88 0.851241 1.87257 2014-04-01 14:55:28,635.635 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1085 16 9276 9260 34.1331 4 0.338287 1.87241 2014-04-01 14:55:29,637.637 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1086 16 9276 9260 34.1016 0 - 1.87241 2014-04-01 14:55:30,636.636 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1087 16 9276 9260 34.0703 0 - 1.87241 2014-04-01 14:55:31,436.436 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5] out_osds: [1] dead_osds: [] live_osds: [0, 3, 2, 4, 1, 5] 2014-04-01 14:55:31,436.436 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:55:31,436.436 INFO:teuthology.task.thrashosds.thrasher:Adding osd 1 2014-04-01 14:55:31,437.437 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 1' 2014-04-01 14:55:31,635.635 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1088 16 9276 9260 34.039 0 - 1.87241 2014-04-01 14:55:32,635.635 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1089 16 9276 9260 34.0077 0 - 1.87241 2014-04-01 14:55:33,401.401 INFO:teuthology.orchestra.run.err:[10.214.133.37]: marked in osd.1. 2014-04-01 14:55:33,410.410 INFO:teuthology.task.thrashosds.thrasher:Added osd 1 2014-04-01 14:55:33,635.635 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1090 16 9299 9283 34.0609 18.4 0.671859 1.87802 2014-04-01 14:55:34,718.718 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1091 16 9306 9290 34.0553 28 0.467176 1.87714 2014-04-01 14:55:35,637.637 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1092 16 9306 9290 34.0241 0 - 1.87714 2014-04-01 14:55:36,637.637 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1093 16 9306 9290 33.993 0 - 1.87714 2014-04-01 14:55:37,637.637 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1094 16 9306 9290 33.9619 0 - 1.87714 2014-04-01 14:55:38,409.409 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 1] out_osds: [] dead_osds: [] live_osds: [0, 3, 2, 4, 1, 5] 2014-04-01 14:55:38,409.409 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:55:38,409.409 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 3 2014-04-01 14:55:38,410.410 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 14:55:38,410.410 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 14:55:38,410.410 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3' 2014-04-01 14:55:38,636.636 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1095 16 9306 9290 33.9309 0 - 1.87714 2014-04-01 14:55:39,636.636 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1096 16 9306 9290 33.9 0 - 1.87714 2014-04-01 14:55:40,638.638 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1097 16 9306 9290 33.8691 0 - 1.87714 2014-04-01 14:55:41,639.639 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1098 16 9319 9303 33.8856 7.42857 0.317623 1.88196 2014-04-01 14:55:42,639.639 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1099 16 9340 9324 33.9312 84 0.253194 1.87868 2014-04-01 14:55:43,641.641 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:55:43.548286min lat: 0.064365 max lat: 110.858 avg lat: 1.87701 2014-04-01 14:55:43,641.641 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:55:43,642.642 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1100 16 9353 9337 33.9476 52 0.367432 1.87701 2014-04-01 14:55:43,877.877 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 1] out_osds: [] dead_osds: [] live_osds: [0, 3, 2, 4, 1, 5] 2014-04-01 14:55:43,877.877 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:55:43,877.877 INFO:teuthology.task.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 2, 3, 4, 5, 1] 2014-04-01 14:55:43,877.877 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 2' 2014-04-01 14:55:44,638.638 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1101 16 9364 9348 33.9567 44 0.402438 1.87557 2014-04-01 14:55:45,148.148 INFO:teuthology.orchestra.run.err:[10.214.133.37]: marked out osd.2. 2014-04-01 14:55:45,637.637 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1102 16 9367 9351 33.9368 12 0.106154 1.87505 2014-04-01 14:55:46,638.638 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1103 16 9367 9351 33.906 0 - 1.87505 2014-04-01 14:55:47,638.638 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1104 16 9367 9351 33.8753 0 - 1.87505 2014-04-01 14:55:48,640.640 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1105 16 9370 9354 33.8555 4 4.02503 1.8758 2014-04-01 14:55:50,162.162 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 1] out_osds: [2] dead_osds: [] live_osds: [0, 3, 2, 4, 1, 5] 2014-04-01 14:55:50,163.163 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:55:50,163.163 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 4 2014-04-01 14:55:50,164.164 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 14:55:50,164.164 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 14:55:50,164.164 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3' 2014-04-01 14:55:50,479.479 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1106 16 9378 9362 33.8538 32 16.6103 1.87831 2014-04-01 14:55:50,638.638 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1107 16 9385 9369 33.8485 28 0.141261 1.87915 2014-04-01 14:55:51,639.639 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1108 16 9399 9383 33.8685 56 0.596839 1.88031 2014-04-01 14:55:52,639.639 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1109 16 9409 9393 33.874 40 0.133161 1.88075 2014-04-01 14:55:53,639.639 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1110 16 9413 9397 33.8579 16 8.70438 1.8829 2014-04-01 14:55:54,639.639 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1111 16 9414 9398 33.8311 4 0.134037 1.88272 2014-04-01 14:55:55,639.639 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1112 16 9430 9414 33.8582 64 0.466026 1.88344 2014-04-01 14:55:55,778.778 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 1] out_osds: [2] dead_osds: [] live_osds: [0, 3, 2, 4, 1, 5] 2014-04-01 14:55:55,779.779 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:55:55,779.779 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool unique_pool_0 2014-04-01 14:55:55,779.779 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:55:56,501.501 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:55:56,655.655 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1113 16 9440 9424 33.8637 40 0.155089 1.88231 2014-04-01 14:55:57,275.275 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set unique_pool_0 pgp_num 26' 2014-04-01 14:55:57,639.639 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1114 16 9441 9425 33.8369 4 0.79612 1.8822 2014-04-01 14:55:58,390.390 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 3 pgp_num to 26 2014-04-01 14:55:58,639.639 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1115 16 9441 9425 33.8065 0 - 1.8822 2014-04-01 14:55:59,639.639 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1116 16 9441 9425 33.7763 0 - 1.8822 2014-04-01 14:56:00,640.640 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1117 16 9441 9425 33.746 0 - 1.8822 2014-04-01 14:56:01,640.640 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1118 16 9452 9436 33.7552 11 0.402277 1.88521 2014-04-01 14:56:02,641.641 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1119 16 9458 9442 33.7465 24 0.1585 1.88417 2014-04-01 14:56:03,402.402 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 1] out_osds: [2] dead_osds: [] live_osds: [0, 3, 2, 4, 1, 5] 2014-04-01 14:56:03,402.402 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:56:03,403.403 INFO:teuthology.task.thrashosds.thrasher:Adding osd 2 2014-04-01 14:56:03,403.403 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 2' 2014-04-01 14:56:03,640.640 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:56:03.551200min lat: 0.064365 max lat: 110.858 avg lat: 1.88277 2014-04-01 14:56:03,640.640 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:56:03,640.640 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1120 16 9471 9455 33.7627 52 0.147113 1.88277 2014-04-01 14:56:04,609.609 INFO:teuthology.orchestra.run.err:[10.214.133.37]: marked in osd.2. 2014-04-01 14:56:04,615.615 INFO:teuthology.task.thrashosds.thrasher:Added osd 2 2014-04-01 14:56:04,640.640 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1121 16 9471 9455 33.7326 0 - 1.88277 2014-04-01 14:56:05,641.641 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1122 16 9474 9458 33.7133 6 2.20354 1.88305 2014-04-01 14:56:06,641.641 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1123 16 9474 9458 33.6832 0 - 1.88305 2014-04-01 14:56:07,641.641 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1124 16 9475 9459 33.6568 2 5.62127 1.88344 2014-04-01 14:56:08,642.642 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1125 16 9477 9461 33.634 8 3.03868 1.88391 2014-04-01 14:56:09,615.615 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 1, 2] out_osds: [] dead_osds: [] live_osds: [0, 3, 2, 4, 1, 5] 2014-04-01 14:56:09,615.615 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:56:09,616.616 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 3 2014-04-01 14:56:09,616.616 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 14:56:09,616.616 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 14:56:09,617.617 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3' 2014-04-01 14:56:09,641.641 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1126 16 9482 9466 33.6219 20 0.128753 1.88474 2014-04-01 14:56:10,641.641 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1127 16 9494 9478 33.6347 48 0.283773 1.88313 2014-04-01 14:56:11,650.650 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1128 16 9506 9490 33.6474 48 16.263 1.89146 2014-04-01 14:56:12,642.642 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1129 16 9510 9494 33.6318 16 16.5374 1.89425 2014-04-01 14:56:13,646.646 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1130 16 9513 9497 33.6126 12 0.321296 1.89704 2014-04-01 14:56:14,654.654 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1131 16 9538 9522 33.6713 100 0.81383 1.8985 2014-04-01 14:56:15,047.047 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 1, 2] out_osds: [] dead_osds: [] live_osds: [0, 3, 2, 4, 1, 5] 2014-04-01 14:56:15,047.047 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:56:15,047.047 INFO:teuthology.task.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 3, 4, 5, 1, 2] 2014-04-01 14:56:15,047.047 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 5' 2014-04-01 14:56:15,641.641 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1132 16 9547 9531 33.6734 36 1.30454 1.89761 2014-04-01 14:56:16,258.258 INFO:teuthology.orchestra.run.err:[10.214.133.37]: marked out osd.5. 2014-04-01 14:56:16,654.654 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1133 16 9554 9538 33.6683 28 0.645451 1.89896 2014-04-01 14:56:17,647.647 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1134 16 9556 9540 33.6457 8 0.102412 1.89865 2014-04-01 14:56:18,650.650 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1135 16 9557 9541 33.6196 4 1.57438 1.89862 2014-04-01 14:56:19,642.642 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1136 16 9558 9542 33.5935 4 3.44333 1.89878 2014-04-01 14:56:20,643.643 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1137 16 9570 9554 33.6062 48 3.64886 1.90168 2014-04-01 14:56:21,268.268 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 3, 4, 1, 2] out_osds: [5] dead_osds: [] live_osds: [0, 3, 2, 4, 1, 5] 2014-04-01 14:56:21,268.268 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:56:21,268.268 INFO:teuthology.task.thrashosds.thrasher:Adding osd 5 2014-04-01 14:56:21,268.268 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 5' 2014-04-01 14:56:21,642.642 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1138 16 9573 9557 33.5872 12 0.689112 1.90135 2014-04-01 14:56:22,642.642 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1139 16 9573 9557 33.5577 0 - 1.90135 2014-04-01 14:56:22,808.808 INFO:teuthology.orchestra.run.err:[10.214.133.37]: marked in osd.5. 2014-04-01 14:56:22,818.818 INFO:teuthology.task.thrashosds.thrasher:Added osd 5 2014-04-01 14:56:23,642.642 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:56:23.553782min lat: 0.064365 max lat: 110.858 avg lat: 1.90135 2014-04-01 14:56:23,642.642 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:56:23,643.643 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1140 16 9573 9557 33.5283 0 - 1.90135 2014-04-01 14:56:24,643.643 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1141 16 9575 9559 33.5059 2.66667 4.47802 1.90204 2014-04-01 14:56:25,644.644 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1142 16 9592 9576 33.5361 68 0.163749 1.90341 2014-04-01 14:56:26,644.644 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1143 16 9593 9577 33.5102 4 0.5989 1.90327 2014-04-01 14:56:27,644.644 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1144 16 9597 9581 33.4949 16 2.62185 1.90556 2014-04-01 14:56:27,819.819 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 3, 4, 1, 2, 5] out_osds: [] dead_osds: [] live_osds: [0, 3, 2, 4, 1, 5] 2014-04-01 14:56:27,819.819 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:56:27,819.819 INFO:teuthology.task.thrashosds.thrasher:Setting osd 5 primary_affinity to 1.000000 2014-04-01 14:56:27,819.819 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 5 1' 2014-04-01 14:56:28,644.644 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1145 16 9610 9594 33.5111 52 0.377213 1.90555 2014-04-01 14:56:28,937.937 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set osd.5 primary-affinity to 1 (8655362) 2014-04-01 14:56:29,643.643 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1146 16 9610 9594 33.4819 0 - 1.90555 2014-04-01 14:56:30,645.645 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1147 16 9618 9602 33.4806 16 0.471153 1.90808 2014-04-01 14:56:31,649.649 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1148 16 9634 9618 33.5071 64 0.31891 1.9093 2014-04-01 14:56:32,649.649 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1149 16 9652 9636 33.5406 72 0.572022 1.90729 2014-04-01 14:56:33,645.645 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1150 16 9661 9645 33.5428 36 1.69116 1.90668 2014-04-01 14:56:33,947.947 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 3, 4, 1, 2, 5] out_osds: [] dead_osds: [] live_osds: [0, 3, 2, 4, 1, 5] 2014-04-01 14:56:33,947.947 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:56:33,947.947 INFO:teuthology.task.thrashosds.thrasher:Killing osd 2, live_osds are [0, 3, 2, 4, 1, 5] 2014-04-01 14:56:33,947.947 DEBUG:teuthology.task.ceph.osd.2:waiting for process to exit 2014-04-01 14:56:34,126.126 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 05:56:34.037890 7fe3b42f3700 0 -- 10.214.133.37:0/1005437 >> 10.214.133.37:6814/6341 pipe(0x7fe3ac025580 sd=14 :0 s=1 pgs=0 cs=0 l=1 c=0x7fe3ac0102d0).fault 2014-04-01 14:56:34,132.132 INFO:teuthology.task.ceph.osd.2:Stopped 2014-04-01 14:56:34,644.644 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1151 16 9664 9648 33.524 12 1.21311 1.90637 2014-04-01 14:56:35,644.644 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1152 16 9664 9648 33.4949 0 - 1.90637 2014-04-01 14:56:36,646.646 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1153 16 9664 9648 33.4659 0 - 1.90637 2014-04-01 14:56:37,648.648 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1154 16 9670 9654 33.4577 8 4.31883 1.9082 2014-04-01 14:56:38,647.647 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1155 16 9677 9661 33.453 28 0.213442 1.90833 2014-04-01 14:56:39,132.132 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 3, 4, 1, 2, 5] out_osds: [] dead_osds: [2] live_osds: [0, 3, 4, 1, 5] 2014-04-01 14:56:39,132.132 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:56:39,132.132 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 5 2014-04-01 14:56:39,132.132 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 14:56:39,132.132 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 14:56:39,133.133 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3' 2014-04-01 14:56:39,645.645 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1156 16 9678 9662 33.4275 4 0.156498 1.90815 2014-04-01 14:56:40,648.648 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1157 16 9678 9662 33.3986 0 - 1.90815 2014-04-01 14:56:41,646.646 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1158 16 9678 9662 33.3697 0 - 1.90815 2014-04-01 14:56:42,653.653 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1159 16 9678 9662 33.341 0 - 1.90815 2014-04-01 14:56:43,646.646 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:56:43.556684min lat: 0.064365 max lat: 110.858 avg lat: 1.90815 2014-04-01 14:56:43,646.646 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:56:43,646.646 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1160 16 9678 9662 33.3122 0 - 1.90815 2014-04-01 14:56:44,556.556 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 3, 4, 1, 2, 5] out_osds: [] dead_osds: [2] live_osds: [0, 3, 4, 1, 5] 2014-04-01 14:56:44,556.556 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:56:44,557.557 INFO:teuthology.task.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 3, 4, 1, 2, 5] 2014-04-01 14:56:44,557.557 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 3' 2014-04-01 14:56:44,646.646 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1161 16 9678 9662 33.2835 0 - 1.90815 2014-04-01 14:56:45,646.646 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1162 16 9678 9662 33.2549 0 - 1.90815 2014-04-01 14:56:46,027.027 INFO:teuthology.orchestra.run.err:[10.214.133.37]: marked out osd.3. 2014-04-01 14:56:46,646.646 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1163 16 9678 9662 33.2263 0 - 1.90815 2014-04-01 14:56:47,647.647 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1164 16 9678 9662 33.1977 0 - 1.90815 2014-04-01 14:56:48,647.647 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1165 16 9678 9662 33.1692 0 - 1.90815 2014-04-01 14:56:49,647.647 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1166 16 9678 9662 33.1408 0 - 1.90815 2014-04-01 14:56:50,647.647 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1167 16 9678 9662 33.1124 0 - 1.90815 2014-04-01 14:56:51,037.037 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 4, 1, 2, 5] out_osds: [3] dead_osds: [2] live_osds: [0, 3, 4, 1, 5] 2014-04-01 14:56:51,037.037 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:56:51,038.038 INFO:teuthology.task.thrashosds.thrasher:Growing pool data 2014-04-01 14:56:51,038.038 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:56:51,838.838 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:56:51,993.993 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1168 16 9678 9662 33.084 0 - 1.90815 2014-04-01 14:56:52,647.647 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1169 16 9678 9662 33.0557 0 - 1.90815 2014-04-01 14:56:53,063.063 INFO:teuthology.task.thrashosds.ceph_manager:increase pool size by 10 2014-04-01 14:56:53,063.063 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set data pg_num 74' 2014-04-01 14:56:53,647.647 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1170 16 9678 9662 33.0275 0 - 1.90815 2014-04-01 14:56:54,654.654 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1171 16 9678 9662 32.9993 0 - 1.90815 2014-04-01 14:56:54,875.875 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 0 pg_num to 74 2014-04-01 14:56:55,652.652 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1172 16 9678 9662 32.9711 0 - 1.90815 2014-04-01 14:56:56,647.647 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1173 16 9678 9662 32.943 0 - 1.90815 2014-04-01 14:56:57,647.647 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1174 16 9678 9662 32.915 0 - 1.90815 2014-04-01 14:56:58,649.649 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1175 16 9678 9662 32.887 0 - 1.90815 2014-04-01 14:56:59,649.649 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1176 16 9678 9662 32.859 0 - 1.90815 2014-04-01 14:56:59,886.886 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 4, 1, 2, 5] out_osds: [3] dead_osds: [2] live_osds: [0, 3, 4, 1, 5] 2014-04-01 14:56:59,886.886 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:56:59,886.886 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool data 2014-04-01 14:56:59,886.886 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:57:00,798.798 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:57:00,962.962 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1177 16 9678 9662 32.8311 0 - 1.90815 2014-04-01 14:57:01,648.648 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1178 16 9678 9662 32.8032 0 - 1.90815 2014-04-01 14:57:02,649.649 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1179 16 9678 9662 32.7754 0 - 1.90815 2014-04-01 14:57:03,652.652 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:57:03.559485min lat: 0.064365 max lat: 110.858 avg lat: 1.9201 2014-04-01 14:57:03,652.652 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:57:03,652.652 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1180 16 9682 9666 32.7612 0.666667 30.2906 1.9201 2014-04-01 14:57:04,650.650 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1181 16 9692 9676 32.7673 40 0.379836 1.93234 2014-04-01 14:57:05,653.653 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1182 16 9696 9680 32.7531 16 2.13549 1.93216 2014-04-01 14:57:06,650.650 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1183 16 9707 9691 32.7626 44 0.319244 1.93405 2014-04-01 14:57:06,723.723 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 4, 1, 2, 5] out_osds: [3] dead_osds: [2] live_osds: [0, 3, 4, 1, 5] 2014-04-01 14:57:06,723.723 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:57:06,723.723 INFO:teuthology.task.thrashosds.thrasher:Growing pool rbd 2014-04-01 14:57:06,723.723 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:57:07,610.610 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:57:07,764.764 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1184 16 9716 9700 32.7653 36 1.48127 1.93337 2014-04-01 14:57:08,649.649 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1185 16 9729 9713 32.7816 52 0.30357 1.93177 2014-04-01 14:57:09,652.652 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1186 16 9733 9717 32.7674 16 0.222119 1.93142 2014-04-01 14:57:10,651.651 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1187 16 9738 9722 32.7566 20 0.422457 1.93114 2014-04-01 14:57:11,650.650 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1188 16 9740 9724 32.7358 8 2.02144 1.93133 2014-04-01 14:57:12,651.651 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1189 16 9744 9728 32.7217 16 1.66504 1.93169 2014-04-01 14:57:13,528.528 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 4, 1, 2, 5] out_osds: [3] dead_osds: [2] live_osds: [0, 3, 4, 1, 5] 2014-04-01 14:57:13,529.529 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:57:13,529.529 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 0 2014-04-01 14:57:13,529.529 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 14:57:13,530.530 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 14:57:13,530.530 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3' 2014-04-01 14:57:13,650.650 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1190 16 9748 9732 32.7077 16 2.923 1.93241 2014-04-01 14:57:14,650.650 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1191 16 9748 9732 32.6802 0 - 1.93241 2014-04-01 14:57:15,650.650 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1192 16 9762 9746 32.6998 28 2.50323 1.93262 2014-04-01 14:57:16,650.650 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1193 16 9768 9752 32.6925 24 0.124042 1.93173 2014-04-01 14:57:17,651.651 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1194 16 9772 9756 32.6785 16 0.291726 1.93152 2014-04-01 14:57:18,652.652 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1195 16 9772 9756 32.6511 0 - 1.93152 2014-04-01 14:57:19,038.038 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 4, 1, 2, 5] out_osds: [3] dead_osds: [2] live_osds: [0, 3, 4, 1, 5] 2014-04-01 14:57:19,038.038 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:57:19,038.038 INFO:teuthology.task.thrashosds.thrasher:inject_pause on 3 2014-04-01 14:57:19,038.038 INFO:teuthology.task.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2014-04-01 14:57:19,039.039 INFO:teuthology.task.thrashosds.thrasher:Checking after 0, should_be_down=False 2014-04-01 14:57:19,039.039 DEBUG:teuthology.orchestra.run:Running [10.214.133.38]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3' 2014-04-01 14:57:19,652.652 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1196 16 9772 9756 32.6238 0 - 1.93152 2014-04-01 14:57:20,663.663 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1197 16 9786 9770 32.6434 18.6667 0.433815 1.93314 2014-04-01 14:57:21,651.651 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1198 16 9791 9775 32.6328 20 1.22465 1.93249 2014-04-01 14:57:21,907.907 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 05:57:21.813644 7fe3a3de0700 0 -- 10.214.133.37:0/1005437 >> 10.214.133.37:6800/4405 pipe(0x15778b0 sd=10 :53303 s=2 pgs=6 cs=1 l=1 c=0x1577b10).injecting socket failure 2014-04-01 14:57:22,752.752 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1199 16 9798 9782 32.6289 28 1.43546 1.93234 2014-04-01 14:57:23,657.657 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:57:23.562309min lat: 0.064365 max lat: 110.858 avg lat: 1.93231 2014-04-01 14:57:23,658.658 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:57:23,658.658 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1200 16 9804 9788 32.6218 24 0.890737 1.93231 2014-04-01 14:57:24,472.472 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 4, 1, 2, 5] out_osds: [3] dead_osds: [2] live_osds: [0, 3, 4, 1, 5] 2014-04-01 14:57:24,472.472 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:57:24,473.473 INFO:teuthology.task.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.298921 2014-04-01 14:57:24,473.473 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 4 0.298921480635' 2014-04-01 14:57:24,705.705 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1201 16 9811 9795 32.6179 28 1.12969 1.93193 2014-04-01 14:57:25,724.724 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1202 16 9815 9799 32.6041 16 2.40939 1.93201 2014-04-01 14:57:26,039.039 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set osd.4 primary-affinity to 0.298921 (8195902) 2014-04-01 14:57:26,745.745 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1203 16 9822 9806 32.6002 28 1.44153 1.93165 2014-04-01 14:57:27,659.659 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1204 16 9824 9808 32.5798 8 1.691 1.93151 2014-04-01 14:57:28,653.653 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1205 16 9826 9810 32.5594 8 2.0935 1.93153 2014-04-01 14:57:29,653.653 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1206 16 9826 9810 32.5324 0 - 1.93153 2014-04-01 14:57:30,653.653 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1207 16 9834 9818 32.532 16 0.328865 1.93279 2014-04-01 14:57:31,039.039 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 4, 1, 2, 5] out_osds: [3] dead_osds: [2] live_osds: [0, 3, 4, 1, 5] 2014-04-01 14:57:31,039.039 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:57:31,039.039 INFO:teuthology.task.thrashosds.thrasher:Setting osd 0 primary_affinity to 1.000000 2014-04-01 14:57:31,039.039 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 0 1' 2014-04-01 14:57:31,653.653 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1208 16 9841 9825 32.5282 28 1.4728 1.93314 2014-04-01 14:57:32,058.058 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set osd.0 primary-affinity to 1 (8655362) 2014-04-01 14:57:32,652.652 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1209 16 9846 9830 32.5179 20 1.81191 1.93291 2014-04-01 14:57:33,652.652 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1210 16 9847 9831 32.4943 4 1.12513 1.93282 2014-04-01 14:57:34,731.731 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1211 16 9856 9840 32.4972 36 2.02711 1.93302 2014-04-01 14:57:35,653.653 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1212 16 9864 9848 32.4968 32 1.07133 1.93299 2014-04-01 14:57:36,654.654 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1213 16 9867 9851 32.4799 12 1.01505 1.93282 2014-04-01 14:57:37,071.071 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 4, 1, 2, 5] out_osds: [3] dead_osds: [2] live_osds: [0, 3, 4, 1, 5] 2014-04-01 14:57:37,071.071 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:57:37,071.071 INFO:teuthology.task.thrashosds.thrasher:fixing pg num pool unique_pool_0 2014-04-01 14:57:37,071.071 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:57:37,631.631 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:57:38,012.012 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1214 16 9871 9855 32.4663 16 2.13509 1.93278 2014-04-01 14:57:38,729.729 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1215 16 9884 9868 32.4824 52 2.81468 1.93174 2014-04-01 14:57:39,041.041 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool set unique_pool_0 pgp_num 26' 2014-04-01 14:57:39,655.655 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1216 16 9888 9872 32.4688 16 0.447082 1.93106 2014-04-01 14:57:40,669.669 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1217 16 9889 9873 32.4454 4 4.9928 1.93137 2014-04-01 14:57:40,691.691 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set pool 3 pgp_num to 26 2014-04-01 14:57:41,694.694 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1218 16 9890 9874 32.4221 4 3.8513 1.93156 2014-04-01 14:57:42,719.719 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1219 16 9891 9875 32.3987 4 3.40905 1.93171 2014-04-01 14:57:43,754.754 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:57:43.564793min lat: 0.064365 max lat: 110.858 avg lat: 1.932 2014-04-01 14:57:43,754.754 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:57:43,755.755 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1220 16 9893 9877 32.3787 8 4.88722 1.932 2014-04-01 14:57:44,664.664 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1221 16 9898 9882 32.3686 20 1.33789 1.93315 2014-04-01 14:57:45,655.655 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1222 16 9902 9886 32.3552 16 0.958304 1.93289 2014-04-01 14:57:45,701.701 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 4, 1, 2, 5] out_osds: [3] dead_osds: [2] live_osds: [0, 3, 4, 1, 5] 2014-04-01 14:57:45,701.701 INFO:teuthology.task.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-04-01 14:57:45,702.702 INFO:teuthology.task.thrashosds.thrasher:Setting osd 0 primary_affinity to 1.000000 2014-04-01 14:57:45,702.702 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd primary-affinity 0 1' 2014-04-01 14:57:46,471.471 INFO:teuthology.orchestra.run.err:[10.214.133.37]: set osd.0 primary-affinity to 1 (8655362) 2014-04-01 14:57:46,654.654 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1223 16 9908 9892 32.3484 24 0.313816 1.93397 2014-04-01 14:57:47,654.654 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1224 16 9909 9893 32.3252 4 8.17848 1.9346 2014-04-01 14:57:48,654.654 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1225 16 9911 9895 32.3054 8 0.38401 1.93431 2014-04-01 14:57:49,656.656 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1226 16 9915 9899 32.2921 16 4.16596 1.93496 2014-04-01 14:57:50,656.656 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1227 16 9916 9900 32.269 4 5.37741 1.93531 2014-04-01 14:57:51,471.471 INFO:teuthology.task.thrashosds.thrasher:in_osds: [0, 4, 1, 2, 5] out_osds: [3] dead_osds: [2] live_osds: [0, 3, 4, 1, 5] 2014-04-01 14:57:51,471.471 INFO:teuthology.task.thrashosds.thrasher:Reviving osd 2 2014-04-01 14:57:51,471.471 INFO:teuthology.task.ceph.osd.2:Restarting 2014-04-01 14:57:51,472.472 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 2' 2014-04-01 14:57:51,655.655 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1228 16 9916 9900 32.2427 0 - 1.93531 2014-04-01 14:57:51,789.789 INFO:teuthology.task.ceph.osd.2:Started 2014-04-01 14:57:51,789.789 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight' 2014-04-01 14:57:52,020.020 INFO:teuthology.task.ceph.osd.2.out:[10.214.133.37]: starting osd.2 at :/0 osd_data /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal 2014-04-01 14:57:52,201.201 INFO:teuthology.orchestra.run.err:[10.214.133.37]: no valid command found; 10 closest matches: 2014-04-01 14:57:52,201.201 INFO:teuthology.orchestra.run.err:[10.214.133.37]: config show 2014-04-01 14:57:52,201.201 INFO:teuthology.orchestra.run.err:[10.214.133.37]: help 2014-04-01 14:57:52,201.201 INFO:teuthology.orchestra.run.err:[10.214.133.37]: log dump 2014-04-01 14:57:52,201.201 INFO:teuthology.orchestra.run.err:[10.214.133.37]: get_command_descriptions 2014-04-01 14:57:52,202.202 INFO:teuthology.orchestra.run.err:[10.214.133.37]: git_version 2014-04-01 14:57:52,208.208 INFO:teuthology.orchestra.run.err:[10.214.133.37]: config set [...] 2014-04-01 14:57:52,208.208 INFO:teuthology.orchestra.run.err:[10.214.133.37]: version 2014-04-01 14:57:52,209.209 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2 2014-04-01 14:57:52,209.209 INFO:teuthology.orchestra.run.err:[10.214.133.37]: config get 2014-04-01 14:57:52,209.209 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 0 2014-04-01 14:57:52,209.209 INFO:teuthology.orchestra.run.err:[10.214.133.37]: admin_socket: invalid command 2014-04-01 14:57:52,209.209 INFO:teuthology.task.thrashosds.ceph_manager:waiting on admin_socket for 2, ['dump_ops_in_flight'] 2014-04-01 14:57:52,486.486 INFO:teuthology.task.ceph.osd.2.err:[10.214.133.37]: 2014-04-01 05:57:52.396474 7f0342203780 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-04-01 14:57:52,656.656 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1229 16 9916 9900 32.2165 0 - 1.93531 2014-04-01 14:57:53,656.656 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1230 16 9916 9900 32.1903 0 - 1.93531 2014-04-01 14:57:54,655.655 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1231 16 9917 9901 32.1674 1 8.88918 1.93601 2014-04-01 14:57:55,657.657 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1232 16 9918 9902 32.1445 4 0.352958 1.93585 2014-04-01 14:57:56,656.656 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1233 16 9918 9902 32.1185 0 - 1.93585 2014-04-01 14:57:57,208.208 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight' 2014-04-01 14:57:57,597.597 INFO:teuthology.task.thrashosds.ceph_manager:waiting for recovery to complete 2014-04-01 14:57:57,597.597 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:57:57,656.656 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1234 16 9918 9902 32.0924 0 - 1.93585 2014-04-01 14:57:58,656.656 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1235 16 9921 9905 32.0762 4 0.319117 1.93784 2014-04-01 14:57:59,656.656 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1236 16 9926 9910 32.0664 20 10.0201 1.9415 2014-04-01 14:58:00,659.659 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1237 16 9941 9925 32.089 60 1.8041 1.9432 2014-04-01 14:58:01,333.333 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:58:01,795.795 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1238 16 9952 9936 32.0986 44 0.918826 1.94211 2014-04-01 14:58:02,433.433 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:58:02,656.656 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1239 16 9971 9955 32.134 76 88.0694 1.94815 2014-04-01 14:58:03,287.287 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:58:03,656.656 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:58:03.567534min lat: 0.064365 max lat: 110.858 avg lat: 1.95555 2014-04-01 14:58:03,656.656 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:58:03,656.656 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1240 16 9980 9964 32.1371 36 0.54803 1.95555 2014-04-01 14:58:04,217.217 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:58:04,656.656 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1241 16 9983 9967 32.1209 12 91.6777 1.96437 2014-04-01 14:58:04,992.992 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:57:42.669772', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438219956, u'health': u'HEALTH_OK', u'kb_used': 10109028, u'store_stats': {u'bytes_total': 3623549, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099245}}, {u'last_updated': u'2014-04-01 05:57:42.980016', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437728840, u'health': u'HEALTH_OK', u'kb_used': 10600144, u'store_stats': {u'bytes_total': 4147910, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099318}}, {u'last_updated': u'2014-04-01 05:57:43.040105', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438219576, u'health': u'HEALTH_OK', u'kb_used': 10109408, u'store_stats': {u'bytes_total': 4147832, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099240}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs peering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'5 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 17748/36147 objects degraded (49.100%); 836/12058 unfound (6.933%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 7425728, u'degraded_objects': 17748, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 50461673828, u'degraded_total': 36147, u'bytes_used': 51059470336, u'unfound_ratio': u'6.933', u'recovering_objects_per_sec': 1, u'write_bytes_sec': 56930586, u'op_per_sec': 13, u'unfound_objects': 836, u'version': 769, u'pgs_by_state': [{u'count': 1, u'state_name': u'peering'}, {u'count': 12, u'state_name': u'active+recovery_wait'}, {u'count': 152, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'49.100', u'bytes_avail': 2284453392384, u'unfound_total': 12058}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:58:04,992.992 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:58:05,657.657 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1242 16 9983 9967 32.095 0 - 1.96437 2014-04-01 14:58:05,841.841 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:58:06,657.657 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1243 16 9983 9967 32.0692 0 - 1.96437 2014-04-01 14:58:07,658.658 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1244 16 9992 9976 32.0724 12 4.13053 1.96684 2014-04-01 14:58:08,657.657 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1245 16 10020 10004 32.1366 112 0.346478 1.96401 2014-04-01 14:58:09,668.668 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1246 16 10027 10011 32.1332 28 1.04681 1.96318 2014-04-01 14:58:09,758.758 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:58:10,688.688 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:58:10,856.856 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1247 16 10035 10019 32.1331 32 1.69534 1.98084 2014-04-01 14:58:11,657.657 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1248 16 10037 10021 32.1138 8 2.0024 1.98092 2014-04-01 14:58:11,734.734 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:58:12,497.497 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:57:42.669772', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438219956, u'health': u'HEALTH_OK', u'kb_used': 10109028, u'store_stats': {u'bytes_total': 3623549, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099245}}, {u'last_updated': u'2014-04-01 05:57:42.980016', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437728840, u'health': u'HEALTH_OK', u'kb_used': 10600144, u'store_stats': {u'bytes_total': 4147910, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099318}}, {u'last_updated': u'2014-04-01 05:57:43.040105', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438219576, u'health': u'HEALTH_OK', u'kb_used': 10109408, u'store_stats': {u'bytes_total': 4147832, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099240}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'3 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 18328/36288 objects degraded (50.507%); 836/12105 unfound (6.906%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 18328, u'num_pgs': 178, u'data_bytes': 50658806116, u'degraded_total': 36288, u'bytes_used': 51368595456, u'unfound_ratio': u'6.906', u'op_per_sec': 6, u'write_bytes_sec': 27387986, u'unfound_objects': 836, u'version': 773, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 152, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'50.507', u'bytes_avail': 2284144267264, u'unfound_total': 12105}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:58:12,498.498 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:58:12,659.659 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1249 16 10037 10021 32.0881 0 - 1.98092 2014-04-01 14:58:13,191.191 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:58:13,662.662 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1250 16 10037 10021 32.0624 0 - 1.98092 2014-04-01 14:58:14,658.658 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1251 16 10037 10021 32.0368 0 - 1.98092 2014-04-01 14:58:15,663.663 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1252 16 10061 10045 32.0878 24 0.265975 1.98453 2014-04-01 14:58:16,716.716 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1253 16 10080 10064 32.1229 76 0.896286 1.98207 2014-04-01 14:58:17,128.128 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:58:17,740.740 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1254 16 10103 10087 32.1706 92 0.545019 1.97895 2014-04-01 14:58:17,743.743 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:58:18,978.978 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1255 16 10107 10091 32.1577 16 0.679253 1.97841 2014-04-01 14:58:19,395.395 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:58:19,658.658 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1256 16 10108 10092 32.1353 4 1.19732 1.97833 2014-04-01 14:58:20,448.448 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:57:42.669772', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438219956, u'health': u'HEALTH_OK', u'kb_used': 10109028, u'store_stats': {u'bytes_total': 3623549, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099245}}, {u'last_updated': u'2014-04-01 05:57:42.980016', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437728840, u'health': u'HEALTH_OK', u'kb_used': 10600144, u'store_stats': {u'bytes_total': 4147910, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099318}}, {u'last_updated': u'2014-04-01 05:57:43.040105', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438219576, u'health': u'HEALTH_OK', u'kb_used': 10109408, u'store_stats': {u'bytes_total': 4147832, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099240}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 18423/36546 objects degraded (50.410%); 276/12191 unfound (2.264%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 11049319, u'degraded_objects': 18423, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 51019516260, u'degraded_total': 36546, u'bytes_used': 51838214144, u'unfound_ratio': u'2.264', u'recovering_objects_per_sec': 2, u'write_bytes_sec': 62981121, u'op_per_sec': 15, u'unfound_objects': 276, u'version': 778, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 152, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'50.410', u'bytes_avail': 2283674648576, u'unfound_total': 12191}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:58:20,449.449 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:58:20,658.658 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1257 16 10108 10092 32.1098 0 - 1.97833 2014-04-01 14:58:21,218.218 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:58:21,660.660 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1258 16 10108 10092 32.0842 0 - 1.97833 2014-04-01 14:58:22,663.663 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1259 16 10123 10107 32.1064 20 3.45709 1.98272 2014-04-01 14:58:23,658.658 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:58:23.570244min lat: 0.064365 max lat: 110.858 avg lat: 1.98192 2014-04-01 14:58:23,658.658 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:58:23,658.658 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1260 16 10128 10112 32.0968 20 0.146012 1.98192 2014-04-01 14:58:24,660.660 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1261 16 10128 10112 32.0713 0 - 1.98192 2014-04-01 14:58:25,190.190 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:58:25,663.663 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1262 16 10160 10144 32.1473 64 0.662765 1.97991 2014-04-01 14:58:26,046.046 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:58:26,662.662 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1263 16 10166 10150 32.1409 24 0.845546 1.97923 2014-04-01 14:58:26,979.979 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:58:27,659.659 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1264 16 10168 10152 32.1218 8 1.3194 1.97909 2014-04-01 14:58:27,695.695 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:57:42.669772', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438219956, u'health': u'HEALTH_OK', u'kb_used': 10109028, u'store_stats': {u'bytes_total': 3623549, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099245}}, {u'last_updated': u'2014-04-01 05:57:42.980016', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437728840, u'health': u'HEALTH_OK', u'kb_used': 10600144, u'store_stats': {u'bytes_total': 4147910, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099318}}, {u'last_updated': u'2014-04-01 05:57:43.040105', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438219576, u'health': u'HEALTH_OK', u'kb_used': 10109408, u'store_stats': {u'bytes_total': 4147832, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099240}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 18458/36645 objects degraded (50.370%); 276/12224 unfound (2.258%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 18470340, u'degraded_objects': 18458, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 51157928292, u'degraded_total': 36645, u'bytes_used': 52141805568, u'unfound_ratio': u'2.258', u'recovering_objects_per_sec': 4, u'write_bytes_sec': 33246612, u'op_per_sec': 7, u'unfound_objects': 276, u'version': 782, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 152, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'50.370', u'bytes_avail': 2283371057152, u'unfound_total': 12224}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:58:27,695.695 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:58:28,404.404 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:58:28,659.659 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1265 16 10168 10152 32.0964 0 - 1.97909 2014-04-01 14:58:29,662.662 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1266 16 10178 10162 32.1026 20 3.68558 1.98098 2014-04-01 14:58:30,658.658 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1267 16 10197 10181 32.1373 76 0.438792 1.97954 2014-04-01 14:58:31,662.662 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1268 16 10198 10182 32.1151 4 0.938145 1.97944 2014-04-01 14:58:32,321.321 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:58:32,666.666 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1269 16 10214 10198 32.1402 64 1.53338 1.97985 2014-04-01 14:58:33,145.145 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:58:33,668.668 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1270 16 10233 10217 32.1747 76 0.254956 1.97698 2014-04-01 14:58:34,177.177 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:58:34,660.660 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1271 16 10234 10218 32.1525 4 0.930225 1.97688 2014-04-01 14:58:34,904.904 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:57:42.669772', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438219956, u'health': u'HEALTH_OK', u'kb_used': 10109028, u'store_stats': {u'bytes_total': 3623549, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099245}}, {u'last_updated': u'2014-04-01 05:57:42.980016', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437728840, u'health': u'HEALTH_OK', u'kb_used': 10600144, u'store_stats': {u'bytes_total': 4147910, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099318}}, {u'last_updated': u'2014-04-01 05:57:43.040105', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438219576, u'health': u'HEALTH_OK', u'kb_used': 10109408, u'store_stats': {u'bytes_total': 4147832, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099240}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 18569/36918 objects degraded (50.298%); 276/12315 unfound (2.241%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 7873078, u'degraded_objects': 18569, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 51539609956, u'degraded_total': 36918, u'bytes_used': 52504641536, u'unfound_ratio': u'2.241', u'recovering_objects_per_sec': 1, u'write_bytes_sec': 56236277, u'op_per_sec': 13, u'unfound_objects': 276, u'version': 787, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 152, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'50.298', u'bytes_avail': 2283008221184, u'unfound_total': 12315}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:58:34,905.905 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:58:35,661.661 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1272 16 10234 10218 32.1273 0 - 1.97688 2014-04-01 14:58:35,809.809 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:58:36,666.666 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1273 16 10234 10218 32.102 0 - 1.97688 2014-04-01 14:58:37,659.659 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1274 16 10263 10247 32.1679 38.6667 0.345329 1.97735 2014-04-01 14:58:38,659.659 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1275 16 10263 10247 32.1426 0 - 1.97735 2014-04-01 14:58:39,662.662 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1276 16 10264 10248 32.1206 2 1.29761 1.97728 2014-04-01 14:58:39,791.791 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:58:40,643.643 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:58:40,808.808 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1277 16 10287 10271 32.1675 92 0.579701 1.97713 2014-04-01 14:58:41,607.607 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:58:41,660.660 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1278 16 10301 10285 32.1861 56 0.45529 1.9753 2014-04-01 14:58:42,330.330 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:57:42.669772', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438219956, u'health': u'HEALTH_OK', u'kb_used': 10109028, u'store_stats': {u'bytes_total': 3623549, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099245}}, {u'last_updated': u'2014-04-01 05:57:42.980016', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437728840, u'health': u'HEALTH_OK', u'kb_used': 10600144, u'store_stats': {u'bytes_total': 4147910, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099318}}, {u'last_updated': u'2014-04-01 05:57:43.040105', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438219576, u'health': u'HEALTH_OK', u'kb_used': 10109408, u'store_stats': {u'bytes_total': 4147832, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099240}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 18625/37053 objects degraded (50.266%); 276/12360 unfound (2.233%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 16555018, u'degraded_objects': 18625, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 51728353636, u'degraded_total': 37053, u'bytes_used': 52936093696, u'unfound_ratio': u'2.233', u'recovering_objects_per_sec': 3, u'write_bytes_sec': 66220072, u'op_per_sec': 15, u'unfound_objects': 276, u'version': 791, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 152, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'50.266', u'bytes_avail': 2282576769024, u'unfound_total': 12360}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:58:42,331.331 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:58:42,661.661 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1279 16 10323 10307 32.2297 88 0.924416 1.97326 2014-04-01 14:58:43,152.152 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:58:43,660.660 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:58:43.573321min lat: 0.064365 max lat: 110.858 avg lat: 1.97283 2014-04-01 14:58:43,660.660 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:58:43,661.661 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1280 16 10326 10310 32.2139 12 0.484312 1.97283 2014-04-01 14:58:44,660.660 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1281 16 10328 10312 32.195 8 1.43661 1.97271 2014-04-01 14:58:45,660.660 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1282 16 10328 10312 32.1699 0 - 1.97271 2014-04-01 14:58:46,660.660 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1283 16 10328 10312 32.1448 0 - 1.97271 2014-04-01 14:58:47,080.080 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:58:47,663.663 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1284 16 10350 10334 32.1883 29.3333 0.593816 1.97498 2014-04-01 14:58:47,858.858 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:58:48,660.660 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1285 16 10358 10342 32.1882 32 0.866243 1.97436 2014-04-01 14:58:48,795.795 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:58:49,517.517 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:58:42.670009', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438110444, u'health': u'HEALTH_OK', u'kb_used': 10218540, u'store_stats': {u'bytes_total': 6244989, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099245}}, {u'last_updated': u'2014-04-01 05:58:42.980258', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437642848, u'health': u'HEALTH_OK', u'kb_used': 10686136, u'store_stats': {u'bytes_total': 7293638, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099318}}, {u'last_updated': u'2014-04-01 05:58:43.040361', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438109956, u'health': u'HEALTH_OK', u'kb_used': 10219028, u'store_stats': {u'bytes_total': 7293560, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099240}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 18722/37248 objects degraded (50.263%); 276/12425 unfound (2.221%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 18722, u'num_pgs': 178, u'data_bytes': 52000983396, u'degraded_total': 37248, u'bytes_used': 53259755520, u'unfound_ratio': u'2.221', u'op_per_sec': 3, u'write_bytes_sec': 12935617, u'unfound_objects': 276, u'version': 795, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 152, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'50.263', u'bytes_avail': 2282253107200, u'unfound_total': 12425}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:58:49,518.518 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:58:49,660.660 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1286 16 10358 10342 32.1631 0 - 1.97436 2014-04-01 14:58:50,257.257 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:58:50,660.660 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1287 16 10363 10347 32.1537 10 3.01353 1.97489 2014-04-01 14:58:51,664.664 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1288 16 10383 10367 32.1908 80 0.643125 1.97452 2014-04-01 14:58:52,660.660 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1289 16 10384 10368 32.169 4 0.401237 1.97437 2014-04-01 14:58:53,661.661 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1290 16 10385 10369 32.1471 4 1.04254 1.97428 2014-04-01 14:58:54,188.188 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:58:54,663.663 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1291 16 10403 10387 32.178 72 0.516191 1.97543 2014-04-01 14:58:54,923.923 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:58:55,661.661 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1292 16 10422 10406 32.2119 76 0.383886 1.97261 2014-04-01 14:58:56,030.030 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:58:56,663.663 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1293 16 10427 10411 32.2024 20 0.229967 1.97189 2014-04-01 14:58:56,857.857 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:58:42.670009', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438110444, u'health': u'HEALTH_OK', u'kb_used': 10218540, u'store_stats': {u'bytes_total': 6244989, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099245}}, {u'last_updated': u'2014-04-01 05:58:42.980258', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437642848, u'health': u'HEALTH_OK', u'kb_used': 10686136, u'store_stats': {u'bytes_total': 7293638, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099318}}, {u'last_updated': u'2014-04-01 05:58:43.040361', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438109956, u'health': u'HEALTH_OK', u'kb_used': 10219028, u'store_stats': {u'bytes_total': 7293560, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099240}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 18787/37440 objects degraded (50.179%); 276/12489 unfound (2.210%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 11856905, u'degraded_objects': 18787, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 52269418852, u'degraded_total': 37440, u'bytes_used': 53651832832, u'unfound_ratio': u'2.210', u'recovering_objects_per_sec': 2, u'write_bytes_sec': 67753744, u'op_per_sec': 16, u'unfound_objects': 276, u'version': 800, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 152, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'50.179', u'bytes_avail': 2281861029888, u'unfound_total': 12489}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:58:56,858.858 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:58:57,654.654 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:58:57,663.663 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1294 16 10427 10411 32.1776 0 - 1.97189 2014-04-01 14:58:58,661.661 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1295 16 10443 10427 32.2021 32 1.63052 1.97329 2014-04-01 14:58:59,662.662 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1296 16 10444 10428 32.1804 4 0.947858 1.97319 2014-04-01 14:59:00,663.663 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1297 16 10444 10428 32.1556 0 - 1.97319 2014-04-01 14:59:01,575.575 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:59:01,661.661 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1298 16 10450 10434 32.1493 12 3.40832 1.97405 2014-04-01 14:59:02,448.448 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:59:02,662.662 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1299 16 10476 10460 32.2046 104 0.649583 1.97295 2014-04-01 14:59:03,421.421 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:59:03,662.662 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:59:03.576270min lat: 0.064365 max lat: 110.858 avg lat: 1.97137 2014-04-01 14:59:03,662.662 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:59:03,663.663 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1300 16 10492 10476 32.229 64 1.06983 1.97137 2014-04-01 14:59:04,219.219 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:58:42.670009', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438110444, u'health': u'HEALTH_OK', u'kb_used': 10218540, u'store_stats': {u'bytes_total': 6244989, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099245}}, {u'last_updated': u'2014-04-01 05:58:42.980258', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437642848, u'health': u'HEALTH_OK', u'kb_used': 10686136, u'store_stats': {u'bytes_total': 7293638, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099318}}, {u'last_updated': u'2014-04-01 05:58:43.040361', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438109956, u'health': u'HEALTH_OK', u'kb_used': 10219028, u'store_stats': {u'bytes_total': 7293560, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099240}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 18846/37575 objects degraded (50.156%); 276/12534 unfound (2.202%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 18846, u'num_pgs': 178, u'data_bytes': 52458162532, u'degraded_total': 37575, u'bytes_used': 54012092416, u'unfound_ratio': u'2.202', u'op_per_sec': 2, u'write_bytes_sec': 10764592, u'unfound_objects': 276, u'version': 804, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 152, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'50.156', u'bytes_avail': 2281500770304, u'unfound_total': 12534}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:59:04,221.221 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:59:04,662.662 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1301 16 10494 10478 32.2104 8 0.649425 1.97109 2014-04-01 14:59:04,963.963 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:59:05,727.727 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1302 16 10494 10478 32.1857 0 - 1.97109 2014-04-01 14:59:06,662.662 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1303 16 10494 10478 32.161 0 - 1.97109 2014-04-01 14:59:07,662.662 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1304 16 10494 10478 32.1363 0 - 1.97109 2014-04-01 14:59:08,662.662 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1305 16 10502 10486 32.1362 8 5.34149 1.9737 2014-04-01 14:59:08,914.914 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:59:09,504.504 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:59:09,817.817 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1306 16 10525 10509 32.182 92 0.528067 1.97362 2014-04-01 14:59:10,663.663 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1307 16 10542 10526 32.2094 68 0.585192 1.97196 2014-04-01 14:59:10,779.779 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:59:11,504.504 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:58:42.670009', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438110444, u'health': u'HEALTH_OK', u'kb_used': 10218540, u'store_stats': {u'bytes_total': 6244989, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099245}}, {u'last_updated': u'2014-04-01 05:58:42.980258', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437642848, u'health': u'HEALTH_OK', u'kb_used': 10686136, u'store_stats': {u'bytes_total': 7293638, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099318}}, {u'last_updated': u'2014-04-01 05:58:43.040361', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438109956, u'health': u'HEALTH_OK', u'kb_used': 10219028, u'store_stats': {u'bytes_total': 7293560, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099240}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 18925/37776 objects degraded (50.098%); 276/12601 unfound (2.190%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 5505332, u'degraded_objects': 18925, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 52739180900, u'degraded_total': 37776, u'bytes_used': 54341812224, u'unfound_ratio': u'2.190', u'recovering_objects_per_sec': 1, u'write_bytes_sec': 22021331, u'op_per_sec': 5, u'unfound_objects': 276, u'version': 809, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 152, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'50.098', u'bytes_avail': 2281171050496, u'unfound_total': 12601}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:59:11,506.506 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:59:11,661.661 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1308 16 10543 10527 32.1878 4 1.12494 1.97188 2014-04-01 14:59:12,238.238 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:59:12,663.663 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1309 16 10547 10531 32.1755 16 0.272545 1.97167 2014-04-01 14:59:13,663.663 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1310 16 10547 10531 32.1509 0 - 1.97167 2014-04-01 14:59:14,666.666 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1311 16 10547 10531 32.1264 0 - 1.97167 2014-04-01 14:59:15,664.664 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1312 16 10547 10531 32.1019 0 - 1.97167 2014-04-01 14:59:16,187.187 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:59:16,664.664 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1313 16 10579 10563 32.1749 32 0.504601 1.97386 2014-04-01 14:59:16,858.858 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:59:17,663.663 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1314 16 10596 10580 32.2022 68 0.846791 1.97157 2014-04-01 14:59:17,967.967 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:59:18,664.664 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1315 16 10611 10595 32.2233 60 1.10962 1.97012 2014-04-01 14:59:18,811.811 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:58:42.670009', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438110444, u'health': u'HEALTH_OK', u'kb_used': 10218540, u'store_stats': {u'bytes_total': 6244989, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099245}}, {u'last_updated': u'2014-04-01 05:58:42.980258', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437642848, u'health': u'HEALTH_OK', u'kb_used': 10686136, u'store_stats': {u'bytes_total': 7293638, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099318}}, {u'last_updated': u'2014-04-01 05:58:43.040361', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438109956, u'health': u'HEALTH_OK', u'kb_used': 10219028, u'store_stats': {u'bytes_total': 7293560, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099240}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 18985/37914 objects degraded (50.074%); 276/12647 unfound (2.182%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 18985, u'num_pgs': 178, u'data_bytes': 52932118884, u'degraded_total': 37914, u'bytes_used': 54705250304, u'unfound_ratio': u'2.182', u'op_per_sec': 2, u'write_bytes_sec': 8611618, u'unfound_objects': 276, u'version': 813, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 152, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'50.074', u'bytes_avail': 2280807612416, u'unfound_total': 12647}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:59:18,811.811 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:59:19,665.665 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1316 16 10612 10596 32.2019 4 0.710054 1.97 2014-04-01 14:59:19,736.736 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:59:20,665.665 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1317 16 10612 10596 32.1774 0 - 1.97 2014-04-01 14:59:21,664.664 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1318 16 10612 10596 32.153 0 - 1.97 2014-04-01 14:59:22,664.664 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1319 16 10612 10596 32.1286 0 - 1.97 2014-04-01 14:59:23,663.663 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:59:23,671.671 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:59:23.579212min lat: 0.064365 max lat: 110.858 avg lat: 1.97405 2014-04-01 14:59:23,672.672 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:59:23,672.672 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1320 16 10627 10611 32.1497 15 5.50116 1.97405 2014-04-01 14:59:24,435.435 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:59:24,753.753 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1321 16 10653 10637 32.2041 104 0.558525 1.97108 2014-04-01 14:59:25,676.676 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1322 16 10668 10652 32.2251 60 1.11372 1.9696 2014-04-01 14:59:25,684.684 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:59:26,478.478 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:58:42.670009', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438110444, u'health': u'HEALTH_OK', u'kb_used': 10218540, u'store_stats': {u'bytes_total': 6244989, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099245}}, {u'last_updated': u'2014-04-01 05:58:42.980258', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437642848, u'health': u'HEALTH_OK', u'kb_used': 10686136, u'store_stats': {u'bytes_total': 7293638, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099318}}, {u'last_updated': u'2014-04-01 05:58:43.040361', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438109956, u'health': u'HEALTH_OK', u'kb_used': 10219028, u'store_stats': {u'bytes_total': 7293560, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099240}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 19101/38172 objects degraded (50.039%); 276/12733 unfound (2.168%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 3311266, u'degraded_objects': 19101, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 53292829028, u'degraded_total': 38172, u'bytes_used': 55055966208, u'unfound_ratio': u'2.168', u'recovering_objects_per_sec': 0, u'write_bytes_sec': 48013370, u'op_per_sec': 11, u'unfound_objects': 276, u'version': 818, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 152, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'50.039', u'bytes_avail': 2280456896512, u'unfound_total': 12733}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:59:26,479.479 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:59:26,664.664 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1323 16 10671 10655 32.2098 12 0.918181 1.96933 2014-04-01 14:59:27,189.189 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:59:27,664.664 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1324 16 10672 10656 32.1885 4 1.49031 1.96928 2014-04-01 14:59:28,665.665 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1325 16 10672 10656 32.1642 0 - 1.96928 2014-04-01 14:59:29,665.665 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1326 16 10672 10656 32.14 0 - 1.96928 2014-04-01 14:59:30,666.666 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1327 16 10678 10662 32.1339 8 5.43602 1.97123 2014-04-01 14:59:31,110.110 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:59:31,665.665 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1328 16 10687 10671 32.1368 36 5.2382 1.9741 2014-04-01 14:59:31,728.728 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:59:32,668.668 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1329 16 10718 10702 32.2059 124 0.399463 1.97072 2014-04-01 14:59:33,005.005 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:59:33,666.666 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1330 16 10734 10718 32.2298 64 0.735513 1.96856 2014-04-01 14:59:33,873.873 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:58:42.670009', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438110444, u'health': u'HEALTH_OK', u'kb_used': 10218540, u'store_stats': {u'bytes_total': 6244989, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099245}}, {u'last_updated': u'2014-04-01 05:58:42.980258', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437642848, u'health': u'HEALTH_OK', u'kb_used': 10686136, u'store_stats': {u'bytes_total': 7293638, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099318}}, {u'last_updated': u'2014-04-01 05:58:43.040361', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438109956, u'health': u'HEALTH_OK', u'kb_used': 10219028, u'store_stats': {u'bytes_total': 7293560, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099240}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 19149/38292 objects degraded (50.008%); 276/12773 unfound (2.161%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 19149, u'num_pgs': 178, u'data_bytes': 53460601188, u'degraded_total': 38292, u'bytes_used': 55387262976, u'unfound_ratio': u'2.161', u'op_per_sec': 1, u'write_bytes_sec': 5398862, u'unfound_objects': 276, u'version': 822, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 152, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'50.008', u'bytes_avail': 2280125599744, u'unfound_total': 12773}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:59:33,873.873 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:59:34,658.658 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:59:35,166.166 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1331 16 10740 10724 32.2236 24 1.23643 1.96812 2014-04-01 14:59:35,773.773 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1332 16 10740 10724 32.1994 0 - 1.96812 2014-04-01 14:59:36,666.666 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1333 16 10740 10724 32.1752 0 - 1.96812 2014-04-01 14:59:37,666.666 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1334 16 10740 10724 32.1511 0 - 1.96812 2014-04-01 14:59:38,749.749 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1335 16 10750 10734 32.157 10 4.38978 1.97081 2014-04-01 14:59:39,181.181 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:59:39,673.673 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1336 16 10755 10739 32.1479 20 1.50821 1.97176 2014-04-01 14:59:40,329.329 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:59:40,689.689 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1337 16 10791 10775 32.2315 144 0.390287 1.96845 2014-04-01 14:59:41,547.547 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:59:41,676.676 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1338 16 10805 10789 32.2493 56 1.07698 1.96681 2014-04-01 14:59:42,453.453 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:58:42.670009', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438110444, u'health': u'HEALTH_OK', u'kb_used': 10218540, u'store_stats': {u'bytes_total': 6244989, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099245}}, {u'last_updated': u'2014-04-01 05:58:42.980258', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437642848, u'health': u'HEALTH_OK', u'kb_used': 10686136, u'store_stats': {u'bytes_total': 7293638, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099318}}, {u'last_updated': u'2014-04-01 05:58:43.040361', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438109956, u'health': u'HEALTH_OK', u'kb_used': 10219028, u'store_stats': {u'bytes_total': 7293560, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099240}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 19265/38520 objects degraded (50.013%); 276/12849 unfound (2.148%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 5341542, u'degraded_objects': 19265, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 53779368292, u'degraded_total': 38520, u'bytes_used': 55780569088, u'unfound_ratio': u'2.148', u'recovering_objects_per_sec': 1, u'write_bytes_sec': 35610284, u'op_per_sec': 8, u'unfound_objects': 276, u'version': 827, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 152, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'50.013', u'bytes_avail': 2279732293632, u'unfound_total': 12849}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:59:42,454.454 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:59:42,671.671 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1339 16 10810 10794 32.2401 20 0.588984 1.96632 2014-04-01 14:59:43,251.251 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:59:43,677.677 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 05:59:43.582100min lat: 0.064365 max lat: 110.858 avg lat: 1.96632 2014-04-01 14:59:43,677.677 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 14:59:43,677.677 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1340 16 10810 10794 32.2161 0 - 1.96632 2014-04-01 14:59:44,678.678 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1341 16 10810 10794 32.1921 0 - 1.96632 2014-04-01 14:59:45,674.674 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1342 16 10810 10794 32.1681 0 - 1.96632 2014-04-01 14:59:46,667.667 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1343 16 10824 10808 32.1858 14 4.47256 1.97013 2014-04-01 14:59:47,307.307 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:59:47,667.667 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1344 16 10835 10819 32.1946 44 0.61696 1.96923 2014-04-01 14:59:48,033.033 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:59:48,667.667 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1345 16 10836 10820 32.1736 4 0.72837 1.96911 2014-04-01 14:59:48,964.964 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:59:49,667.667 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1346 16 10852 10836 32.1973 64 0.522944 1.96934 2014-04-01 14:59:49,772.772 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:59:42.670300', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438031416, u'health': u'HEALTH_OK', u'kb_used': 10297568, u'store_stats': {u'bytes_total': 8342141, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099245}}, {u'last_updated': u'2014-04-01 05:59:42.980497', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437570480, u'health': u'HEALTH_OK', u'kb_used': 10758504, u'store_stats': {u'bytes_total': 10439366, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099318}}, {u'last_updated': u'2014-04-01 05:59:43.040653', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438031248, u'health': u'HEALTH_OK', u'kb_used': 10297736, u'store_stats': {u'bytes_total': 10439288, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099240}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 18877/38748 objects degraded (48.717%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 2195003, u'degraded_objects': 18877, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 54098135396, u'degraded_total': 38748, u'bytes_used': 56204730368, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 15365025, u'op_per_sec': 3, u'version': 832, u'pgs_by_state': [{u'count': 13, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 12, u'state_name': u'active+recovering'}], u'degraded_ratio': u'48.717', u'bytes_avail': 2279308132352}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:59:49,773.773 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:59:50,550.550 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:59:50,851.851 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1347 16 10858 10842 32.1912 24 0.649293 1.96863 2014-04-01 14:59:51,667.667 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1348 16 10858 10842 32.1673 0 - 1.96863 2014-04-01 14:59:52,668.668 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1349 16 10858 10842 32.1434 0 - 1.96863 2014-04-01 14:59:53,668.668 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1350 16 10858 10842 32.1196 0 - 1.96863 2014-04-01 14:59:54,614.614 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:59:54,669.669 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1351 16 10888 10872 32.1847 30 0.370334 1.97016 2014-04-01 14:59:55,333.333 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:59:55,673.673 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1352 16 10904 10888 32.2082 64 0.397162 1.9684 2014-04-01 14:59:56,283.283 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 14:59:56,668.668 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1353 16 10910 10894 32.2021 24 0.662837 1.96771 2014-04-01 14:59:57,029.029 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:59:42.670300', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438031416, u'health': u'HEALTH_OK', u'kb_used': 10297568, u'store_stats': {u'bytes_total': 8342141, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099245}}, {u'last_updated': u'2014-04-01 05:59:42.980497', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437570480, u'health': u'HEALTH_OK', u'kb_used': 10758504, u'store_stats': {u'bytes_total': 10439366, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099318}}, {u'last_updated': u'2014-04-01 05:59:43.040653', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438031248, u'health': u'HEALTH_OK', u'kb_used': 10297736, u'store_stats': {u'bytes_total': 10439288, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099240}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 18884/38835 objects degraded (48.626%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 33774340, u'degraded_objects': 18884, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 54219770212, u'degraded_total': 38835, u'bytes_used': 56501088256, u'recovering_objects_per_sec': 8, u'write_bytes_sec': 30021635, u'op_per_sec': 7, u'version': 836, u'pgs_by_state': [{u'count': 12, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'48.626', u'bytes_avail': 2279011774464}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 14:59:57,030.030 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 14:59:57,667.667 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1354 16 10911 10895 32.1813 4 0.349588 1.96757 2014-04-01 14:59:57,828.828 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 14:59:58,668.668 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1355 16 10911 10895 32.1576 0 - 1.96757 2014-04-01 14:59:59,668.668 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1356 16 10911 10895 32.1338 0 - 1.96757 2014-04-01 15:00:00,670.670 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1357 16 10911 10895 32.1102 0 - 1.96757 2014-04-01 15:00:01,669.669 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1358 16 10934 10918 32.1542 23 0.553247 1.97125 2014-04-01 15:00:01,742.742 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:00:02,346.346 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:00:02,669.669 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1359 16 10955 10939 32.1924 84 0.272307 1.96911 2014-04-01 15:00:03,447.447 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:00:03,668.668 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:00:03.584967min lat: 0.064365 max lat: 110.858 avg lat: 1.96702 2014-04-01 15:00:03,668.668 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:00:03,668.668 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1360 16 10973 10957 32.2217 72 0.519879 1.96702 2014-04-01 15:00:04,172.172 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:59:42.670300', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438031416, u'health': u'HEALTH_OK', u'kb_used': 10297568, u'store_stats': {u'bytes_total': 8342141, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099245}}, {u'last_updated': u'2014-04-01 05:59:42.980497', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437570480, u'health': u'HEALTH_OK', u'kb_used': 10758504, u'store_stats': {u'bytes_total': 10439366, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099318}}, {u'last_updated': u'2014-04-01 05:59:43.040653', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438031248, u'health': u'HEALTH_OK', u'kb_used': 10297736, u'store_stats': {u'bytes_total': 10439288, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099240}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 18950/39009 objects degraded (48.579%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 18329792, u'degraded_objects': 18950, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 54463039844, u'degraded_total': 39009, u'bytes_used': 58842210304, u'recovering_objects_per_sec': 4, u'write_bytes_sec': 7547561, u'op_per_sec': 1, u'version': 840, u'pgs_by_state': [{u'count': 12, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'48.579', u'bytes_avail': 2276670652416}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:00:04,172.172 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:00:04,668.668 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1361 16 10976 10960 32.2068 12 0.48935 1.96673 2014-04-01 15:00:05,147.147 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:00:05,669.669 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1362 16 10976 10960 32.1831 0 - 1.96673 2014-04-01 15:00:06,668.668 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1363 16 10976 10960 32.1595 0 - 1.96673 2014-04-01 15:00:07,670.670 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1364 16 10976 10960 32.136 0 - 1.96673 2014-04-01 15:00:08,673.673 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1365 16 10985 10969 32.1388 9 5.30816 1.96941 2014-04-01 15:00:09,070.070 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:00:09,673.673 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1366 16 11006 10990 32.1767 84 0.970124 1.96955 2014-04-01 15:00:10,000.000 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:00:10,669.669 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1367 16 11022 11006 32.2 64 0.950138 1.96747 2014-04-01 15:00:11,089.089 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:00:11,669.669 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1368 16 11037 11021 32.2203 60 1.67521 1.96621 2014-04-01 15:00:11,866.866 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:59:42.670300', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438031416, u'health': u'HEALTH_OK', u'kb_used': 10297568, u'store_stats': {u'bytes_total': 8342141, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099245}}, {u'last_updated': u'2014-04-01 05:59:42.980497', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437570480, u'health': u'HEALTH_OK', u'kb_used': 10758504, u'store_stats': {u'bytes_total': 10439366, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099318}}, {u'last_updated': u'2014-04-01 05:59:43.040653', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438031248, u'health': u'HEALTH_OK', u'kb_used': 10297736, u'store_stats': {u'bytes_total': 10439288, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099240}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 19021/39222 objects degraded (48.496%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 38023127, u'degraded_objects': 19021, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 54760835428, u'degraded_total': 39222, u'bytes_used': 59243810816, u'recovering_objects_per_sec': 9, u'write_bytes_sec': 38023127, u'op_per_sec': 9, u'version': 845, u'pgs_by_state': [{u'count': 12, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'48.496', u'bytes_avail': 2276269051904}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:00:11,866.866 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:00:12,606.606 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:00:12,907.907 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1369 16 11041 11025 32.2085 16 1.15988 1.96602 2014-04-01 15:00:13,670.670 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1370 16 11042 11026 32.1879 4 0.342024 1.96588 2014-04-01 15:00:14,670.670 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1371 16 11042 11026 32.1644 0 - 1.96588 2014-04-01 15:00:15,670.670 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1372 16 11042 11026 32.141 0 - 1.96588 2014-04-01 15:00:16,521.521 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:00:16,669.669 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1373 16 11062 11046 32.1758 26.6667 0.352197 1.96806 2014-04-01 15:00:17,270.270 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:00:17,669.669 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1374 16 11082 11066 32.2106 80 0.310274 1.96661 2014-04-01 15:00:18,207.207 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:00:18,671.671 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1375 16 11094 11078 32.2221 48 1.32913 1.96557 2014-04-01 15:00:19,061.061 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:59:42.670300', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438031416, u'health': u'HEALTH_OK', u'kb_used': 10297568, u'store_stats': {u'bytes_total': 8342141, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099245}}, {u'last_updated': u'2014-04-01 05:59:42.980497', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437570480, u'health': u'HEALTH_OK', u'kb_used': 10758504, u'store_stats': {u'bytes_total': 10439366, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099318}}, {u'last_updated': u'2014-04-01 05:59:43.040653', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438031248, u'health': u'HEALTH_OK', u'kb_used': 10297736, u'store_stats': {u'bytes_total': 10439288, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099240}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 19102/39405 objects degraded (48.476%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 16225359, u'degraded_objects': 19102, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 55016687972, u'degraded_total': 39405, u'bytes_used': 59472003072, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 12980287, u'op_per_sec': 3, u'version': 849, u'pgs_by_state': [{u'count': 12, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'48.476', u'bytes_avail': 2276040859648}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:00:19,062.062 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:00:19,670.670 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1376 16 11103 11087 32.2248 36 0.465929 1.96467 2014-04-01 15:00:20,002.002 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:00:20,669.669 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1377 16 11104 11088 32.2043 4 0.577083 1.96455 2014-04-01 15:00:21,671.671 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1378 16 11104 11088 32.181 0 - 1.96455 2014-04-01 15:00:22,671.671 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1379 16 11104 11088 32.1576 0 - 1.96455 2014-04-01 15:00:23,671.671 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:00:23.587762min lat: 0.064365 max lat: 110.858 avg lat: 1.96763 2014-04-01 15:00:23,671.671 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:00:23,672.672 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1380 16 11118 11102 32.1749 18.6667 0.186743 1.96763 2014-04-01 15:00:23,921.921 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:00:24,520.520 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:00:24,835.835 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1381 16 11142 11126 32.2211 96 0.371188 1.9655 2014-04-01 15:00:25,615.615 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:00:25,670.670 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1382 16 11156 11140 32.2383 56 0.753955 1.96396 2014-04-01 15:00:26,368.368 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:59:42.670300', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438031416, u'health': u'HEALTH_OK', u'kb_used': 10297568, u'store_stats': {u'bytes_total': 8342141, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099245}}, {u'last_updated': u'2014-04-01 05:59:42.980497', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437570480, u'health': u'HEALTH_OK', u'kb_used': 10758504, u'store_stats': {u'bytes_total': 10439366, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099318}}, {u'last_updated': u'2014-04-01 05:59:43.040653', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438031248, u'health': u'HEALTH_OK', u'kb_used': 10297736, u'store_stats': {u'bytes_total': 10439288, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099240}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'4 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 19206/39645 objects degraded (48.445%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 25626081, u'degraded_objects': 19206, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 55352232292, u'degraded_total': 39645, u'bytes_used': 59995881472, u'recovering_objects_per_sec': 6, u'write_bytes_sec': 48048902, u'op_per_sec': 11, u'version': 854, u'pgs_by_state': [{u'count': 12, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'48.445', u'bytes_avail': 2275516981248}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:00:26,369.369 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:00:26,671.671 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1383 16 11157 11141 32.2179 4 0.835506 1.96386 2014-04-01 15:00:27,177.177 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:00:27,671.671 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1384 16 11158 11142 32.1975 4 2.48744 1.96391 2014-04-01 15:00:28,671.671 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1385 16 11158 11142 32.1743 0 - 1.96391 2014-04-01 15:00:29,675.675 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1386 16 11158 11142 32.151 0 - 1.96391 2014-04-01 15:00:30,671.671 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1387 16 11158 11142 32.1279 0 - 1.96391 2014-04-01 15:00:31,089.089 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:00:31,672.672 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1388 16 11184 11168 32.1796 26 0.177856 1.9669 2014-04-01 15:00:31,831.831 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:00:32,673.673 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1389 16 11199 11183 32.1997 60 1.3652 1.96574 2014-04-01 15:00:32,754.754 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:00:33,570.570 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:59:42.670300', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438031416, u'health': u'HEALTH_OK', u'kb_used': 10297568, u'store_stats': {u'bytes_total': 8342141, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099245}}, {u'last_updated': u'2014-04-01 05:59:42.980497', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437570480, u'health': u'HEALTH_OK', u'kb_used': 10758504, u'store_stats': {u'bytes_total': 10439366, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099318}}, {u'last_updated': u'2014-04-01 05:59:43.040653', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438031248, u'health': u'HEALTH_OK', u'kb_used': 10297736, u'store_stats': {u'bytes_total': 10439288, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099240}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'14 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 19217/39729 objects degraded (48.370%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 40167875, u'degraded_objects': 19217, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 55469672804, u'degraded_total': 39729, u'bytes_used': 60272279552, u'recovering_objects_per_sec': 9, u'write_bytes_sec': 51122750, u'op_per_sec': 12, u'version': 857, u'pgs_by_state': [{u'count': 12, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'48.370', u'bytes_avail': 2275240583168}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:00:33,571.571 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:00:33,670.670 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1390 16 11213 11197 32.2168 56 0.912127 1.96446 2014-04-01 15:00:34,343.343 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:00:34,970.970 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1391 16 11221 11205 32.2166 32 2.16852 1.96392 2014-04-01 15:00:35,672.672 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1392 16 11223 11207 32.1992 8 1.60045 1.96383 2014-04-01 15:00:36,672.672 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1393 16 11226 11210 32.1847 12 1.03122 1.96383 2014-04-01 15:00:37,672.672 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1394 16 11226 11210 32.1616 0 - 1.96383 2014-04-01 15:00:38,429.429 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:00:38,671.671 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1395 16 11238 11222 32.173 24 3.5159 1.96695 2014-04-01 15:00:39,019.019 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:00:39,672.672 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1396 16 11264 11248 32.2244 104 0.67796 1.96419 2014-04-01 15:00:40,260.260 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:00:40,672.672 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1397 16 11270 11254 32.2185 24 0.536689 1.96347 2014-04-01 15:00:41,176.176 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 05:59:42.670300', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438031416, u'health': u'HEALTH_OK', u'kb_used': 10297568, u'store_stats': {u'bytes_total': 8342141, u'bytes_log': 5177344, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099245}}, {u'last_updated': u'2014-04-01 05:59:42.980497', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437570480, u'health': u'HEALTH_OK', u'kb_used': 10758504, u'store_stats': {u'bytes_total': 10439366, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099318}}, {u'last_updated': u'2014-04-01 05:59:43.040653', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 438031248, u'health': u'HEALTH_OK', u'kb_used': 10297736, u'store_stats': {u'bytes_total': 10439288, u'bytes_log': 7274496, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3099240}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'14 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 19326/39993 objects degraded (48.323%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 30100686, u'degraded_objects': 19326, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 55838771556, u'degraded_total': 39993, u'bytes_used': 60707315712, u'recovering_objects_per_sec': 7, u'write_bytes_sec': 56660115, u'op_per_sec': 13, u'version': 863, u'pgs_by_state': [{u'count': 12, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'48.323', u'bytes_avail': 2274805547008}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:00:41,177.177 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:00:41,671.671 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1398 16 11273 11257 32.2041 12 1.55074 1.96326 2014-04-01 15:00:41,947.947 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:00:42,673.673 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1399 16 11278 11262 32.1953 20 3.2571 1.96378 2014-04-01 15:00:43,673.673 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:00:43.590232min lat: 0.064365 max lat: 110.858 avg lat: 1.96384 2014-04-01 15:00:43,674.674 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:00:43,674.674 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1400 16 11283 11267 32.1866 20 0.198704 1.96384 2014-04-01 15:00:44,673.673 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1401 16 11283 11267 32.1637 0 - 1.96384 2014-04-01 15:00:45,672.672 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1402 16 11285 11269 32.1464 4 5.75402 1.96452 2014-04-01 15:00:46,016.016 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:00:46,736.736 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:00:46,890.890 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1403 16 11301 11285 32.1691 64 0.251883 1.96662 2014-04-01 15:00:47,678.678 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1404 16 11307 11291 32.1633 24 1.30038 1.96637 2014-04-01 15:00:47,883.883 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:00:48,678.678 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1405 16 11335 11319 32.2201 112 0.337225 1.96362 2014-04-01 15:00:48,824.824 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 06:00:48.732222 7f15751d2700 0 monclient: hunting for new mon 2014-04-01 15:00:48,975.975 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:00:42.670585', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437955048, u'health': u'HEALTH_OK', u'kb_used': 10373936, u'store_stats': {u'bytes_total': 3947232, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:00:42.980766', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437514380, u'health': u'HEALTH_OK', u'kb_used': 10814604, u'store_stats': {u'bytes_total': 3947230, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:00:43.040968', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437953748, u'health': u'HEALTH_OK', u'kb_used': 10375236, u'store_stats': {u'bytes_total': 3947174, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'16 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 19354/40101 objects degraded (48.263%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 16149155, u'degraded_objects': 19354, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 55989766500, u'degraded_total': 40101, u'bytes_used': 61047111680, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 8612883, u'op_per_sec': 2, u'version': 867, u'pgs_by_state': [{u'count': 12, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'48.263', u'bytes_avail': 2274465751040}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:00:48,975.975 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:00:49,680.680 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1406 16 11336 11320 32.2 4 0.912313 1.96352 2014-04-01 15:00:49,958.958 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:00:50,783.783 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1407 16 11336 11320 32.1772 0 - 1.96352 2014-04-01 15:00:51,676.676 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1408 16 11336 11320 32.1543 0 - 1.96352 2014-04-01 15:00:52,673.673 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1409 16 11336 11320 32.1315 0 - 1.96352 2014-04-01 15:00:53,678.678 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1410 16 11361 11345 32.1796 25 0.619035 1.96569 2014-04-01 15:00:53,971.971 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:00:54,677.677 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1411 16 11363 11347 32.1625 8 1.10838 1.96548 2014-04-01 15:00:54,995.995 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:00:55,686.686 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1412 16 11384 11368 32.1992 84 0.19237 1.9646 2014-04-01 15:00:56,031.031 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:00:56,681.681 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1413 16 11396 11380 32.2104 48 0.818934 1.963 2014-04-01 15:00:56,799.799 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:00:42.670585', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437955048, u'health': u'HEALTH_OK', u'kb_used': 10373936, u'store_stats': {u'bytes_total': 3947232, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:00:42.980766', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437514380, u'health': u'HEALTH_OK', u'kb_used': 10814604, u'store_stats': {u'bytes_total': 3947230, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:00:43.040968', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437953748, u'health': u'HEALTH_OK', u'kb_used': 10375236, u'store_stats': {u'bytes_total': 3947174, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'19 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 19423/40332 objects degraded (48.158%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 32798492, u'degraded_objects': 19423, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 56312727908, u'degraded_total': 40332, u'bytes_used': 61491773440, u'recovering_objects_per_sec': 7, u'write_bytes_sec': 47375600, u'op_per_sec': 11, u'version': 872, u'pgs_by_state': [{u'count': 12, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'48.158', u'bytes_avail': 2274021089280}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:00:56,799.799 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:00:57,553.553 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:00:57,703.703 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1414 16 11399 11383 32.1961 12 1.02305 1.9627 2014-04-01 15:00:58,682.682 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1415 16 11399 11383 32.1733 0 - 1.9627 2014-04-01 15:00:59,676.676 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1416 16 11399 11383 32.1506 0 - 1.9627 2014-04-01 15:01:00,677.677 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1417 16 11411 11395 32.1618 16 4.77829 1.9658 2014-04-01 15:01:01,491.491 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:01:01,673.673 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1418 16 11421 11405 32.1673 40 0.145262 1.96532 2014-04-01 15:01:02,267.267 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:01:02,674.674 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1419 16 11430 11414 32.17 36 1.82599 1.96532 2014-04-01 15:01:03,195.195 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:01:03,674.674 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:01:03.592975min lat: 0.064365 max lat: 110.858 avg lat: 1.96408 2014-04-01 15:01:03,674.674 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:01:03,674.674 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1420 16 11446 11430 32.1924 64 2.19988 1.96408 2014-04-01 15:01:04,002.002 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:00:42.670585', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437955048, u'health': u'HEALTH_OK', u'kb_used': 10373936, u'store_stats': {u'bytes_total': 3947232, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:00:42.980766', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437514380, u'health': u'HEALTH_OK', u'kb_used': 10814604, u'store_stats': {u'bytes_total': 3947230, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:00:43.040968', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437953748, u'health': u'HEALTH_OK', u'kb_used': 10375236, u'store_stats': {u'bytes_total': 3947174, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'19 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 19471/40473 objects degraded (48.109%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 16158882, u'degraded_objects': 19471, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 56509860196, u'degraded_total': 40473, u'bytes_used': 61847379968, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 8618070, u'op_per_sec': 2, u'version': 876, u'pgs_by_state': [{u'count': 12, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'48.109', u'bytes_avail': 2273665482752}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:01:04,002.002 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:01:04,674.674 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1421 16 11458 11442 32.2035 48 1.83438 1.96284 2014-04-01 15:01:04,893.893 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:01:05,674.674 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1422 16 11462 11446 32.1921 16 1.69159 1.96269 2014-04-01 15:01:06,675.675 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1423 16 11462 11446 32.1695 0 - 1.96269 2014-04-01 15:01:07,674.674 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1424 16 11462 11446 32.1469 0 - 1.96269 2014-04-01 15:01:08,675.675 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1425 16 11462 11446 32.1243 0 - 1.96269 2014-04-01 15:01:08,815.815 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:01:09,399.399 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:01:09,715.715 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1426 16 11484 11468 32.1635 22 0.913619 1.96559 2014-04-01 15:01:10,493.493 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:01:10,674.674 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1427 16 11513 11497 32.2223 116 0.645866 1.96212 2014-04-01 15:01:11,527.527 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:00:42.670585', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437955048, u'health': u'HEALTH_OK', u'kb_used': 10373936, u'store_stats': {u'bytes_total': 3947232, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:00:42.980766', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437514380, u'health': u'HEALTH_OK', u'kb_used': 10814604, u'store_stats': {u'bytes_total': 3947230, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:00:43.040968', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437953748, u'health': u'HEALTH_OK', u'kb_used': 10375236, u'store_stats': {u'bytes_total': 3947174, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'19 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 19527/40668 objects degraded (48.016%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 48746115, u'degraded_objects': 19527, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 56782489956, u'degraded_total': 40668, u'bytes_used': 62259322880, u'recovering_objects_per_sec': 11, u'write_bytes_sec': 30256209, u'op_per_sec': 7, u'version': 881, u'pgs_by_state': [{u'count': 12, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'48.016', u'bytes_avail': 2273253539840}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:01:11,528.528 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:01:11,680.680 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1428 16 11520 11504 32.2193 28 1.27074 1.9614 2014-04-01 15:01:12,237.237 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:01:12,675.675 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1429 16 11520 11504 32.1967 0 - 1.9614 2014-04-01 15:01:13,676.676 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1430 16 11522 11506 32.1798 4 1.38496 1.96137 2014-04-01 15:01:14,679.679 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1431 16 11531 11515 32.1825 36 3.84535 1.96296 2014-04-01 15:01:15,676.676 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1432 16 11538 11522 32.1796 28 0.346913 1.96308 2014-04-01 15:01:16,161.161 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:01:16,676.676 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1433 16 11538 11522 32.1571 0 - 1.96308 2014-04-01 15:01:16,908.908 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:01:17,677.677 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1434 16 11538 11522 32.1347 0 - 1.96308 2014-04-01 15:01:18,013.013 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:01:18,677.677 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1435 16 11561 11545 32.1764 30.6667 0.328818 1.96418 2014-04-01 15:01:18,894.894 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:00:42.670585', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437955048, u'health': u'HEALTH_OK', u'kb_used': 10373936, u'store_stats': {u'bytes_total': 3947232, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:00:42.980766', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437514380, u'health': u'HEALTH_OK', u'kb_used': 10814604, u'store_stats': {u'bytes_total': 3947230, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:00:43.040968', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437953748, u'health': u'HEALTH_OK', u'kb_used': 10375236, u'store_stats': {u'bytes_total': 3947174, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'19 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 19600/40836 objects degraded (47.997%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 16885558, u'degraded_objects': 19600, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 57017370980, u'degraded_total': 40836, u'bytes_used': 62538268672, u'recovering_objects_per_sec': 4, u'write_bytes_sec': 7879927, u'op_per_sec': 1, u'version': 885, u'pgs_by_state': [{u'count': 12, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.997', u'bytes_avail': 2272974594048}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:01:18,895.895 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:01:19,677.677 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1436 16 11567 11551 32.1707 24 0.499508 1.96357 2014-04-01 15:01:19,912.912 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:01:20,678.678 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1437 16 11581 11565 32.1873 56 2.1991 1.96339 2014-04-01 15:01:21,679.679 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1438 16 11593 11577 32.1983 48 0.505643 1.96214 2014-04-01 15:01:22,678.678 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1439 16 11595 11579 32.1815 8 1.40868 1.96202 2014-04-01 15:01:23,677.677 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:01:23.595763min lat: 0.064365 max lat: 110.858 avg lat: 1.96188 2014-04-01 15:01:23,677.677 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:01:23,678.678 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1440 16 11596 11580 32.1619 4 0.307287 1.96188 2014-04-01 15:01:23,828.828 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:01:24,452.452 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:01:24,767.767 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1441 16 11596 11580 32.1396 0 - 1.96188 2014-04-01 15:01:25,541.541 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:01:25,676.676 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1442 16 11612 11596 32.1617 32 0.409126 1.96474 2014-04-01 15:01:26,262.262 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:00:42.670585', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437955048, u'health': u'HEALTH_OK', u'kb_used': 10373936, u'store_stats': {u'bytes_total': 3947232, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:00:42.980766', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437514380, u'health': u'HEALTH_OK', u'kb_used': 10814604, u'store_stats': {u'bytes_total': 3947230, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:00:43.040968', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437953748, u'health': u'HEALTH_OK', u'kb_used': 10375236, u'store_stats': {u'bytes_total': 3947174, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'19 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 19659/41055 objects degraded (47.885%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 36638691, u'degraded_objects': 19659, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 57323555172, u'degraded_total': 41055, u'bytes_used': 62966915072, u'recovering_objects_per_sec': 8, u'write_bytes_sec': 62285775, u'op_per_sec': 14, u'version': 890, u'pgs_by_state': [{u'count': 12, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.885', u'bytes_avail': 2272545947648}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:01:26,263.263 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:01:26,677.677 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1443 16 11616 11600 32.1505 16 0.112948 1.9642 2014-04-01 15:01:26,959.959 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:01:27,733.733 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1444 16 11625 11609 32.1531 36 2.20111 1.96448 2014-04-01 15:01:28,677.677 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1445 16 11652 11636 32.2056 108 0.43846 1.96176 2014-04-01 15:01:29,678.678 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1446 16 11657 11641 32.1971 20 0.73744 1.96122 2014-04-01 15:01:30,678.678 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1447 16 11657 11641 32.1749 0 - 1.96122 2014-04-01 15:01:30,881.881 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:01:31,645.645 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:01:31,803.803 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1448 16 11657 11641 32.1527 0 - 1.96122 2014-04-01 15:01:32,573.573 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:01:32,677.677 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1449 16 11670 11654 32.1664 17.3333 0.417691 1.96249 2014-04-01 15:01:33,330.330 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:00:42.670585', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437955048, u'health': u'HEALTH_OK', u'kb_used': 10373936, u'store_stats': {u'bytes_total': 3947232, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:00:42.980766', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437514380, u'health': u'HEALTH_OK', u'kb_used': 10814604, u'store_stats': {u'bytes_total': 3947230, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:00:43.040968', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437953748, u'health': u'HEALTH_OK', u'kb_used': 10375236, u'store_stats': {u'bytes_total': 3947174, u'bytes_log': 983040, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'25 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 19705/41202 objects degraded (47.825%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 39197191, u'degraded_objects': 19705, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 57529076068, u'degraded_total': 41202, u'bytes_used': 63313764352, u'recovering_objects_per_sec': 9, u'write_bytes_sec': 78394383, u'op_per_sec': 18, u'version': 893, u'pgs_by_state': [{u'count': 12, u'state_name': u'active+recovery_wait'}, {u'count': 153, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.825', u'bytes_avail': 2272199098368}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:01:33,331.331 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:01:34,427.427 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1450 16 11687 11671 32.1911 68 1.67004 1.9625 2014-04-01 15:01:34,678.678 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1451 16 11712 11696 32.2378 100 0.64528 1.9594 2014-04-01 15:01:35,679.679 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1452 16 11718 11702 32.2321 24 1.06111 1.9588 2014-04-01 15:01:36,703.703 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1453 16 11718 11702 32.2099 0 - 1.9588 2014-04-01 15:01:37,729.729 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1454 16 11725 11709 32.207 14 2.39619 1.95946 2014-04-01 15:01:38,063.063 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:01:38,709.709 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1455 16 11727 11711 32.1904 8 0.430509 1.95919 2014-04-01 15:01:39,683.683 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1456 16 11727 11711 32.1683 0 - 1.95919 2014-04-01 15:01:40,679.679 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1457 16 11732 11716 32.1599 10 5.76978 1.96088 2014-04-01 15:01:41,682.682 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1458 16 11766 11750 32.2311 136 0.331027 1.95953 2014-04-01 15:01:41,959.959 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:01:42,679.679 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1459 16 11783 11767 32.2557 68 1.09508 1.95773 2014-04-01 15:01:42,687.687 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:01:43,680.680 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:01:43.598720min lat: 0.064365 max lat: 110.858 avg lat: 1.95736 2014-04-01 15:01:43,681.681 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:01:43,681.681 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1460 16 11788 11772 32.2473 20 1.17989 1.95736 2014-04-01 15:01:43,768.768 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:01:44,605.605 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.066412', u'skew': u'-0.145152', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.066184', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:01:42.670848', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437856212, u'health': u'HEALTH_OK', u'kb_used': 10472772, u'store_stats': {u'bytes_total': 6044384, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:01:42.981037', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437449436, u'health': u'HEALTH_OK', u'kb_used': 10879548, u'store_stats': {u'bytes_total': 7092958, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:01:43.041201', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437855224, u'health': u'HEALTH_OK', u'kb_used': 10473760, u'store_stats': {u'bytes_total': 7092902, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'12 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 19706/41439 objects degraded (47.554%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 19706, u'num_pgs': 178, u'data_bytes': 57860426084, u'degraded_total': 41439, u'bytes_used': 63893827584, u'op_per_sec': 1, u'write_bytes_sec': 5587449, u'version': 900, u'pgs_by_state': [{u'count': 12, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 12, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.554', u'bytes_avail': 2271619035136}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:01:44,605.605 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:01:44,679.679 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1461 16 11788 11772 32.2252 0 - 1.95736 2014-04-01 15:01:45,303.303 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:01:45,909.909 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1462 16 11788 11772 32.2031 0 - 1.95736 2014-04-01 15:01:46,680.680 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1463 16 11788 11772 32.1811 0 - 1.95736 2014-04-01 15:01:47,681.681 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1464 16 11788 11772 32.1592 0 - 1.95736 2014-04-01 15:01:48,682.682 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1465 15 11817 11802 32.2191 24 0.5504 1.95987 2014-04-01 15:01:49,374.374 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:01:49,680.680 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1466 16 11831 11815 32.2326 52 0.166104 1.95826 2014-04-01 15:01:50,045.045 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:01:50,680.680 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1467 16 11845 11829 32.2488 56 0.662559 1.95711 2014-04-01 15:01:51,154.154 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:01:51,680.680 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1468 16 11845 11829 32.2268 0 - 1.95711 2014-04-01 15:01:51,921.921 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:01:42.670848', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437856212, u'health': u'HEALTH_OK', u'kb_used': 10472772, u'store_stats': {u'bytes_total': 6044384, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:01:42.981037', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437449436, u'health': u'HEALTH_OK', u'kb_used': 10879548, u'store_stats': {u'bytes_total': 7092958, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:01:43.041201', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437855224, u'health': u'HEALTH_OK', u'kb_used': 10473760, u'store_stats': {u'bytes_total': 7092902, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 19839/41712 objects degraded (47.562%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 1746727, u'degraded_objects': 19839, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 58242107748, u'degraded_total': 41712, u'bytes_used': 64189636608, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 47161651, u'op_per_sec': 11, u'version': 905, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.562', u'bytes_avail': 2271323226112}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:01:51,921.921 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:01:52,615.615 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:01:52,681.681 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1469 16 11845 11829 32.2049 0 - 1.95711 2014-04-01 15:01:53,680.680 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1470 16 11845 11829 32.183 0 - 1.95711 2014-04-01 15:01:54,683.683 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1471 16 11845 11829 32.1611 0 - 1.95711 2014-04-01 15:01:55,685.685 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1472 16 11853 11837 32.161 6.4 5.82432 1.95976 2014-04-01 15:01:56,548.548 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:01:56,682.682 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1473 16 11883 11867 32.2206 120 0.542709 1.95919 2014-04-01 15:01:57,491.491 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:01:57,683.683 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1474 16 11893 11877 32.2259 40 0.978811 1.95824 2014-04-01 15:01:58,442.442 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:01:58,681.681 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1475 16 11896 11880 32.2122 12 0.952296 1.95798 2014-04-01 15:01:59,145.145 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:01:42.670848', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437856212, u'health': u'HEALTH_OK', u'kb_used': 10472772, u'store_stats': {u'bytes_total': 6044384, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:01:42.981037', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437449436, u'health': u'HEALTH_OK', u'kb_used': 10879548, u'store_stats': {u'bytes_total': 7092958, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:01:43.041201', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437855224, u'health': u'HEALTH_OK', u'kb_used': 10473760, u'store_stats': {u'bytes_total': 7092902, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 19880/41814 objects degraded (47.544%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 19880, u'num_pgs': 178, u'data_bytes': 58384714084, u'degraded_total': 41814, u'bytes_used': 64562065408, u'op_per_sec': 1, u'write_bytes_sec': 4270423, u'version': 909, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.544', u'bytes_avail': 2270950797312}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:01:59,146.146 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:01:59,681.681 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1476 16 11896 11880 32.1903 0 - 1.95798 2014-04-01 15:01:59,950.950 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:02:00,717.717 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1477 16 11896 11880 32.1685 0 - 1.95798 2014-04-01 15:02:01,714.714 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1478 16 11896 11880 32.1468 0 - 1.95798 2014-04-01 15:02:02,682.682 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1479 16 11896 11880 32.125 0 - 1.95798 2014-04-01 15:02:03,694.694 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:02:03.601500min lat: 0.064365 max lat: 110.858 avg lat: 1.96113 2014-04-01 15:02:03,694.694 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:02:03,694.694 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1480 16 11925 11909 32.1817 23.2 0.488371 1.96113 2014-04-01 15:02:03,865.865 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:02:04,508.508 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:02:04,901.901 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1481 16 11946 11930 32.2167 84 0.843369 1.95858 2014-04-01 15:02:05,694.694 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1482 16 11954 11938 32.2165 32 0.843861 1.95782 2014-04-01 15:02:05,815.815 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:02:06,645.645 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:01:42.670848', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437856212, u'health': u'HEALTH_OK', u'kb_used': 10472772, u'store_stats': {u'bytes_total': 6044384, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:01:42.981037', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437449436, u'health': u'HEALTH_OK', u'kb_used': 10879548, u'store_stats': {u'bytes_total': 7092958, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:01:43.041201', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437855224, u'health': u'HEALTH_OK', u'kb_used': 10473760, u'store_stats': {u'bytes_total': 7092902, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 19975/42051 objects degraded (47.502%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 19975, u'num_pgs': 178, u'data_bytes': 58716064100, u'degraded_total': 42051, u'bytes_used': 66704683008, u'op_per_sec': 14, u'write_bytes_sec': 59799598, u'version': 914, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.502', u'bytes_avail': 2268808179712}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:02:06,646.646 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:02:06,693.693 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1483 16 11954 11938 32.1948 0 - 1.95782 2014-04-01 15:02:07,419.419 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:02:07,687.687 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1484 16 11954 11938 32.1731 0 - 1.95782 2014-04-01 15:02:08,693.693 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1485 16 11954 11938 32.1515 0 - 1.95782 2014-04-01 15:02:09,693.693 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1486 16 11954 11938 32.1298 0 - 1.95782 2014-04-01 15:02:10,701.701 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1487 16 11964 11948 32.1351 8 6.34406 1.96134 2014-04-01 15:02:11,410.410 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:02:11,697.697 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1488 16 11995 11979 32.1968 124 0.192217 1.95974 2014-04-01 15:02:12,180.180 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:02:12,683.683 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1489 16 12009 11993 32.2128 56 1.27625 1.95818 2014-04-01 15:02:13,261.261 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:02:13,682.682 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1490 16 12021 12005 32.2234 48 1.7859 1.95739 2014-04-01 15:02:13,979.979 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:01:42.670848', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437856212, u'health': u'HEALTH_OK', u'kb_used': 10472772, u'store_stats': {u'bytes_total': 6044384, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:01:42.981037', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437449436, u'health': u'HEALTH_OK', u'kb_used': 10879548, u'store_stats': {u'bytes_total': 7092958, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:01:43.041201', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437855224, u'health': u'HEALTH_OK', u'kb_used': 10473760, u'store_stats': {u'bytes_total': 7092902, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 20009/42141 objects degraded (47.481%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 20009, u'num_pgs': 178, u'data_bytes': 58841893220, u'degraded_total': 42141, u'bytes_used': 67012964352, u'op_per_sec': 0, u'write_bytes_sec': 2153475, u'version': 918, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.481', u'bytes_avail': 2268499898368}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:02:13,979.979 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:02:14,683.683 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1491 16 12021 12005 32.2018 0 - 1.95739 2014-04-01 15:02:14,775.775 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:02:15,683.683 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1492 16 12021 12005 32.1802 0 - 1.95739 2014-04-01 15:02:16,683.683 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1493 16 12021 12005 32.1587 0 - 1.95739 2014-04-01 15:02:17,684.684 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1494 16 12021 12005 32.1371 0 - 1.95739 2014-04-01 15:02:18,685.685 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:02:18,687.687 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1495 16 12039 12023 32.1638 14.4 0.341065 1.96119 2014-04-01 15:02:19,601.601 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:02:19,754.754 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1496 16 12067 12051 32.2171 112 0.570985 1.95764 2014-04-01 15:02:20,536.536 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:02:20,684.684 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1497 16 12078 12062 32.225 44 1.51978 1.98516 2014-04-01 15:02:21,273.273 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:01:42.670848', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437856212, u'health': u'HEALTH_OK', u'kb_used': 10472772, u'store_stats': {u'bytes_total': 6044384, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:01:42.981037', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437449436, u'health': u'HEALTH_OK', u'kb_used': 10879548, u'store_stats': {u'bytes_total': 7092958, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:01:43.041201', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437855224, u'health': u'HEALTH_OK', u'kb_used': 10473760, u'store_stats': {u'bytes_total': 7092902, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 20149/42405 objects degraded (47.516%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 5064357, u'degraded_objects': 20149, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 59210991972, u'degraded_total': 42405, u'bytes_used': 67339927552, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 50643571, u'op_per_sec': 12, u'version': 923, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.516', u'bytes_avail': 2268172935168}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:02:21,273.273 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:02:21,684.684 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1498 16 12085 12069 32.2222 28 1.90867 1.98476 2014-04-01 15:02:21,977.977 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:02:22,684.684 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1499 16 12085 12069 32.2007 0 - 1.98476 2014-04-01 15:02:23,684.684 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:02:23.604226min lat: 0.064365 max lat: 341.507 avg lat: 1.98476 2014-04-01 15:02:23,684.684 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:02:23,685.685 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1500 16 12085 12069 32.1792 0 - 1.98476 2014-04-01 15:02:24,686.686 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1501 16 12085 12069 32.1578 0 - 1.98476 2014-04-01 15:02:25,689.689 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1502 16 12085 12069 32.1364 0 - 1.98476 2014-04-01 15:02:25,899.899 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:02:26,523.523 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:02:26,841.841 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1503 16 12113 12097 32.1895 22.4 0.453515 1.98793 2014-04-01 15:02:27,616.616 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:02:27,685.685 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1504 16 12129 12113 32.2106 64 0.224896 1.98616 2014-04-01 15:02:28,369.369 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:01:42.670848', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437856212, u'health': u'HEALTH_OK', u'kb_used': 10472772, u'store_stats': {u'bytes_total': 6044384, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:01:42.981037', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437449436, u'health': u'HEALTH_OK', u'kb_used': 10879548, u'store_stats': {u'bytes_total': 7092958, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:01:43.041201', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437855224, u'health': u'HEALTH_OK', u'kb_used': 10473760, u'store_stats': {u'bytes_total': 7092902, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 20207/42531 objects degraded (47.511%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 3648418, u'degraded_objects': 20207, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 59387152740, u'degraded_total': 42531, u'bytes_used': 67670659072, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 71144160, u'op_per_sec': 16, u'version': 926, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.511', u'bytes_avail': 2267842203648}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:02:28,369.369 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:02:28,686.686 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1505 16 12146 12130 32.2344 68 1.30876 1.98476 2014-04-01 15:02:29,087.087 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:02:29,685.685 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1506 16 12153 12137 32.2316 28 0.938066 1.98421 2014-04-01 15:02:30,685.685 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1507 16 12153 12137 32.2102 0 - 1.98421 2014-04-01 15:02:31,687.687 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1508 16 12153 12137 32.1889 0 - 1.98421 2014-04-01 15:02:32,687.687 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1509 16 12153 12137 32.1675 0 - 1.98421 2014-04-01 15:02:33,013.013 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:02:33,654.654 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:02:33,968.968 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1510 16 12161 12145 32.1674 8 5.5751 1.98637 2014-04-01 15:02:34,689.689 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1511 16 12199 12183 32.2467 152 0.448169 1.98434 2014-04-01 15:02:34,742.742 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:02:35,587.587 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:01:42.670848', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437856212, u'health': u'HEALTH_OK', u'kb_used': 10472772, u'store_stats': {u'bytes_total': 6044384, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:01:42.981037', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437449436, u'health': u'HEALTH_OK', u'kb_used': 10879548, u'store_stats': {u'bytes_total': 7092958, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:01:43.041201', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437855224, u'health': u'HEALTH_OK', u'kb_used': 10473760, u'store_stats': {u'bytes_total': 7092902, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 20313/42747 objects degraded (47.519%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 3121174, u'degraded_objects': 20313, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 59689142628, u'degraded_total': 42747, u'bytes_used': 67955617792, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 9363523, u'op_per_sec': 2, u'version': 931, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.519', u'bytes_avail': 2267557244928}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:02:35,588.588 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:02:35,686.686 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1512 16 12209 12193 32.2518 40 1.20105 1.98343 2014-04-01 15:02:36,440.440 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:02:36,753.753 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1513 16 12222 12206 32.2649 52 1.17808 1.98248 2014-04-01 15:02:37,687.687 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1514 16 12225 12209 32.2515 12 2.07257 1.98237 2014-04-01 15:02:38,687.687 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1515 16 12225 12209 32.2302 0 - 1.98237 2014-04-01 15:02:39,690.690 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1516 16 12225 12209 32.2089 0 - 1.98237 2014-04-01 15:02:40,375.375 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:02:40,687.687 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1517 16 12225 12209 32.1877 0 - 1.98237 2014-04-01 15:02:41,443.443 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:02:41,691.691 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1518 16 12240 12224 32.206 15 0.157616 1.98583 2014-04-01 15:02:42,374.374 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:02:42,687.687 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1519 16 12260 12244 32.2375 80 0.216772 1.98478 2014-04-01 15:02:43,172.172 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:02:42.671107', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437780108, u'health': u'HEALTH_OK', u'kb_used': 10548876, u'store_stats': {u'bytes_total': 7092960, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:02:42.981247', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437396488, u'health': u'HEALTH_OK', u'kb_used': 10932496, u'store_stats': {u'bytes_total': 9190110, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:02:43.041452', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437779648, u'health': u'HEALTH_OK', u'kb_used': 10549336, u'store_stats': {u'bytes_total': 9190054, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 20407/42945 objects degraded (47.519%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 5411002, u'degraded_objects': 20407, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 59965966692, u'degraded_total': 42945, u'bytes_used': 68297404416, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 91987035, u'op_per_sec': 21, u'version': 935, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.519', u'bytes_avail': 2267215458304}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:02:43,172.172 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:02:43,688.688 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:02:43.607129min lat: 0.064365 max lat: 341.507 avg lat: 1.98302 2014-04-01 15:02:43,688.688 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:02:43,688.688 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1520 16 12277 12261 32.261 68 0.401833 1.98302 2014-04-01 15:02:43,891.891 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:02:44,687.687 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1521 16 12279 12263 32.2451 8 0.571108 1.9828 2014-04-01 15:02:45,688.688 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1522 16 12279 12263 32.2239 0 - 1.9828 2014-04-01 15:02:46,689.689 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1523 16 12279 12263 32.2027 0 - 1.9828 2014-04-01 15:02:47,689.689 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1524 16 12279 12263 32.1816 0 - 1.9828 2014-04-01 15:02:47,810.810 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:02:48,519.519 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:02:48,689.689 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1525 15 12281 12266 32.1683 3 5.95981 1.98376 2014-04-01 15:02:49,446.446 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:02:49,688.688 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1526 16 12302 12286 32.1997 80 0.861489 1.98685 2014-04-01 15:02:50,353.353 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:02:42.671107', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437780108, u'health': u'HEALTH_OK', u'kb_used': 10548876, u'store_stats': {u'bytes_total': 7092960, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:02:42.981247', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437396488, u'health': u'HEALTH_OK', u'kb_used': 10932496, u'store_stats': {u'bytes_total': 9190110, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:02:43.041452', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437779648, u'health': u'HEALTH_OK', u'kb_used': 10549336, u'store_stats': {u'bytes_total': 9190054, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 20482/43107 objects degraded (47.514%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 3223180, u'degraded_objects': 20482, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 60192459108, u'degraded_total': 43107, u'bytes_used': 68621770752, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 4297573, u'op_per_sec': 1, u'version': 940, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.514', u'bytes_avail': 2266891091968}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:02:50,354.354 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:02:50,689.689 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1527 16 12320 12304 32.2257 72 0.425873 1.985 2014-04-01 15:02:51,178.178 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:02:51,690.690 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1528 16 12339 12323 32.2544 76 0.732316 1.98351 2014-04-01 15:02:52,689.689 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1529 16 12344 12328 32.2464 20 0.684579 1.983 2014-04-01 15:02:53,689.689 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1530 16 12344 12328 32.2253 0 - 1.983 2014-04-01 15:02:54,691.691 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1531 16 12344 12328 32.2042 0 - 1.983 2014-04-01 15:02:55,126.126 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:02:55,690.690 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1532 16 12344 12328 32.1832 0 - 1.983 2014-04-01 15:02:55,886.886 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:02:56,690.690 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1533 16 12360 12344 32.204 16 4.31095 1.9869 2014-04-01 15:02:56,813.813 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:02:57,647.647 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:02:42.671107', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437780108, u'health': u'HEALTH_OK', u'kb_used': 10548876, u'store_stats': {u'bytes_total': 7092960, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:02:42.981247', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437396488, u'health': u'HEALTH_OK', u'kb_used': 10932496, u'store_stats': {u'bytes_total': 9190110, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:02:43.041452', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437779648, u'health': u'HEALTH_OK', u'kb_used': 10549336, u'store_stats': {u'bytes_total': 9190054, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 20574/43296 objects degraded (47.519%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 1843232, u'degraded_objects': 20574, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 60456700260, u'degraded_total': 43296, u'bytes_used': 68861386752, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 44237588, u'op_per_sec': 10, u'version': 944, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.519', u'bytes_avail': 2266651475968}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:02:57,647.647 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:02:57,691.691 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1534 16 12377 12361 32.2273 68 1.22599 1.98533 2014-04-01 15:02:58,415.415 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:02:58,719.719 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1535 16 12397 12381 32.2584 80 0.390604 1.98333 2014-04-01 15:02:59,691.691 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1536 16 12403 12387 32.253 24 1.03214 1.98284 2014-04-01 15:03:00,691.691 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1537 16 12403 12387 32.232 0 - 1.98284 2014-04-01 15:03:01,692.692 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1538 16 12403 12387 32.2111 0 - 1.98284 2014-04-01 15:03:02,334.334 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:03:02,691.691 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1539 16 12403 12387 32.1902 0 - 1.98284 2014-04-01 15:03:03,096.096 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:03:03,691.691 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:03:03.610103min lat: 0.064365 max lat: 341.507 avg lat: 1.98312 2014-04-01 15:03:03,691.691 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:03:03,691.691 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1540 16 12404 12388 32.1718 1 5.41735 1.98312 2014-04-01 15:03:04,021.021 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:03:04,693.693 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1541 16 12425 12409 32.2055 84 0.884035 1.98648 2014-04-01 15:03:04,887.887 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:02:42.671107', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437780108, u'health': u'HEALTH_OK', u'kb_used': 10548876, u'store_stats': {u'bytes_total': 7092960, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:02:42.981247', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437396488, u'health': u'HEALTH_OK', u'kb_used': 10932496, u'store_stats': {u'bytes_total': 9190110, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:02:43.041452', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437779648, u'health': u'HEALTH_OK', u'kb_used': 10549336, u'store_stats': {u'bytes_total': 9190054, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 20662/43449 objects degraded (47.555%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 20662, u'num_pgs': 178, u'data_bytes': 60670609764, u'degraded_total': 43449, u'bytes_used': 69197246464, u'op_per_sec': 4, u'write_bytes_sec': 19349479, u'version': 948, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.555', u'bytes_avail': 2266315616256}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:03:04,887.887 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:03:05,677.677 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:03:05,698.698 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1542 16 12457 12441 32.2676 128 0.447474 1.98306 2014-04-01 15:03:06,692.692 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1543 16 12466 12450 32.27 36 0.832091 1.9822 2014-04-01 15:03:07,692.692 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1544 16 12466 12450 32.2491 0 - 1.9822 2014-04-01 15:03:08,693.693 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1545 16 12466 12450 32.2282 0 - 1.9822 2014-04-01 15:03:09,612.612 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:03:09,692.692 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1546 16 12466 12450 32.2074 0 - 1.9822 2014-04-01 15:03:10,374.374 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:03:10,693.693 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1547 16 12466 12450 32.1866 0 - 1.9822 2014-04-01 15:03:11,303.303 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:03:11,692.692 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1548 16 12483 12467 32.2097 13.6 0.410037 1.98571 2014-04-01 15:03:12,053.053 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:02:42.671107', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437780108, u'health': u'HEALTH_OK', u'kb_used': 10548876, u'store_stats': {u'bytes_total': 7092960, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:02:42.981247', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437396488, u'health': u'HEALTH_OK', u'kb_used': 10932496, u'store_stats': {u'bytes_total': 9190110, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:02:43.041452', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437779648, u'health': u'HEALTH_OK', u'kb_used': 10549336, u'store_stats': {u'bytes_total': 9190054, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 20767/43668 objects degraded (47.557%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 12640467, u'degraded_objects': 20767, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 60976793956, u'degraded_total': 43668, u'bytes_used': 69470466048, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 101123741, u'op_per_sec': 24, u'version': 953, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.557', u'bytes_avail': 2266042396672}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:03:12,053.053 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:03:12,697.697 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1549 16 12496 12480 32.2225 52 1.69238 1.98562 2014-04-01 15:03:12,781.781 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:03:13,696.696 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1550 16 12520 12504 32.2636 96 0.724896 1.98295 2014-04-01 15:03:14,713.713 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1551 16 12535 12519 32.2815 60 0.852866 1.98163 2014-04-01 15:03:15,702.702 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1552 16 12536 12520 32.2633 4 2.19245 1.98165 2014-04-01 15:03:16,720.720 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1553 16 12536 12520 32.2425 0 - 1.98165 2014-04-01 15:03:16,767.767 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:03:17,522.522 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:03:17,700.700 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1554 16 12536 12520 32.2217 0 - 1.98165 2014-04-01 15:03:18,511.511 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:03:18,693.693 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1555 16 12536 12520 32.201 0 - 1.98165 2014-04-01 15:03:19,331.331 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:02:42.671107', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437780108, u'health': u'HEALTH_OK', u'kb_used': 10548876, u'store_stats': {u'bytes_total': 7092960, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:02:42.981247', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437396488, u'health': u'HEALTH_OK', u'kb_used': 10932496, u'store_stats': {u'bytes_total': 9190110, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:02:43.041452', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437779648, u'health': u'HEALTH_OK', u'kb_used': 10549336, u'store_stats': {u'bytes_total': 9190054, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 20855/43827 objects degraded (47.585%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 20855, u'num_pgs': 178, u'data_bytes': 61199092068, u'degraded_total': 43827, u'bytes_used': 69843959808, u'op_per_sec': 4, u'write_bytes_sec': 18864983, u'version': 957, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.585', u'bytes_avail': 2265668902912}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:03:19,331.331 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:03:19,710.710 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1556 16 12545 12529 32.2034 9 4.99676 1.98427 2014-04-01 15:03:20,122.122 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:03:20,711.711 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1557 16 12574 12558 32.2573 116 0.477346 1.98358 2014-04-01 15:03:21,695.695 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1558 16 12599 12583 32.3007 100 0.338774 1.98099 2014-04-01 15:03:22,702.702 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1559 16 12604 12588 32.2928 20 0.922822 1.98054 2014-04-01 15:03:23,708.708 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:03:23.612978min lat: 0.064365 max lat: 341.507 avg lat: 1.98054 2014-04-01 15:03:23,708.708 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:03:23,708.708 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1560 16 12604 12588 32.2721 0 - 1.98054 2014-04-01 15:03:24,245.245 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:03:24,696.696 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1561 16 12604 12588 32.2515 0 - 1.98054 2014-04-01 15:03:25,123.123 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:03:25,694.694 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1562 16 12610 12594 32.2462 8 3.83647 1.98116 2014-04-01 15:03:26,042.042 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:03:26,694.694 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1563 16 12610 12594 32.2255 0 - 1.98116 2014-04-01 15:03:26,817.817 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:02:42.671107', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437780108, u'health': u'HEALTH_OK', u'kb_used': 10548876, u'store_stats': {u'bytes_total': 7092960, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:02:42.981247', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437396488, u'health': u'HEALTH_OK', u'kb_used': 10932496, u'store_stats': {u'bytes_total': 9190110, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:02:43.041452', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437779648, u'health': u'HEALTH_OK', u'kb_used': 10549336, u'store_stats': {u'bytes_total': 9190054, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 20968/44079 objects degraded (47.569%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 13793048, u'degraded_objects': 20968, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 61551413604, u'degraded_total': 44079, u'bytes_used': 70209769472, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 101723733, u'op_per_sec': 24, u'version': 962, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.569', u'bytes_avail': 2265303093248}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:03:26,818.818 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:03:27,662.662 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:03:27,697.697 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1564 16 12635 12619 32.2689 50 0.401883 1.98299 2014-04-01 15:03:28,695.695 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1565 16 12653 12637 32.2943 72 0.290051 1.98107 2014-04-01 15:03:29,696.696 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1566 16 12664 12648 32.3017 44 1.22939 1.98008 2014-04-01 15:03:30,695.695 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1567 16 12670 12654 32.2964 24 0.222152 1.97966 2014-04-01 15:03:31,586.586 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:03:31,695.695 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1568 16 12670 12654 32.2758 0 - 1.97966 2014-04-01 15:03:32,355.355 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:03:32,695.695 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1569 16 12670 12654 32.2553 0 - 1.97966 2014-04-01 15:03:33,287.287 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:03:33,726.726 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1570 16 12670 12654 32.2347 0 - 1.97966 2014-04-01 15:03:34,136.136 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:02:42.671107', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437780108, u'health': u'HEALTH_OK', u'kb_used': 10548876, u'store_stats': {u'bytes_total': 7092960, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:02:42.981247', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437396488, u'health': u'HEALTH_OK', u'kb_used': 10932496, u'store_stats': {u'bytes_total': 9190110, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:02:43.041452', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437779648, u'health': u'HEALTH_OK', u'kb_used': 10549336, u'store_stats': {u'bytes_total': 9190054, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 21039/44220 objects degraded (47.578%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 21039, u'num_pgs': 178, u'data_bytes': 61748545892, u'degraded_total': 44220, u'bytes_used': 70605549568, u'op_per_sec': 2, u'write_bytes_sec': 11863546, u'version': 966, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.578', u'bytes_avail': 2264907313152}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:03:34,136.136 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:03:34,771.771 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1571 16 12684 12668 32.2498 14 5.64089 1.98352 2014-04-01 15:03:35,283.283 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:03:35,902.902 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1572 16 12697 12681 32.2624 52 1.17508 1.98313 2014-04-01 15:03:36,720.720 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1573 16 12727 12711 32.3182 120 0.762925 1.97984 2014-04-01 15:03:37,741.741 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1574 16 12731 12715 32.3078 16 0.477369 1.97942 2014-04-01 15:03:38,765.765 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1575 16 12731 12715 32.2873 0 - 1.97942 2014-04-01 15:03:39,720.720 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:03:39,788.788 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1576 16 12731 12715 32.2668 0 - 1.97942 2014-04-01 15:03:40,475.475 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:03:40,697.697 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1577 16 12731 12715 32.2463 0 - 1.97942 2014-04-01 15:03:41,550.550 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:03:41,697.697 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1578 16 12731 12715 32.2259 0 - 1.97942 2014-04-01 15:03:42,308.308 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:02:42.671107', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437780108, u'health': u'HEALTH_OK', u'kb_used': 10548876, u'store_stats': {u'bytes_total': 7092960, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:02:42.981247', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437396488, u'health': u'HEALTH_OK', u'kb_used': 10932496, u'store_stats': {u'bytes_total': 9190110, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:02:43.041452', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437779648, u'health': u'HEALTH_OK', u'kb_used': 10549336, u'store_stats': {u'bytes_total': 9190054, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 21151/44451 objects degraded (47.583%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 11022766, u'degraded_objects': 21151, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 62071507300, u'degraded_total': 44451, u'bytes_used': 70878216192, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 99204895, u'op_per_sec': 23, u'version': 971, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.583', u'bytes_avail': 2264634646528}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:03:42,308.308 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:03:42,697.697 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1579 16 12754 12738 32.2637 18.4 0.439911 1.98308 2014-04-01 15:03:43,162.162 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:03:43,697.697 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:03:43.615957min lat: 0.064365 max lat: 341.507 avg lat: 1.98117 2014-04-01 15:03:43,697.697 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:03:43,697.697 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1580 16 12773 12757 32.2914 76 0.414664 1.98117 2014-04-01 15:03:44,698.698 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1581 16 12788 12772 32.3089 60 1.64781 1.97985 2014-04-01 15:03:45,698.698 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1582 16 12791 12775 32.2961 12 0.28434 1.97952 2014-04-01 15:03:46,697.697 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1583 16 12791 12775 32.2757 0 - 1.97952 2014-04-01 15:03:47,082.082 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:03:47,697.697 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1584 16 12791 12775 32.2553 0 - 1.97952 2014-04-01 15:03:47,979.979 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:03:48,698.698 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1585 16 12791 12775 32.235 0 - 1.97952 2014-04-01 15:03:48,898.898 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:03:49,698.698 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1586 16 12809 12793 32.26 18 0.477383 1.98351 2014-04-01 15:03:49,737.737 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:03:42.671390', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437705220, u'health': u'HEALTH_OK', u'kb_used': 10623764, u'store_stats': {u'bytes_total': 9190112, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:03:42.981473', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437345400, u'health': u'HEALTH_OK', u'kb_used': 10983584, u'store_stats': {u'bytes_total': 12335838, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:03:43.041765', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437703900, u'health': u'HEALTH_OK', u'kb_used': 10625084, u'store_stats': {u'bytes_total': 12335782, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 21231/44595 objects degraded (47.608%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 21231, u'num_pgs': 178, u'data_bytes': 62272833892, u'degraded_total': 44595, u'bytes_used': 71166631936, u'op_per_sec': 3, u'write_bytes_sec': 13873920, u'version': 975, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.608', u'bytes_avail': 2264346230784}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:03:49,738.738 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:03:50,464.464 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:03:50,702.702 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1587 16 12835 12819 32.3052 104 0.664812 1.98056 2014-04-01 15:03:51,698.698 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1588 16 12853 12837 32.3302 72 0.554682 1.97928 2014-04-01 15:03:52,699.699 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1589 16 12860 12844 32.3275 28 0.897711 1.97856 2014-04-01 15:03:53,702.702 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1590 16 12861 12845 32.3097 4 0.603595 1.97846 2014-04-01 15:03:54,375.375 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:03:54,698.698 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1591 16 12861 12845 32.2894 0 - 1.97846 2014-04-01 15:03:55,095.095 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:03:55,699.699 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1592 16 12861 12845 32.2691 0 - 1.97846 2014-04-01 15:03:56,016.016 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:03:56,699.699 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1593 16 12861 12845 32.2488 0 - 1.97846 2014-04-01 15:03:56,732.732 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:03:42.671390', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437705220, u'health': u'HEALTH_OK', u'kb_used': 10623764, u'store_stats': {u'bytes_total': 9190112, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:03:42.981473', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437345400, u'health': u'HEALTH_OK', u'kb_used': 10983584, u'store_stats': {u'bytes_total': 12335838, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:03:43.041765', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437703900, u'health': u'HEALTH_OK', u'kb_used': 10625084, u'store_stats': {u'bytes_total': 12335782, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 21339/44847 objects degraded (47.582%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 11095288, u'degraded_objects': 21339, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 62625155428, u'degraded_total': 44847, u'bytes_used': 71629701120, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 110952889, u'op_per_sec': 26, u'version': 980, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.582', u'bytes_avail': 2263883161600}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:03:56,732.732 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:03:57,604.604 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:03:57,764.764 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1594 16 12882 12866 32.2813 21 0.527061 1.98229 2014-04-01 15:03:58,699.699 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1595 16 12901 12885 32.3087 76 0.450379 1.98014 2014-04-01 15:03:59,699.699 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1596 16 12915 12899 32.3235 56 1.37335 1.97919 2014-04-01 15:04:00,699.699 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1597 16 12916 12900 32.3058 4 0.288657 1.97906 2014-04-01 15:04:01,526.526 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:04:01,700.700 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1598 16 12916 12900 32.2856 0 - 1.97906 2014-04-01 15:04:02,339.339 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:04:02,700.700 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1599 16 12916 12900 32.2654 0 - 1.97906 2014-04-01 15:04:03,285.285 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:04:03,700.700 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:04:03.618818min lat: 0.064365 max lat: 341.507 avg lat: 1.97906 2014-04-01 15:04:03,700.700 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:04:03,700.700 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1600 16 12916 12900 32.2452 0 - 1.97906 2014-04-01 15:04:04,117.117 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:03:42.671390', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437705220, u'health': u'HEALTH_OK', u'kb_used': 10623764, u'store_stats': {u'bytes_total': 9190112, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:03:42.981473', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437345400, u'health': u'HEALTH_OK', u'kb_used': 10983584, u'store_stats': {u'bytes_total': 12335838, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:03:43.041765', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437703900, u'health': u'HEALTH_OK', u'kb_used': 10625084, u'store_stats': {u'bytes_total': 12335782, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 21417/45003 objects degraded (47.590%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 5032126, u'degraded_objects': 21417, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 62843259236, u'degraded_total': 45003, u'bytes_used': 73831538688, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 24154209, u'op_per_sec': 5, u'version': 984, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.590', u'bytes_avail': 2261681324032}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:04:04,118.118 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:04:04,700.700 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1601 16 12916 12900 32.2251 0 - 1.97906 2014-04-01 15:04:05,127.127 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:04:05,704.704 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1602 16 12955 12939 32.3023 31.2 0.367083 1.98098 2014-04-01 15:04:06,702.702 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1603 16 12968 12952 32.3146 52 0.847878 1.9797 2014-04-01 15:04:07,702.702 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1604 16 12980 12964 32.3244 48 0.730289 1.97888 2014-04-01 15:04:08,703.703 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1605 16 12980 12964 32.3042 0 - 1.97888 2014-04-01 15:04:09,123.123 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:04:09,701.701 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1606 16 12980 12964 32.2841 0 - 1.97888 2014-04-01 15:04:09,970.970 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:04:10,701.701 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1607 16 12980 12964 32.264 0 - 1.97888 2014-04-01 15:04:10,893.893 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:04:11,621.621 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:03:42.671390', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437705220, u'health': u'HEALTH_OK', u'kb_used': 10623764, u'store_stats': {u'bytes_total': 9190112, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:03:42.981473', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437345400, u'health': u'HEALTH_OK', u'kb_used': 10983584, u'store_stats': {u'bytes_total': 12335838, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:03:43.041765', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437703900, u'health': u'HEALTH_OK', u'kb_used': 10625084, u'store_stats': {u'bytes_total': 12335782, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 21500/45195 objects degraded (47.572%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 3681154, u'degraded_objects': 21500, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 63111694692, u'degraded_total': 45195, u'bytes_used': 74298769408, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 95710022, u'op_per_sec': 22, u'version': 989, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.572', u'bytes_avail': 2261214093312}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:04:11,621.621 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:04:11,702.702 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1608 16 12980 12964 32.244 0 - 1.97888 2014-04-01 15:04:12,382.382 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:04:12,748.748 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1609 16 13013 12997 32.306 26.4 0.584094 1.98076 2014-04-01 15:04:13,702.702 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1610 16 13029 13013 32.3256 64 0.886475 1.97924 2014-04-01 15:04:14,702.702 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1611 16 13038 13022 32.3279 36 0.091503 1.97828 2014-04-01 15:04:15,702.702 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1612 16 13038 13022 32.3079 0 - 1.97828 2014-04-01 15:04:16,317.317 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:04:16,702.702 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1613 16 13038 13022 32.2878 0 - 1.97828 2014-04-01 15:04:17,023.023 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:04:17,702.702 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1614 16 13038 13022 32.2678 0 - 1.97828 2014-04-01 15:04:17,940.940 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:04:18,703.703 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1615 16 13038 13022 32.2479 0 - 1.97828 2014-04-01 15:04:18,746.746 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:03:42.671390', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437705220, u'health': u'HEALTH_OK', u'kb_used': 10623764, u'store_stats': {u'bytes_total': 9190112, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:03:42.981473', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437345400, u'health': u'HEALTH_OK', u'kb_used': 10983584, u'store_stats': {u'bytes_total': 12335838, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:03:43.041765', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437703900, u'health': u'HEALTH_OK', u'kb_used': 10625084, u'store_stats': {u'bytes_total': 12335782, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 21598/45366 objects degraded (47.608%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 21598, u'num_pgs': 178, u'data_bytes': 63350770020, u'degraded_total': 45366, u'bytes_used': 74518007808, u'op_per_sec': 4, u'write_bytes_sec': 17241800, u'version': 993, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.608', u'bytes_avail': 2260994854912}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:04:18,747.747 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:04:19,502.502 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:04:19,702.702 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1616 16 13042 13026 32.2378 3.2 6.41266 1.97964 2014-04-01 15:04:20,703.703 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1617 16 13072 13056 32.2921 120 0.614695 1.98143 2014-04-01 15:04:21,704.704 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1618 16 13092 13076 32.3215 80 0.187382 1.97954 2014-04-01 15:04:22,703.703 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1619 16 13092 13076 32.3016 0 - 1.97954 2014-04-01 15:04:23,429.429 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:04:23,703.703 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:04:23.621535min lat: 0.064365 max lat: 341.507 avg lat: 1.97954 2014-04-01 15:04:23,703.703 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:04:23,703.703 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1620 16 13092 13076 32.2816 0 - 1.97954 2014-04-01 15:04:24,137.137 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:04:24,703.703 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1621 16 13092 13076 32.2617 0 - 1.97954 2014-04-01 15:04:25,067.067 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:04:25,703.703 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1622 16 13092 13076 32.2418 0 - 1.97954 2014-04-01 15:04:25,812.812 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:03:42.671390', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437705220, u'health': u'HEALTH_OK', u'kb_used': 10623764, u'store_stats': {u'bytes_total': 9190112, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:03:42.981473', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437345400, u'health': u'HEALTH_OK', u'kb_used': 10983584, u'store_stats': {u'bytes_total': 12335838, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:03:43.041765', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437703900, u'health': u'HEALTH_OK', u'kb_used': 10625084, u'store_stats': {u'bytes_total': 12335782, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 21665/45528 objects degraded (47.586%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 5511079, u'degraded_objects': 21665, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 63577262436, u'degraded_total': 45528, u'bytes_used': 74734206976, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 55110794, u'op_per_sec': 13, u'version': 997, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.586', u'bytes_avail': 2260778655744}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:04:25,812.812 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:04:26,608.608 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:04:26,712.712 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1623 16 13097 13081 32.2343 4 6.38768 1.98108 2014-04-01 15:04:27,186.186 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.133.37]: 2014-04-01 06:04:27.088576 7fe3a2fd6700 0 -- 10.214.133.37:0/1005437 >> 10.214.133.37:6810/18895 pipe(0x7fe3ac017210 sd=10 :42551 s=2 pgs=2 cs=1 l=1 c=0x7fe3ac0136b0).injecting socket failure 2014-04-01 15:04:27,760.760 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1624 16 13125 13109 32.2834 112 0.662763 1.98209 2014-04-01 15:04:28,709.709 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1625 16 13133 13117 32.2832 32 0.449749 1.98134 2014-04-01 15:04:29,706.706 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1626 16 13145 13129 32.2929 48 0.528007 1.98055 2014-04-01 15:04:30,683.683 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:04:30,727.727 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1627 16 13147 13131 32.2779 8 0.269851 1.98034 2014-04-01 15:04:31,486.486 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:04:31,707.707 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1628 16 13147 13131 32.2581 0 - 1.98034 2014-04-01 15:04:32,708.708 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1629 16 13147 13131 32.2383 0 - 1.98034 2014-04-01 15:04:32,766.766 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:04:33,688.688 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:03:42.671390', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437705220, u'health': u'HEALTH_OK', u'kb_used': 10623764, u'store_stats': {u'bytes_total': 9190112, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:03:42.981473', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437345400, u'health': u'HEALTH_OK', u'kb_used': 10983584, u'store_stats': {u'bytes_total': 12335838, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:03:43.041765', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437703900, u'health': u'HEALTH_OK', u'kb_used': 10625084, u'store_stats': {u'bytes_total': 12335782, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 21742/45675 objects degraded (47.602%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 3670087, u'degraded_objects': 21742, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 63782783332, u'degraded_total': 45675, u'bytes_used': 75004465152, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 80741916, u'op_per_sec': 19, u'version': 1001, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.602', u'bytes_avail': 2260508397568}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:04:33,689.689 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:04:33,720.720 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1630 16 13147 13131 32.2185 0 - 1.98034 2014-04-01 15:04:34,547.547 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:04:34,715.715 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1631 16 13151 13135 32.2086 4 6.4508 1.98167 2014-04-01 15:04:35,714.714 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1632 16 13177 13161 32.2526 104 0.846817 1.98381 2014-04-01 15:04:36,716.716 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1633 16 13194 13178 32.2745 68 0.48046 1.98217 2014-04-01 15:04:37,706.706 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1634 16 13205 13189 32.2816 44 1.61831 1.98137 2014-04-01 15:04:38,601.601 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:04:38,706.706 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1635 16 13210 13194 32.2741 20 0.589524 1.98105 2014-04-01 15:04:39,479.479 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:04:39,710.710 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1636 16 13210 13194 32.2544 0 - 1.98105 2014-04-01 15:04:40,719.719 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:04:40,720.720 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1637 16 13210 13194 32.2347 0 - 1.98105 2014-04-01 15:04:41,582.582 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:03:42.671390', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437705220, u'health': u'HEALTH_OK', u'kb_used': 10623764, u'store_stats': {u'bytes_total': 9190112, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898640}}, {u'last_updated': u'2014-04-01 06:03:42.981473', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437345400, u'health': u'HEALTH_OK', u'kb_used': 10983584, u'store_stats': {u'bytes_total': 12335838, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898638}}, {u'last_updated': u'2014-04-01 06:03:43.041765', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437703900, u'health': u'HEALTH_OK', u'kb_used': 10625084, u'store_stats': {u'bytes_total': 12335782, u'bytes_log': 9371648, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 2898582}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 21844/45897 objects degraded (47.594%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 12648025, u'degraded_objects': 21844, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 64093161828, u'degraded_total': 45897, u'bytes_used': 75319287808, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 106604789, u'op_per_sec': 25, u'version': 1007, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.594', u'bytes_avail': 2260193574912}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:04:41,582.582 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:04:41,730.730 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1638 16 13210 13194 32.215 0 - 1.98105 2014-04-01 15:04:42,711.711 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:04:42,918.918 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1639 16 13234 13218 32.2539 24 0.600208 1.98393 2014-04-01 15:04:43,729.729 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:04:43.624378min lat: 0.064365 max lat: 341.507 avg lat: 1.9811 2014-04-01 15:04:43,730.730 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:04:43,730.730 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1640 16 13258 13242 32.2928 96 0.859819 1.9811 2014-04-01 15:04:44,709.709 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1641 16 13268 13252 32.2975 40 1.59594 1.98033 2014-04-01 15:04:45,708.708 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1642 16 13279 13263 32.3046 44 0.844879 1.97978 2014-04-01 15:04:46,712.712 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1643 16 13279 13263 32.2849 0 - 1.97978 2014-04-01 15:04:46,852.852 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:04:47,628.628 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:04:47,864.864 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1644 16 13279 13263 32.2653 0 - 1.97978 2014-04-01 15:04:48,821.821 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1645 16 13279 13263 32.2457 0 - 1.97978 2014-04-01 15:04:48,829.829 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:04:49,622.622 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:04:42.671722', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437647528, u'health': u'HEALTH_OK', u'kb_used': 10681456, u'store_stats': {u'bytes_total': 4068682, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544378}}, {u'last_updated': u'2014-04-01 06:04:43.055608', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437300736, u'health': u'HEALTH_OK', u'kb_used': 11028248, u'store_stats': {u'bytes_total': 4068567, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544263}}, {u'last_updated': u'2014-04-01 06:04:43.042086', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437645808, u'health': u'HEALTH_OK', u'kb_used': 10683176, u'store_stats': {u'bytes_total': 4068568, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544264}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 21928/46059 objects degraded (47.609%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 21928, u'num_pgs': 178, u'data_bytes': 64319654244, u'degraded_total': 46059, u'bytes_used': 75769581568, u'op_per_sec': 4, u'write_bytes_sec': 19895709, u'version': 1011, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.609', u'bytes_avail': 2259743281152}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:04:49,622.622 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:04:49,723.723 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1646 16 13284 13268 32.2382 5 5.61397 1.98115 2014-04-01 15:04:50,324.324 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:04:50,711.711 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1647 16 13303 13287 32.2648 76 1.02356 1.98287 2014-04-01 15:04:52,053.053 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1648 16 13327 13311 32.3035 96 0.833342 1.98066 2014-04-01 15:04:52,743.743 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1649 16 13333 13317 32.2984 24 0.9689 1.98023 2014-04-01 15:04:53,726.726 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1650 16 13333 13317 32.2789 0 - 1.98023 2014-04-01 15:04:54,737.737 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1651 16 13333 13317 32.2593 0 - 1.98023 2014-04-01 15:04:55,360.360 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:04:55,709.709 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1652 16 13333 13317 32.2398 0 - 1.98023 2014-04-01 15:04:56,026.026 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:04:56,722.722 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1653 16 13333 13317 32.2203 0 - 1.98023 2014-04-01 15:04:57,058.058 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:04:57,711.711 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1654 16 13350 13334 32.2419 13.6 5.53951 1.98463 2014-04-01 15:04:57,903.903 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:04:42.671722', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437647528, u'health': u'HEALTH_OK', u'kb_used': 10681456, u'store_stats': {u'bytes_total': 4068682, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544378}}, {u'last_updated': u'2014-04-01 06:04:43.055608', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437300736, u'health': u'HEALTH_OK', u'kb_used': 11028248, u'store_stats': {u'bytes_total': 4068567, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544263}}, {u'last_updated': u'2014-04-01 06:04:43.042086', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437645808, u'health': u'HEALTH_OK', u'kb_used': 10683176, u'store_stats': {u'bytes_total': 4068568, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544264}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 22020/46272 objects degraded (47.588%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 7128681, u'degraded_objects': 22020, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 64617449828, u'degraded_total': 46272, u'bytes_used': 76025839616, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 94455027, u'op_per_sec': 22, u'version': 1016, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.588', u'bytes_avail': 2259487023104}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:04:57,903.903 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:04:58,727.727 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1655 16 13375 13359 32.2828 100 0.42635 1.98224 2014-04-01 15:04:58,746.746 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:04:59,708.708 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1656 16 13388 13372 32.2947 52 0.953843 1.98085 2014-04-01 15:05:00,708.708 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1657 16 13393 13377 32.2873 20 0.508009 1.98041 2014-04-01 15:05:01,709.709 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1658 16 13393 13377 32.2678 0 - 1.98041 2014-04-01 15:05:02,670.670 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:05:02,709.709 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1659 16 13393 13377 32.2484 0 - 1.98041 2014-04-01 15:05:03,430.430 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:05:03,709.709 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:05:03.627238min lat: 0.064365 max lat: 341.507 avg lat: 1.98041 2014-04-01 15:05:03,709.709 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:05:03,709.709 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1660 16 13393 13377 32.229 0 - 1.98041 2014-04-01 15:05:04,348.348 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:05:04,711.711 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1661 16 13393 13377 32.2096 0 - 1.98041 2014-04-01 15:05:05,250.250 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:04:42.671722', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437647528, u'health': u'HEALTH_OK', u'kb_used': 10681456, u'store_stats': {u'bytes_total': 4068682, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544378}}, {u'last_updated': u'2014-04-01 06:04:43.055608', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437300736, u'health': u'HEALTH_OK', u'kb_used': 11028248, u'store_stats': {u'bytes_total': 4068567, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544263}}, {u'last_updated': u'2014-04-01 06:04:43.042086', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437645808, u'health': u'HEALTH_OK', u'kb_used': 10683176, u'store_stats': {u'bytes_total': 4068568, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544264}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 22079/46443 objects degraded (47.540%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 4198847, u'degraded_objects': 22079, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 64856525156, u'degraded_total': 46443, u'bytes_used': 76486107136, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 19944523, u'op_per_sec': 4, u'version': 1021, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.540', u'bytes_avail': 2259026755584}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:05:05,250.250 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:05:05,712.712 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1662 16 13425 13409 32.2672 25.6 0.585004 1.98323 2014-04-01 15:05:06,201.201 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:05:06,709.709 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1663 16 13440 13424 32.2839 60 0.75728 1.98158 2014-04-01 15:05:07,710.710 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1664 16 13456 13440 32.3029 64 0.279132 1.98046 2014-04-01 15:05:08,710.710 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1665 16 13457 13441 32.2859 4 0.226181 1.98033 2014-04-01 15:05:09,710.710 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1666 16 13457 13441 32.2665 0 - 1.98033 2014-04-01 15:05:10,264.264 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:05:10,710.710 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1667 16 13457 13441 32.2472 0 - 1.98033 2014-04-01 15:05:11,027.027 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:05:11,710.710 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1668 16 13457 13441 32.2278 0 - 1.98033 2014-04-01 15:05:11,949.949 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:05:12,710.710 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1669 16 13462 13446 32.2205 5 6.52425 1.98191 2014-04-01 15:05:12,765.765 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:04:42.671722', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437647528, u'health': u'HEALTH_OK', u'kb_used': 10681456, u'store_stats': {u'bytes_total': 4068682, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544378}}, {u'last_updated': u'2014-04-01 06:04:43.055608', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437300736, u'health': u'HEALTH_OK', u'kb_used': 11028248, u'store_stats': {u'bytes_total': 4068567, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544263}}, {u'last_updated': u'2014-04-01 06:04:43.042086', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437645808, u'health': u'HEALTH_OK', u'kb_used': 10683176, u'store_stats': {u'bytes_total': 4068568, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544264}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'11 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'24 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 22186/46641 objects degraded (47.568%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 5499361, u'degraded_objects': 22186, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 65133349220, u'degraded_total': 46641, u'bytes_used': 76554874880, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 106320979, u'op_per_sec': 25, u'version': 1025, u'pgs_by_state': [{u'count': 11, u'state_name': u'active+recovery_wait'}, {u'count': 154, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'47.568', u'bytes_avail': 2258957987840}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:05:12,765.765 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:05:13,716.716 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1670 16 13493 13477 32.2755 124 0.68641 1.98254 2014-04-01 15:05:13,744.744 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:05:14,715.715 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1671 16 13509 13493 32.2944 64 0.981226 1.98141 2014-04-01 15:05:15,711.711 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1672 16 13513 13497 32.2847 16 0.409625 1.98096 2014-04-01 15:05:16,711.711 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1673 16 13513 13497 32.2654 0 - 1.98096 2014-04-01 15:05:17,679.679 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:05:17,711.711 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1674 16 13516 13500 32.2533 6 2.3034 1.98106 2014-04-01 15:05:18,108.108 INFO:teuthology.orchestra.run.err:[10.214.133.37]: 2014-04-01 06:05:18.026360 7f0f7a5cc700 0 monclient: hunting for new mon 2014-04-01 15:05:18,235.235 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:05:18,711.711 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1675 16 13516 13500 32.234 0 - 1.98106 2014-04-01 15:05:19,473.473 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:05:19,711.711 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1676 16 13516 13500 32.2148 0 - 1.98106 2014-04-01 15:05:20,231.231 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:04:42.671722', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437647528, u'health': u'HEALTH_OK', u'kb_used': 10681456, u'store_stats': {u'bytes_total': 4068682, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544378}}, {u'last_updated': u'2014-04-01 06:04:43.055608', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437300736, u'health': u'HEALTH_OK', u'kb_used': 11028248, u'store_stats': {u'bytes_total': 4068567, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544263}}, {u'last_updated': u'2014-04-01 06:04:43.042086', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437645808, u'health': u'HEALTH_OK', u'kb_used': 10683176, u'store_stats': {u'bytes_total': 4068568, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544264}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'23 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 21815/46806 objects degraded (46.607%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 8298446, u'degraded_objects': 21815, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 65364035940, u'degraded_total': 46806, u'bytes_used': 77051305984, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 27266324, u'op_per_sec': 6, u'version': 1030, u'pgs_by_state': [{u'count': 10, u'state_name': u'active+recovery_wait'}, {u'count': 155, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'46.607', u'bytes_avail': 2258461556736}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:05:20,231.231 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:05:20,713.713 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1677 16 13530 13514 32.229 18.6667 3.3149 1.98474 2014-04-01 15:05:21,035.035 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:05:21,714.714 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1678 16 13551 13535 32.2598 84 0.729607 1.98313 2014-04-01 15:05:22,719.719 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1679 16 13565 13549 32.274 56 0.679986 1.98208 2014-04-01 15:05:23,712.712 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:05:23.629996min lat: 0.064365 max lat: 341.507 avg lat: 1.98188 2014-04-01 15:05:23,712.712 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:05:23,712.712 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1680 16 13567 13551 32.2595 8 0.218329 1.98188 2014-04-01 15:05:24,712.712 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1681 16 13567 13551 32.2403 0 - 1.98188 2014-04-01 15:05:25,106.106 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:05:25,812.812 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:05:25,970.970 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1682 16 13567 13551 32.2212 0 - 1.98188 2014-04-01 15:05:26,712.712 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1683 16 13567 13551 32.202 0 - 1.98188 2014-04-01 15:05:26,759.759 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:05:27,459.459 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:04:42.671722', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437647528, u'health': u'HEALTH_OK', u'kb_used': 10681456, u'store_stats': {u'bytes_total': 4068682, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544378}}, {u'last_updated': u'2014-04-01 06:04:43.055608', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437300736, u'health': u'HEALTH_OK', u'kb_used': 11028248, u'store_stats': {u'bytes_total': 4068567, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544263}}, {u'last_updated': u'2014-04-01 06:04:43.042086', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437645808, u'health': u'HEALTH_OK', u'kb_used': 10683176, u'store_stats': {u'bytes_total': 4068568, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544264}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'23 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 21883/46971 objects degraded (46.588%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 15270691, u'degraded_objects': 21883, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 65594722660, u'degraded_total': 46971, u'bytes_used': 77303758848, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 89927402, u'op_per_sec': 21, u'version': 1034, u'pgs_by_state': [{u'count': 10, u'state_name': u'active+recovery_wait'}, {u'count': 155, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'46.588', u'bytes_avail': 2258209103872}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:05:27,460.460 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:05:27,713.713 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1684 16 13572 13556 32.1948 5 6.28062 1.9834 2014-04-01 15:05:28,715.715 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1685 16 13605 13589 32.254 132 0.452814 1.98398 2014-04-01 15:05:29,713.713 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1686 16 13617 13601 32.2633 48 0.172348 1.98278 2014-04-01 15:05:30,713.713 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1687 16 13618 13602 32.2466 4 0.591349 1.98268 2014-04-01 15:05:31,316.316 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:05:31,713.713 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1688 16 13618 13602 32.2275 0 - 1.98268 2014-04-01 15:05:32,713.713 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1689 16 13618 13602 32.2084 0 - 1.98268 2014-04-01 15:05:33,739.739 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1690 16 13634 13618 32.2272 21.3333 3.91066 1.98504 2014-04-01 15:05:34,784.784 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1691 16 13654 13638 32.2554 80 0.891435 1.9836 2014-04-01 15:05:35,238.238 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:05:35,715.715 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1692 16 13665 13649 32.2624 44 0.730089 1.98276 2014-04-01 15:05:36,114.114 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:05:36,733.733 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1693 16 13681 13665 32.2811 64 0.315266 1.98154 2014-04-01 15:05:37,562.562 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:05:37,756.756 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1694 16 13689 13673 32.2809 32 0.289512 1.98077 2014-04-01 15:05:38,411.411 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:04:42.671722', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437647528, u'health': u'HEALTH_OK', u'kb_used': 10681456, u'store_stats': {u'bytes_total': 4068682, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544378}}, {u'last_updated': u'2014-04-01 06:04:43.055608', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437300736, u'health': u'HEALTH_OK', u'kb_used': 11028248, u'store_stats': {u'bytes_total': 4068567, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544263}}, {u'last_updated': u'2014-04-01 06:04:43.042086', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437645808, u'health': u'HEALTH_OK', u'kb_used': 10683176, u'store_stats': {u'bytes_total': 4068568, u'bytes_log': 458752, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544264}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'23 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 21953/47175 objects degraded (46.535%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 11043865, u'degraded_objects': 21953, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 65879935332, u'degraded_total': 47175, u'bytes_used': 77766754304, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 42597767, u'op_per_sec': 10, u'version': 1040, u'pgs_by_state': [{u'count': 10, u'state_name': u'active+recovery_wait'}, {u'count': 155, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'46.535', u'bytes_avail': 2257746108416}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:05:38,411.411 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:05:38,777.777 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1695 16 13689 13673 32.2619 0 - 1.98077 2014-04-01 15:05:39,201.201 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:05:39,831.831 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1696 16 13689 13673 32.2429 0 - 1.98077 2014-04-01 15:05:40,714.714 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1697 16 13689 13673 32.2239 0 - 1.98077 2014-04-01 15:05:41,714.714 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1698 16 13692 13676 32.212 3 5.77657 1.98173 2014-04-01 15:05:42,714.714 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1699 16 13724 13708 32.2683 128 0.437186 1.98294 2014-04-01 15:05:43,589.589 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:05:43,715.715 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:05:43.632714min lat: 0.064365 max lat: 341.507 avg lat: 1.98168 2014-04-01 15:05:43,715.715 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:05:43,715.715 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1700 16 13736 13720 32.2776 48 0.831141 1.98168 2014-04-01 15:05:44,525.525 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:05:44,715.715 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1701 16 13736 13720 32.2586 0 - 1.98168 2014-04-01 15:05:45,445.445 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:05:45,715.715 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1702 16 13736 13720 32.2397 0 - 1.98168 2014-04-01 15:05:46,163.163 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:05:42.671994', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437569588, u'health': u'HEALTH_OK', u'kb_used': 10759396, u'store_stats': {u'bytes_total': 5641546, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544378}}, {u'last_updated': u'2014-04-01 06:05:43.055875', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437238132, u'health': u'HEALTH_OK', u'kb_used': 11090852, u'store_stats': {u'bytes_total': 6690007, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544263}}, {u'last_updated': u'2014-04-01 06:05:43.042335', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437568596, u'health': u'HEALTH_OK', u'kb_used': 10760388, u'store_stats': {u'bytes_total': 6690008, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544264}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'23 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 22062/47478 objects degraded (46.468%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 14404654, u'degraded_objects': 22062, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 66303560036, u'degraded_total': 47478, u'bytes_used': 78207492096, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 77425016, u'op_per_sec': 18, u'version': 1046, u'pgs_by_state': [{u'count': 10, u'state_name': u'active+recovery_wait'}, {u'count': 155, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'46.468', u'bytes_avail': 2257305370624}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:05:46,163.163 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:05:46,715.715 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1703 16 13736 13720 32.2207 0 - 1.98168 2014-04-01 15:05:46,870.870 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:05:47,715.715 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1704 16 13755 13739 32.2464 19 0.440024 1.98437 2014-04-01 15:05:48,716.716 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1705 16 13775 13759 32.2744 80 0.710509 1.98227 2014-04-01 15:05:49,716.716 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1706 16 13790 13774 32.2907 60 0.284307 1.98095 2014-04-01 15:05:50,716.716 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1707 16 13800 13784 32.2952 40 1.47212 1.98068 2014-04-01 15:05:50,794.794 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:05:51,521.521 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:05:51,716.716 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1708 16 13802 13786 32.2809 8 0.913622 1.98052 2014-04-01 15:05:52,451.451 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:05:52,716.716 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1709 16 13802 13786 32.2621 0 - 1.98052 2014-04-01 15:05:54,160.160 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1710 16 13802 13786 32.2432 0 - 1.98052 2014-04-01 15:05:54,160.160 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:05:42.671994', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437569588, u'health': u'HEALTH_OK', u'kb_used': 10759396, u'store_stats': {u'bytes_total': 5641546, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544378}}, {u'last_updated': u'2014-04-01 06:05:43.055875', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437238132, u'health': u'HEALTH_OK', u'kb_used': 11090852, u'store_stats': {u'bytes_total': 6690007, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544263}}, {u'last_updated': u'2014-04-01 06:05:43.042335', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437568596, u'health': u'HEALTH_OK', u'kb_used': 10760388, u'store_stats': {u'bytes_total': 6690008, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544264}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'23 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 22093/47580 objects degraded (46.433%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 8523913, u'degraded_objects': 22093, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 66446166372, u'degraded_total': 47580, u'bytes_used': 78439358464, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 52848261, u'op_per_sec': 12, u'version': 1049, u'pgs_by_state': [{u'count': 10, u'state_name': u'active+recovery_wait'}, {u'count': 155, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'46.433', u'bytes_avail': 2257073504256}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:05:54,161.161 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:05:54,717.717 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1711 16 13804 13788 32.229 2.66667 5.9007 1.98109 2014-04-01 15:05:54,893.893 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:05:55,717.717 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1712 16 13837 13821 32.2873 132 0.395609 1.98195 2014-04-01 15:05:56,722.722 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1713 16 13852 13836 32.3035 60 1.01509 1.98075 2014-04-01 15:05:57,716.716 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1714 16 13865 13849 32.315 52 1.02417 1.9798 2014-04-01 15:05:58,717.717 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1715 16 13867 13851 32.3008 8 1.2242 1.97977 2014-04-01 15:05:58,845.845 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:05:59,582.582 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:05:59,739.739 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1716 16 13867 13851 32.2819 0 - 1.97977 2014-04-01 15:06:00,595.595 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:06:00,747.747 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1717 16 13867 13851 32.2631 0 - 1.97977 2014-04-01 15:06:01,302.302 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:05:42.671994', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437569588, u'health': u'HEALTH_OK', u'kb_used': 10759396, u'store_stats': {u'bytes_total': 5641546, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544378}}, {u'last_updated': u'2014-04-01 06:05:43.055875', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437238132, u'health': u'HEALTH_OK', u'kb_used': 11090852, u'store_stats': {u'bytes_total': 6690007, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544263}}, {u'last_updated': u'2014-04-01 06:05:43.042335', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437568596, u'health': u'HEALTH_OK', u'kb_used': 10760388, u'store_stats': {u'bytes_total': 6690008, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544264}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'23 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 22201/47862 objects degraded (46.385%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 14503382, u'degraded_objects': 22201, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 66840430948, u'degraded_total': 47862, u'bytes_used': 81078816768, u'recovering_objects_per_sec': 3, u'write_bytes_sec': 110588294, u'op_per_sec': 26, u'version': 1055, u'pgs_by_state': [{u'count': 10, u'state_name': u'active+recovery_wait'}, {u'count': 155, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'46.385', u'bytes_avail': 2254434045952}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:06:01,302.302 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:06:01,732.732 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1718 16 13867 13851 32.2444 0 - 1.97977 2014-04-01 15:06:02,152.152 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:06:02,859.859 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1719 16 13884 13868 32.2652 17 0.516434 1.98286 2014-04-01 15:06:03,732.732 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:06:03.635317min lat: 0.064365 max lat: 341.507 avg lat: 1.98032 2014-04-01 15:06:03,732.732 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:06:03,732.732 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1720 16 13910 13894 32.3069 104 0.962071 1.98032 2014-04-01 15:06:04,727.727 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1721 16 13920 13904 32.3113 40 0.298699 1.97949 2014-04-01 15:06:05,732.732 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1722 16 13927 13911 32.3088 28 0.788879 1.9792 2014-04-01 15:06:06,221.221 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:06:06,733.733 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1723 16 13933 13917 32.304 24 1.69577 1.97934 2014-04-01 15:06:06,938.938 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:06:07,737.737 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1724 16 13936 13920 32.2922 12 2.10434 1.97932 2014-04-01 15:06:08,330.330 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:06:08,728.728 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1725 16 13936 13920 32.2735 0 - 1.97932 2014-04-01 15:06:09,138.138 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:05:42.671994', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437569588, u'health': u'HEALTH_OK', u'kb_used': 10759396, u'store_stats': {u'bytes_total': 5641546, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544378}}, {u'last_updated': u'2014-04-01 06:05:43.055875', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437238132, u'health': u'HEALTH_OK', u'kb_used': 11090852, u'store_stats': {u'bytes_total': 6690007, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544263}}, {u'last_updated': u'2014-04-01 06:05:43.042335', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437568596, u'health': u'HEALTH_OK', u'kb_used': 10760388, u'store_stats': {u'bytes_total': 6690008, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544264}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'23 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 22259/47991 objects degraded (46.382%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 22259, u'num_pgs': 178, u'data_bytes': 67020786020, u'degraded_total': 47991, u'bytes_used': 81350754304, u'op_per_sec': 4, u'write_bytes_sec': 20920497, u'version': 1059, u'pgs_by_state': [{u'count': 10, u'state_name': u'active+recovery_wait'}, {u'count': 155, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'46.382', u'bytes_avail': 2254162108416}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:06:09,138.138 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:06:09,728.728 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1726 16 13936 13920 32.2548 0 - 1.97932 2014-04-01 15:06:10,212.212 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:06:10,720.720 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1727 16 13961 13945 32.294 33.3333 0.381695 1.98156 2014-04-01 15:06:11,755.755 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1728 16 13974 13958 32.3054 52 1.05181 1.98022 2014-04-01 15:06:12,719.719 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1729 16 13988 13972 32.3191 56 0.397301 1.97942 2014-04-01 15:06:13,719.719 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1730 16 13989 13973 32.3027 4 0.913483 1.97935 2014-04-01 15:06:14,265.265 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:06:14,731.731 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1731 16 13989 13973 32.2841 0 - 1.97935 2014-04-01 15:06:15,107.107 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:06:15,742.742 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1732 16 13989 13973 32.2654 0 - 1.97935 2014-04-01 15:06:16,327.327 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:06:16,730.730 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1733 16 13989 13973 32.2468 0 - 1.97935 2014-04-01 15:06:17,302.302 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:05:42.671994', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437569588, u'health': u'HEALTH_OK', u'kb_used': 10759396, u'store_stats': {u'bytes_total': 5641546, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544378}}, {u'last_updated': u'2014-04-01 06:05:43.055875', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437238132, u'health': u'HEALTH_OK', u'kb_used': 11090852, u'store_stats': {u'bytes_total': 6690007, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544263}}, {u'last_updated': u'2014-04-01 06:05:43.042335', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437568596, u'health': u'HEALTH_OK', u'kb_used': 10760388, u'store_stats': {u'bytes_total': 6690008, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544264}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'23 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 22356/48234 objects degraded (46.349%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 7327402, u'degraded_objects': 22356, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 67360524644, u'degraded_total': 48234, u'bytes_used': 81691623424, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 106247333, u'op_per_sec': 25, u'version': 1064, u'pgs_by_state': [{u'count': 10, u'state_name': u'active+recovery_wait'}, {u'count': 155, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'46.349', u'bytes_avail': 2253821239296}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:06:17,302.302 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:06:17,723.723 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1734 16 13993 13977 32.2375 4 5.60646 1.98056 2014-04-01 15:06:18,119.119 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:06:18,723.723 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1735 16 14020 14004 32.2811 108 0.202449 1.98207 2014-04-01 15:06:19,748.748 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1736 16 14038 14022 32.304 72 0.771887 1.98051 2014-04-01 15:06:20,758.758 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1737 16 14052 14036 32.3176 56 1.72284 1.9797 2014-04-01 15:06:21,749.749 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1738 16 14062 14046 32.322 40 0.664367 1.97907 2014-04-01 15:06:22,238.238 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:06:22,720.720 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1739 16 14065 14049 32.3104 12 0.72266 1.9788 2014-04-01 15:06:23,199.199 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:06:23,729.729 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:06:23.638151min lat: 0.064365 max lat: 341.507 avg lat: 1.9788 2014-04-01 15:06:23,729.729 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:06:23,729.729 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1740 16 14065 14049 32.2918 0 - 1.9788 2014-04-01 15:06:24,728.728 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1741 16 14065 14049 32.2732 0 - 1.9788 2014-04-01 15:06:24,973.973 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:06:25,822.822 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1742 16 14072 14056 32.2708 9.33333 5.55938 1.98043 2014-04-01 15:06:25,822.822 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:05:42.671994', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437569588, u'health': u'HEALTH_OK', u'kb_used': 10759396, u'store_stats': {u'bytes_total': 5641546, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544378}}, {u'last_updated': u'2014-04-01 06:05:43.055875', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437238132, u'health': u'HEALTH_OK', u'kb_used': 11090852, u'store_stats': {u'bytes_total': 6690007, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544263}}, {u'last_updated': u'2014-04-01 06:05:43.042335', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437568596, u'health': u'HEALTH_OK', u'kb_used': 10760388, u'store_stats': {u'bytes_total': 6690008, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544264}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'23 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 22401/48432 objects degraded (46.252%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 7877204, u'degraded_objects': 22401, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 67637348708, u'degraded_total': 48432, u'bytes_used': 82154545152, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 39386020, u'op_per_sec': 9, u'version': 1069, u'pgs_by_state': [{u'count': 10, u'state_name': u'active+recovery_wait'}, {u'count': 155, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'46.252', u'bytes_avail': 2253358317568}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:06:25,822.822 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:06:26,725.725 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1743 16 14100 14084 32.3165 112 0.719395 1.97999 2014-04-01 15:06:27,090.090 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:06:27,735.735 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1744 16 14115 14099 32.3324 60 1.14103 1.97893 2014-04-01 15:06:28,722.722 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1745 16 14116 14100 32.3162 4 0.460794 1.97882 2014-04-01 15:06:29,721.721 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1746 16 14116 14100 32.2976 0 - 1.97882 2014-04-01 15:06:30,721.721 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1747 16 14116 14100 32.2792 0 - 1.97882 2014-04-01 15:06:31,454.454 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:06:31,721.721 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1748 16 14126 14110 32.2836 13.3333 4.31566 1.98051 2014-04-01 15:06:32,539.539 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:06:33,661.661 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1749 16 14147 14131 32.3131 84 0.673418 1.97985 2014-04-01 15:06:34,359.359 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1750 16 14167 14151 32.3404 80 1.47921 1.97834 2014-04-01 15:06:35,030.030 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1751 16 14167 14151 32.3219 0 - 1.97834 2014-04-01 15:06:35,281.281 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:06:35,722.722 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1752 16 14168 14152 32.3057 2 1.68741 1.97832 2014-04-01 15:06:36,044.044 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:05:42.671994', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437569588, u'health': u'HEALTH_OK', u'kb_used': 10759396, u'store_stats': {u'bytes_total': 5641546, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544378}}, {u'last_updated': u'2014-04-01 06:05:43.055875', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437238132, u'health': u'HEALTH_OK', u'kb_used': 11090852, u'store_stats': {u'bytes_total': 6690007, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544263}}, {u'last_updated': u'2014-04-01 06:05:43.042335', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437568596, u'health': u'HEALTH_OK', u'kb_used': 10760388, u'store_stats': {u'bytes_total': 6690008, u'bytes_log': 3080192, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544264}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'23 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 22544/48753 objects degraded (46.241%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 9233986, u'degraded_objects': 22544, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 68086139236, u'degraded_total': 48753, u'bytes_used': 82710315008, u'recovering_objects_per_sec': 2, u'write_bytes_sec': 86799476, u'op_per_sec': 20, u'version': 1076, u'pgs_by_state': [{u'count': 10, u'state_name': u'active+recovery_wait'}, {u'count': 155, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'46.241', u'bytes_avail': 2252802547712}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:06:36,045.045 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:06:36,722.722 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1753 16 14168 14152 32.2873 0 - 1.97832 2014-04-01 15:06:36,759.759 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:06:37,722.722 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1754 16 14168 14152 32.2689 0 - 1.97832 2014-04-01 15:06:38,722.722 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1755 16 14168 14152 32.2505 0 - 1.97832 2014-04-01 15:06:39,725.725 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1756 16 14200 14184 32.305 32 0.577521 1.98093 2014-04-01 15:06:40,723.723 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1757 16 14218 14202 32.3276 72 0.957492 1.97912 2014-04-01 15:06:40,744.744 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:06:41,560.560 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:06:41,754.754 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1758 16 14226 14210 32.3274 32 0.813035 1.97863 2014-04-01 15:06:42,522.522 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:06:42,723.723 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1759 16 14233 14217 32.325 28 0.405604 1.97808 2014-04-01 15:06:43,240.240 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 12, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.053336', u'skew': u'-0.116514', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.053098', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:06:42.672214', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437485700, u'health': u'HEALTH_OK', u'kb_used': 10843284, u'store_stats': {u'bytes_total': 7738698, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544378}}, {u'last_updated': u'2014-04-01 06:06:43.056151', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437170552, u'health': u'HEALTH_OK', u'kb_used': 11158432, u'store_stats': {u'bytes_total': 9835735, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544263}}, {u'last_updated': u'2014-04-01 06:06:43.042591', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437485608, u'health': u'HEALTH_OK', u'kb_used': 10843376, u'store_stats': {u'bytes_total': 9835736, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544264}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'23 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 22535/48795 objects degraded (46.183%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 19894220, u'degraded_objects': 22535, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 68144859492, u'degraded_total': 48795, u'bytes_used': 82878705664, u'recovering_objects_per_sec': 4, u'write_bytes_sec': 21702786, u'op_per_sec': 5, u'version': 1079, u'pgs_by_state': [{u'count': 10, u'state_name': u'active+recovery_wait'}, {u'count': 155, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'46.183', u'bytes_avail': 2252634157056}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:06:43,241.241 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:06:43,723.723 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:06:43.640718min lat: 0.064365 max lat: 341.507 avg lat: 1.97808 2014-04-01 15:06:43,723.723 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:06:43,723.723 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1760 16 14233 14217 32.3066 0 - 1.97808 2014-04-01 15:06:43,940.940 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:06:44,724.724 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1761 16 14233 14217 32.2883 0 - 1.97808 2014-04-01 15:06:45,723.723 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1762 16 14233 14217 32.2699 0 - 1.97808 2014-04-01 15:06:46,725.725 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1763 16 14246 14230 32.2811 13 4.91113 1.98146 2014-04-01 15:06:47,723.723 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1764 16 14271 14255 32.3195 100 0.549303 1.97973 2014-04-01 15:06:47,844.844 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:06:48,458.458 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:06:48,766.766 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1765 16 14283 14267 32.3284 48 1.26909 1.97877 2014-04-01 15:06:49,539.539 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:06:49,725.725 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1766 16 14294 14278 32.335 44 0.828386 1.97819 2014-04-01 15:06:50,266.266 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.042926', u'skew': u'-0.093227', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.042614', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:06:42.672214', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437485700, u'health': u'HEALTH_OK', u'kb_used': 10843284, u'store_stats': {u'bytes_total': 7738698, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544378}}, {u'last_updated': u'2014-04-01 06:06:43.056151', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437170552, u'health': u'HEALTH_OK', u'kb_used': 11158432, u'store_stats': {u'bytes_total': 9835735, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544263}}, {u'last_updated': u'2014-04-01 06:06:43.042591', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437485608, u'health': u'HEALTH_OK', u'kb_used': 10843376, u'store_stats': {u'bytes_total': 9835736, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544264}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'23 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 22653/49077 objects degraded (46.158%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 4155861, u'degraded_objects': 22653, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 68539124068, u'degraded_total': 49077, u'bytes_used': 83260616704, u'recovering_objects_per_sec': 0, u'write_bytes_sec': 39480681, u'op_per_sec': 9, u'version': 1084, u'pgs_by_state': [{u'count': 10, u'state_name': u'active+recovery_wait'}, {u'count': 155, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'46.158', u'bytes_avail': 2252252246016}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:06:50,267.267 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:06:50,726.726 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1767 16 14294 14278 32.3167 0 - 1.97819 2014-04-01 15:06:50,968.968 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:06:51,725.725 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1768 16 14294 14278 32.2984 0 - 1.97819 2014-04-01 15:06:52,725.725 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1769 16 14294 14278 32.2802 0 - 1.97819 2014-04-01 15:06:53,725.725 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1770 16 14294 14278 32.2619 0 - 1.97819 2014-04-01 15:06:54,725.725 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1771 16 14310 14294 32.2798 12.8 5.02127 1.98199 2014-04-01 15:06:54,883.883 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:06:55,496.496 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:06:55,836.836 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1772 16 14341 14325 32.3316 124 0.365614 1.97916 2014-04-01 15:06:56,727.727 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1773 16 14351 14335 32.3359 40 1.29328 1.97842 2014-04-01 15:06:56,822.822 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:06:57,600.600 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.042926', u'skew': u'-0.093227', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.042614', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:06:42.672214', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437485700, u'health': u'HEALTH_OK', u'kb_used': 10843284, u'store_stats': {u'bytes_total': 7738698, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544378}}, {u'last_updated': u'2014-04-01 06:06:43.056151', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437170552, u'health': u'HEALTH_OK', u'kb_used': 11158432, u'store_stats': {u'bytes_total': 9835735, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544263}}, {u'last_updated': u'2014-04-01 06:06:43.042591', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437485608, u'health': u'HEALTH_OK', u'kb_used': 10843376, u'store_stats': {u'bytes_total': 9835736, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544264}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'23 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 22674/49161 objects degraded (46.122%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 17045193, u'degraded_objects': 22674, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 68656564580, u'degraded_total': 49161, u'bytes_used': 83641049088, u'recovering_objects_per_sec': 4, u'write_bytes_sec': 30681348, u'op_per_sec': 7, u'version': 1088, u'pgs_by_state': [{u'count': 10, u'state_name': u'active+recovery_wait'}, {u'count': 155, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'46.122', u'bytes_avail': 2251871813632}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:06:57,601.601 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:06:57,725.725 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1774 16 14365 14349 32.3492 56 0.692826 1.97752 2014-04-01 15:06:58,302.302 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:06:58,726.726 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1775 16 14366 14350 32.3333 4 0.707197 1.97743 2014-04-01 15:06:59,726.726 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1776 16 14366 14350 32.3151 0 - 1.97743 2014-04-01 15:07:00,727.727 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1777 16 14366 14350 32.2969 0 - 1.97743 2014-04-01 15:07:01,727.727 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1778 16 14382 14366 32.3147 21.3333 3.41832 1.98016 2014-04-01 15:07:02,225.225 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:07:02,727.727 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1779 16 14398 14382 32.3325 64 0.525146 1.97878 2014-04-01 15:07:03,006.006 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:07:03,728.728 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:07:03.643391min lat: 0.064365 max lat: 341.507 avg lat: 1.97773 2014-04-01 15:07:03,728.728 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:07:03,728.728 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1780 16 14413 14397 32.3481 60 1.4569 1.97773 2014-04-01 15:07:04,082.082 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:07:04,727.727 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1781 16 14414 14398 32.3321 4 0.783683 1.97765 2014-04-01 15:07:04,844.844 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.042926', u'skew': u'-0.093227', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.042614', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:06:42.672214', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437485700, u'health': u'HEALTH_OK', u'kb_used': 10843284, u'store_stats': {u'bytes_total': 7738698, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544378}}, {u'last_updated': u'2014-04-01 06:06:43.056151', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437170552, u'health': u'HEALTH_OK', u'kb_used': 11158432, u'store_stats': {u'bytes_total': 9835735, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544263}}, {u'last_updated': u'2014-04-01 06:06:43.042591', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437485608, u'health': u'HEALTH_OK', u'kb_used': 10843376, u'store_stats': {u'bytes_total': 9835736, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544264}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'23 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 22787/49449 objects degraded (46.082%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 5726279, u'degraded_objects': 22787, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 69059217764, u'degraded_total': 49449, u'bytes_used': 84108099584, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 32067166, u'op_per_sec': 7, u'version': 1093, u'pgs_by_state': [{u'count': 10, u'state_name': u'active+recovery_wait'}, {u'count': 155, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'46.082', u'bytes_avail': 2251404763136}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:07:04,845.845 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:07:05,538.538 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:07:05,728.728 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1782 16 14414 14398 32.314 0 - 1.97765 2014-04-01 15:07:06,727.727 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1783 16 14414 14398 32.2959 0 - 1.97765 2014-04-01 15:07:07,729.729 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1784 16 14414 14398 32.2778 0 - 1.97765 2014-04-01 15:07:08,727.727 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1785 16 14414 14398 32.2597 0 - 1.97765 2014-04-01 15:07:09,615.615 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:07:09,730.730 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1786 16 14448 14432 32.3178 27.2 0.632602 1.98013 2014-04-01 15:07:10,520.520 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:07:10,830.830 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1787 16 14465 14449 32.3377 68 0.987532 1.97846 2014-04-01 15:07:11,471.471 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:07:11,728.728 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1788 16 14479 14463 32.3509 56 1.61976 1.97757 2014-04-01 15:07:12,248.248 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.042926', u'skew': u'-0.093227', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.042614', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:06:42.672214', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437485700, u'health': u'HEALTH_OK', u'kb_used': 10843284, u'store_stats': {u'bytes_total': 7738698, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544378}}, {u'last_updated': u'2014-04-01 06:06:43.056151', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437170552, u'health': u'HEALTH_OK', u'kb_used': 11158432, u'store_stats': {u'bytes_total': 9835735, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544263}}, {u'last_updated': u'2014-04-01 06:06:43.042591', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437485608, u'health': u'HEALTH_OK', u'kb_used': 10843376, u'store_stats': {u'bytes_total': 9835736, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544264}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'23 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 22814/49542 objects degraded (46.050%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'recovering_bytes_per_sec': 7307324, u'degraded_objects': 22814, u'num_pgs': 178, u'recovering_keys_per_sec': 0, u'data_bytes': 69189241188, u'degraded_total': 49542, u'bytes_used': 84298776576, u'recovering_objects_per_sec': 1, u'write_bytes_sec': 34709793, u'op_per_sec': 8, u'version': 1097, u'pgs_by_state': [{u'count': 10, u'state_name': u'active+recovery_wait'}, {u'count': 155, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'46.050', u'bytes_avail': 2251214086144}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:07:12,249.249 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:07:12,730.730 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1789 16 14484 14468 32.344 20 1.23976 1.97727 2014-04-01 15:07:13,017.017 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:07:13,930.930 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1790 16 14487 14471 32.3327 12 0.31211 1.97703 2014-04-01 15:07:14,728.728 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1791 16 14487 14471 32.3146 0 - 1.97703 2014-04-01 15:07:15,728.728 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1792 16 14487 14471 32.2966 0 - 1.97703 2014-04-01 15:07:16,733.733 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1793 16 14508 14492 32.3254 28 3.59029 1.97967 2014-04-01 15:07:16,934.934 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:07:17,800.800 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:07:17,960.960 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1794 16 14522 14506 32.3386 56 0.771301 1.97831 2014-04-01 15:07:18,724.724 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-04-01 15:07:18,730.730 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1795 16 14530 14514 32.3384 32 0.281145 1.97772 2014-04-01 15:07:19,638.638 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'a', u'rank': 0, u'addr': u'10.214.133.37:6789/0'}, {u'name': u'b', u'rank': 1, u'addr': u'10.214.133.38:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.133.37:6790/0'}], u'modified': u'2014-04-01 05:35:09.229885', u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf', u'created': u'2014-04-01 05:35:09.229885'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 14, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.042926', u'skew': u'-0.093227', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.042614', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-04-01 06:06:42.672214', u'name': u'a', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437485700, u'health': u'HEALTH_OK', u'kb_used': 10843284, u'store_stats': {u'bytes_total': 7738698, u'bytes_log': 4128768, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544378}}, {u'last_updated': u'2014-04-01 06:06:43.056151', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437170552, u'health': u'HEALTH_OK', u'kb_used': 11158432, u'store_stats': {u'bytes_total': 9835735, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544263}}, {u'last_updated': u'2014-04-01 06:06:43.042591', u'name': u'c', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 437485608, u'health': u'HEALTH_OK', u'kb_used': 10843376, u'store_stats': {u'bytes_total': 9835736, u'bytes_log': 6225920, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 3544264}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'13 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'10 pgs recovery_wait'}, {u'severity': u'HEALTH_WARN', u'summary': u'23 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 22888/49722 objects degraded (46.032%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 74 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 44 > pgp_num 34'}]}, u'pgmap': {u'bytes_total': 2460623708160, u'degraded_objects': 22888, u'num_pgs': 178, u'data_bytes': 69440899428, u'degraded_total': 49722, u'bytes_used': 84666044416, u'op_per_sec': 3, u'write_bytes_sec': 13327746, u'version': 1101, u'pgs_by_state': [{u'count': 10, u'state_name': u'active+recovery_wait'}, {u'count': 155, u'state_name': u'active+clean'}, {u'count': 13, u'state_name': u'active+recovering'}], u'degraded_ratio': u'46.032', u'bytes_avail': 2250846818304}, u'quorum_names': [u'a', u'b', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 130, u'num_in_osds': 5}}, u'fsid': u'4fc4b977-b888-4422-ba3c-54f89f334abf'} 2014-04-01 15:07:19,639.639 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:07:19,729.729 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1796 16 14541 14525 32.3449 44 0.189717 1.97733 2014-04-01 15:07:20,400.400 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:07:20,729.729 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1797 16 14541 14525 32.3269 0 - 1.97733 2014-04-01 15:07:21,729.729 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1798 16 14541 14525 32.3089 0 - 1.97733 2014-04-01 15:07:22,730.730 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1799 16 14541 14525 32.291 0 - 1.97733 2014-04-01 15:07:23,730.730 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 06:07:23.646225min lat: 0.064365 max lat: 341.507 avg lat: 1.97759 2014-04-01 15:07:23,730.730 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 2014-04-01 15:07:23,731.731 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 1800 16 14542 14526 32.2753 1 5.73665 1.97759 2014-04-01 15:07:24,481.481 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-04-01 15:07:24,730.730 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: Total time run: 1800.602111 2014-04-01 15:07:24,730.730 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: Total writes made: 14542 2014-04-01 15:07:24,731.731 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: Write size: 4194304 2014-04-01 15:07:24,731.731 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: Bandwidth (MB/sec): 32.305 2014-04-01 15:07:24,731.731 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: 2014-04-01 15:07:24,731.731 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: Stddev Bandwidth: 33.8368 2014-04-01 15:07:24,731.731 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: Max bandwidth (MB/sec): 164 2014-04-01 15:07:24,731.731 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: Min bandwidth (MB/sec): 0 2014-04-01 15:07:24,732.732 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: Average Latency: 1.98109 2014-04-01 15:07:24,732.732 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: Stddev Latency: 4.84059 2014-04-01 15:07:24,732.732 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: Max latency: 341.507 2014-04-01 15:07:24,732.732 INFO:teuthology.task.radosbench.radosbench.0.out:[10.214.133.37]: Min latency: 0.064365 2014-04-01 15:07:25,283.283 INFO:teuthology.orchestra.run.err:[10.214.133.37]: dumped all in format json 2014-04-01 15:07:26,232.232 DEBUG:teuthology.orchestra.run:Running [10.214.133.37]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=js