Ceph » RADOS

Run: http://pulpito.ceph.com/teuthology-2017-09-15_17:30:33-upgrade:luminous-x-master-distro-basic-smithi/
Jobs: many
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2017-09-15_17:30:33-upgrade:luminous-x-master-distro-basic-smithi/1636248/teuthology.log

2017-09-15T17:40:55.564 INFO:teuthology.orchestra.run.smithi100:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2'
2017-09-15T17:40:55.565 INFO:tasks.ceph.osd.1.smithi100.stdout:starting osd.1 at - osd_data /var/lib/ceph/osd/ceph-1 /var/lib/ceph/osd/ceph-1/journal
2017-09-15T17:40:55.567 INFO:tasks.ceph.osd.1.smithi100.stderr:2017-09-15 17:40:55.569051 7f74c94631c0 -1 WARNING: all dangerous and experimental features are enabled.
2017-09-15T17:40:55.567 INFO:tasks.ceph.osd.2:Started
2017-09-15T17:40:55.567 INFO:teuthology.orchestra.run.smithi100:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 0'
2017-09-15T17:40:55.612 INFO:tasks.ceph.osd.2.smithi100.stderr:2017-09-15 17:40:55.611529 7fb15f2ab1c0 -1 WARNING: all dangerous and experimental features are enabled.
2017-09-15T17:40:55.612 INFO:tasks.ceph.osd.2.smithi100.stderr:2017-09-15 17:40:55.611674 7fb15f2ab1c0 -1 WARNING: all dangerous and experimental features are enabled.
2017-09-15T17:40:55.612 INFO:tasks.ceph.osd.2.smithi100.stdout:starting osd.2 at - osd_data /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal
2017-09-15T17:40:55.612 INFO:tasks.ceph.osd.2.smithi100.stderr:2017-09-15 17:40:55.614455 7fb15f2ab1c0 -1 WARNING: all dangerous and experimental features are enabled.
2017-09-15T17:40:55.639 INFO:teuthology.orchestra.run.smithi100.stderr:2017-09-15 17:40:55.641517 7f707df44700 -1 WARNING: all dangerous and experimental features are enabled.
2017-09-15T17:40:55.649 INFO:teuthology.orchestra.run.smithi100.stderr:2017-09-15 17:40:55.651011 7f707df44700 -1 WARNING: all dangerous and experimental features are enabled.
2017-09-15T17:40:55.829 INFO:tasks.ceph.osd.0.smithi100.stderr:2017-09-15 17:40:55.831611 7fcb0de8f1c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) fsck error: free extent 0x2000~2000 intersects allocated blocks
2017-09-15T17:40:55.894 INFO:tasks.ceph.osd.1.smithi100.stderr:2017-09-15 17:40:55.896752 7f74c94631c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) fsck error: free extent 0x2000~2000 intersects allocated blocks
2017-09-15T17:40:55.925 INFO:tasks.ceph.osd.2.smithi100.stderr:2017-09-15 17:40:55.927309 7fb15f2ab1c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) fsck error: free extent 0x2000~2000 intersects allocated blocks
2017-09-15T17:40:56.340 INFO:tasks.ceph.osd.0.smithi100.stderr:2017-09-15 17:40:56.341993 7fcb0de8f1c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _mount fsck found 1 errors
017-09-15T17:40:56.340 INFO:tasks.ceph.osd.0.smithi100.stderr:2017-09-15 17:40:56.342000 7fcb0de8f1c0 -1 osd.0 0 OSD:init: unable to mount object store
2017-09-15T17:40:56.340 INFO:tasks.ceph.osd.0.smithi100.stderr:2017-09-15 17:40:56.342007 7fcb0de8f1c0 -1 [0;31m ** ERROR: osd init failed: (5) Input/output error[0m
2017-09-15T17:40:56.412 INFO:tasks.ceph.osd.1.smithi100.stderr:2017-09-15 17:40:56.413972 7f74c94631c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _mount fsck found 1 errors
2017-09-15T17:40:56.412 INFO:tasks.ceph.osd.1.smithi100.stderr:2017-09-15 17:40:56.413980 7f74c94631c0 -1 osd.1 0 OSD:init: unable to mount object store
2017-09-15T17:40:56.412 INFO:tasks.ceph.osd.1.smithi100.stderr:2017-09-15 17:40:56.413987 7f74c94631c0 -1 [0;31m ** ERROR: osd init failed: (5) Input/output error[0m
2017-09-15T17:40:56.448 INFO:tasks.ceph.osd.2.smithi100.stderr:2017-09-15 17:40:56.449997 7fb15f2ab1c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _mount fsck found 1 errors
2017-09-15T17:40:56.448 INFO:tasks.ceph.osd.2.smithi100.stderr:2017-09-15 17:40:56.450004 7fb15f2ab1c0 -1 osd.2 0 OSD:init: unable to mount object store
2017-09-15T17:40:56.448 INFO:tasks.ceph.osd.2.smithi100.stderr:2017-09-15 17:40:56.450010 7fb15f2ab1c0 -1 [0;31m ** ERROR: osd init failed: (5) Input/output error[0m
2017-09-15T17:40:56.500 INFO:tasks.ceph.osd.0.smithi100.stderr:daemon-helper: command failed with exit status 1
2017-09-15T17:40:56.543 INFO:tasks.ceph.osd.1.smithi100.stderr:daemon-helper: command failed with exit status 1
2017-09-15T17:40:56.595 INFO:tasks.ceph.osd.2.smithi100.stderr:daemon-helper: command failed with exit status 1
2017-09-15T17:40:57.625 INFO:teuthology.orchestra.run.smithi100.stderr:marked down osd.0.
2017-09-15T17:40:57.642 INFO:teuthology.orchestra.run.smithi100:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 1'
2017-09-15T17:40:57.805 INFO:teuthology.orchestra.run.smithi100.stderr:2017-09-15 17:40:57.807445 7ff0677d5700 -1 WARNING: all dangerous and experimental features are enabled.
2017-09-15T17:40:57.825 INFO:teuthology.orchestra.run.smithi100.stderr:2017-09-15 17:40:57.827287 7ff0677d5700 -1 WARNING: all dangerous and experimental features are enabled.
2017-09-15T17:40:57.914 INFO:teuthology.orchestra.run.smithi100.stderr:osd.1 is already down.

<2017-09-15T17:40:57.924 INFO:teuthology.orchestra.run.smithi100:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 2'
2017-09-15T17:40:57.994 INFO:teuthology.orchestra.run.smithi100.stderr:2017-09-15 17:40:57.995836 7f45653c4700 -1 WARNING: all dangerous and experimental features are enabled.
2017-09-15T17:40:58.004 INFO:teuthology.orchestra.run.smithi100.stderr:2017-09-15 17:40:58.005996 7f45653c4700 -1 WARNING: all dangerous and experimental features are enabled.
2017-09-15T17:40:58.107 INFO:teuthology.orchestra.run.smithi100.stderr:osd.2 is already down.
2017-09-15T17:40:58.118 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean...
2017-09-15T17:40:58.118 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available
2017-09-15T17:40:58.118 INFO:teuthology.orchestra.run.smithi100:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json'
2017-09-15T17:40:58.211 INFO:teuthology.orchestra.run.smithi100.stderr:2017-09-15 17:40:58.213075 7fed8ec92700 -1 WARNING: all dangerous and experimental features are enabled.
2017-09-15T17:40:58.221 INFO:teuthology.orchestra.run.smithi100.stderr:2017-09-15 17:40:58.223046 7fed8ec92700 -1 WARNING: all dangerous and experimental features are enabled.
2017-09-15T17:40:58.352 INFO:teuthology.orchestra.run.smithi100.stdout:
2017-09-15T17:40:58.352 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":5,"active_gid":44097,"active_name":"x","active_addr":"-","available":false,"standbys":[],"modules":["restful","status"],"available_modules":["balancer","dashboard","prometheus","restful","status","zabbix"]}
2017-09-15T17:41:01.353 INFO:teuthology.orchestra.run.smithi100:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json'
2017-09-15T17:41:01.468 INFO:teuthology.orchestra.run.smithi100.stderr:2017-09-15 17:41:01.470538 7f2762a87700 -1 WARNING: all dangerous and experimental features are enabled.
2017-09-15T17:41:01.490 INFO:teuthology.orchestra.run.smithi100.stderr:2017-09-15 17:41:01.492581 7f2762a87700 -1 WARNING: all dangerous and experimental features are enabled.
2017-09-15T17:41:01.656 INFO:teuthology.orchestra.run.smithi100.stdout:
2017-09-15T17:41:01.656 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":6,"active_gid":44097,"active_name":"x","active_addr":"172.21.15.100:6800/124301","available":true,"standbys":[],"modules":["restful","status"],"available_modules":["balancer","dashboard","prometheus","restful","status","zabbix"]}
2017-09-15T17:41:01.656 INFO:tasks.ceph.ceph_manager.ceph:mgr available!
2017-09-15T17:41:01.657 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 89, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 1378, in restart
    healthy(ctx=ctx, config=dict(cluster=cluster))
  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 1246, in healthy
    ceph_cluster=cluster_name,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/misc.py", line 924, in wait_until_osds_up
    daemon.check_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/daemon.py", line 151, in check_status
    return self.proc.poll()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 205, in poll
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 177, in _raise_for_status
    node=self.hostname, label=self.label
CommandFailedError: Command failed on smithi100 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1'

/pre>