Project

General

Profile

Backport #18383

Updated by Nathan Cutler about 6 years ago

This is plaguing hammer 0.94.10 integration testing

http://qa-proxy.ceph.com/teuthology/smithfarm-2017-01-02_12:24:43-rados-wip-hammer-backports-distro-basic-smithi/683042/teuthology.log

<pre>
2017-01-02T12:28:31.814 INFO:teuthology.orchestra.run.smithi046:Running: 'sudo mkdir -p /var/lib/ceph/osd/ceph-0' 2017-01-02T12:31:42.789 INFO:teuthology.misc:trying to connect to ubuntu@smithi046.front.sepia.ceph.com
2017-01-02T12:28:31.830 INFO:tasks.ceph:{} 2017-01-02T12:31:42.792 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi046.front.sepia.ceph.com', 'timeout': 60}
2017-01-02T12:28:31.833 INFO:tasks.ceph:0 2017-01-02T12:32:41.826 DEBUG:teuthology.orchestra.remote:timed out
2017-01-02T12:28:31.836 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/nvme0n1p2 on 2017-01-02T12:32:41.829 DEBUG:teuthology.misc:waited 71.4516630173
2017-01-02T12:32:42.795 DEBUG:teuthology.orchestra.remote:timed out
2017-01-02T12:32:42.799 DEBUG:teuthology.misc:waited 71.1596288681
2017-01-02T12:32:42.832 INFO:teuthology.misc:trying to connect to ubuntu@smithi062.front.sepia.ceph.com
2017-01-02T12:32:42.843 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi062.front.sepia.ceph.com', 'timeout': 60}
2017-01-02T12:32:43.802 INFO:teuthology.misc:trying to connect to
ubuntu@smithi046.front.sepia.ceph.com
2017-01-02T12:28:31.838 INFO:teuthology.orchestra.run.smithi046:Running: 'yes | sudo mkfs.xfs -f -i size=2048 /dev/nvme0n1p2' 2017-01-02T12:32:43.805 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi046.front.sepia.ceph.com', 'timeout': 60}
2017-01-02T12:28:32.390 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/nvme0n1p2 isize=2048 agcount=4, agsize=6104832 blks 2017-01-02T12:33:42.897 DEBUG:teuthology.orchestra.remote:timed out
2017-01-02T12:28:32.393 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=0 2017-01-02T12:33:42.900 DEBUG:teuthology.misc:waited 132.523020983
2017-01-02T12:28:32.396 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=24419328, imaxpct=25 2017-01-02T12:33:43.809 DEBUG:teuthology.orchestra.remote:timed out
2017-01-02T12:28:32.399 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2017-01-02T12:33:43.812 DEBUG:teuthology.misc:waited 132.173267841
2017-01-02T12:28:32.402 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0 2017-01-02T12:33:43.903 INFO:teuthology.misc:trying to connect to ubuntu@smithi062.front.sepia.ceph.com
2017-01-02T12:28:32.404 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=11923, version=2 2017-01-02T12:33:43.907 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi062.front.sepia.ceph.com', 'timeout': 60}
2017-01-02T12:28:32.407 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2017-01-02T12:33:43.910 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.62
2017-01-02T12:28:32.410 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2017-01-02T12:33:43.913 DEBUG:teuthology.misc:waited 133.535454988
2017-01-02T12:28:32.413 INFO:tasks.ceph:mount /dev/nvme0n1p2 on 2017-01-02T12:33:44.816 INFO:teuthology.misc:trying to connect to ubuntu@smithi046.front.sepia.ceph.com -o noatime
2017-01-02T12:28:32.416 INFO:teuthology.orchestra.run.smithi046:Running: 'sudo mount -t xfs -o noatime /dev/nvme0n1p2 /var/lib/ceph/osd/ceph-0' 2017-01-02T12:33:44.819 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi046.front.sepia.ceph.com', 'timeout': 60}
2017-01-02T12:28:32.452 ERROR:teuthology.contextutil:Saw exception from nested tasks 2017-01-02T12:33:44.916 INFO:teuthology.misc:trying to connect to ubuntu@smithi062.front.sepia.ceph.com
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 30, in nested
vars.append(enter())
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-hammer-backports/qa/tasks/ceph.py", line 595, in cluster
'sudo', '/sbin/restorecon', mnt_point,
2017-01-02T12:33:44.922 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi062.front.sepia.ceph.com', 'timeout': 60}
NameError: global name 'mnt_point' is not defined 2017-01-02T12:33:45.229 INFO:teuthology.orchestra.run.smithi062:Running: 'true'
2017-01-02T12:33:45.415 DEBUG:teuthology.misc:waited 135.03777504
</pre>

Yet, somewhat later I can ssh into these two hosts. Maybe it just needs to wait longer?

Back