Infrastructure

It seems on all releases luminous, mimic and nautilus

Runs:
http://pulpito.ceph.com/yuriw-2019-11-08_14:54:42-smoke-nautilus-distro-basic-smithi/
http://pulpito.ceph.com/yuriw-2019-11-08_14:54:32-smoke-mimic-distro-basic-smithi/
http://pulpito.ceph.com/yuriw-2019-11-08_14:54:19-smoke-luminous-distro-basic-smithi/

Jobs: see failed and dead `SSH connection to smithi081 was lost: 'sudo rm rf - /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0'`

Log: http://qa-proxy.ceph.com/teuthology/yuriw-2019-11-08_14:54:42-smoke-nautilus-distro-basic-smithi/4484163/teuthology.log

2019-11-08T15:50:17.214 INFO:teuthology.orchestra.run.smithi047:> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dd7922d9362fe3f8587e38f50383b76b1fbbee77 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io/test_short_dio_read
2019-11-08T15:50:17.267 INFO:tasks.workunit.client.0.smithi047.stdout:writing first 3 bytes of 10k file
2019-11-08T15:50:17.268 INFO:tasks.workunit.client.0.smithi047.stdout:reading O_DIRECT
2019-11-08T15:50:17.268 INFO:tasks.workunit.client.0.smithi047.stdout:got 10000
2019-11-08T15:50:17.269 INFO:teuthology.orchestra.run:Running command with timeout 3600
2019-11-08T15:50:17.269 INFO:teuthology.orchestra.run.smithi047:Running:
2019-11-08T15:50:17.269 INFO:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp
2019-11-08T15:50:17.369 INFO:tasks.workunit:Running workunit direct_io/test_sync_io...
2019-11-08T15:50:17.369 INFO:teuthology.orchestra.run.smithi047:Running (workunit test direct_io/test_sync_io):
2019-11-08T15:50:17.369 INFO:teuthology.orchestra.run.smithi047:> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dd7922d9362fe3f8587e38f50383b76b1fbbee77 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io/test_sync_io
2019-11-08T15:50:18.561 INFO:tasks.workunit.client.0.smithi047.stdout:writing pattern
2019-11-08T15:50:18.561 INFO:tasks.workunit.client.0.smithi047.stdout:read_direct buf_align 0 offset 4190208 len 1024
2019-11-08T15:50:18.562 INFO:tasks.workunit.client.0.smithi047.stdout:read_file buf_align 0 offset 4190208 len 1024
2019-11-08T15:50:18.562 INFO:tasks.workunit.client.0.smithi047.stdout:read_sync buf_align 0 offset 4190208 len 1024
2019-11-08T15:50:18.562 INFO:tasks.workunit.client.0.smithi047.stdout:read_file buf_align 0 offset 4190208 len 1024
2019-11-08T15:50:18.562 INFO:tasks.workunit.client.0.smithi047.stdout:read_direct buf_align 0 offset 4190208 len 2048
..........
2019-11-08T16:06:16.002 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi047.front.sepia.ceph.com', 'timeout': 60}
2019-11-08T16:06:16.004 DEBUG:tasks.ceph:Missed logrotate, host unreachable
2019-11-08T16:06:19.067 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.47
2019-11-08T16:06:19.067 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 86, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/github.com_ceph_ceph_nautilus/qa/tasks/workunit.py", line 136, in task
    cleanup=cleanup)
  File "/home/teuthworker/src/github.com_ceph_ceph_nautilus/qa/tasks/workunit.py", line 286, in _spawn_on_all_clients
    timeout=timeout)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 85, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 99, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 22, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/github.com_ceph_ceph_nautilus/qa/tasks/workunit.py", line 420, in _run_tests
    args=args,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 205, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 420, in run
    raise ConnectionLostError(command=quote(args), node=name)
ConnectionLostError: SSH connection to smithi047 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0'