Bug #50569
Updated by Ramana Raja about 3 years ago
Description: fs/verify/{begin centos_latest clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress validater/valgrind}
See this repeatedly failing in nautilus Yuri's runs,
https://pulpito.ceph.com/yuriw-2021-04-20_21:38:51-fs-wip-yuri8-testing-2021-04-20-0734-nautilus-distro-basic-smithi/6060287/
https://pulpito.ceph.com/yuriw-2021-04-20_21:38:51-fs-wip-yuri8-testing-2021-04-20-0734-nautilus-distro-basic-smithi/6060336/
https://pulpito.ceph.com/yuriw-2021-04-21_16:09:02-fs-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062296/
https://pulpito.ceph.com/yuriw-2021-04-21_16:09:02-fs-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062345/
https://pulpito.ceph.com/yuriw-2021-04-21_16:09:02-fs-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062391/
I see the same "tasks/cfuse_workunit_suites_fsstress" pass when run one without "validater/valgrind"
Going through the teuthology.log of one of the runs, I find it hard to narrow down the failure. There are multiple Tracebacks.
http://qa-proxy.ceph.com/teuthology/yuriw-2021-04-21_16:09:02-fs-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062296/teuthology.log
<pre>
2021-04-22T11:53:10.562 INFO:tasks.ceph.mon.c.smithi112.stderr:==00:03:04:36.151 34897== Warning: unimplemented fcntl command: 1036
2021-04-22T11:53:10.573 INFO:tasks.ceph.mon.b.smithi112.stderr:==00:03:04:36.163 34899== Warning: unimplemented fcntl command: 1036
2021-04-22T11:53:10.623 INFO:tasks.ceph.mon.c.smithi112.stderr:==00:03:04:36.212 34897== Warning: unimplemented fcntl command: 1036
2021-04-22T11:53:12.800 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2021-04-22T11:53:12.804 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2021-04-22T11:53:42.923 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2021-04-22T11:53:42.928 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2021-04-22T11:54:13.062 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2021-04-22T11:54:13.065 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2021-04-22T11:54:43.169 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2021-04-22T11:54:43.173 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2021-04-22T11:55:13.286 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2021-04-22T11:55:13.314 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2021-04-22T11:55:21.052 ERROR:teuthology:Uncaught exception (Hub)
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr
out = self.in_stderr_buffer.read(nbytes, self.timeout)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read
raise PipeTimeout()
paramiko.buffered_pipe.PipeTimeout
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/run.py", line 323, in copy_file_to
copy_to_log(src, logger, capture=stream, quiet=quiet)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/run.py", line 276, in copy_to_log
for line in f:
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__
line = self.readline()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline
new_data = self._read(n)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read
return self.channel.recv_stderr(size)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr
raise socket.timeout()
socket.timeout
2021-04-22T11:55:21.061 ERROR:teuthology:Uncaught exception (Hub)
</pre>
And later in the log,
<pre>
021-04-22T12:04:36.778 INFO:tasks.ceph.mon.c.smithi112.stderr:==00:03:16:02.368 34897== Warning: unimplemented fcntl command: 1036
2021-04-22T12:04:36.800 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:03:16:02.273 35222== Warning: unimplemented fcntl command: 1036
2021-04-22T12:06:17.006 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2021-04-22T12:06:17.011 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2021-04-22T12:06:47.245 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2021-04-22T12:06:47.248 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2021-04-22T12:07:17.345 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2021-04-22T12:07:17.350 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2021-04-22T12:07:47.490 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2021-04-22T12:07:47.494 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2021-04-22T12:08:17.673 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2021-04-22T12:08:17.678 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2021-04-22T12:08:31.073 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0...
2021-04-22T12:08:31.074 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0
2021-04-22T12:08:31.459 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr
out = self.in_stderr_buffer.read(nbytes, self.timeout)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read
raise PipeTimeout()
paramiko.buffered_pipe.PipeTimeout
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/run_tasks.py", line 91, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/run_tasks.py", line 70, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/github.com_ceph_ceph-c_b0d014fe25986033c2db9422289a173f45eea553/qa/tasks/workunit.py", line 138, in task
cleanup=cleanup)
File "/home/teuthworker/src/github.com_ceph_ceph-c_b0d014fe25986033c2db9422289a173f45eea553/qa/tasks/workunit.py", line 288, in _spawn_on_all_clients
timeout=timeout)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/parallel.py", line 84, in __exit__
for result in self:
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/parallel.py", line 98, in __next__
resurrect_traceback(result)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/parallel.py", line 30, in resurrect_traceback
raise exc.exc_info[1]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/parallel.py", line 23, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/github.com_ceph_ceph-c_b0d014fe25986033c2db9422289a173f45eea553/qa/tasks/workunit.py", line 415, in _run_tests
remote.run(logger=log.getChild(role), args=args, timeout=(60*60))
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/remote.py", line 215, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/run.py", line 455, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/run.py", line 148, in wait
greenlet.get(block=True,timeout=60)
File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get
File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise
raise value.with_traceback(tb)
File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/run.py", line 323, in copy_file_to
copy_to_log(src, logger, capture=stream, quiet=quiet)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/run.py", line 276, in copy_to_log
for line in f:
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__
line = self.readline()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline
new_data = self._read(n)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read
return self.channel.recv_stderr(size)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr
raise socket.timeout()
socket.timeout
2021-04-22T12:08:31.573 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=5e645ce7410e49219e7ebffc525e2242
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr
out = self.in_stderr_buffer.read(nbytes, self.timeout)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read
raise PipeTimeout()
paramiko.buffered_pipe.PipeTimeout
</pre>