Project

General

Profile

Actions
Copy link

Bug #11986

closed

logs changing during tarball generation at end of job

Added by John Spray almost 9 years ago. Updated almost 7 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
Testing
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

http://pulpito.ceph.com/teuthology-2015-06-08_23:04:01-fs-master---basic-multi/926320/

2015-06-12T03:08:59.651 DEBUG:teuthology.misc:Transferring archived files from plana39:/var/log/ceph to /var/lib/teuthworker/archive/teuthology-2015-06-08_23:04:01-fs-master---basic-multi/926320/remote/plana39/log
2015-06-12T03:08:59.690 INFO:teuthology.orchestra.run.plana39:Running: "python -c 'import os; import tempfile; import sys;(fd,fname) = tempfile.mkstemp();os.close(fd);sys.stdout.write(fname.rstrip());sys.stdout.flush()'" 
2015-06-12T03:09:00.844 INFO:teuthology.orchestra.run.plana39:Running: 'sudo tar cz -f /tmp/tmpfc8lg3 -C /var/log/ceph -- .'
2015-06-12T03:10:34.531 INFO:teuthology.orchestra.run.plana39.stderr:tar: ./ceph-mds.a.log-20150612.gz: file changed as we read it
2015-06-12T03:10:43.569 INFO:teuthology.orchestra.run.plana39.stderr:tar: ./ceph-mds.a.log-20150612: File removed before we read it
2015-06-12T03:10:43.579 INFO:teuthology.orchestra.run.plana39.stderr:tar: .: file changed as we read it
2015-06-12T03:10:43.590 ERROR:teuthology.run_tasks:Manager failed: ceph
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 125, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/ceph.py", line 1360, in task
    osd_scrub_pgs(ctx, config)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 44, in nested
    if exit(*exc):
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/ceph.py", line 181, in ceph_log
    os.path.join(sub, 'log'))
  File "/home/teuthworker/src/teuthology_master/teuthology/misc.py", line 735, in pull_directory
    remote.get_tar(remotedir, local_tarfile, sudo=True)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 291, in get_tar
    self.run(args=args)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 156, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 378, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 114, in wait
    label=self.label)

Could be because the log rotation thread isn't waiting on completion of its RemoteProcess before completing.

Actions
Copy link

Also available in: Atom PDF