Project

General

Profile

Actions

Bug #11986

closed

logs changing during tarball generation at end of job

Added by John Spray almost 9 years ago. Updated almost 7 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
Testing
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

http://pulpito.ceph.com/teuthology-2015-06-08_23:04:01-fs-master---basic-multi/926320/

2015-06-12T03:08:59.651 DEBUG:teuthology.misc:Transferring archived files from plana39:/var/log/ceph to /var/lib/teuthworker/archive/teuthology-2015-06-08_23:04:01-fs-master---basic-multi/926320/remote/plana39/log
2015-06-12T03:08:59.690 INFO:teuthology.orchestra.run.plana39:Running: "python -c 'import os; import tempfile; import sys;(fd,fname) = tempfile.mkstemp();os.close(fd);sys.stdout.write(fname.rstrip());sys.stdout.flush()'" 
2015-06-12T03:09:00.844 INFO:teuthology.orchestra.run.plana39:Running: 'sudo tar cz -f /tmp/tmpfc8lg3 -C /var/log/ceph -- .'
2015-06-12T03:10:34.531 INFO:teuthology.orchestra.run.plana39.stderr:tar: ./ceph-mds.a.log-20150612.gz: file changed as we read it
2015-06-12T03:10:43.569 INFO:teuthology.orchestra.run.plana39.stderr:tar: ./ceph-mds.a.log-20150612: File removed before we read it
2015-06-12T03:10:43.579 INFO:teuthology.orchestra.run.plana39.stderr:tar: .: file changed as we read it
2015-06-12T03:10:43.590 ERROR:teuthology.run_tasks:Manager failed: ceph
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 125, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/ceph.py", line 1360, in task
    osd_scrub_pgs(ctx, config)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 44, in nested
    if exit(*exc):
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/ceph.py", line 181, in ceph_log
    os.path.join(sub, 'log'))
  File "/home/teuthworker/src/teuthology_master/teuthology/misc.py", line 735, in pull_directory
    remote.get_tar(remotedir, local_tarfile, sudo=True)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 291, in get_tar
    self.run(args=args)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 156, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 378, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 114, in wait
    label=self.label)

Could be because the log rotation thread isn't waiting on completion of its RemoteProcess before completing.

Actions #1

Updated by Greg Farnum almost 9 years ago

  • Subject changed from Log rotation interferes with transfer of logs at end of test to logs changing during tarball generation at end of job

Actually this job doesn't have log rotation enabled at all.

Actions #2

Updated by Patrick Donnelly almost 7 years ago

  • Status changed from New to Closed

Haven't seen this one recently. Closing.

Actions

Also available in: Atom PDF