Project

General

Profile

Actions
Copy link

Bug #20735

open

mds: stderr:gzip: /var/log/ceph/ceph-mds.f.log: file size changed while zipping

Added by Patrick Donnelly almost 7 years ago. Updated over 4 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Testing
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
4 - irritation
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS, qa-suite
Labels (FS):
qa
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Some logs are still being written to after the MDS is terminated. This only happens with valgrind and tasks/cfuse_workunit_suites_dbench.yaml:

Failure: reached maximum tries (50) after waiting for 300 seconds
4 jobs: ['1419908', '1419880', '1419824', '1419852']
suites intersection: ['clusters/9-mds.yaml', 'frag_enable.yaml', 'mon-debug.yaml', 'multimds/verify/{begin.yaml', 'overrides/{fuse-default-perm-no.yaml', 'tasks/cfuse_workunit_suites_dbench.yaml', 'validater/valgrind.yaml}', 'verify/{debug.yaml', 'whitelist_wrongly_marked_down.yaml}}']
suites union: ['clusters/9-mds.yaml', 'frag_enable.yaml', 'mon-debug.yaml', 'mount/fuse.yaml', 'mount/kclient.yaml', 'multimds/verify/{begin.yaml', 'objectstore/bluestore.yaml', 'objectstore/filestore-xfs.yaml', 'overrides/{fuse-default-perm-no.yaml', 'tasks/cfuse_workunit_suites_dbench.yaml', 'validater/valgrind.yaml}', 'verify/{debug.yaml', 'whitelist_wrongly_marked_down.yaml}}']

2017-07-19T23:04:06.336 INFO:teuthology.orchestra.run.smithi011:Running: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" 
2017-07-19T23:04:06.342 INFO:teuthology.orchestra.run.smithi033:Running: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" 
2017-07-19T23:04:06.347 INFO:teuthology.orchestra.run.smithi142:Running: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" 
2017-07-19T23:04:14.682 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-mds.f.log: file size changed while zipping
2017-07-19T23:04:32.298 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-mds.h.log: file size changed while zipping
2017-07-19T23:04:35.331 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-mds.i.log: file size changed while zipping
2017-07-19T23:04:41.299 INFO:tasks.ceph:Archiving logs...
2017-07-19T23:04:41.307 DEBUG:teuthology.misc:Transferring archived files from smithi142:/var/log/ceph to /home/teuthworker/archive/pdonnell-2017-07-19_19:04:52-multimds-wip-pdonnell-testing-20170718-testing-basic-smithi/1419908/remote/smithi142/log
2017-07-19T23:04:41.310 INFO:teuthology.orchestra.run.smithi142:Running: 'sudo tar cz -f - -C /var/log/ceph -- .'
2017-07-19T23:04:43.136 DEBUG:teuthology.misc:Transferring archived files from smithi011:/var/log/ceph to /home/teuthworker/archive/pdonnell-2017-07-19_19:04:52-multimds-wip-pdonnell-testing-20170718-testing-basic-smithi/1419908/remote/smithi011/log
2017-07-19T23:04:43.140 INFO:teuthology.orchestra.run.smithi011:Running: 'sudo tar cz -f - -C /var/log/ceph -- .'
2017-07-19T23:04:52.587 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/log/ceph to /home/teuthworker/archive/pdonnell-2017-07-19_19:04:52-multimds-wip-pdonnell-testing-20170718-testing-basic-smithi/1419908/remote/smithi033/log
2017-07-19T23:04:52.592 INFO:teuthology.orchestra.run.smithi033:Running: 'sudo tar cz -f - -C /var/log/ceph -- .'
2017-07-19T23:04:59.081 ERROR:teuthology.run_tasks:Manager failed: ceph
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 159, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20170718/qa/tasks/ceph.py", line 1645, in task
    '--no-mon-health-to-clog',
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 46, in nested
    if exit(*exc):
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20170718/qa/tasks/ceph.py", line 1198, in run_daemon
    teuthology.stop_daemons_of_type(ctx, type_, cluster_name)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/misc.py", line 1184, in stop_daemons_of_type
    daemon.stop()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/daemon.py", line 48, in stop
    run.wait([self.proc], timeout=timeout)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 432, in wait
    check_time()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 132, in __call__
    raise MaxWhileTries(error_msg)
MaxWhileTries: reached maximum tries (50) after waiting for 300 seconds
2017-07-19T23:04:59.083 DEBUG:teuthology.run_tasks:Unwinding manager install

From: /ceph/teuthology-archive/pdonnell-2017-07-19_19:04:52-multimds-wip-pdonnell-testing-20170718-testing-basic-smithi/1419908/teuthology.log

Actions
Copy link

Also available in: Atom PDF