Project

General

Profile

Bug #20735

mds: stderr:gzip: /var/log/ceph/ceph-mds.f.log: file size changed while zipping

Added by Patrick Donnelly over 6 years ago. Updated over 4 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Testing
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
4 - irritation
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS, qa-suite
Labels (FS):
qa
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Some logs are still being written to after the MDS is terminated. This only happens with valgrind and tasks/cfuse_workunit_suites_dbench.yaml:

Failure: reached maximum tries (50) after waiting for 300 seconds
4 jobs: ['1419908', '1419880', '1419824', '1419852']
suites intersection: ['clusters/9-mds.yaml', 'frag_enable.yaml', 'mon-debug.yaml', 'multimds/verify/{begin.yaml', 'overrides/{fuse-default-perm-no.yaml', 'tasks/cfuse_workunit_suites_dbench.yaml', 'validater/valgrind.yaml}', 'verify/{debug.yaml', 'whitelist_wrongly_marked_down.yaml}}']
suites union: ['clusters/9-mds.yaml', 'frag_enable.yaml', 'mon-debug.yaml', 'mount/fuse.yaml', 'mount/kclient.yaml', 'multimds/verify/{begin.yaml', 'objectstore/bluestore.yaml', 'objectstore/filestore-xfs.yaml', 'overrides/{fuse-default-perm-no.yaml', 'tasks/cfuse_workunit_suites_dbench.yaml', 'validater/valgrind.yaml}', 'verify/{debug.yaml', 'whitelist_wrongly_marked_down.yaml}}']
2017-07-19T23:04:06.336 INFO:teuthology.orchestra.run.smithi011:Running: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" 
2017-07-19T23:04:06.342 INFO:teuthology.orchestra.run.smithi033:Running: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" 
2017-07-19T23:04:06.347 INFO:teuthology.orchestra.run.smithi142:Running: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" 
2017-07-19T23:04:14.682 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-mds.f.log: file size changed while zipping
2017-07-19T23:04:32.298 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-mds.h.log: file size changed while zipping
2017-07-19T23:04:35.331 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-mds.i.log: file size changed while zipping
2017-07-19T23:04:41.299 INFO:tasks.ceph:Archiving logs...
2017-07-19T23:04:41.307 DEBUG:teuthology.misc:Transferring archived files from smithi142:/var/log/ceph to /home/teuthworker/archive/pdonnell-2017-07-19_19:04:52-multimds-wip-pdonnell-testing-20170718-testing-basic-smithi/1419908/remote/smithi142/log
2017-07-19T23:04:41.310 INFO:teuthology.orchestra.run.smithi142:Running: 'sudo tar cz -f - -C /var/log/ceph -- .'
2017-07-19T23:04:43.136 DEBUG:teuthology.misc:Transferring archived files from smithi011:/var/log/ceph to /home/teuthworker/archive/pdonnell-2017-07-19_19:04:52-multimds-wip-pdonnell-testing-20170718-testing-basic-smithi/1419908/remote/smithi011/log
2017-07-19T23:04:43.140 INFO:teuthology.orchestra.run.smithi011:Running: 'sudo tar cz -f - -C /var/log/ceph -- .'
2017-07-19T23:04:52.587 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/log/ceph to /home/teuthworker/archive/pdonnell-2017-07-19_19:04:52-multimds-wip-pdonnell-testing-20170718-testing-basic-smithi/1419908/remote/smithi033/log
2017-07-19T23:04:52.592 INFO:teuthology.orchestra.run.smithi033:Running: 'sudo tar cz -f - -C /var/log/ceph -- .'
2017-07-19T23:04:59.081 ERROR:teuthology.run_tasks:Manager failed: ceph
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 159, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20170718/qa/tasks/ceph.py", line 1645, in task
    '--no-mon-health-to-clog',
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 46, in nested
    if exit(*exc):
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20170718/qa/tasks/ceph.py", line 1198, in run_daemon
    teuthology.stop_daemons_of_type(ctx, type_, cluster_name)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/misc.py", line 1184, in stop_daemons_of_type
    daemon.stop()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/daemon.py", line 48, in stop
    run.wait([self.proc], timeout=timeout)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 432, in wait
    check_time()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 132, in __call__
    raise MaxWhileTries(error_msg)
MaxWhileTries: reached maximum tries (50) after waiting for 300 seconds
2017-07-19T23:04:59.083 DEBUG:teuthology.run_tasks:Unwinding manager install

From: /ceph/teuthology-archive/pdonnell-2017-07-19_19:04:52-multimds-wip-pdonnell-testing-20170718-testing-basic-smithi/1419908/teuthology.log

History

#1 Updated by Patrick Donnelly almost 6 years ago

  • Priority changed from High to Normal
  • Target version set to v14.0.0
  • Severity changed from 3 - minor to 4 - irritation
  • Component(FS) MDS, qa-suite added
  • Labels (FS) qa added

#2 Updated by Patrick Donnelly about 5 years ago

  • Target version changed from v14.0.0 to v15.0.0

#3 Updated by Patrick Donnelly about 5 years ago

  • Target version deleted (v15.0.0)

#4 Updated by Neha Ojha over 4 years ago

Here the same happened with the mon, with valgrind.

2019-11-08T03:02:59.341 INFO:teuthology.orchestra.run.smithi144:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --
2019-11-08T03:03:01.104 INFO:teuthology.orchestra.run.smithi143.stderr:gzip: /var/log/ceph/ceph-mon.c.log: file size changed while zipping
2019-11-08T03:03:02.594 INFO:tasks.ceph:Archiving logs...
2019-11-08T03:03:02.594 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/log/ceph to /home/teuthworker/archive/yuriw-2019-11-08_00:57:06-rados-wip-yuri5-testing-2019-11-07-2203-nautilus-distro-basic-smithi/4481070/remote/smithi143/log
2019-11-08T03:03:02.595 INFO:teuthology.orchestra.run.smithi143:Running:
2019-11-08T03:03:02.596 INFO:teuthology.orchestra.run.smithi143:> sudo tar cz -f - -C /var/log/ceph -- .
2019-11-08T03:03:03.629 DEBUG:teuthology.misc:Transferring archived files from smithi144:/var/log/ceph to /home/teuthworker/archive/yuriw-2019-11-08_00:57:06-rados-wip-yuri5-testing-2019-11-07-2203-nautilus-distro-basic-smithi/4481070/remote/smithi144/log
2019-11-08T03:03:03.630 INFO:teuthology.orchestra.run.smithi144:Running:
2019-11-08T03:03:03.630 INFO:teuthology.orchestra.run.smithi144:> sudo tar cz -f - -C /var/log/ceph -- .
2019-11-08T03:03:06.135 ERROR:teuthology.run_tasks:Manager failed: ceph
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 159, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri5-testing-2019-11-07-2203-nautilus/qa/tasks/ceph.py", line 1952, in task
    '--no-mon-health-to-clog',
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 46, in nested
    if exit(*exc):
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri5-testing-2019-11-07-2203-nautilus/qa/tasks/ceph.py", line 1445, in run_daemon
    teuthology.stop_daemons_of_type(ctx, type_, cluster_name)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/misc.py", line 1171, in stop_daemons_of_type
    daemon.stop()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/daemon/state.py", line 138, in stop
    run.wait([self.proc], timeout=timeout)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 455, in wait
    check_time()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 132, in __call__
    raise MaxWhileTries(error_msg)
MaxWhileTries: reached maximum tries (50) after waiting for 300 seconds
2019-11-08T03:03:06.156 DEBUG:teuthology.run_tasks:Unwinding manager install

/a/yuriw-2019-11-08_00:57:06-rados-wip-yuri5-testing-2019-11-07-2203-nautilus-distro-basic-smithi/4481070/

Also available in: Atom PDF