Bug #20735
mds: stderr:gzip: /var/log/ceph/ceph-mds.f.log: file size changed while zipping
Status:
New
Priority:
Normal
Assignee:
-
Category:
Testing
Target version:
-
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
4 - irritation
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS, qa-suite
Labels (FS):
qa
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Some logs are still being written to after the MDS is terminated. This only happens with valgrind and tasks/cfuse_workunit_suites_dbench.yaml:
Failure: reached maximum tries (50) after waiting for 300 seconds 4 jobs: ['1419908', '1419880', '1419824', '1419852'] suites intersection: ['clusters/9-mds.yaml', 'frag_enable.yaml', 'mon-debug.yaml', 'multimds/verify/{begin.yaml', 'overrides/{fuse-default-perm-no.yaml', 'tasks/cfuse_workunit_suites_dbench.yaml', 'validater/valgrind.yaml}', 'verify/{debug.yaml', 'whitelist_wrongly_marked_down.yaml}}'] suites union: ['clusters/9-mds.yaml', 'frag_enable.yaml', 'mon-debug.yaml', 'mount/fuse.yaml', 'mount/kclient.yaml', 'multimds/verify/{begin.yaml', 'objectstore/bluestore.yaml', 'objectstore/filestore-xfs.yaml', 'overrides/{fuse-default-perm-no.yaml', 'tasks/cfuse_workunit_suites_dbench.yaml', 'validater/valgrind.yaml}', 'verify/{debug.yaml', 'whitelist_wrongly_marked_down.yaml}}']
2017-07-19T23:04:06.336 INFO:teuthology.orchestra.run.smithi011:Running: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" 2017-07-19T23:04:06.342 INFO:teuthology.orchestra.run.smithi033:Running: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" 2017-07-19T23:04:06.347 INFO:teuthology.orchestra.run.smithi142:Running: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" 2017-07-19T23:04:14.682 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-mds.f.log: file size changed while zipping 2017-07-19T23:04:32.298 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-mds.h.log: file size changed while zipping 2017-07-19T23:04:35.331 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-mds.i.log: file size changed while zipping 2017-07-19T23:04:41.299 INFO:tasks.ceph:Archiving logs... 2017-07-19T23:04:41.307 DEBUG:teuthology.misc:Transferring archived files from smithi142:/var/log/ceph to /home/teuthworker/archive/pdonnell-2017-07-19_19:04:52-multimds-wip-pdonnell-testing-20170718-testing-basic-smithi/1419908/remote/smithi142/log 2017-07-19T23:04:41.310 INFO:teuthology.orchestra.run.smithi142:Running: 'sudo tar cz -f - -C /var/log/ceph -- .' 2017-07-19T23:04:43.136 DEBUG:teuthology.misc:Transferring archived files from smithi011:/var/log/ceph to /home/teuthworker/archive/pdonnell-2017-07-19_19:04:52-multimds-wip-pdonnell-testing-20170718-testing-basic-smithi/1419908/remote/smithi011/log 2017-07-19T23:04:43.140 INFO:teuthology.orchestra.run.smithi011:Running: 'sudo tar cz -f - -C /var/log/ceph -- .' 2017-07-19T23:04:52.587 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/log/ceph to /home/teuthworker/archive/pdonnell-2017-07-19_19:04:52-multimds-wip-pdonnell-testing-20170718-testing-basic-smithi/1419908/remote/smithi033/log 2017-07-19T23:04:52.592 INFO:teuthology.orchestra.run.smithi033:Running: 'sudo tar cz -f - -C /var/log/ceph -- .' 2017-07-19T23:04:59.081 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 159, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ self.gen.next() File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20170718/qa/tasks/ceph.py", line 1645, in task '--no-mon-health-to-clog', File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ self.gen.next() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ self.gen.next() File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20170718/qa/tasks/ceph.py", line 1198, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/misc.py", line 1184, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/daemon.py", line 48, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 432, in wait check_time() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 132, in __call__ raise MaxWhileTries(error_msg) MaxWhileTries: reached maximum tries (50) after waiting for 300 seconds 2017-07-19T23:04:59.083 DEBUG:teuthology.run_tasks:Unwinding manager install
From: /ceph/teuthology-archive/pdonnell-2017-07-19_19:04:52-multimds-wip-pdonnell-testing-20170718-testing-basic-smithi/1419908/teuthology.log
History
#1 Updated by Patrick Donnelly almost 6 years ago
- Priority changed from High to Normal
- Target version set to v14.0.0
- Severity changed from 3 - minor to 4 - irritation
- Component(FS) MDS, qa-suite added
- Labels (FS) qa added
#2 Updated by Patrick Donnelly about 5 years ago
- Target version changed from v14.0.0 to v15.0.0
#3 Updated by Patrick Donnelly about 5 years ago
- Target version deleted (
v15.0.0)
#4 Updated by Neha Ojha over 4 years ago
Here the same happened with the mon, with valgrind.
2019-11-08T03:02:59.341 INFO:teuthology.orchestra.run.smithi144:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2019-11-08T03:03:01.104 INFO:teuthology.orchestra.run.smithi143.stderr:gzip: /var/log/ceph/ceph-mon.c.log: file size changed while zipping 2019-11-08T03:03:02.594 INFO:tasks.ceph:Archiving logs... 2019-11-08T03:03:02.594 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/log/ceph to /home/teuthworker/archive/yuriw-2019-11-08_00:57:06-rados-wip-yuri5-testing-2019-11-07-2203-nautilus-distro-basic-smithi/4481070/remote/smithi143/log 2019-11-08T03:03:02.595 INFO:teuthology.orchestra.run.smithi143:Running: 2019-11-08T03:03:02.596 INFO:teuthology.orchestra.run.smithi143:> sudo tar cz -f - -C /var/log/ceph -- . 2019-11-08T03:03:03.629 DEBUG:teuthology.misc:Transferring archived files from smithi144:/var/log/ceph to /home/teuthworker/archive/yuriw-2019-11-08_00:57:06-rados-wip-yuri5-testing-2019-11-07-2203-nautilus-distro-basic-smithi/4481070/remote/smithi144/log 2019-11-08T03:03:03.630 INFO:teuthology.orchestra.run.smithi144:Running: 2019-11-08T03:03:03.630 INFO:teuthology.orchestra.run.smithi144:> sudo tar cz -f - -C /var/log/ceph -- . 2019-11-08T03:03:06.135 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 159, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ self.gen.next() File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri5-testing-2019-11-07-2203-nautilus/qa/tasks/ceph.py", line 1952, in task '--no-mon-health-to-clog', File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ self.gen.next() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ self.gen.next() File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri5-testing-2019-11-07-2203-nautilus/qa/tasks/ceph.py", line 1445, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/misc.py", line 1171, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/daemon/state.py", line 138, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 455, in wait check_time() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 132, in __call__ raise MaxWhileTries(error_msg) MaxWhileTries: reached maximum tries (50) after waiting for 300 seconds 2019-11-08T03:03:06.156 DEBUG:teuthology.run_tasks:Unwinding manager install
/a/yuriw-2019-11-08_00:57:06-rados-wip-yuri5-testing-2019-11-07-2203-nautilus-distro-basic-smithi/4481070/