Bug #53302
qa: sudo logrotate /etc/logrotate.d/ceph-test.conf failed with status 1
Status:
Triaged
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
fs
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Seen in teuthology run while running tasks/kernel_cfuse_workunits_dbench_iozone
The traceback is as below.
2021-11-09T00:21:46.690 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph-c_af62a1db6cc922d2e0b2aeafa6345763c002a05b/qa/tasks/ceph.py", line 188, in invoke_logrotate run.wait(procs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_27954452159076fda5642b6c0eb0c4998b99d2e4/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_27954452159076fda5642b6c0eb0c4998b99d2e4/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_27954452159076fda5642b6c0eb0c4998b99d2e4/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi156 with status 1: 'sudo logrotate /etc/logrotate.d/ceph-test.conf'
Seen in the following pacific run
http://pulpito.front.sepia.ceph.com/yuriw-2021-11-08_15:19:37-fs-wip-yuri2-testing-2021-11-06-1322-pacific-distro-basic-smithi/6491150/
Log: /ceph/teuthology-archive/yuriw-2021-11-08_15:19:37-fs-wip-yuri2-testing-2021-11-06-1322-pacific-distro-basic-smithi/6491150/teuthology.log
misc log
2021-11-09T00:00:34.892232+00:00 smithi156 systemd[1]: Starting Rotate log files... 2021-11-09T00:00:34.973248+00:00 smithi156 systemd[1]: Starting Daily man-db regeneration... 2021-11-09T00:00:35.124411+00:00 smithi156 logrotate[23970]: error: ceph-test.conf:1 duplicate log entry for /var/log/ceph/ceph-mds.b.log 2021-11-09T00:00:35.142840+00:00 smithi156 logrotate[23970]: error: ceph-test.conf:14 duplicate log entry for /var/log/ceph/ceph-osd.4.log 2021-11-09T00:00:35.142919+00:00 smithi156 logrotate[23970]: error: found error in file ceph-test.conf, skipping 2021-11-09T00:00:39.040813+00:00 smithi156 systemd[1]: man-db.service: Succeeded. 2021-11-09T00:00:39.045145+00:00 smithi156 systemd[1]: Finished Daily man-db regeneration. 2021-11-09T00:00:44.265087+00:00 smithi156 rsyslogd: [origin software="rsyslogd" swVersion="8.2001.0" x-pid="1340" x-info="https://www.rsyslog.com"] rsyslogd was HUPed 2021-11-09T00:00:44.316197+00:00 smithi156 systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE 2021-11-09T00:00:44.316611+00:00 smithi156 systemd[1]: logrotate.service: Failed with result 'exit-code'. 2021-11-09T00:00:44.317588+00:00 smithi156 systemd[1]: Failed to start Rotate log files. 2021-11-09T00:04:22.560434+00:00 smithi156 sudo: pam_unix(sudo:session): session closed for user root 2021-11-09T00:04:50.011911+00:00 smithi156 smartd[538]: Device: /dev/sda [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 72 to 81 2021-11-09T00:04:50.012409+00:00 smithi156 smartd[538]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 67 to 66 2021-11-09T00:04:50.012645+00:00 smithi156 smartd[538]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 33 to 34 2021-11-09T00:05:01.283499+00:00 smithi156 CRON[24002]: pam_unix(cron:session): session opened for user root by (uid=0) 2021-11-09T00:05:01.285340+00:00 smithi156 CRON[24003]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1) 2021-11-09T00:05:01.293933+00:00 smithi156 CRON[24002]: pam_unix(cron:session): session closed for user root 2021-11-09T00:15:01.309316+00:00 smithi156 CRON[24008]: pam_unix(cron:session): session opened for user root by (uid=0) 2021-11-09T00:15:01.343246+00:00 smithi156 CRON[24009]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1) 2021-11-09T00:15:01.343508+00:00 smithi156 CRON[24008]: pam_unix(cron:session): session closed for user root 2021-11-09T00:17:01.334210+00:00 smithi156 CRON[24012]: pam_unix(cron:session): session opened for user root by (uid=0) 2021-11-09T00:17:01.337804+00:00 smithi156 CRON[24013]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) 2021-11-09T00:17:01.355170+00:00 smithi156 CRON[24012]: pam_unix(cron:session): session closed for user root 2021-11-09T00:21:46.741443+00:00 smithi156 sudo: ubuntu : TTY=unknown ; PWD=/home/ubuntu ; USER=root ; COMMAND=/usr/bin/rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-11-09T00:21:46.784179+00:00 smithi156 sudo: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-11-09T00:21:46.794788+00:00 smithi156 sudo: pam_unix(sudo:session): session closed for user root 2021-11-09T00:21:47.240482+00:00 smithi156 sudo: ubuntu : TTY=unknown ; PWD=/home/ubuntu ; USER=root ; COMMAND=/usr/bin/rm -f -- /etc/rsyslog.d/80-cephtest.conf 2021-11-09T00:21:47.241616+00:00 smithi156 sudo: pam_unix(sudo:session): session opened for user root by (uid=0) 2021-11-09T00:21:47.250184+00:00 smithi156 sudo: pam_unix(sudo:session): session closed for user root 2021-11-09T00:21:47.264798+00:00 smithi156 sudo: ubuntu : TTY=unknown ; PWD=/home/ubuntu ; USER=root ; COMMAND=/usr/sbin/service rsyslog restart 2021-11-09T00:21:47.266513+00:00 smithi156 sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
History
#1 Updated by Kotresh Hiremath Ravishankar over 2 years ago
- Description updated (diff)
#2 Updated by Kotresh Hiremath Ravishankar over 2 years ago
- Project changed from CephFS to teuthology
#3 Updated by Josh Durgin about 2 years ago
Is this still occurring?
#5 Updated by Kotresh Hiremath Ravishankar 11 months ago
#7 Updated by Zack Cerza 7 months ago
Looks like there's a problem with the logrotate configuration set up by the ceph task.
#8 Updated by Zack Cerza 7 months ago
- Project changed from teuthology to Ceph
- Status changed from New to Triaged