Project

General

Profile

Bug #13151

mira105: rsyslog restart failure

Added by Greg Farnum almost 4 years ago. Updated almost 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Test Node
Target version:
-
Start date:
09/18/2015
Due date:
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:

Description

2015-09-17T17:08:42.478 INFO:teuthology.orchestra.run.mira105:Running: 'sudo service rsyslog restart'
2015-09-17T17:08:42.481 INFO:teuthology.orchestra.run.plana25:Running: 'sudo service rsyslog restart'
2015-09-17T17:08:42.496 INFO:teuthology.orchestra.run.mira105.stderr:stop: Unknown instance:
2015-09-17T17:08:42.516 INFO:teuthology.orchestra.run.mira105.stderr:start: Job failed to start
2015-09-17T17:08:42.550 INFO:teuthology.task.internal:Shutting down syslog monitoring...
2015-09-17T17:08:42.550 INFO:teuthology.orchestra.run.mira105:Running: 'sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart'
2015-09-17T17:08:42.561 INFO:teuthology.orchestra.run.plana25:Running: 'sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart'
2015-09-17T17:08:42.570 INFO:teuthology.orchestra.run.plana25.stdout:rsyslog stop/waiting
2015-09-17T17:08:42.581 INFO:teuthology.orchestra.run.mira105.stderr:stop: Unknown instance:
2015-09-17T17:08:42.591 INFO:teuthology.orchestra.run.plana25.stdout:rsyslog start/running, process 1858
2015-09-17T17:08:42.602 INFO:teuthology.orchestra.run.mira105.stderr:start: Job failed to start
2015-09-17T17:08:42.603 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 56, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/internal.py", line 668, in syslog
    wait=False,
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 401, in wait
    proc.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 114, in wait
    label=self.label)
CommandFailedError: Command failed on mira105 with status 1: 'sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart

Last time this happened that I can see was #8561.

History

#1 Updated by Dan Mick almost 4 years ago

fascinating. /var/log/upstart/rsyslog.log says:

Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.Feature buffer full.

#2 Updated by Dan Mick almost 4 years ago

See also #11126. The log amazed me then too :)

#3 Updated by Dan Mick almost 4 years ago

...and it appears to be an apparmor problem? ... O_o https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/1214979

#4 Updated by Dan Mick almost 4 years ago

also appears in these upstart logs:

network-interface-security-network-interface_eth0.log
network-interface-security-network-interface_eth1.log
network-interface-security-network-interface_lo.log

#5 Updated by Dan Mick almost 4 years ago

Ah. mira105 (and 047 and 052) were machines that got recently repaired; they moved from "down for repair" to "back in service", but we forgot to also make them "trusty". I'll lock and reimage.

#6 Updated by Dan Mick almost 4 years ago

  • Status changed from New to In Progress
  • Assignee set to Dan Mick

#7 Updated by Dan Mick almost 4 years ago

  • Status changed from In Progress to Resolved

Reimaged all three.

Also available in: Atom PDF