Bug #53257: mgr logs do not reopen after respawn - Orchestrator - Ceph

Actions

Copy link

Bug #53257

closed

mgr logs do not reopen after respawn

Added by Sage Weil over 2 years ago. Updated over 2 years ago.

Status:

Resolved

Priority:

High

Assignee:

Category:

Target version:

% Done:

Source:

Tags:

Backport:

pacific

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

43929

Crash signature (v1):

Crash signature (v2):

Description

...
2021-11-12T18:54:30.584+0000 7f88af992700  1 mgr handle_mgr_map respawning because set of enabled modules changed!
2021-11-12T18:54:30.584+0000 7f88af992700  1 mgr respawn  e: '/usr/bin/ceph-mgr'
2021-11-12T18:54:30.584+0000 7f88af992700  1 mgr respawn  0: '/usr/bin/ceph-mgr'
2021-11-12T18:54:30.584+0000 7f88af992700  1 mgr respawn  1: '-n'
2021-11-12T18:54:30.584+0000 7f88af992700  1 mgr respawn  2: 'mgr.smithi007.qbdtuk'
2021-11-12T18:54:30.584+0000 7f88af992700  1 mgr respawn  3: '-f'
2021-11-12T18:54:30.584+0000 7f88af992700  1 mgr respawn  4: '--setuser'
2021-11-12T18:54:30.584+0000 7f88af992700  1 mgr respawn  5: 'ceph'
2021-11-12T18:54:30.584+0000 7f88af992700  1 mgr respawn  6: '--setgroup'
2021-11-12T18:54:30.584+0000 7f88af992700  1 mgr respawn  7: 'ceph'
2021-11-12T18:54:30.584+0000 7f88af992700  1 mgr respawn  8: '--default-log-to-file=false'
2021-11-12T18:54:30.584+0000 7f88af992700  1 mgr respawn  9: '--default-log-to-journald=true'
2021-11-12T18:54:30.584+0000 7f88af992700  1 mgr respawn  10: '--default-log-to-stderr=false'
2021-11-12T18:54:30.584+0000 7f88af992700  1 mgr respawn respawning with exe /usr/bin/ceph-mgr
2021-11-12T18:54:30.584+0000 7f88af992700  1 mgr respawn  exe_path /proc/self/exe
2021-11-12T18:54:30.584+0000 7f884f722700  1 -- 172.21.15.7:0/1055722056 <== mon.0 v2:172.21.15.7:3300/0 12 ==== mgrmap(e 14) v1 ==== 40924+0+0 (secure 0 0 0) 0x56518502cb00 con 0x56518483d400
...end of file...

/a/sage-2021-11-12_17:44:21-orch:cephadm-wip-sage2-testing-2021-11-10-1508-distro-basic-smithi/6500038

this seems to happen pretty consistently. not on every respawn, but on many.

Actions

Copy link

Updated by Sage Weil over 2 years ago

Description updated (diff)

Actions

Copy link

Updated by Sage Weil over 2 years ago

[ceph: root@dael /]# cd /var/log/ceph
[ceph: root@dael ceph]# ls -al
total 1364
drwxrwx---. 2 gluster printadmin     118 Nov 15 14:57 .
drwxr-xr-x. 1 root    root            18 Nov 15 14:57 ..
-rw-r--r--. 1 ceph    ceph          7059 Nov 15 14:57 ceph-mgr.dael.cxikzt.log
-rw-r--r--. 1 ceph    ceph          1227 Nov 15 14:57 ceph-mgr.dael.pdiimz.log
-rw-r--r--. 1 ceph    ceph         42528 Nov 15 15:03 ceph-mon.dael.log
-rw-r--r--. 1 root    root       1338516 Nov 15 14:58 ceph-volume.log
[ceph: root@dael ceph]# stat .
  File: .
  Size: 118           Blocks: 0          IO Block: 4096   directory
Device: fd00h/64768d    Inode: 67891614    Links: 2
Access: (0770/drwxrwx---)  Uid: (  998/ gluster)   Gid: (  996/printadmin)
Access: 2021-11-15 15:03:08.294104133 +0000
Modify: 2021-11-15 14:57:40.103784294 +0000
Change: 2021-11-15 14:58:38.885453848 +0000
 Birth: -
[ceph: root@dael ceph]# chown ceph.ceph .
[ceph: root@dael ceph]# ceph daemon mgr.dael.cxikzt log reopen
{}
[ceph: root@dael ceph]# stat .
  File: .
  Size: 118           Blocks: 0          IO Block: 4096   directory
Device: fd00h/64768d    Inode: 67891614    Links: 2
Access: (0770/drwxrwx---)  Uid: (  167/    ceph)   Gid: (  167/    ceph)
Access: 2021-11-15 15:03:08.294104133 +0000
Modify: 2021-11-15 14:57:40.103784294 +0000
Change: 2021-11-15 15:03:17.574367657 +0000
 Birth: -

something is screwing up the ownership of /var/log/ceph

Actions

Copy link