Ceph » Orchestrator

stderr:Reconfig daemon osd.16 ... 
Traceback (most recent call last):
 File "/var/lib/ceph/fsid/cephadm.170...482", line 7931, in <module> 
     main()
 File "/var/lib/ceph/fsid/cephadm.170...482", line 7919, in main 
     r = ctx.func(ctx)
 File "/var/lib/ceph/fsid/cephadm.170...482", line 1717, in defaultimage 
     return func(ctx)
 File "/var/lib/ceph/fsid/cephadm.170...482", line 4162, in command_deploy 
     c = get_container(ctx, ctx.fsid, daemon_type, daemon_id,
 File "/var/lib/ceph/fsid/cephadm.170...482", line 2451, in get_container 
     volume_mounts=get_container_mounts(ctx, fsid, daemon_type, daemon_id),
 File "/var/lib/ceph/fsid/cephadm.170...482", line 2292, in get_container_mounts 
     if HostFacts(ctx).selinux_enabled:
 File "/var/lib/ceph/fsid/cephadm.170...482", line 6451, in selinux_enabled 
     return (self.kernel_security['type'] == 'SELinux') and \
 File "/var/lib/ceph/fsid/cephadm.170...482", line 6434, in kernel_security 
     ret = _fetch_apparmor()
 File "/var/lib/ceph/fsid/cephadm.170...482", line 6415, in _fetch_apparmor 
     item, mode = line.split(' ') 
ValueError: not enough values to unpack (expected 2, got 1) 

Traceback (most recent call last):
 File "/usr/share/ceph/mgr/cephadm/serve.py", line 1172, in _remote_connection 
     yield (conn, connr)
 File "/usr/share/ceph/mgr/cephadm/serve.py", line 1087, in _run_cephadm 
     code, '\n'.join(err))) 
orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Reconfig daemon osd.16 ... 

workaround is to replace

/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482

in line 6451 with

return False and \

This works, cause we're not actually verifying the hash of this file, if it already exists.

I did as suggested but the upgrade still fails with the following new error

cephadm exited with an error code: 1, stderr:Reconfig daemon osd.24 ... Traceback (most recent call last): File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7931, in <module> main() File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7919, in main r = ctx.func(ctx) File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 1717, in _default_image return func(ctx) File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 4162, in command_deploy c = get_container(ctx, ctx.fsid, daemon_type, daemon_id, File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 2451, in get_container volume_mounts=get_container_mounts(ctx, fsid, daemon_type, daemon_id), File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 2292, in get_container_mounts if HostFacts(ctx).selinux_enabled: File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6451, in selinux_enabled return (self.kernel_security['type'] == 'SELinux') and \ File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6434, in kernel_security ret = _fetch_apparmor() File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6415, in _fetch_apparmor item, mode = line.split(' ') ValueError: not enough values to unpack (expected 2, got 1) Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1172, in _remote_connection yield (conn, connr) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1087, in _run_cephadm code, '\n'.join(err))) orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Reconfig daemon osd.24 ... Traceback (most recent call last): File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7931, in <module> main() File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7919, in main r = ctx.func(ctx) File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 1717, in _default_image return func(ctx) File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 4162, in command_deploy c = get_container(ctx, ctx.fsid, daemon_type, daemon_id, File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 2451, in get_container volume_mounts=get_container_mounts(ctx, fsid, daemon_type, daemon_id), File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 2292, in get_container_mounts if HostFacts(ctx).selinux_enabled: File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6451, in selinux_enabled return (self.kernel_security['type'] == 'SELinux') and \ File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6434, in kernel_security ret = _fetch_apparmor() File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6415, in _fetch_apparmor item, mode = line.split(' ') ValueError: not enough values to unpack (expected 2, got 1)

BTW Sorry for the formating cant work out what one handles it the best.

Just to confirm this is how the section looks after my edit

    def selinux_enabled(self):
        return False and \
               (self.kernel_security['description'] != 'SELinux: Disabled')

I did a reboot of the server hosting the cephadm just to make sure the edits where loaded.

#50693 explains the root-cause, which is that /sys/kernel/security/apparmor/profiles is an empty file and cephadm is expecting at least one valid line in that file.

The work-around for me was to install the apparmor-profiles package (on Debian) which populates the profiles file and prevents this error from occurring.

Sam Overton wrote:

#50693 explains the root-cause, which is that /sys/kernel/security/apparmor/profiles is an empty file and cephadm is expecting at least one valid line in that file.

The work-around for me was to install the apparmor-profiles package (on Debian) which populates the profiles file and prevents this error from occurring.

Perfect running "apt install apparmor-profiles" on each node removed the error showing in the CEPH Log, and allowed the upgrade to complete.

So looks more like CEPH is expecting something to be installed which isn't always by default / not set to handle when it's not installed.