Bug #50616
closedValueError: not enough values to unpack (expected 2, got 1) during upgrade from 15.2.8 -> 16.2.1
0%
Description
Started an upgrade from 15.2.8 to 16.2.1 via cephadm running on Ubuntu 20.04 & Docker.
MON/MGR/MDS upgraded fine along with some OSD's however it then failed to bring one OSD back up and halted the upgrade.
The OSD in question is down and seems to be it failed to pull in the new docker image on that node, checking the log's it points to "ValueError: not enough values to unpack (expected 2, got 1)"
I have since stopped the upgrade and have this OSD still offline, however, the cluster log's are now full of the following also repeating
cephadm exited with an error code: 1, stderr:Reconfig daemon osd.16 ... Traceback (most recent call last): File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7931, in <module> main() File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7919, in main r = ctx.func(ctx) File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 1717, in _default_image return func(ctx) File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 4162, in command_deploy c = get_container(ctx, ctx.fsid, daemon_type, daemon_id, File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 2451, in get_container volume_mounts=get_container_mounts(ctx, fsid, daemon_type, daemon_id), File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 2292, in get_container_mounts if HostFacts(ctx).selinux_enabled: File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6451, in selinux_enabled return (self.kernel_security['type'] == 'SELinux') and \ File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6434, in kernel_security ret = _fetch_apparmor() File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6415, in _fetch_apparmor item, mode = line.split(' ') ValueError: not enough values to unpack (expected 2, got 1) Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1172, in _remote_connection yield (conn, connr) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1087, in _run_cephadm code, '\n'.join(err))) orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Reconfig daemon osd.16 ... Traceback (most recent call last): File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7931, in <module> main() File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7919, in main r = ctx.func(ctx) File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 1717, in _default_image return func(ctx) File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 4162, in command_deploy c = get_container(ctx, ctx.fsid, daemon_type, daemon_id, File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 2451, in get_container volume_mounts=get_container_mounts(ctx, fsid, daemon_type, daemon_id), File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 2292, in get_container_mounts if HostFacts(ctx).selinux_enabled: File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6451, in selinux_enabled return (self.kernel_security['type'] == 'SELinux') and \ File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6434, in kernel_security ret = _fetch_apparmor() File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6415, in _fetch_apparmor item, mode = line.split(' ') ValueError: not enough values to unpack (expected 2, got 1)
Having a look via google for the ValueError seems it was a known bug in 15.2.8 : https://tracker.ceph.com/issues/49522
Not sure if this is a new bug factor my MGR/MON's are already on 16.2.1 but currently it is stopping me from bringing up the down OSD and finishing the upgrade to 16.2.1.
Let me know if require any extra logs
Updated by Sebastian Wagner about 3 years ago
- Project changed from Ceph to Orchestrator
- Category deleted (
common) - Priority changed from Normal to Urgent
Updated by Sebastian Wagner about 3 years ago
stderr:Reconfig daemon osd.16 ... Traceback (most recent call last): File "/var/lib/ceph/fsid/cephadm.170...482", line 7931, in <module> main() File "/var/lib/ceph/fsid/cephadm.170...482", line 7919, in main r = ctx.func(ctx) File "/var/lib/ceph/fsid/cephadm.170...482", line 1717, in defaultimage return func(ctx) File "/var/lib/ceph/fsid/cephadm.170...482", line 4162, in command_deploy c = get_container(ctx, ctx.fsid, daemon_type, daemon_id, File "/var/lib/ceph/fsid/cephadm.170...482", line 2451, in get_container volume_mounts=get_container_mounts(ctx, fsid, daemon_type, daemon_id), File "/var/lib/ceph/fsid/cephadm.170...482", line 2292, in get_container_mounts if HostFacts(ctx).selinux_enabled: File "/var/lib/ceph/fsid/cephadm.170...482", line 6451, in selinux_enabled return (self.kernel_security['type'] == 'SELinux') and \ File "/var/lib/ceph/fsid/cephadm.170...482", line 6434, in kernel_security ret = _fetch_apparmor() File "/var/lib/ceph/fsid/cephadm.170...482", line 6415, in _fetch_apparmor item, mode = line.split(' ') ValueError: not enough values to unpack (expected 2, got 1) Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1172, in _remote_connection yield (conn, connr) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1087, in _run_cephadm code, '\n'.join(err))) orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Reconfig daemon osd.16 ...
Updated by Sebastian Wagner about 3 years ago
workaround is to replace
/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482
in line 6451 with
return False and \
This works, cause we're not actually verifying the hash of this file, if it already exists.
Updated by Ashley Merrick about 3 years ago
I did as suggested but the upgrade still fails with the following new error
cephadm exited with an error code: 1, stderr:Reconfig daemon osd.24 ... Traceback (most recent call last): File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7931, in <module> main() File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7919, in main r = ctx.func(ctx) File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 1717, in _default_image return func(ctx) File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 4162, in command_deploy c = get_container(ctx, ctx.fsid, daemon_type, daemon_id, File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 2451, in get_container volume_mounts=get_container_mounts(ctx, fsid, daemon_type, daemon_id), File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 2292, in get_container_mounts if HostFacts(ctx).selinux_enabled: File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6451, in selinux_enabled return (self.kernel_security['type'] == 'SELinux') and \ File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6434, in kernel_security ret = _fetch_apparmor() File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6415, in _fetch_apparmor item, mode = line.split(' ') ValueError: not enough values to unpack (expected 2, got 1) Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1172, in _remote_connection yield (conn, connr) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1087, in _run_cephadm code, '\n'.join(err))) orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Reconfig daemon osd.24 ... Traceback (most recent call last): File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7931, in <module> main() File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7919, in main r = ctx.func(ctx) File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 1717, in _default_image return func(ctx) File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 4162, in command_deploy c = get_container(ctx, ctx.fsid, daemon_type, daemon_id, File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 2451, in get_container volume_mounts=get_container_mounts(ctx, fsid, daemon_type, daemon_id), File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 2292, in get_container_mounts if HostFacts(ctx).selinux_enabled: File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6451, in selinux_enabled return (self.kernel_security['type'] == 'SELinux') and \ File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6434, in kernel_security ret = _fetch_apparmor() File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6415, in _fetch_apparmor item, mode = line.split(' ') ValueError: not enough values to unpack (expected 2, got 1)
BTW Sorry for the formating cant work out what one handles it the best.
Updated by Ashley Merrick about 3 years ago
Just to confirm this is how the section looks after my edit
def selinux_enabled(self): return False and \ (self.kernel_security['description'] != 'SELinux: Disabled')
I did a reboot of the server hosting the cephadm just to make sure the edits where loaded.
Updated by Sam Overton almost 3 years ago
#50693 explains the root-cause, which is that /sys/kernel/security/apparmor/profiles
is an empty file and cephadm is expecting at least one valid line in that file.
The work-around for me was to install the apparmor-profiles
package (on Debian) which populates the profiles file and prevents this error from occurring.
Updated by Ashley Merrick almost 3 years ago
Sam Overton wrote:
#50693 explains the root-cause, which is that
/sys/kernel/security/apparmor/profiles
is an empty file and cephadm is expecting at least one valid line in that file.The work-around for me was to install the
apparmor-profiles
package (on Debian) which populates the profiles file and prevents this error from occurring.
Perfect running "apt install apparmor-profiles" on each node removed the error showing in the CEPH Log, and allowed the upgrade to complete.
So looks more like CEPH is expecting something to be installed which isn't always by default / not set to handle when it's not installed.
Updated by Sebastian Wagner almost 3 years ago
- Is duplicate of Bug #50693: cephadm: commands fail with "ValueError: not enough values to unpack (expected 2, got 1)" when apparmor profiles file is empty added
Updated by Sebastian Wagner almost 3 years ago
- Status changed from New to Duplicate