Project

General

Profile

Actions
Copy link

Bug #50616

closed

ValueError: not enough values to unpack (expected 2, got 1) during upgrade from 15.2.8 -> 16.2.1

Added by Ashley Merrick almost 3 years ago. Updated almost 3 years ago.

Status:
Duplicate
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Started an upgrade from 15.2.8 to 16.2.1 via cephadm running on Ubuntu 20.04 & Docker.

MON/MGR/MDS upgraded fine along with some OSD's however it then failed to bring one OSD back up and halted the upgrade.

The OSD in question is down and seems to be it failed to pull in the new docker image on that node, checking the log's it points to "ValueError: not enough values to unpack (expected 2, got 1)"

I have since stopped the upgrade and have this OSD still offline, however, the cluster log's are now full of the following also repeating

cephadm exited with an error code: 1, stderr:Reconfig daemon osd.16 ... Traceback (most recent call last): File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7931, in <module> main() File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7919, in main r = ctx.func(ctx) File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 1717, in _default_image return func(ctx) File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 4162, in command_deploy c = get_container(ctx, ctx.fsid, daemon_type, daemon_id, File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 2451, in get_container volume_mounts=get_container_mounts(ctx, fsid, daemon_type, daemon_id), File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 2292, in get_container_mounts if HostFacts(ctx).selinux_enabled: File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6451, in selinux_enabled return (self.kernel_security['type'] == 'SELinux') and \ File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6434, in kernel_security ret = _fetch_apparmor() File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6415, in _fetch_apparmor item, mode = line.split(' ') ValueError: not enough values to unpack (expected 2, got 1) Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1172, in _remote_connection yield (conn, connr) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1087, in _run_cephadm code, '\n'.join(err))) orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Reconfig daemon osd.16 ... Traceback (most recent call last): File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7931, in <module> main() File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7919, in main r = ctx.func(ctx) File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 1717, in _default_image return func(ctx) File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 4162, in command_deploy c = get_container(ctx, ctx.fsid, daemon_type, daemon_id, File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 2451, in get_container volume_mounts=get_container_mounts(ctx, fsid, daemon_type, daemon_id), File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 2292, in get_container_mounts if HostFacts(ctx).selinux_enabled: File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6451, in selinux_enabled return (self.kernel_security['type'] == 'SELinux') and \ File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6434, in kernel_security ret = _fetch_apparmor() File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6415, in _fetch_apparmor item, mode = line.split(' ') ValueError: not enough values to unpack (expected 2, got 1)

Having a look via google for the ValueError seems it was a known bug in 15.2.8 : https://tracker.ceph.com/issues/49522

Not sure if this is a new bug factor my MGR/MON's are already on 16.2.1 but currently it is stopping me from bringing up the down OSD and finishing the upgrade to 16.2.1.

Let me know if require any extra logs

Related issues 1 (0 open — 1 closed)

Is duplicate of Orchestrator - Bug #50693: cephadm: commands fail with "ValueError: not enough values to unpack (expected 2, got 1)" when apparmor profiles file is emptyResolved

Actions
Actions
Copy link
 #1

Updated by Sebastian Wagner almost 3 years ago

  • Project changed from Ceph to Orchestrator
  • Category deleted (common)
  • Priority changed from Normal to Urgent
Actions
Copy link
 #2

Updated by Sebastian Wagner almost 3 years ago 

stderr:Reconfig daemon osd.16 ... 
Traceback (most recent call last):
 File "/var/lib/ceph/fsid/cephadm.170...482", line 7931, in <module> 
     main()
 File "/var/lib/ceph/fsid/cephadm.170...482", line 7919, in main 
     r = ctx.func(ctx)
 File "/var/lib/ceph/fsid/cephadm.170...482", line 1717, in defaultimage 
     return func(ctx)
 File "/var/lib/ceph/fsid/cephadm.170...482", line 4162, in command_deploy 
     c = get_container(ctx, ctx.fsid, daemon_type, daemon_id,
 File "/var/lib/ceph/fsid/cephadm.170...482", line 2451, in get_container 
     volume_mounts=get_container_mounts(ctx, fsid, daemon_type, daemon_id),
 File "/var/lib/ceph/fsid/cephadm.170...482", line 2292, in get_container_mounts 
     if HostFacts(ctx).selinux_enabled:
 File "/var/lib/ceph/fsid/cephadm.170...482", line 6451, in selinux_enabled 
     return (self.kernel_security['type'] == 'SELinux') and \
 File "/var/lib/ceph/fsid/cephadm.170...482", line 6434, in kernel_security 
     ret = _fetch_apparmor()
 File "/var/lib/ceph/fsid/cephadm.170...482", line 6415, in _fetch_apparmor 
     item, mode = line.split(' ') 
ValueError: not enough values to unpack (expected 2, got 1) 

Traceback (most recent call last):
 File "/usr/share/ceph/mgr/cephadm/serve.py", line 1172, in _remote_connection 
     yield (conn, connr)
 File "/usr/share/ceph/mgr/cephadm/serve.py", line 1087, in _run_cephadm 
     code, '\n'.join(err))) 
orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Reconfig daemon osd.16 ...
Actions
Copy link
 #3

Updated by Sebastian Wagner almost 3 years ago

workaround is to replace

/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482

in line 6451 with

return False and \

This works, cause we're not actually verifying the hash of this file, if it already exists.

Actions
Copy link
 #4

Updated by Ashley Merrick almost 3 years ago

I did as suggested but the upgrade still fails with the following new error

cephadm exited with an error code: 1, stderr:Reconfig daemon osd.24 ... Traceback (most recent call last): File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7931, in <module> main() File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7919, in main r = ctx.func(ctx) File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 1717, in _default_image return func(ctx) File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 4162, in command_deploy c = get_container(ctx, ctx.fsid, daemon_type, daemon_id, File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 2451, in get_container volume_mounts=get_container_mounts(ctx, fsid, daemon_type, daemon_id), File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 2292, in get_container_mounts if HostFacts(ctx).selinux_enabled: File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6451, in selinux_enabled return (self.kernel_security['type'] == 'SELinux') and \ File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6434, in kernel_security ret = _fetch_apparmor() File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6415, in _fetch_apparmor item, mode = line.split(' ') ValueError: not enough values to unpack (expected 2, got 1) Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1172, in _remote_connection yield (conn, connr) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1087, in _run_cephadm code, '\n'.join(err))) orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Reconfig daemon osd.24 ... Traceback (most recent call last): File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7931, in <module> main() File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7919, in main r = ctx.func(ctx) File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 1717, in _default_image return func(ctx) File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 4162, in command_deploy c = get_container(ctx, ctx.fsid, daemon_type, daemon_id, File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 2451, in get_container volume_mounts=get_container_mounts(ctx, fsid, daemon_type, daemon_id), File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 2292, in get_container_mounts if HostFacts(ctx).selinux_enabled: File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6451, in selinux_enabled return (self.kernel_security['type'] == 'SELinux') and \ File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6434, in kernel_security ret = _fetch_apparmor() File "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6415, in _fetch_apparmor item, mode = line.split(' ') ValueError: not enough values to unpack (expected 2, got 1)

BTW Sorry for the formating cant work out what one handles it the best.

Actions
Copy link
 #5

Updated by Ashley Merrick almost 3 years ago

Just to confirm this is how the section looks after my edit

    def selinux_enabled(self):
        return False and \
               (self.kernel_security['description'] != 'SELinux: Disabled')

I did a reboot of the server hosting the cephadm just to make sure the edits where loaded.

Actions
Copy link
 #6

Updated by Sam Overton almost 3 years ago

#50693 explains the root-cause, which is that /sys/kernel/security/apparmor/profiles is an empty file and cephadm is expecting at least one valid line in that file.

The work-around for me was to install the apparmor-profiles package (on Debian) which populates the profiles file and prevents this error from occurring.

Actions
Copy link
 #7

Updated by Ashley Merrick almost 3 years ago

Sam Overton wrote:

#50693 explains the root-cause, which is that /sys/kernel/security/apparmor/profiles is an empty file and cephadm is expecting at least one valid line in that file.

The work-around for me was to install the apparmor-profiles package (on Debian) which populates the profiles file and prevents this error from occurring.

Perfect running "apt install apparmor-profiles" on each node removed the error showing in the CEPH Log, and allowed the upgrade to complete.

So looks more like CEPH is expecting something to be installed which isn't always by default / not set to handle when it's not installed.

Actions
Copy link
 #8

Updated by Sebastian Wagner almost 3 years ago

  • Is duplicate of Bug #50693: cephadm: commands fail with "ValueError: not enough values to unpack (expected 2, got 1)" when apparmor profiles file is empty added
Actions
Copy link
 #9

Updated by Sebastian Wagner almost 3 years ago

  • Status changed from New to Duplicate
Actions
Copy link

Also available in: Atom PDF