Actions
Bug #50693
closedcephadm: commands fail with "ValueError: not enough values to unpack (expected 2, got 1)" when apparmor profiles file is empty
Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
cephadm
Target version:
-
% Done:
0%
Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Occurs with ceph/cephadm 16.2.1 running on a clean Debian 10.9 install.
The following error is from a failed OSD DriveSpec applied from the admin dashboard, but the failure is not specific to this command:
May 5, 2021, 1:29:18 AM [ERR] Failed to apply osd.dashboard-admin-1620170688575 spec DriveGroupSpec(name=dashboard-admin-1620170688575->placement=PlacementSpec(host_pattern='*'), service_id='dashboard-admin-1620170688575', service_type='osd', data_devices=DeviceSelection(size='8GB', all=False), osd_id_claims={}, unmanaged=False, filter_logic='AND', preview_only=False): cephadm exited with an error code: 1, stderr:Traceback (most recent call last):
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7931, in <module>
main()
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7919, in main
r = ctx.func(ctx)
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 1611, in _infer_fsid
return func(ctx)
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 1695, in _infer_image
return func(ctx)
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 4413, in command_ceph_volume
mounts = get_container_mounts(ctx, ctx.fsid, 'osd', None)
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 2292, in get_container_mounts
if HostFacts(ctx).selinux_enabled:
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6451, in selinux_enabled
return (self.kernel_security['type'] == 'SELinux') and \
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6434, in kernel_security
ret = _fetch_apparmor()
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6415, in _fetch_apparmor
item, mode = line.split(' ')
ValueError: not enough values to unpack (expected 2, got 1)
Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/serve.py", line 465, in _apply_all_services
if self._apply_service(spec):
File "/usr/share/ceph/mgr/cephadm/serve.py", line 522, in _apply_service
self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec))
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 68, in create_from_spec
ret = create_from_spec_one(self.prepare_drivegroup(drive_group))
File "/usr/share/ceph/mgr/cephadm/utils.py", line 79, in forall_hosts_wrapper
return CephadmOrchestrator.instance._worker_pool.map(do_work, vals)
File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get
raise self._value
File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/usr/share/ceph/mgr/cephadm/utils.py", line 73, in do_work
return f(*arg)
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 60, in create_from_spec_one
replace_osd_ids=osd_id_claims.get(host, []), env_vars=env_vars
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 86, in create_single_host
code, '\n'.join(err)))
RuntimeError: cephadm exited with an error code: 1, stderr:Traceback (most recent call last):
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7931, in <module>
main()
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7919, in main
r = ctx.func(ctx)
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 1611, in _infer_fsid
return func(ctx)
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 1695, in _infer_image
return func(ctx)
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 4413, in command_ceph_volume
mounts = get_container_mounts(ctx, ctx.fsid, 'osd', None)
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 2292, in get_container_mounts
if HostFacts(ctx).selinux_enabled:
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6451, in selinux_enabled
return (self.kernel_security['type'] == 'SELinux') and \
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6434, in kernel_security
ret = _fetch_apparmor()
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6415, in _fetch_apparmor
item, mode = line.split(' ')
ValueError: not enough values to unpack (expected 2, got 1)
Disabling kernel_security check does not disable this code-path, so the error still occurs:
May 5, 2021, 1:29:33 AM [INF] updated config check 'kernel_security' : disabled
May 5, 2021, 1:30:19 AM [ERR] cephadm exited with an error code: 1, stderr:Traceback (most recent call last):
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7931, in <module>
main()
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7919, in main
r = ctx.func(ctx)
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 1611, in _infer_fsid
return func(ctx)
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 1695, in _infer_image
return func(ctx)
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 4413, in command_ceph_volume
mounts = get_container_mounts(ctx, ctx.fsid, 'osd', None)
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 2292, in get_container_mounts
if HostFacts(ctx).selinux_enabled:
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6451, in selinux_enabled
return (self.kernel_security['type'] == 'SELinux') and \
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6434, in kernel_security
ret = _fetch_apparmor()
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6415, in _fetch_apparmor
item, mode = line.split(' ')
ValueError: not enough values to unpack (expected 2, got 1)
Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/serve.py", line 1172, in _remote_connection
yield (conn, connr)
File "/usr/share/ceph/mgr/cephadm/serve.py", line 1087, in _run_cephadm
code, '\n'.join(err)))
orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Traceback (most recent call last):
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7931, in <module>
main()
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7919, in main
r = ctx.func(ctx)
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 1611, in _infer_fsid
return func(ctx)
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 1695, in _infer_image
return func(ctx)
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 4413, in command_ceph_volume
mounts = get_container_mounts(ctx, ctx.fsid, 'osd', None)
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 2292, in get_container_mounts
if HostFacts(ctx).selinux_enabled:
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6451, in selinux_enabled
return (self.kernel_security['type'] == 'SELinux') and \
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6434, in kernel_security
ret = _fetch_apparmor()
File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6415, in _fetch_apparmor
item, mode = line.split(' ')
ValueError: not enough values to unpack (expected 2, got 1)
This occurs because the kernel reports apparmor support, but the apparmor profiles file is empty:
$ cat /sys/kernel/security/lsm
capability,yama,apparmor
$ ls -l /etc/apparmor
total 4
drwxr-xr-x 3 root root 4096 May 4 21:05 init
Empty profiles file:
$ sudo ls -l /sys/kernel/security/apparmor/profiles
-r--r--r-- 1 root root 0 May 4 20:07 /sys/kernel/security/apparmor/profiles
The offending code in cephadm assumes that an extant profiles file will contain at least one space separated line and does not handle an empty file:
def _fetch_apparmor() -> Dict[str, str]:
"""Read the apparmor profiles directly, returning an overview of AppArmor status"""
security = {}
for apparmor_path in HostFacts._apparmor_path_list:
if os.path.exists(apparmor_path):
security['type'] = 'AppArmor'
security['description'] = 'AppArmor: Enabled'
try:
profiles = read_file(['/sys/kernel/security/apparmor/profiles'])
except OSError:
pass
else:
summary = {} # type: Dict[str, int]
for line in profiles.split('\n'):
item, mode = line.split(' ')
mode = mode.strip('()')
if mode in summary:
summary[mode] += 1
else:
summary[mode] = 0
summary_str = ','.join(['{} {}'.format(v, k) for k, v in summary.items()])
security = {**security, **summary} # type: ignore
security['description'] += '({})'.format(summary_str)
return security
return {}
The workaround is to install the apparmor-profiles
package which populates /sys/kernel/security/apparmor/profiles
with the expected lines.
Actions