Ceph » Orchestrator

Occurs with ceph/cephadm 16.2.1 running on a clean Debian 10.9 install.

The following error is from a failed OSD DriveSpec applied from the admin dashboard, but the failure is not specific to this command:

May 5, 2021, 1:29:18 AM    [ERR]    Failed to apply osd.dashboard-admin-1620170688575 spec DriveGroupSpec(name=dashboard-admin-1620170688575->placement=PlacementSpec(host_pattern='*'), service_id='dashboard-admin-1620170688575', service_type='osd', data_devices=DeviceSelection(size='8GB', all=False), osd_id_claims={}, unmanaged=False, filter_logic='AND', preview_only=False): cephadm exited with an error code: 1, stderr:Traceback (most recent call last):
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7931, in <module>
    main()
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7919, in main
    r = ctx.func(ctx)
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 1611, in _infer_fsid
    return func(ctx)
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 1695, in _infer_image
    return func(ctx)
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 4413, in command_ceph_volume
    mounts = get_container_mounts(ctx, ctx.fsid, 'osd', None)
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 2292, in get_container_mounts
    if HostFacts(ctx).selinux_enabled:
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6451, in selinux_enabled
    return (self.kernel_security['type'] == 'SELinux') and \
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6434, in kernel_security
    ret = _fetch_apparmor()
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6415, in _fetch_apparmor
    item, mode = line.split(' ')
ValueError: not enough values to unpack (expected 2, got 1)
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/cephadm/serve.py", line 465, in _apply_all_services
    if self._apply_service(spec):
  File "/usr/share/ceph/mgr/cephadm/serve.py", line 522, in _apply_service
    self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec))
  File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 68, in create_from_spec
    ret = create_from_spec_one(self.prepare_drivegroup(drive_group))
  File "/usr/share/ceph/mgr/cephadm/utils.py", line 79, in forall_hosts_wrapper
    return CephadmOrchestrator.instance._worker_pool.map(do_work, vals)
  File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
  File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/usr/share/ceph/mgr/cephadm/utils.py", line 73, in do_work
    return f(*arg)
  File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 60, in create_from_spec_one
    replace_osd_ids=osd_id_claims.get(host, []), env_vars=env_vars
  File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 86, in create_single_host
    code, '\n'.join(err)))
RuntimeError: cephadm exited with an error code: 1, stderr:Traceback (most recent call last):
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7931, in <module>
    main()
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7919, in main
    r = ctx.func(ctx)
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 1611, in _infer_fsid
    return func(ctx)
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 1695, in _infer_image
    return func(ctx)
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 4413, in command_ceph_volume
    mounts = get_container_mounts(ctx, ctx.fsid, 'osd', None)
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 2292, in get_container_mounts
    if HostFacts(ctx).selinux_enabled:
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6451, in selinux_enabled
    return (self.kernel_security['type'] == 'SELinux') and \
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6434, in kernel_security
    ret = _fetch_apparmor()
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6415, in _fetch_apparmor
    item, mode = line.split(' ')
ValueError: not enough values to unpack (expected 2, got 1)

Disabling kernel_security check does not disable this code-path, so the error still occurs:

May 5, 2021, 1:29:33 AM    [INF]    updated config check 'kernel_security' : disabled

May 5, 2021, 1:30:19 AM    [ERR]    cephadm exited with an error code: 1, stderr:Traceback (most recent call last):
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7931, in <module>
    main()
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7919, in main
    r = ctx.func(ctx)
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 1611, in _infer_fsid
    return func(ctx)
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 1695, in _infer_image
    return func(ctx)
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 4413, in command_ceph_volume
    mounts = get_container_mounts(ctx, ctx.fsid, 'osd', None)
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 2292, in get_container_mounts
    if HostFacts(ctx).selinux_enabled:
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6451, in selinux_enabled
    return (self.kernel_security['type'] == 'SELinux') and \
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6434, in kernel_security
    ret = _fetch_apparmor()
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6415, in _fetch_apparmor
    item, mode = line.split(' ')
ValueError: not enough values to unpack (expected 2, got 1)
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/cephadm/serve.py", line 1172, in _remote_connection
    yield (conn, connr)
  File "/usr/share/ceph/mgr/cephadm/serve.py", line 1087, in _run_cephadm
    code, '\n'.join(err)))
orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Traceback (most recent call last):
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7931, in <module>
    main()
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 7919, in main
    r = ctx.func(ctx)
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 1611, in _infer_fsid
    return func(ctx)
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 1695, in _infer_image
    return func(ctx)
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 4413, in command_ceph_volume
    mounts = get_container_mounts(ctx, ctx.fsid, 'osd', None)
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 2292, in get_container_mounts
    if HostFacts(ctx).selinux_enabled:
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6451, in selinux_enabled
    return (self.kernel_security['type'] == 'SELinux') and \
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6434, in kernel_security
    ret = _fetch_apparmor()
  File "/var/lib/ceph/e5836280-ad05-11eb-95c1-08002772f6b2/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6415, in _fetch_apparmor
    item, mode = line.split(' ')
ValueError: not enough values to unpack (expected 2, got 1)

This occurs because the kernel reports apparmor support, but the apparmor profiles file is empty:

$ cat /sys/kernel/security/lsm 
capability,yama,apparmor

$ ls -l /etc/apparmor
total 4
drwxr-xr-x 3 root root 4096 May  4 21:05 init

Empty profiles file:

$ sudo ls -l /sys/kernel/security/apparmor/profiles
-r--r--r-- 1 root root 0 May  4 20:07 /sys/kernel/security/apparmor/profiles

The offending code in cephadm assumes that an extant profiles file will contain at least one space separated line and does not handle an empty file:

def _fetch_apparmor() -> Dict[str, str]:
            """Read the apparmor profiles directly, returning an overview of AppArmor status""" 
            security = {}
            for apparmor_path in HostFacts._apparmor_path_list:
                if os.path.exists(apparmor_path):
                    security['type'] = 'AppArmor'
                    security['description'] = 'AppArmor: Enabled'
                    try:
                        profiles = read_file(['/sys/kernel/security/apparmor/profiles'])
                    except OSError:
                        pass
                    else:
                        summary = {}  # type: Dict[str, int]
                        for line in profiles.split('\n'):
                            item, mode = line.split(' ')
                            mode = mode.strip('()')
                            if mode in summary:
                                summary[mode] += 1
                            else:
                                summary[mode] = 0
                        summary_str = ','.join(['{} {}'.format(v, k) for k, v in summary.items()])
                        security = {**security, **summary}  # type: ignore
                        security['description'] += '({})'.format(summary_str)

                    return security
            return {}

The workaround is to install the apparmor-profiles package which populates /sys/kernel/security/apparmor/profiles with the expected lines.