Bug #50691

cephadm: bootstrap fails with "IndexError: list index out of range" during cephadm set-user

Added by Sam Overton about 1 month ago. Updated 23 days ago.

Target version:
% Done:


Community (user)
3 - minor
Affected Versions:
Pull request ID:
Crash signature (v1):
Crash signature (v2):


Running on a cleanly installed Debian 10.9 host with ceph/cephadm 16.2.3.

The same command in 16.2.1, running on the same base Debian image completes successfully, so this is a regression.

$ sudo cephadm bootstrap --ssh-user cephadm --mon-ip
Verifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...
Unit ntp.service is enabled and running
Repeating the final host check...
podman|docker (/usr/bin/docker) is present
systemctl is present
lvcreate is present
Unit ntp.service is enabled and running
Host looks OK
Cluster fsid: 2f6bc44e-af65-11eb-9236-08002772f6b2
Verifying IP port 3300 ...
Verifying IP port 6789 ...
Mon IP is in CIDR network
- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network
Pulling container image
Ceph version: ceph version 16.2.3 (381b476cb3900f9a92eb95d03b4850b953cfd79a) pacific (stable)
Extracting ceph user uid/gid from container image...
Creating initial keys...
Creating initial monmap...
Creating mon...
Waiting for mon to start...
Waiting for mon...
mon is available
Assimilating anything we can from ceph.conf...
Generating new minimal ceph.conf...
Restarting the monitor...
Setting mon public_network to
Wrote config to /etc/ceph/ceph.conf
Wrote keyring to /etc/ceph/ceph.client.admin.keyring
Creating mgr...
Verifying port 9283 ...
Waiting for mgr to start...
Waiting for mgr...
mgr not available, waiting (1/15)...
mgr not available, waiting (2/15)...
mgr not available, waiting (3/15)...
mgr is available
Enabling cephadm module...
Waiting for the mgr to restart...
Waiting for mgr epoch 5...
mgr epoch 5 is available
Setting orchestrator backend to cephadm...
Non-zero exit code 22 from /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e -e NODE_NAME=cephhost -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2f6bc44e-af65-11eb-9236-08002772f6b2:/var/log/ceph:z -v /tmp/ceph-tmpi7r_6_3q:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpm1a5ietn:/etc/ceph/ceph.conf:z cephadm set-user cephadm
/usr/bin/ceph: stderr Error EINVAL: Traceback (most recent call last):
/usr/bin/ceph: stderr   File "/usr/share/ceph/mgr/", line 1335, in _handle_command
/usr/bin/ceph: stderr     return self.handle_command(inbuf, cmd)
/usr/bin/ceph: stderr   File "/usr/share/ceph/mgr/orchestrator/", line 167, in handle_command
/usr/bin/ceph: stderr     return dispatch[cmd['prefix']].call(self, cmd, inbuf)
/usr/bin/ceph: stderr   File "/usr/share/ceph/mgr/", line 389, in call
/usr/bin/ceph: stderr     return self.func(mgr, **kwargs)
/usr/bin/ceph: stderr   File "/usr/share/ceph/mgr/orchestrator/", line 107, in <lambda>
/usr/bin/ceph: stderr     wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs)  # noqa: E731
/usr/bin/ceph: stderr   File "/usr/share/ceph/mgr/orchestrator/", line 96, in wrapper
/usr/bin/ceph: stderr     return func(*args, **kwargs)
/usr/bin/ceph: stderr   File "/usr/share/ceph/mgr/cephadm/", line 836, in set_ssh_user
/usr/bin/ceph: stderr     host = self.cache.get_hosts()[0]
/usr/bin/ceph: stderr IndexError: list index out of range
/usr/bin/ceph: stderr 
Traceback (most recent call last):
  File "/usr/sbin/cephadm", line 8029, in <module>
  File "/usr/sbin/cephadm", line 8017, in main
    r = ctx.func(ctx)
  File "/usr/sbin/cephadm", line 1760, in _default_image
    return func(ctx)
  File "/usr/sbin/cephadm", line 4066, in command_bootstrap
    prepare_ssh(ctx, cli, wait_for_mgr_restart)
  File "/usr/sbin/cephadm", line 3673, in prepare_ssh
    cli(['cephadm', 'set-user', ctx.ssh_user])
  File "/usr/sbin/cephadm", line 4010, in cli
  File "/usr/sbin/cephadm", line 3225, in run
    desc=self.entrypoint, timeout=timeout)
  File "/usr/sbin/cephadm", line 1464, in call_throws
    raise RuntimeError('Failed command: %s' % ' '.join(command))
RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e -e NODE_NAME=cephhost -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2f6bc44e-af65-11eb-9236-08002772f6b2:/var/log/ceph:z -v /tmp/ceph-tmpi7r_6_3q:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpm1a5ietn:/etc/ceph/ceph.conf:z cephadm set-user cephadm


#1 Updated by Sebastian Wagner about 1 month ago

  • Priority changed from Normal to Urgent

#3 Updated by Sebastian Wagner about 1 month ago

  • Assignee set to Sebastian Wagner

#4 Updated by Sebastian Wagner about 1 month ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 41278

#5 Updated by Sebastian Wagner about 1 month ago

workaround: Do not specify the ssh user when bootstrapping, but later on.

#6 Updated by Kefu Chai 23 days ago

  • Status changed from Fix Under Review to Resolved

Also available in: Atom PDF