Project

General

Profile

Actions
Copy link

Bug #54620

closed

failed to write ceph conf due to permission issue after ceph bootstrapped with ssh-user and ssh-keys option.

Added by SUNIL KUMAR NAGARAJU about 2 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Adam King
Category:
cephadm
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
quincy
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
Ceph - v17.0.0
ceph-qa-suite:
Pull request ID:
45735
Crash signature (v1):
Crash signature (v2):

Description

Bootstrap with ssh-user and ssh-keys options

$ cephadm --verbose --image quay.ceph.io/ceph-ci/ceph:quincy bootstrap --registry-json /tmp/tmp3ewifw5b.json --dashboard-key /root/dashboard.key --dashboard-crt /root/dashboard.crt --mon-ip 10.0.211.167 --initial-dashboard-user admin123 --initial-dashboard-password admin@123 --dashboard-password-noupdate --fsid f64f341c-655d-11eb-8778-fa163e914bcc --skip-monitoring-stack --orphan-initial-daemons --ssl-dashboard-port 8445 --ssh-user cephuser --ssh-public-key /home/cephuser/.ssh/id_rsa.pub --ssh-private-key /home/cephuser/.ssh/id_rsa --apply-spec /tmp/tmp129jpady.yaml
...
...
For more information see:

        https://docs.ceph.com/docs/master/mgr/telemetry/

2022-03-18 08:36:23,567 7f5e83037b80 INFO Bootstrap complete.

MGR logs:

2022-03-18T12:42:44.683+0000 7f84d37dc700  0 [cephadm INFO cephadm.serve] Updating ceph-sunil01-yf5qw5-node1-installer:/etc/ceph/ceph.conf
2022-03-18T12:42:44.683+0000 7f84d37dc700  0 log_channel(cephadm) log [INF] : Updating ceph-sunil01-yf5qw5-node1-installer:/etc/ceph/ceph.conf
2022-03-18T12:42:44.686+0000 7f84dcfef700  0 log_channel(cluster) log [DBG] : pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail
2022-03-18T12:42:44.915+0000 7f84cefd3700  0 [cephadm ERROR cephadm.ssh] Unable to write ceph-sunil01-yf5qw5-node1-installer:/etc/ceph/ceph.conf: scp: /tmp/etc/ceph/ceph.conf.new: Permission denied
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/cephadm/ssh.py", line 221, in _write_remote_file
    await asyncssh.scp(f.name, (conn, tmp_path))
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp
    await source.run(srcpath)
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run
    self.handle_error(exc)
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error
    raise exc from None
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run
    await self._send_files(path, b'')
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in _send_files
    self.handle_error(exc)
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error
    raise exc from None
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in _send_files
    await self._send_file(srcpath, dstpath, attrs)
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in _send_file
    await self._make_cd_request(b'C', attrs, size, srcpath)
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in _make_cd_request
    self._fs.basename(path))
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in make_request
    raise exc
asyncssh.sftp.SFTPFailure: scp: /tmp/etc/ceph/ceph.conf.new: Permission denied
2022-03-18T12:42:44.915+0000 7f84cefd3700 -1 log_channel(cephadm) log [ERR] : Unable to write ceph-sunil01-yf5qw5-node1-installer:/etc/ceph/ceph.conf: scp: /tmp/etc/ceph/ceph.conf.new: Permission denied
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/cephadm/ssh.py", line 221, in _write_remote_file
    await asyncssh.scp(f.name, (conn, tmp_path))
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp
    await source.run(srcpath)
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run
    self.handle_error(exc)
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error
    raise exc from None
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run
    await self._send_files(path, b'')
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in _send_files
    self.handle_error(exc)
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error
    raise exc from None
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in _send_files
    await self._send_file(srcpath, dstpath, attrs)
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in _send_file
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in _send_file
    await self._make_cd_request(b'C', attrs, size, srcpath)
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in _make_cd_request
    self._fs.basename(path))
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in make_request
    raise exc
asyncssh.sftp.SFTPFailure: scp: /tmp/etc/ceph/ceph.conf.new: Permission denied
2022-03-18T12:42:44.916+0000 7f84d37dc700  0 [cephadm ERROR cephadm.utils] executing refresh((['ceph-sunil01-yf5qw5-node1-installer', 'ceph-sunil01-yf5qw5-node2', 'ceph-sunil01-yf5qw5-node3'],)) failed.
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/cephadm/ssh.py", line 221, in _write_remote_file
    await asyncssh.scp(f.name, (conn, tmp_path))
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp
    await source.run(srcpath)
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run
    self.handle_error(exc)
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error
    raise exc from None
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run
    await self._send_files(path, b'')
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in _send_files
    self.handle_error(exc)
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error
    raise exc from None
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in _send_files
    await self._send_file(srcpath, dstpath, attrs)
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in _send_file
    await self._make_cd_request(b'C', attrs, size, srcpath)
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in _make_cd_request
    self._fs.basename(path))
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in make_request
    raise exc
asyncssh.sftp.SFTPFailure: scp: /tmp/etc/ceph/ceph.conf.new: Permission denied

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work
    return f(*arg)
  File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh
    self._write_client_files(client_files, host)
  File "/usr/share/ceph/mgr/cephadm/serve.py", line 1052, in _write_client_files
    self.mgr.ssh.write_remote_file(host, path, content, mode, uid, gid)
  File "/usr/share/ceph/mgr/cephadm/ssh.py", line 238, in write_remote_file
    host, path, content, mode, uid, gid, addr))
  File "/usr/share/ceph/mgr/cephadm/module.py", line 569, in wait_async
    return self.event_loop.get_result(coro)
  File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result
    raise self._exception
  File "/usr/share/ceph/mgr/cephadm/ssh.py", line 226, in _write_remote_file
    raise OrchestratorError(msg)
orchestrator._interface.OrchestratorError: Unable to write ceph-sunil01-yf5qw5-node1-installer:/etc/ceph/ceph.conf: scp: /tmp/etc/ceph/ceph.conf.new: Permission denied
2022-03-18T12:42:44.916+0000 7f84d37dc700 -1 log_channel(cephadm) log [ERR] : executing refresh((['ceph-sunil01-yf5qw5-node1-installer', 'ceph-sunil01-yf5qw5-node2', 'ceph-sunil01-yf5qw5-node3'],)) failed.
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/cephadm/ssh.py", line 221, in _write_remote_file
    await asyncssh.scp(f.name, (conn, tmp_path))
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp
    await source.run(srcpath)
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run
    self.handle_error(exc)
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error
    raise exc from None
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run
    await self._send_files(path, b'')
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in _send_files
    self.handle_error(exc)
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error
    raise exc from None
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in _send_files
    await self._send_file(srcpath, dstpath, attrs)
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in _send_file
    await self._make_cd_request(b'C', attrs, size, srcpath)
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in _make_cd_request
    self._fs.basename(path))
  File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in make_request
    raise exc
asyncssh.sftp.SFTPFailure: scp: /tmp/etc/ceph/ceph.conf.new: Permission denied

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work
    return f(*arg)
  File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh
    self._write_client_files(client_files, host)
  File "/usr/share/ceph/mgr/cephadm/serve.py", line 1052, in _write_client_files
    self.mgr.ssh.write_remote_file(host, path, content, mode, uid, gid)
  File "/usr/share/ceph/mgr/cephadm/ssh.py", line 238, in write_remote_file
    host, path, content, mode, uid, gid, addr))
  File "/usr/share/ceph/mgr/cephadm/module.py", line 569, in wait_async
    return self.event_loop.get_result(coro)
  File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result
    return asyncio.run_coroutine_threadsafe(coro, self._loop).result()
  File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/share/ceph/mgr/cephadm/ssh.py", line 226, in _write_remote_file
    raise OrchestratorError(msg)
orchestrator._interface.OrchestratorError: Unable to write ceph-sunil01-yf5qw5-node1-installer:/etc/ceph/ceph.conf: scp: /tmp/etc/ceph/ceph.conf.new: Permission denied
2022-03-18T12:42:46.687+0000 7f84dcfef700  0 log_channel(cluster) log [DBG] : pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail
2022-03-18T12:42:48.687+0000 7f84dcfef700  0 log_channel(cluster) log [DBG] : pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail

Build:
https://2.chacra.ceph.com/repos/ceph/quincy/1b309fef1b6e8e1f4cfe6a3c9ccacc08a6a844aa/centos/8/flavors/default/repo
quay.ceph.io/ceph-ci/ceph:quincy

Actions
Copy link
 #1

Updated by Adam King about 2 years ago

Did some digging into this and it looks like for most commands we do over ssh in quincy we prepend "sudo" but specifically for writing files we use asyncssh's scp command and no attempt is made to do so with higher privileges. So if the default ssh user we use isn't root we might be running this with too low of permissions. At least that's what I think is going on right now.

Actions
Copy link
 #2

Updated by Adam King about 2 years ago

  • Assignee set to Adam King
Actions
Copy link
 #3

Updated by Christoph Glaubitz about 2 years ago

Hey Adam,

came along this issue during upgrade of a 16.2.7 to 17.1.0 with different ssh_user as well.
My patch would be to change permissions to the new file to the ssh_user, write/scp the file, and restore the real permissions.

See https://github.com/ceph/ceph/pull/45735

Would love to have your opinion on it!

Actions
Copy link
 #4

Updated by Adam King about 2 years ago

  • Status changed from New to Pending Backport
  • Backport set to quincy
  • Pull request ID set to 45735
Actions
Copy link
 #5

Updated by Redouane Kachach Elhichou almost 2 years ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link

Also available in: Atom PDF