Bug #37503
closedAudit log: mgr module passwords set on CLI written as plaintext in log files
100%
Description
A number of mgr modules need passwords set for one reason or another, either to authenticate with external systems (deepsea, influx, diskprediction), or to define credentials for users of those modules (dashboard, restful).
In all cases, these passwords are set from the command line, either via module-specific commands (`ceph dashboard ac-user-create`
, `deepsea config-set salt_api_password`
, etc.) or via `ceph config set`
with some particular key (e.g.: mgr/influx/passsword)
All module-specific commands go through DaemonServer::_handle_command()
, which then logs the command via audit_clog->debug()
(or audit_clog->info()
in case of access denied). This all ends up written to /var/log/ceph/ceph-mgr.$ID.log
, which is world-readable, e.g.:
2018-12-03 10:45:28.864 7f67e7f8f700 0 log_channel(audit) log [DBG] : from='client.343880 172.16.1.254:39896/3560370796' entity='client.admin' cmd=[{"prefix": "deepsea config-set", "key": "salt_api_password", "value": "foo", "target": ["mgr", ""]}]: dispatch
Additionally, anything that results in a "config set" lands in the mon log, e.g.:
2018-12-03 10:45:28.881552 [INF] from='mgr.295252 172.16.1.21:56636/175641' entity='mgr.data1' cmd='[{"prefix":"config set","who":"mgr","name":"mgr/deepsea/salt_api_password","value":"foo"}]': finished
This also appears in the Audit log in the Dashboard.
Some things that land in the mon log probably don't matter; for any module that hashes passwords before saving them, only the hashed password should land in the mon log. But there's still the problem of the CLI commands in the mgr log, and in any case, modules that need to authenticate with external services will need to store plaintext passwords.
ISTM we need to either never log these things, or somehow keep the command logging, but filter the passwords out, so it renders the value as "*****" instead of the actual password.
I'm not sure how best to approach this, given the way command logging is structured. At the point commands are logged, the commands themselves are just strings. Admittedly, they're strings of JSON, but they're effectively opaque at that point - we'd have to parse the JSON, then look for things that might be passwords, blank them out, and turn the whole lot back into a string. Yuck.
Updated by Sebastian Wagner over 5 years ago
I would expect the `diskprediction_cloud` module to also be affected by this. See http://docs.ceph.com/docs/master/mgr/diskprediction/#connection-settings
Updated by Sebastian Wagner over 4 years ago
- Related to Bug #41320: mgr/dashboard: passwords and other sensitive information is written to logs added
Updated by Sebastian Wagner over 4 years ago
- Project changed from mgr to Ceph
- Subject changed from mgr module passwords set on CLI written as plaintext in log files to Audit log: mgr module passwords set on CLI written as plaintext in log files
Updated by Lenz Grimmer over 4 years ago
- Severity changed from 3 - minor to 2 - major
Updated by Juan Miguel Olmo MartÃnez about 4 years ago
I think that is going to be very hard to detect when a command parameter should be hidden or not. Because trying to deduce this point using only the name of the parameter probably will drive us to wrong decisions.
In my view, we have two options:
1. To have a predefined list of parameters names whose value must be hidden always in the logs.
2. To add a new optional parameter for all the Ceph manager commands ( --nolog for example ) to avoid to log the command (or just log the command without parameters)
Updated by Ernesto Puerta about 4 years ago
For the current log trace:
cmd=[{"prefix": "dashboard ac-user-create", "force_password": true, "username": "admin", "password": "admin", "rolename": "administrator", "target": ["mon-mgr", ""]}]
I see some ways of dealing with it:
- Forcing sensitive information not to be passed by command line (e.g.: read from file): as commands are kept in shell history and procfs also keeps cmdline in memory.
- Per-command: Adding a new MonCommand FLAG (
src/mon/MonCommand.h
), likeFLAG_SANITIZE_CMDARGS
. It'd print the command trace but with everything except theprefix
redacted out:cmd=[{"prefix": "dashboard ac-user-create", "force_password": SANITIZED, "username": SANITIZED, "password": SANITIZED, "rolename": SANITIZED, "target": ["mon-mgr", ""]}]
- Per-argument 1 (new flag) (as we may still want to log some command args): Adding a new command option to cmddesc, like
name=password,type=CephString,sanitized=true
cmd=[{"prefix": "dashboard ac-user-create", "force_password": true, "username": "admin", "password": SANITIZED, "rolename": "administrator", "target": ["mon-mgr", ""]}]
- Per-argument 2 (new Ceph Type): Adding a new CephType, like
name=password,type=CephSecret
cmd=[{"prefix": "dashboard ac-user-create", "force_password": true, "username": "admin", "password": SANITIZED, "rolename": "administrator", "target": ["mon-mgr", ""]}]
- New Ceph Option Flag: Adding a new flag for Ceph Options (like
FLAG_DO_NOT_LOG
):cmd=[{"prefix": "dashboard ac-user-create", "force_password": true, "username": "admin", "password": SANITIZED, "rolename": "administrator", "target": ["mon-mgr", ""]}]
This would be required both for ceph-mgr commands and also for ceph-mon config/config-key commands, as these Mgr commands end up stored in the Mon KV store.
Updated by Travis Nielsen almost 4 years ago
Rook really would benefit from at least a simple solution that sanitizes the log for setting the dashboard password. Rook doesn't need to worry about shell history either.
What's the simplest solution to avoid logging the dashboard username/password? Couldn't the log at least be sanitized if the prefix matches certain commands like "dashboard set-login-credentials"? Let's at least get something simple for the commands that currently exist instead of waiting for the perfect solution.
Updated by Neha Ojha over 3 years ago
- Status changed from New to Fix Under Review
- Assignee set to Neha Ojha
- Backport changed from nautilus to nautilus,octopus
- Pull request ID set to 38479
Updated by Neha Ojha over 3 years ago
- Status changed from Fix Under Review to Pending Backport
Updated by Backport Bot over 3 years ago
- Copied to Backport #48614: nautilus: Audit log: mgr module passwords set on CLI written as plaintext in log files added
Updated by Backport Bot over 3 years ago
- Copied to Backport #48615: octopus: Audit log: mgr module passwords set on CLI written as plaintext in log files added
Updated by Ernesto Puerta over 3 years ago
- Blocks Bug #48623: mgr/dashboard: Dashboard logs e2e tests are failing added
Updated by Nathan Cutler over 3 years ago
- Status changed from Pending Backport to Resolved
While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".