Bug #37503

Audit log: mgr module passwords set on CLI written as plaintext in log files

Added by Tim Serong over 1 year ago. Updated 4 months ago.

Target version:
% Done:


2 - major
Affected Versions:
Pull request ID:
Crash signature:


A number of mgr modules need passwords set for one reason or another, either to authenticate with external systems (deepsea, influx, diskprediction), or to define credentials for users of those modules (dashboard, restful).

In all cases, these passwords are set from the command line, either via module-specific commands (`ceph dashboard ac-user-create`, `deepsea config-set salt_api_password`, etc.) or via `ceph config set` with some particular key (e.g.: mgr/influx/passsword)

All module-specific commands go through DaemonServer::_handle_command(), which then logs the command via audit_clog->debug() (or audit_clog->info() in case of access denied). This all ends up written to /var/log/ceph/ceph-mgr.$ID.log, which is world-readable, e.g.:

2018-12-03 10:45:28.864 7f67e7f8f700  0 log_channel(audit) log [DBG] : from='client.343880' entity='client.admin' cmd=[{"prefix": "deepsea config-set", "key": "salt_api_password", "value": "foo", "target": ["mgr", ""]}]: dispatch

Additionally, anything that results in a "config set" lands in the mon log, e.g.:

2018-12-03 10:45:28.881552 [INF]  from='mgr.295252' entity='mgr.data1' cmd='[{"prefix":"config set","who":"mgr","name":"mgr/deepsea/salt_api_password","value":"foo"}]': finished 

This also appears in the Audit log in the Dashboard.

Some things that land in the mon log probably don't matter; for any module that hashes passwords before saving them, only the hashed password should land in the mon log. But there's still the problem of the CLI commands in the mgr log, and in any case, modules that need to authenticate with external services will need to store plaintext passwords.

ISTM we need to either never log these things, or somehow keep the command logging, but filter the passwords out, so it renders the value as "*****" instead of the actual password.

I'm not sure how best to approach this, given the way command logging is structured. At the point commands are logged, the commands themselves are just strings. Admittedly, they're strings of JSON, but they're effectively opaque at that point - we'd have to parse the JSON, then look for things that might be passwords, blank them out, and turn the whole lot back into a string. Yuck.

Related issues

Related to mgr - Bug #41320: mgr/dashboard: passwords and other sensitive information is written to logs Resolved 08/16/2019


#1 Updated by Sebastian Wagner over 1 year ago

I would expect the `diskprediction_cloud` module to also be affected by this. See

#2 Updated by Sebastian Wagner 10 months ago

  • Related to Bug #41320: mgr/dashboard: passwords and other sensitive information is written to logs added

#3 Updated by Sebastian Wagner 10 months ago

  • Status changed from New to 12

#4 Updated by Sebastian Wagner 10 months ago

  • Project changed from mgr to Ceph
  • Subject changed from mgr module passwords set on CLI written as plaintext in log files to Audit log: mgr module passwords set on CLI written as plaintext in log files

#5 Updated by Lenz Grimmer 10 months ago

  • Backport set to nautilus

#6 Updated by Patrick Donnelly 7 months ago

  • Status changed from 12 to New

#7 Updated by Lenz Grimmer 6 months ago

  • Severity changed from 3 - minor to 2 - major

#8 Updated by Juan Miguel Olmo Martínez 4 months ago

I think that is going to be very hard to detect when a command parameter should be hidden or not. Because trying to deduce this point using only the name of the parameter probably will drive us to wrong decisions.

In my view, we have two options:
1. To have a predefined list of parameters names whose value must be hidden always in the logs.
2. To add a new optional parameter for all the Ceph manager commands ( --nolog for example ) to avoid to log the command (or just log the command without parameters)

#9 Updated by Ernesto Puerta 4 months ago

For the current log trace:

cmd=[{"prefix": "dashboard ac-user-create", "force_password": true, "username": "admin", "password": "admin", "rolename": "administrator", "target": ["mon-mgr", ""]}]
I see 3 ways of dealing with it:
  • Forcing sensitive information not to be passed by command line (e.g.: read from file): as commands are kept in shell history and procfs also keeps cmdline in memory.
  • Per-command: Adding a new MonCommand FLAG (src/mon/MonCommand.h), like FLAG_SANITIZE_CMDARGS. It'd print the command trace but with everything except the prefix redacted out:
    cmd=[{"prefix": "dashboard ac-user-create", "force_password": SANITIZED, "username": SANITIZED, "password": SANITIZED, "rolename": SANITIZED, "target": ["mon-mgr", ""]}]
  • Per-argument (as we may still want to log some command args): Adding a new command option to cmddesc, like name=password,type=CephString,sanitized=true
    cmd=[{"prefix": "dashboard ac-user-create", "force_password": true, "username": "admin", "password": SANITIZED, "rolename": "administrator", "target": ["mon-mgr", ""]}]

However, none of the above would fix this when directly setting config options via config set unless that command log is fully sanitized.

#10 Updated by Sage Weil 4 months ago

  • Priority changed from Normal to Urgent

#11 Updated by Sage Weil 4 months ago

  • Priority changed from Urgent to High

Also available in: Atom PDF