Bug #37503

Audit log: mgr module passwords set on CLI written as plaintext in log files

Added by Tim Serong almost 2 years ago. Updated 4 months ago.

Target version:
% Done:


2 - major
Affected Versions:
Pull request ID:
Crash signature:


A number of mgr modules need passwords set for one reason or another, either to authenticate with external systems (deepsea, influx, diskprediction), or to define credentials for users of those modules (dashboard, restful).

In all cases, these passwords are set from the command line, either via module-specific commands (`ceph dashboard ac-user-create`, `deepsea config-set salt_api_password`, etc.) or via `ceph config set` with some particular key (e.g.: mgr/influx/passsword)

All module-specific commands go through DaemonServer::_handle_command(), which then logs the command via audit_clog->debug() (or audit_clog->info() in case of access denied). This all ends up written to /var/log/ceph/ceph-mgr.$ID.log, which is world-readable, e.g.:

2018-12-03 10:45:28.864 7f67e7f8f700  0 log_channel(audit) log [DBG] : from='client.343880' entity='client.admin' cmd=[{"prefix": "deepsea config-set", "key": "salt_api_password", "value": "foo", "target": ["mgr", ""]}]: dispatch

Additionally, anything that results in a "config set" lands in the mon log, e.g.:

2018-12-03 10:45:28.881552 [INF]  from='mgr.295252' entity='mgr.data1' cmd='[{"prefix":"config set","who":"mgr","name":"mgr/deepsea/salt_api_password","value":"foo"}]': finished 

This also appears in the Audit log in the Dashboard.

Some things that land in the mon log probably don't matter; for any module that hashes passwords before saving them, only the hashed password should land in the mon log. But there's still the problem of the CLI commands in the mgr log, and in any case, modules that need to authenticate with external services will need to store plaintext passwords.

ISTM we need to either never log these things, or somehow keep the command logging, but filter the passwords out, so it renders the value as "*****" instead of the actual password.

I'm not sure how best to approach this, given the way command logging is structured. At the point commands are logged, the commands themselves are just strings. Admittedly, they're strings of JSON, but they're effectively opaque at that point - we'd have to parse the JSON, then look for things that might be passwords, blank them out, and turn the whole lot back into a string. Yuck.

Related issues

Related to mgr - Bug #41320: mgr/dashboard: passwords and other sensitive information is written to logs Resolved


#1 Updated by Sebastian Wagner almost 2 years ago

I would expect the `diskprediction_cloud` module to also be affected by this. See

#2 Updated by Sebastian Wagner about 1 year ago

  • Related to Bug #41320: mgr/dashboard: passwords and other sensitive information is written to logs added

#3 Updated by Sebastian Wagner about 1 year ago

  • Status changed from New to 12

#4 Updated by Sebastian Wagner about 1 year ago

  • Project changed from mgr to Ceph
  • Subject changed from mgr module passwords set on CLI written as plaintext in log files to Audit log: mgr module passwords set on CLI written as plaintext in log files

#5 Updated by Lenz Grimmer about 1 year ago

  • Backport set to nautilus

#6 Updated by Patrick Donnelly 12 months ago

  • Status changed from 12 to New

#7 Updated by Lenz Grimmer 10 months ago

  • Severity changed from 3 - minor to 2 - major

#8 Updated by Juan Miguel Olmo Martínez 9 months ago

I think that is going to be very hard to detect when a command parameter should be hidden or not. Because trying to deduce this point using only the name of the parameter probably will drive us to wrong decisions.

In my view, we have two options:
1. To have a predefined list of parameters names whose value must be hidden always in the logs.
2. To add a new optional parameter for all the Ceph manager commands ( --nolog for example ) to avoid to log the command (or just log the command without parameters)

#9 Updated by Ernesto Puerta 9 months ago

For the current log trace:

cmd=[{"prefix": "dashboard ac-user-create", "force_password": true, "username": "admin", "password": "admin", "rolename": "administrator", "target": ["mon-mgr", ""]}]
I see some ways of dealing with it:
  • Forcing sensitive information not to be passed by command line (e.g.: read from file): as commands are kept in shell history and procfs also keeps cmdline in memory.
  • Per-command: Adding a new MonCommand FLAG (src/mon/MonCommand.h), like FLAG_SANITIZE_CMDARGS. It'd print the command trace but with everything except the prefix redacted out:
    cmd=[{"prefix": "dashboard ac-user-create", "force_password": SANITIZED, "username": SANITIZED, "password": SANITIZED, "rolename": SANITIZED, "target": ["mon-mgr", ""]}]
  • Per-argument 1 (new flag) (as we may still want to log some command args): Adding a new command option to cmddesc, like name=password,type=CephString,sanitized=true
    cmd=[{"prefix": "dashboard ac-user-create", "force_password": true, "username": "admin", "password": SANITIZED, "rolename": "administrator", "target": ["mon-mgr", ""]}]
  • Per-argument 2 (new Ceph Type): Adding a new CephType, like name=password,type=CephSecret
    cmd=[{"prefix": "dashboard ac-user-create", "force_password": true, "username": "admin", "password": SANITIZED, "rolename": "administrator", "target": ["mon-mgr", ""]}]
  • New Ceph Option Flag: Adding a new flag for Ceph Options (like FLAG_DO_NOT_LOG):
    cmd=[{"prefix": "dashboard ac-user-create", "force_password": true, "username": "admin", "password": SANITIZED, "rolename": "administrator", "target": ["mon-mgr", ""]}]

This would be required both for ceph-mgr commands and also for ceph-mon config/config-key commands, as these Mgr commands end up stored in the Mon KV store.

#10 Updated by Sage Weil 9 months ago

  • Priority changed from Normal to Urgent

#11 Updated by Sage Weil 9 months ago

  • Priority changed from Urgent to High

#12 Updated by Travis Nielsen 4 months ago

Rook really would benefit from at least a simple solution that sanitizes the log for setting the dashboard password. Rook doesn't need to worry about shell history either.

What's the simplest solution to avoid logging the dashboard username/password? Couldn't the log at least be sanitized if the prefix matches certain commands like "dashboard set-login-credentials"? Let's at least get something simple for the commands that currently exist instead of waiting for the perfect solution.

Also available in: Atom PDF