Project

General

Profile

Actions

Bug #37503

closed

Audit log: mgr module passwords set on CLI written as plaintext in log files

Added by Tim Serong over 5 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
-
% Done:

100%

Source:
Tags:
Backport:
nautilus,octopus
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

A number of mgr modules need passwords set for one reason or another, either to authenticate with external systems (deepsea, influx, diskprediction), or to define credentials for users of those modules (dashboard, restful).

In all cases, these passwords are set from the command line, either via module-specific commands (`ceph dashboard ac-user-create`, `deepsea config-set salt_api_password`, etc.) or via `ceph config set` with some particular key (e.g.: mgr/influx/passsword)

All module-specific commands go through DaemonServer::_handle_command(), which then logs the command via audit_clog->debug() (or audit_clog->info() in case of access denied). This all ends up written to /var/log/ceph/ceph-mgr.$ID.log, which is world-readable, e.g.:

2018-12-03 10:45:28.864 7f67e7f8f700  0 log_channel(audit) log [DBG] : from='client.343880 172.16.1.254:39896/3560370796' entity='client.admin' cmd=[{"prefix": "deepsea config-set", "key": "salt_api_password", "value": "foo", "target": ["mgr", ""]}]: dispatch

Additionally, anything that results in a "config set" lands in the mon log, e.g.:

2018-12-03 10:45:28.881552 [INF]  from='mgr.295252 172.16.1.21:56636/175641' entity='mgr.data1' cmd='[{"prefix":"config set","who":"mgr","name":"mgr/deepsea/salt_api_password","value":"foo"}]': finished 

This also appears in the Audit log in the Dashboard.

Some things that land in the mon log probably don't matter; for any module that hashes passwords before saving them, only the hashed password should land in the mon log. But there's still the problem of the CLI commands in the mgr log, and in any case, modules that need to authenticate with external services will need to store plaintext passwords.

ISTM we need to either never log these things, or somehow keep the command logging, but filter the passwords out, so it renders the value as "*****" instead of the actual password.

I'm not sure how best to approach this, given the way command logging is structured. At the point commands are logged, the commands themselves are just strings. Admittedly, they're strings of JSON, but they're effectively opaque at that point - we'd have to parse the JSON, then look for things that might be passwords, blank them out, and turn the whole lot back into a string. Yuck.


Subtasks 1 (0 open1 closed)

Dashboard - Subtask #48355: mgr/dashboard: CLI commands: read passwords from fileResolvedAlfonso Martínez

Actions

Related issues 4 (0 open4 closed)

Related to Dashboard - Bug #41320: mgr/dashboard: passwords and other sensitive information is written to logsResolvedKefu Chai

Actions
Blocks Dashboard - Bug #48623: mgr/dashboard: Dashboard logs e2e tests are failingResolvedNizamudeen A

Actions
Copied to Ceph - Backport #48614: nautilus: Audit log: mgr module passwords set on CLI written as plaintext in log filesResolvedNeha OjhaActions
Copied to Ceph - Backport #48615: octopus: Audit log: mgr module passwords set on CLI written as plaintext in log filesResolvedNathan CutlerActions
Actions #1

Updated by Sebastian Wagner over 5 years ago

I would expect the `diskprediction_cloud` module to also be affected by this. See http://docs.ceph.com/docs/master/mgr/diskprediction/#connection-settings

Actions #2

Updated by Sebastian Wagner over 4 years ago

  • Related to Bug #41320: mgr/dashboard: passwords and other sensitive information is written to logs added
Actions #3

Updated by Sebastian Wagner over 4 years ago

  • Status changed from New to 12
Actions #4

Updated by Sebastian Wagner over 4 years ago

  • Project changed from mgr to Ceph
  • Subject changed from mgr module passwords set on CLI written as plaintext in log files to Audit log: mgr module passwords set on CLI written as plaintext in log files
Actions #5

Updated by Lenz Grimmer over 4 years ago

  • Backport set to nautilus
Actions #6

Updated by Patrick Donnelly over 4 years ago

  • Status changed from 12 to New
Actions #7

Updated by Lenz Grimmer over 4 years ago

  • Severity changed from 3 - minor to 2 - major
Actions #8

Updated by Juan Miguel Olmo Martínez about 4 years ago

I think that is going to be very hard to detect when a command parameter should be hidden or not. Because trying to deduce this point using only the name of the parameter probably will drive us to wrong decisions.

In my view, we have two options:
1. To have a predefined list of parameters names whose value must be hidden always in the logs.
2. To add a new optional parameter for all the Ceph manager commands ( --nolog for example ) to avoid to log the command (or just log the command without parameters)

Actions #9

Updated by Ernesto Puerta about 4 years ago

For the current log trace:

cmd=[{"prefix": "dashboard ac-user-create", "force_password": true, "username": "admin", "password": "admin", "rolename": "administrator", "target": ["mon-mgr", ""]}]
I see some ways of dealing with it:
  • Forcing sensitive information not to be passed by command line (e.g.: read from file): as commands are kept in shell history and procfs also keeps cmdline in memory.
  • Per-command: Adding a new MonCommand FLAG (src/mon/MonCommand.h), like FLAG_SANITIZE_CMDARGS. It'd print the command trace but with everything except the prefix redacted out:
    cmd=[{"prefix": "dashboard ac-user-create", "force_password": SANITIZED, "username": SANITIZED, "password": SANITIZED, "rolename": SANITIZED, "target": ["mon-mgr", ""]}]
    
  • Per-argument 1 (new flag) (as we may still want to log some command args): Adding a new command option to cmddesc, like name=password,type=CephString,sanitized=true
    cmd=[{"prefix": "dashboard ac-user-create", "force_password": true, "username": "admin", "password": SANITIZED, "rolename": "administrator", "target": ["mon-mgr", ""]}]
    
  • Per-argument 2 (new Ceph Type): Adding a new CephType, like name=password,type=CephSecret
    cmd=[{"prefix": "dashboard ac-user-create", "force_password": true, "username": "admin", "password": SANITIZED, "rolename": "administrator", "target": ["mon-mgr", ""]}]
    
  • New Ceph Option Flag: Adding a new flag for Ceph Options (like FLAG_DO_NOT_LOG):
    cmd=[{"prefix": "dashboard ac-user-create", "force_password": true, "username": "admin", "password": SANITIZED, "rolename": "administrator", "target": ["mon-mgr", ""]}]
    

This would be required both for ceph-mgr commands and also for ceph-mon config/config-key commands, as these Mgr commands end up stored in the Mon KV store.

Actions #10

Updated by Sage Weil about 4 years ago

  • Priority changed from Normal to Urgent
Actions #11

Updated by Sage Weil about 4 years ago

  • Priority changed from Urgent to High
Actions #12

Updated by Travis Nielsen almost 4 years ago

Rook really would benefit from at least a simple solution that sanitizes the log for setting the dashboard password. Rook doesn't need to worry about shell history either.

What's the simplest solution to avoid logging the dashboard username/password? Couldn't the log at least be sanitized if the prefix matches certain commands like "dashboard set-login-credentials"? Let's at least get something simple for the commands that currently exist instead of waiting for the perfect solution.

Actions #13

Updated by Neha Ojha over 3 years ago

  • Status changed from New to Fix Under Review
  • Assignee set to Neha Ojha
  • Backport changed from nautilus to nautilus,octopus
  • Pull request ID set to 38479
Actions #14

Updated by Neha Ojha over 3 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions #15

Updated by Backport Bot over 3 years ago

  • Copied to Backport #48614: nautilus: Audit log: mgr module passwords set on CLI written as plaintext in log files added
Actions #16

Updated by Backport Bot over 3 years ago

  • Copied to Backport #48615: octopus: Audit log: mgr module passwords set on CLI written as plaintext in log files added
Actions #17

Updated by Ernesto Puerta over 3 years ago

  • Blocks Bug #48623: mgr/dashboard: Dashboard logs e2e tests are failing added
Actions #18

Updated by Nathan Cutler over 3 years ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Actions

Also available in: Atom PDF