Bug #54159: cephadm: chown the uid/gid of the files under the prometheus data dir - Orchestrator - Ceph

Actions

Copy link

Bug #54159

closed

cephadm: chown the uid/gid of the files under the prometheus data dir

Added by Michael Fritch about 2 years ago. Updated almost 2 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Michael Fritch

Category:

cephadm (binary)

Target version:

% Done:

Source:

Community (dev)

Tags:

Backport:

quincy,pacific

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

44905

Crash signature (v1):

Crash signature (v2):

Description

some builds of prometheus run with a uid 65534 (nobody) whereas other builds run prometheus with a uid of 0 (root)

manually setting the mgr/cephadm/container_image_prometheus key to a different prometheus image and a redeploy fails to start the container with the following:

Feb 04 13:21:12 mfritch3 conmon[15704]: ts=2022-02-04T20:21:12.258Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.1, branch=HEAD, revision=4e08110891fd5177f9174c4179bc38d789985a13)" 
Feb 04 13:21:12 mfritch3 conmon[15704]: ts=2022-02-04T20:21:12.258Z caller=main.go:517 level=info build_context="(go=go1.17.6, user=root@37fc1ebac798, date=20220202-15:23:18)" 
Feb 04 13:21:12 mfritch3 conmon[15704]: ts=2022-02-04T20:21:12.259Z caller=main.go:518 level=info host_details="(Linux 5.15.12-1-default #1 SMP Wed Dec 29 14:50:16 UTC 2021 (375fcb8) x86_64 mfritch3 )" 
Feb 04 13:21:12 mfritch3 conmon[15704]: ts=2022-02-04T20:21:12.259Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 
Feb 04 13:21:12 mfritch3 conmon[15704]: ts=2022-02-04T20:21:12.259Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 
Feb 04 13:21:12 mfritch3 conmon[15704]: ts=2022-02-04T20:21:12.260Z caller=query_logger.go:86 level=error component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied" 
Feb 04 13:21:12 mfritch3 conmon[15704]: panic: Unable to create mmap-ed active query log
Feb 04 13:21:12 mfritch3 conmon[15704]: 
Feb 04 13:21:12 mfritch3 conmon[15704]: goroutine 1 [running]:
Feb 04 13:21:12 mfritch3 conmon[15704]: github.com/prometheus/prometheus/promql.NewActiveQueryTracker({0x7ffe4d5feee6, 0xb}, 0x14, {0x35edfa0, 0xc0005990e0})
Feb 04 13:21:12 mfritch3 conmon[15704]:         /app/promql/query_logger.go:116 +0x3d7
Feb 04 13:21:12 mfritch3 conmon[15704]: main.main()
Feb 04 13:21:12 mfritch3 conmon[15704]:         /app/cmd/prometheus/main.go:564 +0x5e73

A similar condition could also occurring during a `ceph orch upgrade ...`