Actions
Bug #49573
closedcephadm: rgw-nfs: stray daemons(s) not managed by cephadm
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Description
node4:~ # ceph -s cluster: id: yxz health: HEALTH_WARN 2 stray daemons(s) not managed by cephadm services: mon: 3 daemons, quorum node6,node7,node5 (age 3d) mgr: node6(active, since 3d), standbys: node7, node5 mds: cephfs:1 {0=cephfs.node8.qivbjw=up:active} 1 up:standby osd: 36 osds: 36 up (since 3d), 36 in (since 3d) rgw: 2 daemons active (default.default.node8.acedkk, default.default.node9.rcjrtg) rgw-nfs: 2 daemons active (nfs.nfs-ganesha.node8-rgw, nfs.nfs-ganesha.node9-rgw) task status: data: pools: 13 pools, 706 pgs objects: 8.51k objects, 10 GiB usage: 310 GiB used, 1.2 TiB / 1.5 TiB avail pgs: 706 active+clean io: client: 2.2 KiB/s rd, 2 op/s rd, 0 op/s wr
node4:~ # ceph health detail HEALTH_WARN 2 stray daemons(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 2 stray daemons(s) not managed by cephadm stray daemon rgw-nfs.nfs.nfs-ganesha.node8-rgw on host node8 not managed by cephadm stray daemon rgw-nfs.nfs.nfs-ganesha.node9-rgw on host node9 not managed by cephadm
node4:~ # ceph orch ps --daemon_type nfs NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID nfs.nfs-ganesha.node8 node8 running (37m) 6m ago 37m 3.3 ceph/ceph:latest f1a7d8e63a7e e1dbe2870987 nfs.nfs-ganesha.node9 node9 running (37m) 6m ago 37m 3.3 ceph/ceph:latest f1a7d8e63a7e 60b83c0e275d
service_type: nfs
service_id: nfs-ganesha
placement:
hosts:
- node8
- node9
spec:
pool: ganesha_config
namespace: ganesha
---
service_type: rgw
service_id: default.default
service_name: rgw.default.default
placement:
hosts:
- node8
- node9
spec:
rgw_realm: default
rgw_zone: default
status:
container_image_id: f1a7d8e63a7eb956904027325e1924fc6d187994fce646a040f2ea8c7b2cec7d
container_image_name: ceph/ceph:latest
created: '2021-02-25T21:48:54.268489'
last_refresh: '2021-03-01T20:56:22.878539'
running: 2
size: 2
---
daemon_type: nfs
daemon_id: nfs-ganesha.node8
hostname: node8
container_id: e1dbe2870987
container_image_id: f1a7d8e63a7eb956904027325e1924fc6d187994fce646a040f2ea8c7b2cec7d
container_image_name: ceph/ceph:latest
version: '3.3'
status: 1
status_desc: running
is_active: false
last_refresh: '2021-03-01T20:56:22.878752'
created: '2021-03-01T19:54:36.099431'
started: '2021-03-01T19:54:35.712779'
events:
- 2021-02-26T18:20:31.198532 daemon:nfs.nfs-ganesha.node8 [INFO] "Deployed nfs.nfs-ganesha.node8
on host 'node8'"
- 2021-02-26T19:51:23.638753 daemon:nfs.nfs-ganesha.node8 [INFO] "restart nfs.nfs-ganesha.node8
from host 'node8'"
---
daemon_type: rgw
daemon_id: default.default.node8.acedkk
hostname: node8
container_id: 96f77433ea02
container_image_id: f1a7d8e63a7eb956904027325e1924fc6d187994fce646a040f2ea8c7b2cec7d
container_image_name: ceph/ceph:latest
version: 15.2.8.80
status: 1
status_desc: running
is_active: false
last_refresh: '2021-03-01T20:56:22.878539'
created: '2021-02-25T02:14:02.159285'
started: '2021-02-26T19:45:03.708309'
# /usr/sbin/cephadm ls
[
{
"style": "cephadm:v1",
"name": "rgw.default.default.node8.acedkk",
"fsid": "fsid-fsid-fsid-fsid",
"systemd_unit": "ceph-fsid-fsid-fsid-fsid@rgw.default.default.node8.acedkk",
"enabled": true,
"state": "running",
"container_id": "96f77433ea02506eb87628a71e637c2224db1642165102574951dc9ef168b154",
"container_image_name": "ceph/ceph:latest",
"container_image_id": "f1a7d8e63a7eb956904027325e1924fc6d187994fce646a040f2ea8c7b2cec7d",
"version": "15.2.8.80",
"started": "2021-02-26T19:45:03.708309",
"created": "2021-02-25T02:14:02.159285",
"deployed": "2021-02-25T02:14:00.819265",
"configured": "2021-02-25T20:55:35.516564"
},
{
"style": "cephadm:v1",
"name": "nfs.nfs-ganesha.node8",
"fsid": "fsid-fsid-fsid-fsid",
"systemd_unit": "ceph-fsid-fsid-fsid-fsid@nfs.nfs-ganesha.node8",
"enabled": true,
"state": "running",
"container_id": "e1dbe287098722c8aa08f7db674cf6f4db980800776b4356181268e81804720d",
"container_image_name": "ceph/ceph:latest",
"container_image_id": "f1a7d8e63a7eb956904027325e1924fc6d187994fce646a040f2ea8c7b2cec7d",
"version": "3.3",
"started": "2021-03-01T19:54:35.712779",
"created": "2021-03-01T19:54:36.099431",
"deployed": "2021-03-01T19:54:33.163387",
"configured": "2021-03-01T19:54:36.099431"
},
]
Updated by Michael Fritch about 3 years ago
- Status changed from New to Fix Under Review
- Assignee set to Michael Fritch
- Pull request ID set to 39825
during the an nfs mount of an rgw bucket, librgw will register a daemon with type `rgw-nfs`:
https://github.com/ceph/ceph/blob/65724b1f97a00ee345db806ebbef3fad5be74dab/src/rgw/librgw.cc#L616
For example:
s = {'type': 'rgw-nfs', 'id': 'nfs.foo.host1-rgw'}
Updated by Michael Fritch about 3 years ago
nfs-ganesha is deployed in such a way that it can host both cephfs and rgw exports from a single daemon.
rgw appears to register itself in the service-map as `rgw` and then will later register another `rgw-nfs` type during the first mount of a bucket from ganesha.
As far as I can tell, cephfs doesn't register anything in relation to nfs mounts.
daemons deployed by cephadm:
$ ceph orch ps NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID crash.host1 host1 running (18h) 4m ago 18h 17.0.0-1275-g5e197a21 docker.io/ceph/daemon-base@sha256:8a9d01990f2601ea3111d84a18ba84a9043275cde1e48956a74b5a12e5e850b3 5c2a636e05b4 05d1f66a69e4 mds.a.host1.rjbglc host1 running (18h) 4m ago 18h 17.0.0-1275-g5e197a21 docker.io/ceph/daemon-base@sha256:8a9d01990f2601ea3111d84a18ba84a9043275cde1e48956a74b5a12e5e850b3 5c2a636e05b4 62610e9a2e16 nfs.foo.host1 host1 running (18h) 4m ago 18h 3.5 docker.io/ceph/daemon-base@sha256:8a9d01990f2601ea3111d84a18ba84a9043275cde1e48956a74b5a12e5e850b3 5c2a636e05b4 440bc305732c rgw.realm.zone.host1.uxxnwr host1 running (18h) 4m ago 18h 17.0.0-1275-g5e197a21 docker.io/ceph/daemon-base@sha256:8a9d01990f2601ea3111d84a18ba84a9043275cde1e48956a74b5a12e5e850b3 5c2a636e05b4 13b1751e5324
ceph service-map:
$ ceph -s cluster: id: cdf3a637-85c6-41d4-9866-c62c93692f7e health: HEALTH_WARN 10 stray daemon(s) not managed by cephadm services: mon: 3 daemons, quorum a,b,c (age 3h) mgr: x(active, since 77m) mds: 1/1 daemons up, 3 standby osd: 3 osds: 3 up (since 3h), 3 in (since 3h) rgw: 1 daemon active (realm.zone.host1.uxxnwr) rgw-nfs: 1 daemon active (nfs.foo.host1-rgw) data: volumes: 1/1 healthy pools: 9 pools, 272 pgs objects: 404 objects, 71 KiB usage: 3.0 GiB used, 300 GiB / 303 GiB avail pgs: 272 active+clean io: client: 85 B/s rd, 0 op/s rd, 0 op/s wr $ ceph service status { "rgw": { "realm.zone.host1.uxxnwr": { "status_stamp": "2021-03-04T09:12:54.023764-0700", "last_beacon": "2021-03-04T09:12:54.023764-0700", "status": { "current_sync": "[]" } } }, "rgw-nfs": { "nfs.foo.host1-rgw": { "status_stamp": "2021-03-04T09:12:49.039145-0700", "last_beacon": "2021-03-04T09:12:54.039491-0700", "status": { "current_sync": "[]" } } } }
Updated by Sage Weil about 3 years ago
- Status changed from Fix Under Review to Resolved
Actions