Project

General

Profile

Actions
Copy link

Bug #49573

closed

cephadm: rgw-nfs: stray daemons(s) not managed by cephadm

Added by Sebastian Wagner about 3 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Michael Fritch
Category:
cephadm/nfs
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
39825
Crash signature (v1):
Crash signature (v2):

Description

node4:~ # ceph -s
cluster:
id: yxz
health: HEALTH_WARN
2 stray daemons(s) not managed by cephadm

services:
mon: 3 daemons, quorum node6,node7,node5 (age 3d)
mgr: node6(active, since 3d), standbys: node7, node5
mds: cephfs:1 {0=cephfs.node8.qivbjw=up:active} 1 up:standby
osd: 36 osds: 36 up (since 3d), 36 in (since 3d)
rgw: 2 daemons active (default.default.node8.acedkk, default.default.node9.rcjrtg)
rgw-nfs: 2 daemons active (nfs.nfs-ganesha.node8-rgw, nfs.nfs-ganesha.node9-rgw)

task status:

data:
pools: 13 pools, 706 pgs
objects: 8.51k objects, 10 GiB
usage: 310 GiB used, 1.2 TiB / 1.5 TiB avail
pgs: 706 active+clean

io:
client: 2.2 KiB/s rd, 2 op/s rd, 0 op/s wr

node4:~ # ceph health detail
HEALTH_WARN 2 stray daemons(s) not managed by cephadm
[WRN] CEPHADM_STRAY_DAEMON: 2 stray daemons(s) not managed by cephadm
stray daemon rgw-nfs.nfs.nfs-ganesha.node8-rgw on host node8 not managed by cephadm
stray daemon rgw-nfs.nfs.nfs-ganesha.node9-rgw on host node9 not managed by cephadm

node4:~ # ceph orch ps --daemon_type nfs
NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID
nfs.nfs-ganesha.node8 node8 running (37m) 6m ago 37m 3.3 ceph/ceph:latest f1a7d8e63a7e e1dbe2870987
nfs.nfs-ganesha.node9 node9 running (37m) 6m ago 37m 3.3 ceph/ceph:latest f1a7d8e63a7e 60b83c0e275d

service_type: nfs
service_id: nfs-ganesha
placement:
hosts:
- node8
- node9
spec:
pool: ganesha_config
namespace: ganesha
---
service_type: rgw
service_id: default.default
service_name: rgw.default.default
placement:
  hosts:
  - node8
  - node9
spec:
  rgw_realm: default
  rgw_zone: default
status:
  container_image_id: f1a7d8e63a7eb956904027325e1924fc6d187994fce646a040f2ea8c7b2cec7d
  container_image_name: ceph/ceph:latest
  created: '2021-02-25T21:48:54.268489'
  last_refresh: '2021-03-01T20:56:22.878539'
  running: 2
  size: 2
---
daemon_type: nfs
daemon_id: nfs-ganesha.node8
hostname: node8
container_id: e1dbe2870987
container_image_id: f1a7d8e63a7eb956904027325e1924fc6d187994fce646a040f2ea8c7b2cec7d
container_image_name: ceph/ceph:latest
version: '3.3'
status: 1
status_desc: running
is_active: false
last_refresh: '2021-03-01T20:56:22.878752'
created: '2021-03-01T19:54:36.099431'
started: '2021-03-01T19:54:35.712779'
events:
- 2021-02-26T18:20:31.198532 daemon:nfs.nfs-ganesha.node8 [INFO] "Deployed nfs.nfs-ganesha.node8
  on host 'node8'" 
- 2021-02-26T19:51:23.638753 daemon:nfs.nfs-ganesha.node8 [INFO] "restart nfs.nfs-ganesha.node8
  from host 'node8'" 
---
daemon_type: rgw
daemon_id: default.default.node8.acedkk
hostname: node8
container_id: 96f77433ea02
container_image_id: f1a7d8e63a7eb956904027325e1924fc6d187994fce646a040f2ea8c7b2cec7d
container_image_name: ceph/ceph:latest
version: 15.2.8.80
status: 1
status_desc: running
is_active: false
last_refresh: '2021-03-01T20:56:22.878539'
created: '2021-02-25T02:14:02.159285'
started: '2021-02-26T19:45:03.708309'


# /usr/sbin/cephadm ls
[
    {
        "style": "cephadm:v1",
        "name": "rgw.default.default.node8.acedkk",
        "fsid": "fsid-fsid-fsid-fsid",
        "systemd_unit": "ceph-fsid-fsid-fsid-fsid@rgw.default.default.node8.acedkk",
        "enabled": true,
        "state": "running",
        "container_id": "96f77433ea02506eb87628a71e637c2224db1642165102574951dc9ef168b154",
        "container_image_name": "ceph/ceph:latest",
        "container_image_id": "f1a7d8e63a7eb956904027325e1924fc6d187994fce646a040f2ea8c7b2cec7d",
        "version": "15.2.8.80",
        "started": "2021-02-26T19:45:03.708309",
        "created": "2021-02-25T02:14:02.159285",
        "deployed": "2021-02-25T02:14:00.819265",
        "configured": "2021-02-25T20:55:35.516564" 
    },
    {
        "style": "cephadm:v1",
        "name": "nfs.nfs-ganesha.node8",
        "fsid": "fsid-fsid-fsid-fsid",
        "systemd_unit": "ceph-fsid-fsid-fsid-fsid@nfs.nfs-ganesha.node8",
        "enabled": true,
        "state": "running",
        "container_id": "e1dbe287098722c8aa08f7db674cf6f4db980800776b4356181268e81804720d",
        "container_image_name": "ceph/ceph:latest",
        "container_image_id": "f1a7d8e63a7eb956904027325e1924fc6d187994fce646a040f2ea8c7b2cec7d",
        "version": "3.3",
        "started": "2021-03-01T19:54:35.712779",
        "created": "2021-03-01T19:54:36.099431",
        "deployed": "2021-03-01T19:54:33.163387",
        "configured": "2021-03-01T19:54:36.099431" 
    },
]
Actions
Copy link
 #1

Updated by Michael Fritch about 3 years ago

  • Status changed from New to Fix Under Review
  • Assignee set to Michael Fritch
  • Pull request ID set to 39825

during the an nfs mount of an rgw bucket, librgw will register a daemon with type `rgw-nfs`:
https://github.com/ceph/ceph/blob/65724b1f97a00ee345db806ebbef3fad5be74dab/src/rgw/librgw.cc#L616

For example:

s = {'type': 'rgw-nfs', 'id': 'nfs.foo.host1-rgw'}

Actions
Copy link
 #2

Updated by Michael Fritch about 3 years ago

nfs-ganesha is deployed in such a way that it can host both cephfs and rgw exports from a single daemon.

rgw appears to register itself in the service-map as `rgw` and then will later register another `rgw-nfs` type during the first mount of a bucket from ganesha.

As far as I can tell, cephfs doesn't register anything in relation to nfs mounts.

daemons deployed by cephadm:

$ ceph orch ps
NAME                            HOST      STATUS         REFRESHED  AGE  VERSION                IMAGE NAME                                                                                          IMAGE ID      CONTAINER ID  
crash.host1                  host1  running (18h)  4m ago     18h  17.0.0-1275-g5e197a21  docker.io/ceph/daemon-base@sha256:8a9d01990f2601ea3111d84a18ba84a9043275cde1e48956a74b5a12e5e850b3  5c2a636e05b4  05d1f66a69e4  
mds.a.host1.rjbglc           host1  running (18h)  4m ago     18h  17.0.0-1275-g5e197a21  docker.io/ceph/daemon-base@sha256:8a9d01990f2601ea3111d84a18ba84a9043275cde1e48956a74b5a12e5e850b3  5c2a636e05b4  62610e9a2e16  
nfs.foo.host1                host1  running (18h)  4m ago     18h  3.5                    docker.io/ceph/daemon-base@sha256:8a9d01990f2601ea3111d84a18ba84a9043275cde1e48956a74b5a12e5e850b3  5c2a636e05b4  440bc305732c  
rgw.realm.zone.host1.uxxnwr  host1  running (18h)  4m ago     18h  17.0.0-1275-g5e197a21  docker.io/ceph/daemon-base@sha256:8a9d01990f2601ea3111d84a18ba84a9043275cde1e48956a74b5a12e5e850b3  5c2a636e05b4  13b1751e5324

ceph service-map:

$ ceph -s
  cluster:
    id:     cdf3a637-85c6-41d4-9866-c62c93692f7e
    health: HEALTH_WARN
            10 stray daemon(s) not managed by cephadm

  services:
    mon:     3 daemons, quorum a,b,c (age 3h)
    mgr:     x(active, since 77m)
    mds:     1/1 daemons up, 3 standby
    osd:     3 osds: 3 up (since 3h), 3 in (since 3h)
    rgw:     1 daemon active (realm.zone.host1.uxxnwr)
    rgw-nfs: 1 daemon active (nfs.foo.host1-rgw)

  data:
    volumes: 1/1 healthy
    pools:   9 pools, 272 pgs
    objects: 404 objects, 71 KiB
    usage:   3.0 GiB used, 300 GiB / 303 GiB avail
    pgs:     272 active+clean

  io:
    client:   85 B/s rd, 0 op/s rd, 0 op/s wr

$ ceph service status
{
    "rgw": {
        "realm.zone.host1.uxxnwr": {
            "status_stamp": "2021-03-04T09:12:54.023764-0700",
            "last_beacon": "2021-03-04T09:12:54.023764-0700",
            "status": {
                "current_sync": "[]" 
            }
        }
    },
    "rgw-nfs": {
        "nfs.foo.host1-rgw": {
            "status_stamp": "2021-03-04T09:12:49.039145-0700",
            "last_beacon": "2021-03-04T09:12:54.039491-0700",
            "status": {
                "current_sync": "[]" 
            }
        }
    }
}

Actions
Copy link
 #3

Updated by Sage Weil about 3 years ago

  • Status changed from Fix Under Review to Resolved
Actions
Copy link

Also available in: Atom PDF