Bug #51111: Pacific: CEPHADM_STRAY_DAEMON after deploying iSCSI gateway with cephadm due to tcmu-runner - Orchestrator - Ceph

Actions

Copy link

Bug #51111

closed

Pacific: CEPHADM_STRAY_DAEMON after deploying iSCSI gateway with cephadm due to tcmu-runner

Added by Paul Giralt almost 3 years ago. Updated about 2 years ago.

Status:

Resolved

Priority:

High

Assignee:

Melissa Li

Category:

cephadm

Target version:

% Done:

Source:

Community (user)

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

Ceph - v16.2.4

ceph-qa-suite:

Pull request ID:

43833

Crash signature (v1):

Crash signature (v2):

Description

Deploy iscsi gateways with the command 'ceph orch apply iscsi.yaml' with the following YAML file (hostname / IP / password have been changed for privacy reasons),

service_type: iscsi
service_id: iscsi
placement:
  hosts:
    - host1.domain.com
    - host2.domain.com
    - host3.domain.com
    - host4.domain.com
spec:
  pool: iscsi-config
  trusted_ip_list: "10.10.10.10,10.10.10.11,10.10.10.12,10.10.10.10.13,10.10.10.14" 
  api_user: admin
  api_password: password_removed
  api_secure: false

All services will work properly, however the cluster health will indicate a warning whenever an image is added to a target - one for each image on each gateway (so total number of warnings will be number of gateways x number of images).

# ceph health detail
HEALTH_WARN 16 stray daemon(s) not managed by cephadm
[WRN] CEPHADM_STRAY_DAEMON: 16 stray daemon(s) not managed by cephadm
    stray daemon tcmu-runner.host2.domain.com:iscsi-pool-0001/iscsi-p0001-img-01 on host host2.domain.com not managed by cephadm
    stray daemon tcmu-runner.host2.domain.com:iscsi-pool-0002/iscsi-p0002-img-01 on host host2.domain.com not managed by cephadm
    stray daemon tcmu-runner.host2.domain.com:iscsi-pool-0003/iscsi-p0003-img-01 on host host2.domain.com not managed by cephadm
    stray daemon tcmu-runner.host2.domain.com:iscsi-pool-0004/iscsi-p0004-img-01 on host host2.domain.com not managed by cephadm
    stray daemon tcmu-runner.host3.domain.com:iscsi-pool-0001/iscsi-p0001-img-01 on host host3.domain.com not managed by cephadm
    stray daemon tcmu-runner.host3.domain.com:iscsi-pool-0002/iscsi-p0002-img-01 on host host3.domain.com not managed by cephadm
    stray daemon tcmu-runner.host3.domain.com:iscsi-pool-0003/iscsi-p0003-img-01 on host host3.domain.com not managed by cephadm
    stray daemon tcmu-runner.host3.domain.com:iscsi-pool-0004/iscsi-p0004-img-01 on host host3.domain.com not managed by cephadm
    stray daemon tcmu-runner.host4.domain.com:iscsi-pool-0001/iscsi-p0001-img-01 on host host4.domain.com not managed by cephadm
    stray daemon tcmu-runner.host4.domain.com:iscsi-pool-0002/iscsi-p0002-img-01 on host host4.domain.com not managed by cephadm
    stray daemon tcmu-runner.host4.domain.com:iscsi-pool-0003/iscsi-p0003-img-01 on host host4.domain.com not managed by cephadm
    stray daemon tcmu-runner.host4.domain.com:iscsi-pool-0004/iscsi-p0004-img-01 on host host4.domain.com not managed by cephadm
    stray daemon tcmu-runner.host5.domain.com:iscsi-pool-0001/iscsi-p0001-img-01 on host host5.domain.com not managed by cephadm
    stray daemon tcmu-runner.host5.domain.com:iscsi-pool-0002/iscsi-p0002-img-01 on host host5.domain.com not managed by cephadm
    stray daemon tcmu-runner.host5.domain.com:iscsi-pool-0003/iscsi-p0003-img-01 on host host5.domain.com not managed by cephadm
    stray daemon tcmu-runner.host5.domain.com:iscsi-pool-0004/iscsi-p0004-img-01 on host host5.domain.com not managed by cephadm

I've reproduced this twice on fresh Pacific 16.2.4 installations and I suspect it might have something to do with using FQDN.

Actions

Copy link

Updated by Loïc Dachary almost 3 years ago

Target version deleted (~~v16.2.5~~)

Actions

Copy link

Updated by Sebastian Wagner almost 3 years ago

Description updated (diff)
Category changed from cephadm/monitoring to cephadm

Actions

Copy link

Updated by Fabian Goebel almost 3 years ago

Hello,
I have the very same Issue on a fresh install of Pacific 16.2.4 on Ubuntu with podman, but I only have used short dns names

root@ceph00:~# ceph health detail
HEALTH_WARN 2 stray daemon(s) not managed by cephadm
[WRN] CEPHADM_STRAY_DAEMON: 2 stray daemon(s) not managed by cephadm
    stray daemon tcmu-runner.ceph01:rbd/isci-test on host ceph01 not managed by cephadm
    stray daemon tcmu-runner.ceph02:rbd/isci-test on host ceph02 not managed by cephadm

Actions

Copy link

Updated by Tobias Fischer almost 3 years ago

same here as well:

HEALTH_WARN 2 stray daemon(s) not managed by cephadm
[WRN] CEPHADM_STRAY_DAEMON: 2 stray daemon(s) not managed by cephadm
    stray daemon tcmu-runner.be-iscsi20p:iscsi/test1 on host be-iscsi20p not managed by cephadm
    stray daemon tcmu-runner.be-iscsi21p:iscsi/test1 on host be-iscsi21p not managed by cephadm

Actions

Copy link

Updated by Tobias Fischer almost 3 years ago

ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)

Actions

Copy link

Updated by icy chan over 2 years ago

Same issue on 16.2.6 with Ubuntu 20.04.
Temporarily silence the WRAN by configuring mgr/cephadm/warn_on_stray_daemons to false.

Actions

Copy link

Updated by Kamil Kuramshin over 2 years ago

icy chan wrote:

Same issue on 16.2.6 with Ubuntu 20.04.
Temporarily silence the WRAN by configuring mgr/cephadm/warn_on_stray_daemons to false.

Can confirm same issue. Fresh installation in Debian 11. ISCSI-Gateways deployed with ceph orch apply -i iscsi.yaml
Error raised after attempt to create new export of rdb-image,
Removing iscsi-service fix this problem.

ceph health detail
HEALTH_WARN 1 stray daemon(s) not managed by cephadm
[WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm
    stray daemon tcmu-runner.cn01:compressed_replicated/iscsi on host cn01 not managed by cephadm

Actions

Copy link