Ceph » Orchestrator

Deploying an iSCSI service doesn't fail

# ceph orch apply iscsi rbd admin admin 192.168.100.10 --placement="label:iscsigws" 

But the container doesn't start at all

# ceph orch ls --service_type iscsi
INFO:cephadm:Inferring fsid 995d67d2-c216-11ea-87c7-fa163e4f94fe
INFO:cephadm:Inferring config /var/lib/ceph/995d67d2-c216-11ea-87c7-fa163e4f94fe/mon.fosnvioja/config
INFO:cephadm:Using recent ceph image docker.io/ceph/daemon:latest-master
NAME         RUNNING  REFRESHED  AGE  PLACEMENT       IMAGE NAME                           IMAGE ID   
iscsi.iscsi      0/1  5m ago     26m  label:iscsigws  docker.io/ceph/daemon:latest-master  <unknown>
# ceph orch ps --daemon_type iscsi
INFO:cephadm:Inferring fsid 995d67d2-c216-11ea-87c7-fa163e4f94fe
INFO:cephadm:Inferring config /var/lib/ceph/995d67d2-c216-11ea-87c7-fa163e4f94fe/mon.fosnvioja/config
INFO:cephadm:Using recent ceph image docker.io/ceph/daemon:latest-master
NAME                          HOST       STATUS  REFRESHED  AGE  VERSION    IMAGE NAME                           IMAGE ID   CONTAINER ID  
iscsi.iscsi.fosnvioja.zmhojt  fosnvioja  error   6m ago     27m  <unknown>  docker.io/ceph/daemon:latest-master  <unknown>  <unknown>
# cephadm logs --name iscsi.iscsi.fosnvioja.zmhojt
INFO:cephadm:Inferring fsid 995d67d2-c216-11ea-87c7-fa163e4f94fe
-- Logs begin at Thu 2020-07-09 18:48:33 UTC, end at Fri 2020-07-10 15:55:41 UTC. --
Jul 10 15:27:47 fosnvioja systemd[1]: Starting Ceph iscsi.iscsi.fosnvioja.zmhojt for 995d67d2-c216-11ea-87c7-fa163e4f94fe...
Jul 10 15:27:47 fosnvioja podman[101210]: Error: no container with name or ID ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe-iscsi.iscsi.fosnvioja.zmhojt found: no such container
Jul 10 15:27:47 fosnvioja systemd[1]: Started Ceph iscsi.iscsi.fosnvioja.zmhojt for 995d67d2-c216-11ea-87c7-fa163e4f94fe.
Jul 10 15:27:47 fosnvioja bash[101221]: Error: Failed to evict container: "": Failed to find container "ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe-iscsi.iscsi.fosnvioja.zmhojt" in state: no container with name or ID ceph-995d67d2-c216-11e>
Jul 10 15:27:47 fosnvioja bash[101221]: ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe-iscsi.iscsi.fosnvioja.zmhojt
Jul 10 15:27:47 fosnvioja bash[101221]: Error: no container with ID or name "ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe-iscsi.iscsi.fosnvioja.zmhojt" found: no such container
Jul 10 15:27:47 fosnvioja bash[101221]: WARNING: The same type, major and minor should not be used for multiple devices.
Jul 10 15:27:48 fosnvioja podman[101249]: 2020-07-10 15:27:48.198264405 +0000 UTC m=+0.255695263 container create f5cbf35e69b9a4c99b97e6fd646556d6516681cf8ff21b6394983e20b17ec05d (image=docker.io/ceph/daemon:latest-master, name=ceph-995d>
Jul 10 15:27:48 fosnvioja podman[101249]: 2020-07-10 15:27:48.740681658 +0000 UTC m=+0.798112569 container remove f5cbf35e69b9a4c99b97e6fd646556d6516681cf8ff21b6394983e20b17ec05d (image=docker.io/ceph/daemon:latest-master, name=ceph-995d>
Jul 10 15:27:48 fosnvioja bash[101221]: Error: relabel failed "/var/lib/ceph/995d67d2-c216-11ea-87c7-fa163e4f94fe/iscsi.iscsi.fosnvioja.zmhojt/configfs": operation not supported
Jul 10 15:27:48 fosnvioja systemd[1]: ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe@iscsi.iscsi.fosnvioja.zmhojt.service: Main process exited, code=exited, status=126/n/a
Jul 10 15:27:48 fosnvioja systemd[1]: ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe@iscsi.iscsi.fosnvioja.zmhojt.service: Failed with result 'exit-code'.
Jul 10 15:27:58 fosnvioja systemd[1]: ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe@iscsi.iscsi.fosnvioja.zmhojt.service: Service RestartSec=10s expired, scheduling restart.
Jul 10 15:27:58 fosnvioja systemd[1]: ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe@iscsi.iscsi.fosnvioja.zmhojt.service: Scheduled restart job, restart counter is at 1.
Jul 10 15:27:58 fosnvioja systemd[1]: Stopped Ceph iscsi.iscsi.fosnvioja.zmhojt for 995d67d2-c216-11ea-87c7-fa163e4f94fe.
Jul 10 15:27:58 fosnvioja systemd[1]: Starting Ceph iscsi.iscsi.fosnvioja.zmhojt for 995d67d2-c216-11ea-87c7-fa163e4f94fe...
Jul 10 15:27:58 fosnvioja podman[101804]: Error: no container with name or ID ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe-iscsi.iscsi.fosnvioja.zmhojt found: no such container
Jul 10 15:27:58 fosnvioja systemd[1]: Started Ceph iscsi.iscsi.fosnvioja.zmhojt for 995d67d2-c216-11ea-87c7-fa163e4f94fe.
Jul 10 15:27:59 fosnvioja bash[101815]: Error: Failed to evict container: "": Failed to find container "ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe-iscsi.iscsi.fosnvioja.zmhojt" in state: no container with name or ID ceph-995d67d2-c216-11e>
Jul 10 15:27:59 fosnvioja bash[101815]: ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe-iscsi.iscsi.fosnvioja.zmhojt
Jul 10 15:27:59 fosnvioja bash[101815]: Error: no container with ID or name "ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe-iscsi.iscsi.fosnvioja.zmhojt" found: no such container
Jul 10 15:27:59 fosnvioja bash[101815]: WARNING: The same type, major and minor should not be used for multiple devices.
Jul 10 15:27:59 fosnvioja podman[101839]: 2020-07-10 15:27:59.503779368 +0000 UTC m=+0.336018950 container create f762ce7b4cb4c5dc64f286cf6efe9f48ca69519a42bdd15fbc4e46ccdb2483f3 (image=docker.io/ceph/daemon:latest-master, name=ceph-995d>
Jul 10 15:27:59 fosnvioja podman[101839]: 2020-07-10 15:27:59.81832938 +0000 UTC m=+0.650569002 container remove f762ce7b4cb4c5dc64f286cf6efe9f48ca69519a42bdd15fbc4e46ccdb2483f3 (image=docker.io/ceph/daemon:latest-master, name=ceph-995d6>
Jul 10 15:27:59 fosnvioja bash[101815]: Error: relabel failed "/var/lib/ceph/995d67d2-c216-11ea-87c7-fa163e4f94fe/iscsi.iscsi.fosnvioja.zmhojt/configfs": operation not supported
Jul 10 15:27:59 fosnvioja systemd[1]: ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe@iscsi.iscsi.fosnvioja.zmhojt.service: Main process exited, code=exited, status=126/n/a
Jul 10 15:27:59 fosnvioja systemd[1]: ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe@iscsi.iscsi.fosnvioja.zmhojt.service: Failed with result 'exit-code'.
Jul 10 15:28:09 fosnvioja systemd[1]: ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe@iscsi.iscsi.fosnvioja.zmhojt.service: Service RestartSec=10s expired, scheduling restart.
Jul 10 15:28:09 fosnvioja systemd[1]: ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe@iscsi.iscsi.fosnvioja.zmhojt.service: Scheduled restart job, restart counter is at 2.
Jul 10 15:28:09 fosnvioja systemd[1]: Stopped Ceph iscsi.iscsi.fosnvioja.zmhojt for 995d67d2-c216-11ea-87c7-fa163e4f94fe.
Jul 10 15:28:09 fosnvioja systemd[1]: Starting Ceph iscsi.iscsi.fosnvioja.zmhojt for 995d67d2-c216-11ea-87c7-fa163e4f94fe...
Jul 10 15:28:09 fosnvioja podman[101912]: Error: no container with name or ID ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe-iscsi.iscsi.fosnvioja.zmhojt found: no such container
Jul 10 15:28:09 fosnvioja systemd[1]: Started Ceph iscsi.iscsi.fosnvioja.zmhojt for 995d67d2-c216-11ea-87c7-fa163e4f94fe.
Jul 10 15:28:10 fosnvioja bash[101924]: Error: Failed to evict container: "": Failed to find container "ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe-iscsi.iscsi.fosnvioja.zmhojt" in state: no container with name or ID ceph-995d67d2-c216-11e>
Jul 10 15:28:10 fosnvioja bash[101924]: ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe-iscsi.iscsi.fosnvioja.zmhojt
Jul 10 15:28:10 fosnvioja bash[101924]: Error: no container with ID or name "ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe-iscsi.iscsi.fosnvioja.zmhojt" found: no such container
Jul 10 15:28:10 fosnvioja bash[101924]: WARNING: The same type, major and minor should not be used for multiple devices.
Jul 10 15:28:10 fosnvioja podman[101950]: 2020-07-10 15:28:10.525956081 +0000 UTC m=+0.356430087 container create 46f7926ba690303af8c976cbdd024705319f32b0e2d96407cece4e29dbf76f33 (image=docker.io/ceph/daemon:latest-master, name=ceph-995d>
Jul 10 15:28:10 fosnvioja podman[101950]: 2020-07-10 15:28:10.729236708 +0000 UTC m=+0.559710803 container remove 46f7926ba690303af8c976cbdd024705319f32b0e2d96407cece4e29dbf76f33 (image=docker.io/ceph/daemon:latest-master, name=ceph-995d>
Jul 10 15:28:10 fosnvioja bash[101924]: Error: relabel failed "/var/lib/ceph/995d67d2-c216-11ea-87c7-fa163e4f94fe/iscsi.iscsi.fosnvioja.zmhojt/configfs": operation not supported
Jul 10 15:28:10 fosnvioja systemd[1]: ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe@iscsi.iscsi.fosnvioja.zmhojt.service: Main process exited, code=exited, status=126/n/a
Jul 10 15:28:10 fosnvioja systemd[1]: ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe@iscsi.iscsi.fosnvioja.zmhojt.service: Failed with result 'exit-code'.
Jul 10 15:28:20 fosnvioja systemd[1]: ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe@iscsi.iscsi.fosnvioja.zmhojt.service: Service RestartSec=10s expired, scheduling restart.
Jul 10 15:28:20 fosnvioja systemd[1]: ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe@iscsi.iscsi.fosnvioja.zmhojt.service: Scheduled restart job, restart counter is at 3.
Jul 10 15:28:20 fosnvioja systemd[1]: Stopped Ceph iscsi.iscsi.fosnvioja.zmhojt for 995d67d2-c216-11ea-87c7-fa163e4f94fe.
Jul 10 15:28:20 fosnvioja systemd[1]: Starting Ceph iscsi.iscsi.fosnvioja.zmhojt for 995d67d2-c216-11ea-87c7-fa163e4f94fe...
Jul 10 15:28:20 fosnvioja podman[102195]: Error: no container with name or ID ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe-iscsi.iscsi.fosnvioja.zmhojt found: no such container
Jul 10 15:28:20 fosnvioja systemd[1]: Started Ceph iscsi.iscsi.fosnvioja.zmhojt for 995d67d2-c216-11ea-87c7-fa163e4f94fe.
Jul 10 15:28:21 fosnvioja bash[102208]: Error: Failed to evict container: "": Failed to find container "ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe-iscsi.iscsi.fosnvioja.zmhojt" in state: no container with name or ID ceph-995d67d2-c216-11e>
Jul 10 15:28:21 fosnvioja bash[102208]: ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe-iscsi.iscsi.fosnvioja.zmhojt
Jul 10 15:28:21 fosnvioja bash[102208]: Error: no container with ID or name "ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe-iscsi.iscsi.fosnvioja.zmhojt" found: no such container
Jul 10 15:28:21 fosnvioja bash[102208]: WARNING: The same type, major and minor should not be used for multiple devices.
Jul 10 15:28:21 fosnvioja podman[102263]: 2020-07-10 15:28:21.952608335 +0000 UTC m=+0.726071179 container create 7bae7f482e1328c510250fd4733f8034a1eec8e3a6059dae8295c0aa8c35ef67 (image=docker.io/ceph/daemon:latest-master, name=ceph-995d>
Jul 10 15:28:22 fosnvioja podman[102263]: 2020-07-10 15:28:22.4389976 +0000 UTC m=+1.212460428 container remove 7bae7f482e1328c510250fd4733f8034a1eec8e3a6059dae8295c0aa8c35ef67 (image=docker.io/ceph/daemon:latest-master, name=ceph-995d67>
Jul 10 15:28:22 fosnvioja bash[102208]: Error: relabel failed "/var/lib/ceph/995d67d2-c216-11ea-87c7-fa163e4f94fe/iscsi.iscsi.fosnvioja.zmhojt/configfs": operation not supported
Jul 10 15:28:22 fosnvioja systemd[1]: ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe@iscsi.iscsi.fosnvioja.zmhojt.service: Main process exited, code=exited, status=126/n/a
Jul 10 15:28:22 fosnvioja systemd[1]: ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe@iscsi.iscsi.fosnvioja.zmhojt.service: Failed with result 'exit-code'.
Jul 10 15:28:32 fosnvioja systemd[1]: ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe@iscsi.iscsi.fosnvioja.zmhojt.service: Service RestartSec=10s expired, scheduling restart.
Jul 10 15:28:32 fosnvioja systemd[1]: ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe@iscsi.iscsi.fosnvioja.zmhojt.service: Scheduled restart job, restart counter is at 4.
Jul 10 15:28:32 fosnvioja systemd[1]: Stopped Ceph iscsi.iscsi.fosnvioja.zmhojt for 995d67d2-c216-11ea-87c7-fa163e4f94fe.
Jul 10 15:28:32 fosnvioja systemd[1]: Starting Ceph iscsi.iscsi.fosnvioja.zmhojt for 995d67d2-c216-11ea-87c7-fa163e4f94fe...
Jul 10 15:28:32 fosnvioja podman[102489]: Error: no container with name or ID ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe-iscsi.iscsi.fosnvioja.zmhojt found: no such container
Jul 10 15:28:32 fosnvioja systemd[1]: Started Ceph iscsi.iscsi.fosnvioja.zmhojt for 995d67d2-c216-11ea-87c7-fa163e4f94fe.
Jul 10 15:28:32 fosnvioja bash[102500]: Error: Failed to evict container: "": Failed to find container "ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe-iscsi.iscsi.fosnvioja.zmhojt" in state: no container with name or ID ceph-995d67d2-c216-11e>
Jul 10 15:28:32 fosnvioja bash[102500]: ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe-iscsi.iscsi.fosnvioja.zmhojt
Jul 10 15:28:32 fosnvioja bash[102500]: Error: no container with ID or name "ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe-iscsi.iscsi.fosnvioja.zmhojt" found: no such container
Jul 10 15:28:32 fosnvioja bash[102500]: WARNING: The same type, major and minor should not be used for multiple devices.
Jul 10 15:28:33 fosnvioja podman[102525]: 2020-07-10 15:28:33.143993257 +0000 UTC m=+0.238219126 container create 0d3790b8581cb9411d457cab12fb8a05b137dc92a8b17fe0ca48030fb0d9a110 (image=docker.io/ceph/daemon:latest-master, name=ceph-995d>
Jul 10 15:28:33 fosnvioja podman[102525]: 2020-07-10 15:28:33.371307008 +0000 UTC m=+0.465532922 container remove 0d3790b8581cb9411d457cab12fb8a05b137dc92a8b17fe0ca48030fb0d9a110 (image=docker.io/ceph/daemon:latest-master, name=ceph-995d>
Jul 10 15:28:33 fosnvioja bash[102500]: Error: relabel failed "/var/lib/ceph/995d67d2-c216-11ea-87c7-fa163e4f94fe/iscsi.iscsi.fosnvioja.zmhojt/configfs": operation not supported
Jul 10 15:28:33 fosnvioja systemd[1]: ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe@iscsi.iscsi.fosnvioja.zmhojt.service: Main process exited, code=exited, status=126/n/a
Jul 10 15:28:33 fosnvioja systemd[1]: ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe@iscsi.iscsi.fosnvioja.zmhojt.service: Failed with result 'exit-code'.
Jul 10 15:28:43 fosnvioja systemd[1]: ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe@iscsi.iscsi.fosnvioja.zmhojt.service: Service RestartSec=10s expired, scheduling restart.
Jul 10 15:28:43 fosnvioja systemd[1]: ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe@iscsi.iscsi.fosnvioja.zmhojt.service: Scheduled restart job, restart counter is at 5.
Jul 10 15:28:43 fosnvioja systemd[1]: Stopped Ceph iscsi.iscsi.fosnvioja.zmhojt for 995d67d2-c216-11ea-87c7-fa163e4f94fe.
Jul 10 15:28:43 fosnvioja systemd[1]: ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe@iscsi.iscsi.fosnvioja.zmhojt.service: Start request repeated too quickly.
Jul 10 15:28:43 fosnvioja systemd[1]: ceph-995d67d2-c216-11ea-87c7-fa163e4f94fe@iscsi.iscsi.fosnvioja.zmhojt.service: Failed with result 'exit-code'.
Jul 10 15:28:43 fosnvioja systemd[1]: Failed to start Ceph iscsi.iscsi.fosnvioja.zmhojt for 995d67d2-c216-11ea-87c7-fa163e4f94fe.

The main issue here is the configfs bindmount which fails (ie: -v /var/lib/ceph/995d67d2-c216-11ea-87c7-fa163e4f94fe/iscsi.iscsi.fosnvioja.zmhojt/configfs:/sys/kernel/config:z)

Error: relabel failed "/var/lib/ceph/995d67d2-c216-11ea-87c7-fa163e4f94fe/iscsi.iscsi.fosnvioja.zmhojt/configfs": operation not supported

I do have configfs mount for that container present on the host (as per the unit.run)

# grep /configfs /proc/mounts 
none /var/lib/ceph/995d67d2-c216-11ea-87c7-fa163e4f94fe/iscsi.iscsi.fosnvioja.zmhojt/configfs configfs rw,relatime 0 0