Project

General

Profile

Bug #46910

cephadm: Unable to create an iSCSI target

Added by Ricardo Marques 10 months ago. Updated 7 months ago.

Status:
Can't reproduce
Priority:
Normal
Category:
cephadm
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

When I try to create an iSCSI target I get the following error:

On `ceph-iscsi` container logs I see:

Aug 11 16:27:02 node1 bash[26919]: debug ::1 - - [11/Aug/2020 14:27:02] "GET /api/_ping HTTP/1.1" 200 -
Aug 11 16:27:03 node1 bash[26919]: debug (LUN.add_dev_to_lio) Adding image 'mypool/myrbd1' to LIO backstore user:rbd
Aug 11 16:27:03 node1 bash[26919]: debug failed to add mypool/myrbd1 to LIO - error([Errno 2] No such file or directory)
Aug 11 16:27:03 node1 bash[26919]: debug LUN alloc problem - failed to add mypool/myrbd1 to LIO - error([Errno 2] No such file or directory)

My service spec:

# tail -14 service_spec_gw.yml
---
service_type: iscsi
service_id: iscsi_service
placement:
    hosts:
        - 'node1'
        - 'node2'
spec:
    pool: rbd
    trusted_ip_list: 10.20.139.201,10.20.139.202,10.20.139.203
    api_port: 5000
    api_user: admin1
    api_password: admin2
    api_secure: False

My test environment was created using the following `sesdev` command:

sesdev create pacific --ceph-salt-repo https://github.com/ceph/ceph-salt.git --ceph-salt-branch master pacific

error-creating-target.png View (72.6 KB) Ricardo Marques, 08/12/2020 11:09 AM

History

#1 Updated by Matthew Oliver 10 months ago

I think I've had this before, if I remember correctly I've seen this when the tcmu-runner container was running but didn't have /dev volume mounted into the container.

What does your tcmu-runner unit.run look like?

#2 Updated by Ricardo Marques 10 months ago

node1:~ # podman ps | grep tcmu
aad4b80fe431  registry.opensuse.org/filesystems/ceph/master/upstream/images/ceph/ceph:latest  /bin/bash --login     3 minutes ago  Up 3 minutes ago         ceph-28341724-dc8b-11ea-bb36-5254009df885-iscsi.iscsi_service.node1.kghzlz-tcmu

node1:~ # podman exec -ti aad4b80fe431 /bin/bash

node1:/ # ps -ef | grep tcmu 
root         1     0  0 13:02 ?        00:00:00 /usr/bin/tcmu-runner
root        48    10  0 13:09 pts/1    00:00:00 grep tcmu
node1:/ # cat /var/log/tcmu-runner.log 
2020-08-13 13:02:31.499 1 [INFO] load_our_module:548: Inserted module 'target_core_user'
2020-08-13 13:04:35.675 1 [ERROR] add_device:456: could not find handler for uio0
2020-08-13 13:04:35.677 1 [ERROR] add_device:456: could not find handler for uio0
node1:/ # ls /dev
autofs                       dm-1       lightnvm          port        stdin   tty18  tty3   tty41  tty53    tty8    ttyS19    ttyS30     vcs    vcsa6  vdc
block                       dri          log          ppp        stdout  tty19  tty30  tty42  tty54    tty9    ttyS2    ttyS31     vcs1    vcsu   vfio
btrfs-control                   fb0          loop-control      psaux     tty     tty2   tty31  tty43  tty55    ttyS0    ttyS20    ttyS4     vcs2    vcsu1  vga_arbiter
bus                       fd          mapper          ptmx        tty0    tty20  tty32  tty44  tty56    ttyS1    ttyS21    ttyS5     vcs3    vcsu2  vhci
ceph-19b86412-4dec-4956-bdef-bde1259f60a3  full       mcelog          pts        tty1    tty21  tty33  tty45  tty57    ttyS10    ttyS22    ttyS6     vcs4    vcsu3  vhost-net
ceph-4d581afd-a93b-4553-95aa-4d31904167b3  fuse       mem          random    tty10   tty22  tty34  tty46  tty58    ttyS11    ttyS23    ttyS7     vcs5    vcsu4  vhost-vsock
char                       hpet       memory_bandwidth      rfkill    tty11   tty23  tty35  tty47  tty59    ttyS12    ttyS24    ttyS8     vcs6    vcsu5  zero
console                       hugepages  mqueue          rtc        tty12   tty24  tty36  tty48  tty6    ttyS13    ttyS25    ttyS9     vcsa    vcsu6
core                       hwrng      net          rtc0        tty13   tty25  tty37  tty49  tty60    ttyS14    ttyS26    udmabuf  vcsa1    vda
cpu_dma_latency                   initctl    network_latency      shm        tty14   tty26  tty38  tty5     tty61    ttyS15    ttyS27    uhid     vcsa2    vda1
cuse                       input      network_throughput  snapshot  tty15   tty27  tty39  tty50  tty62    ttyS16    ttyS28    uinput     vcsa3    vda2
disk                       kmsg       null          snd        tty16   tty28  tty4   tty51  tty63    ttyS17    ttyS29    urandom  vcsa4    vda3
dm-0                       kvm          nvram          stderr    tty17   tty29  tty40  tty52  tty7    ttyS18    ttyS3    userio     vcsa5    vdb

node1:/ # exit

node1:~ # ls /dev
autofs                       dm-1       lightnvm          port        stdin   tty18  tty3   tty41  tty53    tty8    ttyS19    ttyS30     vcs    vcsa6  vdc
block                       dri          log          ppp        stdout  tty19  tty30  tty42  tty54    tty9    ttyS2    ttyS31     vcs1    vcsu   vfio
btrfs-control                   fb0          loop-control      psaux     tty     tty2   tty31  tty43  tty55    ttyS0    ttyS20    ttyS4     vcs2    vcsu1  vga_arbiter
bus                       fd          mapper          ptmx        tty0    tty20  tty32  tty44  tty56    ttyS1    ttyS21    ttyS5     vcs3    vcsu2  vhci
ceph-19b86412-4dec-4956-bdef-bde1259f60a3  full       mcelog          pts        tty1    tty21  tty33  tty45  tty57    ttyS10    ttyS22    ttyS6     vcs4    vcsu3  vhost-net
ceph-4d581afd-a93b-4553-95aa-4d31904167b3  fuse       mem          random    tty10   tty22  tty34  tty46  tty58    ttyS11    ttyS23    ttyS7     vcs5    vcsu4  vhost-vsock
char                       hpet       memory_bandwidth      rfkill    tty11   tty23  tty35  tty47  tty59    ttyS12    ttyS24    ttyS8     vcs6    vcsu5  zero
console                       hugepages  mqueue          rtc        tty12   tty24  tty36  tty48  tty6    ttyS13    ttyS25    ttyS9     vcsa    vcsu6
core                       hwrng      net          rtc0        tty13   tty25  tty37  tty49  tty60    ttyS14    ttyS26    udmabuf  vcsa1    vda
cpu_dma_latency                   initctl    network_latency      shm        tty14   tty26  tty38  tty5     tty61    ttyS15    ttyS27    uhid     vcsa2    vda1
cuse                       input      network_throughput  snapshot  tty15   tty27  tty39  tty50  tty62    ttyS16    ttyS28    uinput     vcsa3    vda2
disk                       kmsg       null          snd        tty16   tty28  tty4   tty51  tty63    ttyS17    ttyS29    urandom  vcsa4    vda3
dm-0                       kvm          nvram          stderr    tty17   tty29  tty40  tty52  tty7    ttyS18    ttyS3    userio     vcsa5    vdb

#3 Updated by Sebastian Wagner 10 months ago

  • Assignee set to Matthew Oliver

#4 Updated by Matthew Oliver 10 months ago

  • Assignee deleted (Matthew Oliver)

Finally tracked down the issue. Turns out our suse ceph image doesn't contain the tcmu-runner-handler-rbd package. If I install that in the container it works.

This is the handler that allows tcmu-runner to talk to librbd.

Next I'll need to go figure out where it's built, I assume obs/ibs.

#5 Updated by Matthew Oliver 10 months ago

  • Assignee set to Matthew Oliver

#6 Updated by Matthew Oliver 10 months ago

Created an open build system (OBS) request (PR) to add the missing pacakage to the ceph-base pattern used in building the Suse container: https://build.opensuse.org/request/show/830139

#7 Updated by Matthew Oliver 10 months ago

The first OBS request has been accepted. The suse product container doesn't suffer from the same problem. It seems it's only the OBS/LEAP container side.

#8 Updated by Sebastian Wagner 7 months ago

  • Status changed from New to Can't reproduce

Also available in: Atom PDF