Ceph » devops

I can't replicate this behavior at all. I do know why that is the error that comes about though.

What happens is that ceph-disk tries to mount the device but before it does that it attempts to detect the device file type:

    try:
        fstype = detect_fstype(dev=dev)
    except (subprocess.CalledProcessError,
            TruncatedLineError,
            TooManyLinesError) as e:
        raise FilesystemTypeError(
            'device {dev}'.format(dev=dev),
            e,
            )

That call to detect_fstype fails because the result is not a single line (there is a check for this) so that is why there is a mixed error in the output.

The reason why the error looks like this (without anything readable after the colon) is because the result is empty:

[hosta][WARNIN] ceph-disk: Cannot discover filesystem type: device /dev/sdb: Line is truncated: 

The handler for that error will add the output to the exception, but since there is no output, the exception looks like it is missing a part of the error.

This is the command that detect_fstype is calling:

sudo blkid -p -s TYPE -o value {device}

The biggest problem here is that we are using subprocess.Popen and later we call communicate() which returns a tuple (stdout and stderr) and we only
use stdout and completely discard stderr.

So in the case of blkid, useful error information is spit out to stderr that we discard. See:

In [21]: result = Popen(['sudo', 'blkid', '-p', '-s', 'TYPE', '-o', 'value', '/dev/sdb1'], stdout=PIPE)

In [22]: out, _ = result.communicate()

In [23]: print out

In [24]: print _
None

Not only '_' is None, it is so because we are not even telling subprocess to capture it, this would look like:

In [25]: result = Popen(['sudo', 'blkid', '-p', '-s', 'TYPE', '-o', 'value', '/dev/sdb1'], stdout=PIPE, stderr=PIPE)

In [26]: out, _ = result.communicate()

In [27]: print out

In [28]: print _
error: /dev/sdb1: No such file or directory

AHA! No such file or directory! THIS IS USEFUL OUTPUT.

The only thing I think I can do here is just improve error reporting in ceph-disk, Unless someone can replicate the behavior from the bug report reliably so I can
make the changes to ceph-disk and see what the actual error is.

One of the differences here is that you cannot call 'activate' without doing a 'prepare' first.

This behavior would be expected because there is the chance that a device's file system will not be
able to be detected.

For example:

ssh node3 sudo fdisk -l

WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted.

Disk /dev/mapper/vg0-root doesn't contain a valid partition table
Disk /dev/mapper/vg0-swap doesn't contain a valid partition table

Disk /dev/sda: 44.0 GB, 44040192000 bytes
255 heads, 63 sectors/track, 5354 cylinders, total 86016000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0007137a

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     1757183      877568   83  Linux
/dev/sda2         1759230    86013951    42127361    5  Extended
/dev/sda5         1759232    86013951    42127360   8e  Linux LVM

Disk /dev/sdb: 8589 MB, 8589934592 bytes
256 heads, 63 sectors/track, 1040 cylinders, total 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1    16777215     8388607+  ee  GPT

So in this case, if we call 'blkid' it will fail to detect what the filesystem is:

vagrant@node3:~$ sudo /sbin/blkid p -s TYPE -o value - /dev/sdb
vagrant@node3:~$ echo $?
2

If I insist and try to 'activate' that device, the error comes up:

ceph-deploy osd activate node3:/dev/sdb
[ceph_deploy.conf][DEBUG ] found configuration file at: /Users/alfredo/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.3): /Users/alfredo/.virtualenvs/ceph-deploy/bin/ceph-deploy osd activate node3:/dev/sdb
[ceph_deploy.osd][DEBUG ] Activating cluster ceph disks node3:/dev/sdb:
[node3][DEBUG ] connected to host: node3
[node3][DEBUG ] detect platform information from remote host
[node3][DEBUG ] detect machine type
[ceph_deploy.osd][INFO  ] Distro info: Ubuntu 14.04 trusty
[ceph_deploy.osd][DEBUG ] activating host node3 disk /dev/sdb
[ceph_deploy.osd][DEBUG ] will use init type: upstart
[node3][INFO  ] Running command: sudo ceph-disk-activate --mark-init upstart --mount /dev/sdb
[node3][WARNIN] ceph-disk: Cannot discover filesystem type: device /dev/sdb: Line is truncated:
[node3][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: ceph-disk-activate --mark-init upstart --mount /dev/sdb

If I call 'create' instead of 'activate' on the whole device, it works:

$ ceph-deploy osd --zap create node3:/dev/sdb
[ceph_deploy.conf][DEBUG ] found configuration file at: /Users/alfredo/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.3): /Users/alfredo/.virtualenvs/ceph-deploy/bin/ceph-deploy osd --zap create node3:/dev/sdb
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks node3:/dev/sdb:
[node3][DEBUG ] connected to host: node3
[node3][DEBUG ] detect platform information from remote host
[node3][DEBUG ] detect machine type
[ceph_deploy.osd][INFO  ] Distro info: Ubuntu 14.04 trusty
[ceph_deploy.osd][DEBUG ] Deploying osd to node3
[node3][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node3][INFO  ] Running command: sudo udevadm trigger --subsystem-match=block --action=add
[ceph_deploy.osd][DEBUG ] Preparing host node3 disk /dev/sdb journal None activate True
[node3][INFO  ] Running command: sudo ceph-disk-prepare --zap-disk --fs-type xfs --cluster ceph -- /dev/sdb
[node3][DEBUG ] ****************************************************************************
[node3][DEBUG ] Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
[node3][DEBUG ] verification and recovery are STRONGLY recommended.
[node3][DEBUG ] ****************************************************************************
[node3][DEBUG ] GPT data structures destroyed! You may now partition the disk using fdisk or
[node3][DEBUG ] other utilities.
[node3][DEBUG ] The operation has completed successfully.
[node3][DEBUG ] The operation has completed successfully.
[node3][DEBUG ] The operation has completed successfully.
[node3][DEBUG ] meta-data=/dev/sdb1              isize=2048   agcount=4, agsize=196543 blks
[node3][DEBUG ]          =                       sectsz=512   attr=2, projid32bit=0
[node3][DEBUG ] data     =                       bsize=4096   blocks=786171, imaxpct=25
[node3][DEBUG ]          =                       sunit=0      swidth=0 blks
[node3][DEBUG ] naming   =version 2              bsize=4096   ascii-ci=0
[node3][DEBUG ] log      =internal log           bsize=4096   blocks=2560, version=2
[node3][DEBUG ]          =                       sectsz=512   sunit=0 blks, lazy-count=1
[node3][DEBUG ] realtime =none                   extsz=4096   blocks=0, rtextents=0
[node3][DEBUG ] The operation has completed successfully.
[node3][WARNIN] Caution: invalid backup GPT header, but valid main header; regenerating
[node3][WARNIN] backup header from main header.
[node3][WARNIN]
[node3][WARNIN] Warning! Main and backup partition tables differ! Use the 'c' and 'e' options
[node3][WARNIN] on the recovery & transformation menu to examine the two tables.
[node3][WARNIN]
[node3][WARNIN] Warning! One or more CRCs don't match. You should repair the disk!
[node3][WARNIN]
[node3][WARNIN] INFO:ceph-disk:Will colocate journal with data on /dev/sdb
[node3][INFO  ] Running command: sudo udevadm trigger --subsystem-match=block --action=add
[node3][INFO  ] checking OSD status...
[node3][INFO  ] Running command: sudo ceph --cluster=ceph osd stat --format=json
[ceph_deploy.osd][DEBUG ] Host node3 is now ready for osd use.

The device is ready and mounted:

$ ceph-deploy osd list node3
[ceph_deploy.conf][DEBUG ] found configuration file at: /Users/alfredo/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.3): /Users/alfredo/.virtualenvs/ceph-deploy/bin/ceph-deploy osd list node3
[node3][DEBUG ] connected to host: node3
[node3][DEBUG ] detect platform information from remote host
[node3][DEBUG ] detect machine type
[node3][INFO  ] Running command: sudo ceph --cluster=ceph osd tree --format=json
[node3][DEBUG ] connected to host: node3
[node3][DEBUG ] detect platform information from remote host
[node3][DEBUG ] detect machine type
[node3][INFO  ] Running command: sudo ceph-disk list
[node3][INFO  ] ----------------------------------------
[node3][INFO  ] ceph-0
[node3][INFO  ] ----------------------------------------
[node3][INFO  ] Path           /var/lib/ceph/osd/ceph-0
[node3][INFO  ] ID             0
[node3][INFO  ] Name           osd.0
[node3][INFO  ] Status         up
[node3][INFO  ] Reweight       1.000000
[node3][INFO  ] Magic          ceph osd volume v026
[node3][INFO  ] Journal_uuid   980f8734-e5c4-4124-bfbc-deb2ad0d5c8e
[node3][INFO  ] Active         ok
[node3][INFO  ] Device         /dev/sdb1
[node3][INFO  ] Whoami         0
[node3][INFO  ] Journal path   /dev/sdb2
[node3][INFO  ] ----------------------------------------

Can someone confirm that calling 'create' would work? (as opposed to activate on a whole device)