Ceph

It looks like custom cluster names are not being handled correctly by the init script.

This is reproducible with ceph-deploy:

$ ceph-deploy --cluster cephtst mon create node2
[ceph_deploy.conf][DEBUG ] found configuration file at: /Users/alfredo/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.2): /Users/alfredo/.virtualenvs/ceph-deploy/bin/ceph-deploy --cluster cephtst mon create node2
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster cephtst hosts node2
[ceph_deploy.mon][DEBUG ] detecting platform for host node2 ...
[node2][DEBUG ] connected to host: node2
[node2][DEBUG ] detect platform information from remote host
[node2][DEBUG ] detect machine type
[ceph_deploy.mon][INFO  ] distro info: CentOS 6.4 Final
[node2][DEBUG ] determining if provided host has same hostname in remote
[node2][DEBUG ] get remote short hostname
[node2][DEBUG ] deploying mon to node2
[node2][DEBUG ] get remote short hostname
[node2][DEBUG ] remote hostname: node2
[node2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node2][DEBUG ] create the mon path if it does not exist
[node2][DEBUG ] checking for done path: /var/lib/ceph/mon/cephtst-node2/done
[node2][DEBUG ] done path does not exist: /var/lib/ceph/mon/cephtst-node2/done
[node2][INFO  ] creating keyring file: /var/lib/ceph/tmp/cephtst-node2.mon.keyring
[node2][DEBUG ] create the monitor keyring file
[node2][INFO  ] Running command: sudo ceph-mon --cluster cephtst --mkfs -i node2 --keyring /var/lib/ceph/tmp/cephtst-node2.mon.keyring
[node2][DEBUG ] ceph-mon: mon.noname-a 192.168.111.101:6789/0 is local, renaming to mon.node2
[node2][DEBUG ] ceph-mon: set fsid to f6cd60a4-bc1a-4d24-ab6f-39de9e59eabd
[node2][DEBUG ] ceph-mon: created monfs at /var/lib/ceph/mon/cephtst-node2 for mon.node2
[node2][INFO  ] unlinking keyring file /var/lib/ceph/tmp/cephtst-node2.mon.keyring
[node2][DEBUG ] create a done file to avoid re-doing the mon deployment
[node2][DEBUG ] create the init path if it does not exist
[node2][DEBUG ] locating the `service` executable...
[node2][INFO  ] Running command: sudo /sbin/service ceph -c /etc/ceph/cephtst.conf start mon.node2
[node2][INFO  ] Running command: sudo ceph --cluster=cephtst --admin-daemon /var/run/ceph/cephtst-mon.node2.asok mon_status
[node2][ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
[node2][WARNIN] monitor: mon.node2, might not be running yet
[node2][INFO  ] Running command: sudo ceph --cluster=cephtst --admin-daemon /var/run/ceph/cephtst-mon.node2.asok mon_status
[node2][ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
[node2][WARNIN] monitor node2 does not exist in monmap
[node2][WARNIN] neither `public_addr` nor `public_network` keys are defined for monitors
[node2][WARNIN] monitors may not be able to form quorum

ceph-deploy does make extensive use of `--cluster` in every command that accepts this, except for when starting the actual deamon (--cluster is not allowed and inferred from the ceph config file)

[vagrant@node2 ~]$ sudo /etc/init.d/ceph --cluster cephtest -c /etc/ceph/cephtst.conf start mon.node2
unrecognized option '--cluster'
usage: /etc/init.d/ceph [options] {start|stop|restart|condrestart} [mon|osd|mds]...
    -c ceph.conf
    --valgrind    run via valgrind
    --hostname [hostname]    override hostname lookup
[vagrant@node2 ~]$ sudo /etc/init.d/ceph --cluster=cephtest -c /etc/ceph/cephtst.conf start mon.node2
unrecognized option '--cluster=cephtest'
usage: /etc/init.d/ceph [options] {start|stop|restart|condrestart} [mon|osd|mds]...
    -c ceph.conf
    --valgrind    run via valgrind
    --hostname [hostname]    override hostname lookup

If the daemon is started manually (without ceph-deploy and without using --cluster) nothing happens. No log gets written, nothing goes to stderr nor stdout
and the exit code is 0.

[vagrant@node2 ~]$ sudo /etc/init.d/ceph -c /etc/ceph/cephtst.conf start mon.node2
[vagrant@node2 ~]$ echo $?
0
[vagrant@node2 ~]$ ls /var/run/ceph/
[vagrant@node2 ~]$ ls /var/log/ceph/

After adding a few print statements on the init script to see where execution stops, it seems to be this line:

    check_host || continue

So check_host fails. So I go to check_host to see what is going on, and the place where this all fails is here:

    # ignore all sections without 'host' defined
    if [ -z "$host" ]; then
        return 1
    fi

So `$host` is never resolved, and that variable comes from this one command a few lines into that function:

    host=`$CCONF -c $conf -n $type.$id host`

For my example above, that translates into this one-liner:

[vagrant@node2 ~]$ /usr/bin/ceph-conf -c /etc/ceph/cephtst.conf --name mon.node2 host
[vagrant@node2 ~]$ echo $?
1

And this is where I am not sure what else to try. This of course does not happen at all when there is no custom cluster name being used.