Ceph

ubuntu@teuthology:/a/teuthology-2013-08-16_01:10:04-ceph-deploy-master-testing-basic-vps/109409$ cat config.yaml 
kernel:
  kdb: true
  sha1: 68d04c9a036bea12d76fdd51e926aef771e8e041
machine_type: vps
nuke-on-error: true
os_type: centos
os_version: '6.4'
overrides:
  admin_socket:
    branch: master
  ceph:
    conf:
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
    log-whitelist:
    - slow request
    sha1: 067d5756a8b17af16ce73a487698471c079a2fb6
  ceph-deploy:
    branch:
      dev: master
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        debug mon: 1
        debug ms: 20
        debug paxos: 20
  install:
    ceph:
      sha1: 067d5756a8b17af16ce73a487698471c079a2fb6
  s3tests:
    branch: master
  workunit:
    sha1: 067d5756a8b17af16ce73a487698471c079a2fb6
roles:
- - mon.a
  - mds.0
  - osd.0
- - mon.b
  - osd.1
- - client.0
targets:
  ubuntu@vpm008.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv1AqKrrBcT/XFYUTzfYxOvAHtw/COM2Q+2/zgyTysCoROGzohhR3dmvKubT6zUGPEXBM8NgSW71AUm8wtcWYPfOFVzYnpWPDdcWaw7zYZWSblm+ufMuHuV4gSojfpXxjXzwYjdXmcFatinqrHw9uU/bVGrfzfim3XyjzjYLsEss0rUUVajKB6EqFTO9BFOGl2xPCEv2rV9GFo3vh7L8Qf3C61Pm7k79nqGx6yGWS9MtnlYBnOE40LVPJGFuli2k9BzQXUuO60huxWc5XLnPu7ObgtMQU3TqSu1VN2cMy7Q1kh76f5/X1DBluiOGQ4MhIqWEDehaMp1zJm/Tdbsigww==
  ubuntu@vpm060.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAtjzg7mdG5R75eqO164QQTXe2w3T8gh70K0FCFtwB908TAzEMs1EiQ70togqyfdcnmEYYv8lH2QTWpvDdpLHRP5a34Jd1sT1FQHlMmQhuBsmG6bBY+gjEffkh0/wy9mIPFwvv8YzHB45py7Qvd2kArhUxqufjuSOmBDb3kbPhuKD+/A79OpemAgLPu1l3UgzcGmY488UxQI3K7FmgEJYi3uM0Cu4VixlgWiOLU3ieOhRTJh2kTM8AIy/SwCYxtvY2YMoTaYdZc3QlzsuoMsoSOHsBR95td8SV4D1yiJFY5cKr+wMCMFj2f+RsgWOh3gE+igiShuGGNxbemc9PeuXsHQ==
  ubuntu@vpm063.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAySgNl2xmwNYQBMEbYtGXCNWtwkGC5EfImlt79FUjtl3x5r5/IxnDSaTaLjNAG9Gv65HROT+/5OcfYtVvz4Y14qkl9kezhY+Zounn1T+tTGR4LUTTYS3FkdB1kasdyZVRSKCknDjVTm58gUK2Ynmp9YzZDgFUdMkIeA3KkmnzmbPfXYOh12s1s32qXwZWLF0VMTqttgW4S/oOsM6kG3PltQSpaeiy4qXIjwezpSrHAYFmgFHtl67GhvhfAVyKQfjm8nNTNXac9oOS/4LQHAlXUyDCUbfoGeHtJebWLmBpQawJhnko46Utpop4TG1mbZizFImwHCdW0RveWI2juSqDLw==
tasks:
- internal.lock_machines:
  - 3
  - vps
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- internal.check_ceph_data: null
- internal.vm_setup: null
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock.check: null
- install:
    extras: true
- ssh_keys: null
- ceph-deploy: null
- workunit:
    clients:
      client.0:
      - rbd/test_librbd.sh
teuthology_branch: master

6.3 fails in the same way:

2013-08-19T02:02:16.748 DEBUG:teuthology.orchestra.run:Running [10.214.138.64]: 'cd /home/ubuntu/cephtest/1288/ceph-deploy && ./ceph-deploy mon create vpm028.front.sepia.ceph.com vpm027.front.sepia.ceph.com'
2013-08-19T02:02:16.974 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mceph_deploy.mon^[[0m][^[[1;34mDEBUG^[[0m ] Deploying mon, cluster ceph hosts vpm028.front.sepia.ceph.com vpm027.front.sepia.ceph.com
2013-08-19T02:02:16.974 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mceph_deploy.mon^[[0m][^[[1;34mDEBUG^[[0m ] detecting platform for host vpm028.front.sepia.ceph.com ...
2013-08-19T02:02:17.330 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mceph_deploy.mon^[[0m][^[[1;37mINFO^[[0m  ] distro info: CentOS 6.3 Final
2013-08-19T02:02:17.330 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;34mDEBUG^[[0m ] deploying mon to vpm028.front.sepia.ceph.com
2013-08-19T02:02:17.334 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;34mDEBUG^[[0m ] remote hostname: vpm028
2013-08-19T02:02:17.346 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;37mINFO^[[0m  ] write cluster configuration to /etc/ceph/{cluster}.conf
2013-08-19T02:02:17.384 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;37mINFO^[[0m  ] creating path: /var/lib/ceph/mon/ceph-vpm028
2013-08-19T02:02:17.388 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;34mDEBUG^[[0m ] checking for done path: /var/lib/ceph/mon/ceph-vpm028/done
2013-08-19T02:02:17.394 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;34mDEBUG^[[0m ] done path does not exist: /var/lib/ceph/mon/ceph-vpm028/done
2013-08-19T02:02:17.400 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;37mINFO^[[0m  ] creating keyring file: /var/lib/ceph/tmp/ceph-vpm028.mon.keyring
2013-08-19T02:02:17.410 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;37mINFO^[[0m  ] create the monitor keyring file
2013-08-19T02:02:17.430 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;37mINFO^[[0m  ] Running command: ceph-mon --cluster ceph --mkfs -i vpm028 --keyring /var/lib/ceph/tmp/ceph-vpm028.mon.keyring
2013-08-19T02:02:17.748 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] Traceback (most recent call last):
2013-08-19T02:02:17.748 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ]   File "/home/ubuntu/cephtest/1288/ceph-deploy/ceph_deploy/hosts/common.py", line 72, in mon_create
2013-08-19T02:02:17.756 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ]   File "/home/ubuntu/cephtest/1288/ceph-deploy/ceph_deploy/util/decorators.py", line 10, in inner
2013-08-19T02:02:17.757 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ]     def inner(*args, **kwargs):
2013-08-19T02:02:17.759 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ]   File "/home/ubuntu/cephtest/1288/ceph-deploy/ceph_deploy/util/wrappers.py", line 6, in remote_call
2013-08-19T02:02:17.761 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ]     This allows us to only remote-execute the actual calls, not whole functions.
2013-08-19T02:02:17.762 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ]   File "/usr/lib64/python2.6/subprocess.py", line 505, in check_call
2013-08-19T02:02:17.764 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ]     raise CalledProcessError(retcode, cmd)
2013-08-19T02:02:17.765 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] CalledProcessError: Command '['ceph-mon', '--cluster', 'ceph', '--mkfs', '-i', 'vpm028', '--keyring', '/var/lib/ceph/tmp/ceph-vpm028.mon.keyring']' returned non-zero exit status -6
2013-08-19T02:02:17.773 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;37mINFO^[[0m  ] ceph-mon: mon.noname-a 10.214.138.64:6789/0 is local, renaming to mon.vpm028
2013-08-19T02:02:17.774 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;37mINFO^[[0m  ] ceph-mon: set fsid to aef793bf-e8ff-42d8-a936-1b24d89ac79e
2013-08-19T02:02:17.774 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] pthread lock: Invalid argument
2013-08-19T02:02:17.774 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] *** Caught signal (Aborted) **
2013-08-19T02:02:17.774 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ]  in thread 7f3bb93ed7a0
2013-08-19T02:02:17.775 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ]  ceph version 0.67-215-g4677041 (4677041da309234feb7b7df28e1d73db99d27c72)
2013-08-19T02:02:17.775 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ]  1: ceph-mon() [0x6483f9]
2013-08-19T02:02:17.775 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ]  2: (()+0xf500) [0x7f3bb8a5b500]
2013-08-19T02:02:17.775 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ]  3: (gsignal()+0x35) [0x7f3bb74678a5]
2013-08-19T02:02:17.775 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ]  4: (abort()+0x175) [0x7f3bb7469085]
2013-08-19T02:02:17.776 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ]  5: (()+0x34d71) [0x7f3bb7f9cd71]
2013-08-19T02:02:17.776 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ]  6: (leveldb::DBImpl::Get(leveldb::ReadOptions const&, leveldb::Slice const&, leveldb::Value*)+0x50) [0x7f3bb7f82120]
2013-08-19T02:02:17.776 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ]  7: (LevelDBStore::_get_iterator()+0x41) [0x632d71]
2013-08-19T02:02:17.776 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ]  8: (MonitorDBStore::exists(std::string const&, std::string const&)+0x28) [0x523898]
2013-08-19T02:02:17.777 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ]  9: (Monitor::check_fsid()+0x59) [0x54f069]
2013-08-19T02:02:17.777 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ]  10: (Monitor::mkfs(ceph::buffer::list&)+0x36) [0x5566e6]
2013-08-19T02:02:17.777 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ]  11: (main()+0x11a3) [0x51e223]
2013-08-19T02:02:17.778 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ]  12: (__libc_start_main()+0xfd) [0x7f3bb7453cdd]
2013-08-19T02:02:17.778 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ]  13: ceph-mon() [0x51bcb9]

Looks like the old leveldb-1.7.0 package works ok with dumpling on centos6.3, but the new leveldb-1.12 package does not.

Testing new ceph 0.67.1 install with a prior install of leveldb from epel:

Installed Packages
leveldb.x86_64 1.7.0-2.el6 @epel

ceph-deploy mon create was successful.

Testing with leveldb 1.12

Installed Packages
leveldb.x86_64 1.12.0-3.el6 installed

ceph-deploy hangs in the mon create command, with this back trace:

(gdb) bt
#0 0x00007fee39985dd0 in __pause_nocancel () from /lib64/libpthread.so.0
#1 0x00007fee3997fffb in __pthread_mutex_lock_full () from /lib64/libpthread.so.0
#2 0x00007fee38ec7e79 in leveldb::port::Mutex::Lock (this=<value optimized out>) at port_posix.cc:26
#3 0x00007fee38ead120 in MutexLock (this=0x7fffbbc27ca0, options=..., key=..., value=0x81cff)
at ../util/mutexlock.h:27
#4 leveldb::DBImpl::Get (this=0x7fffbbc27ca0, options=..., key=..., value=0x81cff) at db_impl.cc:1098
#5 0x0000000000630551 in LevelDBStore::_get_iterator() ()
#6 0x0000000000523368 in MonitorDBStore::exists(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
#7 0x000000000054c499 in Monitor::check_fsid() ()
#8 0x000000000055a6a6 in Monitor::mkfs(ceph::buffer::list&) ()
#9 0x000000000051dcf3 in main ()

Removed ceph and leveldb-1.12 and resintalled. leveldb dependencies was resolved from epel:

Dependencies Resolved

==========================================================================================================
Package Arch Version Repository Size ==========================================================================================================
Installing:
ceph x86_64 0.67.1-0.el6 ceph 15 M
Installing for dependencies:
leveldb x86_64 1.7.0-2.el6 epel 158 k
libcephfs1 x86_64 0.67.1-0.el6 ceph 1.7 M
librados2 x86_64 0.67.1-0.el6 ceph 1.5 M
librbd1 x86_64 0.67.1-0.el6 ceph 268 k
python-ceph x86_64 0.67.1-0.el6 ceph 120 k

ceph-deploy create mon worked correctly:

[glowell@gary-centos-01 test]$ ceph-deploy new gary-centos-01
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][DEBUG ] Resolving host gary-centos-01
[ceph_deploy.new][DEBUG ] Monitor gary-centos-01 at 10.214.140.136
[ceph_deploy.new][DEBUG ] Monitor initial members are ['gary-centos-01']
[ceph_deploy.new][DEBUG ] Monitor addrs are ['10.214.140.136']
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
[glowell@gary-centos-01 test]$
[glowell@gary-centos-01 test]$
[glowell@gary-centos-01 test]$ ceph-deploy mon create gary-centos-01
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts gary-centos-01
[ceph_deploy.mon][DEBUG ] detecting platform for host gary-centos-01 ...
[ceph_deploy.mon][INFO ] distro info: CentOS 6.3 Final
[gary-centos-01][DEBUG ] deploying mon to gary-centos-01
[gary-centos-01][DEBUG ] remote hostname: gary-centos-01
[gary-centos-01][INFO ] write cluster configuration to /etc/ceph/{cluster}.conf
[gary-centos-01][INFO ] creating path: /var/lib/ceph/mon/ceph-gary-centos-01
[gary-centos-01][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-gary-centos-01/done
[gary-centos-01][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-gary-centos-01/done
[gary-centos-01][INFO ] creating keyring file: /var/lib/ceph/tmp/ceph-gary-centos-01.mon.keyring
[gary-centos-01][INFO ] create the monitor keyring file
[gary-centos-01][INFO ] Running command: ceph-mon --cluster ceph --mkfs -i gary-centos-01 --keyring /var/lib/ceph/tmp/ceph-gary-centos-01.mon.keyring
[gary-centos-01][INFO ] ceph-mon: mon.noname-a 10.214.140.136:6789/0 is local, renaming to mon.gary-centos-01
[gary-centos-01][INFO ] ceph-mon: set fsid to 16efbd8e-190c-4782-a1c8-de122600b710
[gary-centos-01][INFO ] ceph-mon: created monfs at /var/lib/ceph/mon/ceph-gary-centos-01 for mon.gary-centos-01
[gary-centos-01][INFO ] unlinking keyring file /var/lib/ceph/tmp/ceph-gary-centos-01.mon.keyring
[gary-centos-01][INFO ] create a done file to avoid re-doing the mon deployment
[gary-centos-01][INFO ] create the init path if it does not exist
[gary-centos-01][INFO ] locating `service` executable...
[gary-centos-01][INFO ] found `service` executable: /sbin/service
[gary-centos-01][INFO ] Running command: /sbin/service ceph start mon.gary-centos-01
[glowell@gary-centos-01 test]$
[glowell@gary-centos-01 test]$ ceph-deploy gatherkeys gary-centos-01
[ceph_deploy.gatherkeys][DEBUG ] Checking gary-centos-01 for /etc/ceph/ceph.client.admin.keyring
[ceph_deploy.gatherkeys][DEBUG ] Got ceph.client.admin.keyring key from gary-centos-01.
[ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring
[ceph_deploy.gatherkeys][DEBUG ] Checking gary-centos-01 for /var/lib/ceph/bootstrap-osd/ceph.keyring
[ceph_deploy.gatherkeys][DEBUG ] Got ceph.bootstrap-osd.keyring key from gary-centos-01.
[ceph_deploy.gatherkeys][DEBUG ] Checking gary-centos-01 for /var/lib/ceph/bootstrap-mds/ceph.keyring
[ceph_deploy.gatherkeys][DEBUG ] Got ceph.bootstrap-mds.keyring key from gary-centos-01.
[glowell@gary-centos-01 test]$
[glowell@gary-centos-01 test]$ yum list installed leveldb
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile * base: mirrors.kernel.org * epel: mirrors.xmission.com * extras: mirrors.syringanetworks.net * updates: centos-distro.cavecreek.net
Installed Packages
leveldb.x86_64 1.7.0-2.el6 @epel
[glowell@gary-centos-01 test]$
[glowell@gary-centos-01 test]$ yum list installed ceph
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile * base: mirrors.kernel.org * epel: mirrors.xmission.com * extras: mirrors.syringanetworks.net * updates: centos-distro.cavecreek.net
Installed Packages
ceph.x86_64 0.67.1-0.el6 @ceph
[glowell@gary-centos-01 test]$

Rebuilding leveldb-1.12 without the Basho patch seems to work ok. This patch is described as:

1. Cherry-picked from Basho's fork
   Patch6: leveldb-0006-allow-Get-calls-to-avoid-copies-into-std-string.patch

[glowell@gary-centos-01 test]$ ceph-deploy new gary-centos-01
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][DEBUG ] Resolving host gary-centos-01
[ceph_deploy.new][DEBUG ] Monitor gary-centos-01 at 10.214.140.136
[ceph_deploy.new][DEBUG ] Monitor initial members are ['gary-centos-01']
[ceph_deploy.new][DEBUG ] Monitor addrs are ['10.214.140.136']
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
[glowell@gary-centos-01 test]$
[glowell@gary-centos-01 test]$ ceph-deploy mon create gary-centos-01
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts gary-centos-01
[ceph_deploy.mon][DEBUG ] detecting platform for host gary-centos-01 ...
[ceph_deploy.mon][INFO ] distro info: CentOS 6.3 Final
[gary-centos-01][DEBUG ] deploying mon to gary-centos-01
[gary-centos-01][DEBUG ] remote hostname: gary-centos-01
[gary-centos-01][INFO ] write cluster configuration to /etc/ceph/{cluster}.conf
[gary-centos-01][INFO ] creating path: /var/lib/ceph/mon/ceph-gary-centos-01
[gary-centos-01][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-gary-centos-01/done
[gary-centos-01][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-gary-centos-01/done
[gary-centos-01][INFO ] creating keyring file: /var/lib/ceph/tmp/ceph-gary-centos-01.mon.keyring
[gary-centos-01][INFO ] create the monitor keyring file
[gary-centos-01][INFO ] Running command: ceph-mon --cluster ceph --mkfs -i gary-centos-01 --keyring /var/lib/ceph/tmp/ceph-gary-centos-01.mon.keyring
[gary-centos-01][INFO ] ceph-mon: mon.noname-a 10.214.140.136:6789/0 is local, renaming to mon.gary-centos-01
[gary-centos-01][INFO ] ceph-mon: set fsid to 68d35035-6547-4544-b5ce-e4cb62c3af7c
[gary-centos-01][INFO ] ceph-mon: created monfs at /var/lib/ceph/mon/ceph-gary-centos-01 for mon.gary-centos-01
[gary-centos-01][INFO ] unlinking keyring file /var/lib/ceph/tmp/ceph-gary-centos-01.mon.keyring
[gary-centos-01][INFO ] create a done file to avoid re-doing the mon deployment
[gary-centos-01][INFO ] create the init path if it does not exist
[gary-centos-01][INFO ] locating `service` executable...
[gary-centos-01][INFO ] found `service` executable: /sbin/service
[gary-centos-01][INFO ] Running command: /sbin/service ceph start mon.gary-centos-01
[glowell@gary-centos-01 test]$