It appears I was able to get further this time, the steps are below.
Key difference is, when I did ceph-deploy new I included all the nodes not just the admin.
Will let you know how the rest of install went.
Thanks...
steps
1. On Each Node:
echo "ceph_user ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/ceph_user
sudo chmod 0440 /etc/sudoers.d/ceph_user
2. On admin node only
a. ssh-keygen
b. Create ~/.ssh/config
Host cwtcph001
Hostname cwtcph001
User ceph_user
Host cwtcph002
Hostname cwtcph002
User ceph_user
Host cwtcph003
Hostname cwtcph003
User ceph_user
c. On each Node:
(including admin node since it will do both admin and monitoring).
ssh-copy-id cewit_user@cwtcph001 (Admin + monitoring)
ssh-copy-id cewit_user@cwtcph002
ssh-copy-id cewit_user@cwtcph003
3. mkdir ceph-cluster cd ceph-cluster
4.
a. wget q -O 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc' | sudo apt-key add -
b. echo deb http://ceph.com/debian-firefly/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
c. sudo apt-get update && sudo apt-get install ceph-deploy (admin node)
5. ceph-deploy new cwtcph001 cwtcph002 cwtcph003
output files = ceph.conf ceph.log ceph.mon.keyring
Log file :
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cewit_admin/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.9): /usr/bin/ceph-deploy new cwtcph001 cwtcph002 cwtcph003
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][DEBUG ] Resolving host cwtcph001
[ceph_deploy.new][DEBUG ] Monitor cwtcph001 at 10.1.0.13
[ceph_deploy.new][INFO ] making sure passwordless SSH succeeds
[ceph_deploy.new][DEBUG ] Resolving host cwtcph002
[ceph_deploy.new][DEBUG ] Monitor cwtcph002 at 10.1.0.14
[ceph_deploy.new][INFO ] making sure passwordless SSH succeeds
[cwtcph002][DEBUG ] connected to host: cwtcph001
[cwtcph002][INFO ] Running command: ssh -CT -o BatchMode=yes cwtcph002
[ceph_deploy.new][DEBUG ] Resolving host cwtcph003
[ceph_deploy.new][DEBUG ] Monitor cwtcph003 at 10.1.0.15
[ceph_deploy.new][INFO ] making sure passwordless SSH succeeds
[cwtcph003][DEBUG ] connected to host: cwtcph001
[cwtcph003][INFO ] Running command: ssh -CT -o BatchMode=yes cwtcph003
[ceph_deploy.new][DEBUG ] Monitor initial members are ['cwtcph001', 'cwtcph002', 'cwtcph003']
[ceph_deploy.new][DEBUG ] Monitor addrs are ['10.1.0.13', '10.1.0.14', '10.1.0.15']
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
6. ceph-deploy install cwtcph001 cwtcph002 cwtcph003 (completed with no errors on all 3 nodes).
Node Files:
a. cwtcph001 = ceph.conf ceph.log ceph.mon.keyring release.asc
b. cwtcph002 = home directory release.asc
c. cwtcph003 = home directory release.asc
7. ceph-deploy mon create cwtcph001 cwtcph002 cwtcph003
Appears to have worked this time.
8. ceph-deploy gatherkeys cwtcph001