Ceph » devops

It looks to me like the last command that mon create ran is finished, but pushy still has its connection open. that function does almost nothing except check_call() (which we see ran and completed), but pushy didn't finish the call. possibly a problem with the close() call?

distro.mon.create(distro, rlogger, args, monitor_keyring)
            distro.sudo_conn.close()

Alfredo, I mentioned that the issue is not reproducible on a single node but the problem still exists with 2 nodes.

Ugh, I keep failing to replicate :(

The error on the second one is the famous ulimit configuration but that is not related to this.

ceph-deploy mon create burnupi27.front.sepia.ceph.com burnupi28.front.sepia.ceph.com
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts burnupi27.front.sepia.ceph.com burnupi28.front.sepia.ceph.com
[ceph_deploy.mon][DEBUG ] detecting platform for host burnupi27.front.sepia.ceph.com ...
[ceph_deploy.mon][INFO  ] distro info: RedHatEnterpriseServer 6.3 Santiago
[burnupi27.front.sepia.ceph.com][DEBUG ] deploying mon to burnupi27.front.sepia.ceph.com
[burnupi27.front.sepia.ceph.com][DEBUG ] remote hostname: burnupi27
[burnupi27.front.sepia.ceph.com][INFO  ] write cluster configuration to /etc/ceph/{cluster}.conf
[burnupi27.front.sepia.ceph.com][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-burnupi27/done
[burnupi27.front.sepia.ceph.com][INFO  ] create a done file to avoid re-doing the mon deployment
[burnupi27.front.sepia.ceph.com][INFO  ] create the init path if it does not exist
[burnupi27.front.sepia.ceph.com][INFO  ] locating `service` executable...
[burnupi27.front.sepia.ceph.com][INFO  ] found `service` executable: /sbin/service
[burnupi27.front.sepia.ceph.com][INFO  ] Running command: /sbin/service ceph start mon.burnupi27
[burnupi27.front.sepia.ceph.com][INFO  ] === mon.burnupi27 ===
[burnupi27.front.sepia.ceph.com][INFO  ] Starting Ceph mon.burnupi27 on burnupi27...
[burnupi27.front.sepia.ceph.com][INFO  ] Starting ceph-create-keys on burnupi27...
[ceph_deploy.mon][DEBUG ] detecting platform for host burnupi28.front.sepia.ceph.com ...
The authenticity of host 'burnupi28.front.sepia.ceph.com (10.214.135.34)' can't be established.
RSA key fingerprint is 74:f1:99:c3:ee:55:45:f0:39:0c:48:ac:f3:09:79:90.
Are you sure you want to continue connecting (yes/no)? yes
[ceph_deploy.mon][INFO  ] distro info: RedHatEnterpriseServer 6.3 Santiago
[burnupi28.front.sepia.ceph.com][DEBUG ] deploying mon to burnupi28.front.sepia.ceph.com
[burnupi28.front.sepia.ceph.com][DEBUG ] remote hostname: burnupi28
[burnupi28.front.sepia.ceph.com][INFO  ] write cluster configuration to /etc/ceph/{cluster}.conf
[burnupi28.front.sepia.ceph.com][INFO  ] creating path: /var/lib/ceph/mon/ceph-burnupi28
[burnupi28.front.sepia.ceph.com][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-burnupi28/done
[burnupi28.front.sepia.ceph.com][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-burnupi28/done
[burnupi28.front.sepia.ceph.com][INFO  ] creating keyring file: /var/lib/ceph/tmp/ceph-burnupi28.mon.keyring
[burnupi28.front.sepia.ceph.com][INFO  ] create the monitor keyring file
[burnupi28.front.sepia.ceph.com][INFO  ] Running command: ceph-mon --cluster ceph --mkfs -i burnupi28 --keyring /var/lib/ceph/tmp/ceph-burnupi28.mon.keyring
[burnupi28.front.sepia.ceph.com][INFO  ] ceph-mon: set fsid to 072e8e7b-d417-4c6a-b383-f5e0c8465bbd
[burnupi28.front.sepia.ceph.com][INFO  ] ceph-mon: created monfs at /var/lib/ceph/mon/ceph-burnupi28 for mon.burnupi28
[burnupi28.front.sepia.ceph.com][INFO  ] unlinking keyring file /var/lib/ceph/tmp/ceph-burnupi28.mon.keyring
[burnupi28.front.sepia.ceph.com][INFO  ] create a done file to avoid re-doing the mon deployment
[burnupi28.front.sepia.ceph.com][INFO  ] create the init path if it does not exist
[burnupi28.front.sepia.ceph.com][INFO  ] locating `service` executable...
[burnupi28.front.sepia.ceph.com][INFO  ] found `service` executable: /sbin/service
[burnupi28.front.sepia.ceph.com][INFO  ] Running command: /sbin/service ceph start mon.burnupi28
[burnupi28.front.sepia.ceph.com][ERROR ] Traceback (most recent call last):
[burnupi28.front.sepia.ceph.com][ERROR ]   File "/Users/alfredo/python/ceph-deploy/ceph_deploy/hosts/centos/mon/create.py", line 16, in create
[burnupi28.front.sepia.ceph.com][ERROR ]   File "/Users/alfredo/python/ceph-deploy/ceph_deploy/util/decorators.py", line 10, in inner
[burnupi28.front.sepia.ceph.com][ERROR ]   File "/Users/alfredo/python/ceph-deploy/ceph_deploy/util/wrappers.py", line 6, in remote_call
[burnupi28.front.sepia.ceph.com][ERROR ]   File "/usr/lib64/python2.6/subprocess.py", line 502, in check_call
[burnupi28.front.sepia.ceph.com][ERROR ]     raise CalledProcessError(retcode, cmd)
[burnupi28.front.sepia.ceph.com][ERROR ] CalledProcessError: Command '['/sbin/service', 'ceph', 'start', 'mon.burnupi28']' returned non-zero exit status 1
[burnupi28.front.sepia.ceph.com][INFO  ] === mon.burnupi28 ===
[burnupi28.front.sepia.ceph.com][INFO  ] Starting Ceph mon.burnupi28 on burnupi28...
[burnupi28.front.sepia.ceph.com][INFO  ] failed: 'ulimit -n 32768;  /usr/bin/ceph-mon -i burnupi28 --pid-file /var/run/ceph/mon.burnupi28.pid -c /etc/ceph/ceph.conf '
[burnupi28.front.sepia.ceph.com][INFO  ] Starting ceph-create-keys on burnupi28...

I opened a bug in Pushy: https://github.com/axw/pushy/issues/45

Still investigating and trying all kinds of things. So so frustrating :(

The only way I see around this (other than pushy fixing this problem) is to avoid capturing the stdout/stderr of the remote end as we are currently doing.

Until pushy decides to get a fix, I am going to implement a flag that will not capture anything so that if this problem comes up again we can just pass a
'no-capture' flag.

Very unfortunate because it was nice to be able to capture everything from the remote end.

I'll merge this pull request but I really want a ticket to stay open reminding us that this needs to be fixed and not just worked around.

Opened #5975; merged https://github.com/ceph/ceph-deploy/pull/44