Ceph » devops

I'm 90% sure this is systemd killing everything in the cgroup when ceph-deploy's ssh session closes. The log file just ends (kill -9?) right when the ceph-deploy command finishes. In the cases where we start several OSDs at once, they all stay running until it finishes and then all die at once, consistent with the ssh close.

The f19 config options don't explicitly enable KillUserProcesses or kill-session-processes (at least that i can see), but i can't figure out how to view what the default is.

I tried to replicate this with ceph-deploy and boy was it a nightmare to get there.

FC19 panics a few seconds after an OSD is deployed. I was able to replicate this behavior reliably by blowing away everything and reinstalling from scratch.

So after several attempts to get FC19 going and failing, Sage suggested I should try FC20, but alas, we do not have any FC20 packages. I attempted to re-use the
FC19 packages though and was not able to get the dependencies resolved:

ceph-deploy install --release fedora20 node{7,8}
[ceph_deploy.cli][INFO  ] Invoked (1.3.5): /Users/alfredo/.virtualenvs/ceph-deploy/bin/ceph-deploy install --release fedora20 node7 node8
[ceph_deploy.install][DEBUG ] Installing stable version fedora20 on cluster ceph hosts node7 node8
[ceph_deploy.install][DEBUG ] Detecting platform for host node7 ...
[node7][DEBUG ] connected to host: node7
[node7][DEBUG ] detect platform information from remote host
[node7][DEBUG ] detect machine type
[ceph_deploy.install][INFO  ] Distro info: Fedora 20 Heisenbug
[node7][INFO  ] installing ceph on node7
[ceph_deploy.install][INFO  ] detected valid custom repositories from config file
[ceph_deploy.install][INFO  ] will use repository from conf: fedora20
[node7][INFO  ] adding custom repository file
[node7][INFO  ] Running command: sudo rpm --import https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc
[node7][DEBUG ] set the contents of repo file to /etc/yum.repos.d/
[node7][INFO  ] Running command: sudo yum -y -q install wget
` [node7][DEBUG ] Package wget-1.14-12.fc20.x86_64 already installed and latest version
[node7][INFO  ] Running command: sudo yum -y -q install ceph
[node7][WARNIN] Error: Package: libcephfs1-0.72.2-0.fc19.x86_64 (fedora20)
[node7][WARNIN]            Requires: libboost_thread-mt.so.1.53.0()(64bit)
[node7][WARNIN] Error: Package: librbd1-0.72.2-0.fc19.x86_64 (fedora20)
[node7][WARNIN]            Requires: libboost_system-mt.so.1.53.0()(64bit)
[node7][WARNIN] Error: Package: ceph-0.72.2-0.fc19.x86_64 (fedora20)
[node7][WARNIN]            Requires: libboost_system-mt.so.1.53.0()(64bit)
[node7][WARNIN] Error: Package: libcephfs1-0.72.2-0.fc19.x86_64 (fedora20)
[node7][WARNIN]            Requires: libboost_system-mt.so.1.53.0()(64bit)
[node7][WARNIN] Error: Package: librbd1-0.72.2-0.fc19.x86_64 (fedora20)
[node7][WARNIN]            Requires: libboost_thread-mt.so.1.53.0()(64bit)
[node7][WARNIN] Error: Package: ceph-0.72.2-0.fc19.x86_64 (fedora20)
[node7][WARNIN]            Requires: libboost_thread-mt.so.1.53.0()(64bit)
[node7][WARNIN] Error: Package: librados2-0.72.2-0.fc19.x86_64 (fedora20)
[node7][WARNIN]            Requires: libboost_thread-mt.so.1.53.0()(64bit)
[node7][WARNIN] Error: Package: librados2-0.72.2-0.fc19.x86_64 (fedora20)
[node7][WARNIN]            Requires: libboost_system-mt.so.1.53.0()(64bit)
[node7][DEBUG ]  You could try using --skip-broken to work around the problem
[node7][DEBUG ]  You could try running: rpm -Va --nofiles --nodigest
[node7][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: yum -y -q install ceph

About the systemd not playing well with ceph-deploy, I am still doubtful because the way ceph-deploy is working with the host is constantly connecting and disconnecting.

It does not stay connected doing a bunch of commands (say monitors and then OSDs). So why would systemd not kill the process for monitors but would totally do that for
OSDs?

I am thinking that this has to do with the fact that I could not get OSDs up and running except with a kernel panic in combination with systemd killing the process.

Opened https://bugzilla.redhat.com/show_bug.cgi?id=1095988

Fedora 19 does not have (nor it provides) systemd-run:

$ systemd --version
systemd 204
+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ

$ sudo yum provides systemd-run
No matches found

According to the Fedora docs it looks like it is there for FC 20: http://linuxmanpages.net/manpages/fedora20/man1/systemd-run.1.html

Alfredo Deza wrote:

Fedora 19 does not have (nor it provides) systemd-run:

[...]

According to the Fedora docs it looks like it is there for FC 20: http://linuxmanpages.net/manpages/fedora20/man1/systemd-run.1.html

I think we should ignore fc19 entirely then and focus on making sure we are doing the right thing with 20+. Hopefully rhel7 has systemd-run?

...and we don't have packages for F20

Going to try and build for F20, but it might take some effort to get there as we depend on setting up hosts for this (as opposed to ephemeral ones). See issue #8325
for a reference on to why this is problematic for now.

I am having a lot of trouble trying to relate the ssh connection with how we start the daemon and the subsequent failure.

We are able to start the monitor correctly without the 'systemd-run' call using just 'service', and that stays up even
after ceph-deploy closes the connection:

ceph-deploy mon create-initial
[ceph_deploy.conf][DEBUG ] found configuration file at: /Users/alfredo/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.4): /Users/alfredo/.virtualenvs/ceph-deploy/bin/ceph-deploy mon create-initial
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts node8
[ceph_deploy.mon][DEBUG ] detecting platform for host node8 ...
[node8][DEBUG ] connected to host: node8
[node8][DEBUG ] detect platform information from remote host
[node8][DEBUG ] detect machine type
[ceph_deploy.mon][INFO  ] distro info: Fedora 20 Heisenbug
[node8][DEBUG ] determining if provided host has same hostname in remote
[node8][DEBUG ] get remote short hostname
[node8][DEBUG ] deploying mon to node8
[node8][DEBUG ] get remote short hostname
[node8][DEBUG ] remote hostname: node8
[node8][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node8][DEBUG ] create the mon path if it does not exist
[node8][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-node8/done
[node8][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-node8/done
[node8][INFO  ] creating keyring file: /var/lib/ceph/tmp/ceph-node8.mon.keyring
[node8][DEBUG ] create the monitor keyring file
[node8][INFO  ] Running command: sudo ceph-mon --cluster ceph --mkfs -i node8 --keyring /var/lib/ceph/tmp/ceph-node8.mon.keyring
[node8][DEBUG ] ceph-mon: mon.noname-a 192.168.111.107:6789/0 is local, renaming to mon.node8
[node8][DEBUG ] ceph-mon: set fsid to ba2b6622-a1bb-490b-9de9-47f263321591
[node8][DEBUG ] ceph-mon: created monfs at /var/lib/ceph/mon/ceph-node8 for mon.node8
[node8][INFO  ] unlinking keyring file /var/lib/ceph/tmp/ceph-node8.mon.keyring
[node8][DEBUG ] create a done file to avoid re-doing the mon deployment
[node8][DEBUG ] create the init path if it does not exist
[node8][DEBUG ] locating the `service` executable...
[node8][INFO  ] Running command: sudo /usr/sbin/service ceph -c /etc/ceph/ceph.conf start mon.node8
[node8][DEBUG ] === mon.node8 ===
[node8][DEBUG ] Starting Ceph mon.node8 on node8...
[node8][DEBUG ] Starting ceph-create-keys on node8...
[node8][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node8.asok mon_status
[node8][DEBUG ] ********************************************************************************
[node8][DEBUG ] status for monitor: mon.node8
[node8][DEBUG ] {
[node8][DEBUG ]   "election_epoch": 2,
[node8][DEBUG ]   "extra_probe_peers": [],
[node8][DEBUG ]   "monmap": {
[node8][DEBUG ]     "created": "0.000000",
[node8][DEBUG ]     "epoch": 1,
[node8][DEBUG ]     "fsid": "ba2b6622-a1bb-490b-9de9-47f263321591",
[node8][DEBUG ]     "modified": "0.000000",
[node8][DEBUG ]     "mons": [
[node8][DEBUG ]       {
[node8][DEBUG ]         "addr": "192.168.111.107:6789/0",
[node8][DEBUG ]         "name": "node8",
[node8][DEBUG ]         "rank": 0
[node8][DEBUG ]       }
[node8][DEBUG ]     ]
[node8][DEBUG ]   },
[node8][DEBUG ]   "name": "node8",
[node8][DEBUG ]   "outside_quorum": [],
[node8][DEBUG ]   "quorum": [
[node8][DEBUG ]     0
[node8][DEBUG ]   ],
[node8][DEBUG ]   "rank": 0,
[node8][DEBUG ]   "state": "leader",
[node8][DEBUG ]   "sync_provider": []
[node8][DEBUG ] }
[node8][DEBUG ] ********************************************************************************
[node8][INFO  ] monitor: mon.node8 is running
[node8][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node8.asok mon_status
[ceph_deploy.mon][INFO  ] processing monitor mon.node8
[node8][DEBUG ] connected to host: node8
[node8][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node8.asok mon_status
[ceph_deploy.mon][INFO  ] mon.node8 monitor has reached quorum!
[ceph_deploy.mon][INFO  ] all initial monitors are running and have formed quorum
[ceph_deploy.mon][INFO  ] Running gatherkeys...
[ceph_deploy.gatherkeys][DEBUG ] Have ceph.client.admin.keyring
[ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring
[ceph_deploy.gatherkeys][DEBUG ] Have ceph.bootstrap-osd.keyring
[ceph_deploy.gatherkeys][DEBUG ] Have ceph.bootstrap-mds.keyring

After that is done I can verify that ceph-mon is still up:

$ ssh node8 ps aux | grep ceph-mon
root     13984  0.1  1.8 133824  9068 ?        Sl   20:49   0:00 /usr/bin/ceph-mon -i node8 --pid-file /var/run/ceph/mon.node8.pid -c /etc/ceph/ceph.conf --cluster ceph

I think that one possibility is that ceph-disk is launching things incorrectly (for Fedora) but I cannot find where that is.

I did add a few 'systemd-run' calls to see if that would take care of it, but that didn't seem to help:

ceph-deploy osd --zap create node8:/dev/sdb
[ceph_deploy.conf][DEBUG ] found configuration file at: /Users/alfredo/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.4): /Users/alfredo/.virtualenvs/ceph-deploy/bin/ceph-deploy osd --zap create node8:/dev/sdb
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks node8:/dev/sdb:
[node8][DEBUG ] connected to host: node8
[node8][DEBUG ] detect platform information from remote host
[node8][DEBUG ] detect machine type
[ceph_deploy.osd][INFO  ] Distro info: Fedora 20 Heisenbug
[ceph_deploy.osd][DEBUG ] Deploying osd to node8
[node8][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node8][INFO  ] Running command: sudo udevadm trigger --subsystem-match=block --action=add
[ceph_deploy.osd][DEBUG ] Preparing host node8 disk /dev/sdb journal None activate True
[node8][INFO  ] Running command: sudo systemd-run ceph-disk-prepare --zap-disk --fs-type xfs --cluster ceph -- /dev/sdb
[node8][WARNIN] Running as unit run-14206.service.
[node8][INFO  ] Running command: sudo udevadm trigger --subsystem-match=block --action=add
[node8][INFO  ] checking OSD status...
[node8][INFO  ] Running command: sudo ceph --cluster=ceph osd stat --format=json
[ceph_deploy.osd][DEBUG ] Host node8 is now ready for osd use.

The 'host is now ready for osd use' is a lie, here is the 'osd stat' result:

{u'epoch': 1,
 u'full': False,
 u'nearfull': False,
 u'num_in_osds': 0,
 u'num_osds': 0,
 u'num_up_osds': 0}

Not sure what else I can try here