Ceph » devops

You could be running an older version of systemd that doesn't support TasksMax=

I just built the Ceph cluster using ceph-deploy with stable jewel.

Problem is not only osd but mon as well. Maybe I filed a bug on this.

It's enabled.

[ceph@octopus conf]$ sudo systemctl is-enabled ceph-mon@octopus.service
enabled

But...

[ceph@octopus conf]$ sudo systemctl status ceph-mon@octopus.service
● ceph-mon@octopus.service - Ceph cluster monitor daemon
Loaded: loaded (/usr/lib/systemd/system/ceph-mon@.service; enabled; vendor preset: disabled)
Active: inactive (dead)

Once I started mon, it's running -;

[ceph@octopus conf]$ sudo systemctl status ceph-mon@octopus.service
● ceph-mon@octopus.service - Ceph cluster monitor daemon
Loaded: loaded (/usr/lib/systemd/system/ceph-mon@.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2016-04-25 18:37:32 EDT; 2s ago
Main PID: 3577 (ceph-mon)
CGroup: /system.slice/system-ceph\x2dmon.slice/ceph-mon@octopus.service
└─3577 /usr/bin/ceph-mon -f --cluster ceph --id octopus --setuser ceph --setgroup ceph

Apr 25 18:37:32 octopus.fullstack.go systemd¹: Started Ceph cluster monitor daemon.
Apr 25 18:37:32 octopus.fullstack.go systemd¹: Starting Ceph cluster monitor daemon...
Apr 25 18:37:32 octopus.fullstack.go ceph-mon³⁵⁷⁷: 2016-04-25 18:37:32.336537 7f591911d4c0 -1 WARNING: the following dangerous and experimental features are enabled: *
Apr 25 18:37:32 octopus.fullstack.go ceph-mon³⁵⁷⁷: 2016-04-25 18:37:32.337119 7f591911d4c0 -1 WARNING: the following dangerous and experimental features are enabled: *
Apr 25 18:37:32 octopus.fullstack.go ceph-mon³⁵⁷⁷: 2016-04-25 18:37:32.385628 7f591911d4c0 -1 WARNING: the following dangerous and experimental features are enabled: *
Apr 25 18:37:32 octopus.fullstack.go ceph-mon³⁵⁷⁷: starting mon.octopus rank 0 at 172.16.0.2:6789/0 mon_data /var/lib/ceph/mon/ceph-octopus fsid dbc219e2-cbd4-4538-8486-6dab65a57c18

Now I understood that it's because mon is NOT running that osd doen't run.

[ceph@octopus conf]$ sudo systemctl status ceph-osd@0.service
● ceph-osd@0.service - Ceph object storage daemon
Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2016-04-25 18:38:26 EDT; 25s ago
Main PID: 3852 (ceph-osd)
CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@0.service
└─3852 /usr/bin/ceph-osd -f --cluster ceph --id 0 --setuser ceph --setgroup ceph

Apr 25 18:38:26 octopus.fullstack.go ceph-osd³⁸⁵²: unsupported, and may result in data corruption, data loss,
Apr 25 18:38:26 octopus.fullstack.go ceph-osd³⁸⁵²: and/or irreparable damage to your cluster. Do not use
Apr 25 18:38:26 octopus.fullstack.go ceph-osd³⁸⁵²: feature with important data.
Apr 25 18:38:27 octopus.fullstack.go ceph-osd³⁸⁵²: 2016-04-25 18:38:27.385077 7fce90b25800 -1 WARNING: experimental feature 'rocksdb' is enabled
Apr 25 18:38:27 octopus.fullstack.go ceph-osd³⁸⁵²: Please be aware that this feature is experimental, untested,
Apr 25 18:38:27 octopus.fullstack.go ceph-osd³⁸⁵²: unsupported, and may result in data corruption, data loss,
Apr 25 18:38:27 octopus.fullstack.go ceph-osd³⁸⁵²: and/or irreparable damage to your cluster. Do not use
Apr 25 18:38:27 octopus.fullstack.go ceph-osd³⁸⁵²: feature with important data.
Apr 25 18:38:27 octopus.fullstack.go ceph-osd³⁸⁵²: 2016-04-25 18:38:27.537116 7fce90b25800 -1 osd.0 0 log_to_monitors {default=true}
Apr 25 18:38:32 octopus.fullstack.go systemd¹: [/usr/lib/systemd/system/ceph-osd@.service:18] Unknown lvalue 'TasksMax' in section 'Service'

Let's have a look.

[ceph@octopus conf]$ sudo reboot

O.k it's not running -;

[ceph@octopus ~]$ sudo systemctl status ceph-mon@octopus.service
● ceph-mon@octopus.service - Ceph cluster monitor daemon
Loaded: loaded (/usr/lib/systemd/system/ceph-mon@.service; enabled; vendor preset: disabled)
Active: inactive (dead)

Apr 25 18:45:28 octopus.fullstack.go systemd¹: [/usr/lib/systemd/system/ceph-mon@.service:24] Unknown lvalue 'TasksMax' in section 'Service'

osd is innocent...

[ceph@octopus ~]$ sudo systemctl status ceph-osd@0.service
● ceph-osd@0.service - Ceph object storage daemon
Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: disabled)
Active: inactive (dead)

Apr 25 18:45:21 octopus.fullstack.go systemd¹: [/usr/lib/systemd/system/ceph-osd@.service:18] Unknown lvalue 'TasksMax' in section 'Service'

$ lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: CentOS
Description: CentOS Linux release 7.2.1511 (Core)
Release: 7.2.1511
Codename: Core

$ uname -r
4.6.0-rc4+

The ceph-mon service not starting after reboot most likely means that ceph-mon.target is disabled. It's likely that ceph-osd.target is also disabled. I have a PR open to fix this.

Please check if these targets are disabled:

systemctl is-enabled ceph-mon.target
systemctl is-enabled ceph-osd.target

If, as I suspect, they are disabled, please enable them and try reboot again.

You're right. By default, it's disabled.

[ceph@octopus ~]$ sudo systemctl is-enabled ceph-mon.target
disabled

[ceph@octopus ~]$ sudo systemctl is-enabled ceph-osd.target
disabled

After enabled,

[ceph@octopus ~]$ sudo systemctl enable ceph-mon.target
Created symlink from /etc/systemd/system/multi-user.target.wants/ceph-mon.target to /usr/lib/systemd/system/ceph-mon.target.
Created symlink from /etc/systemd/system/ceph.target.wants/ceph-mon.target to /usr/lib/systemd/system/ceph-mon.target.

[ceph@octopus ~]$ sudo systemctl enable ceph-osd.target
Created symlink from /etc/systemd/system/multi-user.target.wants/ceph-osd.target to /usr/lib/systemd/system/ceph-osd.target.
Created symlink from /etc/systemd/system/ceph.target.wants/ceph-osd.target to /usr/lib/systemd/system/ceph-osd.target.

[ceph@octopus ~]$ sudo systemctl status ceph-mon@octopus.service
● ceph-mon@octopus.service - Ceph cluster monitor daemon
Loaded: loaded (/usr/lib/systemd/system/ceph-mon@.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2016-04-26 18:16:45 EDT; 9min ago
Main PID: 1703 (ceph-mon)
CGroup: /system.slice/system-ceph\x2dmon.slice/ceph-mon@octopus.service
└─1703 /usr/bin/ceph-mon -f --cluster ceph --id octopus --setuser ceph --setgroup ceph

Apr 26 18:16:45 octopus.fullstack.go systemd¹: Started Ceph cluster monitor daemon.
Apr 26 18:16:45 octopus.fullstack.go systemd¹: Starting Ceph cluster monitor daemon...
Apr 26 18:16:46 octopus.fullstack.go ceph-mon¹⁷⁰³: 2016-04-26 18:16:46.374772 7efc5ee7e4c0 -1 WARNING: the following dangerous and experimental features are enabled: *
Apr 26 18:16:46 octopus.fullstack.go ceph-mon¹⁷⁰³: 2016-04-26 18:16:46.382433 7efc5ee7e4c0 -1 WARNING: the following dangerous and experimental features are enabled: *
Apr 26 18:16:46 octopus.fullstack.go ceph-mon¹⁷⁰³: 2016-04-26 18:16:46.488410 7efc5ee7e4c0 -1 WARNING: the following dangerous and experimental features are enabled: *
Apr 26 18:16:46 octopus.fullstack.go ceph-mon¹⁷⁰³: starting mon.octopus rank 0 at 172.16.0.2:6789/0 mon_data /var/lib/ceph/mon/ceph-octopus fsid dbc219e2-c...b65a57c18
Apr 26 18:16:51 octopus.fullstack.go systemd¹: [/usr/lib/systemd/system/ceph-mon@.service:24] Unknown lvalue 'TasksMax' in section 'Service'
Hint: Some lines were ellipsized, use -l to show in full.

[ceph@octopus ~]$ sudo systemctl status ceph-osd@0.service
● ceph-osd@0.service - Ceph object storage daemon
Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2016-04-26 18:16:47 EDT; 9min ago
Main PID: 2941 (ceph-osd)
CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@0.service
└─2941 /usr/bin/ceph-osd -f --cluster ceph --id 0 --setuser ceph --setgroup ceph

Apr 26 18:16:47 octopus.fullstack.go ceph-osd²⁹⁴¹: unsupported, and may result in data corruption, data loss,
Apr 26 18:16:47 octopus.fullstack.go ceph-osd²⁹⁴¹: and/or irreparable damage to your cluster. Do not use
Apr 26 18:16:47 octopus.fullstack.go ceph-osd²⁹⁴¹: feature with important data.
Apr 26 18:16:48 octopus.fullstack.go ceph-osd²⁹⁴¹: 2016-04-26 18:16:48.186222 7f01135b4800 -1 WARNING: experimental feature 'rocksdb' is enabled
Apr 26 18:16:48 octopus.fullstack.go ceph-osd²⁹⁴¹: Please be aware that this feature is experimental, untested,
Apr 26 18:16:48 octopus.fullstack.go ceph-osd²⁹⁴¹: unsupported, and may result in data corruption, data loss,
Apr 26 18:16:48 octopus.fullstack.go ceph-osd²⁹⁴¹: and/or irreparable damage to your cluster. Do not use
Apr 26 18:16:48 octopus.fullstack.go ceph-osd²⁹⁴¹: feature with important data.
Apr 26 18:16:48 octopus.fullstack.go ceph-osd²⁹⁴¹: 2016-04-26 18:16:48.466579 7f01135b4800 -1 osd.0 27 log_to_monitors {default=true}
Apr 26 18:16:51 octopus.fullstack.go systemd¹: [/usr/lib/systemd/system/ceph-osd@.service:18] Unknown lvalue 'TasksMax' in section 'Service'

So now we need these kind of dependencies since Jewel.

ceph.target
  -> ceph-mon.target
    -> ceph-mon@.service

Why do we need ceph-{mon|osd}.target?