Ceph

I had a node deployed using ceph-deploy. 7 disks in total, the journals are
on files on an SSD.

After rebooting the node, only some of them would come back.

-3 2.61 host signina
5 0.45 osd.5 up 1
6 0.45 osd.6 up 1
7 0.27 osd.7 down 0
8 0.27 osd.8 down 0
9 0.27 osd.9 down 0
10 0.45 osd.10 up 1
11 0.45 osd.11 down 0

In dmesg I see the following:

[ 42.072796] init: ceph-osd (ceph/8) main process (4118) terminated with status 134
[ 42.072824] init: ceph-osd (ceph/8) main process ended, respawning
[ 42.982971] init: ceph-osd (ceph/9) main process (4117) killed by ABRT signal
[ 42.983009] init: ceph-osd (ceph/9) main process ended, respawning
[ 42.987865] init: ceph-osd (ceph/7) main process (4124) terminated with status 134
[ 42.987892] init: ceph-osd (ceph/7) main process ended, respawning
[ 43.960877] init: ceph-osd (ceph/8) main process (8146) terminated with status 134
[ 43.960906] init: ceph-osd (ceph/8) main process ended, respawning
[ 44.963873] init: ceph-osd (ceph/9) main process (8355) terminated with status 134
[ 44.963901] init: ceph-osd (ceph/9) main process ended, respawning
[ 44.967048] init: ceph-osd (ceph/7) main process (8360) terminated with status 134
[ 44.967089] init: ceph-osd (ceph/7) main process ended, respawning

and this keeps repeating for 7,8,9,11.

They are, however, mounted:

/dev/sde1 on /var/lib/ceph/osd/ceph-9 type xfs (rw)
/dev/sdd1 on /var/lib/ceph/osd/ceph-8 type xfs (rw)
/dev/sdb1 on /var/lib/ceph/osd/ceph-6 type xfs (rw)
/dev/sdf1 on /var/lib/ceph/osd/ceph-10 type xfs (rw)
/dev/sdc1 on /var/lib/ceph/osd/ceph-7 type xfs (rw)
/dev/sdg1 on /var/lib/ceph/osd/ceph-11 type xfs (rw)
/dev/sda1 on /var/lib/ceph/osd/ceph-5 type xfs (rw)

Attaching the logs for these.

The journals are available at the specified location:

zoltan@signina:~$ ls las /osdjournal/
total 30787668
4 drwxr-xr-x 3 root root 4096 Aug 13 14:51 .
4 drwxr-xr-x 24 root root 4096 Aug 13 14:34 ..
16 drwx----- 2 root root 16384 Aug 13 14:34 lost+found
5242888 rw-r--r- 1 root root 5368709120 Aug 15 20:56 sda
5242888 rw-r--r- 1 root root 5368709120 Aug 15 20:56 sdb
3409264 rw-r--r- 1 root root 5368709120 Aug 15 20:50 sdc
3585292 rw-r--r- 1 root root 5368709120 Aug 15 20:50 sdd
3413500 rw-r--r- 1 root root 5368709120 Aug 15 20:50 sde
5242888 rw-r--r- 1 root root 5368709120 Aug 15 20:56 sdf
4650924 rw-r--r- 1 root root 5368709120 Aug 15 20:51 sdg
zoltan@signina:~$