Project

General

Profile

Actions
Copy link

Bug #13781

closed

Random Ceph-osd processes doesn't start upon system restart (rhel 7.1)

Added by Vimal A.R over 8 years ago. Updated about 7 years ago.

Status:
Resolved
Priority:
High
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
osd
Backport:
Regression:
Severity:
Reviewed:
Affected Versions:
v0.94.3
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Random ceph-osd processes doesn't get started up when the OSD node restarts. This can happen in any of the OSD nodes, and for any random OSD.

The OSD however is mounted at the mount point under /var/lib/ceph/osd/. Manually starting the OSD process after the machine is up and running, works fine.

/var/log/messages have logs on other OSD processes starting, but not anything on the ones that are not. No crashes, or anything that indicates a problem.

This is Hammer on RHEL7.1.

~~~
$ lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: RedHatEnterpriseServer
Description: Red Hat Enterprise Linux Server release 7.1 (Maipo)
Release: 7.1
Codename: Maipo

$ uname -a
Linux <hostname> 3.10.0-229.14.1.el7.x86_64 #1 SMP Tue Aug 25 11:21:22 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

$ rpm -qa | grep ceph
ceph-0.94.3-0.el7.x86_64
libcephfs1-0.94.3-0.el7.x86_64
python-cephfs-0.94.3-0.el7.x86_64
ceph-common-0.94.3-0.el7.x86_64
~~~

Attaching /var/log/messages, as an example since the problem was reproducible on this OSD for the latest reboot.

Files

Download all files
messages (574 KB) messages /var/log/messages Vimal A.R, 11/12/2015 11:49 AM
ceph-osd.10.log (1010 KB) ceph-osd.10.log Vimal A.R, 11/12/2015 11:51 AM
Actions
Copy link
 #1

Updated by Nathan Cutler over 8 years ago

  • Tracker changed from Tasks to Bug
  • Project changed from Stable releases to Ceph
  • Tags changed from osd, ceph to osd
Actions
Copy link
 #2

Updated by Loïc Dachary over 8 years ago

Is the corresponding systemd service enabled ? I kind of remember a similar problem where all was good, only the service was not enabled and after enabling it all was better at boot time.

Actions
Copy link
 #3

Updated by Nathan Cutler over 8 years ago

Loic, see description - this is Hammer (no systemd).

Actions
Copy link
 #4

Updated by Loïc Dachary over 8 years ago

  • Subject changed from Random Ceph-osd processes doesn't start upon system restart to Random Ceph-osd processes doesn't start upon system restart (rhel 7.1)
Actions
Copy link
 #5

Updated by Sage Weil over 8 years ago

messages shows no attempt to start osd.10. :/

Actions
Copy link
 #6

Updated by Loïc Dachary about 8 years ago

  • Status changed from New to Need More Info

@Nathan Weinberg : rhel7.1 has hammer and systemd

@Vimal could you please confirm that the service was enabled ? If so it would be great to get as many details as possible about how you installed the machine.

Actions
Copy link
 #7

Updated by Nathan Cutler about 8 years ago

@Loïc Dachary: What I was trying to say was that ceph.spec.in in hammer does not support systemd.

Actions
Copy link
 #8

Updated by Vimal A.R about 8 years ago

Loic,

Yes, the service is enabled. This comes up for random OSDs, when the machine is rebooted after kernel updates etc.. Do you have any specific information you're interested in, related to the installation? Please let me know.

Vimal

Actions
Copy link
 #9

Updated by Kefu Chai about 8 years ago

it sounds like a dup of #13422, but it's not. because as sage pointed out

messages shows no attempt to start osd.10.

Actions
Copy link
 #10

Updated by Sage Weil about 7 years ago

  • Status changed from Need More Info to Resolved

i believe we've resolved these issues with changes to udev, ceph-disk, and systemd interaction

Actions
Copy link

Also available in: Atom PDF