Project

General

Profile

Bug #7117

centOS: init script silently dies due to missing '/usr/lib64/ceph/ceph_common.sh'

Added by Alfredo Deza over 9 years ago. Updated over 9 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Ken Dreyer
Category:
Monitor
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

One of the failing tests: http://qa-proxy.ceph.com/teuthology/teuthology-2014-01-05_01:10:06-ceph-deploy-master-testing-basic-vps/27104/teuthology.log

2014-01-07T08:50:35.516 INFO:teuthology.orchestra.run.err:[10.214.138.93]: Warning: Permanently added 'vpm034.front.sepia.ceph.com,10.214.138.93' (RSA) to the list of known hosts.
2014-01-07T08:50:35.645 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [vpm034.front.sepia.ceph.com][DEBUG ] connected to host: vpm034.front.sepia.ceph.com
2014-01-07T08:50:35.645 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [vpm034.front.sepia.ceph.com][DEBUG ] detect platform information from remote host
2014-01-07T08:50:35.666 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [vpm034.front.sepia.ceph.com][DEBUG ] detect machine type
2014-01-07T08:50:35.670 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [vpm034.front.sepia.ceph.com][DEBUG ] fetch remote file
2014-01-07T08:50:35.672 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [ceph_deploy.gatherkeys][WARNIN] Unable to find /var/lib/ceph/bootstrap-osd/ceph.keyring on ['vpm034.front.sepia.ceph.com']
2014-01-07T08:50:35.673 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [ceph_deploy.gatherkeys][DEBUG ] Checking vpm034.front.sepia.ceph.com for /var/lib/ceph/bootstrap-mds/ceph.keyring
2014-01-07T08:50:35.706 INFO:teuthology.orchestra.run.err:[10.214.138.93]: Warning: Permanently added 'vpm034.front.sepia.ceph.com,10.214.138.93' (RSA) to the list of known hosts.
2014-01-07T08:50:35.829 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [vpm034.front.sepia.ceph.com][DEBUG ] connected to host: vpm034.front.sepia.ceph.com
2014-01-07T08:50:35.829 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [vpm034.front.sepia.ceph.com][DEBUG ] detect platform information from remote host
2014-01-07T08:50:35.850 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [vpm034.front.sepia.ceph.com][DEBUG ] detect machine type
2014-01-07T08:50:35.854 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [vpm034.front.sepia.ceph.com][DEBUG ] fetch remote file
2014-01-07T08:50:35.857 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [ceph_deploy.gatherkeys][WARNIN] Unable to find /var/lib/ceph/bootstrap-mds/ceph.keyring on ['vpm034.front.sepia.ceph.com']
2014-01-07T08:50:45.882 DEBUG:teuthology.orchestra.run:Running [10.214.138.93]: 'cd /home/ubuntu/cephtest/ceph-deploy && ./ceph-deploy gatherkeys vpm034.front.sepia.ceph.com'
2014-01-07T08:50:46.044 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [ceph_deploy.cli][INFO  ] Invoked (1.3.4): ./ceph-deploy gatherkeys vpm034.front.sepia.ceph.com
2014-01-07T08:50:46.044 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [ceph_deploy.gatherkeys][DEBUG ] Checking vpm034.front.sepia.ceph.com for /etc/ceph/ceph.client.admin.keyring
2014-01-07T08:50:46.100 INFO:teuthology.orchestra.run.err:[10.214.138.93]: Warning: Permanently added 'vpm034.front.sepia.ceph.com,10.214.138.93' (RSA) to the list of known hosts.
2014-01-07T08:50:46.217 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [vpm034.front.sepia.ceph.com][DEBUG ] connected to host: vpm034.front.sepia.ceph.com
2014-01-07T08:50:46.218 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [vpm034.front.sepia.ceph.com][DEBUG ] detect platform information from remote host
2014-01-07T08:50:46.237 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [vpm034.front.sepia.ceph.com][DEBUG ] detect machine type
2014-01-07T08:50:46.241 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [vpm034.front.sepia.ceph.com][DEBUG ] fetch remote file
2014-01-07T08:50:46.243 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [ceph_deploy.gatherkeys][WARNIN] Unable to find /etc/ceph/ceph.client.admin.keyring on ['vpm034.front.sepia.ceph.com']
2014-01-07T08:50:46.244 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring
2014-01-07T08:50:46.244 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [ceph_deploy.gatherkeys][DEBUG ] Checking vpm034.front.sepia.ceph.com for /var/lib/ceph/bootstrap-osd/ceph.keyring
2014-01-07T08:50:46.277 INFO:teuthology.orchestra.run.err:[10.214.138.93]: Warning: Permanently added 'vpm034.front.sepia.ceph.com,10.214.138.93' (RSA) to the list of known hosts.
2014-01-07T08:50:46.395 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [vpm034.front.sepia.ceph.com][DEBUG ] connected to host: vpm034.front.sepia.ceph.com
2014-01-07T08:50:46.395 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [vpm034.front.sepia.ceph.com][DEBUG ] detect platform information from remote host
2014-01-07T08:50:46.414 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [vpm034.front.sepia.ceph.com][DEBUG ] detect machine type
2014-01-07T08:50:46.418 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [vpm034.front.sepia.ceph.com][DEBUG ] fetch remote file
2014-01-07T08:50:46.420 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [ceph_deploy.gatherkeys][WARNIN] Unable to find /var/lib/ceph/bootstrap-osd/ceph.keyring on ['vpm034.front.sepia.ceph.com']
2014-01-07T08:50:46.421 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [ceph_deploy.gatherkeys][DEBUG ] Checking vpm034.front.sepia.ceph.com for /var/lib/ceph/bootstrap-mds/ceph.keyring
2014-01-07T08:50:46.455 INFO:teuthology.orchestra.run.err:[10.214.138.93]: Warning: Permanently added 'vpm034.front.sepia.ceph.com,10.214.138.93' (RSA) to the list of known hosts.
2014-01-07T08:50:46.574 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [vpm034.front.sepia.ceph.com][DEBUG ] connected to host: vpm034.front.sepia.ceph.com
2014-01-07T08:50:46.575 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [vpm034.front.sepia.ceph.com][DEBUG ] detect platform information from remote host
2014-01-07T08:50:46.594 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [vpm034.front.sepia.ceph.com][DEBUG ] detect machine type
2014-01-07T08:50:46.598 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [vpm034.front.sepia.ceph.com][DEBUG ] fetch remote file
2014-01-07T08:50:46.600 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [ceph_deploy.gatherkeys][WARNIN] Unable to find /var/lib/ceph/bootstrap-mds/ceph.keyring on ['vpm034.front.sepia.ceph.com']

Associated revisions

Revision 2d0d48b8 (diff)
Added by Ken Dreyer over 9 years ago

packaging: ship libdir/ceph

Automake puts ceph_common.sh into libdir/ceph, but the Red Hat packaging
was not capturing this file.

Add the libdir/ceph location to the RPM packaging.

Fixes #7117

Revision 4c968534 (diff)
Added by Ken Dreyer over 9 years ago

packaging: ship libdir/ceph

Automake puts ceph_common.sh into libdir/ceph, but the Red Hat packaging
was not capturing this file.

Add the libdir/ceph location to the RPM packaging.

Fixes #7117

(cherry picked from commit 2d0d48b829bd5721b7058ec43f61481fe8542b12)

History

#1 Updated by Alfredo Deza over 9 years ago

I replicated this problem locally using the most current `master` branch. The issue is not really `gatherkeys` but the fact that the monitors don't even start on CentOS hosts.

#2 Updated by Ian Colle over 9 years ago

  • Project changed from teuthology to Ceph
  • Subject changed from ceph-deploy fails to gatherkeys to monitors fail to start on CentOS hosts
  • Category set to Monitor
  • Assignee changed from Alfredo Deza to Joao Eduardo Luis

#3 Updated by Joao Eduardo Luis over 9 years ago

  • Subject changed from monitors fail to start on CentOS hosts to centOS: init script silently dies due to missing '/usr/lib64/ceph/ceph_common.sh'

AFAICT, the reason is that the init-script is not even starting any daemon due to failing on

# behave if we are not completely installed (e.g., Debian "removed,
# config remains" state)
test -f $LIBDIR/ceph_common.sh || exit 0

considering that LIBDIR = /usr/lib64/ceph

[ubuntu@vpm025 ~]$ ls /usr/lib64/ceph/
erasure-code
[ubuntu@vpm025 ~]$

So, no ceph_common.sh, no init script. No init script, no daemons come up using 'service'.
Also, the init script is silently exiting in this situation.

The mons work just fine when ran manually though.

#4 Updated by Ian Colle over 9 years ago

  • Assignee changed from Joao Eduardo Luis to Ken Dreyer

Something with the make file?

#5 Updated by Ken Dreyer over 9 years ago

I noticed the following warning today while building v0.75.

http://jenkins.ceph.com/job/ceph-build/Arch=x86_64,Distro=fedora19/100/console

Checking for unpackaged file(s): /usr/lib/rpm/check-files /home/jenkins-build/build/workspace/ceph-build/Arch/x86_64/Distro/fedora19/release/0.75/rpm/fc19/BUILDROOT/ceph-0.75-0.fc19.x86_64

  warning: Installed (but unpackaged) file(s) found:
    /usr/bin/ceph_test_c_headers
    /usr/bin/ceph_test_get_blkdev_size
    /usr/lib64/ceph/ceph_common.sh

So it looks like automake is installing ceph_common.sh appropriately, but the RPMs are not packaging the /usr/lib64/ceph directory in %files.

#6 Updated by Ken Dreyer over 9 years ago

I've pushed an experimental patch to https://github.com/ceph/ceph/tree/wip-7117-redhat

I need to test and see what happens in gitbuilder.

#7 Updated by Alfredo Deza over 9 years ago

  • Priority changed from High to Urgent

This is now affecting every test that runs on a system with that init script.

For example, all OSDs deployed in this test come back unhealthy because the mons never start:

http://qa-proxy.ceph.com/teuthology/teuthology-2014-01-12_01:10:02-ceph-deploy-master-testing-basic-vps/35701/teuthology.log

#8 Updated by Ken Dreyer over 9 years ago

  • Status changed from 12 to Fix Under Review

I've verified that with this patch, the RPM packaging properly captures the ceph_common.sh file in /usr/lib64/ceph/.

https://github.com/ceph/ceph/pull/1094

#9 Updated by Sage Weil over 9 years ago

  • Status changed from Fix Under Review to Resolved

Also available in: Atom PDF