Project

General

Profile

Actions
Copy link

Bug #7117

closed

centOS: init script silently dies due to missing '/usr/lib64/ceph/ceph_common.sh'

Added by Alfredo Deza over 10 years ago. Updated about 10 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Ken Dreyer
Category:
Monitor
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

One of the failing tests: http://qa-proxy.ceph.com/teuthology/teuthology-2014-01-05_01:10:06-ceph-deploy-master-testing-basic-vps/27104/teuthology.log

2014-01-07T08:50:35.516 INFO:teuthology.orchestra.run.err:[10.214.138.93]: Warning: Permanently added 'vpm034.front.sepia.ceph.com,10.214.138.93' (RSA) to the list of known hosts.
2014-01-07T08:50:35.645 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mvpm034.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] connected to host: vpm034.front.sepia.ceph.com
2014-01-07T08:50:35.645 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mvpm034.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] detect platform information from remote host
2014-01-07T08:50:35.666 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mvpm034.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] detect machine type
2014-01-07T08:50:35.670 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mvpm034.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] fetch remote file
2014-01-07T08:50:35.672 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mceph_deploy.gatherkeys[0m][[1;33mWARNIN[0m] Unable to find /var/lib/ceph/bootstrap-osd/ceph.keyring on ['vpm034.front.sepia.ceph.com']
2014-01-07T08:50:35.673 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mceph_deploy.gatherkeys[0m][[1;34mDEBUG[0m ] Checking vpm034.front.sepia.ceph.com for /var/lib/ceph/bootstrap-mds/ceph.keyring
2014-01-07T08:50:35.706 INFO:teuthology.orchestra.run.err:[10.214.138.93]: Warning: Permanently added 'vpm034.front.sepia.ceph.com,10.214.138.93' (RSA) to the list of known hosts.
2014-01-07T08:50:35.829 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mvpm034.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] connected to host: vpm034.front.sepia.ceph.com
2014-01-07T08:50:35.829 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mvpm034.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] detect platform information from remote host
2014-01-07T08:50:35.850 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mvpm034.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] detect machine type
2014-01-07T08:50:35.854 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mvpm034.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] fetch remote file
2014-01-07T08:50:35.857 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mceph_deploy.gatherkeys[0m][[1;33mWARNIN[0m] Unable to find /var/lib/ceph/bootstrap-mds/ceph.keyring on ['vpm034.front.sepia.ceph.com']
2014-01-07T08:50:45.882 DEBUG:teuthology.orchestra.run:Running [10.214.138.93]: 'cd /home/ubuntu/cephtest/ceph-deploy && ./ceph-deploy gatherkeys vpm034.front.sepia.ceph.com'
2014-01-07T08:50:46.044 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mceph_deploy.cli[0m][[1;37mINFO[0m  ] Invoked (1.3.4): ./ceph-deploy gatherkeys vpm034.front.sepia.ceph.com
2014-01-07T08:50:46.044 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mceph_deploy.gatherkeys[0m][[1;34mDEBUG[0m ] Checking vpm034.front.sepia.ceph.com for /etc/ceph/ceph.client.admin.keyring
2014-01-07T08:50:46.100 INFO:teuthology.orchestra.run.err:[10.214.138.93]: Warning: Permanently added 'vpm034.front.sepia.ceph.com,10.214.138.93' (RSA) to the list of known hosts.
2014-01-07T08:50:46.217 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mvpm034.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] connected to host: vpm034.front.sepia.ceph.com
2014-01-07T08:50:46.218 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mvpm034.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] detect platform information from remote host
2014-01-07T08:50:46.237 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mvpm034.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] detect machine type
2014-01-07T08:50:46.241 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mvpm034.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] fetch remote file
2014-01-07T08:50:46.243 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mceph_deploy.gatherkeys[0m][[1;33mWARNIN[0m] Unable to find /etc/ceph/ceph.client.admin.keyring on ['vpm034.front.sepia.ceph.com']
2014-01-07T08:50:46.244 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mceph_deploy.gatherkeys[0m][[1;34mDEBUG[0m ] Have ceph.mon.keyring
2014-01-07T08:50:46.244 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mceph_deploy.gatherkeys[0m][[1;34mDEBUG[0m ] Checking vpm034.front.sepia.ceph.com for /var/lib/ceph/bootstrap-osd/ceph.keyring
2014-01-07T08:50:46.277 INFO:teuthology.orchestra.run.err:[10.214.138.93]: Warning: Permanently added 'vpm034.front.sepia.ceph.com,10.214.138.93' (RSA) to the list of known hosts.
2014-01-07T08:50:46.395 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mvpm034.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] connected to host: vpm034.front.sepia.ceph.com
2014-01-07T08:50:46.395 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mvpm034.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] detect platform information from remote host
2014-01-07T08:50:46.414 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mvpm034.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] detect machine type
2014-01-07T08:50:46.418 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mvpm034.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] fetch remote file
2014-01-07T08:50:46.420 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mceph_deploy.gatherkeys[0m][[1;33mWARNIN[0m] Unable to find /var/lib/ceph/bootstrap-osd/ceph.keyring on ['vpm034.front.sepia.ceph.com']
2014-01-07T08:50:46.421 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mceph_deploy.gatherkeys[0m][[1;34mDEBUG[0m ] Checking vpm034.front.sepia.ceph.com for /var/lib/ceph/bootstrap-mds/ceph.keyring
2014-01-07T08:50:46.455 INFO:teuthology.orchestra.run.err:[10.214.138.93]: Warning: Permanently added 'vpm034.front.sepia.ceph.com,10.214.138.93' (RSA) to the list of known hosts.
2014-01-07T08:50:46.574 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mvpm034.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] connected to host: vpm034.front.sepia.ceph.com
2014-01-07T08:50:46.575 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mvpm034.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] detect platform information from remote host
2014-01-07T08:50:46.594 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mvpm034.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] detect machine type
2014-01-07T08:50:46.598 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mvpm034.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] fetch remote file
2014-01-07T08:50:46.600 INFO:teuthology.orchestra.run.err:[10.214.138.93]: [[1mceph_deploy.gatherkeys[0m][[1;33mWARNIN[0m] Unable to find /var/lib/ceph/bootstrap-mds/ceph.keyring on ['vpm034.front.sepia.ceph.com']
Actions
Copy link
 #1

Updated by Alfredo Deza over 10 years ago

I replicated this problem locally using the most current `master` branch. The issue is not really `gatherkeys` but the fact that the monitors don't even start on CentOS hosts.

Actions
Copy link
 #2

Updated by Ian Colle over 10 years ago

  • Project changed from teuthology to Ceph
  • Subject changed from ceph-deploy fails to gatherkeys to monitors fail to start on CentOS hosts
  • Category set to Monitor
  • Assignee changed from Alfredo Deza to Joao Eduardo Luis
Actions
Copy link
 #3

Updated by Joao Eduardo Luis over 10 years ago

  • Subject changed from monitors fail to start on CentOS hosts to centOS: init script silently dies due to missing '/usr/lib64/ceph/ceph_common.sh'

AFAICT, the reason is that the init-script is not even starting any daemon due to failing on

# behave if we are not completely installed (e.g., Debian "removed,
# config remains" state)
test -f $LIBDIR/ceph_common.sh || exit 0

considering that LIBDIR = /usr/lib64/ceph

[ubuntu@vpm025 ~]$ ls /usr/lib64/ceph/
erasure-code
[ubuntu@vpm025 ~]$

So, no ceph_common.sh, no init script. No init script, no daemons come up using 'service'.
Also, the init script is silently exiting in this situation.

The mons work just fine when ran manually though.

Actions
Copy link
 #4

Updated by Ian Colle over 10 years ago

  • Assignee changed from Joao Eduardo Luis to Ken Dreyer

Something with the make file?

Actions
Copy link
 #5

Updated by Ken Dreyer over 10 years ago

I noticed the following warning today while building v0.75.

http://jenkins.ceph.com/job/ceph-build/Arch=x86_64,Distro=fedora19/100/console

Checking for unpackaged file(s): /usr/lib/rpm/check-files /home/jenkins-build/build/workspace/ceph-build/Arch/x86_64/Distro/fedora19/release/0.75/rpm/fc19/BUILDROOT/ceph-0.75-0.fc19.x86_64

  warning: Installed (but unpackaged) file(s) found:
    /usr/bin/ceph_test_c_headers
    /usr/bin/ceph_test_get_blkdev_size
    /usr/lib64/ceph/ceph_common.sh

So it looks like automake is installing ceph_common.sh appropriately, but the RPMs are not packaging the /usr/lib64/ceph directory in %files.

Actions
Copy link
 #6

Updated by Ken Dreyer over 10 years ago

I've pushed an experimental patch to https://github.com/ceph/ceph/tree/wip-7117-redhat

I need to test and see what happens in gitbuilder.

Actions
Copy link
 #7

Updated by Alfredo Deza over 10 years ago

  • Priority changed from High to Urgent

This is now affecting every test that runs on a system with that init script.

For example, all OSDs deployed in this test come back unhealthy because the mons never start:

http://qa-proxy.ceph.com/teuthology/teuthology-2014-01-12_01:10:02-ceph-deploy-master-testing-basic-vps/35701/teuthology.log

Actions
Copy link
 #8

Updated by Ken Dreyer over 10 years ago

  • Status changed from 12 to Fix Under Review

I've verified that with this patch, the RPM packaging properly captures the ceph_common.sh file in /usr/lib64/ceph/.

https://github.com/ceph/ceph/pull/1094

Actions
Copy link
 #9

Updated by Sage Weil about 10 years ago

  • Status changed from Fix Under Review to Resolved
Actions
Copy link

Also available in: Atom PDF