Project

General

Profile

Actions
Copy link

Bug #7254

closed

"pthread lock: Invalid argument" when deploying monitors

Added by Alfredo Deza about 10 years ago. Updated about 10 years ago.

Status:
Closed
Priority:
High
Assignee:
Alfredo Deza
Category:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

2014-01-28T02:38:28.846 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm099[0m][[1;37mINFO[0m  ] Running command: sudo ceph --version
2014-01-28T02:38:28.966 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm099[0m][[1;34mDEBUG[0m ] ceph version 0.75-239-g9e52398 (9e523984a9e134e370153e5c47b13c49f5024e56)
2014-01-28T02:38:28.982 DEBUG:teuthology.orchestra.run:Running [10.214.138.101]: 'cd /home/ubuntu/cephtest/ceph-deploy && ./ceph-deploy mon create-initial'
2014-01-28T02:38:29.229 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mceph_deploy.cli[0m][[1;37mINFO[0m  ] Invoked (1.3.4): ./ceph-deploy mon create-initial
2014-01-28T02:38:29.231 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mceph_deploy.mon[0m][[1;34mDEBUG[0m ] Deploying mon, cluster ceph hosts vpm096 vpm055 vpm098
2014-01-28T02:38:29.231 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mceph_deploy.mon[0m][[1;34mDEBUG[0m ] detecting platform for host vpm096 ...
2014-01-28T02:38:29.291 INFO:teuthology.orchestra.run.err:[10.214.138.101]: Warning: Permanently added 'vpm096,10.214.138.143' (RSA) to the list of known hosts.
2014-01-28T02:38:29.517 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] connected to host: vpm096
2014-01-28T02:38:29.518 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] detect platform information from remote host
2014-01-28T02:38:29.543 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] detect machine type
2014-01-28T02:38:29.549 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mceph_deploy.mon[0m][[1;37mINFO[0m  ] distro info: Fedora 19 Schrödinger’s Cat
2014-01-28T02:38:29.549 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] determining if provided host has same hostname in remote
2014-01-28T02:38:29.549 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] get remote short hostname
2014-01-28T02:38:29.551 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] deploying mon to vpm096
2014-01-28T02:38:29.551 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] get remote short hostname
2014-01-28T02:38:29.553 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] remote hostname: vpm096
2014-01-28T02:38:29.554 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] write cluster configuration to /etc/ceph/{cluster}.conf
2014-01-28T02:38:29.557 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] create the mon path if it does not exist
2014-01-28T02:38:29.558 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] checking for done path: /var/lib/ceph/mon/ceph-vpm096/done
2014-01-28T02:38:29.559 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] done path does not exist: /var/lib/ceph/mon/ceph-vpm096/done
2014-01-28T02:38:29.561 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;37mINFO[0m  ] creating keyring file: /var/lib/ceph/tmp/ceph-vpm096.mon.keyring
2014-01-28T02:38:29.561 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] create the monitor keyring file
2014-01-28T02:38:29.565 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;37mINFO[0m  ] Running command: sudo ceph-mon --cluster ceph --mkfs -i vpm096 --keyring /var/lib/ceph/tmp/ceph-vpm096.mon.keyring
2014-01-28T02:38:29.740 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m] pthread lock: Invalid argument
2014-01-28T02:38:29.740 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m] *** Caught signal (Aborted) **
2014-01-28T02:38:29.741 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  in thread 7f6465b427c0
2014-01-28T02:38:29.741 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  ceph version 0.75-239-g9e52398 (9e523984a9e134e370153e5c47b13c49f5024e56)
2014-01-28T02:38:29.741 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  1: ceph-mon() [0x82ff9e]
2014-01-28T02:38:29.741 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  2: (()+0xefa0) [0x7f6464f44fa0]
2014-01-28T02:38:29.741 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  3: (gsignal()+0x39) [0x7f6463b17a19]
2014-01-28T02:38:29.741 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  4: (abort()+0x148) [0x7f6463b19128]
2014-01-28T02:38:29.741 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  5: (()+0x13eb3) [0x7f64646d1eb3]
2014-01-28T02:38:29.742 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  6: (leveldb::port::Mutex::Unlock()+0) [0x7f64646f4340]
2014-01-28T02:38:29.743 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  7: (leveldb::DBImpl::~DBImpl()+0x2a) [0x7f64646d770a]
2014-01-28T02:38:29.743 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  8: (leveldb::DBImpl::~DBImpl()+0x9) [0x7f64646d7b29]
2014-01-28T02:38:29.743 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  9: (LevelDBStore::LevelDBWholeSpaceIteratorImpl::lower_bound(std::string const&, std::string const&)+0x39) [0x801659]
2014-01-28T02:38:29.743 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  10: (MonitorDBStore::exists(std::string const&, std::string const&)+0xd9) [0x534199]
2014-01-28T02:38:29.743 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  11: (Monitor::check_fsid()+0x4d) [0x54ac7d]
2014-01-28T02:38:29.743 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  12: (Monitor::mkfs(ceph::buffer::list&)+0x34) [0x54c394]
2014-01-28T02:38:29.744 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  13: (main()+0x703) [0x523b13]
2014-01-28T02:38:29.744 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  14: (__libc_start_main()+0xf5) [0x7f6463b03b75]
2014-01-28T02:38:29.745 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  15: ceph-mon() [0x530899]
2014-01-28T02:38:29.745 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m] 2014-01-28 10:38:29.724910 7f6465b427c0 -1 *** Caught signal (Aborted) **
2014-01-28T02:38:29.745 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  in thread 7f6465b427c0
2014-01-28T02:38:29.745 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]
2014-01-28T02:38:29.747 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  ceph version 0.75-239-g9e52398 (9e523984a9e134e370153e5c47b13c49f5024e56)
2014-01-28T02:38:29.748 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  1: ceph-mon() [0x82ff9e]
2014-01-28T02:38:29.748 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  2: (()+0xefa0) [0x7f6464f44fa0]
2014-01-28T02:38:29.748 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  3: (gsignal()+0x39) [0x7f6463b17a19]
2014-01-28T02:38:29.748 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  4: (abort()+0x148) [0x7f6463b19128]
2014-01-28T02:38:29.748 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  5: (()+0x13eb3) [0x7f64646d1eb3]
2014-01-28T02:38:29.748 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  6: (leveldb::port::Mutex::Unlock()+0) [0x7f64646f4340]
2014-01-28T02:38:29.749 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  7: (leveldb::DBImpl::~DBImpl()+0x2a) [0x7f64646d770a]
2014-01-28T02:38:29.751 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  8: (leveldb::DBImpl::~DBImpl()+0x9) [0x7f64646d7b29]
2014-01-28T02:38:29.751 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  9: (LevelDBStore::LevelDBWholeSpaceIteratorImpl::lower_bound(std::string const&, std::string const&)+0x39) [0x801659]
2014-01-28T02:38:29.751 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  10: (MonitorDBStore::exists(std::string const&, std::string const&)+0xd9) [0x534199]
2014-01-28T02:38:29.751 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  11: (Monitor::check_fsid()+0x4d) [0x54ac7d]
2014-01-28T02:38:29.751 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  12: (Monitor::mkfs(ceph::buffer::list&)+0x34) [0x54c394]
2014-01-28T02:38:29.751 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  13: (main()+0x703) [0x523b13]
2014-01-28T02:38:29.752 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  14: (__libc_start_main()+0xf5) [0x7f6463b03b75]
2014-01-28T02:38:29.752 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  15: ceph-mon() [0x530899]
2014-01-28T02:38:29.752 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2014-01-28T02:38:29.752 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]
2014-01-28T02:38:29.752 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]      0> 2014-01-28 10:38:29.724910 7f6465b427c0 -1 *** Caught signal (Aborted) **
2014-01-28T02:38:29.752 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  in thread 7f6465b427c0
2014-01-28T02:38:29.754 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]
2014-01-28T02:38:29.754 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  ceph version 0.75-239-g9e52398 (9e523984a9e134e370153e5c47b13c49f5024e56)
2014-01-28T02:38:29.754 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  1: ceph-mon() [0x82ff9e]
2014-01-28T02:38:29.754 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  2: (()+0xefa0) [0x7f6464f44fa0]
2014-01-28T02:38:29.754 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  3: (gsignal()+0x39) [0x7f6463b17a19]
2014-01-28T02:38:29.755 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  4: (abort()+0x148) [0x7f6463b19128]
2014-01-28T02:38:29.755 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  5: (()+0x13eb3) [0x7f64646d1eb3]
2014-01-28T02:38:29.755 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  6: (leveldb::port::Mutex::Unlock()+0) [0x7f64646f4340]
2014-01-28T02:38:29.755 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  7: (leveldb::DBImpl::~DBImpl()+0x2a) [0x7f64646d770a]
2014-01-28T02:38:29.755 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  8: (leveldb::DBImpl::~DBImpl()+0x9) [0x7f64646d7b29]
2014-01-28T02:38:29.755 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  9: (LevelDBStore::LevelDBWholeSpaceIteratorImpl::lower_bound(std::string const&, std::string const&)+0x39) [0x801659]
2014-01-28T02:38:29.757 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  10: (MonitorDBStore::exists(std::string const&, std::string const&)+0xd9) [0x534199]
2014-01-28T02:38:29.757 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  11: (Monitor::check_fsid()+0x4d) [0x54ac7d]
2014-01-28T02:38:29.757 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  12: (Monitor::mkfs(ceph::buffer::list&)+0x34) [0x54c394]
2014-01-28T02:38:29.757 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  13: (main()+0x703) [0x523b13]
2014-01-28T02:38:29.757 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  14: (__libc_start_main()+0xf5) [0x7f6463b03b75]
2014-01-28T02:38:29.758 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  15: ceph-mon() [0x530899]
2014-01-28T02:38:29.758 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2014-01-28T02:38:29.758 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]
2014-01-28T02:38:29.758 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] ceph-mon: mon.noname-a 10.214.138.143:6789/0 is local, renaming to mon.vpm096
2014-01-28T02:38:29.758 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] ceph-mon: set fsid to 8492734e-1530-4eea-9680-039d94556a64
2014-01-28T02:38:29.759 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;31mERROR[0m ] RuntimeError: command returned non-zero exit status: -6
2014-01-28T02:38:29.759 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mceph_deploy.mon[0m][[1;31mERROR[0m ] Failed to execute command: ceph-mon --cluster ceph --mkfs -i vpm096 --keyring /var/lib/ceph/tmp/ceph-vpm096.mon.keyring

Failure log: http://qa-proxy.ceph.com/teuthology/teuthology-2014-01-28_01:10:01-ceph-deploy-next-testing-basic-vps/56490/teuthology.log

Actions
Copy link
 #1

Updated by Alfredo Deza about 10 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by Alfredo Deza about 10 years ago

  • Priority changed from Normal to High

Still seeing same failures in ceph-deploy tests, a similar issue happened a while ago when a patched leveldb was used (see issue #6022)

Actions
Copy link
 #3

Updated by Joao Eduardo Luis about 10 years ago

  • Status changed from New to In Progress
Actions
Copy link
 #4

Updated by Joao Eduardo Luis about 10 years ago

  • Status changed from In Progress to 4

issue appears to be that the fedora 19 machines are running with a 1 year old leveldb version (1.9.0). This causes problems when running the monitor, regardless of doing it manually or through ceph-deploy.

Upgrading the package manually on the vms to 1.12 makes the problem go away.

Ticket #7429 has been filed and it should fix this issue.

Actions
Copy link
 #5

Updated by Ian Colle about 10 years ago

  • Assignee changed from Joao Eduardo Luis to Alfredo Deza

Alfredo are you still seeing this?

Actions
Copy link
 #6

Updated by Alfredo Deza about 10 years ago

  • Status changed from 4 to Closed

This is no longer happening.

Actions
Copy link

Also available in: Atom PDF