Bug #7254: "pthread lock: Invalid argument" when deploying monitors - teuthology - Ceph

Bug #7254

Updated by Alfredo Deza over 10 years ago

<pre> 
 2014-01-28T02:38:28.846 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm099[0m][[1;37mINFO[0m    ] Running command: sudo ceph --version 
 2014-01-28T02:38:28.966 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm099[0m][[1;34mDEBUG[0m ] ceph version 0.75-239-g9e52398 (9e523984a9e134e370153e5c47b13c49f5024e56) 
 2014-01-28T02:38:28.982 DEBUG:teuthology.orchestra.run:Running [10.214.138.101]: 'cd /home/ubuntu/cephtest/ceph-deploy && ./ceph-deploy mon create-initial' 
 2014-01-28T02:38:29.229 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mceph_deploy.cli[0m][[1;37mINFO[0m    ] Invoked (1.3.4): ./ceph-deploy mon create-initial 
 2014-01-28T02:38:29.231 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mceph_deploy.mon[0m][[1;34mDEBUG[0m ] Deploying mon, cluster ceph hosts vpm096 vpm055 vpm098 
 2014-01-28T02:38:29.231 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mceph_deploy.mon[0m][[1;34mDEBUG[0m ] detecting platform for host vpm096 ... 
 2014-01-28T02:38:29.291 INFO:teuthology.orchestra.run.err:[10.214.138.101]: Warning: Permanently added 'vpm096,10.214.138.143' (RSA) to the list of known hosts. 
 2014-01-28T02:38:29.517 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] connected to host: vpm096 
 2014-01-28T02:38:29.518 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] detect platform information from remote host 
 2014-01-28T02:38:29.543 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] detect machine type 
 2014-01-28T02:38:29.549 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mceph_deploy.mon[0m][[1;37mINFO[0m    ] distro info: Fedora 19 SchrÃ¶dingerâ€™s Cat 
 2014-01-28T02:38:29.549 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] determining if provided host has same hostname in remote 
 2014-01-28T02:38:29.549 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] get remote short hostname 
 2014-01-28T02:38:29.551 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] deploying mon to vpm096 
 2014-01-28T02:38:29.551 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] get remote short hostname 
 2014-01-28T02:38:29.553 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] remote hostname: vpm096 
 2014-01-28T02:38:29.554 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] write cluster configuration to /etc/ceph/{cluster}.conf 
 2014-01-28T02:38:29.557 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] create the mon path if it does not exist 
 2014-01-28T02:38:29.558 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] checking for done path: /var/lib/ceph/mon/ceph-vpm096/done 
 2014-01-28T02:38:29.559 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] done path does not exist: /var/lib/ceph/mon/ceph-vpm096/done 
 2014-01-28T02:38:29.561 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;37mINFO[0m    ] creating keyring file: /var/lib/ceph/tmp/ceph-vpm096.mon.keyring 
 2014-01-28T02:38:29.561 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] create the monitor keyring file 
 2014-01-28T02:38:29.565 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;37mINFO[0m    ] Running command: sudo ceph-mon --cluster ceph --mkfs -i vpm096 --keyring /var/lib/ceph/tmp/ceph-vpm096.mon.keyring 
 2014-01-28T02:38:29.740 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m] pthread lock: Invalid argument 
 2014-01-28T02:38:29.740 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m] *** Caught signal (Aborted) ** 
 2014-01-28T02:38:29.741 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    in thread 7f6465b427c0 
 2014-01-28T02:38:29.741 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    ceph version 0.75-239-g9e52398 (9e523984a9e134e370153e5c47b13c49f5024e56) 
 2014-01-28T02:38:29.741 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    1: ceph-mon() [0x82ff9e] 
 2014-01-28T02:38:29.741 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    2: (()+0xefa0) [0x7f6464f44fa0] 
 2014-01-28T02:38:29.741 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    3: (gsignal()+0x39) [0x7f6463b17a19] 
 2014-01-28T02:38:29.741 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    4: (abort()+0x148) [0x7f6463b19128] 
 2014-01-28T02:38:29.741 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    5: (()+0x13eb3) [0x7f64646d1eb3] 
 2014-01-28T02:38:29.742 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    6: (leveldb::port::Mutex::Unlock()+0) [0x7f64646f4340] 
 2014-01-28T02:38:29.743 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    7: (leveldb::DBImpl::~DBImpl()+0x2a) [0x7f64646d770a] 
 2014-01-28T02:38:29.743 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    8: (leveldb::DBImpl::~DBImpl()+0x9) [0x7f64646d7b29] 
 2014-01-28T02:38:29.743 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    9: (LevelDBStore::LevelDBWholeSpaceIteratorImpl::lower_bound(std::string const&, std::string const&)+0x39) [0x801659] 
 2014-01-28T02:38:29.743 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    10: (MonitorDBStore::exists(std::string const&, std::string const&)+0xd9) [0x534199] 
 2014-01-28T02:38:29.743 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    11: (Monitor::check_fsid()+0x4d) [0x54ac7d] 
 2014-01-28T02:38:29.743 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    12: (Monitor::mkfs(ceph::buffer::list&)+0x34) [0x54c394] 
 2014-01-28T02:38:29.744 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    13: (main()+0x703) [0x523b13] 
 2014-01-28T02:38:29.744 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    14: (__libc_start_main()+0xf5) [0x7f6463b03b75] 
 2014-01-28T02:38:29.745 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    15: ceph-mon() [0x530899] 
 2014-01-28T02:38:29.745 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m] 2014-01-28 10:38:29.724910 7f6465b427c0 -1 *** Caught signal (Aborted) ** 
 2014-01-28T02:38:29.745 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    in thread 7f6465b427c0 
 2014-01-28T02:38:29.745 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m] 
 2014-01-28T02:38:29.747 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    ceph version 0.75-239-g9e52398 (9e523984a9e134e370153e5c47b13c49f5024e56) 
 2014-01-28T02:38:29.748 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    1: ceph-mon() [0x82ff9e] 
 2014-01-28T02:38:29.748 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    2: (()+0xefa0) [0x7f6464f44fa0] 
 2014-01-28T02:38:29.748 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    3: (gsignal()+0x39) [0x7f6463b17a19] 
 2014-01-28T02:38:29.748 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    4: (abort()+0x148) [0x7f6463b19128] 
 2014-01-28T02:38:29.748 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    5: (()+0x13eb3) [0x7f64646d1eb3] 
 2014-01-28T02:38:29.748 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    6: (leveldb::port::Mutex::Unlock()+0) [0x7f64646f4340] 
 2014-01-28T02:38:29.749 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    7: (leveldb::DBImpl::~DBImpl()+0x2a) [0x7f64646d770a] 
 2014-01-28T02:38:29.751 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    8: (leveldb::DBImpl::~DBImpl()+0x9) [0x7f64646d7b29] 
 2014-01-28T02:38:29.751 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    9: (LevelDBStore::LevelDBWholeSpaceIteratorImpl::lower_bound(std::string const&, std::string const&)+0x39) [0x801659] 
 2014-01-28T02:38:29.751 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    10: (MonitorDBStore::exists(std::string const&, std::string const&)+0xd9) [0x534199] 
 2014-01-28T02:38:29.751 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    11: (Monitor::check_fsid()+0x4d) [0x54ac7d] 
 2014-01-28T02:38:29.751 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    12: (Monitor::mkfs(ceph::buffer::list&)+0x34) [0x54c394] 
 2014-01-28T02:38:29.751 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    13: (main()+0x703) [0x523b13] 
 2014-01-28T02:38:29.752 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    14: (__libc_start_main()+0xf5) [0x7f6463b03b75] 
 2014-01-28T02:38:29.752 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    15: ceph-mon() [0x530899] 
 2014-01-28T02:38:29.752 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 
 2014-01-28T02:38:29.752 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m] 
 2014-01-28T02:38:29.752 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]        0> 2014-01-28 10:38:29.724910 7f6465b427c0 -1 *** Caught signal (Aborted) ** 
 2014-01-28T02:38:29.752 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    in thread 7f6465b427c0 
 2014-01-28T02:38:29.754 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m] 
 2014-01-28T02:38:29.754 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    ceph version 0.75-239-g9e52398 (9e523984a9e134e370153e5c47b13c49f5024e56) 
 2014-01-28T02:38:29.754 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    1: ceph-mon() [0x82ff9e] 
 2014-01-28T02:38:29.754 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    2: (()+0xefa0) [0x7f6464f44fa0] 
 2014-01-28T02:38:29.754 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    3: (gsignal()+0x39) [0x7f6463b17a19] 
 2014-01-28T02:38:29.755 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    4: (abort()+0x148) [0x7f6463b19128] 
 2014-01-28T02:38:29.755 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    5: (()+0x13eb3) [0x7f64646d1eb3] 
 2014-01-28T02:38:29.755 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    6: (leveldb::port::Mutex::Unlock()+0) [0x7f64646f4340] 
 2014-01-28T02:38:29.755 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    7: (leveldb::DBImpl::~DBImpl()+0x2a) [0x7f64646d770a] 
 2014-01-28T02:38:29.755 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    8: (leveldb::DBImpl::~DBImpl()+0x9) [0x7f64646d7b29] 
 2014-01-28T02:38:29.755 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    9: (LevelDBStore::LevelDBWholeSpaceIteratorImpl::lower_bound(std::string const&, std::string const&)+0x39) [0x801659] 
 2014-01-28T02:38:29.757 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    10: (MonitorDBStore::exists(std::string const&, std::string const&)+0xd9) [0x534199] 
 2014-01-28T02:38:29.757 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    11: (Monitor::check_fsid()+0x4d) [0x54ac7d] 
 2014-01-28T02:38:29.757 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    12: (Monitor::mkfs(ceph::buffer::list&)+0x34) [0x54c394] 
 2014-01-28T02:38:29.757 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    13: (main()+0x703) [0x523b13] 
 2014-01-28T02:38:29.757 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    14: (__libc_start_main()+0xf5) [0x7f6463b03b75] 
 2014-01-28T02:38:29.758 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    15: ceph-mon() [0x530899] 
 2014-01-28T02:38:29.758 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m]    NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 
 2014-01-28T02:38:29.758 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;33mWARNIN[0m] 
 2014-01-28T02:38:29.758 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] ceph-mon: mon.noname-a 10.214.138.143:6789/0 is local, renaming to mon.vpm096 
 2014-01-28T02:38:29.758 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;34mDEBUG[0m ] ceph-mon: set fsid to 8492734e-1530-4eea-9680-039d94556a64 
 2014-01-28T02:38:29.759 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mvpm096[0m][[1;31mERROR[0m ] RuntimeError: command returned non-zero exit status: -6 
 2014-01-28T02:38:29.759 INFO:teuthology.orchestra.run.err:[10.214.138.101]: [[1mceph_deploy.mon[0m][[1;31mERROR[0m ] Failed to execute command: ceph-mon --cluster ceph --mkfs -i vpm096 --keyring /var/lib/ceph/tmp/ceph-vpm096.mon.keyring 
 </pre> 

 Failure log: http://qa-proxy.ceph.com/teuthology/teuthology-2014-01-28_01:10:01-ceph-deploy-next-testing-basic-vps/56490/teuthology.log

Back

Project

General

Profile

Tools » teuthology

Bug #7254