Bug #6022
monitor crashed during ceph-deploy mon create on centos 6.4 and 6.3
0%
Description
logs: ubuntu@teuthology:/a/teuthology-2013-08-16_01:10:04-ceph-deploy-master-testing-basic-vps/109409
2013-08-16T03:41:39.847 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm008^[[0m][^[[1;37mINFO^[[0m ] Running command: ceph --version 2013-08-16T03:41:40.061 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm008^[[0m][^[[1;37mINFO^[[0m ] ceph version 0.67-180-g067d575 (067d5756a8b17af16ce73a487698471c079a2fb6) 2013-08-16T03:41:40.114 DEBUG:teuthology.misc:with jobid basedir: 109409 2013-08-16T03:41:40.115 DEBUG:teuthology.orchestra.run:Running [10.214.138.140]: 'cd /home/ubuntu/cephtest/109409/ceph-deploy && ./ceph-deploy mon create vpm063.front.sepia.ceph.com vpm060.front.sepia.ceph.com' 2013-08-16T03:41:40.283 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mceph_deploy.mon^[[0m][^[[1;34mDEBUG^[[0m ] Deploying mon, cluster ceph hosts vpm063.front.sepia.ceph.com vpm060.front.sepia.ceph.com 2013-08-16T03:41:40.283 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mceph_deploy.mon^[[0m][^[[1;34mDEBUG^[[0m ] detecting platform for host vpm063.front.sepia.ceph.com ... 2013-08-16T03:41:40.808 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mceph_deploy.mon^[[0m][^[[1;37mINFO^[[0m ] distro info: CentOS 6.4 Final 2013-08-16T03:41:40.809 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;34mDEBUG^[[0m ] deploying mon to vpm063.front.sepia.ceph.com 2013-08-16T03:41:40.813 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;34mDEBUG^[[0m ] remote hostname: vpm063 2013-08-16T03:41:40.824 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;37mINFO^[[0m ] write cluster configuration to /etc/ceph/{cluster}.conf 2013-08-16T03:41:40.855 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;37mINFO^[[0m ] creating path: /var/lib/ceph/mon/ceph-vpm063 2013-08-16T03:41:40.859 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;34mDEBUG^[[0m ] checking for done path: /var/lib/ceph/mon/ceph-vpm063/done 2013-08-16T03:41:40.865 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;34mDEBUG^[[0m ] done path does not exist: /var/lib/ceph/mon/ceph-vpm063/done 2013-08-16T03:41:40.870 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;37mINFO^[[0m ] creating keyring file: /var/lib/ceph/tmp/ceph-vpm063.mon.keyring 2013-08-16T03:41:40.880 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;37mINFO^[[0m ] create the monitor keyring file 2013-08-16T03:41:40.899 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;37mINFO^[[0m ] Running command: ceph-mon --cluster ceph --mkfs -i vpm063 --keyring /var/lib/ceph/tmp/ceph-vpm063.mon.keyring 2013-08-16T03:41:41.035 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] Traceback (most recent call last): 2013-08-16T03:41:41.035 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] File "/home/ubuntu/cephtest/109409/ceph-deploy/ceph_deploy/hosts/common.py", line 72, in mon_create 2013-08-16T03:41:41.040 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] File "/home/ubuntu/cephtest/109409/ceph-deploy/ceph_deploy/util/decorators.py", line 10, in inner 2013-08-16T03:41:41.041 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] def inner(*args, **kwargs): 2013-08-16T03:41:41.043 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] File "/home/ubuntu/cephtest/109409/ceph-deploy/ceph_deploy/util/wrappers.py", line 6, in remote_call 2013-08-16T03:41:41.044 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] This allows us to only remote-execute the actual calls, not whole functions. 2013-08-16T03:41:41.046 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] File "/usr/lib64/python2.6/subprocess.py", line 505, in check_call 2013-08-16T03:41:41.047 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] raise CalledProcessError(retcode, cmd) 2013-08-16T03:41:41.049 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] CalledProcessError: Command '['ceph-mon', '--cluster', 'ceph', '--mkfs', '-i', 'vpm063', '--keyring', '/var/lib/ceph/tmp/ceph-vpm063.mon.keyring']' returned non-zero exit status -6 2013-08-16T03:41:41.057 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;37mINFO^[[0m ] ceph-mon: mon.noname-a 10.214.138.140:6789/0 is local, renaming to mon.vpm063 2013-08-16T03:41:41.057 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;37mINFO^[[0m ] ceph-mon: set fsid to 11528781-fddb-42b9-a57b-6c01f879676c 2013-08-16T03:41:41.057 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] pthread lock: Invalid argument 2013-08-16T03:41:41.058 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] *** Caught signal (Aborted) ** 2013-08-16T03:41:41.058 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] in thread 7ff8a1d807a0 2013-08-16T03:41:41.058 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] ceph version 0.67-180-g067d575 (067d5756a8b17af16ce73a487698471c079a2fb6) 2013-08-16T03:41:41.059 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] 1: ceph-mon() [0x6483f9] 2013-08-16T03:41:41.059 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] 2: (()+0xf500) [0x7ff8a13f0500] 2013-08-16T03:41:41.061 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] 3: (gsignal()+0x35) [0x7ff89fdfc8a5] 2013-08-16T03:41:41.061 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] 4: (abort()+0x175) [0x7ff89fdfe085] 2013-08-16T03:41:41.062 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] 5: (()+0x34d71) [0x7ff8a0931d71] 2013-08-16T03:41:41.062 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] 6: (leveldb::DBImpl::Get(leveldb::ReadOptions const&, leveldb::Slice const&, leveldb::Value*)+0x50) [0x7ff8a0917120] 2013-08-16T03:41:41.062 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] 7: (LevelDBStore::_get_iterator()+0x41) [0x632d71] 2013-08-16T03:41:41.062 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] 8: (MonitorDBStore::exists(std::string const&, std::string const&)+0x28) [0x523898] 2013-08-16T03:41:41.062 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] 9: (Monitor::check_fsid()+0x59) [0x54f069] 2013-08-16T03:41:41.062 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] 10: (Monitor::mkfs(ceph::buffer::list&)+0x36) [0x5566e6] 2013-08-16T03:41:41.063 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] 11: (main()+0x11a3) [0x51e223] 2013-08-16T03:41:41.063 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] 12: (__libc_start_main()+0xfd) [0x7ff89fde8cdd] 2013-08-16T03:41:41.063 INFO:teuthology.orchestra.run.err:[10.214.138.140]: [^[[1mvpm063.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] 13: ceph-mon() [0x51bcb9]
History
#1 Updated by Tamilarasi muthamizhan over 10 years ago
ubuntu@teuthology:/a/teuthology-2013-08-16_01:10:04-ceph-deploy-master-testing-basic-vps/109409$ cat config.yaml kernel: kdb: true sha1: 68d04c9a036bea12d76fdd51e926aef771e8e041 machine_type: vps nuke-on-error: true os_type: centos os_version: '6.4' overrides: admin_socket: branch: master ceph: conf: mon: debug mon: 20 debug ms: 1 debug paxos: 20 log-whitelist: - slow request sha1: 067d5756a8b17af16ce73a487698471c079a2fb6 ceph-deploy: branch: dev: master conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: debug mon: 1 debug ms: 20 debug paxos: 20 install: ceph: sha1: 067d5756a8b17af16ce73a487698471c079a2fb6 s3tests: branch: master workunit: sha1: 067d5756a8b17af16ce73a487698471c079a2fb6 roles: - - mon.a - mds.0 - osd.0 - - mon.b - osd.1 - - client.0 targets: ubuntu@vpm008.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv1AqKrrBcT/XFYUTzfYxOvAHtw/COM2Q+2/zgyTysCoROGzohhR3dmvKubT6zUGPEXBM8NgSW71AUm8wtcWYPfOFVzYnpWPDdcWaw7zYZWSblm+ufMuHuV4gSojfpXxjXzwYjdXmcFatinqrHw9uU/bVGrfzfim3XyjzjYLsEss0rUUVajKB6EqFTO9BFOGl2xPCEv2rV9GFo3vh7L8Qf3C61Pm7k79nqGx6yGWS9MtnlYBnOE40LVPJGFuli2k9BzQXUuO60huxWc5XLnPu7ObgtMQU3TqSu1VN2cMy7Q1kh76f5/X1DBluiOGQ4MhIqWEDehaMp1zJm/Tdbsigww== ubuntu@vpm060.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAtjzg7mdG5R75eqO164QQTXe2w3T8gh70K0FCFtwB908TAzEMs1EiQ70togqyfdcnmEYYv8lH2QTWpvDdpLHRP5a34Jd1sT1FQHlMmQhuBsmG6bBY+gjEffkh0/wy9mIPFwvv8YzHB45py7Qvd2kArhUxqufjuSOmBDb3kbPhuKD+/A79OpemAgLPu1l3UgzcGmY488UxQI3K7FmgEJYi3uM0Cu4VixlgWiOLU3ieOhRTJh2kTM8AIy/SwCYxtvY2YMoTaYdZc3QlzsuoMsoSOHsBR95td8SV4D1yiJFY5cKr+wMCMFj2f+RsgWOh3gE+igiShuGGNxbemc9PeuXsHQ== ubuntu@vpm063.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAySgNl2xmwNYQBMEbYtGXCNWtwkGC5EfImlt79FUjtl3x5r5/IxnDSaTaLjNAG9Gv65HROT+/5OcfYtVvz4Y14qkl9kezhY+Zounn1T+tTGR4LUTTYS3FkdB1kasdyZVRSKCknDjVTm58gUK2Ynmp9YzZDgFUdMkIeA3KkmnzmbPfXYOh12s1s32qXwZWLF0VMTqttgW4S/oOsM6kG3PltQSpaeiy4qXIjwezpSrHAYFmgFHtl67GhvhfAVyKQfjm8nNTNXac9oOS/4LQHAlXUyDCUbfoGeHtJebWLmBpQawJhnko46Utpop4TG1mbZizFImwHCdW0RveWI2juSqDLw== tasks: - internal.lock_machines: - 3 - vps - internal.save_config: null - internal.check_lock: null - internal.connect: null - internal.check_conflict: null - internal.check_ceph_data: null - internal.vm_setup: null - internal.base: null - internal.archive: null - internal.coredump: null - internal.syslog: null - internal.timer: null - chef: null - clock.check: null - install: extras: true - ssh_keys: null - ceph-deploy: null - workunit: clients: client.0: - rbd/test_librbd.sh teuthology_branch: master
#2 Updated by Sage Weil over 10 years ago
- Status changed from New to 12
#3 Updated by Sage Weil over 10 years ago
- Status changed from 12 to New
verified that i can reproduce this. my first guess is a problem with the leveldb package on 6.4. 6.3 passes?
#4 Updated by Alfredo Deza over 10 years ago
6.3 fails in the same way:
2013-08-19T02:02:16.748 DEBUG:teuthology.orchestra.run:Running [10.214.138.64]: 'cd /home/ubuntu/cephtest/1288/ceph-deploy && ./ceph-deploy mon create vpm028.front.sepia.ceph.com vpm027.front.sepia.ceph.com' 2013-08-19T02:02:16.974 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mceph_deploy.mon^[[0m][^[[1;34mDEBUG^[[0m ] Deploying mon, cluster ceph hosts vpm028.front.sepia.ceph.com vpm027.front.sepia.ceph.com 2013-08-19T02:02:16.974 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mceph_deploy.mon^[[0m][^[[1;34mDEBUG^[[0m ] detecting platform for host vpm028.front.sepia.ceph.com ... 2013-08-19T02:02:17.330 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mceph_deploy.mon^[[0m][^[[1;37mINFO^[[0m ] distro info: CentOS 6.3 Final 2013-08-19T02:02:17.330 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;34mDEBUG^[[0m ] deploying mon to vpm028.front.sepia.ceph.com 2013-08-19T02:02:17.334 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;34mDEBUG^[[0m ] remote hostname: vpm028 2013-08-19T02:02:17.346 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;37mINFO^[[0m ] write cluster configuration to /etc/ceph/{cluster}.conf 2013-08-19T02:02:17.384 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;37mINFO^[[0m ] creating path: /var/lib/ceph/mon/ceph-vpm028 2013-08-19T02:02:17.388 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;34mDEBUG^[[0m ] checking for done path: /var/lib/ceph/mon/ceph-vpm028/done 2013-08-19T02:02:17.394 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;34mDEBUG^[[0m ] done path does not exist: /var/lib/ceph/mon/ceph-vpm028/done 2013-08-19T02:02:17.400 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;37mINFO^[[0m ] creating keyring file: /var/lib/ceph/tmp/ceph-vpm028.mon.keyring 2013-08-19T02:02:17.410 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;37mINFO^[[0m ] create the monitor keyring file 2013-08-19T02:02:17.430 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;37mINFO^[[0m ] Running command: ceph-mon --cluster ceph --mkfs -i vpm028 --keyring /var/lib/ceph/tmp/ceph-vpm028.mon.keyring 2013-08-19T02:02:17.748 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] Traceback (most recent call last): 2013-08-19T02:02:17.748 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] File "/home/ubuntu/cephtest/1288/ceph-deploy/ceph_deploy/hosts/common.py", line 72, in mon_create 2013-08-19T02:02:17.756 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] File "/home/ubuntu/cephtest/1288/ceph-deploy/ceph_deploy/util/decorators.py", line 10, in inner 2013-08-19T02:02:17.757 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] def inner(*args, **kwargs): 2013-08-19T02:02:17.759 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] File "/home/ubuntu/cephtest/1288/ceph-deploy/ceph_deploy/util/wrappers.py", line 6, in remote_call 2013-08-19T02:02:17.761 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] This allows us to only remote-execute the actual calls, not whole functions. 2013-08-19T02:02:17.762 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] File "/usr/lib64/python2.6/subprocess.py", line 505, in check_call 2013-08-19T02:02:17.764 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] raise CalledProcessError(retcode, cmd) 2013-08-19T02:02:17.765 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] CalledProcessError: Command '['ceph-mon', '--cluster', 'ceph', '--mkfs', '-i', 'vpm028', '--keyring', '/var/lib/ceph/tmp/ceph-vpm028.mon.keyring']' returned non-zero exit status -6 2013-08-19T02:02:17.773 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;37mINFO^[[0m ] ceph-mon: mon.noname-a 10.214.138.64:6789/0 is local, renaming to mon.vpm028 2013-08-19T02:02:17.774 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;37mINFO^[[0m ] ceph-mon: set fsid to aef793bf-e8ff-42d8-a936-1b24d89ac79e 2013-08-19T02:02:17.774 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] pthread lock: Invalid argument 2013-08-19T02:02:17.774 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] *** Caught signal (Aborted) ** 2013-08-19T02:02:17.774 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] in thread 7f3bb93ed7a0 2013-08-19T02:02:17.775 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] ceph version 0.67-215-g4677041 (4677041da309234feb7b7df28e1d73db99d27c72) 2013-08-19T02:02:17.775 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] 1: ceph-mon() [0x6483f9] 2013-08-19T02:02:17.775 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] 2: (()+0xf500) [0x7f3bb8a5b500] 2013-08-19T02:02:17.775 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] 3: (gsignal()+0x35) [0x7f3bb74678a5] 2013-08-19T02:02:17.775 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] 4: (abort()+0x175) [0x7f3bb7469085] 2013-08-19T02:02:17.776 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] 5: (()+0x34d71) [0x7f3bb7f9cd71] 2013-08-19T02:02:17.776 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] 6: (leveldb::DBImpl::Get(leveldb::ReadOptions const&, leveldb::Slice const&, leveldb::Value*)+0x50) [0x7f3bb7f82120] 2013-08-19T02:02:17.776 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] 7: (LevelDBStore::_get_iterator()+0x41) [0x632d71] 2013-08-19T02:02:17.776 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] 8: (MonitorDBStore::exists(std::string const&, std::string const&)+0x28) [0x523898] 2013-08-19T02:02:17.777 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] 9: (Monitor::check_fsid()+0x59) [0x54f069] 2013-08-19T02:02:17.777 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] 10: (Monitor::mkfs(ceph::buffer::list&)+0x36) [0x5566e6] 2013-08-19T02:02:17.777 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] 11: (main()+0x11a3) [0x51e223] 2013-08-19T02:02:17.778 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] 12: (__libc_start_main()+0xfd) [0x7f3bb7453cdd] 2013-08-19T02:02:17.778 INFO:teuthology.orchestra.run.err:[10.214.138.64]: [^[[1mvpm028.front.sepia.ceph.com^[[0m][^[[1;31mERROR^[[0m ] 13: ceph-mon() [0x51bcb9]
#5 Updated by Alfredo Deza over 10 years ago
- Subject changed from monitor crashed during ceph-deploy mon create on centos 6.4 to monitor crashed during ceph-deploy mon create on centos 6.4 and 6.3
#6 Updated by Sage Weil over 10 years ago
- Priority changed from Urgent to Immediate
#7 Updated by Ian Colle over 10 years ago
- Assignee set to Anonymous
#8 Updated by Anonymous over 10 years ago
Looks like the old leveldb-1.7.0 package works ok with dumpling on centos6.3, but the new leveldb-1.12 package does not.
Testing new ceph 0.67.1 install with a prior install of leveldb from epel:
Installed Packages
leveldb.x86_64 1.7.0-2.el6 @epel
ceph-deploy mon create was successful.
Testing with leveldb 1.12
Installed Packages
leveldb.x86_64 1.12.0-3.el6 installed
ceph-deploy hangs in the mon create command, with this back trace:
(gdb) bt
#0 0x00007fee39985dd0 in __pause_nocancel () from /lib64/libpthread.so.0
#1 0x00007fee3997fffb in __pthread_mutex_lock_full () from /lib64/libpthread.so.0
#2 0x00007fee38ec7e79 in leveldb::port::Mutex::Lock (this=<value optimized out>) at port_posix.cc:26
#3 0x00007fee38ead120 in MutexLock (this=0x7fffbbc27ca0, options=..., key=..., value=0x81cff)
at ../util/mutexlock.h:27
#4 leveldb::DBImpl::Get (this=0x7fffbbc27ca0, options=..., key=..., value=0x81cff) at db_impl.cc:1098
#5 0x0000000000630551 in LevelDBStore::_get_iterator() ()
#6 0x0000000000523368 in MonitorDBStore::exists(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
#7 0x000000000054c499 in Monitor::check_fsid() ()
#8 0x000000000055a6a6 in Monitor::mkfs(ceph::buffer::list&) ()
#9 0x000000000051dcf3 in main ()
Removed ceph and leveldb-1.12 and resintalled. leveldb dependencies was resolved from epel:
Dependencies Resolved
==========================================================================================================
Package Arch Version Repository Size
==========================================================================================================
Installing:
ceph x86_64 0.67.1-0.el6 ceph 15 M
Installing for dependencies:
leveldb x86_64 1.7.0-2.el6 epel 158 k
libcephfs1 x86_64 0.67.1-0.el6 ceph 1.7 M
librados2 x86_64 0.67.1-0.el6 ceph 1.5 M
librbd1 x86_64 0.67.1-0.el6 ceph 268 k
python-ceph x86_64 0.67.1-0.el6 ceph 120 k
ceph-deploy create mon worked correctly:
[glowell@gary-centos-01 test]$ ceph-deploy new gary-centos-01
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][DEBUG ] Resolving host gary-centos-01
[ceph_deploy.new][DEBUG ] Monitor gary-centos-01 at 10.214.140.136
[ceph_deploy.new][DEBUG ] Monitor initial members are ['gary-centos-01']
[ceph_deploy.new][DEBUG ] Monitor addrs are ['10.214.140.136']
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
[glowell@gary-centos-01 test]$
[glowell@gary-centos-01 test]$
[glowell@gary-centos-01 test]$ ceph-deploy mon create gary-centos-01
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts gary-centos-01
[ceph_deploy.mon][DEBUG ] detecting platform for host gary-centos-01 ...
[ceph_deploy.mon][INFO ] distro info: CentOS 6.3 Final
[gary-centos-01][DEBUG ] deploying mon to gary-centos-01
[gary-centos-01][DEBUG ] remote hostname: gary-centos-01
[gary-centos-01][INFO ] write cluster configuration to /etc/ceph/{cluster}.conf
[gary-centos-01][INFO ] creating path: /var/lib/ceph/mon/ceph-gary-centos-01
[gary-centos-01][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-gary-centos-01/done
[gary-centos-01][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-gary-centos-01/done
[gary-centos-01][INFO ] creating keyring file: /var/lib/ceph/tmp/ceph-gary-centos-01.mon.keyring
[gary-centos-01][INFO ] create the monitor keyring file
[gary-centos-01][INFO ] Running command: ceph-mon --cluster ceph --mkfs -i gary-centos-01 --keyring /var/lib/ceph/tmp/ceph-gary-centos-01.mon.keyring
[gary-centos-01][INFO ] ceph-mon: mon.noname-a 10.214.140.136:6789/0 is local, renaming to mon.gary-centos-01
[gary-centos-01][INFO ] ceph-mon: set fsid to 16efbd8e-190c-4782-a1c8-de122600b710
[gary-centos-01][INFO ] ceph-mon: created monfs at /var/lib/ceph/mon/ceph-gary-centos-01 for mon.gary-centos-01
[gary-centos-01][INFO ] unlinking keyring file /var/lib/ceph/tmp/ceph-gary-centos-01.mon.keyring
[gary-centos-01][INFO ] create a done file to avoid re-doing the mon deployment
[gary-centos-01][INFO ] create the init path if it does not exist
[gary-centos-01][INFO ] locating `service` executable...
[gary-centos-01][INFO ] found `service` executable: /sbin/service
[gary-centos-01][INFO ] Running command: /sbin/service ceph start mon.gary-centos-01
[glowell@gary-centos-01 test]$
[glowell@gary-centos-01 test]$ ceph-deploy gatherkeys gary-centos-01
[ceph_deploy.gatherkeys][DEBUG ] Checking gary-centos-01 for /etc/ceph/ceph.client.admin.keyring
[ceph_deploy.gatherkeys][DEBUG ] Got ceph.client.admin.keyring key from gary-centos-01.
[ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring
[ceph_deploy.gatherkeys][DEBUG ] Checking gary-centos-01 for /var/lib/ceph/bootstrap-osd/ceph.keyring
[ceph_deploy.gatherkeys][DEBUG ] Got ceph.bootstrap-osd.keyring key from gary-centos-01.
[ceph_deploy.gatherkeys][DEBUG ] Checking gary-centos-01 for /var/lib/ceph/bootstrap-mds/ceph.keyring
[ceph_deploy.gatherkeys][DEBUG ] Got ceph.bootstrap-mds.keyring key from gary-centos-01.
[glowell@gary-centos-01 test]$
[glowell@gary-centos-01 test]$ yum list installed leveldb
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.kernel.org
* epel: mirrors.xmission.com
* extras: mirrors.syringanetworks.net
* updates: centos-distro.cavecreek.net
Installed Packages
leveldb.x86_64 1.7.0-2.el6 @epel
[glowell@gary-centos-01 test]$
[glowell@gary-centos-01 test]$ yum list installed ceph
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.kernel.org
* epel: mirrors.xmission.com
* extras: mirrors.syringanetworks.net
* updates: centos-distro.cavecreek.net
Installed Packages
ceph.x86_64 0.67.1-0.el6 @ceph
[glowell@gary-centos-01 test]$
#9 Updated by Anonymous over 10 years ago
Rebuilding leveldb-1.12 without the Basho patch seems to work ok. This patch is described as:
- Cherry-picked from Basho's fork
Patch6: leveldb-0006-allow-Get-calls-to-avoid-copies-into-std-string.patch
[glowell@gary-centos-01 test]$ ceph-deploy new gary-centos-01
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][DEBUG ] Resolving host gary-centos-01
[ceph_deploy.new][DEBUG ] Monitor gary-centos-01 at 10.214.140.136
[ceph_deploy.new][DEBUG ] Monitor initial members are ['gary-centos-01']
[ceph_deploy.new][DEBUG ] Monitor addrs are ['10.214.140.136']
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
[glowell@gary-centos-01 test]$
[glowell@gary-centos-01 test]$ ceph-deploy mon create gary-centos-01
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts gary-centos-01
[ceph_deploy.mon][DEBUG ] detecting platform for host gary-centos-01 ...
[ceph_deploy.mon][INFO ] distro info: CentOS 6.3 Final
[gary-centos-01][DEBUG ] deploying mon to gary-centos-01
[gary-centos-01][DEBUG ] remote hostname: gary-centos-01
[gary-centos-01][INFO ] write cluster configuration to /etc/ceph/{cluster}.conf
[gary-centos-01][INFO ] creating path: /var/lib/ceph/mon/ceph-gary-centos-01
[gary-centos-01][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-gary-centos-01/done
[gary-centos-01][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-gary-centos-01/done
[gary-centos-01][INFO ] creating keyring file: /var/lib/ceph/tmp/ceph-gary-centos-01.mon.keyring
[gary-centos-01][INFO ] create the monitor keyring file
[gary-centos-01][INFO ] Running command: ceph-mon --cluster ceph --mkfs -i gary-centos-01 --keyring /var/lib/ceph/tmp/ceph-gary-centos-01.mon.keyring
[gary-centos-01][INFO ] ceph-mon: mon.noname-a 10.214.140.136:6789/0 is local, renaming to mon.gary-centos-01
[gary-centos-01][INFO ] ceph-mon: set fsid to 68d35035-6547-4544-b5ce-e4cb62c3af7c
[gary-centos-01][INFO ] ceph-mon: created monfs at /var/lib/ceph/mon/ceph-gary-centos-01 for mon.gary-centos-01
[gary-centos-01][INFO ] unlinking keyring file /var/lib/ceph/tmp/ceph-gary-centos-01.mon.keyring
[gary-centos-01][INFO ] create a done file to avoid re-doing the mon deployment
[gary-centos-01][INFO ] create the init path if it does not exist
[gary-centos-01][INFO ] locating `service` executable...
[gary-centos-01][INFO ] found `service` executable: /sbin/service
[gary-centos-01][INFO ] Running command: /sbin/service ceph start mon.gary-centos-01
[glowell@gary-centos-01 test]$
#10 Updated by Sage Weil over 10 years ago
- Priority changed from Immediate to Urgent
#11 Updated by Anonymous over 10 years ago
- Status changed from New to Resolved
Deleting leveldb-1.12 from the ceph-extras repo, and from the local mirror used by teuthology falls back to the leveldb-1.7.0 version from the epel repository. This version does not hang in mon create.
Reopening bug #5847 for the the leveldb port. We'll need to create new backport for the rpm platforms.