Bug #52320
Updated by Sebastian Wagner over 2 years ago
<pre> 2021-08-17T04:38:36.144 DEBUG:teuthology.orchestra.run.smithi012:> kubectl -n rook-ceph exec rook-ceph-tools-744c55f865-h6tqq -- ceph osd stat -f json 2021-08-17T04:38:46.606 INFO:tasks.rook.operator.smithi012.stdout:2021-08-17 04:38:46.609719 I | op-osd: OSD orchestration status for PVC scratch-data-9n2r7x is "completed" 2021-08-17T04:38:46.607 INFO:tasks.rook.operator.smithi012.stdout:2021-08-17 04:38:46.609758 I | op-osd: creating OSD 11 on PVC "scratch-data-9n2r7x" 2021-08-17T04:38:46.607 INFO:tasks.rook.operator.smithi012.stdout:2021-08-17 04:38:46.609775 I | op-osd: OSD will have its main bluestore block on "scratch-data-9n2r7x" 2021-08-17T04:38:46.808 INFO:tasks.rook.operator.smithi012.stdout:2021-08-17 04:38:46.791084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph osds: 3 failures encountered while running osds on nodes in namespace "rook-ceph". 2021-08-17T04:38:46.809 INFO:tasks.rook.operator.smithi012.stdout:failed to provision OSD(s) on PVC scratch-data-8bsdlq. &{OSDs:[] Status:failed PvcBackedOSD:true Message:failed to configure devices: failed to initialize devices on PVC: failed to run ceph-volume. Running command: /usr/bin/ceph-authtool --gen-print-key 2021-08-17T04:38:46.809 INFO:tasks.rook.operator.smithi012.stdout:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e0592531-b099-495e-9797-bfe8fa64eb5a 2021-08-17T04:38:46.809 INFO:tasks.rook.operator.smithi012.stdout: stderr: unable to get monitor info from DNS SRV with service name: ceph-mon 2021-08-17T04:38:46.809 INFO:tasks.rook.operator.smithi012.stdout: stderr: 2021-08-17T03:56:39.559+0000 7f945bcf3700 -1 failed for service _ceph-mon._tcp 2021-08-17T04:38:46.810 INFO:tasks.rook.operator.smithi012.stdout: stderr: 2021-08-17T03:56:39.559+0000 7f945bcf3700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2021-08-17T04:38:46.810 INFO:tasks.rook.operator.smithi012.stdout: stderr: [errno 2] RADOS object not found (error connecting to the cluster) 2021-08-17T04:38:46.810 INFO:tasks.rook.operator.smithi012.stdout:Traceback (most recent call last): 2021-08-17T04:38:46.810 INFO:tasks.rook.operator.smithi012.stdout: File "/usr/sbin/ceph-volume", line 11, in <module> 2021-08-17T04:38:46.811 INFO:tasks.rook.operator.smithi012.stdout: load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2021-08-17T04:38:46.811 INFO:tasks.rook.operator.smithi012.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 40, in __init__ 2021-08-17T04:38:46.811 INFO:tasks.rook.operator.smithi012.stdout: self.main(self.argv) 2021-08-17T04:38:46.811 INFO:tasks.rook.operator.smithi012.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2021-08-17T04:38:46.811 INFO:tasks.rook.operator.smithi012.stdout: return f(*a, **kw) 2021-08-17T04:38:46.812 INFO:tasks.rook.operator.smithi012.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 152, in main 2021-08-17T04:38:46.812 INFO:tasks.rook.operator.smithi012.stdout: terminal.dispatch(self.mapper, subcommand_args) 2021-08-17T04:38:46.812 INFO:tasks.rook.operator.smithi012.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2021-08-17T04:38:46.812 INFO:tasks.rook.operator.smithi012.stdout: instance.main() 2021-08-17T04:38:46.813 INFO:tasks.rook.operator.smithi012.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main 2021-08-17T04:38:46.813 INFO:tasks.rook.operator.smithi012.stdout: terminal.dispatch(self.mapper, self.argv) 2021-08-17T04:38:46.813 INFO:tasks.rook.operator.smithi012.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2021-08-17T04:38:46.813 INFO:tasks.rook.operator.smithi012.stdout: instance.main() 2021-08-17T04:38:46.813 INFO:tasks.rook.operator.smithi012.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main 2021-08-17T04:38:46.814 INFO:tasks.rook.operator.smithi012.stdout: self.safe_prepare(self.args) 2021-08-17T04:38:46.814 INFO:tasks.rook.operator.smithi012.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare 2021-08-17T04:38:46.814 INFO:tasks.rook.operator.smithi012.stdout: self.prepare() 2021-08-17T04:38:46.814 INFO:tasks.rook.operator.smithi012.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2021-08-17T04:38:46.815 INFO:tasks.rook.operator.smithi012.stdout: return func(*a, **kw) 2021-08-17T04:38:46.815 INFO:tasks.rook.operator.smithi012.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare 2021-08-17T04:38:46.815 INFO:tasks.rook.operator.smithi012.stdout: osd_fsid, json.dumps(secrets)) 2021-08-17T04:38:46.815 INFO:tasks.rook.operator.smithi012.stdout: File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id 2021-08-17T04:38:46.816 INFO:tasks.rook.operator.smithi012.stdout: raise RuntimeError('Unable to create a new OSD id') 2021-08-17T04:38:46.816 INFO:tasks.rook.operator.smithi012.stdout:RuntimeError: Unable to create a new OSD id. debug logs below: </pre> https://pulpito.ceph.com/yuriw-2021-08-16_21:15:00-rados-wip-yuri-testing-master-8.16.21-distro-basic-smithi/6342214 - hits max job timeout