Ceph » Orchestrator

2021-08-17T04:38:36.144 DEBUG:teuthology.orchestra.run.smithi012:> kubectl -n rook-ceph exec rook-ceph-tools-744c55f865-h6tqq -- ceph osd stat -f json
2021-08-17T04:38:46.606 INFO:tasks.rook.operator.smithi012.stdout:2021-08-17 04:38:46.609719 I | op-osd: OSD orchestration status for PVC scratch-data-9n2r7x is "completed" 
2021-08-17T04:38:46.607 INFO:tasks.rook.operator.smithi012.stdout:2021-08-17 04:38:46.609758 I | op-osd: creating OSD 11 on PVC "scratch-data-9n2r7x" 
2021-08-17T04:38:46.607 INFO:tasks.rook.operator.smithi012.stdout:2021-08-17 04:38:46.609775 I | op-osd: OSD will have its main bluestore block on "scratch-data-9n2r7x" 
2021-08-17T04:38:46.808 INFO:tasks.rook.operator.smithi012.stdout:2021-08-17 04:38:46.791084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph osds: 3 failures encountered while running osds on nodes in namespace "rook-ceph".
2021-08-17T04:38:46.809 INFO:tasks.rook.operator.smithi012.stdout:failed to provision OSD(s) on PVC scratch-data-8bsdlq. &{OSDs:[] Status:failed PvcBackedOSD:true Message:failed to configure devices: failed to initialize devices on PVC: failed to run ceph-volume. Running command: /usr/bin/ceph-authtool --gen-print-key
2021-08-17T04:38:46.809 INFO:tasks.rook.operator.smithi012.stdout:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e0592531-b099-495e-9797-bfe8fa64eb5a
2021-08-17T04:38:46.809 INFO:tasks.rook.operator.smithi012.stdout: stderr: unable to get monitor info from DNS SRV with service name: ceph-mon
2021-08-17T04:38:46.809 INFO:tasks.rook.operator.smithi012.stdout: stderr: 2021-08-17T03:56:39.559+0000 7f945bcf3700 -1 failed for service _ceph-mon._tcp
2021-08-17T04:38:46.810 INFO:tasks.rook.operator.smithi012.stdout: stderr: 2021-08-17T03:56:39.559+0000 7f945bcf3700 -1 monclient: get_monmap_and_config cannot identify monitors to contact
2021-08-17T04:38:46.810 INFO:tasks.rook.operator.smithi012.stdout: stderr: [errno 2] RADOS object not found (error connecting to the cluster)
2021-08-17T04:38:46.810 INFO:tasks.rook.operator.smithi012.stdout:Traceback (most recent call last):
2021-08-17T04:38:46.810 INFO:tasks.rook.operator.smithi012.stdout:  File "/usr/sbin/ceph-volume", line 11, in <module>
2021-08-17T04:38:46.811 INFO:tasks.rook.operator.smithi012.stdout:    load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()
2021-08-17T04:38:46.811 INFO:tasks.rook.operator.smithi012.stdout:  File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 40, in __init__
2021-08-17T04:38:46.811 INFO:tasks.rook.operator.smithi012.stdout:    self.main(self.argv)
2021-08-17T04:38:46.811 INFO:tasks.rook.operator.smithi012.stdout:  File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
2021-08-17T04:38:46.811 INFO:tasks.rook.operator.smithi012.stdout:    return f(*a, **kw)
2021-08-17T04:38:46.812 INFO:tasks.rook.operator.smithi012.stdout:  File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 152, in main
2021-08-17T04:38:46.812 INFO:tasks.rook.operator.smithi012.stdout:    terminal.dispatch(self.mapper, subcommand_args)
2021-08-17T04:38:46.812 INFO:tasks.rook.operator.smithi012.stdout:  File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
2021-08-17T04:38:46.812 INFO:tasks.rook.operator.smithi012.stdout:    instance.main()
2021-08-17T04:38:46.813 INFO:tasks.rook.operator.smithi012.stdout:  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main
2021-08-17T04:38:46.813 INFO:tasks.rook.operator.smithi012.stdout:    terminal.dispatch(self.mapper, self.argv)
2021-08-17T04:38:46.813 INFO:tasks.rook.operator.smithi012.stdout:  File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
2021-08-17T04:38:46.813 INFO:tasks.rook.operator.smithi012.stdout:    instance.main()
2021-08-17T04:38:46.813 INFO:tasks.rook.operator.smithi012.stdout:  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main
2021-08-17T04:38:46.814 INFO:tasks.rook.operator.smithi012.stdout:    self.safe_prepare(self.args)
2021-08-17T04:38:46.814 INFO:tasks.rook.operator.smithi012.stdout:  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare
2021-08-17T04:38:46.814 INFO:tasks.rook.operator.smithi012.stdout:    self.prepare()
2021-08-17T04:38:46.814 INFO:tasks.rook.operator.smithi012.stdout:  File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
2021-08-17T04:38:46.815 INFO:tasks.rook.operator.smithi012.stdout:    return func(*a, **kw)
2021-08-17T04:38:46.815 INFO:tasks.rook.operator.smithi012.stdout:  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 125, in prepare
2021-08-17T04:38:46.815 INFO:tasks.rook.operator.smithi012.stdout:    osd_fsid, json.dumps(secrets))
2021-08-17T04:38:46.815 INFO:tasks.rook.operator.smithi012.stdout:  File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 173, in create_id
2021-08-17T04:38:46.816 INFO:tasks.rook.operator.smithi012.stdout:    raise RuntimeError('Unable to create a new OSD id')
2021-08-17T04:38:46.816 INFO:tasks.rook.operator.smithi012.stdout:RuntimeError: Unable to create a new OSD id. debug logs below:

https://pulpito.ceph.com/yuriw-2021-08-16_21:15:00-rados-wip-yuri-testing-master-8.16.21-distro-basic-smithi/6342214 - hits max job timeout