Ceph » Orchestrator

/a/yuriw-2021-12-17_22:45:37-rados-wip-yuri10-testing-2021-12-17-1119-distro-default-smithi/6569344/

2021-12-18T01:35:01.041 INFO:teuthology.orchestra.run.smithi185.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1b23219043771c7aeaf383e73d414c625d03fae66d69ca7172f26ede96eefd1d", "container_image_name": "quay.ceph.io/ceph-ci/ceph:91fdab49fed87aa0a3dbbceccc27e84ab4f80130", "created": "2021-12-18T01:11:19.000000Z", "last_refresh": "2021-12-18T01:35:00.947402Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1b23219043771c7aeaf383e73d414c625d03fae66d69ca7172f26ede96eefd1d", "container_image_name": "quay.ceph.io/ceph-ci/ceph:91fdab49fed87aa0a3dbbceccc27e84ab4f80130", "created": "2021-12-18T01:02:05.000000Z", "last_refresh": "2021-12-18T01:35:00.947402Z", "running": 1, "size": 1}}, {"placement": {"count": 1}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1b23219043771c7aeaf383e73d414c625d03fae66d69ca7172f26ede96eefd1d", "container_image_name": "quay.ceph.io/ceph-ci/ceph:91fdab49fed87aa0a3dbbceccc27e84ab4f80130", "created": "2021-12-18T01:01:38.000000Z", "last_refresh": "2021-12-18T01:35:00.947402Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1b23219043771c7aeaf383e73d414c625d03fae66d69ca7172f26ede96eefd1d", "container_image_name": "quay.ceph.io/ceph-ci/ceph:91fdab49fed87aa0a3dbbceccc27e84ab4f80130", "created": "2021-12-18T01:03:41.000000Z", "last_refresh": "2021-12-18T01:35:00.947402Z", "running": 8, "size": 4}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2021-12-18T01:35:00.947402Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_id": "1b23219043771c7aeaf383e73d414c625d03fae66d69ca7172f26ede96eefd1d", "container_image_name": "quay.ceph.io/ceph-ci/ceph:91fdab49fed87aa0a3dbbceccc27e84ab4f80130", "created": "2021-12-18T01:13:57.000000Z", "last_refresh": "2021-12-18T01:35:00.947402Z", "running": 1, "size": 1}}]
2021-12-18T01:35:01.064 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/teuthology/contextutil.py", line 33, in nested
    yield vars
  File "/home/teuthworker/src/github.com_ceph_ceph-c_91fdab49fed87aa0a3dbbceccc27e84ab4f80130/qa/tasks/rook.py", line 669, in task
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/teuthology/contextutil.py", line 133, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds
2021-12-18T01:35:01.065 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds
Traceback (most recent call last):
  File "/home/teuthworker/src/github.com_ceph_ceph-c_91fdab49fed87aa0a3dbbceccc27e84ab4f80130/qa/tasks/rook.py", line 530, in ceph_config_keyring
    yield
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/teuthology/contextutil.py", line 33, in nested
    yield vars
  File "/home/teuthworker/src/github.com_ceph_ceph-c_91fdab49fed87aa0a3dbbceccc27e84ab4f80130/qa/tasks/rook.py", line 669, in task
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/teuthology/contextutil.py", line 133, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds
2021-12-18T01:35:01.065 INFO:tasks.rook:Cleaning up config and client.admin keyring
2021-12-18T01:35:01.066 DEBUG:teuthology.orchestra.run.smithi185:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring
2021-12-18T01:35:01.079 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds
Traceback (most recent call last):
  File "/home/teuthworker/src/github.com_ceph_ceph-c_91fdab49fed87aa0a3dbbceccc27e84ab4f80130/qa/tasks/rook.py", line 478, in rook_post_config
    yield
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/teuthology/contextutil.py", line 33, in nested
    yield vars
  File "/home/teuthworker/src/github.com_ceph_ceph-c_91fdab49fed87aa0a3dbbceccc27e84ab4f80130/qa/tasks/rook.py", line 669, in task
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/teuthology/contextutil.py", line 133, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds
2021-12-18T01:35:01.080 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds
Traceback (most recent call last):
  File "/home/teuthworker/src/github.com_ceph_ceph-c_91fdab49fed87aa0a3dbbceccc27e84ab4f80130/qa/tasks/rook.py", line 442, in rook_toolbox
    yield
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/teuthology/contextutil.py", line 33, in nested
    yield vars
  File "/home/teuthworker/src/github.com_ceph_ceph-c_91fdab49fed87aa0a3dbbceccc27e84ab4f80130/qa/tasks/rook.py", line 669, in task
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/teuthology/contextutil.py", line 133, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds
2021-12-18T01:35:01.141 DEBUG:teuthology.orchestra.remote:smithi185:rook/cluster/examples/kubernetes/ceph/operator.yaml is 22KB
2021-12-18T01:35:01.191 DEBUG:teuthology.orchestra.run.smithi185:> kubectl delete -f rook/cluster/examples/kubernetes/ceph/toolbox.yaml
2021-12-18T01:35:01.255 INFO:teuthology.orchestra.run.smithi185.stdout:deployment.apps "rook-ceph-tools" deleted
2021-12-18T01:35:01.287 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds
Traceback (most recent call last):
  File "/home/teuthworker/src/github.com_ceph_ceph-c_91fdab49fed87aa0a3dbbceccc27e84ab4f80130/qa/tasks/rook.py", line 379, in rook_cluster
    yield
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/teuthology/contextutil.py", line 33, in nested
    yield vars
  File "/home/teuthworker/src/github.com_ceph_ceph-c_91fdab49fed87aa0a3dbbceccc27e84ab4f80130/qa/tasks/rook.py", line 669, in task
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/teuthology/contextutil.py", line 133, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds
2021-12-18T01:35:01.288 DEBUG:teuthology.orchestra.run.smithi185:> kubectl delete -f cluster.yaml
2021-12-18T01:35:01.380 INFO:teuthology.orchestra.run.smithi185.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted
2021-12-18T01:35:01.383 INFO:tasks.rook.operator.smithi185.stdout:2021-12-18 01:35:01.383570 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration
2021-12-18T01:35:01.685 INFO:tasks.rook.operator.smithi185.stdout:2021-12-18 01:35:01.685102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo]
2021-12-18T01:35:01.696 INFO:tasks.rook.operator.smithi185.stdout:2021-12-18 01:35:01.696075 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo]
2021-12-18T01:35:01.696 INFO:tasks.rook.operator.smithi185.stdout:2021-12-18 01:35:01.696111 I | op-k8sutil: Reporting Event rook-ceph:rook-ceph Warning:ReconcileFailed:CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStores: [foo]