|
2021-09-09T17:07:30.160 INFO:root:teuthology version: 1.1.0-df69d6db
|
|
2021-09-09T17:07:30.254 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba001.front.sepia.ceph.com' reimaging
|
|
2021-09-09T17:07:30.255 INFO:teuthology.lock.ops:Updating [ubuntu@gibba001.front.sepia.ceph.com]: reset os type and version on server
|
|
2021-09-09T17:07:30.255 INFO:teuthology.lock.ops:Updating gibba001.front.sepia.ceph.com on lock server
|
|
2021-09-09T17:07:30.269 INFO:teuthology.lock.ops:Node 'ubuntu@gibba001.front.sepia.ceph.com' reimaging is complete
|
|
2021-09-09T17:07:30.270 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba041.front.sepia.ceph.com' reimaging
|
|
2021-09-09T17:07:30.270 INFO:teuthology.lock.ops:Updating [ubuntu@gibba041.front.sepia.ceph.com]: reset os type and version on server
|
|
2021-09-09T17:07:30.270 INFO:teuthology.lock.ops:Updating gibba041.front.sepia.ceph.com on lock server
|
|
2021-09-09T17:07:30.285 INFO:teuthology.lock.ops:Node 'ubuntu@gibba041.front.sepia.ceph.com' reimaging is complete
|
|
2021-09-09T17:07:30.488 INFO:teuthology.provision.fog.gibba001:Scheduling deploy of centos 8.2
|
|
2021-09-09T17:07:30.494 INFO:teuthology.provision.fog.gibba041:Scheduling deploy of centos 8.2
|
|
2021-09-09T17:07:30.829 INFO:teuthology.orchestra.console:Power off gibba001
|
|
2021-09-09T17:07:30.843 INFO:teuthology.orchestra.console:Power off gibba041
|
|
2021-09-09T17:07:30.979 INFO:teuthology.orchestra.console:Power off for gibba001 completed
|
|
2021-09-09T17:07:30.988 INFO:teuthology.orchestra.console:Power off for gibba041 completed
|
|
2021-09-09T17:07:31.080 INFO:teuthology.orchestra.console:Power on gibba001
|
|
2021-09-09T17:07:31.089 INFO:teuthology.orchestra.console:Power on gibba041
|
|
2021-09-09T17:07:35.453 INFO:teuthology.orchestra.console:Power on for gibba041 completed
|
|
2021-09-09T17:07:35.554 INFO:teuthology.provision.fog.gibba041:Waiting for deploy to finish
|
|
2021-09-09T17:07:39.466 INFO:teuthology.orchestra.console:Power on for gibba001 completed
|
|
2021-09-09T17:07:39.568 INFO:teuthology.provision.fog.gibba001:Waiting for deploy to finish
|
|
2021-09-09T17:11:36.668 INFO:teuthology.orchestra.run:Running command with timeout 600
|
|
2021-09-09T17:11:47.217 INFO:teuthology.provision.fog.gibba001:Node is ready
|
|
2021-09-09T17:11:47.230 INFO:teuthology.orchestra.run.gibba001.stdout:gibba031
|
|
2021-09-09T17:11:47.282 INFO:teuthology.orchestra.run.gibba001.stdout:172.21.2.131 gibba031.front.sepia.ceph.com gibba031
|
|
2021-09-09T17:11:47.571 INFO:teuthology.provision.fog.gibba001:Deploy complete!
|
|
2021-09-09T17:12:07.405 INFO:teuthology.orchestra.run:Running command with timeout 600
|
|
2021-09-09T17:12:07.643 INFO:teuthology.provision.fog.gibba041:Node is ready
|
|
2021-09-09T17:12:07.653 INFO:teuthology.orchestra.run.gibba041.stdout:gibba041.front.sepia.ceph.com
|
|
2021-09-09T17:12:07.706 INFO:teuthology.orchestra.run.gibba041.stdout:172.21.2.141 gibba041.front.sepia.ceph.com gibba041
|
|
2021-09-09T17:12:07.989 INFO:teuthology.provision.fog.gibba041:Deploy complete!
|
|
2021-09-09T17:12:08.115 INFO:teuthology.lock.ops:Checking gibba001.front.sepia.ceph.com
|
|
2021-09-09T17:12:08.116 INFO:teuthology.lock.ops:New key found. Updating...
|
|
2021-09-09T17:12:08.132 INFO:teuthology.lock.ops:Checking gibba041.front.sepia.ceph.com
|
|
2021-09-09T17:12:08.132 INFO:teuthology.lock.ops:New key found. Updating...
|
|
2021-09-09T17:12:08.152 INFO:teuthology.lock.ops:Updating [gibba001.front.sepia.ceph.com]: set os type and version on server
|
|
2021-09-09T17:12:08.153 INFO:teuthology.orchestra.remote:Trying to reconnect to host
|
|
2021-09-09T17:12:08.381 INFO:teuthology.orchestra.run.gibba001.stdout:x86_64
|
|
2021-09-09T17:12:08.433 INFO:teuthology.orchestra.run.gibba001.stdout:NAME="CentOS Linux"
|
|
2021-09-09T17:12:08.434 INFO:teuthology.orchestra.run.gibba001.stdout:VERSION="8 (Core)"
|
|
2021-09-09T17:12:08.434 INFO:teuthology.orchestra.run.gibba001.stdout:ID="centos"
|
|
2021-09-09T17:12:08.434 INFO:teuthology.orchestra.run.gibba001.stdout:ID_LIKE="rhel fedora"
|
|
2021-09-09T17:12:08.435 INFO:teuthology.orchestra.run.gibba001.stdout:VERSION_ID="8"
|
|
2021-09-09T17:12:08.435 INFO:teuthology.orchestra.run.gibba001.stdout:PLATFORM_ID="platform:el8"
|
|
2021-09-09T17:12:08.435 INFO:teuthology.orchestra.run.gibba001.stdout:PRETTY_NAME="CentOS Linux 8 (Core)"
|
|
2021-09-09T17:12:08.436 INFO:teuthology.orchestra.run.gibba001.stdout:ANSI_COLOR="0;31"
|
|
2021-09-09T17:12:08.436 INFO:teuthology.orchestra.run.gibba001.stdout:CPE_NAME="cpe:/o:centos:centos:8"
|
|
2021-09-09T17:12:08.436 INFO:teuthology.orchestra.run.gibba001.stdout:HOME_URL="https://www.centos.org/"
|
|
2021-09-09T17:12:08.436 INFO:teuthology.orchestra.run.gibba001.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
|
|
2021-09-09T17:12:08.436 INFO:teuthology.orchestra.run.gibba001.stdout:
|
|
2021-09-09T17:12:08.437 INFO:teuthology.orchestra.run.gibba001.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
|
|
2021-09-09T17:12:08.437 INFO:teuthology.orchestra.run.gibba001.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
|
|
2021-09-09T17:12:08.437 INFO:teuthology.orchestra.run.gibba001.stdout:REDHAT_SUPPORT_PRODUCT="centos"
|
|
2021-09-09T17:12:08.437 INFO:teuthology.orchestra.run.gibba001.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8"
|
|
2021-09-09T17:12:08.437 INFO:teuthology.orchestra.run.gibba001.stdout:
|
|
2021-09-09T17:12:08.439 INFO:teuthology.lock.ops:Updating gibba001.front.sepia.ceph.com on lock server
|
|
2021-09-09T17:12:08.453 INFO:teuthology.lock.ops:Updating [gibba041.front.sepia.ceph.com]: set os type and version on server
|
|
2021-09-09T17:12:08.454 INFO:teuthology.orchestra.remote:Trying to reconnect to host
|
|
2021-09-09T17:12:08.675 INFO:teuthology.orchestra.run.gibba041.stdout:x86_64
|
|
2021-09-09T17:12:08.725 INFO:teuthology.orchestra.run.gibba041.stdout:NAME="CentOS Linux"
|
|
2021-09-09T17:12:08.726 INFO:teuthology.orchestra.run.gibba041.stdout:VERSION="8 (Core)"
|
|
2021-09-09T17:12:08.726 INFO:teuthology.orchestra.run.gibba041.stdout:ID="centos"
|
|
2021-09-09T17:12:08.726 INFO:teuthology.orchestra.run.gibba041.stdout:ID_LIKE="rhel fedora"
|
|
2021-09-09T17:12:08.726 INFO:teuthology.orchestra.run.gibba041.stdout:VERSION_ID="8"
|
|
2021-09-09T17:12:08.727 INFO:teuthology.orchestra.run.gibba041.stdout:PLATFORM_ID="platform:el8"
|
|
2021-09-09T17:12:08.727 INFO:teuthology.orchestra.run.gibba041.stdout:PRETTY_NAME="CentOS Linux 8 (Core)"
|
|
2021-09-09T17:12:08.727 INFO:teuthology.orchestra.run.gibba041.stdout:ANSI_COLOR="0;31"
|
|
2021-09-09T17:12:08.727 INFO:teuthology.orchestra.run.gibba041.stdout:CPE_NAME="cpe:/o:centos:centos:8"
|
|
2021-09-09T17:12:08.727 INFO:teuthology.orchestra.run.gibba041.stdout:HOME_URL="https://www.centos.org/"
|
|
2021-09-09T17:12:08.728 INFO:teuthology.orchestra.run.gibba041.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
|
|
2021-09-09T17:12:08.728 INFO:teuthology.orchestra.run.gibba041.stdout:
|
|
2021-09-09T17:12:08.728 INFO:teuthology.orchestra.run.gibba041.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
|
|
2021-09-09T17:12:08.728 INFO:teuthology.orchestra.run.gibba041.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
|
|
2021-09-09T17:12:08.728 INFO:teuthology.orchestra.run.gibba041.stdout:REDHAT_SUPPORT_PRODUCT="centos"
|
|
2021-09-09T17:12:08.729 INFO:teuthology.orchestra.run.gibba041.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8"
|
|
2021-09-09T17:12:08.729 INFO:teuthology.orchestra.run.gibba041.stdout:
|
|
2021-09-09T17:12:08.729 INFO:teuthology.lock.ops:Updating gibba041.front.sepia.ceph.com on lock server
|
|
2021-09-09T17:12:08.799 INFO:teuthology.dispatcher.supervisor:Running job 6369338
|
|
2021-09-09T17:12:08.799 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-09-01_03:30:03-rados-octopus-distro-basic-gibba/6369338 --name teuthology-2021-09-01_03:30:03-rados-octopus-distro-basic-gibba --description rados/cephadm/upgrade/{1-start-distro/1-start-centos_8 2-repo_digest/defaut 3-start-upgrade 4-wait fixed-2} -- /home/teuthworker/archive/teuthology-2021-09-01_03:30:03-rados-octopus-distro-basic-gibba/6369338/orig.config.yaml
|
|
2021-09-09T17:12:08.803 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-09-01_03:30:03-rados-octopus-distro-basic-gibba/6369338
|
|
2021-09-09T17:12:08.804 INFO:teuthology.dispatcher.supervisor:Job PID: 20266
|
|
2021-09-09T17:12:08.804 INFO:teuthology.dispatcher.supervisor:Running with watchdog
|
|
2021-09-10T05:12:53.550 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
|
|
2021-09-10T05:12:53.761 INFO:teuthology.kill:Killing Pids: {20266}
|
|
2021-09-10T05:12:54.855 INFO:teuthology.kill:Nuking machines: ['gibba001', 'gibba041']
|
|
2021-09-10T05:12:55.711 INFO:teuthology.kill:2021-09-10 05:12:55,711.711 INFO:teuthology.nuke:targets:
|
|
2021-09-10T05:12:55.712 INFO:teuthology.kill: gibba001.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD7Dwz5eZsjA4lRqJ5lAnlTHpnVPHYhTsrPLQy6CKb8Eoc5+gX05oyvpeKzNpGBsXw+R/hlp/Ti0e6teKxrTupL+HIDE6GwanV2J2iwcdWzr9iA2tXuJ4egbfrNpLDJbQnkX59D6Ia6uyEceS2N32CDP4vkwkJ4P64+K20+G3a63fiYWcYGxEGWJi3sWILuxS8rOPLy55GOY6GOntGV3sWluDsxxrT1JrZPG1tKM1JtT7vZT9ofe7N1Jro/FyUdW8vp9coRShM4RHIea/mj0QCmnohhoO+pS2ZnmOJNTAdMudsV/z3Ru8d/n+0sS5VjHwiC/hutTNZMaTkOFYSsIR8j+rSbeX3AICU22wSTFlyVVDaKByOt2qhMLxY9vRJVaxSKyFg0OWniK7igo/TxBtvMvVhn2Vpizb6w9H5Z74XExwxBtQx+TKEJncfGai7nkyR9Yhf0xOz+4489Tx/Ad8XzHKrY7B32KkaIu9e7XrJBW9grjwsWGrvuL5b7OrE5lss=
|
|
2021-09-10T05:12:55.713 INFO:teuthology.kill: gibba041.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcwXnlx05+pkQzUNij/F8XjjFK2w7q0rbnVzlOXaDW9fUbFHz+uWDlL+vR3LAZrqFLW7XXGazyyacjchn4+v9B8WlLT8pJysCnfukpQYNViL8tPU1+IK+mGG8pIzhKXVSEhsDHk9xvmwYBsq1JONz7Nw3hJrTCbEZD77XtA/JnOK1ksWwP1ZEOjZnQGmq4s7VPlUDb8HlH0dcd7IBXshWsTTDLk52y+CrJd/6VcgIeNDVsXXkzXwBwsKcQSvM20J3x0+ne8ETP6jFVa46eNXZUHcIwOyZ/6g1Sl0hasb6vOeiIwZafVte5H7an2GCrMbsGUkQ+nRizzH6fOZXRym3rlIDxUd8uKwCZpof9hrjkkhdmkb+cfwcdtTuPDEfZCaTKM8cBn7Wi46Enic0mdvR0AmnGGtwP47cCcLU4bPTqh54V3eSaFApNwnyWKY9iN/JSQ164GpbHCs26ixabDyNRUfGkdz44Qq9bOs0ZfFhghZ1YZD4gGTC7NshKsxChiE=
|
|
2021-09-10T05:12:55.713 INFO:teuthology.kill:2021-09-10 05:12:55,711.711 INFO:teuthology.task.internal.check_lock:Checking locks...
|
|
2021-09-10T05:12:55.715 INFO:teuthology.kill:2021-09-10 05:12:55,715.715 INFO:teuthology.task.internal.check_lock:Checking locks...
|
|
2021-09-10T05:12:55.775 INFO:teuthology.kill:2021-09-10 05:12:55,775.775 INFO:teuthology.orchestra.console:Power off gibba041
|
|
2021-09-10T05:12:55.788 INFO:teuthology.kill:2021-09-10 05:12:55,787.787 INFO:teuthology.orchestra.console:Power off gibba001
|
|
2021-09-10T05:13:08.280 INFO:teuthology.kill:2021-09-10 05:13:08,280.280 INFO:teuthology.orchestra.console:Power off for gibba001 completed
|
|
2021-09-10T05:13:08.285 INFO:teuthology.kill:2021-09-10 05:13:08,285.285 INFO:teuthology.orchestra.console:Power off for gibba041 completed
|
|
2021-09-10T05:13:08.474 INFO:teuthology.task.internal:roles: ubuntu@gibba001.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a']
|
|
2021-09-10T05:13:08.475 INFO:teuthology.task.internal:roles: ubuntu@gibba041.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b']
|
|
2021-09-10T05:13:08.475 INFO:teuthology.misc:Compressing logs...
|
|
2021-09-10T05:13:08.476 INFO:teuthology.orchestra.remote:Trying to reconnect to host
|
|
2021-09-10T05:14:08.524 ERROR:teuthology.dispatcher.supervisor:Could not save logs
|
|
Traceback (most recent call last):
|
|
File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog
|
|
teuth_config.archive_base, job_config)
|
|
File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives
|
|
compress_logs(ctx, log_path)
|
|
File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/misc.py", line 1378, in compress_logs
|
|
wait=False,
|
|
File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/orchestra/cluster.py", line 64, in run
|
|
return [remote.run(**kwargs) for remote in remotes]
|
|
File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/orchestra/cluster.py", line 64, in <listcomp>
|
|
return [remote.run(**kwargs) for remote in remotes]
|
|
File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/orchestra/remote.py", line 509, in run
|
|
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
|
|
File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/orchestra/run.py", line 434, in run
|
|
transport = client.get_transport()
|
|
AttributeError: 'NoneType' object has no attribute 'get_transport'
|
|
2021-09-10T05:14:08.562 INFO:teuthology.kill:No teuthology processes running
|
|
2021-09-10T05:14:09.539 INFO:teuthology.kill:Nuking machines: ['gibba001', 'gibba041']
|
|
2021-09-10T05:14:10.376 INFO:teuthology.kill:2021-09-10 05:14:10,375.375 INFO:teuthology.nuke:targets:
|
|
2021-09-10T05:14:10.376 INFO:teuthology.kill: gibba001.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD7Dwz5eZsjA4lRqJ5lAnlTHpnVPHYhTsrPLQy6CKb8Eoc5+gX05oyvpeKzNpGBsXw+R/hlp/Ti0e6teKxrTupL+HIDE6GwanV2J2iwcdWzr9iA2tXuJ4egbfrNpLDJbQnkX59D6Ia6uyEceS2N32CDP4vkwkJ4P64+K20+G3a63fiYWcYGxEGWJi3sWILuxS8rOPLy55GOY6GOntGV3sWluDsxxrT1JrZPG1tKM1JtT7vZT9ofe7N1Jro/FyUdW8vp9coRShM4RHIea/mj0QCmnohhoO+pS2ZnmOJNTAdMudsV/z3Ru8d/n+0sS5VjHwiC/hutTNZMaTkOFYSsIR8j+rSbeX3AICU22wSTFlyVVDaKByOt2qhMLxY9vRJVaxSKyFg0OWniK7igo/TxBtvMvVhn2Vpizb6w9H5Z74XExwxBtQx+TKEJncfGai7nkyR9Yhf0xOz+4489Tx/Ad8XzHKrY7B32KkaIu9e7XrJBW9grjwsWGrvuL5b7OrE5lss=
|
|
2021-09-10T05:14:10.377 INFO:teuthology.kill: gibba041.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcwXnlx05+pkQzUNij/F8XjjFK2w7q0rbnVzlOXaDW9fUbFHz+uWDlL+vR3LAZrqFLW7XXGazyyacjchn4+v9B8WlLT8pJysCnfukpQYNViL8tPU1+IK+mGG8pIzhKXVSEhsDHk9xvmwYBsq1JONz7Nw3hJrTCbEZD77XtA/JnOK1ksWwP1ZEOjZnQGmq4s7VPlUDb8HlH0dcd7IBXshWsTTDLk52y+CrJd/6VcgIeNDVsXXkzXwBwsKcQSvM20J3x0+ne8ETP6jFVa46eNXZUHcIwOyZ/6g1Sl0hasb6vOeiIwZafVte5H7an2GCrMbsGUkQ+nRizzH6fOZXRym3rlIDxUd8uKwCZpof9hrjkkhdmkb+cfwcdtTuPDEfZCaTKM8cBn7Wi46Enic0mdvR0AmnGGtwP47cCcLU4bPTqh54V3eSaFApNwnyWKY9iN/JSQ164GpbHCs26ixabDyNRUfGkdz44Qq9bOs0ZfFhghZ1YZD4gGTC7NshKsxChiE=
|
|
2021-09-10T05:14:10.396 INFO:teuthology.kill:2021-09-10 05:14:10,396.396 INFO:teuthology.task.internal.check_lock:Checking locks...
|
|
2021-09-10T05:14:10.400 INFO:teuthology.kill:2021-09-10 05:14:10,399.399 INFO:teuthology.task.internal.check_lock:Checking locks...
|
|
2021-09-10T05:14:10.456 INFO:teuthology.kill:2021-09-10 05:14:10,455.455 INFO:teuthology.orchestra.console:Power off gibba041
|
|
2021-09-10T05:14:10.466 INFO:teuthology.kill:2021-09-10 05:14:10,466.466 INFO:teuthology.orchestra.console:Power off gibba001
|
|
2021-09-10T05:14:10.602 INFO:teuthology.kill:2021-09-10 05:14:10,602.602 INFO:teuthology.orchestra.console:Power off for gibba041 completed
|
|
2021-09-10T05:14:10.611 INFO:teuthology.kill:2021-09-10 05:14:10,611.611 INFO:teuthology.orchestra.console:Power off for gibba001 completed
|
|
2021-09-10T05:14:10.741 INFO:teuthology.kill:2021-09-10 05:14:10,740.740 INFO:teuthology.lock.ops:unlocked gibba041.front.sepia.ceph.com
|
|
2021-09-10T05:14:10.747 INFO:teuthology.kill:2021-09-10 05:14:10,747.747 INFO:teuthology.lock.ops:unlocked gibba001.front.sepia.ceph.com
|
|
2021-09-10T05:16:10.954 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
|
|
2021-09-10T05:16:11.015 INFO:teuthology.dispatcher.supervisor:Nuking machines...
|
|
2021-09-10T05:16:11.016 INFO:teuthology.nuke:Checking targets against current locks
|
|
2021-09-10T05:16:11.093 INFO:teuthology.nuke:Not nuking gibba041.front.sepia.ceph.com because description doesn't match
|
|
2021-09-10T05:16:11.093 INFO:teuthology.nuke:Not nuking gibba001.front.sepia.ceph.com because description doesn't match
|