Project

General

Profile

supervisor.6369338.log

David Galloway, 09/10/2021 12:55 PM

Download (15.6 KB)

 
1
2021-09-09T17:07:30.160 INFO:root:teuthology version: 1.1.0-df69d6db
2
2021-09-09T17:07:30.254 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba001.front.sepia.ceph.com' reimaging
3
2021-09-09T17:07:30.255 INFO:teuthology.lock.ops:Updating [ubuntu@gibba001.front.sepia.ceph.com]: reset os type and version on server
4
2021-09-09T17:07:30.255 INFO:teuthology.lock.ops:Updating gibba001.front.sepia.ceph.com on lock server
5
2021-09-09T17:07:30.269 INFO:teuthology.lock.ops:Node 'ubuntu@gibba001.front.sepia.ceph.com' reimaging is complete
6
2021-09-09T17:07:30.270 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba041.front.sepia.ceph.com' reimaging
7
2021-09-09T17:07:30.270 INFO:teuthology.lock.ops:Updating [ubuntu@gibba041.front.sepia.ceph.com]: reset os type and version on server
8
2021-09-09T17:07:30.270 INFO:teuthology.lock.ops:Updating gibba041.front.sepia.ceph.com on lock server
9
2021-09-09T17:07:30.285 INFO:teuthology.lock.ops:Node 'ubuntu@gibba041.front.sepia.ceph.com' reimaging is complete
10
2021-09-09T17:07:30.488 INFO:teuthology.provision.fog.gibba001:Scheduling deploy of centos 8.2
11
2021-09-09T17:07:30.494 INFO:teuthology.provision.fog.gibba041:Scheduling deploy of centos 8.2
12
2021-09-09T17:07:30.829 INFO:teuthology.orchestra.console:Power off gibba001
13
2021-09-09T17:07:30.843 INFO:teuthology.orchestra.console:Power off gibba041
14
2021-09-09T17:07:30.979 INFO:teuthology.orchestra.console:Power off for gibba001 completed
15
2021-09-09T17:07:30.988 INFO:teuthology.orchestra.console:Power off for gibba041 completed
16
2021-09-09T17:07:31.080 INFO:teuthology.orchestra.console:Power on gibba001
17
2021-09-09T17:07:31.089 INFO:teuthology.orchestra.console:Power on gibba041
18
2021-09-09T17:07:35.453 INFO:teuthology.orchestra.console:Power on for gibba041 completed
19
2021-09-09T17:07:35.554 INFO:teuthology.provision.fog.gibba041:Waiting for deploy to finish
20
2021-09-09T17:07:39.466 INFO:teuthology.orchestra.console:Power on for gibba001 completed
21
2021-09-09T17:07:39.568 INFO:teuthology.provision.fog.gibba001:Waiting for deploy to finish
22
2021-09-09T17:11:36.668 INFO:teuthology.orchestra.run:Running command with timeout 600
23
2021-09-09T17:11:47.217 INFO:teuthology.provision.fog.gibba001:Node is ready
24
2021-09-09T17:11:47.230 INFO:teuthology.orchestra.run.gibba001.stdout:gibba031
25
2021-09-09T17:11:47.282 INFO:teuthology.orchestra.run.gibba001.stdout:172.21.2.131 gibba031.front.sepia.ceph.com gibba031
26
2021-09-09T17:11:47.571 INFO:teuthology.provision.fog.gibba001:Deploy complete!
27
2021-09-09T17:12:07.405 INFO:teuthology.orchestra.run:Running command with timeout 600
28
2021-09-09T17:12:07.643 INFO:teuthology.provision.fog.gibba041:Node is ready
29
2021-09-09T17:12:07.653 INFO:teuthology.orchestra.run.gibba041.stdout:gibba041.front.sepia.ceph.com
30
2021-09-09T17:12:07.706 INFO:teuthology.orchestra.run.gibba041.stdout:172.21.2.141 gibba041.front.sepia.ceph.com gibba041
31
2021-09-09T17:12:07.989 INFO:teuthology.provision.fog.gibba041:Deploy complete!
32
2021-09-09T17:12:08.115 INFO:teuthology.lock.ops:Checking gibba001.front.sepia.ceph.com
33
2021-09-09T17:12:08.116 INFO:teuthology.lock.ops:New key found. Updating...
34
2021-09-09T17:12:08.132 INFO:teuthology.lock.ops:Checking gibba041.front.sepia.ceph.com
35
2021-09-09T17:12:08.132 INFO:teuthology.lock.ops:New key found. Updating...
36
2021-09-09T17:12:08.152 INFO:teuthology.lock.ops:Updating [gibba001.front.sepia.ceph.com]: set os type and version on server
37
2021-09-09T17:12:08.153 INFO:teuthology.orchestra.remote:Trying to reconnect to host
38
2021-09-09T17:12:08.381 INFO:teuthology.orchestra.run.gibba001.stdout:x86_64
39
2021-09-09T17:12:08.433 INFO:teuthology.orchestra.run.gibba001.stdout:NAME="CentOS Linux"
40
2021-09-09T17:12:08.434 INFO:teuthology.orchestra.run.gibba001.stdout:VERSION="8 (Core)"
41
2021-09-09T17:12:08.434 INFO:teuthology.orchestra.run.gibba001.stdout:ID="centos"
42
2021-09-09T17:12:08.434 INFO:teuthology.orchestra.run.gibba001.stdout:ID_LIKE="rhel fedora"
43
2021-09-09T17:12:08.435 INFO:teuthology.orchestra.run.gibba001.stdout:VERSION_ID="8"
44
2021-09-09T17:12:08.435 INFO:teuthology.orchestra.run.gibba001.stdout:PLATFORM_ID="platform:el8"
45
2021-09-09T17:12:08.435 INFO:teuthology.orchestra.run.gibba001.stdout:PRETTY_NAME="CentOS Linux 8 (Core)"
46
2021-09-09T17:12:08.436 INFO:teuthology.orchestra.run.gibba001.stdout:ANSI_COLOR="0;31"
47
2021-09-09T17:12:08.436 INFO:teuthology.orchestra.run.gibba001.stdout:CPE_NAME="cpe:/o:centos:centos:8"
48
2021-09-09T17:12:08.436 INFO:teuthology.orchestra.run.gibba001.stdout:HOME_URL="https://www.centos.org/"
49
2021-09-09T17:12:08.436 INFO:teuthology.orchestra.run.gibba001.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
50
2021-09-09T17:12:08.436 INFO:teuthology.orchestra.run.gibba001.stdout:
51
2021-09-09T17:12:08.437 INFO:teuthology.orchestra.run.gibba001.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
52
2021-09-09T17:12:08.437 INFO:teuthology.orchestra.run.gibba001.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
53
2021-09-09T17:12:08.437 INFO:teuthology.orchestra.run.gibba001.stdout:REDHAT_SUPPORT_PRODUCT="centos"
54
2021-09-09T17:12:08.437 INFO:teuthology.orchestra.run.gibba001.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8"
55
2021-09-09T17:12:08.437 INFO:teuthology.orchestra.run.gibba001.stdout:
56
2021-09-09T17:12:08.439 INFO:teuthology.lock.ops:Updating gibba001.front.sepia.ceph.com on lock server
57
2021-09-09T17:12:08.453 INFO:teuthology.lock.ops:Updating [gibba041.front.sepia.ceph.com]: set os type and version on server
58
2021-09-09T17:12:08.454 INFO:teuthology.orchestra.remote:Trying to reconnect to host
59
2021-09-09T17:12:08.675 INFO:teuthology.orchestra.run.gibba041.stdout:x86_64
60
2021-09-09T17:12:08.725 INFO:teuthology.orchestra.run.gibba041.stdout:NAME="CentOS Linux"
61
2021-09-09T17:12:08.726 INFO:teuthology.orchestra.run.gibba041.stdout:VERSION="8 (Core)"
62
2021-09-09T17:12:08.726 INFO:teuthology.orchestra.run.gibba041.stdout:ID="centos"
63
2021-09-09T17:12:08.726 INFO:teuthology.orchestra.run.gibba041.stdout:ID_LIKE="rhel fedora"
64
2021-09-09T17:12:08.726 INFO:teuthology.orchestra.run.gibba041.stdout:VERSION_ID="8"
65
2021-09-09T17:12:08.727 INFO:teuthology.orchestra.run.gibba041.stdout:PLATFORM_ID="platform:el8"
66
2021-09-09T17:12:08.727 INFO:teuthology.orchestra.run.gibba041.stdout:PRETTY_NAME="CentOS Linux 8 (Core)"
67
2021-09-09T17:12:08.727 INFO:teuthology.orchestra.run.gibba041.stdout:ANSI_COLOR="0;31"
68
2021-09-09T17:12:08.727 INFO:teuthology.orchestra.run.gibba041.stdout:CPE_NAME="cpe:/o:centos:centos:8"
69
2021-09-09T17:12:08.727 INFO:teuthology.orchestra.run.gibba041.stdout:HOME_URL="https://www.centos.org/"
70
2021-09-09T17:12:08.728 INFO:teuthology.orchestra.run.gibba041.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
71
2021-09-09T17:12:08.728 INFO:teuthology.orchestra.run.gibba041.stdout:
72
2021-09-09T17:12:08.728 INFO:teuthology.orchestra.run.gibba041.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
73
2021-09-09T17:12:08.728 INFO:teuthology.orchestra.run.gibba041.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
74
2021-09-09T17:12:08.728 INFO:teuthology.orchestra.run.gibba041.stdout:REDHAT_SUPPORT_PRODUCT="centos"
75
2021-09-09T17:12:08.729 INFO:teuthology.orchestra.run.gibba041.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8"
76
2021-09-09T17:12:08.729 INFO:teuthology.orchestra.run.gibba041.stdout:
77
2021-09-09T17:12:08.729 INFO:teuthology.lock.ops:Updating gibba041.front.sepia.ceph.com on lock server
78
2021-09-09T17:12:08.799 INFO:teuthology.dispatcher.supervisor:Running job 6369338
79
2021-09-09T17:12:08.799 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-09-01_03:30:03-rados-octopus-distro-basic-gibba/6369338 --name teuthology-2021-09-01_03:30:03-rados-octopus-distro-basic-gibba --description rados/cephadm/upgrade/{1-start-distro/1-start-centos_8 2-repo_digest/defaut 3-start-upgrade 4-wait fixed-2} -- /home/teuthworker/archive/teuthology-2021-09-01_03:30:03-rados-octopus-distro-basic-gibba/6369338/orig.config.yaml
80
2021-09-09T17:12:08.803 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-09-01_03:30:03-rados-octopus-distro-basic-gibba/6369338
81
2021-09-09T17:12:08.804 INFO:teuthology.dispatcher.supervisor:Job PID: 20266
82
2021-09-09T17:12:08.804 INFO:teuthology.dispatcher.supervisor:Running with watchdog
83
2021-09-10T05:12:53.550 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
84
2021-09-10T05:12:53.761 INFO:teuthology.kill:Killing Pids: {20266}
85
2021-09-10T05:12:54.855 INFO:teuthology.kill:Nuking machines: ['gibba001', 'gibba041']
86
2021-09-10T05:12:55.711 INFO:teuthology.kill:2021-09-10 05:12:55,711.711 INFO:teuthology.nuke:targets:
87
2021-09-10T05:12:55.712 INFO:teuthology.kill:  gibba001.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD7Dwz5eZsjA4lRqJ5lAnlTHpnVPHYhTsrPLQy6CKb8Eoc5+gX05oyvpeKzNpGBsXw+R/hlp/Ti0e6teKxrTupL+HIDE6GwanV2J2iwcdWzr9iA2tXuJ4egbfrNpLDJbQnkX59D6Ia6uyEceS2N32CDP4vkwkJ4P64+K20+G3a63fiYWcYGxEGWJi3sWILuxS8rOPLy55GOY6GOntGV3sWluDsxxrT1JrZPG1tKM1JtT7vZT9ofe7N1Jro/FyUdW8vp9coRShM4RHIea/mj0QCmnohhoO+pS2ZnmOJNTAdMudsV/z3Ru8d/n+0sS5VjHwiC/hutTNZMaTkOFYSsIR8j+rSbeX3AICU22wSTFlyVVDaKByOt2qhMLxY9vRJVaxSKyFg0OWniK7igo/TxBtvMvVhn2Vpizb6w9H5Z74XExwxBtQx+TKEJncfGai7nkyR9Yhf0xOz+4489Tx/Ad8XzHKrY7B32KkaIu9e7XrJBW9grjwsWGrvuL5b7OrE5lss=
88
2021-09-10T05:12:55.713 INFO:teuthology.kill:  gibba041.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcwXnlx05+pkQzUNij/F8XjjFK2w7q0rbnVzlOXaDW9fUbFHz+uWDlL+vR3LAZrqFLW7XXGazyyacjchn4+v9B8WlLT8pJysCnfukpQYNViL8tPU1+IK+mGG8pIzhKXVSEhsDHk9xvmwYBsq1JONz7Nw3hJrTCbEZD77XtA/JnOK1ksWwP1ZEOjZnQGmq4s7VPlUDb8HlH0dcd7IBXshWsTTDLk52y+CrJd/6VcgIeNDVsXXkzXwBwsKcQSvM20J3x0+ne8ETP6jFVa46eNXZUHcIwOyZ/6g1Sl0hasb6vOeiIwZafVte5H7an2GCrMbsGUkQ+nRizzH6fOZXRym3rlIDxUd8uKwCZpof9hrjkkhdmkb+cfwcdtTuPDEfZCaTKM8cBn7Wi46Enic0mdvR0AmnGGtwP47cCcLU4bPTqh54V3eSaFApNwnyWKY9iN/JSQ164GpbHCs26ixabDyNRUfGkdz44Qq9bOs0ZfFhghZ1YZD4gGTC7NshKsxChiE=
89
2021-09-10T05:12:55.713 INFO:teuthology.kill:2021-09-10 05:12:55,711.711 INFO:teuthology.task.internal.check_lock:Checking locks...
90
2021-09-10T05:12:55.715 INFO:teuthology.kill:2021-09-10 05:12:55,715.715 INFO:teuthology.task.internal.check_lock:Checking locks...
91
2021-09-10T05:12:55.775 INFO:teuthology.kill:2021-09-10 05:12:55,775.775 INFO:teuthology.orchestra.console:Power off gibba041
92
2021-09-10T05:12:55.788 INFO:teuthology.kill:2021-09-10 05:12:55,787.787 INFO:teuthology.orchestra.console:Power off gibba001
93
2021-09-10T05:13:08.280 INFO:teuthology.kill:2021-09-10 05:13:08,280.280 INFO:teuthology.orchestra.console:Power off for gibba001 completed
94
2021-09-10T05:13:08.285 INFO:teuthology.kill:2021-09-10 05:13:08,285.285 INFO:teuthology.orchestra.console:Power off for gibba041 completed
95
2021-09-10T05:13:08.474 INFO:teuthology.task.internal:roles: ubuntu@gibba001.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a']
96
2021-09-10T05:13:08.475 INFO:teuthology.task.internal:roles: ubuntu@gibba041.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b']
97
2021-09-10T05:13:08.475 INFO:teuthology.misc:Compressing logs...
98
2021-09-10T05:13:08.476 INFO:teuthology.orchestra.remote:Trying to reconnect to host
99
2021-09-10T05:14:08.524 ERROR:teuthology.dispatcher.supervisor:Could not save logs
100
Traceback (most recent call last):
101
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog
102
    teuth_config.archive_base, job_config)
103
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives
104
    compress_logs(ctx, log_path)
105
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/misc.py", line 1378, in compress_logs
106
    wait=False,
107
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/orchestra/cluster.py", line 64, in run
108
    return [remote.run(**kwargs) for remote in remotes]
109
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/orchestra/cluster.py", line 64, in <listcomp>
110
    return [remote.run(**kwargs) for remote in remotes]
111
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/orchestra/remote.py", line 509, in run
112
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
113
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/orchestra/run.py", line 434, in run
114
    transport = client.get_transport()
115
AttributeError: 'NoneType' object has no attribute 'get_transport'
116
2021-09-10T05:14:08.562 INFO:teuthology.kill:No teuthology processes running
117
2021-09-10T05:14:09.539 INFO:teuthology.kill:Nuking machines: ['gibba001', 'gibba041']
118
2021-09-10T05:14:10.376 INFO:teuthology.kill:2021-09-10 05:14:10,375.375 INFO:teuthology.nuke:targets:
119
2021-09-10T05:14:10.376 INFO:teuthology.kill:  gibba001.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD7Dwz5eZsjA4lRqJ5lAnlTHpnVPHYhTsrPLQy6CKb8Eoc5+gX05oyvpeKzNpGBsXw+R/hlp/Ti0e6teKxrTupL+HIDE6GwanV2J2iwcdWzr9iA2tXuJ4egbfrNpLDJbQnkX59D6Ia6uyEceS2N32CDP4vkwkJ4P64+K20+G3a63fiYWcYGxEGWJi3sWILuxS8rOPLy55GOY6GOntGV3sWluDsxxrT1JrZPG1tKM1JtT7vZT9ofe7N1Jro/FyUdW8vp9coRShM4RHIea/mj0QCmnohhoO+pS2ZnmOJNTAdMudsV/z3Ru8d/n+0sS5VjHwiC/hutTNZMaTkOFYSsIR8j+rSbeX3AICU22wSTFlyVVDaKByOt2qhMLxY9vRJVaxSKyFg0OWniK7igo/TxBtvMvVhn2Vpizb6w9H5Z74XExwxBtQx+TKEJncfGai7nkyR9Yhf0xOz+4489Tx/Ad8XzHKrY7B32KkaIu9e7XrJBW9grjwsWGrvuL5b7OrE5lss=
120
2021-09-10T05:14:10.377 INFO:teuthology.kill:  gibba041.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcwXnlx05+pkQzUNij/F8XjjFK2w7q0rbnVzlOXaDW9fUbFHz+uWDlL+vR3LAZrqFLW7XXGazyyacjchn4+v9B8WlLT8pJysCnfukpQYNViL8tPU1+IK+mGG8pIzhKXVSEhsDHk9xvmwYBsq1JONz7Nw3hJrTCbEZD77XtA/JnOK1ksWwP1ZEOjZnQGmq4s7VPlUDb8HlH0dcd7IBXshWsTTDLk52y+CrJd/6VcgIeNDVsXXkzXwBwsKcQSvM20J3x0+ne8ETP6jFVa46eNXZUHcIwOyZ/6g1Sl0hasb6vOeiIwZafVte5H7an2GCrMbsGUkQ+nRizzH6fOZXRym3rlIDxUd8uKwCZpof9hrjkkhdmkb+cfwcdtTuPDEfZCaTKM8cBn7Wi46Enic0mdvR0AmnGGtwP47cCcLU4bPTqh54V3eSaFApNwnyWKY9iN/JSQ164GpbHCs26ixabDyNRUfGkdz44Qq9bOs0ZfFhghZ1YZD4gGTC7NshKsxChiE=
121
2021-09-10T05:14:10.396 INFO:teuthology.kill:2021-09-10 05:14:10,396.396 INFO:teuthology.task.internal.check_lock:Checking locks...
122
2021-09-10T05:14:10.400 INFO:teuthology.kill:2021-09-10 05:14:10,399.399 INFO:teuthology.task.internal.check_lock:Checking locks...
123
2021-09-10T05:14:10.456 INFO:teuthology.kill:2021-09-10 05:14:10,455.455 INFO:teuthology.orchestra.console:Power off gibba041
124
2021-09-10T05:14:10.466 INFO:teuthology.kill:2021-09-10 05:14:10,466.466 INFO:teuthology.orchestra.console:Power off gibba001
125
2021-09-10T05:14:10.602 INFO:teuthology.kill:2021-09-10 05:14:10,602.602 INFO:teuthology.orchestra.console:Power off for gibba041 completed
126
2021-09-10T05:14:10.611 INFO:teuthology.kill:2021-09-10 05:14:10,611.611 INFO:teuthology.orchestra.console:Power off for gibba001 completed
127
2021-09-10T05:14:10.741 INFO:teuthology.kill:2021-09-10 05:14:10,740.740 INFO:teuthology.lock.ops:unlocked gibba041.front.sepia.ceph.com
128
2021-09-10T05:14:10.747 INFO:teuthology.kill:2021-09-10 05:14:10,747.747 INFO:teuthology.lock.ops:unlocked gibba001.front.sepia.ceph.com
129
2021-09-10T05:16:10.954 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
130
2021-09-10T05:16:11.015 INFO:teuthology.dispatcher.supervisor:Nuking machines...
131
2021-09-10T05:16:11.016 INFO:teuthology.nuke:Checking targets against current locks
132
2021-09-10T05:16:11.093 INFO:teuthology.nuke:Not nuking gibba041.front.sepia.ceph.com because description doesn't match
133
2021-09-10T05:16:11.093 INFO:teuthology.nuke:Not nuking gibba001.front.sepia.ceph.com because description doesn't match