Actions
Bug #50191
closedAttributeError: 'NoneType' object has no attribute 'get'
Status:
Resolved
Priority:
High
Assignee:
-
Category:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):
Description
2021-04-06T15:52:24.956 INFO:teuthology.task.internal.syslog:Compressing syslogs...
2021-04-06T15:52:24.957 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --
2021-04-06T15:52:24.980 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --
2021-04-06T15:52:25.025 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/run_tasks.py", line 94, in run_tasks
manager.__enter__()
File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/task/internal/syslog.py", line 50, in syslog
rem.chcon(log_path, log_context)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/remote.py", line 326, in chcon
if teuthology.lock.query.is_vm(self.shortname):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/lock/query.py", line 48, in is_vm
return status.get('is_vm', False)
AttributeError: 'NoneType' object has no attribute 'get'
2021-04-06T15:52:25.311 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=36a24ffeb7394377ace8e9fb4f11414e
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/run_tasks.py", line 94, in run_tasks
manager.__enter__()
File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/task/internal/syslog.py", line 50, in syslog
rem.chcon(log_path, log_context)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/remote.py", line 326, in chcon
if teuthology.lock.query.is_vm(self.shortname):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/lock/query.py", line 48, in is_vm
return status.get('is_vm', False)
AttributeError: 'NoneType' object has no attribute 'get'
/a/sage-2021-04-06_12:42:11-rados-wip-sage3-testing-2021-04-05-1538-distro-basic-smithi/6024169
/a/sage-2021-04-06_12:42:11-rados-wip-sage3-testing-2021-04-05-1538-distro-basic-smithi/6024168
/a/sage-2021-04-06_12:42:11-rados-wip-sage3-testing-2021-04-05-1538-distro-basic-smithi/6023965
/a/sage-2021-04-06_12:42:11-rados-wip-sage3-testing-2021-04-05-1538-distro-basic-smithi/6023960
Updated by Deepika Upadhyay about 3 years ago
/ceph/teuthology-archive/ideepika-2021-04-06_14:26:42-rados-wip-deepika-testing-2021-04-05-0643-distro-basic-smithi/6024583/teuthology.log
Updated by Josh Durgin about 3 years ago
The paddles worker is hanging somehow and timing out, causing this issue:
[2021-04-07 02:30:41 +0000] [8397] [CRITICAL] WORKER TIMEOUT (pid:27312)
I'm thinking we should add retries for teuthology in this code path
since it's where we keep running into this error.
I think we'll need a bunch more logging in paddles to be able to
diagnose why it's hanging.
Actions