Actions
Bug #53262
closed"teuthology-queue -m smithi -D xx" fails "404 Client Error: Not Found for url"
Status:
Can't reproduce
Priority:
Normal
Assignee:
-
Category:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):
Description
teuthology-queue -m smithi -D teuthology-2021-11
2021-11-13 16:01:22,064.064 INFO:teuthology.beanstalk:Checking Beanstalk Queue...
/home/yuriw/teuthology/virtualenv/lib/python3.6/site-packages/beanstalkc.py:141: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
return self._parse_yaml(body)
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501050
2021-11-13 16:01:51,539.539 ERROR:teuthology.report:Job deletion failed
Traceback (most recent call last):
File "/home/yuriw/teuthology/teuthology/report.py", line 550, in try_delete_job
reporter.delete_job(run_name, job_id)
File "/home/yuriw/teuthology/teuthology/report.py", line 433, in delete_job
response.raise_for_status()
File "/home/yuriw/teuthology/virtualenv/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://paddles.front.sepia.ceph.com:80/runs/teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/jobs/6501050/
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501051
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501052
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501053
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501054
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501055
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501056
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501057
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501058
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501059
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501060
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501061
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501062
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501063
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501064
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501065
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501066
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501067
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501068
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501069
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501070
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501071
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501072
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501073
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501074
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501075
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501076
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501077
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501078
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501079
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501080
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501081
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501082
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501083
Deleting teuthology-2021-11-13_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/6501084
2021-11-13 16:01:54,363.363 ERROR:teuthology.report:Job deletion failed
Traceback (most recent call last):
File "/home/yuriw/teuthology/teuthology/report.py", line 550, in try_delete_job
reporter.delete_job(run_name, job_id)
File "/home/yuriw/teuthology/teuthology/report.py", line 433, in delete_job
response.raise_for_status()
File "/home/yuriw/teuthology/virtualenv/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://paddles.front.sepia.ceph.com:80/runs/teuthology-2021-11-13_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/jobs/6501084/
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501085
Deleting teuthology-2021-11-13_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/6501086
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501087
Deleting teuthology-2021-11-13_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/6501088
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501089
Deleting teuthology-2021-11-13_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/6501090
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501091
Deleting teuthology-2021-11-13_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/6501092
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501093
Deleting teuthology-2021-11-13_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/6501094
Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501095
Traceback (most recent call last):
File "/home/yuriw/teuthology/virtualenv/bin/teuthology-queue", line 33, in <module>
sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-queue')())
File "/home/yuriw/teuthology/scripts/queue.py", line 36, in main
teuthology.beanstalk.main(args)
File "/home/yuriw/teuthology/teuthology/beanstalk.py", line 204, in main
JobDeleter(delete))
File "/home/yuriw/teuthology/teuthology/beanstalk.py", line 59, in walk_jobs
processor.add_job(job_id, job_config, job)
File "/home/yuriw/teuthology/teuthology/beanstalk.py", line 145, in add_job
super(JobDeleter, self).add_job(job_id, job_config, job_obj)
File "/home/yuriw/teuthology/teuthology/beanstalk.py", line 91, in add_job
self.process_job(job_id)
File "/home/yuriw/teuthology/teuthology/beanstalk.py", line 157, in process_job
report.try_delete_jobs(job_name, job_id)
File "/home/yuriw/teuthology/teuthology/report.py", line 539, in try_delete_jobs
got_jobs = reporter.get_jobs(run_name, fields=['job_id'])
File "/home/yuriw/teuthology/teuthology/report.py", line 373, in get_jobs
response.raise_for_status()
File "/home/yuriw/teuthology/virtualenv/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://paddles.front.sepia.ceph.com:80/runs/teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/jobs/?fields=job_id
Updated by Yuri Weinstein over 2 years ago
- Related to Bug #46840: errors while using "teuthology-queue -D" added
Updated by Kyrylo Shatskyy over 2 years ago
This bug is unrelated to YAMLLoadWarning issue.
Updated by Josh Durgin about 2 years ago
- Status changed from New to Can't reproduce
Updated by Yuri Weinstein about 1 year ago
- Status changed from Can't reproduce to New
Easy to see now.
yuriw@teuthology ~ [20:26:02]> teuthology-queue -m smithi -D yuriw-2023-03-08_20:24:37-fs-wip-yuri2-testing-2023-03-08-0800-pacific-distro-default-smithi
2023-03-08 20:26:20,143.143 INFO:teuthology.beanstalk:Checking Beanstalk Queue...
Deleting yuriw-2023-03-08_20:24:37-fs-wip-yuri2-testing-2023-03-08-0800-pacific-distro-default-smithi/7197832
2023-03-08 20:26:25,772.772 ERROR:teuthology.report:Job deletion failed
Traceback (most recent call last):
File "/home/yuriw/teuthology/teuthology/report.py", line 553, in try_delete_job
reporter.delete_job(run_name, job_id)
File "/home/yuriw/teuthology/teuthology/report.py", line 436, in delete_job
response.raise_for_status()
File "/home/yuriw/teuthology/virtualenv/lib/python3.8/site-packages/requests/models.py", line 960, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://paddles.front.sepia.ceph.com:80/runs/yuriw-2023-03-08_20:24:37-fs-wip-yuri2-testing-2023-03-08-0800-pacific-distro-default-smithi/jobs/7197832/
Deleting yuriw-2023-03-08_20:24:37-fs-wip-yuri2-testing-2023-03-08-0800-pacific-distro-default-smithi/7197833
Deleting yuriw-2023-03-08_20:24:37-fs-wip-yuri2-testing-2023-03-08-0800-pacific-distro-default-smithi/7197834
Deleting yuriw-2023-03-08_20:24:37-fs-wip-yuri2-testing-2023-03-08-0800-pacific-distro-default-smithi/7197835
Deleting yuriw-2023-03-08_20:24:37-fs-wip-yuri2-testing-2023-03-08-0800-pacific-distro-default-smithi/7197836
Loading 370/370
Updated by Laura Flores about 1 year ago
Reproduced it by scheduling a new teuthology:noop run and marking it dead:
lflores@teuthology:~$ ./teuthology/virtualenv/bin/teuthology-suite -v -m smithi -c main -s teuthology:nop -p 100
/home/lflores/teuthology/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
from cryptography.hazmat.backends import default_backend
2023-03-08 21:16:25,731.731 INFO:teuthology.suite:Using random seed=1827
2023-03-08 21:16:25,732.732 INFO:teuthology.suite.run:kernel sha1: distro
...
/home/lflores/teuthology/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
from cryptography.hazmat.backends import default_backend
Job scheduled with name lflores-2023-03-08_21:16:25-teuthology:nop-main-distro-default-smithi and ID 7198361
2023-03-08 21:16:29,042.042 INFO:teuthology.suite.run:Scheduling teuthology:nop/{all/nop}
/home/lflores/teuthology/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
from cryptography.hazmat.backends import default_backend
Job scheduled with name lflores-2023-03-08_21:16:25-teuthology:nop-main-distro-default-smithi and ID 7198362
2023-03-08 21:16:30,200.200 INFO:teuthology.suite.run:Suite teuthology:nop in /home/lflores/src/git.ceph.com_ceph-c_main/qa/suites/teuthology/nop scheduled 1 jobs.
2023-03-08 21:16:30,200.200 INFO:teuthology.suite.run:0/1 jobs were filtered out.
2023-03-08 21:16:30,200.200 INFO:teuthology.suite.run:Scheduled 1 jobs in total.
/home/lflores/teuthology/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
from cryptography.hazmat.backends import default_backend
Job scheduled with name lflores-2023-03-08_21:16:25-teuthology:nop-main-distro-default-smithi and ID 7198363
2023-03-08 21:16:31,294.294 INFO:teuthology.suite.run:Test results viewable at http://pulpito.front.sepia.ceph.com:80/lflores-2023-03-08_21:16:25-teuthology:nop-main-distro-default-smithi/
lflores@teuthology:~$ ./teuthology/virtualenv/bin/teuthology-queue -m smithi
Job: 809 priority: 100 lflores-2023-03-08_21:16:25-teuthology:nop-main-distro-default-smithi/7198361
Job: 810 priority: 100 lflores-2023-03-08_21:16:25-teuthology:nop-main-distro-default-smithi/7198362
Job: 811 priority: 100 lflores-2023-03-08_21:16:25-teuthology:nop-main-distro-default-smithi/7198363
lflores@teuthology:~$ ./teuthology/virtualenv/bin/teuthology-queue -m smithi -D lflores-2023-03-08_21:16:25-teuthology:nop-main-distro-default-smithi
/home/lflores/teuthology/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
from cryptography.hazmat.backends import default_backend
2023-03-08 21:20:51,328.328 INFO:teuthology.beanstalk:Checking Beanstalk Queue...
Deleting lflores-2023-03-08_21:16:25-teuthology:nop-main-distro-default-smithi/7198361
2023-03-08 21:21:10,533.533 ERROR:teuthology.report:Job deletion failed
Traceback (most recent call last):
File "/home/lflores/teuthology/teuthology/report.py", line 553, in try_delete_job
reporter.delete_job(run_name, job_id)
File "/home/lflores/teuthology/teuthology/report.py", line 436, in delete_job
response.raise_for_status()
File "/home/lflores/teuthology/virtualenv/lib/python3.6/site-packages/requests/models.py", line 960, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://paddles.front.sepia.ceph.com:80/runs/lflores-2023-03-08_21:16:25-teuthology:nop-main-distro-default-smithi/jobs/7198361/
Deleting lflores-2023-03-08_21:16:25-teuthology:nop-main-distro-default-smithi/7198362
Deleting lflores-2023-03-08_21:16:25-teuthology:nop-main-distro-default-smithi/7198363
Traceback (most recent call last):
File "./teuthology/virtualenv/bin/teuthology-queue", line 33, in <module>
sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-queue')())
File "/home/lflores/teuthology/scripts/queue.py", line 36, in main
teuthology.beanstalk.main(args)
File "/home/lflores/teuthology/teuthology/beanstalk.py", line 205, in main
JobDeleter(delete))
File "/home/lflores/teuthology/teuthology/beanstalk.py", line 60, in walk_jobs
processor.add_job(job_id, job_config, job)
File "/home/lflores/teuthology/teuthology/beanstalk.py", line 146, in add_job
super(JobDeleter, self).add_job(job_id, job_config, job_obj)
File "/home/lflores/teuthology/teuthology/beanstalk.py", line 92, in add_job
self.process_job(job_id)
File "/home/lflores/teuthology/teuthology/beanstalk.py", line 158, in process_job
report.try_delete_jobs(job_name, job_id)
File "/home/lflores/teuthology/teuthology/report.py", line 542, in try_delete_jobs
got_jobs = reporter.get_jobs(run_name, fields=['job_id'])
File "/home/lflores/teuthology/teuthology/report.py", line 373, in get_jobs
response.raise_for_status()
File "/home/lflores/teuthology/virtualenv/lib/python3.6/site-packages/requests/models.py", line 960, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://paddles.front.sepia.ceph.com:80/runs/lflores-2023-03-08_21:16:25-teuthology:nop-main-distro-default-smithi/jobs/?fields=job_id
Can't include an image for some reason, but this is the job that was queued. Notice that only one job is scheduled, but there are three jobs in the teuthology-queue. It successfully deletes 7198362, which is the job reflected in pulpito, but hits a Traceback with 7198361 and 7198363.
lflores-2023-03-08_21:16:25-teuthology:nop-main-distro-default-smithi
See other runs of suite 'teuthology:nop' on branch 'main'?
See other runs on branch 'main'?
See other runs scheduled on 2023-03-08?
See detail view?
queued 7198362 2023-03-08 21:16:30 2023-03-08 21:16:30 smithi main —
Updated by Kamoltat (Junior) Sirivadhna 7 months ago
- Status changed from New to Can't reproduce
Since we will be changing the queue to paddles soon and also the issues hasn't happen in a long time, closing this issue and marking it as cannot reproduce.
Actions