Actions
Bug #53262
closed"teuthology-queue -m smithi -D xx" fails "404 Client Error: Not Found for url"
Status:
Can't reproduce
Priority:
Normal
Assignee:
-
Category:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):
Description
teuthology-queue -m smithi -D teuthology-2021-11 2021-11-13 16:01:22,064.064 INFO:teuthology.beanstalk:Checking Beanstalk Queue... /home/yuriw/teuthology/virtualenv/lib/python3.6/site-packages/beanstalkc.py:141: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. return self._parse_yaml(body) Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501050 2021-11-13 16:01:51,539.539 ERROR:teuthology.report:Job deletion failed Traceback (most recent call last): File "/home/yuriw/teuthology/teuthology/report.py", line 550, in try_delete_job reporter.delete_job(run_name, job_id) File "/home/yuriw/teuthology/teuthology/report.py", line 433, in delete_job response.raise_for_status() File "/home/yuriw/teuthology/virtualenv/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://paddles.front.sepia.ceph.com:80/runs/teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/jobs/6501050/ Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501051 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501052 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501053 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501054 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501055 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501056 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501057 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501058 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501059 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501060 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501061 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501062 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501063 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501064 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501065 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501066 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501067 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501068 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501069 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501070 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501071 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501072 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501073 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501074 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501075 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501076 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501077 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501078 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501079 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501080 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501081 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501082 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501083 Deleting teuthology-2021-11-13_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/6501084 2021-11-13 16:01:54,363.363 ERROR:teuthology.report:Job deletion failed Traceback (most recent call last): File "/home/yuriw/teuthology/teuthology/report.py", line 550, in try_delete_job reporter.delete_job(run_name, job_id) File "/home/yuriw/teuthology/teuthology/report.py", line 433, in delete_job response.raise_for_status() File "/home/yuriw/teuthology/virtualenv/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://paddles.front.sepia.ceph.com:80/runs/teuthology-2021-11-13_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/jobs/6501084/ Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501085 Deleting teuthology-2021-11-13_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/6501086 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501087 Deleting teuthology-2021-11-13_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/6501088 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501089 Deleting teuthology-2021-11-13_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/6501090 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501091 Deleting teuthology-2021-11-13_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/6501092 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501093 Deleting teuthology-2021-11-13_14:23:02-upgrade:nautilus-x-pacific-distro-default-smithi/6501094 Deleting teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6501095 Traceback (most recent call last): File "/home/yuriw/teuthology/virtualenv/bin/teuthology-queue", line 33, in <module> sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-queue')()) File "/home/yuriw/teuthology/scripts/queue.py", line 36, in main teuthology.beanstalk.main(args) File "/home/yuriw/teuthology/teuthology/beanstalk.py", line 204, in main JobDeleter(delete)) File "/home/yuriw/teuthology/teuthology/beanstalk.py", line 59, in walk_jobs processor.add_job(job_id, job_config, job) File "/home/yuriw/teuthology/teuthology/beanstalk.py", line 145, in add_job super(JobDeleter, self).add_job(job_id, job_config, job_obj) File "/home/yuriw/teuthology/teuthology/beanstalk.py", line 91, in add_job self.process_job(job_id) File "/home/yuriw/teuthology/teuthology/beanstalk.py", line 157, in process_job report.try_delete_jobs(job_name, job_id) File "/home/yuriw/teuthology/teuthology/report.py", line 539, in try_delete_jobs got_jobs = reporter.get_jobs(run_name, fields=['job_id']) File "/home/yuriw/teuthology/teuthology/report.py", line 373, in get_jobs response.raise_for_status() File "/home/yuriw/teuthology/virtualenv/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://paddles.front.sepia.ceph.com:80/runs/teuthology-2021-11-13_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/jobs/?fields=job_id
Updated by Yuri Weinstein over 2 years ago
- Related to Bug #46840: errors while using "teuthology-queue -D" added
Updated by Kyrylo Shatskyy over 2 years ago
This bug is unrelated to YAMLLoadWarning issue.
Updated by Josh Durgin about 2 years ago
- Status changed from New to Can't reproduce
Updated by Yuri Weinstein about 1 year ago
- Status changed from Can't reproduce to New
Easy to see now.
yuriw@teuthology ~ [20:26:02]> teuthology-queue -m smithi -D yuriw-2023-03-08_20:24:37-fs-wip-yuri2-testing-2023-03-08-0800-pacific-distro-default-smithi
2023-03-08 20:26:20,143.143 INFO:teuthology.beanstalk:Checking Beanstalk Queue...
Deleting yuriw-2023-03-08_20:24:37-fs-wip-yuri2-testing-2023-03-08-0800-pacific-distro-default-smithi/7197832
2023-03-08 20:26:25,772.772 ERROR:teuthology.report:Job deletion failed
Traceback (most recent call last):
File "/home/yuriw/teuthology/teuthology/report.py", line 553, in try_delete_job
reporter.delete_job(run_name, job_id)
File "/home/yuriw/teuthology/teuthology/report.py", line 436, in delete_job
response.raise_for_status()
File "/home/yuriw/teuthology/virtualenv/lib/python3.8/site-packages/requests/models.py", line 960, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://paddles.front.sepia.ceph.com:80/runs/yuriw-2023-03-08_20:24:37-fs-wip-yuri2-testing-2023-03-08-0800-pacific-distro-default-smithi/jobs/7197832/
Deleting yuriw-2023-03-08_20:24:37-fs-wip-yuri2-testing-2023-03-08-0800-pacific-distro-default-smithi/7197833
Deleting yuriw-2023-03-08_20:24:37-fs-wip-yuri2-testing-2023-03-08-0800-pacific-distro-default-smithi/7197834
Deleting yuriw-2023-03-08_20:24:37-fs-wip-yuri2-testing-2023-03-08-0800-pacific-distro-default-smithi/7197835
Deleting yuriw-2023-03-08_20:24:37-fs-wip-yuri2-testing-2023-03-08-0800-pacific-distro-default-smithi/7197836
Loading 370/370
Updated by Laura Flores about 1 year ago
Reproduced it by scheduling a new teuthology:noop run and marking it dead:
lflores@teuthology:~$ ./teuthology/virtualenv/bin/teuthology-suite -v -m smithi -c main -s teuthology:nop -p 100
/home/lflores/teuthology/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
from cryptography.hazmat.backends import default_backend
2023-03-08 21:16:25,731.731 INFO:teuthology.suite:Using random seed=1827
2023-03-08 21:16:25,732.732 INFO:teuthology.suite.run:kernel sha1: distro
...
/home/lflores/teuthology/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
from cryptography.hazmat.backends import default_backend
Job scheduled with name lflores-2023-03-08_21:16:25-teuthology:nop-main-distro-default-smithi and ID 7198361
2023-03-08 21:16:29,042.042 INFO:teuthology.suite.run:Scheduling teuthology:nop/{all/nop}
/home/lflores/teuthology/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
from cryptography.hazmat.backends import default_backend
Job scheduled with name lflores-2023-03-08_21:16:25-teuthology:nop-main-distro-default-smithi and ID 7198362
2023-03-08 21:16:30,200.200 INFO:teuthology.suite.run:Suite teuthology:nop in /home/lflores/src/git.ceph.com_ceph-c_main/qa/suites/teuthology/nop scheduled 1 jobs.
2023-03-08 21:16:30,200.200 INFO:teuthology.suite.run:0/1 jobs were filtered out.
2023-03-08 21:16:30,200.200 INFO:teuthology.suite.run:Scheduled 1 jobs in total.
/home/lflores/teuthology/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
from cryptography.hazmat.backends import default_backend
Job scheduled with name lflores-2023-03-08_21:16:25-teuthology:nop-main-distro-default-smithi and ID 7198363
2023-03-08 21:16:31,294.294 INFO:teuthology.suite.run:Test results viewable at http://pulpito.front.sepia.ceph.com:80/lflores-2023-03-08_21:16:25-teuthology:nop-main-distro-default-smithi/
lflores@teuthology:~$ ./teuthology/virtualenv/bin/teuthology-queue -m smithi
Job: 809 priority: 100 lflores-2023-03-08_21:16:25-teuthology:nop-main-distro-default-smithi/7198361
Job: 810 priority: 100 lflores-2023-03-08_21:16:25-teuthology:nop-main-distro-default-smithi/7198362
Job: 811 priority: 100 lflores-2023-03-08_21:16:25-teuthology:nop-main-distro-default-smithi/7198363
lflores@teuthology:~$ ./teuthology/virtualenv/bin/teuthology-queue -m smithi -D lflores-2023-03-08_21:16:25-teuthology:nop-main-distro-default-smithi
/home/lflores/teuthology/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
from cryptography.hazmat.backends import default_backend
2023-03-08 21:20:51,328.328 INFO:teuthology.beanstalk:Checking Beanstalk Queue...
Deleting lflores-2023-03-08_21:16:25-teuthology:nop-main-distro-default-smithi/7198361
2023-03-08 21:21:10,533.533 ERROR:teuthology.report:Job deletion failed
Traceback (most recent call last):
File "/home/lflores/teuthology/teuthology/report.py", line 553, in try_delete_job
reporter.delete_job(run_name, job_id)
File "/home/lflores/teuthology/teuthology/report.py", line 436, in delete_job
response.raise_for_status()
File "/home/lflores/teuthology/virtualenv/lib/python3.6/site-packages/requests/models.py", line 960, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://paddles.front.sepia.ceph.com:80/runs/lflores-2023-03-08_21:16:25-teuthology:nop-main-distro-default-smithi/jobs/7198361/
Deleting lflores-2023-03-08_21:16:25-teuthology:nop-main-distro-default-smithi/7198362
Deleting lflores-2023-03-08_21:16:25-teuthology:nop-main-distro-default-smithi/7198363
Traceback (most recent call last):
File "./teuthology/virtualenv/bin/teuthology-queue", line 33, in <module>
sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-queue')())
File "/home/lflores/teuthology/scripts/queue.py", line 36, in main
teuthology.beanstalk.main(args)
File "/home/lflores/teuthology/teuthology/beanstalk.py", line 205, in main
JobDeleter(delete))
File "/home/lflores/teuthology/teuthology/beanstalk.py", line 60, in walk_jobs
processor.add_job(job_id, job_config, job)
File "/home/lflores/teuthology/teuthology/beanstalk.py", line 146, in add_job
super(JobDeleter, self).add_job(job_id, job_config, job_obj)
File "/home/lflores/teuthology/teuthology/beanstalk.py", line 92, in add_job
self.process_job(job_id)
File "/home/lflores/teuthology/teuthology/beanstalk.py", line 158, in process_job
report.try_delete_jobs(job_name, job_id)
File "/home/lflores/teuthology/teuthology/report.py", line 542, in try_delete_jobs
got_jobs = reporter.get_jobs(run_name, fields=['job_id'])
File "/home/lflores/teuthology/teuthology/report.py", line 373, in get_jobs
response.raise_for_status()
File "/home/lflores/teuthology/virtualenv/lib/python3.6/site-packages/requests/models.py", line 960, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://paddles.front.sepia.ceph.com:80/runs/lflores-2023-03-08_21:16:25-teuthology:nop-main-distro-default-smithi/jobs/?fields=job_id
Can't include an image for some reason, but this is the job that was queued. Notice that only one job is scheduled, but there are three jobs in the teuthology-queue. It successfully deletes 7198362, which is the job reflected in pulpito, but hits a Traceback with 7198361 and 7198363.
lflores-2023-03-08_21:16:25-teuthology:nop-main-distro-default-smithi
See other runs of suite 'teuthology:nop' on branch 'main'?
See other runs on branch 'main'?
See other runs scheduled on 2023-03-08?
See detail view?
queued 7198362 2023-03-08 21:16:30 2023-03-08 21:16:30 smithi main —
Updated by Kamoltat (Junior) Sirivadhna 8 months ago
- Status changed from New to Can't reproduce
Since we will be changing the queue to paddles soon and also the issues hasn't happen in a long time, closing this issue and marking it as cannot reproduce.
Actions