Project

General

Profile

Bug #46797

ERROR: test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest)

Added by Sebastian Wagner about 2 months ago. Updated about 2 months ago.

Status:
New
Priority:
Normal
Assignee:
Category:
API Tests
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

2020-07-31 05:16:37,145.145 INFO:__main__:----------------------------------------------------------------------
2020-07-31 05:16:37,145.145 INFO:__main__:Traceback (most recent call last):
2020-07-31 05:16:37,146.146 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/test_pool.py", line 327, in test_pool_update_metadata
2020-07-31 05:16:37,146.146 INFO:__main__:    with self.__yield_pool(pool_name):
2020-07-31 05:16:37,146.146 INFO:__main__:  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
2020-07-31 05:16:37,146.146 INFO:__main__:    return next(self.gen)
2020-07-31 05:16:37,146.146 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/test_pool.py", line 58, in __yield_pool
2020-07-31 05:16:37,147.147 INFO:__main__:    data = self._create_pool(name, data)
2020-07-31 05:16:37,147.147 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/test_pool.py", line 77, in _create_pool
2020-07-31 05:16:37,147.147 INFO:__main__:    self._task_post('/api/pool/', data)
2020-07-31 05:16:37,147.147 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/helper.py", line 337, in _task_post
2020-07-31 05:16:37,147.147 INFO:__main__:    return cls._task_request('POST', url, data, timeout)
2020-07-31 05:16:37,148.148 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/helper.py", line 317, in _task_request
2020-07-31 05:16:37,148.148 INFO:__main__:    .format(task_name, task_metadata, _res))
2020-07-31 05:16:37,148.148 INFO:__main__:Exception: Waiting for task (pool/create, {'pool_name': 'pool_update_metadata'}) to finish timed out. {'executing_tasks': [{'name': 'pool/create', 'metadata': {'pool_name': 'pool_update_metadata'}, 'begin_time': '2020-07-31T05:14:56.624619Z', 'progress': 0}, {'name': 'progress/PG autoscaler decreasing pool 11 PGs from 32 to 8', 'metadata': {'pool': 11}, 'begin_time': '2020-07-31T05:12:39.386827Z', 'progress': 45}], 'finished_tasks': [{'name': 'pool/create', 'metadata': {'pool_name': 'pool_update_configuration'}, 'begin_time': '2020-07-31T05:14:35.870663Z', 'end_time': '2020-07-31T05:14:43.978899Z', 'duration': 8.108235836029053, 'progress': 100, 'success': True, 'ret_value': None, 'exception': None}, {'name': 'pool/create', 'metadata': {'pool_name': 'pool_update_compression'}, 'begin_time': '2020-07-31T05:14:08.247703Z', 'end_time': '2020-07-31T05:14:18.308763Z', 'duration': 10.061060190200806, 'progress': 100, 'success': True, 'ret_value': None, 'exception': None}, {'name': 'progress/PG autoscaler increasing pool 15 PGs from 8 to 32', 'metadata': {'pool': 15}, 'begin_time': '2020-07-31T05:12:40.733971Z', 'end_time': '2020-07-31T05:13:40.765727Z', 'duration': 60.031755685806274, 'progress': 100, 'success': True, 'ret_value': None, 'exception': None}, {'name': 'pool/create', 'metadata': {'pool_name': 'dashboard_pool_quota2'}, 'begin_time': '2020-07-31T05:13:06.829904Z', 'end_time': '2020-07-31T05:13:11.590168Z', 'duration': 4.760263919830322, 'progress': 100, 'success': True, 'ret_value': None, 'exception': None}, {'name': 'pool/create', 'metadata': {'pool_name': 'dashboard_pool_quota1'}, 'begin_time': '2020-07-31T05:13:02.254149Z', 'end_time': '2020-07-31T05:13:03.345344Z', 'duration': 1.0911946296691895, 'progress': 100, 'success': True, 'ret_value': None, 'exception': None}, {'name': 'pool/create', 'metadata': {'pool_name': 'dashboard_pool3'}, 'begin_time': '2020-07-31T05:12:26.802483Z', 'end_time': '2020-07-31T05:12:42.674605Z', 'duration': 15.872122049331665, 'progress': 100, 'success': True, 'ret_value': None, 'exception': None}, {'name': 'pool/create', 'metadata': {'pool_name': 'sadfs'}, 'begin_time': '2020-07-31T05:12:22.019784Z', 'end_time': '2020-07-31T05:12:22.030813Z', 'duration': 0.011028766632080078, 'progress': 0, 'success': False, 'ret_value': None, 'exception': {'detail': "[errno -2] specified rule dnf doesn't exist"}}, {'name': 'progress/Rebalancing after osd.0 marked in', 'metadata': {'osd': 0}, 'begin_time': '2020-07-31T05:09:13.529738Z', 'end_time': '2020-07-31T05:09:22.037584Z', 'duration': 8.507845878601074, 'progress': 100, 'success': True, 'ret_value': None, 'exception': None}, {'name': 'progress/Rebalancing after osd.0 marked out', 'metadata': {'osd': 0}, 'begin_time': '2020-07-31T05:09:12.471868Z', 'end_time': '2020-07-31T05:09:13.529236Z', 'duration': 1.057368516921997, 'progress': 100, 'success': True, 'ret_value': None, 'exception': None}, {'name': 'progress/apply_drivesgroups', 'metadata': {'origin': 'orchestrator'}, 'begin_time': '2020-07-31T05:08:51.012334Z', 'end_time': '2020-07-31T05:08:51.014928Z', 'duration': 0.0025937557220458984, 'progress': 100, 'success': True, 'ret_value': None, 'exception': None}]}
2020-07-31 05:16:37,148.148 INFO:__main__:
2020-07-31 05:16:37,149.149 INFO:__main__:----------------------------------------------------------------------
2020-07-31 05:16:37,149.149 INFO:__main__:Ran 138 tests in 2056.558s
2020-07-31 05:16:37,149.149 INFO:__main__:
2020-07-31 05:16:37,149.149 INFO:__main__:

https://jenkins.ceph.com/job/ceph-dashboard-pr-backend/4785/

History

#2 Updated by Alfonso Martínez about 2 months ago

Ceph API nightly job was successful making requests against:
https://slave-ubuntu07.front.sepia.ceph.com:7790

5 hours later, the PR job failed making requests against:
https://slave-ubuntu07.front.sepia.ceph.com:7789

Is it possible that by that time the machine was not in a clean state?

API nightly:

2020-07-31 00:26:03,498.498 INFO:__main__:Starting test: test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest)
2020-07-31 00:26:03,498.498 INFO:__main__:Running ['./bin/ceph', 'log', 'Starting test tasks.mgr.dashboard.test_pool.PoolTest.test_pool_update_metadata']
2020-07-31 00:26:04,510.510 INFO:__main__:Running ['./bin/ceph', 'health', '--format=json']
2020-07-31 00:26:05,135.135 INFO:tasks.mgr.dashboard.helper:Request POST to https://slave-ubuntu07.front.sepia.ceph.com:7790/api/pool/
/tmp/tmp.liU0zW9PcX/venv/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
2020-07-31 00:26:07,155.155 INFO:tasks.mgr.dashboard.helper:task (pool/create, {'pool_name': 'pool_update_metadata'}) is still executing
2020-07-31 00:26:08,156.156 INFO:tasks.mgr.dashboard.helper:Request GET to https://slave-ubuntu07.front.sepia.ceph.com:7790/api/task?name=pool/create
/tmp/tmp.liU0zW9PcX/venv/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)

PR job:

2020-07-31 05:14:54,528.528 INFO:__main__:Starting test: test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest)
2020-07-31 05:14:54,529.529 INFO:__main__:Running ['./bin/ceph', 'log', 'Starting test tasks.mgr.dashboard.test_pool.PoolTest.test_pool_update_metadata']
2020-07-31 05:14:55,922.922 INFO:__main__:Running ['./bin/ceph', 'health', '--format=json']
2020-07-31 05:14:56,614.614 INFO:tasks.mgr.dashboard.helper:Request POST to https://slave-ubuntu07.front.sepia.ceph.com:7789/api/pool/
/tmp/tmp.AsU0lnpFHd/venv/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)

#3 Updated by David Galloway about 2 months ago

I see the mon, mgr, and osd logs get tailed at the end of the job. Is there a way to see more debug from the dashboard or whatever is behind https://slave-ubuntu07.front.sepia.ceph.com:7789/api?

Only one job can run at a time on each builder. If slave-ubuntu07 wasn't in a clean state, it is every job's responsibility to clean up after itself.

Also available in: Atom PDF