Project

General

Profile

Actions
Copy link

Bug #45717

closed

FAIL: test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest)

Added by Brad Hubbard almost 4 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
High
Assignee:
Kefu Chai
Category:
Testing - API
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
nautilus, octopus
Backport:
nautilus, octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v16.0.0
ceph-qa-suite:
Pull request ID:
35871
Crash signature (v1):
Crash signature (v2):

Description

/a/yuriw-2020-05-22_19:55:53-rados-wip-yuri-master_5.22.20-distro-basic-smithi/5083246

2020-05-22T23:15:05.259 INFO:tasks.mgr.dashboard.helper:task (pool/edit, {u'pool_name': u'pool_update_metadata'}) is still executing
2020-05-22T23:15:06.259 INFO:tasks.mgr.dashboard.helper:Request GET to https://smithi074:7791/api/task?name=pool/edit
2020-05-22T23:15:06.271 INFO:tasks.mgr.dashboard.helper:task (pool/edit, {u'pool_name': u'pool_update_metadata'}) finished
2020-05-22T23:15:11.272 INFO:tasks.mgr.dashboard.helper:Request GET to https://smithi074:7791/api/pool/pool_update_metadata
2020-05-22T23:15:11.290 INFO:tasks.mgr.dashboard.helper:Request GET to https://smithi074:7791/api/health/minimal
2020-05-22T23:15:11.386 INFO:tasks.mgr.dashboard.helper:Request PUT to https://smithi074:7791/api/pool/pool_update_metadata
2020-05-22T23:15:13.486 INFO:tasks.mgr.dashboard.helper:task (pool/edit, {u'pool_name': u'pool_update_metadata'}) is still executing
2020-05-22T23:15:14.487 INFO:tasks.mgr.dashboard.helper:Request GET to https://smithi074:7791/api/task?name=pool/edit
2020-05-22T23:15:14.495 INFO:tasks.mgr.dashboard.helper:task (pool/edit, {u'pool_name': u'pool_update_metadata'}) finished
2020-05-22T23:15:19.496 INFO:tasks.mgr.dashboard.helper:Request GET to https://smithi074:7791/api/pool/pool_update_metadata
2020-05-22T23:15:19.516 INFO:tasks.mgr.dashboard.helper:Request GET to https://smithi074:7791/api/health/minimal
2020-05-22T23:15:19.609 INFO:tasks.mgr.dashboard.helper:Request PUT to https://smithi074:7791/api/pool/pool_update_metadata
2020-05-22T23:15:21.708 INFO:tasks.mgr.dashboard.helper:task (pool/edit, {u'pool_name': u'pool_update_metadata'}) is still executing
2020-05-22T23:15:22.709 INFO:tasks.mgr.dashboard.helper:Request GET to https://smithi074:7791/api/task?name=pool/edit
2020-05-22T23:15:22.717 INFO:tasks.mgr.dashboard.helper:task (pool/edit, {u'pool_name': u'pool_update_metadata'}) finished
2020-05-22T23:15:27.718 INFO:tasks.mgr.dashboard.helper:Request GET to https://smithi074:7791/api/pool/pool_update_metadata
2020-05-22T23:15:27.739 INFO:tasks.mgr.dashboard.helper:Request GET to https://smithi074:7791/api/health/minimal
2020-05-22T23:15:27.837 INFO:tasks.cephfs_test_runner:test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest) ... FAIL
...
2020-05-22T23:15:29.048 INFO:tasks.cephfs_test_runner:FAIL: test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest)
2020-05-22T23:15:29.048 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-05-22T23:15:29.048 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-05-22T23:15:29.048 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri-master_5.22.20/qa/tasks/mgr/dashboard/test_pool.py", line 333, in test_pool_update_metadata
2020-05-22T23:15:29.048 INFO:tasks.cephfs_test_runner:    self._validate_pool_properties(properties, self._get_pool(pool_name))
2020-05-22T23:15:29.049 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri-master_5.22.20/qa/tasks/mgr/dashboard/test_pool.py", line 119, in _validate_pool_properties
2020-05-22T23:15:29.049 INFO:tasks.cephfs_test_runner:    self.assertEqual(health['status'], 'HEALTH_OK', msg='health={}'.format(health))
2020-05-22T23:15:29.049 INFO:tasks.cephfs_test_runner:AssertionError: health={u'status': u'HEALTH_WARN', u'checks': [{u'muted': False, u'detail': [{u'message': u'pg 5.37 is stuck peering since forever, current state peering, last acting [2,0]'}, {u'message': u'pg 6.d is stuck peering since forever, current state peering, last acting [3,1]'}], u'type': u'PG_AVAILABILITY', u'severity': u'HEALTH_WARN', u'summary': {u'count': 4, u'message': u'Reduced data availability: 2 pgs inactive, 2 pgs peering'}}], u'mutes': []}
2020-05-22T23:15:29.049 INFO:tasks.cephfs_test_runner:
2020-05-22T23:15:29.049 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-05-22T23:15:29.049 INFO:tasks.cephfs_test_runner:Ran 122 tests in 1424.264s
2020-05-22T23:15:29.049 INFO:tasks.cephfs_test_runner:
2020-05-22T23:15:29.049 INFO:tasks.cephfs_test_runner:FAILED (failures=1)
2020-05-22T23:15:29.049 INFO:tasks.cephfs_test_runner:
2020-05-22T23:15:29.049 INFO:tasks.cephfs_test_runner:======================================================================
2020-05-22T23:15:29.050 INFO:tasks.cephfs_test_runner:FAIL: test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest)
2020-05-22T23:15:29.050 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------

Related issues 4 (0 open — 4 closed)

Has duplicate Dashboard - Bug #45918: mgr/dashboard: "ceph-api-nightly-master-backend" failure: test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest)Resolved

Actions
Copied to mgr - Backport #46393: nautilus: FAIL: test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest)ResolvedLaura PaduanoActions
Copied to mgr - Backport #46394: octopus: FAIL: test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest)ResolvedLaura PaduanoActions
Copied to Dashboard - Bug #46573: FAIL: test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest)ResolvedTatjana Dehler

Actions
Actions
Copy link
 #1

Updated by Alfonso Martínez almost 4 years ago

Hi Brad,

I saw that the last ceph-api-nightly-master-backend job runs are successful:
https://jenkins.ceph.com/view/all/job/ceph-api-nightly-master-backend/

So maybe this failure is indicating a negative impact on API/dashboard coming from the tested PR.

It would be great to know the exact PR tested in order to run "jenkins test dashboard backend"
jenkins job and see the result.

Actions
Copy link
 #2

Updated by Brad Hubbard almost 4 years ago

Hi Alfonso,

You can see the branch is "master" in Yuri's test run so this was a baseline run on the HEAD of master at the time (c3321b7686f181e1bcb805a1fb24baced390ae4c). We run these periodically to see what issues pop up.

Actions
Copy link
 #3

Updated by Laura Paduano almost 4 years ago

Also "tasks.mgr.dashboard.test_pool.PoolTest" failures happening in Jenkins now (but the Traceback differs from the original issue, I think I need to create a separate tracker issue):
- https://jenkins.ceph.com/view/mgr-dashboard/job/ceph-api-nightly-master-backend/191/

test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest)
2020-06-07 23:34:49,912.912 INFO:__main__:----------------------------------------------------------------------
2020-06-07 23:34:49,912.912 INFO:__main__:Traceback (most recent call last):
2020-06-07 23:34:49,912.912 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-api-nightly-master-backend/qa/tasks/mgr/dashboard/test_pool.py", line 333, in test_pool_update_metadata
2020-06-07 23:34:49,912.912 INFO:__main__:    self._validate_pool_properties(properties, self._get_pool(pool_name))
2020-06-07 23:34:49,912.912 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-api-nightly-master-backend/qa/tasks/mgr/dashboard/test_pool.py", line 119, in _validate_pool_properties
2020-06-07 23:34:49,912.912 INFO:__main__:    self.assertEqual(health['status'], 'HEALTH_OK', msg='health={}'.format(health))
2020-06-07 23:34:49,912.912 INFO:__main__:AssertionError: 'HEALTH_WARN' != 'HEALTH_OK'
2020-06-07 23:34:49,912.912 INFO:__main__:- HEALTH_WARN
2020-06-07 23:34:49,913.913 INFO:__main__:+ HEALTH_OK
2020-06-07 23:34:49,913.913 INFO:__main__: : health={'status': 'HEALTH_WARN', 'checks': [{'severity': 'HEALTH_WARN', 'summary': {'message': '1 pool(s) do not have an application enabled', 'count': 1}, 'detail': [{'message': "application not enabled on pool 'pool_update_metadata'"}, {'message': "use 'ceph osd pool application enable  ', where  is 'cephfs', 'rbd', 'rgw', or freeform for custom applications."}], 'muted': False, 'type': 'POOL_APP_NOT_ENABLED'}], 'mutes': []}
2020-06-07 23:34:49,913.913 INFO:__main__:
2020-06-07 23:34:49,913.913 INFO:__main__:----------------------------------------------------------------------
2020-06-07 23:34:49,913.913 INFO:__main__:Ran 122 tests in 2099.146s
2020-06-07 23:34:49,913.913 INFO:__main__:
2020-06-07 23:34:49,914.914 INFO:__main__:
Actions
Copy link
 #4

Updated by Sebastian Wagner almost 4 years ago

https://jenkins.ceph.com/job/ceph-dashboard-pr-backend/2200/

test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest)
2020-06-24 08:57:32,473.473 INFO:__main__:----------------------------------------------------------------------
2020-06-24 08:57:32,473.473 INFO:__main__:Traceback (most recent call last):
2020-06-24 08:57:32,473.473 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/test_pool.py", line 318, in test_pool_update_metadata
2020-06-24 08:57:32,473.473 INFO:__main__:    self._validate_pool_properties(props, self._get_pool(pool_name))
2020-06-24 08:57:32,473.473 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/test_pool.py", line 119, in _validate_pool_properties
2020-06-24 08:57:32,473.473 INFO:__main__:    self.assertEqual(health['status'], 'HEALTH_OK', msg='health={}'.format(health))
2020-06-24 08:57:32,473.473 INFO:__main__:AssertionError: 'HEALTH_WARN' != 'HEALTH_OK'
2020-06-24 08:57:32,474.474 INFO:__main__:- HEALTH_WARN
2020-06-24 08:57:32,474.474 INFO:__main__:+ HEALTH_OK
2020-06-24 08:57:32,474.474 INFO:__main__: : health={'status': 'HEALTH_WARN', 'checks': [{'severity': 'HEALTH_WARN', 'summary': {'message': 'Reduced data availability: 1 pg inactive', 'count': 1}, 'detail': [{'message': 'pg 11.14 is stuck inactive for 3m, current state activating, last acting [0,2,3]'}], 'muted': False, 'type': 'PG_AVAILABILITY'}], 'mutes': []}
2020-06-24 08:57:32,474.474 INFO:__main__:
2020-06-24 08:57:32,474.474 INFO:__main__:----------------------------------------------------------------------
2020-06-24 08:57:32,474.474 INFO:__main__:Ran 122 tests in 1536.051s
2020-06-24 08:57:32,474.474 INFO:__main__:
2020-06-24 08:57:32,475.475 INFO:__main__:
Actions
Copy link
 #5

Updated by Sebastian Wagner almost 4 years ago

https://jenkins.ceph.com/job/ceph-dashboard-pr-backend/1919/

test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest)
2020-06-19 16:20:45,941.941 INFO:__main__:----------------------------------------------------------------------
2020-06-19 16:20:45,941.941 INFO:__main__:Traceback (most recent call last):
2020-06-19 16:20:45,941.941 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/test_pool.py", line 333, in test_pool_update_metadata
2020-06-19 16:20:45,941.941 INFO:__main__:    self._validate_pool_properties(properties, self._get_pool(pool_name))
2020-06-19 16:20:45,941.941 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/test_pool.py", line 119, in _validate_pool_properties
2020-06-19 16:20:45,941.941 INFO:__main__:    self.assertEqual(health['status'], 'HEALTH_OK', msg='health={}'.format(health))
2020-06-19 16:20:45,942.942 INFO:__main__:AssertionError: 'HEALTH_WARN' != 'HEALTH_OK'
2020-06-19 16:20:45,942.942 INFO:__main__:- HEALTH_WARN
2020-06-19 16:20:45,942.942 INFO:__main__:+ HEALTH_OK
2020-06-19 16:20:45,942.942 INFO:__main__: : health={'status': 'HEALTH_WARN', 'checks': [{'severity': 'HEALTH_WARN', 'summary': {'message': 'Reduced data availability: 1 pg peering', 'count': 1}, 'detail': [{'message': 'pg 11.13 is stuck peering for 15m, current state peering, last acting [3,0,1]'}], 'muted': False, 'type': 'PG_AVAILABILITY'}], 'mutes': []}
2020-06-19 16:20:45,942.942 INFO:__main__:
2020-06-19 16:20:45,942.942 INFO:__main__:----------------------------------------------------------------------
2020-06-19 16:20:45,943.943 INFO:__main__:Ran 122 tests in 1674.284s
2020-06-19 16:20:45,943.943 INFO:__main__:
2020-06-19 16:20:45,943.943 INFO:__main__:
Actions
Copy link
 #6

Updated by Sebastian Wagner almost 4 years ago

  • Priority changed from Normal to High
Actions
Copy link
 #7

Updated by Sebastian Wagner almost 4 years ago

  • Has duplicate Bug #45918: mgr/dashboard: "ceph-api-nightly-master-backend" failure: test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest) added
Actions
Copy link
 #10

Updated by Sebastian Wagner almost 4 years ago

https://jenkins.ceph.com/job/ceph-dashboard-pr-backend/2267/

test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest)
2020-06-24 19:17:03,989.989 INFO:__main__:----------------------------------------------------------------------
2020-06-24 19:17:03,989.989 INFO:__main__:Traceback (most recent call last):
2020-06-24 19:17:03,989.989 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/test_pool.py", line 333, in test_pool_update_metadata
2020-06-24 19:17:03,990.990 INFO:__main__:    self._validate_pool_properties(properties, self._get_pool(pool_name))
2020-06-24 19:17:03,990.990 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/test_pool.py", line 119, in _validate_pool_properties
2020-06-24 19:17:03,990.990 INFO:__main__:    self.assertEqual(health['status'], 'HEALTH_OK', msg='health={}'.format(health))
2020-06-24 19:17:03,990.990 INFO:__main__:AssertionError: 'HEALTH_WARN' != 'HEALTH_OK'
2020-06-24 19:17:03,990.990 INFO:__main__:- HEALTH_WARN
2020-06-24 19:17:03,990.990 INFO:__main__:+ HEALTH_OK
2020-06-24 19:17:03,990.990 INFO:__main__: : health={'status': 'HEALTH_WARN', 'checks': [{'severity': 'HEALTH_WARN', 'summary': {'message': '1 pool(s) do not have an application enabled', 'count': 1}, 'detail': [{'message': "application not enabled on pool 'pool_update_metadata'"}, {'message': "use 'ceph osd pool application enable  ', where  is 'cephfs', 'rbd', 'rgw', or freeform for custom applications."}], 'muted': False, 'type': 'POOL_APP_NOT_ENABLED'}], 'mutes': []}
2020-06-24 19:17:03,991.991 INFO:__main__:
2020-06-24 19:17:03,991.991 INFO:__main__:----------------------------------------------------------------------
2020-06-24 19:17:03,991.991 INFO:__main__:Ran 122 tests in 2162.395s
2020-06-24 19:17:03,991.991 INFO:__main__:
2020-06-24 19:17:03,991.991 INFO:__main__:
Actions
Copy link
 #11

Updated by Kefu Chai almost 4 years ago

print osd log as well in https://github.com/ceph/ceph/pull/35812 for the root cause of peering.

Actions
Copy link
 #12

Updated by Ernesto Puerta almost 4 years ago

  • Status changed from New to In Progress
  • Assignee set to Ernesto Puerta
Actions
Copy link
 #14

Updated by Kefu Chai almost 4 years ago

  • Status changed from In Progress to Fix Under Review
  • Tags set to nautilus, octpus
  • Pull request ID set to 35871
Actions
Copy link
 #15

Updated by Kefu Chai almost 4 years ago

  • Assignee changed from Ernesto Puerta to Kefu Chai
Actions
Copy link
 #16

Updated by Kefu Chai almost 4 years ago

  • Category changed from 137 to ceph-mgr
Actions
Copy link
 #17

Updated by Alfonso Martínez almost 4 years ago

  • Tags changed from nautilus, octpus to nautilus, octopus
Actions
Copy link
 #18

Updated by Kefu Chai almost 4 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions
Copy link
 #19

Updated by Laura Paduano almost 4 years ago

  • Backport set to nautilus, octopus
Actions
Copy link
 #20

Updated by Laura Paduano almost 4 years ago

  • Copied to Backport #46393: nautilus: FAIL: test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest) added
Actions
Copy link
 #21

Updated by Laura Paduano almost 4 years ago

  • Copied to Backport #46394: octopus: FAIL: test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest) added
Actions
Copy link
 #22

Updated by Laura Paduano almost 4 years ago

Sebastian Wagner wrote:

https://jenkins.ceph.com/job/ceph-dashboard-pr-backend/2200/

[...]

Today's build on a PR: https://jenkins.ceph.com/job/ceph-dashboard-pr-backend/3187/

test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest)
2020-07-08 14:49:55,097.097 INFO:__main__:----------------------------------------------------------------------
2020-07-08 14:49:55,098.098 INFO:__main__:Traceback (most recent call last):
2020-07-08 14:49:55,098.098 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/test_pool.py", line 318, in test_pool_update_metadata
2020-07-08 14:49:55,098.098 INFO:__main__:    self._validate_pool_properties(props, self._get_pool(pool_name))
2020-07-08 14:49:55,098.098 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/test_pool.py", line 119, in _validate_pool_properties
2020-07-08 14:49:55,098.098 INFO:__main__:    self.assertEqual(health['status'], 'HEALTH_OK', msg='health={}'.format(health))
2020-07-08 14:49:55,098.098 INFO:__main__:AssertionError: 'HEALTH_WARN' != 'HEALTH_OK'
2020-07-08 14:49:55,098.098 INFO:__main__:- HEALTH_WARN
2020-07-08 14:49:55,098.098 INFO:__main__:+ HEALTH_OK
2020-07-08 14:49:55,099.099 INFO:__main__: : health={'status': 'HEALTH_WARN', 'checks': [{'severity': 'HEALTH_WARN', 'summary': {'message': 'Reduced data availability: 1 pg inactive, 1 pg peering', 'count': 2}, 'detail': [{'message': 'pg 11.16 is stuck peering for 3m, current state peering, last acting [0,2,1]'}], 'muted': False, 'type': 'PG_AVAILABILITY'}], 'mutes': []}
2020-07-08 14:49:55,099.099 INFO:__main__:
2020-07-08 14:49:55,099.099 INFO:__main__:----------------------------------------------------------------------
2020-07-08 14:49:55,099.099 INFO:__main__:Ran 122 tests in 1536.917s
2020-07-08 14:49:55,099.099 INFO:__main__:
2020-07-08 14:49:55,099.099 INFO:__main__:

Actions
Copy link
 #23

Updated by Ernesto Puerta almost 4 years ago

  • Copied to Bug #46573: FAIL: test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest) added
Actions
Copy link
 #24

Updated by Ernesto Puerta almost 4 years ago

  • Category changed from ceph-mgr to API Tests
Actions
Copy link
 #25

Updated by Nathan Cutler almost 4 years ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Actions
Copy link
 #26

Updated by Ernesto Puerta about 3 years ago

  • Project changed from mgr to Dashboard
  • Category changed from API Tests to Testing - API
Actions
Copy link

Also available in: Atom PDF