Project

General

Profile

Bug #45717

FAIL: test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest)

Added by Brad Hubbard 8 months ago. Updated 6 months ago.

Status:
Resolved
Priority:
High
Assignee:
Kefu Chai
Category:
API Tests
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
nautilus, octopus
Backport:
nautilus, octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v16.0.0
ceph-qa-suite:
Pull request ID:
35871
Crash signature:

Description

/a/yuriw-2020-05-22_19:55:53-rados-wip-yuri-master_5.22.20-distro-basic-smithi/5083246

2020-05-22T23:15:05.259 INFO:tasks.mgr.dashboard.helper:task (pool/edit, {u'pool_name': u'pool_update_metadata'}) is still executing
2020-05-22T23:15:06.259 INFO:tasks.mgr.dashboard.helper:Request GET to https://smithi074:7791/api/task?name=pool/edit
2020-05-22T23:15:06.271 INFO:tasks.mgr.dashboard.helper:task (pool/edit, {u'pool_name': u'pool_update_metadata'}) finished
2020-05-22T23:15:11.272 INFO:tasks.mgr.dashboard.helper:Request GET to https://smithi074:7791/api/pool/pool_update_metadata
2020-05-22T23:15:11.290 INFO:tasks.mgr.dashboard.helper:Request GET to https://smithi074:7791/api/health/minimal
2020-05-22T23:15:11.386 INFO:tasks.mgr.dashboard.helper:Request PUT to https://smithi074:7791/api/pool/pool_update_metadata
2020-05-22T23:15:13.486 INFO:tasks.mgr.dashboard.helper:task (pool/edit, {u'pool_name': u'pool_update_metadata'}) is still executing
2020-05-22T23:15:14.487 INFO:tasks.mgr.dashboard.helper:Request GET to https://smithi074:7791/api/task?name=pool/edit
2020-05-22T23:15:14.495 INFO:tasks.mgr.dashboard.helper:task (pool/edit, {u'pool_name': u'pool_update_metadata'}) finished
2020-05-22T23:15:19.496 INFO:tasks.mgr.dashboard.helper:Request GET to https://smithi074:7791/api/pool/pool_update_metadata
2020-05-22T23:15:19.516 INFO:tasks.mgr.dashboard.helper:Request GET to https://smithi074:7791/api/health/minimal
2020-05-22T23:15:19.609 INFO:tasks.mgr.dashboard.helper:Request PUT to https://smithi074:7791/api/pool/pool_update_metadata
2020-05-22T23:15:21.708 INFO:tasks.mgr.dashboard.helper:task (pool/edit, {u'pool_name': u'pool_update_metadata'}) is still executing
2020-05-22T23:15:22.709 INFO:tasks.mgr.dashboard.helper:Request GET to https://smithi074:7791/api/task?name=pool/edit
2020-05-22T23:15:22.717 INFO:tasks.mgr.dashboard.helper:task (pool/edit, {u'pool_name': u'pool_update_metadata'}) finished
2020-05-22T23:15:27.718 INFO:tasks.mgr.dashboard.helper:Request GET to https://smithi074:7791/api/pool/pool_update_metadata
2020-05-22T23:15:27.739 INFO:tasks.mgr.dashboard.helper:Request GET to https://smithi074:7791/api/health/minimal
2020-05-22T23:15:27.837 INFO:tasks.cephfs_test_runner:test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest) ... FAIL
...
2020-05-22T23:15:29.048 INFO:tasks.cephfs_test_runner:FAIL: test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest)
2020-05-22T23:15:29.048 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-05-22T23:15:29.048 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-05-22T23:15:29.048 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri-master_5.22.20/qa/tasks/mgr/dashboard/test_pool.py", line 333, in test_pool_update_metadata
2020-05-22T23:15:29.048 INFO:tasks.cephfs_test_runner:    self._validate_pool_properties(properties, self._get_pool(pool_name))
2020-05-22T23:15:29.049 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri-master_5.22.20/qa/tasks/mgr/dashboard/test_pool.py", line 119, in _validate_pool_properties
2020-05-22T23:15:29.049 INFO:tasks.cephfs_test_runner:    self.assertEqual(health['status'], 'HEALTH_OK', msg='health={}'.format(health))
2020-05-22T23:15:29.049 INFO:tasks.cephfs_test_runner:AssertionError: health={u'status': u'HEALTH_WARN', u'checks': [{u'muted': False, u'detail': [{u'message': u'pg 5.37 is stuck peering since forever, current state peering, last acting [2,0]'}, {u'message': u'pg 6.d is stuck peering since forever, current state peering, last acting [3,1]'}], u'type': u'PG_AVAILABILITY', u'severity': u'HEALTH_WARN', u'summary': {u'count': 4, u'message': u'Reduced data availability: 2 pgs inactive, 2 pgs peering'}}], u'mutes': []}
2020-05-22T23:15:29.049 INFO:tasks.cephfs_test_runner:
2020-05-22T23:15:29.049 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-05-22T23:15:29.049 INFO:tasks.cephfs_test_runner:Ran 122 tests in 1424.264s
2020-05-22T23:15:29.049 INFO:tasks.cephfs_test_runner:
2020-05-22T23:15:29.049 INFO:tasks.cephfs_test_runner:FAILED (failures=1)
2020-05-22T23:15:29.049 INFO:tasks.cephfs_test_runner:
2020-05-22T23:15:29.049 INFO:tasks.cephfs_test_runner:======================================================================
2020-05-22T23:15:29.050 INFO:tasks.cephfs_test_runner:FAIL: test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest)
2020-05-22T23:15:29.050 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------

Related issues

Duplicated by mgr - Bug #45918: mgr/dashboard: "ceph-api-nightly-master-backend" failure: test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest) Resolved
Copied to mgr - Backport #46393: nautilus: FAIL: test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest) Resolved
Copied to mgr - Backport #46394: octopus: FAIL: test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest) Resolved
Copied to mgr - Bug #46573: FAIL: test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest) Resolved

History

#1 Updated by Alfonso Martínez 8 months ago

Hi Brad,

I saw that the last ceph-api-nightly-master-backend job runs are successful:
https://jenkins.ceph.com/view/all/job/ceph-api-nightly-master-backend/

So maybe this failure is indicating a negative impact on API/dashboard coming from the tested PR.

It would be great to know the exact PR tested in order to run "jenkins test dashboard backend"
jenkins job and see the result.

#2 Updated by Brad Hubbard 8 months ago

Hi Alfonso,

You can see the branch is "master" in Yuri's test run so this was a baseline run on the HEAD of master at the time (c3321b7686f181e1bcb805a1fb24baced390ae4c). We run these periodically to see what issues pop up.

#3 Updated by Laura Paduano 7 months ago

Also "tasks.mgr.dashboard.test_pool.PoolTest" failures happening in Jenkins now (but the Traceback differs from the original issue, I think I need to create a separate tracker issue):
- https://jenkins.ceph.com/view/mgr-dashboard/job/ceph-api-nightly-master-backend/191/

test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest)
2020-06-07 23:34:49,912.912 INFO:__main__:----------------------------------------------------------------------
2020-06-07 23:34:49,912.912 INFO:__main__:Traceback (most recent call last):
2020-06-07 23:34:49,912.912 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-api-nightly-master-backend/qa/tasks/mgr/dashboard/test_pool.py", line 333, in test_pool_update_metadata
2020-06-07 23:34:49,912.912 INFO:__main__:    self._validate_pool_properties(properties, self._get_pool(pool_name))
2020-06-07 23:34:49,912.912 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-api-nightly-master-backend/qa/tasks/mgr/dashboard/test_pool.py", line 119, in _validate_pool_properties
2020-06-07 23:34:49,912.912 INFO:__main__:    self.assertEqual(health['status'], 'HEALTH_OK', msg='health={}'.format(health))
2020-06-07 23:34:49,912.912 INFO:__main__:AssertionError: 'HEALTH_WARN' != 'HEALTH_OK'
2020-06-07 23:34:49,912.912 INFO:__main__:- HEALTH_WARN
2020-06-07 23:34:49,913.913 INFO:__main__:+ HEALTH_OK
2020-06-07 23:34:49,913.913 INFO:__main__: : health={'status': 'HEALTH_WARN', 'checks': [{'severity': 'HEALTH_WARN', 'summary': {'message': '1 pool(s) do not have an application enabled', 'count': 1}, 'detail': [{'message': "application not enabled on pool 'pool_update_metadata'"}, {'message': "use 'ceph osd pool application enable  ', where  is 'cephfs', 'rbd', 'rgw', or freeform for custom applications."}], 'muted': False, 'type': 'POOL_APP_NOT_ENABLED'}], 'mutes': []}
2020-06-07 23:34:49,913.913 INFO:__main__:
2020-06-07 23:34:49,913.913 INFO:__main__:----------------------------------------------------------------------
2020-06-07 23:34:49,913.913 INFO:__main__:Ran 122 tests in 2099.146s
2020-06-07 23:34:49,913.913 INFO:__main__:
2020-06-07 23:34:49,914.914 INFO:__main__:

#4 Updated by Sebastian Wagner 7 months ago

https://jenkins.ceph.com/job/ceph-dashboard-pr-backend/2200/

test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest)
2020-06-24 08:57:32,473.473 INFO:__main__:----------------------------------------------------------------------
2020-06-24 08:57:32,473.473 INFO:__main__:Traceback (most recent call last):
2020-06-24 08:57:32,473.473 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/test_pool.py", line 318, in test_pool_update_metadata
2020-06-24 08:57:32,473.473 INFO:__main__:    self._validate_pool_properties(props, self._get_pool(pool_name))
2020-06-24 08:57:32,473.473 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/test_pool.py", line 119, in _validate_pool_properties
2020-06-24 08:57:32,473.473 INFO:__main__:    self.assertEqual(health['status'], 'HEALTH_OK', msg='health={}'.format(health))
2020-06-24 08:57:32,473.473 INFO:__main__:AssertionError: 'HEALTH_WARN' != 'HEALTH_OK'
2020-06-24 08:57:32,474.474 INFO:__main__:- HEALTH_WARN
2020-06-24 08:57:32,474.474 INFO:__main__:+ HEALTH_OK
2020-06-24 08:57:32,474.474 INFO:__main__: : health={'status': 'HEALTH_WARN', 'checks': [{'severity': 'HEALTH_WARN', 'summary': {'message': 'Reduced data availability: 1 pg inactive', 'count': 1}, 'detail': [{'message': 'pg 11.14 is stuck inactive for 3m, current state activating, last acting [0,2,3]'}], 'muted': False, 'type': 'PG_AVAILABILITY'}], 'mutes': []}
2020-06-24 08:57:32,474.474 INFO:__main__:
2020-06-24 08:57:32,474.474 INFO:__main__:----------------------------------------------------------------------
2020-06-24 08:57:32,474.474 INFO:__main__:Ran 122 tests in 1536.051s
2020-06-24 08:57:32,474.474 INFO:__main__:
2020-06-24 08:57:32,475.475 INFO:__main__:

#5 Updated by Sebastian Wagner 7 months ago

https://jenkins.ceph.com/job/ceph-dashboard-pr-backend/1919/

test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest)
2020-06-19 16:20:45,941.941 INFO:__main__:----------------------------------------------------------------------
2020-06-19 16:20:45,941.941 INFO:__main__:Traceback (most recent call last):
2020-06-19 16:20:45,941.941 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/test_pool.py", line 333, in test_pool_update_metadata
2020-06-19 16:20:45,941.941 INFO:__main__:    self._validate_pool_properties(properties, self._get_pool(pool_name))
2020-06-19 16:20:45,941.941 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/test_pool.py", line 119, in _validate_pool_properties
2020-06-19 16:20:45,941.941 INFO:__main__:    self.assertEqual(health['status'], 'HEALTH_OK', msg='health={}'.format(health))
2020-06-19 16:20:45,942.942 INFO:__main__:AssertionError: 'HEALTH_WARN' != 'HEALTH_OK'
2020-06-19 16:20:45,942.942 INFO:__main__:- HEALTH_WARN
2020-06-19 16:20:45,942.942 INFO:__main__:+ HEALTH_OK
2020-06-19 16:20:45,942.942 INFO:__main__: : health={'status': 'HEALTH_WARN', 'checks': [{'severity': 'HEALTH_WARN', 'summary': {'message': 'Reduced data availability: 1 pg peering', 'count': 1}, 'detail': [{'message': 'pg 11.13 is stuck peering for 15m, current state peering, last acting [3,0,1]'}], 'muted': False, 'type': 'PG_AVAILABILITY'}], 'mutes': []}
2020-06-19 16:20:45,942.942 INFO:__main__:
2020-06-19 16:20:45,942.942 INFO:__main__:----------------------------------------------------------------------
2020-06-19 16:20:45,943.943 INFO:__main__:Ran 122 tests in 1674.284s
2020-06-19 16:20:45,943.943 INFO:__main__:
2020-06-19 16:20:45,943.943 INFO:__main__:

#6 Updated by Sebastian Wagner 7 months ago

  • Priority changed from Normal to High

#7 Updated by Sebastian Wagner 7 months ago

  • Duplicated by Bug #45918: mgr/dashboard: "ceph-api-nightly-master-backend" failure: test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest) added

#10 Updated by Sebastian Wagner 7 months ago

https://jenkins.ceph.com/job/ceph-dashboard-pr-backend/2267/

test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest)
2020-06-24 19:17:03,989.989 INFO:__main__:----------------------------------------------------------------------
2020-06-24 19:17:03,989.989 INFO:__main__:Traceback (most recent call last):
2020-06-24 19:17:03,989.989 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/test_pool.py", line 333, in test_pool_update_metadata
2020-06-24 19:17:03,990.990 INFO:__main__:    self._validate_pool_properties(properties, self._get_pool(pool_name))
2020-06-24 19:17:03,990.990 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/test_pool.py", line 119, in _validate_pool_properties
2020-06-24 19:17:03,990.990 INFO:__main__:    self.assertEqual(health['status'], 'HEALTH_OK', msg='health={}'.format(health))
2020-06-24 19:17:03,990.990 INFO:__main__:AssertionError: 'HEALTH_WARN' != 'HEALTH_OK'
2020-06-24 19:17:03,990.990 INFO:__main__:- HEALTH_WARN
2020-06-24 19:17:03,990.990 INFO:__main__:+ HEALTH_OK
2020-06-24 19:17:03,990.990 INFO:__main__: : health={'status': 'HEALTH_WARN', 'checks': [{'severity': 'HEALTH_WARN', 'summary': {'message': '1 pool(s) do not have an application enabled', 'count': 1}, 'detail': [{'message': "application not enabled on pool 'pool_update_metadata'"}, {'message': "use 'ceph osd pool application enable  ', where  is 'cephfs', 'rbd', 'rgw', or freeform for custom applications."}], 'muted': False, 'type': 'POOL_APP_NOT_ENABLED'}], 'mutes': []}
2020-06-24 19:17:03,991.991 INFO:__main__:
2020-06-24 19:17:03,991.991 INFO:__main__:----------------------------------------------------------------------
2020-06-24 19:17:03,991.991 INFO:__main__:Ran 122 tests in 2162.395s
2020-06-24 19:17:03,991.991 INFO:__main__:
2020-06-24 19:17:03,991.991 INFO:__main__:

#11 Updated by Kefu Chai 7 months ago

print osd log as well in https://github.com/ceph/ceph/pull/35812 for the root cause of peering.

#12 Updated by Ernesto Puerta 7 months ago

  • Status changed from New to In Progress
  • Assignee set to Ernesto Puerta

#14 Updated by Kefu Chai 7 months ago

  • Status changed from In Progress to Fix Under Review
  • Tags set to nautilus, octpus
  • Pull request ID set to 35871

#15 Updated by Kefu Chai 7 months ago

  • Assignee changed from Ernesto Puerta to Kefu Chai

#16 Updated by Kefu Chai 7 months ago

  • Category changed from dashboard/pools to ceph-mgr

#17 Updated by Alfonso Martínez 7 months ago

  • Tags changed from nautilus, octpus to nautilus, octopus

#18 Updated by Kefu Chai 7 months ago

  • Status changed from Fix Under Review to Pending Backport

#19 Updated by Laura Paduano 6 months ago

  • Backport set to nautilus, octopus

#20 Updated by Laura Paduano 6 months ago

  • Copied to Backport #46393: nautilus: FAIL: test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest) added

#21 Updated by Laura Paduano 6 months ago

  • Copied to Backport #46394: octopus: FAIL: test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest) added

#22 Updated by Laura Paduano 6 months ago

Sebastian Wagner wrote:

https://jenkins.ceph.com/job/ceph-dashboard-pr-backend/2200/

[...]

Today's build on a PR: https://jenkins.ceph.com/job/ceph-dashboard-pr-backend/3187/

test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest)
2020-07-08 14:49:55,097.097 INFO:__main__:----------------------------------------------------------------------
2020-07-08 14:49:55,098.098 INFO:__main__:Traceback (most recent call last):
2020-07-08 14:49:55,098.098 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/test_pool.py", line 318, in test_pool_update_metadata
2020-07-08 14:49:55,098.098 INFO:__main__:    self._validate_pool_properties(props, self._get_pool(pool_name))
2020-07-08 14:49:55,098.098 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/test_pool.py", line 119, in _validate_pool_properties
2020-07-08 14:49:55,098.098 INFO:__main__:    self.assertEqual(health['status'], 'HEALTH_OK', msg='health={}'.format(health))
2020-07-08 14:49:55,098.098 INFO:__main__:AssertionError: 'HEALTH_WARN' != 'HEALTH_OK'
2020-07-08 14:49:55,098.098 INFO:__main__:- HEALTH_WARN
2020-07-08 14:49:55,098.098 INFO:__main__:+ HEALTH_OK
2020-07-08 14:49:55,099.099 INFO:__main__: : health={'status': 'HEALTH_WARN', 'checks': [{'severity': 'HEALTH_WARN', 'summary': {'message': 'Reduced data availability: 1 pg inactive, 1 pg peering', 'count': 2}, 'detail': [{'message': 'pg 11.16 is stuck peering for 3m, current state peering, last acting [0,2,1]'}], 'muted': False, 'type': 'PG_AVAILABILITY'}], 'mutes': []}
2020-07-08 14:49:55,099.099 INFO:__main__:
2020-07-08 14:49:55,099.099 INFO:__main__:----------------------------------------------------------------------
2020-07-08 14:49:55,099.099 INFO:__main__:Ran 122 tests in 1536.917s
2020-07-08 14:49:55,099.099 INFO:__main__:
2020-07-08 14:49:55,099.099 INFO:__main__:

#23 Updated by Ernesto Puerta 6 months ago

  • Copied to Bug #46573: FAIL: test_pool_update_metadata (tasks.mgr.dashboard.test_pool.PoolTest) added

#24 Updated by Ernesto Puerta 6 months ago

  • Category changed from ceph-mgr to API Tests

#25 Updated by Nathan Cutler 6 months ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Also available in: Atom PDF