Project

General

Profile

Actions
Copy link

Bug #45149

closed

mgr/dashboard: ceph-api-nightly-master-backend and ceph-api-nightly-octopus-backend RuntimeError "test_purge_trash (tasks.mgr.dashboard.test_rbd.RbdTest)"

Added by Laura Paduano about 4 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Kiefer Chang
Category:
Testing & QA
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
octopus,nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v14.2.9, Ceph - v15.2.2, Ceph - v16.0.0
ceph-qa-suite:
Pull request ID:
34705
Crash signature (v1):
Crash signature (v2):

Description

2020-04-18 23:38:34,597.597 INFO:__main__:test_purge_trash (tasks.mgr.dashboard.test_rbd.RbdTest) ... ERROR
2020-04-18 23:38:34,599.599 INFO:__main__:Running ['./bin/ceph', 'log', 'Ended test tasks.mgr.dashboard.test_rbd.RbdTest.test_purge_trash']
2020-04-18 23:38:36,449.449 INFO:__main__:Stopped test: test_purge_trash (tasks.mgr.dashboard.test_rbd.RbdTest) in 67.035623s
2020-04-18 23:38:36,451.451 INFO:__main__:Running ['./bin/ceph', 'osd', 'pool', 'delete', 'rbd', 'rbd', '--yes-i-really-really-mean-it']
2020-04-18 23:38:37,550.550 INFO:tasks.mgr.dashboard.helper:command result: 
2020-04-18 23:38:37,551.551 INFO:__main__:Running ['./bin/ceph', 'osd', 'pool', 'delete', 'rbd_iscsi', 'rbd_iscsi', '--yes-i-really-really-mean-it']
2020-04-18 23:38:38,601.601 INFO:tasks.mgr.dashboard.helper:command result: 
2020-04-18 23:38:38,602.602 INFO:__main__:Running ['./bin/ceph', 'osd', 'pool', 'delete', 'rbd_data', 'rbd_data', '--yes-i-really-really-mean-it']
2020-04-18 23:38:39,186.186 INFO:tasks.mgr.dashboard.helper:command result: 
2020-04-18 23:38:39,187.187 INFO:__main__:
2020-04-18 23:38:39,187.187 INFO:__main__:======================================================================
2020-04-18 23:38:39,187.187 INFO:__main__:ERROR: test_purge_trash (tasks.mgr.dashboard.test_rbd.RbdTest)
2020-04-18 23:38:39,188.188 INFO:__main__:----------------------------------------------------------------------
2020-04-18 23:38:39,188.188 INFO:__main__:Traceback (most recent call last):
2020-04-18 23:38:39,188.188 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-api-nightly-master-backend/qa/tasks/mgr/dashboard/test_rbd.py", line 854, in test_purge_trash
2020-04-18 23:38:39,188.188 INFO:__main__:    self.wait_until_equal(lambda: self.get_trash('rbd', id_expired), None, 60)
2020-04-18 23:38:39,189.189 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-api-nightly-master-backend/qa/tasks/ceph_test_case.py", line 176, in wait_until_equal
2020-04-18 23:38:39,189.189 INFO:__main__:    elapsed, expect_val, val
2020-04-18 23:38:39,189.189 INFO:__main__:RuntimeError: Timed out after 60 seconds waiting for None (currently {u'pool_name': u'rbd', u'name': u'test_rbd_expired', u'deletion_time': u'2020-04-19T03:37:31Z', u'namespace': u'', u'deferment_end_time': u'2020-04-19T03:37:31Z', u'source': u'USER', u'id': u'3cffa06dbe3'})
2020-04-18 23:38:39,189.189 INFO:__main__:
2020-04-18 23:38:39,190.190 INFO:__main__:----------------------------------------------------------------------
2020-04-18 23:38:39,190.190 INFO:__main__:Ran 161 tests in 2346.200s
2020-04-18 23:38:39,190.190 INFO:__main__:
2020-04-18 23:38:39,191.191 INFO:__main__:FAILED (errors=1)
2020-04-18 23:38:39,191.191 INFO:__main__:
2020-04-18 23:38:39,191.191 INFO:__main__:======================================================================
2020-04-18 23:38:39,191.191 INFO:__main__:ERROR: test_purge_trash (tasks.mgr.dashboard.test_rbd.RbdTest)
2020-04-18 23:38:39,192.192 INFO:__main__:----------------------------------------------------------------------
2020-04-18 23:38:39,192.192 INFO:__main__:Traceback (most recent call last):
2020-04-18 23:38:39,192.192 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-api-nightly-master-backend/qa/tasks/mgr/dashboard/test_rbd.py", line 854, in test_purge_trash
2020-04-18 23:38:39,193.193 INFO:__main__:    self.wait_until_equal(lambda: self.get_trash('rbd', id_expired), None, 60)
2020-04-18 23:38:39,193.193 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-api-nightly-master-backend/qa/tasks/ceph_test_case.py", line 176, in wait_until_equal
2020-04-18 23:38:39,193.193 INFO:__main__:    elapsed, expect_val, val
2020-04-18 23:38:39,193.193 INFO:__main__:RuntimeError: Timed out after 60 seconds waiting for None (currently {u'pool_name': u'rbd', u'name': u'test_rbd_expired', u'deletion_time': u'2020-04-19T03:37:31Z', u'namespace': u'', u'deferment_end_time': u'2020-04-19T03:37:31Z', u'source': u'USER', u'id': u'3cffa06dbe3'})

- https://jenkins.ceph.com/view/all/job/ceph-api-nightly-master-backend/140/
- https://jenkins.ceph.com/view/all/job/ceph-api-nightly-master-backend/139/

Related issues 2 (0 open — 2 closed)

Copied to Dashboard - Backport #45272: octopus: mgr/dashboard: ceph-api-nightly-master-backend and ceph-api-nightly-octopus-backend RuntimeError "test_purge_trash (tasks.mgr.dashboard.test_rbd.RbdTest)"ResolvedLaura PaduanoActions
Copied to Dashboard - Backport #45273: nautilus: mgr/dashboard: ceph-api-nightly-master-backend and ceph-api-nightly-octopus-backend RuntimeError "test_purge_trash (tasks.mgr.dashboard.test_rbd.RbdTest)"ResolvedKiefer ChangActions
Actions
Copy link
 #1

Updated by Lenz Grimmer about 4 years ago

Looks like this test has been failing since April 11th: https://jenkins.ceph.com/view/all/job/ceph-api-nightly-master-backend/131/ - something seems to have changed in master that day that causes the RBD deletion to fail.

Actions
Copy link
 #2

Updated by Kiefer Chang about 4 years ago

Some thoughts from the log files:

  • In POST /api/block/image/trash/purge call, NO trash images are deleted. So no matter how long we wait, the trashed image will be there.
  • Initially I suspected that's because we comparing datetime strings here, but I think this should work if the strings are in isoformats. Comparing datetime objects might be a better approach.
  • The call loops RBD trash list, which is encapsulated within a ViewCache. But from the log, cache is not used, a call is initialized to get fresh RBD trash list.
Actions
Copy link
 #3

Updated by Laura Paduano about 4 years ago

  • Subject changed from mgr/dashboard: ceph-api-nightly-master-backend RuntimeError "test_purge_trash (tasks.mgr.dashboard.test_rbd.RbdTest)" to mgr/dashboard: ceph-api-nightly-master-backend and ceph-api-nightly-octopus-backend RuntimeError "test_purge_trash (tasks.mgr.dashboard.test_rbd.RbdTest)"
  • Affected Versions v15.2.2 added
Actions
Copy link
 #4

Updated by Kiefer Chang about 4 years ago

Adding some debugging messages (PR) and test on jenkins.
The debugging log is:

2020-04-23T02:00:11.961-0400 7f3eb2a6f700  0 [dashboard INFO dashboard.controllers.rbd] now: 2020-04-23T02:00:11.958301Z
2020-04-23T02:00:11.961-0400 7f3eb2a6f700  0 [dashboard INFO dashboard.controllers.rbd] pools: [{'status': 0, 'value': [{'id': '127c2a273b8e', 'name': 'test_rbd_expired', 'source': 'USER', 
'deletion_time': '2020-04-23T06:00:10Z', 'deferment_end_time': '2020-04-23T06:00:10Z', 'pool_name': 'rbd', 'namespace': ''}, {'id': '127c4e48a834', 'name': 'test_rbd', 'source': 'USER', 'de
letion_time': '2020-04-23T06:00:10Z', 'deferment_end_time': '2020-04-23T08:46:49Z', 'pool_name': 'rbd', 'namespace': ''}], 'pool_name': 'rbd'}]

Looks like one timestamp is in UTC and another is local time:

'deferment_end_time': '2020-04-23T06:00:10Z'

vs.
now: 2020-04-23T02:00:11.958301Z

Actions
Copy link
 #5

Updated by Kiefer Chang about 4 years ago

  • Status changed from New to Fix Under Review
  • Assignee set to Kiefer Chang
  • Backport set to octopus,nautilus
  • Pull request ID set to 34705
  • Affected Versions v14.2.9 added
Actions
Copy link
 #6

Updated by Laura Paduano about 4 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions
Copy link
 #7

Updated by Laura Paduano about 4 years ago

  • Copied to Backport #45272: octopus: mgr/dashboard: ceph-api-nightly-master-backend and ceph-api-nightly-octopus-backend RuntimeError "test_purge_trash (tasks.mgr.dashboard.test_rbd.RbdTest)" added
Actions
Copy link
 #8

Updated by Laura Paduano about 4 years ago

  • Copied to Backport #45273: nautilus: mgr/dashboard: ceph-api-nightly-master-backend and ceph-api-nightly-octopus-backend RuntimeError "test_purge_trash (tasks.mgr.dashboard.test_rbd.RbdTest)" added
Actions
Copy link
 #9

Updated by Kiefer Chang almost 4 years ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link
 #10

Updated by Ernesto Puerta about 3 years ago

  • Project changed from mgr to Dashboard
  • Category changed from 151 to Testing & QA
Actions
Copy link

Also available in: Atom PDF