Bug #43912
Updated by Sage Weil about 4 years ago
<pre>
2020-01-29T22:28:24.867 INFO:tasks.ceph.mgr.x.smithi189.stderr:2020-01-29T22:28:24.864+0000 7f69f9aae700 -1 mgr.server reply reply (11) Resource temporarily unavailable 1 pgs have unknown state; cannot draw any conclusions
2020-01-29T22:28:24.869 INFO:tasks.cephfs_test_runner:test_safe_to_destroy (tasks.mgr.dashboard.test_osd.OsdTest) ... FAIL
2020-01-29T22:28:24.869 INFO:tasks.mgr.dashboard.helper:Request POST to https://smithi189:7789/api/osd/0/mark_in
2020-01-29T22:28:24.972 INFO:tasks.cephfs_test_runner:
2020-01-29T22:28:24.972 INFO:tasks.cephfs_test_runner:======================================================================
2020-01-29T22:28:24.972 INFO:tasks.cephfs_test_runner:FAIL: test_safe_to_destroy (tasks.mgr.dashboard.test_osd.OsdTest)
2020-01-29T22:28:24.972 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-01-29T22:28:24.973 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-01-29T22:28:24.973 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2020-01-29-1034/qa/tasks/mgr/dashboard/test_osd.py", line 113, in test_safe_to_destroy
2020-01-29T22:28:24.973 INFO:tasks.cephfs_test_runner: 'stored_pgs': [],
2020-01-29T22:28:24.973 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2020-01-29-1034/qa/tasks/mgr/dashboard/helper.py", line 343, in assertJsonBody
2020-01-29T22:28:24.973 INFO:tasks.cephfs_test_runner: self.assertEqual(body, data)
2020-01-29T22:28:24.973 INFO:tasks.cephfs_test_runner:AssertionError: {u'is_safe_to_destroy': False, u'message': u'[errno -11] 1 pgs have unknown stat [truncated]... != {'active': [], 'is_safe_to_destroy': True, 'stored_pgs': [], 'safe_to_destroy': [truncated]...
2020-01-29T22:28:24.973 INFO:tasks.cephfs_test_runner:+ {'active': [],
2020-01-29T22:28:24.973 INFO:tasks.cephfs_test_runner:- {u'is_safe_to_destroy': False,
2020-01-29T22:28:24.974 INFO:tasks.cephfs_test_runner:? ^^ ^^^^
2020-01-29T22:28:24.974 INFO:tasks.cephfs_test_runner:
2020-01-29T22:28:24.974 INFO:tasks.cephfs_test_runner:+ 'is_safe_to_destroy': True,
2020-01-29T22:28:24.974 INFO:tasks.cephfs_test_runner:? ^ ^^^
2020-01-29T22:28:24.974 INFO:tasks.cephfs_test_runner:
2020-01-29T22:28:24.975 INFO:tasks.cephfs_test_runner:- u'message': u'[errno -11] 1 pgs have unknown state; cannot draw any conclusions'}
2020-01-29T22:28:24.975 INFO:tasks.cephfs_test_runner:+ 'missing_stats': [],
2020-01-29T22:28:24.975 INFO:tasks.cephfs_test_runner:+ 'safe_to_destroy': [13],
2020-01-29T22:28:24.975 INFO:tasks.cephfs_test_runner:+ 'stored_pgs': []}
2020-01-29T22:28:24.975 INFO:tasks.cephfs_test_runner:
2020-01-29T22:28:24.976 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
</pre>
/a/sage-2020-01-29_20:14:58-rados-wip-sage-testing-2020-01-29-1034-distro-basic-smithi/4717950
/a/sage-2020-01-29_20:14:58-rados-wip-sage-testing-2020-01-29-1034-distro-basic-smithi/4718032
this looks like a flaky test? or very flaky--there were 2 failues in one run.
test.. it needs to retry a few times if it sees 'unknown' in the reason, and/or wait until pg state is not 'unknown' before doing the test