Project

General

Profile

Bug #38539

tasks.mgr.test_insights.TestInsights#test_health_history fails

Added by Sebastian Wagner about 2 years ago. Updated over 1 year ago.

Status:
New
Priority:
Normal
Assignee:
Brad Hubbard
Category:
insights module
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2019-03-01T15:01:23.607 INFO:tasks.cephfs_test_runner:test_health_history (tasks.mgr.test_insights.TestInsights) ... FAIL
2019-03-01T15:01:23.607 INFO:tasks.cephfs_test_runner:
2019-03-01T15:01:23.608 INFO:tasks.cephfs_test_runner:======================================================================
2019-03-01T15:01:23.608 INFO:tasks.cephfs_test_runner:FAIL: test_health_history (tasks.mgr.test_insights.TestInsights)
2019-03-01T15:01:23.608 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2019-03-01T15:01:23.608 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2019-03-01T15:01:23.608 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-swagner-testing/qa/tasks/mgr/test_insights.py", line 164, in test_health_history
2019-03-01T15:01:23.608 INFO:tasks.cephfs_test_runner:    self.assertFalse(report["health"]["history"]["checks"])
2019-03-01T15:01:23.608 INFO:tasks.cephfs_test_runner:AssertionError: {u'insights_health_check_0': {u'HEALTH_WARN': {u'detail': [u'detail'], u'summary': [u'summary']}}, u'insights_health_check': {u'HEALTH_WARN': {u'detail': [u'detail'], u'summary': [u'summary']}}} is not false
2019-03-01T15:01:23.608 INFO:tasks.cephfs_test_runner:
2019-03-01T15:01:23.609 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2019-03-01T15:01:23.609 INFO:tasks.cephfs_test_runner:Ran 2 tests in 160.209s
2019-03-01T15:01:23.609 INFO:tasks.cephfs_test_runner:
2019-03-01T15:01:23.609 INFO:tasks.cephfs_test_runner:FAILED (failures=1)

http://qa-proxy.ceph.com/teuthology/swagner-2019-03-01_13:11:37-rados:mgr-wip-swagner-testing-distro-basic-mira/3653500/teuthology.log

History

#2 Updated by Sebastian Wagner almost 2 years ago

might be a rare race?

#4 Updated by Sebastian Wagner over 1 year ago

  • Category set to insights module

#5 Updated by Brad Hubbard over 1 year ago

  • Assignee set to Brad Hubbard

#6 Updated by Neha Ojha over 1 year ago

/a/nojha-2019-11-22_01:55:39-rados-fix-merge-vs-backoff-with-revert-distro-basic-smithi/4531358/

#7 Updated by Brad Hubbard over 1 year ago

I can reproduce this and am trying to track it down.

#8 Updated by Brad Hubbard over 1 year ago

This sort of a race really should not be happening in a single threaded module like insights.

I've added the following decorator to try and get more information about concurrent calls to the insights methods and I am trying to reproduce with this code in place.

def log_enter_exit(fn):                                                                                                                                                                                                                      
    from functools import wraps                                                                                                                                                                                                              
    @wraps(fn)                                                                                                                                                                                                                               
    def wrapper(*args, **kwargs):                                                                                                                                                                                                            
        logger = logging.getLogger("insights")                                                                                                                                                                                               
        logger.info('insights-trace: entering {}'.format(fn.__name__))                                                                                                                                                                       

        out = fn(*args, **kwargs)                                                                                                                                                                                                            

        logger.info('insights-trace: exiting {}'.format(fn.__name__))                                                                                                                                                                        
        return out                                                                                                                                                                                                                           
    return wrapper

#9 Updated by Brad Hubbard over 1 year ago

  • Subject changed from tasks.mgr.test_insights.TestInsights#test_health_history failes to tasks.mgr.test_insights.TestInsights#test_health_history fails

#10 Updated by Patrick Donnelly over 1 year ago

  • Status changed from 12 to New

#11 Updated by Kefu Chai over 1 year ago

/a/kchai-2019-12-26_05:49:41-rados-wip-kefu-testing-2019-12-26-1055-distro-basic-smithi/4631691

Also available in: Atom PDF