Project

General

Profile

Actions

Bug #38539

closed

tasks.mgr.test_insights.TestInsights#test_health_history fails

Added by Sebastian Wagner about 5 years ago. Updated over 2 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
insights module
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2019-03-01T15:01:23.607 INFO:tasks.cephfs_test_runner:test_health_history (tasks.mgr.test_insights.TestInsights) ... FAIL
2019-03-01T15:01:23.607 INFO:tasks.cephfs_test_runner:
2019-03-01T15:01:23.608 INFO:tasks.cephfs_test_runner:======================================================================
2019-03-01T15:01:23.608 INFO:tasks.cephfs_test_runner:FAIL: test_health_history (tasks.mgr.test_insights.TestInsights)
2019-03-01T15:01:23.608 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2019-03-01T15:01:23.608 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2019-03-01T15:01:23.608 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-swagner-testing/qa/tasks/mgr/test_insights.py", line 164, in test_health_history
2019-03-01T15:01:23.608 INFO:tasks.cephfs_test_runner:    self.assertFalse(report["health"]["history"]["checks"])
2019-03-01T15:01:23.608 INFO:tasks.cephfs_test_runner:AssertionError: {u'insights_health_check_0': {u'HEALTH_WARN': {u'detail': [u'detail'], u'summary': [u'summary']}}, u'insights_health_check': {u'HEALTH_WARN': {u'detail': [u'detail'], u'summary': [u'summary']}}} is not false
2019-03-01T15:01:23.608 INFO:tasks.cephfs_test_runner:
2019-03-01T15:01:23.609 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2019-03-01T15:01:23.609 INFO:tasks.cephfs_test_runner:Ran 2 tests in 160.209s
2019-03-01T15:01:23.609 INFO:tasks.cephfs_test_runner:
2019-03-01T15:01:23.609 INFO:tasks.cephfs_test_runner:FAILED (failures=1)

http://qa-proxy.ceph.com/teuthology/swagner-2019-03-01_13:11:37-rados:mgr-wip-swagner-testing-distro-basic-mira/3653500/teuthology.log

Actions #2

Updated by Sebastian Wagner almost 5 years ago

might be a rare race?

Actions #4

Updated by Sebastian Wagner over 4 years ago

  • Category set to insights module
Actions #5

Updated by Brad Hubbard over 4 years ago

  • Assignee set to Brad Hubbard
Actions #6

Updated by Neha Ojha over 4 years ago

/a/nojha-2019-11-22_01:55:39-rados-fix-merge-vs-backoff-with-revert-distro-basic-smithi/4531358/

Actions #7

Updated by Brad Hubbard over 4 years ago

I can reproduce this and am trying to track it down.

Actions #8

Updated by Brad Hubbard over 4 years ago

This sort of a race really should not be happening in a single threaded module like insights.

I've added the following decorator to try and get more information about concurrent calls to the insights methods and I am trying to reproduce with this code in place.

def log_enter_exit(fn):                                                                                                                                                                                                                      
    from functools import wraps                                                                                                                                                                                                              
    @wraps(fn)                                                                                                                                                                                                                               
    def wrapper(*args, **kwargs):                                                                                                                                                                                                            
        logger = logging.getLogger("insights")                                                                                                                                                                                               
        logger.info('insights-trace: entering {}'.format(fn.__name__))                                                                                                                                                                       

        out = fn(*args, **kwargs)                                                                                                                                                                                                            

        logger.info('insights-trace: exiting {}'.format(fn.__name__))                                                                                                                                                                        
        return out                                                                                                                                                                                                                           
    return wrapper 
Actions #9

Updated by Brad Hubbard over 4 years ago

  • Subject changed from tasks.mgr.test_insights.TestInsights#test_health_history failes to tasks.mgr.test_insights.TestInsights#test_health_history fails
Actions #10

Updated by Patrick Donnelly over 4 years ago

  • Status changed from 12 to New
Actions #11

Updated by Kefu Chai over 4 years ago

/a/kchai-2019-12-26_05:49:41-rados-wip-kefu-testing-2019-12-26-1055-distro-basic-smithi/4631691

Actions #12

Updated by Brad Hubbard over 2 years ago

  • Status changed from New to Closed

Closing this since we haven't seen it in a while. Please reopen it if this issue reappears.

Actions

Also available in: Atom PDF