Bug #52316: qa/tasks/mon_thrash.py: _do_thrash AssertionError len(s['quorum']) == len(mons) - RADOS - Ceph

Actions

Copy link

Bug #52316

closed

qa/tasks/mon_thrash.py: _do_thrash AssertionError len(s['quorum']) == len(mons)

Added by Aishwarya Mathuria over 2 years ago. Updated 6 months ago.

Status:

Resolved

Priority:

Normal

Assignee:

Nitzan Mordechai

Category:

Target version:

% Done:

100%

Source:

Tags:

backport_processed

Backport:

pacific,quincy,reef

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(RADOS):

Monitor

Pull request ID:

51570

Crash signature (v1):

Crash signature (v2):

Description

2021-08-17T03:12:45.055 INFO:tasks.workunit.client.0.smithi135.stderr:2021-08-17T03:12:45.052+0000 7f27d941a700 1 -- 172.21.15.135:0/2440531935 >> v1:172.21.15.13:6790/0 conn(0x7f27b000a460 legacy=0x7f27b000a840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.13:6790/0
2021-08-17T03:12:45.055 INFO:tasks.workunit.client.0.smithi135.stderr:2021-08-17T03:12:45.052+0000 7f27d8c19700 1 -- 172.21.15.135:0/2440531935 >> v1:172.21.15.135:6789/0 conn(0x7f27b000db60 legacy=0x7f27b000f1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.135:6789/0
2021-08-17T03:12:45.055 INFO:tasks.workunit.client.0.smithi135.stderr:2021-08-17T03:12:45.052+0000 7f27d8c19700 1 -- 172.21.15.135:0/2440531935 >> v1:172.21.15.13:6789/0 conn(0x7f27b0012880 legacy=0x7f27b000e6b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.13:6789/0
2021-08-17T03:12:52.741 DEBUG:teuthology.orchestra.run:got remote process result: 124
2021-08-17T03:12:52.743 INFO:tasks.workunit:Stopping ['mon/test_mon_osdmap_prune.sh'] on client.0...
2021-08-17T03:12:52.743 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm ~~rf -~~ /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0
2021-08-17T03:12:52.992 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/run_tasks.py", line 91, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/run_tasks.py", line 70, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/github.com_ceph_ceph-c_a00b573203353a8db1f3e59f48e827ed27479b62/qa/tasks/workunit.py", line 134, in task
coverage_and_limits=not config.get('no_coverage_and_limits', None))
File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/parallel.py", line 84, in exit
for result in self:
File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/parallel.py", line 98, in next
resurrect_traceback(result)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/parallel.py", line 30, in resurrect_traceback
raise exc.exc_info¹
File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/parallel.py", line 23, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/github.com_ceph_ceph-c_a00b573203353a8db1f3e59f48e827ed27479b62/qa/tasks/workunit.py", line 426, in _run_tests
label="workunit test {workunit}".format(workunit=workunit)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/remote.py", line 509, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/run.py", line 455, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/run.py", line 183, in _raise_for_status
node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed (workunit test mon/test_mon_osdmap_prune.sh) on smithi135 with status 124: 'mkdir ~~p -~~ /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a00b573203353a8db1f3e59f48e827ed27479b62 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/test_mon_osdmap_prune.sh'

Teuthology logs: https://pulpito.ceph.com/yuriw-2021-08-16_21:15:00-rados-wip-yuri-testing-master-8.16.21-distro-basic-smithi/6341993/

Related issues 3 (0 open — 3 closed)

Actions

Copy link

Updated by Neha Ojha over 2 years ago

Subject changed from mon/test_mon_osdmap_prune.sh fails to qa/tasks/mon_thrash.py: _do_thrash AssertionError len(s['quorum']) == len(mons)

2021-08-17T00:22:05.475 ERROR:tasks.mon_thrash.mon_thrasher:exception:
Traceback (most recent call last):
  File "/home/teuthworker/src/github.com_ceph_ceph-c_a00b573203353a8db1f3e59f48e827ed27479b62/qa/tasks/mon_thrash.py", line 232, in do_thrash
    self._do_thrash()
  File "/home/teuthworker/src/github.com_ceph_ceph-c_a00b573203353a8db1f3e59f48e827ed27479b62/qa/tasks/mon_thrash.py", line 266, in _do_thrash
    assert len(s['quorum']) == len(mons)
AssertionError

Before this:

2021-08-17T00:21:52.681 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3
2021-08-17T00:21:52.681 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum
2021-08-17T00:21:52.682 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status
...
2021-08-17T00:21:53.071 INFO:teuthology.orchestra.run.smithi013.stdout:{
2021-08-17T00:21:53.071 INFO:teuthology.orchestra.run.smithi013.stdout:    "name": "a",
2021-08-17T00:21:53.071 INFO:teuthology.orchestra.run.smithi013.stdout:    "rank": 0,
2021-08-17T00:21:53.072 INFO:teuthology.orchestra.run.smithi013.stdout:    "state": "leader",
2021-08-17T00:21:53.072 INFO:teuthology.orchestra.run.smithi013.stdout:    "election_epoch": 42,
2021-08-17T00:21:53.072 INFO:teuthology.orchestra.run.smithi013.stdout:    "quorum": [
2021-08-17T00:21:53.072 INFO:teuthology.orchestra.run.smithi013.stdout:        0,
2021-08-17T00:21:53.072 INFO:teuthology.orchestra.run.smithi013.stdout:        1,
2021-08-17T00:21:53.073 INFO:teuthology.orchestra.run.smithi013.stdout:        2
2021-08-17T00:21:53.073 INFO:teuthology.orchestra.run.smithi013.stdout:    ],
...
2021-08-17T00:21:53.179 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status
...
2021-08-17T00:22:05.419 INFO:teuthology.orchestra.run.smithi013.stdout:{
2021-08-17T00:22:05.419 INFO:teuthology.orchestra.run.smithi013.stdout:    "name": "c",
2021-08-17T00:22:05.419 INFO:teuthology.orchestra.run.smithi013.stdout:    "rank": 2,
2021-08-17T00:22:05.420 INFO:teuthology.orchestra.run.smithi013.stdout:    "state": "peon",
2021-08-17T00:22:05.420 INFO:teuthology.orchestra.run.smithi013.stdout:    "election_epoch": 46,
2021-08-17T00:22:05.420 INFO:teuthology.orchestra.run.smithi013.stdout:    "quorum": [
2021-08-17T00:22:05.425 INFO:teuthology.orchestra.run.smithi013.stdout:        0,
2021-08-17T00:22:05.425 INFO:teuthology.orchestra.run.smithi013.stdout:        2
2021-08-17T00:22:05.426 INFO:teuthology.orchestra.run.smithi013.stdout:    ],

The assertion error is seen because peon mon.c reports 2 mons (not 3) in quorum.

Actions

Copy link

Updated by Laura Flores almost 2 years ago

Backport set to octopus

/a/yuriw-2022-05-13_14:13:55-rados-wip-yuri3-testing-2022-05-12-1609-oc 2022-05-14T10:26:57.539 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-05-14T10:26:57.540 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-05-14T10:26:57.540 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-05-14T10:26:57.540 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-05-14T10:26:57.540 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-05-14T10:26:57.541 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-05-14T10:26:57.541 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-05-14T10:26:57.541 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-05-14T10:26:57.541 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-05-14T10:26:57.542 INFO:teuthology.orchestra.run.smithi003.stdout: ... but mon.b only reports 2: 2022-05-14T10:27:10.038 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-05-14T10:27:10.038 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-05-14T10:27:10.038 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-05-14T10:27:10.039 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-05-14T10:27:10.039 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-05-14T10:27:10.039 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-05-14T10:27:10.039 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-05-14T10:27:10.040 INFO:teuthology.orchestra.run.smithi003.stdout: 2022-05-14T10:27:10.040 INFO:teuthology.orchestra.run.smithi003.stdout: ... Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_aa4a6ef62a4724eafd self._do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph-c_aa4a6ef62a4724eafd assert len(s['quorum']) == len(mons) AssertionError topus-distro-default-smithi/6832711

mon.c reports 3 mons in quorum: { "name": "c", "rank": 2, "state": "peon", "election_epoch": 90, "quorum": [ 0, 1, 2 ], { "name": "b", "rank": 1, "state": "peon", "election_epoch": 96, "quorum": [ 0, 1 ], 160ebcc111d3b5d7550b84/qa/tasks/mon_thrash.py", line 232, in do_thrash 160ebcc111d3b5d7550b84/qa/tasks/mon_thrash.py", line 266, in _do_thrash


  
  
    
    
      ActionsCopy link
      #3
    
    
      
      Updated by Laura Flores almost 2 years ago
      
      
    

    /a/yuriw-2022-06-13_16:36:31-rados-wip-yuri7-testing-2022-06-13-0706-distro-default-smithi/6876523


	Description: rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/sync-many workloads/rados_api_tests}
    
  
  
  
    
    
      ActionsCopy link
      #4
    
    
      
      Updated by Laura Flores about 1 year ago
      
      
    

    
       Translation missing: en.field_tag_list set to test-failure
    
    /a/lflores-2023-02-09_16:38:16-rados-wip-lflores-testing-2023-02-06-1529-distro-default-smithi/7164042
    
  
  
  
    
    
      ActionsCopy link
      #5
    
    
      
      Updated by Laura Flores about 1 year ago
      
      
    

    /a/yuriw-2023-03-08_23:00:31-rados-wip-yuri11-testing-2023-03-08-1220-distro-default-smithi/7198899
    
  
  
  
    
    
      ActionsCopy link
      #6
    
    
      
      Updated by Nitzan Mordechai 12 months ago
      
      
    

    
       Status changed from New to In Progress
       Assignee set to Nitzan Mordechai
    
    Since we are failing at the first assert of the check for quorum, and we had few iterators over the thrashing, it looks like we are not delaying enough between the thrashing.
we can see from the quorum_age that not all the monitors ticks update the quorum on the time of check.


	will increase the thrash_delay from 1 to the default of mon_tick_interval which is 5
    
  
  
  
    
    
      ActionsCopy link
      #7
    
    
      
      Updated by Nitzan Mordechai 12 months ago
      
      
    

    
       Status changed from In Progress to Fix Under Review
       Pull request ID set to 51570
    
    
    
  
  
  
    
    
      ActionsCopy link
      #8
    
    
      
      Updated by Radoslaw Zarzynski 12 months ago
      
      
    

    
       Status changed from Fix Under Review to Pending Backport
       Backport changed from octopus to pacific,quincy,reef
    
    
    
  
  
  
    
    
      ActionsCopy link
      #9
    
    
      
      Updated by Backport Bot 12 months ago
      
      
    

    
       Copied to Backport #61450: pacific: qa/tasks/mon_thrash.py: _do_thrash AssertionError len(s['quorum']) == len(mons) added
    
    
    
  
  
  
    
    
      ActionsCopy link
      #10
    
    
      
      Updated by Backport Bot 12 months ago
      
      
    

    
       Copied to Backport #61451: quincy: qa/tasks/mon_thrash.py: _do_thrash AssertionError len(s['quorum']) == len(mons) added
    
    
    
  
  
  
    
    
      ActionsCopy link
      #11
    
    
      
      Updated by Backport Bot 12 months ago
      
      
    

    
       Copied to Backport #61452: reef: qa/tasks/mon_thrash.py: _do_thrash AssertionError len(s['quorum']) == len(mons) added
    
    
    
  
  
  
    
    
      ActionsCopy link
      #12
    
    
      
      Updated by Backport Bot 12 months ago
      
      
    

    
       Tags set to backport_processed
    
    
    
  
  
  
    
    
      ActionsCopy link
      #13
    
    
      
      Updated by Konstantin Shalygin 6 months ago
      
      
    

    
       Status changed from Pending Backport to Resolved
       % Done changed from 0 to 100









Actions
  Copy link
  






Also available in:  Atom
  PDF

Project

General

Profile

Ceph » RADOS

Custom queries

Bug #52316

qa/tasks/mon_thrash.py: _do_thrash AssertionError len(s['quorum']) == len(mons)

Updated by Neha Ojha over 2 years ago

Updated by Laura Flores almost 2 years ago

Updated by Laura Flores almost 2 years ago

Updated by Laura Flores about 1 year ago

Updated by Laura Flores about 1 year ago

Updated by Nitzan Mordechai 12 months ago

Updated by Nitzan Mordechai 12 months ago

Updated by Radoslaw Zarzynski 12 months ago

Updated by Backport Bot 12 months ago

Updated by Backport Bot 12 months ago

Updated by Backport Bot 12 months ago

Updated by Backport Bot 12 months ago

Updated by Konstantin Shalygin 6 months ago