Ceph » RADOS

This failure seems sporadic. I reran the same job that failed 50 times, and all succeeded except one, which failed for a different reason.
http://pulpito.front.sepia.ceph.com/lflores-2023-02-09_19:24:50-rados-wip-lflores-testing-2023-02-06-1529-distro-default-smithi/

    def primary_affinity(self, osd=None):
        self.log("primary_affinity")
        if osd is None:
            osd = random.choice(self.in_osds)
        if random.random() >= .5:
            pa = random.random()
        elif random.random() >= .5:
            pa = 1
        else:
            pa = 0
        self.log('Setting osd %s primary_affinity to %f' % (str(osd), pa))
        self.ceph_manager.raw_cluster_cmd('osd', 'primary-affinity',
                                          str(osd), str(pa))

>>> random.choice(None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.6/random.py", line 258, in choice
    i = self._randbelow(len(seq))
TypeError: object of type 'NoneType' has no len()
>>> random.choice([])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.6/random.py", line 260, in choice
    raise IndexError('Cannot choose from an empty sequence') from None
IndexError: Cannot choose from an empty sequence

So self.in_osds is [].

class OSDThrasher(Thrasher):
    """ 
    Object used to thrash Ceph
    """ 
    def __init__(self, manager, config, name, logger):
        # ...
        self.in_osds = osd_status['in']

So the question is why those OSDs haven't joined the cluster.

While thrashing OSDs in _do_thrash() (qa/tasks/ceph_manager.py), the task didn't add OSDs back into the cluster after killing/removing them:

lflores@teuthology:/a/lflores-2023-02-08_20:25:06-rados-wip-lflores-testing-2023-02-06-1529-distro-default-smithi/7162184$ cat teuthology.log | grep "Adding osd" 
2023-02-09T06:16:22.945 INFO:tasks.thrashosds.thrasher:Adding osd 1
2023-02-09T06:19:34.997 INFO:tasks.thrashosds.thrasher:Adding osd 7

There should be way more lines of the task adding OSDs back into the cluster. The thrashing code normally checks if there are out OSDs and adds them back in. Continuing to dig into why this didn't happen as expected.