Ceph » Messengers

I have a script that I'm been using to try and recreate this issue. I ran just the COnnectionRaceTest 2500 times without an issue or pause. I think changed the script to run the whole set of ceph_test_msgr tests over and over again until it fails, with the thinking that maybe one of the other tests doesn't always clean up after itself.. well left it going to day. Currently at 585 times and still no failure.

Either my script isn't stopping (but I've used it before, it'll stop on a return code > 0).

So maybe the error is more subtle? Maybe happens in teuthology envs?

• left it going for a few days.

Sage's teuthology run took place in November 2019. The problem might have been fixed since then.

Another possibility is that the bug only ever existed in Sage's wip branch.

Be that as it may, I looked up the run in question:

https://pulpito.ceph.com/sage-2019-11-07_22:38:52-rados-wip-sage-testing-2019-11-07-1412-distro-basic-smithi/4480741/

Now, there's nothing particularly special about it. It deploys ceph on a single-node cluster with the following roles:

[u'mon.a', u'mgr.x', u'osd.0', u'osd.1', u'client.0']

Once the cluster gets to HEALTH_OK, it runs the following two binaries from the ceph-test RPM:

ceph_test_async_driver
ceph_test_msgr

The logs are gone. Most likely they were lost in the recent mass-deletion event on the LRC.

Cool, thanks Nathan. Will create a new script that will use sesdev (though Sage probably uses centos/redhat) to create a single node deploy and run test binaries. Let's see what happens :)

Nathan Cutler wrote:
...

[...]

Although, I thought the `ceph_test_msgr` just ran the msgr unit tests in general, not actually tested against the built cluster. But it's been a few weeks since I looked at this, and a lots happened in that time. So will just revisit it and see :)

As I'm pretty sure `ceph_test_msgr` is the binary I've been running without a cluster (just compiled).

Although, I thought the `ceph_test_msgr` just ran the msgr unit tests in general, not actually tested against the built cluster. But it's been a few weeks since I looked at this, and a lots happened in that time. So will just revisit it and see :)

As I'm pretty sure `ceph_test_msgr` is the binary I've been running without a cluster (just compiled).

The developers try pretty hard to keep the unit tests (in master) in a state where they are always passing. (PRs are not supposed to be merged unless they pass the "make check" Jenkins test, for example.) So... if it's failing (in master) I doubt it's a unit test.

To investigate that possibility further I ran the following commands in top-level directory of the master source tree:

find src/ -name '*CMakeLists*' -exec grep -3 -H add_ceph_test {} \; | grep ceph_test_msgr
find src/ -name '*.sh' -exec grep ceph_test_msgr {} \;

The first command looks for a unit test that runs the ceph_test_msgr executable directly.

Since many unit tests are implemented as shell scripts, the second command looks for shell scripts that trigger ceph_test_msgr.

Neither of these commands returned any output, so at this point I cannot say for sure that ceph_test_msgr is used in any unit test.

It's also not clear how useful it would be to test the messenger in a unit test which, by definition, only ever runs on a single node?