Ceph » CephFS

IMO, for this we can lower down the concurent threads 128 --> 32 and can try it serveral times. From my test without Jeff's shutdown race fixing patches, it is very easy to reproduce with even 10 concurent threads.

Xiubo Li wrote:

IMO, for this we can lower down the concurent threads 128 --> 32 and can try it serveral times. From my test without Jeff's shutdown race fixing patches, it is very easy to reproduce with even 10 concurent threads.

On my box, the hard limit for open files is 1048576. It seems sort of crazy that we end up with 8k fds per thread. I have to wonder if there is a fd leak or something.

Does teuthology have a lower rlim_max for this? It might be good in the interim to change this code to dump out what the RLIMIT_NOFILE value is so we can sanity check this.

Jeff Layton wrote:

Xiubo Li wrote:

IMO, for this we can lower down the concurent threads 128 --> 32 and can try it serveral times. From my test without Jeff's shutdown race fixing patches, it is very easy to reproduce with even 10 concurent threads.

On my box, the hard limit for open files is 1048576. It seems sort of crazy that we end up with 8k fds per thread. I have to wonder if there is a fd leak or something.

That's possible.

Does teuthology have a lower rlim_max for this? It might be good in the interim to change this code to dump out what the RLIMIT_NOFILE value is so we can sanity check this.

This and perhaps a dump of `/proc/self/fd` so we can see what fds are in use and how they are growing (Detect a leak).

Logging the RLIMIT_NOFILE we set should be no problem.

It may be tough to get a file descriptor in the same process to open /proc/self/fd when this occurs. This is also occurring via exception, AFAICT, so we don't really know what the testcase was doing when it triggered.

Do we get coredumps when these failures occur? We might be better off opening those up in gdb and seeing what was going on.

We still have the coredump from this test failure, but the x86_64 binaries have been reaped so we can't analyze it. I've proposed a PR to make the test log the RLIMIT_NOFILE before we start the test, but I don't really expect it to be smaller than 1M.

What we'll probably need to do is just wait for this to happen again, and take a look at the open file table in the coredump. That might give us a clearer picture of why this keeps happening.

It seems the tests will fire many event works, which will open many fds, the last issue about this was cause by the event work code.

Xiubo Li wrote:

It seems the tests will fire many event works, which will open many fds, the last issue about this was cause by the event work code.

Maybe that is it. I haven't looked at the event handling code in detail.

I wonder too if it's possible that some fds are being lazily cleaned up after ceph_shutdown() is called, and so we end up with a bunch of unused fds still open after several loop iterations.

Debugging this is tricky since we can't count on being able to open new files from the process once this happens. I looked at the core, but getting proper debuginfo from the testcase is is hard since that's built during the test. We'd need a separate watchdog process to run lsof on the testcase when this occurs.

Reducing the number of threads sounds like a reasonable thing to do for now. I'll also note that I tried to run this test in a podman container and I hit cgroup thread limits. I'll start with 32 threads.