Ceph » CephFS

Should be fixed by https://tracker.ceph.com/issues/54049.

Xiubo Li wrote:

Should be fixed by https://tracker.ceph.com/issues/54049.

Looks the same. However, I'm not sure if the client used to crash with the bracktrace as in this tracker.

Venky,

Please check this one https://tracker.ceph.com/issues/56532. It should be the same bug with this one.

This only exist in the v17.1.0 and the logic has been changed after [1]²³ below. When trying remount to invalidate the kernel dcache, it may fail dues to the mount is still busy. After v17.1.0 it will be allowed to retry it at most of mds_max_retries_on_remount_failure time instead of crash the client in _do_remout() directly.

[1] https://tracker.ceph.com/issues/27657
[2] https://tracker.ceph.com/issues/54049
[2] https://tracker.ceph.com/issues/56532

This could be closed directly.

Xiubo Li wrote:

This only exist in the v17.1.0 and the logic has been changed after [1]²³ below. When trying remount to invalidate the kernel dcache, it may fail dues to the mount is still busy. After v17.1.0 it will be allowed to retry it at most of mds_max_retries_on_remount_failure time instead of crash the client in _do_remout() directly.

[1] https://tracker.ceph.com/issues/27657
[2] https://tracker.ceph.com/issues/54049
[2] https://tracker.ceph.com/issues/56532

This could be closed directly.

I might be wrong, actually the [2] introduced one bug, that is when the remount failed for more than mds_max_retries_on_remount_failure time we should abort the client to avoid the client cannot do invalidate the dentry cache and then the MDS cache cannot shrink which can cause the MDS to fail.

Went through the kernel code I couldn't find any case in our case could cause the failure.

And from https://tracker.ceph.com/issues/56532#note-1:

$ grep 'to trim kernel dentries' client.* -rn
client.0.774355.log:98:2022-07-14T14:24:03.984+0530 7fbbd5ffb640 -1 client.4495 failed to remount (to trim kernel dentries): return code = 1
client.0.792060.log:137:2022-07-14T14:34:49.030+0530 7f46017fa640 -1 client.10512 failed to remount (to trim kernel dentries): return code = 32
client.0.793575.log:137:2022-07-14T14:35:32.033+0530 7f779aff5640 -1 client.10990 failed to remount (to trim kernel dentries): return code = 32
client.0.795072.log:137:2022-07-14T14:36:15.771+0530 7f899e7fc640 -1 client.11499 failed to remount (to trim kernel dentries): return code = 32
client.1.774461.log:98:2022-07-14T14:24:05.570+0530 7f766bfff640 -1 client.4505 failed to remount (to trim kernel dentries): return code = 1

The return code 1 is:

1      incorrect invocation or permissions

And from util-linux, the 1 is util-linux, just suppose we may passed invalidate options. And then went through the client/Client.cc and ceph_fuse.cc code more carefully, found that it's possible we may do use-after-free for the mountpoint:

1254 static int remount_cb(void *handle)
1255 { 
1256   // used for trimming kernel dcache. when remounting a file system, linux kernel
1257   // trims all unused dentries in the file system
1258   char cmd[128+PATH_MAX];
1259   CephFuse::Handle *cfuse = (CephFuse::Handle *)handle;
1260   snprintf(cmd, sizeof(cmd), "LIBMOUNT_FSTAB=/dev/null mount -i -o remount %s",                                                                   
1261 #if FUSE_VERSION >= FUSE_MAKE_VERSION(3, 0)
1262                   cfuse->opts.mountpoint);
1263 #else
1264                   cfuse->mountpoint);
1265 #endif
1266   int r = system(cmd);
1267   if (r != 0 && r != -1) {    
1268     r = WEXITSTATUS(r);
1269   }
1270   
1271   return r;
1272 } 
1273   

Because in ceph_fuse.cc it will do client->unmount() and then do cfuse->finalize(), which will free the mountpoint and then call client->shutdown(), which will stop the remount_finisher queue.

 92 int main(int argc, const char **argv, const char *envp[]) {

...
333   out_client_unmount:
334     client->unmount();                                                                                                                             
335     cfuse->finalize();
336   out_shutdown:
337     icp.stop();
338     client->shutdown();
339   out_init_failed:
...
}

The remount_finisher queue is a queue will run the remount_cb().

That means after the mountpoint freed the remount_finisher maybe not empty.