Ceph » Linux kernel client

Using v17.2.7 with linux kernel version 5.15.0-105-generic, we've been having issues with rsync hanging on writes, and apache hanging on reads of files on our ceph storage. Rsync just quit with no logging info, but apache provided the following trace in dmesg:

[ 3772.321407] libceph: osd103 (1)172.100.200.5:6881 socket closed (con state V1_BANNER)
[ 3867.731198] INFO: task apache2:2369 blocked for more than 1208 seconds.
[ 3867.731210] Not tainted 5.15.0-105-generic #115-Ubuntu
[ 3867.731213] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3867.731215] task:apache2 state:D stack: 0 pid: 2369 ppid: 1029 flags:0x00000002
[ 3867.731223] Call Trace:
[ 3867.731225] <TASK>
[ 3867.731242] __schedule+0x24e/0x590
[ 3867.731250] ? kvm_sched_clock_read+0x11/0x20
[ 3867.731256] schedule+0x69/0x110
[ 3867.731260] rwsem_down_write_slowpath+0x230/0x3e0
[ 3867.731265] ? kmem_cache_free+0x24f/0x290
[ 3867.731271] ? putname+0x59/0x70
[ 3867.731277] down_write+0x47/0x60
[ 3867.731298] ceph_llseek+0x47/0x120 [ceph]
[ 3867.731325] ksys_lseek+0x85/0xc0
[ 3867.731330] __x64_sys_lseek+0x18/0x20
[ 3867.731333] do_syscall_64+0x59/0xc0
[ 3867.731337] ? exit_to_user_mode_prepare+0x96/0xb0
[ 3867.731341] ? syscall_exit_to_user_mode+0x35/0x50
[ 3867.731344] ? do_syscall_64+0x69/0xc0
[ 3867.731346] ? do_syscall_64+0x69/0xc0
[ 3867.731349] ? do_syscall_64+0x69/0xc0
[ 3867.731351] entry_SYSCALL_64_after_hwframe+0x62/0xcc
[ 3867.731354] RIP: 0033:0x7fd69a01691b
[ 3867.731362] RSP: 002b:00007ffeab5af9c8 EFLAGS: 00000202 ORIG_RAX: 0000000000000008
[ 3867.731365] RAX: ffffffffffffffda RBX: 00007fd696abfb88 RCX: 00007fd69a01691b
[ 3867.731367] RDX: 0000000000000000 RSI: 00000000237f585e RDI: 0000000000000010
[ 3867.731369] RBP: 00007ffeab5afa90 R08: 0000000000000010 R09: 00000000237f585e
[ 3867.731370] R10: 00007fd699f127b8 R11: 0000000000000202 R12: 0000000000000000
[ 3867.731372] R13: 0000000000000000 R14: 00007fd696ede048 R15: 000000000bb11c87
[ 3867.731375] </TASK>

(The libceph: osd103 (1)172.100.200.5:6881 socket closed messages occurred about 1208 seconds before each trace, I've just included the last one before the selected trace here)
The trace appears to correspond to https://elixir.bootlin.com/linux/v5.15.105/source/fs/ceph/file.c#L1902 and looks to me (by no means a systems programmer) like the ceph_llseek waits forever trying to lock the inode.

Talking to our administrators, it sounds like they use 172.100.200.0/24 for internal communication among the nodes, but it is not accessible by the client (and to make matters worse, actually routes to some poor user's cable modem in upstate New York) all of the other OSDs had correctly routable addresses. Adding a manual routing rule to the client immediately resolved the issue.

Expected behavior (given the mis-configured network): libceph prevents mounting of FS with OSD which it can't communicate with, or at the very least throws an IO error when trying to access stuff on that OSD.