Ceph » Linux kernel client

I will also note here that enabling lockdep in the kernel on the client side makes the problem go away for some reason. No logs or reports from lockdep either.

Sorry about that - I've raised the upload issue internally. For now, just email the logs directly to me.

You've trimmed too much - I need the context for tid 322452, so I'm looking for "client.1246958" ":322452" or so. The easiest thing would be to compress and send the whole log from 2016-07-26 23:06:46 to 2016-07-27 00:56:59.

Also, just to validate the log - that image has hundreds of snapshots, right?

I'll attach the wider log to an email.

Yes, it has 697 snapshots. The overall average snapshot count per rdb is 355. The lowest number is 3 and the highest number is 708. Because of our load testing the number of snapshots could

Interestingly enough we can't map that rbd anymore.

$ rbd info test-rbd/3f370dbabff91bbb7ff23ae7a96e5cb414cac3408013cefed6d4b627b5eed9c7-willnotmap
rbd image '3f370dbabff91bbb7ff23ae7a96e5cb414cac3408013cefed6d4b627b5eed9c7-willnotmap':
size 9536 MB in 2385 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.11e601379f78e7
format: 2
features: layering, striping
flags:
stripe unit: 4096 kB
stripe count: 1

When we try to mount it, it hangs here:

setsockopt(3, SOL_SOCKET, SO_PASSCRED, [1], 4) = 0
open("/sys/bus/rbd/add_single_major", O_WRONLY) = 4
write(4, "192.168.2.63:6789,192.168.3.63:6789,"..., 147

Other rbds mount fine.

The cluster health is HEALTH_OK.

From Cliff:

Sifting through the osd2's log for "d960431b30e2f" I see this:

2016-07-27 00:47:47.199252 7f1ac7dbb700 1 -- 10.9.228.101:6814/25892 <== client.1252427 10.9.72.23:0/273648291 13 ==== osd_op(client.1252427.0:19 rbd_header.d960431b30e2f [call lock.lock] 37.a819967 ondisk+write+known_if_redirected e107453) v6 ==== 210+0+51 (4053684278 0 659001106) 0x55c9ec362000 con 0x55c9f0101600
2016-07-27 00:47:47.199515 7f1ad3539700 1 -- 10.9.228.101:6814/25892 --> 10.9.72.23:0/273648291 -- osd_op_reply(19 rbd_header.d960431b30e2f [call lock.lock] v0'0 uv0 ondisk = 16 ((16) Device or resource busy)) v6 - ?+0 0x55c9ee4b8b00 con 0x55c9f0101600

Because of our load testing the number of snapshots could

very well make it to 10k per rbd.

That's exactly the problem.

For obscure implementation reasons, kernel client currently supports only 510 snapshots per image. Mapping an image with more than that hangs

[ 4986.527680] libceph: get_reply osd2 tid 322452 data 5332 > preallocated 4092, skipping

because the reply message is simply dropped on the floor and "rbd map" needs to be Ctrl-C'ed. I guess you just didn't notice it between all those ext4 warnings.

Lifting this limit is on the TODO list, but having this many snapshots is going to affect your bandwidth, among other things - snapshot contexts are transmitted with every write.
If you aren't just stress-testing and have a real-world use case, please share it.

Thanks, I did see those messages but since there's so much noise from all the activity it was hard to go hey that's the problem specially since it looks harmless.

For our use case we're going to limit it to 300ish snapshots for now and we may end up going down to 50ish snapshots (which we do need them for various reasons) after we do some benchmarking.

As for the EXT4 errors we have code to fsck them as part of the mounting process, it's just disabled so we could see what kind of corruptions we could see. Most of them seem to be from all the hangs (that forced us to reset systems to unmap them) that have been damaging file systems.