Ceph » Linux kernel client

Is there anything else in dmesg? What is general state of the cluster, any warnings in ceph -s, osd logs?

Do you use rbd snapshots and/or clones?

How exactly are you increasing the number of PGs (commands you are issuing)? Are those errors transient or do they persist?

Can you reproduce with "debug ms = 1" set on the OSDs?

We doesn't use snapshots and/or clones.

our dmsg log is clean, and doens't have any suspicious records(only boot log, thats all). When rbd erros happens last time (we launch massivly rsync). ceph will by in health state, like now:

    cluster df060192-7369-42dd-bb9c-db18bb6f3355
     health HEALTH_OK
     monmap e5: 6 mons at {ceph1=192.168.30.101:6789/0,ceph2=192.168.30.102:6789/0,ceph3=192.168.30.103:6789/0,ceph4=192.168.30.104:6789/0,ceph5=192.168.30.105:6789/0,ceph6=192.168.30.106:6789/0}
            election epoch 222, quorum 0,1,2,3,4,5 ceph1,ceph2,ceph3,ceph4,ceph5,ceph6
     osdmap e5982: 49 osds: 49 up, 49 in
            flags sortbitwise
      pgmap v1069321: 1024 pgs, 1 pools, 6193 GB data, 1564 kobjects
            12411 GB used, 38762 GB / 51174 GB avail
                1024 active+clean
  client io 6403 B/s wr, 1 op/s

We found follow warn in osd logs when we make massive rsync

2016-02-26 11:06:25.969815 7f16750af700  0 log_channel(cluster) log [WRN] : client.157872 192.168.31.220:0/3211780995 misdirected client.157872.1:795379 pg 2
.ebfbb119 to osd.19 not [27,19] in e5982/5982
2016-02-26 11:06:26.046774 7f16750af700  0 log_channel(cluster) log [WRN] : client.157872 192.168.31.220:0/3211780995 misdirected client.157872.1:795381 pg 2
.ebfbb119 to osd.19 not [27,19] in e5982/5982
2016-02-26 11:06:26.800276 7f16728aa700  0 log_channel(cluster) log [WRN] : client.157872 192.168.31.220:0/3211780995 misdirected client.157872.1:795390 pg 2
.c5547424 to osd.19 not [22,19] in e5982/5982
2016-02-26 11:06:26.907875 7f16750af700  0 log_channel(cluster) log [WRN] : client.157872 192.168.31.220:0/3211780995 misdirected client.157872.1:795393 pg 2
.c5547424 to osd.19 not [22,19] in e5982/5982
2016-02-26 11:06:51.353003 7f16728aa700  0 log_channel(cluster) log [WRN] : client.157872 192.168.31.220:0/3211780995 misdirected client.157872.1:795598 pg 2
.76468424 to osd.19 not [22,19] in e5982/5982
2016-02-26 11:06:51.558168 7f1669e82700  0 -- 192.168.30.103:6805/1974 >> 192.168.30.101:6825/2555 pipe(0x7f169c13e000 sd=97 :6805 s=0 pgs=0 cs=0 l=0 c=0x7f1
6aa043b80).accept connect_seq 325 vs existing 325 state standby
2016-02-26 11:06:51.558786 7f1669e82700  0 -- 192.168.30.103:6805/1974 >> 192.168.30.101:6825/2555 pipe(0x7f169c13e000 sd=97 :6805 s=0 pgs=0 cs=0 l=0 c=0x7f1
6aa043b80).accept connect_seq 326 vs existing 325 state standby
2016-02-26 11:06:52.166995 7f16728aa700  0 log_channel(cluster) log [WRN] : client.157872 192.168.31.220:0/3211780995 misdirected client.157872.1:795601 pg 2
.76468424 to osd.19 not [22,19] in e5982/5982
2016-02-26 11:15:17.370983 7f165f45f700  0 -- 192.168.30.103:6805/1974 >> 192.168.30.101:6817/2398 pipe(0x7f169f2de000 sd=211 :6805 s=0 pgs=0 cs=0 l=0 c=0x7f
16aa0426e0).accept connect_seq 351 vs existing 351 state standby

192.168.31.220 is an ip of rbd maped instance

Hmm. Can you do the same for each from 5970 to 5981 and attach a .tar with those?

Was the massive rsync you've provided warnings for done during rebalancing or after rebalancing completed? Do you have more instances of these warnings? If so, provide all of them, please.

Could you also attach more osdmaps? 5982..6000 and, say, 5800..5969?

This seems to be more serious than I thought. Could you send over as many osdmaps going back from 5800 as possible (they are periodically trimmed, but I think you should be able to get down to ~5600)? Another thing that could help is the full syslog for Feb 26 +/- a day.

What about a syslog for the night of Feb 26 - the attached one starts at ~7 o'clock? I need to see syslogs for the entire 26th from both 31.220 and 30.4.

Sorry but more old logs alredy rotated and we haven't them

It's not immediately obvious what the problem is - there are a few different possibilities and not enough information to go on. Would you be willing to try to reproduce this with logging enabled? Ideally, I would like you to install a custom kernel (.deb package built on our gitbuilders) on 31.220 and 30.4 (or at least one of them) and enable more verbose logging on the OSD side. If installing a custom kernel is something you can't do, verbose OSD logs and some limited logs from the kernel you've got installed might still help...

We will try! But where we can get custom kernels, and whats is scenario of reproduce? Set weight of some osd to 0? Or something else?

If I had a reproducer, I wouldn't be asking you to reproduce ;) I would suggest trying to replicate what you described above as close as possible; changing weights to trigger rebalancing is a good idea. I tried those things on my local cluster but didn't see a single misdirected request - probably because my cluster is too small or I'm doing something different from what you did.

Would you be open to trying to replicate one of the occurrences (increasing PG count, OSD failure, adding OSDs) on that same cluster without installing a custom kernel, but with logging enabled? "debug ms = 1" for MONs and OSDs and

# modprobe rbd
# echo 'file osdmap.c +pt' >/sys/kernel/debug/dynamic_debug/control
# echo 'file osd_client.c +pt' >/sys/kernel/debug/dynamic_debug/control
# echo 'format get_osd -pt' >/sys/kernel/debug/dynamic_debug/control
# echo 'format put_osd -pt' >/sys/kernel/debug/dynamic_debug/control

on the kernel client boxes? That might be just enough and we won't have to bother with a custom kernel.

Hello

We have not very new hdd that may fail quite often, so what you need from us to hel to you? Debug output consume very big space, so we can't hold it alltime.

Nothing - I think I fixed it. The patch just needs to be reviewed and will be merged ASAP, hopefully into the final 4.7 kernel.

Perhaps a stupid question. Have we any chanse to use try this new rbd driver on kernel 4.4? We use Ubuntu 14.4 and this it difficult to us use unstable kernels

Not stupid at all. The fix is set to be backported to 3.15+, it might take a bit more time for it to propagate to Ubuntu kernels, but it'll get there eventually.

libceph: apply new_state before new_up_client on incrementals

merged into 4.7-final. Additional backport for 3.10-3.14 submitted.

Is there any way to know when backport will hit 4.4 Ubuntu kernel?

A couple of weeks from now? It's up to the Ubuntu kernel team.

In 3.16.39.