Ceph » Linux kernel client

This weekend i tried to rsync kernel.org to my cluster, just to get some data on it.

At first it all seems to go fine, until i started to rain the following messages in my dmesg:

[89636.033739] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c32dc8 1099511929817
[89636.033743] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c33320 1099511929818
[89636.033746] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c33878 1099511929819
[89636.033750] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c33dd0 1099511929820
[89636.033754] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c34328 1099511929821
[89636.033758] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c34880 1099511929822
[89636.033761] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c34dd8 1099511929823
[89636.033765] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c35330 1099511929824
[89636.033769] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c35888 1099511929825
[89636.033773] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c36338 1099511929826
[89636.033777] ceph:  dropping dirty+flushing AxFx state for ffff8800b0c36890 1099511929827
[89636.033781] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c36de8 1099511929828
[89636.033785] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c37340 1099511929829
[89636.033788] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c37898 1099511929830
[89636.033792] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c38308 1099511929831
[89636.033796] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c38860 1099511929832
[89636.033800] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c38db8 1099511929833
[89636.033804] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c39310 1099511929834
[89636.033808] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c39868 1099511929835
[89636.033811] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c39dc0 1099511929836
[89636.033815] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c3a318 1099511929837
[89636.033819] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c3a870 1099511929838
[89636.033822] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c3adc8 1099511929839
[89636.033826] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c3b320 1099511929840
[89636.033830] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c3b878 1099511929841
[89636.033834] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c3bdd0 1099511929842
[89636.033838] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c3c328 1099511929843
[89636.033842] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c3c880 1099511929844
[89636.033845] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c3cdd8 1099511929845
[89636.033849] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c3d330 1099511929846
[89636.033853] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c3d888 1099511929847
[89636.033857] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c3dde0 1099511929848
[89636.033861] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c3e338 1099511929849
[89636.033865] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c3e890 1099511929850
[89636.033868] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c3ede8 1099511929851
[89636.033872] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c3f340 1099511929852
[89636.033876] ceph:  dropping dirty+flushing AxFxw state for ffff8800b0c3f898 1099511929853
[89636.033880] ceph:  dropping dirty+flushing AxFxw state for ffff880027540308 1099511929854
[89636.033883] ceph:  dropping dirty+flushing AxFxw state for ffff880027540860 1099511929855
[89636.033887] ceph:  dropping dirty+flushing AxFxw state for ffff880027540db8 1099511929856
[89636.033891] ceph:  dropping dirty+flushing AxFxw state for ffff880027541310 1099511929857
[89636.033895] ceph:  dropping dirty+flushing AxFxw state for ffff880027541868 1099511929858
[89636.033898] ceph:  dropping dirty+flushing AxFxw state for ffff880027541dc0 1099511929859
[89636.033902] ceph:  dropping dirty+flushing AxFxw state for ffff880027542318 1099511929860
[89636.033906] ceph:  dropping dirty+flushing AxFxw state for ffff880027542870 1099511929861
[89636.033909] ceph:  dropping dirty+flushing AxFxw state for ffff880027542dc8 1099511929862
[89636.033913] ceph:  dropping dirty+flushing AxFxw state for ffff880027543320 1099511929863
[89636.033917] ceph:  dropping dirty+flushing AxFxw state for ffff880027543878 1099511929864
[89636.033921] ceph:  dropping dirty+flushing AxFxw state for ffff880027543dd0 1099511929865
[89636.033924] ceph:  dropping dirty+flushing AxFxw state for ffff880027544328 1099511929866
[89636.033928] ceph:  dropping dirty+flushing AxFxw state for ffff880027544880 1099511929867
[89636.033932] ceph:  dropping dirty+flushing AxFxw state for ffff880027544dd8 1099511929868
[89636.033935] ceph:  dropping dirty+flushing AxFxw state for ffff880027545330 1099511929869
[89636.033939] ceph:  dropping dirty+flushing AxFxw state for ffff880027545888 1099511929870
[89636.033943] ceph:  dropping dirty+flushing AxFxw state for ffff880027545de0 1099511929871
[89636.033947] ceph:  dropping dirty+flushing AxFxw state for ffff880027546338 1099511929872
[89636.033951] ceph:  dropping dirty+flushing AxFxw state for ffff880027546890 1099511929873
[89636.033955] ceph:  dropping dirty+flushing AxFxw state for ffff880027546de8 1099511929874
[89636.033958] ceph:  dropping dirty+flushing AxFxw state for ffff880027547340 1099511929875
[89636.033962] ceph:  dropping dirty+flushing AxFxw state for ffff880027547898 1099511929876
[89636.033966] ceph:  dropping dirty+flushing AxFxw state for ffff880027548308 1099511929877
[89636.033970] ceph:  dropping dirty+flushing AxFxw state for ffff880027548860 1099511929878
[89636.033973] ceph:  dropping dirty+flushing AxFxw state for ffff880027548db8 1099511929879
[89636.033977] ceph:  dropping dirty+flushing AxFxw state for ffff880027549310 1099511929880
[89636.033981] ceph:  dropping dirty+flushing AxFxw state for ffff880027549868 1099511929881
[89636.033985] ceph:  dropping dirty+flushing AxFxw state for ffff880027549dc0 1099511929882
[89636.033988] ceph:  dropping dirty+flushing AxFxw state for ffff88002754a318 1099511929883
[89636.033992] ceph:  dropping dirty+flushing AxFxw state for ffff88002754a870 1099511929884
[89636.033996] ceph:  dropping dirty+flushing AxFxw state for ffff88002754adc8 1099511929885
[89636.033999] ceph:  dropping dirty+flushing AxFxw state for ffff88002754b320 1099511929886
[89636.034003] ceph:  dropping dirty+flushing AxFxw state for ffff88002754b878 1099511929887
[89636.034007] ceph:  dropping dirty+flushing AxFxw state for ffff88002754bdd0 1099511929888
[89636.034010] ceph:  dropping dirty+flushing AxFxw state for ffff88002754c328 1099511929889
[89636.034014] ceph:  dropping dirty+flushing AxFxw state for ffff88002754c880 1099511929890
[89636.034018] ceph:  dropping dirty+flushing Fx state for ffff88005bb37898 1099511928796
[89636.034022] ceph:  dropping dirty+flushing Fx state for ffff88005bb35330 1099511927341
[89636.034026] ceph:  dropping dirty+flushing AxFxw state for ffff88002754cdd8 1099511929891
[89636.034030] ceph:  dropping dirty+flushing AxFxw state for ffff88002754d330 1099511929892
[89636.034034] ceph:  dropping dirty+flushing AxFxw state for ffff88002754d888 1099511929893
[89636.034038] ceph:  dropping dirty+flushing AxFxw state for ffff88002754dde0 1099511929894
[89636.034042] ceph:  dropping dirty+flushing AxFxw state for ffff88002754e338 1099511929895
[89636.034045] ceph:  dropping dirty+flushing AxFxw state for ffff88002754e890 1099511929896
[89636.034049] ceph:  dropping dirty+flushing AxFxw state for ffff88002754ede8 1099511929897
[89636.034059] ceph:  dropping dirty+flushing Fx state for ffff88007b39cdd8 1099511633194
[89636.034064] ceph:  dropping dirty+flushing Fx state for ffff88007b380860 1099511927313

This kept on going, until the rsync stalled.

I was able to cancel the rsync with CTRL+C, but then i tried a "sync", which then failed:

[146281.380053] INFO: task sync:11834 blocked for more than 120 seconds.
[146281.381687] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message

Reading data from the cluster was still possible, but writes all hung.

After a reboot of the client, i could continue the rsync and it finished.

Kernel: 2.6.35-rc1
Client: master branch (Running IPv6 only)