Actions
Bug #7139
opendead tcp connections during connection negotiation may lead to hangs
% Done:
0%
Source:
Development
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):
Description
Nothing too suspicious in the dmesg:
[ 5098.025842] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5098.148598] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5098.241094] libceph: osd1 10.214.135.140:6810 socket closed (con state OPEN) [ 5098.357377] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5098.372193] libceph: osd2 10.214.135.140:6801 socket closed (con state CONNECTING) [ 5098.450011] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5098.457433] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5098.467518] libceph: osd3 10.214.137.118:6811 socket closed (con state CONNECTING) [ 5098.555893] libceph: osd0 10.214.135.140:6800 socket closed (con state OPEN) [ 5098.575337] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5098.733510] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5098.800067] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5098.807896] libceph: osd5 10.214.137.118:6800 socket closed (con state NEGOTIATING) [ 5099.135166] libceph: osd1 10.214.135.140:6810 socket closed (con state OPEN) [ 5099.599966] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5099.640669] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5099.655517] libceph: osd2 10.214.135.140:6801 socket closed (con state CONNECTING) [ 5100.430724] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5100.494605] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5100.722882] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5100.800884] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5100.847695] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5100.861701] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5102.051646] libceph: osd0 10.214.135.140:6800 socket closed (con state OPEN) [ 5102.073919] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5102.105544] libceph: osd1 10.214.135.140:6810 socket closed (con state OPEN) [ 5102.113505] libceph: osd1 10.214.135.140:6810 socket closed (con state CONNECTING) [ 5102.571221] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5102.628832] libceph: osd4 10.214.137.118:6806 socket closed (con state OPEN) [ 5102.640384] libceph: osd4 10.214.137.118:6806 socket closed (con state NEGOTIATING) [ 5102.673485] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5102.710885] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5102.727342] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5102.743146] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5102.758222] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5102.813666] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5102.824270] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5102.835681] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5102.844449] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5102.857421] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5102.871608] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5102.897644] libceph: osd1 10.214.135.140:6810 socket closed (con state OPEN) [ 5102.911173] libceph: osd1 10.214.135.140:6810 socket closed (con state OPEN) [ 5102.927357] libceph: osd1 10.214.135.140:6810 socket closed (con state OPEN) [ 5102.937252] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5102.950653] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5102.965256] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5102.980300] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5102.997695] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5103.258028] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5103.415366] libceph: osd4 10.214.137.118:6806 socket closed (con state OPEN) [ 5103.432445] libceph: osd4 10.214.137.118:6806 socket closed (con state OPEN) [ 5103.450256] libceph: osd0 10.214.135.140:6800 socket closed (con state OPEN) [ 5103.575996] libceph: osd0 10.214.135.140:6800 socket closed (con state OPEN) [ 5103.586414] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5103.598168] libceph: osd0 10.214.135.140:6800 socket closed (con state OPEN) [ 5103.690927] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5103.895867] libceph: osd1 10.214.135.140:6810 socket closed (con state OPEN) [ 5103.932000] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5103.947610] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5103.961415] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5103.972534] libceph: osd2 10.214.135.140:6801 socket closed (con state NEGOTIATING) [ 5104.039621] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5104.053701] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5104.067095] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5104.080263] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5104.091906] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5104.102006] libceph: osd3 10.214.137.118:6811 socket closed (con state CONNECTING) [ 5104.195537] libceph: osd1 10.214.135.140:6810 socket closed (con state OPEN) [ 5104.209328] libceph: osd1 10.214.135.140:6810 socket closed (con state OPEN) [ 5104.254971] libceph: osd0 10.214.135.140:6800 socket closed (con state OPEN) [ 5104.336663] libceph: osd0 10.214.135.140:6800 socket closed (con state OPEN) [ 5104.348690] libceph: osd0 10.214.135.140:6800 socket closed (con state OPEN) [ 5104.360047] libceph: osd0 10.214.135.140:6800 socket closed (con state OPEN) [ 5104.412737] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5104.420384] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5104.431733] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5104.447075] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5104.460807] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5104.472447] libceph: osd3 10.214.137.118:6811 socket closed (con state NEGOTIATING) [ 5104.657574] libceph: osd0 10.214.135.140:6800 socket closed (con state OPEN) [ 5104.669905] libceph: osd0 10.214.135.140:6800 socket closed (con state OPEN) [ 5104.681815] libceph: osd0 10.214.135.140:6800 socket closed (con state OPEN) [ 5104.693929] libceph: osd0 10.214.135.140:6800 socket closed (con state OPEN) [ 5104.706516] libceph: osd0 10.214.135.140:6800 socket closed (con state OPEN) [ 5104.720341] libceph: osd0 10.214.135.140:6800 socket closed (con state OPEN) [ 5104.862544] libceph: osd0 10.214.135.140:6800 socket closed (con state OPEN) [ 5104.928707] libceph: osd0 10.214.135.140:6800 socket closed (con state OPEN) [ 5104.990845] libceph: osd0 10.214.135.140:6800 socket closed (con state OPEN) [ 5105.003574] libceph: osd0 10.214.135.140:6800 socket closed (con state OPEN) [ 5105.080290] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5105.151979] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5105.174872] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5105.188595] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5105.200425] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5105.209389] libceph: osd5 10.214.137.118:6800 socket closed (con state NEGOTIATING) [ 5105.281461] libceph: osd0 10.214.135.140:6800 socket closed (con state OPEN) [ 5105.314909] libceph: osd4 10.214.137.118:6806 socket closed (con state OPEN) [ 5105.412843] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5105.424958] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5105.438760] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5105.453470] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5105.465574] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5105.478010] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5105.487983] libceph: osd3 10.214.137.118:6811 socket closed (con state NEGOTIATING) [ 5105.511040] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5105.523820] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5105.614595] libceph: osd1 10.214.135.140:6810 socket closed (con state OPEN) [ 5105.765591] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5106.078535] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5106.207335] libceph: osd4 10.214.137.118:6806 socket closed (con state OPEN) [ 5106.214523] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5106.222437] libceph: osd4 10.214.137.118:6806 socket closed (con state CONNECTING) [ 5106.303083] libceph: osd1 10.214.135.140:6810 socket closed (con state OPEN) [ 5106.324583] libceph: osd0 10.214.135.140:6800 socket closed (con state OPEN) [ 5106.335553] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5106.409447] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5106.417026] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5106.426765] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5106.438423] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5106.445964] libceph: osd2 10.214.135.140:6801 socket closed (con state OPEN) [ 5106.458375] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5106.484096] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5106.496926] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5106.509655] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5106.519872] libceph: osd3 10.214.137.118:6811 socket closed (con state NEGOTIATING) [ 5106.543784] libceph: osd0 10.214.135.140:6800 socket closed (con state OPEN) [ 5106.609766] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5106.618302] libceph: osd5 10.214.137.118:6800 socket closed (con state NEGOTIATING) [ 5107.408045] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5107.419109] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5107.430492] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5107.442760] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5107.457489] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5107.464938] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5107.475003] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5107.482565] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5107.495374] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5107.584191] libceph: osd0 10.214.135.140:6800 socket closed (con state OPEN) [ 5107.686589] libceph: osd0 10.214.135.140:6800 socket closed (con state OPEN) [ 5107.939780] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5107.957655] libceph: osd0 10.214.135.140:6800 socket closed (con state OPEN) [ 5108.016870] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5108.030903] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5108.057509] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5108.070417] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5108.084448] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5108.097539] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5108.111334] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5108.124832] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5108.250686] libceph: osd0 10.214.135.140:6800 socket closed (con state OPEN) [ 5108.428992] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5108.441859] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5108.451195] libceph: osd0 10.214.135.140:6800 socket closed (con state OPEN) [ 5108.458490] libceph: osd5 10.214.137.118:6800 socket closed (con state CONNECTING) [ 5108.586855] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5108.598022] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5108.744464] libceph: osd4 10.214.137.118:6806 socket closed (con state OPEN) [ 5109.045563] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5109.054636] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5109.064023] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5109.166271] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5109.174872] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5109.183485] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5109.196264] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5109.204838] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5109.226753] libceph: osd1 10.214.135.140:6810 socket closed (con state OPEN) [ 5109.304206] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5109.313212] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5109.321318] libceph: osd3 10.214.137.118:6811 socket closed (con state NEGOTIATING) [ 5109.410153] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5109.423732] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5109.434367] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5109.531307] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5109.540051] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5109.548952] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5109.591881] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5109.600609] libceph: osd5 10.214.137.118:6800 socket closed (con state CONNECTING) [ 5110.406015] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5110.415183] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5110.422462] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5110.431208] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5110.438463] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5110.447142] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5110.454417] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5110.462849] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5110.470108] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5110.521715] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5110.534760] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5110.548948] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5110.562669] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5110.572077] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5110.581371] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5110.589594] libceph: osd5 10.214.137.118:6800 socket closed (con state CONNECTING) [ 5110.783743] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5110.826682] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5110.835791] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5110.843665] libceph: osd3 10.214.137.118:6811 socket closed (con state NEGOTIATING) [ 5111.405678] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5111.414925] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5111.425431] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5111.457101] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5111.467574] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5111.579619] libceph: osd1 10.214.135.140:6810 socket closed (con state OPEN) [ 5111.665909] libceph: osd1 10.214.135.140:6810 socket closed (con state OPEN) [ 5111.689322] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5111.969165] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5111.979138] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5111.988965] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5111.998549] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5112.008848] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5112.019917] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5112.031394] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5112.183188] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5112.195072] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5112.205947] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5112.240789] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5112.249314] libceph: osd5 10.214.137.118:6800 socket closed (con state CONNECTING) [ 5112.419413] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5112.428474] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5112.437305] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5112.699846] libceph: osd0 10.214.135.140:6800 socket closed (con state OPEN) [ 5112.806825] libceph: osd1 10.214.135.140:6810 socket closed (con state OPEN) [ 5113.409197] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5113.417957] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5113.426753] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5113.435578] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5113.449030] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5113.486317] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5113.495329] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5113.503761] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5113.512353] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5113.521010] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5113.548793] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5113.604737] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5113.614149] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5113.623189] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5113.657498] libceph: osd1 10.214.135.140:6810 socket closed (con state OPEN) [ 5113.789057] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5113.798114] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5113.806569] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5114.075775] libceph: osd3 10.214.137.118:6811 socket closed (con state OPEN) [ 5114.099620] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5114.151283] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5161.819242] libceph: osd4 10.214.137.118:6806 socket closed (con state OPEN) [ 5161.826468] libceph: osd5 10.214.137.118:6800 socket closed (con state OPEN) [ 5271.979146] libceph: mon1 10.214.137.118:6789 socket closed (con state OPEN) [ 5271.986241] libceph: mon1 10.214.137.118:6789 session lost, hunting for new mon [ 5271.997013] libceph: mon2 10.214.135.140:6790 session established [ 5278.827358] INFO: task dd:25643 blocked for more than 120 seconds. [ 5278.833634] Not tainted 3.13.0-rc3-ceph-00058-gdbf7fcc #17 [ 5278.839723] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 5278.847669] dd D ffff88040cfb3fe0 0 25643 25606 0x00000000 [ 5278.854841] ffff880358d539a8 0000000000000086 ffff880345780000 0000000000000000 [ 5278.862491] ffff88043fd130c0 ffff8804297e0000 00000000000130c0 ffff880358d53fd8 [ 5278.870131] 00000000000130c0 ffff8804297e0000 ffff880358d539a8 ffff88043fd13978 [ 5278.877781] Call Trace: [ 5278.880320] [<ffffffff816b1959>] schedule+0x29/0x70 [ 5278.885346] [<ffffffff816b1a2f>] io_schedule+0x8f/0xd0 [ 5278.890632] [<ffffffff811d2364>] do_blockdev_direct_IO+0xda4/0x2040 [ 5278.897067] [<ffffffffa0342940>] ? __xfs_get_blocks+0x580/0x580 [xfs] [ 5278.903651] [<ffffffff811d3655>] __blockdev_direct_IO+0x55/0x60 [ 5278.909725] [<ffffffffa0342940>] ? __xfs_get_blocks+0x580/0x580 [xfs] [ 5278.916318] [<ffffffffa034149c>] xfs_vm_direct_IO+0x16c/0x170 [xfs] [ 5278.922744] [<ffffffffa0342940>] ? __xfs_get_blocks+0x580/0x580 [xfs] [ 5278.929330] [<ffffffff81198753>] ? __sb_end_write+0x33/0x70 [ 5278.935043] [<ffffffff811b5578>] ? __mnt_want_write+0x58/0x70 [ 5278.940933] [<ffffffff811308eb>] generic_file_aio_read+0x6cb/0x710 [ 5278.947299] [<ffffffff811d0832>] ? dio_complete+0xb2/0x140 [ 5278.952933] [<ffffffff81096558>] ? up_read+0x18/0x30 [ 5278.958066] [<ffffffffa039e62c>] ? xfs_ilock+0x17c/0x190 [xfs] [ 5278.964118] [<ffffffffa034ede0>] xfs_file_aio_read+0x1c0/0x2e0 [xfs] [ 5278.970629] [<ffffffff81195d0a>] do_sync_read+0x5a/0x90 [ 5278.975992] [<ffffffff81196d7b>] vfs_read+0xab/0x180 [ 5278.981097] [<ffffffff81197002>] SyS_read+0x52/0xa0 [ 5278.986120] [<ffffffff816b8ade>] ? do_page_fault+0xe/0x10 [ 5278.991668] [<ffffffff816bd096>] system_call_fastpath+0x1a/0x1f ...
~85 tasks hung, all because of osd2:
48162 osd2 2.4aade9fc rb.0.100b.38437fdb.000000000077 read 48166 osd2 2.19bf5c94 rb.0.100b.38437fdb.0000000000ae read 48168 osd2 2.3f6453b0 rb.0.100b.38437fdb.0000000000c1 read 48172 osd2 2.29c6b76c rb.0.100b.38437fdb.000000000084 read 48176 osd2 2.72c6f84e rb.0.100b.38437fdb.0000000000c2 read 48182 osd2 2.6388e7b0 rb.0.100b.38437fdb.000000000034 read 48185 osd2 2.a5a655b0 rb.0.100b.38437fdb.0000000000c8 read 48186 osd2 2.2528df9c rb.0.100b.38437fdb.0000000000f9 read 48189 osd2 2.f31bda8e rb.0.100b.38437fdb.0000000000d9 read 48190 osd2 2.d2f0ea54 rb.0.100b.38437fdb.0000000000f3 read 48192 osd2 2.6b532bdc rb.0.100b.38437fdb.000000000086 read 48195 osd2 2.af2e22a1 rb.0.100b.38437fdb.0000000000ca read 48201 osd2 2.35bd913c rb.0.100b.38437fdb.000000000041 read 48205 osd2 2.36b3bdac rb.0.100b.38437fdb.000000000064 read 48208 osd2 2.15b711b0 rb.0.100b.38437fdb.000000000028 read 48211 osd2 2.fff3a36e rb.0.100b.38437fdb.0000000000d1 read 48212 osd2 2.44de4614 rb.0.100b.38437fdb.000000000037 read 48215 osd2 2.a5a655b0 rb.0.100b.38437fdb.0000000000c8 read 48218 osd2 2.e75699a1 rb.0.100b.38437fdb.00000000002e read 48222 osd2 2.13b03e0e rb.0.100b.38437fdb.0000000000e6 read 48225 osd2 2.41bc206c rb.0.100b.38437fdb.0000000000b5 read 48226 osd2 2.3f2437f0 rb.0.100b.38437fdb.000000000082 read 48234 osd2 2.efa4a59e rb.0.100b.38437fdb.0000000000a5 read 48236 osd2 2.197508fc rb.0.100b.38437fdb.000000000059 read 48239 osd2 2.f3c7a101 rb.0.100b.38437fdb.000000000050 read 48247 osd2 2.1687ea54 rb.0.100b.38437fdb.000000000091 read 48250 osd2 2.a4f74df4 rb.0.100b.38437fdb.0000000000a9 read 48252 osd2 2.d66f89dc rb.0.100b.38437fdb.00000000009c read 48254 osd2 2.40857bd4 rb.0.100b.38437fdb.0000000000fd read 48255 osd2 2.bc647370 rb.0.100b.38437fdb.000000000081 read 48263 osd2 2.7a7a23fc rb.0.100b.38437fdb.000000000039 read 48265 osd2 2.3cabe4ae rb.0.100b.38437fdb.0000000000b3 read 48268 osd2 2.7477e3dc rb.0.100b.38437fdb.000000000095 read 48269 osd2 2.44ccacd4 rb.0.100b.38437fdb.0000000000a6 read 48273 osd2 2.29c6b76c rb.0.100b.38437fdb.000000000084 read 48279 osd2 2.3de52bc rb.0.100b.38437fdb.000000000087 read 48281 osd2 2.c78b847c rb.0.100b.38437fdb.0000000000e5 read 48283 osd2 2.c33ababc rb.0.100b.38437fdb.00000000009a read 48286 osd2 2.da53abbe rb.0.100b.38437fdb.0000000000cd read 48287 osd2 2.1f000d2c rb.0.100b.38437fdb.00000000008b read 48288 osd2 2.71e795a1 rb.0.100b.38437fdb.000000000044 read 48289 osd2 2.49fdb5b0 rb.0.100b.38437fdb.000000000031 read 48290 osd2 2.de246abc rb.0.100b.38437fdb.000000000071 read 48291 osd2 2.d2f0ea54 rb.0.100b.38437fdb.0000000000f3 read 48292 osd2 2.de246abc rb.0.100b.38437fdb.000000000071 read 48302 osd2 2.20f71b90 rb.0.100b.38437fdb.000000000042 read 48305 osd2 2.491ddbee rb.0.100b.38437fdb.000000000024 read 48352 osd2 2.6b532bdc rb.0.100b.38437fdb.000000000086 read 48364 osd2 2.c78b847c rb.0.100b.38437fdb.0000000000e5 read 48434 osd2 2.a4f74df4 rb.0.100b.38437fdb.0000000000a9 read 48521 osd2 2.36b3bdac rb.0.100b.38437fdb.000000000064 read 48536 osd2 2.29c6b76c rb.0.100b.38437fdb.000000000084 read 48544 osd2 2.41bc206c rb.0.100b.38437fdb.0000000000b5 read 48547 osd2 2.af2e22a1 rb.0.100b.38437fdb.0000000000ca read 48555 osd2 2.31377e6e rb.0.100b.38437fdb.000000000094 read 48561 osd2 2.6b532bdc rb.0.100b.38437fdb.000000000086 read 48563 osd2 2.c33ababc rb.0.100b.38437fdb.00000000009a read 48565 osd2 2.4aade9fc rb.0.100b.38437fdb.000000000077 read 48580 osd2 2.efa4a59e rb.0.100b.38437fdb.0000000000a5 read 48612 osd2 2.46de4150 rb.0.100b.38437fdb.000000000063 read 48671 osd2 2.c78b847c rb.0.100b.38437fdb.0000000000e5 read 48696 osd2 2.d2f0ea54 rb.0.100b.38437fdb.0000000000f3 read 48870 osd2 2.41bc206c rb.0.100b.38437fdb.0000000000b5 read 48905 osd2 2.efa4a59e rb.0.100b.38437fdb.0000000000a5 read 48926 osd2 2.491ddbee rb.0.100b.38437fdb.000000000024 read 48937 osd2 2.362255bc rb.0.100b.38437fdb.0000000000d0 read 48959 osd2 2.52893b9e rb.0.100b.38437fdb.000000000046 read 48980 osd2 2.88ddda0c rb.0.100b.38437fdb.000000000073 read 49004 osd2 2.1850bd9c rb.0.100b.38437fdb.00000000005c read 49011 osd2 2.696eab7e rb.0.100b.38437fdb.00000000000f read 49083 osd2 2.a5a655b0 rb.0.100b.38437fdb.0000000000c8 read 49095 osd2 2.af2e22a1 rb.0.100b.38437fdb.0000000000ca read 49120 osd2 2.197508fc rb.0.100b.38437fdb.000000000059 read 49124 osd2 2.3cabe4ae rb.0.100b.38437fdb.0000000000b3 read 49140 osd2 2.c78b847c rb.0.100b.38437fdb.0000000000e5 read 49145 osd2 2.a5a655b0 rb.0.100b.38437fdb.0000000000c8 read 49146 osd2 2.c33ababc rb.0.100b.38437fdb.00000000009a read 49149 osd2 2.d2f0ea54 rb.0.100b.38437fdb.0000000000f3 read 49173 osd2 2.197508fc rb.0.100b.38437fdb.000000000059 read 49179 osd2 2.35bd913c rb.0.100b.38437fdb.000000000041 read 49208 osd2 2.bacf1350 rb.0.100b.38437fdb.00000000001b read 49209 osd2 2.491ddbee rb.0.100b.38437fdb.000000000024 read 49216 osd2 2.35bd913c rb.0.100b.38437fdb.000000000041 read 49217 osd2 2.197508fc rb.0.100b.38437fdb.000000000059 read 49254 osd2 2.f31bda8e rb.0.100b.38437fdb.0000000000d9 write 49255 osd2 2.f31bda8e rb.0.100b.38437fdb.0000000000d9 write 49398 osd2 2.4e2d49cc test_image-0.rbd 3'25769803776 watch
osd2 seems to be doing nothing other than exchanging pings with other osds.
Updated by Ilya Dryomov over 10 years ago
- Project changed from rbd to Linux kernel client
- Subject changed from krbd: xfstest 224 hung to dead tcp connections during connection negotiation may lead to hangs
Updated by Ilya Dryomov over 10 years ago
- Status changed from New to 12
Reproduced it a couple days ago, kernel client had a half-open socket:
10.214.136.140 == mira091 (kernel client) 10.214.135.140 == mira073 (osd2, two mons) ubuntu@mira091:~$ sudo netstat -anp | grep 10.214.135.140 tcp 0 0 10.214.136.140:55422 10.214.135.140:6801 ESTABLISHED - <-- half-open tcp 0 0 10.214.136.140:46237 10.214.135.140:6789 ESTABLISHED - ubuntu@mira073:~$ sudo netstat -anp | grep 10.214.136.140 tcp 0 0 10.214.135.140:6789 10.214.136.140:46237 ESTABLISHED 8420/ceph-mon
Updated by Ilya Dryomov over 10 years ago
- Status changed from 12 to Fix Under Review
Updated by Ilya Dryomov about 3 years ago
- Related to Bug #48895: ceph: krbd client con get stuck in CON_SOCK_STATE_CONNECTING state added
Updated by Ilya Dryomov about 3 years ago
- Category set to libceph
- Status changed from Fix Under Review to In Progress
- Priority changed from Normal to Urgent
Updated by Ilya Dryomov about 3 years ago
- Related to deleted (Bug #48895: ceph: krbd client con get stuck in CON_SOCK_STATE_CONNECTING state)
Updated by Ilya Dryomov about 3 years ago
- Has duplicate Bug #48895: ceph: krbd client con get stuck in CON_SOCK_STATE_CONNECTING state added
Actions