2015-01-18T22:04:42+01:00 kaa-35 kernel: [253032.495748] ceph: mds0 caps stale 2015-01-18T22:04:50+01:00 kaa-35 kernel: [253040.364505] ceph: mds0 reconnect start 2015-01-18T22:04:50+01:00 kaa-35 kernel: [253040.774964] ceph: mds0 reconnect success 2015-01-18T22:05:42+01:00 kaa-35 kernel: [253092.544327] ceph: mds0 caps stale 2015-01-18T22:06:02+01:00 kaa-35 kernel: [253112.560527] ceph: mds0 caps stale 2015-01-18T22:09:52+01:00 kaa-35 kernel: [253342.730742] ceph: mds0 hung 2015-01-18T22:28:06+01:00 kaa-35 kernel: [254437.879175] ceph: mds0 reconnect start 2015-01-18T22:28:08+01:00 kaa-35 kernel: [254439.742705] ceph: mds0 reconnect success 2015-01-18T22:29:02+01:00 kaa-35 kernel: [254493.677913] ceph: mds0 caps stale 2015-01-18T22:29:22+01:00 kaa-35 kernel: [254513.694101] ceph: mds0 caps stale 2015-01-18T22:33:12+01:00 kaa-35 kernel: [254743.864335] ceph: mds0 hung 2015-01-18T22:54:06+01:00 kaa-35 kernel: [255999.645870] ceph: mds0 recovery completed 2015-01-18T22:54:07+01:00 kaa-35 kernel: [255999.734940] ceph: mds0 came back 2015-01-18T22:54:07+01:00 kaa-35 kernel: [255999.908427] ceph: mds0 caps renewed 2015-01-19T03:11:54+01:00 kaa-35 kernel: [271479.852745] libceph: wrong peer, want 192.168.1.54:6800/10216, got 192.168.1.54:6800/13878 2015-01-19T03:11:54+01:00 kaa-35 kernel: [271479.852749] libceph: osd27 192.168.1.54:6800 socket error on read 2015-01-19T03:11:55+01:00 kaa-35 kernel: [271480.430621] libceph: wrong peer, want 192.168.1.54:6800/10216, got 192.168.1.54:6800/13878 2015-01-19T03:11:55+01:00 kaa-35 kernel: [271480.430626] libceph: osd27 192.168.1.54:6800 socket error on read 2015-01-19T03:11:56+01:00 kaa-35 kernel: [271481.369314] libceph: wrong peer, want 192.168.1.54:6800/10216, got 192.168.1.54:6800/13878 2015-01-19T03:11:56+01:00 kaa-35 kernel: [271481.369318] libceph: osd27 192.168.1.54:6800 socket error on read 2015-01-19T03:11:58+01:00 kaa-35 kernel: [271483.344892] libceph: wrong peer, want 192.168.1.54:6800/10216, got 192.168.1.54:6800/13878 2015-01-19T03:11:58+01:00 kaa-35 kernel: [271483.344896] libceph: osd27 192.168.1.54:6800 socket error on read 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.545623] libceph: wrong peer, want 192.168.1.67:6800/3497, got 192.168.1.67:6800/11207 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.545626] libceph: osd44 192.168.1.67:6800 socket error on read 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.875097] libceph: osd6 weight 0x10000 (in) 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.875273] libceph: osd6 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.876008] libceph: osd38 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.876065] libceph: osd38 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.876262] libceph: osd38 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.876318] libceph: osd38 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.876549] libceph: osd38 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.876591] libceph: osd38 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.876909] libceph: osd38 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.876952] libceph: osd38 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.877091] libceph: osd38 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.877125] libceph: osd38 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.877560] libceph: osd38 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.877589] libceph: osd38 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.878183] libceph: osd38 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.878200] libceph: osd38 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.878524] libceph: osd38 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.878610] libceph: osd38 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.879064] libceph: osd38 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.879127] libceph: osd38 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.879647] libceph: osd38 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.879662] libceph: osd38 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.880562] libceph: osd38 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.880621] libceph: osd38 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.880826] libceph: wrong peer, want 192.168.1.47:6800/3982, got 192.168.1.47:6800/18077 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.880830] libceph: osd36 192.168.1.47:6800 socket error on read 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.880857] libceph: osd38 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.880900] libceph: osd38 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.880942] libceph: osd74 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.880985] libceph: osd74 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881054] libceph: osd38 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881097] libceph: osd38 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881163] libceph: osd78 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881178] libceph: osd8 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881179] libceph: osd9 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881181] libceph: osd69 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881182] libceph: osd71 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881183] libceph: osd72 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881185] libceph: osd73 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881186] libceph: osd74 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881187] libceph: osd75 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881189] libceph: osd76 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881190] libceph: osd77 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881191] libceph: osd79 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881193] libceph: osd80 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881194] libceph: osd81 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881209] libceph: osd50 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881216] libceph: osd51 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881219] libceph: osd58 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881220] libceph: osd59 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881222] libceph: osd60 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881223] libceph: osd70 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881238] libceph: osd5 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881240] libceph: osd6 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881241] libceph: osd17 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881242] libceph: osd18 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881244] libceph: osd35 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881245] libceph: osd36 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881246] libceph: osd37 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881248] libceph: osd38 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881249] libceph: osd39 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881250] libceph: osd41 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881252] libceph: osd42 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881253] libceph: osd43 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881254] libceph: osd44 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881255] libceph: osd45 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881257] libceph: osd46 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881258] libceph: osd47 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881259] libceph: osd49 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881261] libceph: osd53 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881262] libceph: osd56 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881263] libceph: osd57 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881265] libceph: osd62 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881266] libceph: osd63 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881267] libceph: osd64 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881269] libceph: osd65 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881270] libceph: osd66 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881272] libceph: osd67 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881273] libceph: osd68 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881292] libceph: osd4 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881293] libceph: osd12 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881294] libceph: osd31 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881296] libceph: osd32 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881297] libceph: osd33 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881298] libceph: osd34 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881314] libceph: osd25 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881315] libceph: osd27 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881317] libceph: osd30 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881318] libceph: osd40 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881319] libceph: osd48 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881350] libceph: osd50 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881364] libceph: osd13 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881379] libceph: osd17 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881381] libceph: osd18 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881396] libceph: osd12 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881397] libceph: osd73 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881399] libceph: osd74 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881400] libceph: osd77 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881401] libceph: osd79 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881403] libceph: osd80 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881418] libceph: osd37 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881419] libceph: osd38 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881420] libceph: osd51 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881422] libceph: osd71 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881423] libceph: osd72 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881425] libceph: osd75 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881426] libceph: osd76 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881428] libceph: osd78 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881429] libceph: osd81 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881431] libceph: osd1 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881432] libceph: osd11 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881447] libceph: osd1 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881453] libceph: osd11 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881458] libceph: osd35 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881463] libceph: osd16 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881469] libceph: osd55 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881490] libceph: osd16 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881491] libceph: osd30 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881493] libceph: osd31 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881494] libceph: osd34 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881495] libceph: osd36 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881497] libceph: osd53 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881498] libceph: osd55 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881500] libceph: osd59 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881501] libceph: osd69 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881505] libceph: osd0 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881510] libceph: osd10 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881516] libceph: osd14 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881521] libceph: osd29 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881526] libceph: osd52 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881532] libceph: osd54 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881550] libceph: osd0 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881551] libceph: osd10 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881555] libceph: osd14 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881558] libceph: osd29 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881560] libceph: osd32 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881563] libceph: osd52 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881565] libceph: osd54 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881567] libceph: osd58 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881570] libceph: osd60 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881572] libceph: osd70 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881574] libceph: osd15 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881577] libceph: osd61 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881592] libceph: osd15 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881594] libceph: osd39 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881597] libceph: osd56 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881599] libceph: osd61 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881614] libceph: osd25 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881617] libceph: osd43 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881619] libceph: osd44 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881621] libceph: osd47 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881624] libceph: osd3 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881626] libceph: osd23 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881642] libceph: osd3 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881649] libceph: osd4 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881649] libceph: osd23 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881650] libceph: osd41 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881650] libceph: osd42 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881651] libceph: osd45 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881652] libceph: osd46 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881652] libceph: osd62 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881653] libceph: osd2 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881667] libceph: osd2 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881668] libceph: osd27 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881671] libceph: osd33 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881673] libceph: osd40 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881675] libceph: osd49 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881678] libceph: osd21 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881694] libceph: osd5 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881694] libceph: osd6 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881695] libceph: osd9 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881697] libceph: osd13 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881700] libceph: osd21 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881702] libceph: osd19 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881704] libceph: osd22 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881720] libceph: osd8 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881721] libceph: osd19 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881721] libceph: osd22 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881722] libceph: osd7 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881722] libceph: osd28 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881739] libceph: osd7 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881740] libceph: osd28 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881741] libceph: osd57 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881741] libceph: osd20 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881756] libceph: osd20 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881756] libceph: osd24 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881771] libceph: osd24 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881772] libceph: osd48 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881773] libceph: osd26 down 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881788] libceph: osd26 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881843] libceph: osd63 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881845] libceph: osd64 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881874] libceph: osd66 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881875] libceph: osd68 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881890] libceph: osd65 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.881892] libceph: osd67 up 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.882049] BUG: unable to handle kernel NULL pointer dereference at (null) 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.886009] IP: [] rb_erase+0x1e5/0x294 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.886016] PGD 1023e4f067 PUD 1026023067 PMD 0 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.886018] Oops: 0002 [#1] SMP 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.886043] Modules linked in: cbc ceph libceph ipmi_watchdog adm1026 w83795 w83793 hwmon_vid jc42 8021q garp stp mrp llc autofs4 btrfs xor raid6_pq ipmi_devintf mgag200 syscopyarea sysfillrect sysimgblt ttm x86_pkg_temp_thermal drm_kms_helper coretemp iTCO_wdt drm crc32c_intel iTCO_vendor_support microcode lpc_ich ipmi_si evdev pcspkr i2c_i801 mfd_core ipmi_msghandler rtc_cmos ioatdma processor dca thermal_sys button rpcsec_gss_krb5 fuse nfsv4 nfs af_packet hid_generic usbhid hid sd_mod bonding ehci_pci ehci_hcd ahci libahci usbcore libata usb_common ipv6 dm_mirror dm_region_hash dm_log dm_mod unix 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.886045] CPU: 6 PID: 8518 Comm: kworker/6:1 Tainted: P O 3.14.28-gentoo #2 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.886046] Hardware name: Supermicro X9DRT-HF+/X9DRT-HF+, BIOS 3.0c 05/21/2014 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.886056] Workqueue: ceph-msgr con_work [libceph] 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.886057] task: ffff880f5d96b1e0 ti: ffff880aaaeb8000 task.ti: ffff880aaaeb8000 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.886059] RIP: 0010:[] [] rb_erase+0x1e5/0x294 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.886060] RSP: 0018:ffff880aaaeb9cd8 EFLAGS: 00010282 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.886060] RAX: ffff8808cdc55019 RBX: ffff8808cdc55018 RCX: 0000000000000000 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.886061] RDX: ffff88100c06a7e8 RSI: ffff8808cdc51818 RDI: ffff8808cdc55018 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.886062] RBP: ffff880aaaeb9cf0 R08: 0000000000000000 R09: ffffea0040415c00 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.886062] R10: ffff880aaaeb9cc0 R11: 000000000000b842 R12: ffff88100c06a7e8 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.886063] R13: 0000000000000000 R14: ffff881023604060 R15: ffffffffa0163100 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.886064] FS: 0000000000000000(0000) GS:ffff88103fac0000(0000) knlGS:0000000000000000 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.886065] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.886066] CR2: 0000000000000000 CR3: 0000000b0d582000 CR4: 00000000001407e0 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.886066] Stack: 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.886067] ffff881023604000 ffff88100c06a768 ffff88100c06a768 ffff880aaaeb9d10 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.886069] ffffffffa0149c68 ffff881023604000 ffff8810236044a0 ffff880aaaeb9d40 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.886070] ffffffffa0149cea ffff88100c06a768 ffff881023604000 ffff880aaaeb9d70 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.886070] Call Trace: 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.886080] [] __remove_osd+0x51/0x73 [libceph] 2015-01-19T03:11:59+01:00 kaa-35 kernel: [271484.886087] [] __reset_osd+0x60/0x117 [libceph] 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271484.886093] [] osd_reset+0x9f/0x248 [libceph] 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271484.886099] [] con_work+0x2140/0x2374 [libceph] 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271484.886103] [] ? arch_vtime_task_switch+0x87/0x8c 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271484.886106] [] ? vtime_common_task_switch+0x25/0x28 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271484.886108] [] process_one_work+0x154/0x221 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271484.886110] [] worker_thread+0x13e/0x1d7 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271484.886112] [] ? cancel_delayed_work_sync+0x10/0x10 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271484.886113] [] kthread+0xb2/0xba 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271484.886115] [] ? __kthread_parkme+0x62/0x62 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271484.886118] [] ret_from_fork+0x7c/0xb0 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271484.886119] [] ? __kthread_parkme+0x62/0x62 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271484.886131] Code: 89 08 e9 ab 00 00 00 48 8b 73 10 f6 06 01 75 27 4c 8b 6e 08 48 89 d8 31 c9 48 83 c8 01 4c 89 e2 48 89 df 4c 89 6b 10 48 89 5e 08 <49> 89 45 00 e8 30 f9 ff ff 4c 89 ee 48 8b 4e 10 48 85 c9 74 05 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271484.886133] RIP [] rb_erase+0x1e5/0x294 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271484.886133] RSP 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271484.886134] CR2: 0000000000000000 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271484.886135] ---[ end trace 0077f2b8b8f10084 ]--- 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002283] BUG: unable to handle kernel paging request at ffffffffffffffd8 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002287] IP: [] kthread_data+0xb/0x11 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002289] PGD 180c067 PUD 180e067 PMD 0 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002291] Oops: 0000 [#2] SMP 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002317] Modules linked in: cbc ceph libceph ipmi_watchdog adm1026 w83795 w83793 hwmon_vid jc42 8021q garp stp mrp llc autofs4 btrfs xor raid6_pq ipmi_devintf mgag200 syscopyarea sysfillrect sysimgblt ttm x86_pkg_temp_thermal drm_kms_helper coretemp iTCO_wdt drm crc32c_intel iTCO_vendor_support microcode lpc_ich ipmi_si evdev pcspkr i2c_i801 mfd_core ipmi_msghandler rtc_cmos ioatdma processor dca thermal_sys button rpcsec_gss_krb5 fuse nfsv4 nfs af_packet hid_generic usbhid hid sd_mod bonding ehci_pci ehci_hcd ahci libahci usbcore libata usb_common ipv6 dm_mirror dm_region_hash dm_log dm_mod unix 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002320] CPU: 6 PID: 8518 Comm: kworker/6:1 Tainted: P D O 3.14.28-gentoo #2 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002321] Hardware name: Supermicro X9DRT-HF+/X9DRT-HF+, BIOS 3.0c 05/21/2014 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002332] task: ffff880f5d96b1e0 ti: ffff880aaaeb8000 task.ti: ffff880aaaeb8000 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002334] RIP: 0010:[] [] kthread_data+0xb/0x11 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002335] RSP: 0018:ffff880aaaeb9910 EFLAGS: 00010002 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002336] RAX: 0000000000000000 RBX: ffff88103fad0d80 RCX: 000000000000000f 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002337] RDX: 000000012503e618 RSI: 0000000000000006 RDI: ffff880f5d96b1e0 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002338] RBP: ffff880aaaeb9910 R08: ffff88103fad0df8 R09: ffffffff8199fa00 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002339] R10: ffff880aaaeb98b8 R11: 000000000000b9f4 R12: 0000000000000006 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002340] R13: 0000000000000001 R14: ffff88103fad0d80 R15: 0000000000000006 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002342] FS: 0000000000000000(0000) GS:ffff88103fac0000(0000) knlGS:0000000000000000 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002343] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002344] CR2: 0000000000000028 CR3: 0000000dd0f2a000 CR4: 00000000001407e0 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002344] Stack: 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002346] ffff880aaaeb9930 ffffffff8104c385 ffff88103fad0d80 ffff880f5d96b728 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002348] ffff880aaaeb9990 ffffffff813c902c ffff88102503e618 ffff880aaaeb9fd8 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002350] ffff880f5d96b1e0 0000000000010d80 ffff880f5d96b1e0 ffff880f5d96b1e0 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002351] Call Trace: 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002354] [] wq_worker_sleeping+0xf/0x83 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002358] [] __schedule+0x14e/0x545 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002360] [] schedule+0x69/0x6b 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002363] [] do_exit+0x915/0x92d 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002367] [] oops_end+0x7d/0x82 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002371] [] no_context+0x26b/0x278 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002374] [] __bad_area_nosemaphore+0x62/0x1b2 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002377] [] bad_area_nosemaphore+0xe/0x10 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002379] [] __do_page_fault+0x175/0x34a 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002382] [] ? enqueue_task_fair+0xa5/0xac 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002385] [] ? check_preempt_curr+0x42/0x65 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002388] [] ? account_entity_dequeue+0x73/0xad 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002392] [] ? native_sched_clock+0x35/0x37 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002394] [] ? native_sched_clock+0x35/0x37 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002398] [] ? sched_clock_cpu+0x1a/0x9e 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002401] [] ? arch_vtime_task_switch+0x87/0x8c 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002404] [] do_page_fault+0x1e/0x57 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002406] [] page_fault+0x22/0x30 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002409] [] ? rb_erase+0x1e5/0x294 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002420] [] __remove_osd+0x51/0x73 [libceph] 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002430] [] __reset_osd+0x60/0x117 [libceph] 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002439] [] osd_reset+0x9f/0x248 [libceph] 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002447] [] con_work+0x2140/0x2374 [libceph] 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002450] [] ? arch_vtime_task_switch+0x87/0x8c 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002453] [] ? vtime_common_task_switch+0x25/0x28 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002456] [] process_one_work+0x154/0x221 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002458] [] worker_thread+0x13e/0x1d7 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002460] [] ? cancel_delayed_work_sync+0x10/0x10 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002462] [] kthread+0xb2/0xba 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002464] [] ? __kthread_parkme+0x62/0x62 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002467] [] ret_from_fork+0x7c/0xb0 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002469] [] ? __kthread_parkme+0x62/0x62 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002489] Code: 65 48 8b 04 25 80 a9 00 00 48 8b 80 d8 04 00 00 48 89 e5 5d 48 8b 40 c8 48 c1 e8 02 83 e0 01 c3 48 8b 87 d8 04 00 00 55 48 89 e5 <48> 8b 40 d8 5d c3 55 ba 08 00 00 00 48 89 e5 50 48 8b b7 d8 04 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002491] RIP [] kthread_data+0xb/0x11 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002491] RSP 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002492] CR2: ffffffffffffffd8 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.002493] ---[ end trace 0077f2b8b8f10085 ]--- 2015-01-19T03:12:00+01:00 kaa-35 kernel: [271485.114636] Fixing recursive fault but reboot is needed! 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.467297] ------------[ cut here ]------------ 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.472987] WARNING: CPU: 0 PID: 17017 at net/sched/sch_generic.c:264 dev_watchdog+0xdc/0x142() 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482553] NETDEV WATCHDOG: eth1 (igb): transmit queue 1 timed out 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482585] Modules linked in: cbc ceph libceph ipmi_watchdog adm1026 w83795 w83793 hwmon_vid jc42 8021q garp stp mrp llc autofs4 btrfs xor raid6_pq ipmi_devintf mgag200 syscopyarea sysfillrect sysimgblt ttm x86_pkg_temp_thermal drm_kms_helper coretemp iTCO_wdt drm crc32c_intel iTCO_vendor_support microcode lpc_ich ipmi_si evdev pcspkr i2c_i801 mfd_core ipmi_msghandler rtc_cmos ioatdma processor dca thermal_sys button rpcsec_gss_krb5 fuse nfsv4 nfs af_packet hid_generic usbhid hid sd_mod bonding ehci_pci ehci_hcd ahci libahci usbcore libata usb_common ipv6 dm_mirror dm_region_hash dm_log dm_mod unix 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482587] CPU: 0 PID: 17017 Comm: kworker/0:2 Tainted: P D O 3.14.28-gentoo #2 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482588] Hardware name: Supermicro X9DRT-HF+/X9DRT-HF+, BIOS 3.0c 05/21/2014 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482600] Workqueue: events handle_osds_timeout [libceph] 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482602] 0000000000000000 ffff88103fa03de8 ffffffff813c6f0f ffff88103fa03e30 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482603] ffff88103fa03e20 ffffffff810392fb ffffffff81346a40 ffff8810249ba000 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482604] ffffffff81346964 ffff88103fa03ee0 ffffffff81346964 ffff88103fa03e88 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482605] Call Trace: 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482610] [] dump_stack+0x45/0x56 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482614] [] warn_slowpath_common+0x75/0x8e 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482615] [] ? dev_watchdog+0xdc/0x142 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482617] [] ? dev_deactivate_queue+0x55/0x55 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482618] [] ? dev_deactivate_queue+0x55/0x55 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482620] [] warn_slowpath_fmt+0x43/0x4b 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482622] [] ? _raw_spin_lock+0x9/0xb 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482623] [] ? netif_tx_lock+0x4c/0x7b 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482624] [] dev_watchdog+0xdc/0x142 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482627] [] call_timer_fn.isra.26+0x18/0x7c 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482629] [] run_timer_softirq+0x1a9/0x1d2 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482631] [] __do_softirq+0xf5/0x211 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482633] [] irq_exit+0x55/0x9d 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482635] [] smp_apic_timer_interrupt+0x30/0x3c 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482637] [] apic_timer_interrupt+0x6a/0x70 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482640] [] ? rep_nop+0x7/0x8 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482642] [] ? mspin_lock+0x38/0x3d 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482644] [] __mutex_lock_slowpath+0x5e/0x1b9 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482645] [] mutex_lock+0x12/0x22 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482646] [] ? mutex_lock+0x12/0x22 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482654] [] handle_osds_timeout+0x90/0x11d [libceph] 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482657] [] process_one_work+0x154/0x221 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482659] [] worker_thread+0x13e/0x1d7 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482660] [] ? cancel_delayed_work_sync+0x10/0x10 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482662] [] kthread+0xb2/0xba 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482663] [] ? __kthread_parkme+0x62/0x62 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482666] [] ret_from_fork+0x7c/0xb0 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482672] [] ? __kthread_parkme+0x62/0x62 2015-01-19T03:12:12+01:00 kaa-35 kernel: [271497.482675] ---[ end trace 0077f2b8b8f10086 ]---