Project

General

Profile

Actions
Copy link

Bug #46823

closed

nautilus: kceph w/ testing branch: mdsc_handle_session corrupt message mds0 len 67

Added by Patrick Donnelly over 3 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
High
Assignee:
Jeff Layton
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Yes
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS, kceph
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

[  145.812885] libceph: mon2 (1)172.21.15.84:6790 session established
[  145.813553] libceph: client8321 fsid 5cf09433-2f5f-422b-acb4-803697e075a8
[  149.694847] libceph: mds0 (1)172.21.15.65:6835 socket closed (con state OPEN)
[  149.729899] libceph: mds0 (1)172.21.15.65:6835 connection reset
[  149.736003] libceph: reset on mds0
[  149.736005] ceph: mds0 closed our session
[  149.736007] ceph: mds0 reconnect start
[  149.737125] ceph: mds0 reconnect denied
[  149.738001] libceph: mds0 (1)172.21.15.65:6835 socket closed (con state NEGOTIATING)
[  150.899734] ceph: mdsc_handle_session corrupt message mds0 len 67
[  150.906018] header: 00000000: 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  150.906023] header: 00000010: 16 00 7f 00 03 00 43 00 00 00 00 00 00 00 00 00  ......C.........
[  150.906026] header: 00000020: 00 00 00 00 02 00 00 00 00 00 00 00 00 01 00 00  ................
[  150.906030] header: 00000030: 00 30 45 8c 74                                   .0E.t
[  150.906033]  front: 00000000: 0b 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  150.906037]  front: 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00  ................
[  150.906040]  front: 00000020: 0c 00 00 00 65 72 72 6f 72 5f 73 74 72 69 6e 67  ....error_string
[  150.906044]  front: 00000030: 0b 00 00 00 62 6c 61 63 6b 6c 69 73 74 65 64 00  ....blacklisted.
[  150.906047]  front: 00000040: 00 00 00                                         ...
[  150.906051] footer: 00000000: d6 ac a0 0c 00 00 00 00 00 00 00 00 3e d3 12 5a  ............>..Z
[  150.906055] footer: 00000010: e0 4d 48 94 05                                   .MH..

From dmesg output on hung node. Teuthology log:

2020-08-03T21:57:12.992 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0...
2020-08-03T21:57:12.992 INFO:teuthology.orchestra.run:Running command with timeout 900
2020-08-03T21:57:12.993 INFO:teuthology.orchestra.run.smithi201:> sudo umount /home/ubuntu/cephtest/mnt.0
2020-08-03T21:57:13.033 INFO:teuthology.orchestra.run.smithi045:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2020-08-03T21:57:13.038 INFO:teuthology.orchestra.run.smithi065:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2020-08-03T21:57:13.040 INFO:teuthology.orchestra.run.smithi084:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2020-08-03T21:57:13.043 INFO:teuthology.orchestra.run.smithi101:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2020-08-03T21:57:13.047 INFO:teuthology.orchestra.run.smithi171:> sudo logrotate /etc/logrotate.d/ceph-test.conf
...

From: /ceph/teuthology-archive/pdonnell-2020-08-03_20:28:49-kcephfs-nautilus-distro-basic-smithi/5284713/teuthology.log

Actions
Copy link

Also available in: Atom PDF