Project

General

Profile

Bug #2759

libceph: crush tree algorithm is not understood

Added by Josh Durgin over 11 years ago. Updated over 8 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
libceph
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

After changing the crush algorithm for a bucket to tree, the kernel client cannot parse the OSDMap:

Jul  9 11:12:06 plana29 kernel: [587716.692437] libceph: osdc handle_map corrupt msg
Jul  9 11:12:06 plana29 kernel: [587716.716772] header: 00000000: 07 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716777] header: 00000010: 29 00 c4 00 01 00 5d 04 00 00 00 00 00 00 00 00  ).....].........
Jul  9 11:12:06 plana29 kernel: [587716.716781] header: 00000020: 00 00 00 00 01 01 00 00 00 00 00 00 00 01 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716785] header: 00000030: 00 d0 e8 8d ed                                   .....
Jul  9 11:12:06 plana29 kernel: [587716.716788]  front: 00000000: ba 9b be 62 86 d5 43 6d a0 82 cd c9 ef 06 8f be  ...b..Cm........
Jul  9 11:12:06 plana29 kernel: [587716.716792]  front: 00000010: 00 00 00 00 01 00 00 00 07 00 00 00 3d 04 00 00  ............=...
Jul  9 11:12:06 plana29 kernel: [587716.716796]  front: 00000020: 05 00 ba 9b be 62 86 d5 43 6d a0 82 cd c9 ef 06  .....b..Cm......
Jul  9 11:12:06 plana29 kernel: [587716.716800]  front: 00000030: 8f be 07 00 00 00 54 f5 f5 4f d8 71 4e 1c 4d 1d  ......T..O.qN.M.
Jul  9 11:12:06 plana29 kernel: [587716.716804]  front: 00000040: fb 4f 80 7c fa 21 03 00 00 00 00 00 00 00 02 01  .O.|.!..........
Jul  9 11:12:06 plana29 kernel: [587716.716807]  front: 00000050: 02 00 02 08 00 00 00 08 00 00 00 00 00 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716811]  front: 00000060: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716815]  front: 00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716818]  front: 00000080: 00 00 00 01 00 00 00 02 01 02 01 02 08 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716823]  front: 00000090: 08 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716828]  front: 000000a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716832]  front: 000000b0: 00 00 00 00 00 00 00 00 00 00 00 00 02 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716836]  front: 000000c0: 02 01 02 02 02 08 00 00 00 08 00 00 00 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716840]  front: 000000d0: 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716844]  front: 000000e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716848]  front: 000000f0: 00 00 00 00 00 03 00 00 00 00 00 00 00 04 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716853]  front: 00000100: 00 64 61 74 61 01 00 00 00 08 00 00 00 6d 65 74  .data........met
Jul  9 11:12:06 plana29 kernel: [587716.716857]  front: 00000110: 61 64 61 74 61 02 00 00 00 03 00 00 00 72 62 64  adata........rbd
Jul  9 11:12:06 plana29 kernel: [587716.716861]  front: 00000120: 02 00 00 00 00 00 00 00 02 00 00 00 02 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716866]  front: 00000130: 03 03 02 00 00 00 00 00 01 00 00 00 01 00 02 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716870]  front: 00000140: 00 00 00 00 00 00 36 54 00 00 00 02 1a 90 0a d6  ......6T........
Jul  9 11:12:06 plana29 kernel: [587716.716874]  front: 00000150: 83 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716878]  front: 00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716883]  front: 00000170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716887]  front: 00000180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716891]  front: 00000190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716895]  front: 000001a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716900]  front: 000001b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716904]  front: 000001c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 06 4b  ...............K
Jul  9 11:12:06 plana29 kernel: [587716.716908]  front: 000001d0: 00 00 00 02 1a 90 0a d6 83 22 00 00 00 00 00 00  ........."......
Jul  9 11:12:06 plana29 kernel: [587716.716913]  front: 000001e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716917]  front: 000001f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716921]  front: 00000200: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716925]  front: 00000210: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716930]  front: 00000220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716934]  front: 00000230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716938]  front: 00000240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716942]  front: 00000250: 00 00 00 00 00 00 03 02 00 00 00 00 01 00 08 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716946]  front: 00000260: 00 00 03 00 00 00 02 00 00 00 04 00 00 00 ff ff  ................
Jul  9 11:12:06 plana29 kernel: [587716.716951]  front: 00000270: ff ff 06 00 04 00 00 00 02 00 01 00 00 00 fd ff  ................
Jul  9 11:12:06 plana29 kernel: [587716.716955]  front: 00000280: ff ff 00 00 02 00 00 00 01 00 03 00 00 00 fe ff  ................
Jul  9 11:12:06 plana29 kernel: [587716.716959]  front: 00000290: ff ff 01 00 03 00 00 00 02 00 02 00 00 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716963]  front: 000002a0: 00 00 01 00 00 00 04 00 00 00 00 00 00 01 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716967]  front: 000002b0: 00 02 00 00 00 01 00 04 00 00 00 fd ff ff ff 02  ................
Jul  9 11:12:06 plana29 kernel: [587716.716971]  front: 000002c0: 00 04 00 00 00 02 00 01 00 00 00 fe ff ff ff 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716975]  front: 000002d0: 00 02 00 00 00 01 00 00 00 00 00 00 00 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716979]  front: 000002e0: 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 03  ................
Jul  9 11:12:06 plana29 kernel: [587716.716983]  front: 000002f0: 00 00 00 00 01 01 0a 01 00 00 00 ff ff ff ff 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.716988]  front: 00000300: 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 04  ................
Jul  9 11:12:06 plana29 kernel: [587716.716992]  front: 00000310: 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 03  ................
Jul  9 11:12:06 plana29 kernel: [587716.716996]  front: 00000320: 00 00 00 01 01 01 0a 01 00 00 00 ff ff ff ff 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.717000]  front: 00000330: 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 04  ................
Jul  9 11:12:06 plana29 kernel: [587716.717004]  front: 00000340: 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 03  ................
Jul  9 11:12:06 plana29 kernel: [587716.717008]  front: 00000350: 00 00 00 02 01 01 0a 01 00 00 00 ff ff ff ff 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.717012]  front: 00000360: 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 04  ................
Jul  9 11:12:06 plana29 kernel: [587716.717017]  front: 00000370: 00 00 00 00 00 00 00 00 00 00 00 07 00 00 00 00  ................
Jul  9 11:12:06 plana29 kernel: [587716.717021]  front: 00000380: 00 00 00 03 00 00 00 6f 73 64 01 00 00 00 04 00  .......osd......
Jul  9 11:12:06 plana29 kernel: [587716.717026]  front: 00000390: 00 00 68 6f 73 74 02 00 00 00 04 00 00 00 72 61  ..host........ra
Jul  9 11:12:06 plana29 kernel: [587716.717030]  front: 000003a0: 63 6b 03 00 00 00 03 00 00 00 72 6f 77 04 00 00  ck........row...
Jul  9 11:12:06 plana29 kernel: [587716.717034]  front: 000003b0: 00 04 00 00 00 72 6f 6f 6d 05 00 00 00 0a 00 00  .....room.......
Jul  9 11:12:06 plana29 kernel: [587716.717038]  front: 000003c0: 00 64 61 74 61 63 65 6e 74 65 72 06 00 00 00 04  .datacenter.....
Jul  9 11:12:06 plana29 kernel: [587716.717042]  front: 000003d0: 00 00 00 70 6f 6f 6c 05 00 00 00 fd ff ff ff 09  ...pool.........
Jul  9 11:12:06 plana29 kernel: [587716.717046]  front: 000003e0: 00 00 00 6c 6f 63 61 6c 72 61 63 6b fe ff ff ff  ...localrack....
Jul  9 11:12:06 plana29 kernel: [587716.717050]  front: 000003f0: 09 00 00 00 6c 6f 63 61 6c 68 6f 73 74 ff ff ff  ....localhost...
Jul  9 11:12:06 plana29 kernel: [587716.717054]  front: 00000400: ff 07 00 00 00 64 65 66 61 75 6c 74 00 00 00 00  .....default....
Jul  9 11:12:06 plana29 kernel: [587716.717058]  front: 00000410: 05 00 00 00 6f 73 64 2e 30 01 00 00 00 05 00 00  ....osd.0.......
Jul  9 11:12:06 plana29 kernel: [587716.717063]  front: 00000420: 00 6f 73 64 2e 31 03 00 00 00 00 00 00 00 04 00  .osd.1..........
Jul  9 11:12:06 plana29 kernel: [587716.717067]  front: 00000430: 00 00 64 61 74 61 01 00 00 00 08 00 00 00 6d 65  ..data........me
Jul  9 11:12:06 plana29 kernel: [587716.717071]  front: 00000440: 74 61 64 61 74 61 02 00 00 00 03 00 00 00 72 62  tadata........rb
Jul  9 11:12:06 plana29 kernel: [587716.717075]  front: 00000450: 64 02 00 00 00 05 00 00 00 13 00 00 00           d............
Jul  9 11:12:06 plana29 kernel: [587716.717079] footer: 00000000: 9b d3 12 b0 00 00 00 00 00 00 00 00 01           .............

The mount command hangs, as described in http://permalink.gmane.org/gmane.comp.file-systems.ceph.devel/7702

History

#1 Updated by Sage Weil about 10 years ago

  • Priority changed from Normal to High

#2 Updated by Ian Colle about 10 years ago

We should be testing CRUSH with multiple algorithms

#3 Updated by Ilya Dryomov over 9 years ago

  • Subject changed from crush: tree algorithm is not understood to libceph: crush tree algorithm is not understood

#4 Updated by Ilya Dryomov over 8 years ago

  • Status changed from 12 to In Progress
  • Assignee set to Ilya Dryomov
  • Regression set to No

#5 Updated by Ilya Dryomov over 8 years ago

  • Status changed from In Progress to Fix Under Review

#6 Updated by Ilya Dryomov over 8 years ago

  • Status changed from Fix Under Review to Resolved

[PATCH] crush: fix a bug in tree bucket decode

Also available in: Atom PDF