Bug #3290: libceph: unable to handle osdmap - Ceph - Ceph

Actions

Copy link

Bug #3290

closed

libceph: unable to handle osdmap

Added by Josh Durgin over 11 years ago. Updated over 11 years ago.

Status:

Resolved

Priority:

High

Assignee:

Sage Weil

Category:

OSD

Target version:

% Done:

Source:

Community (user)

Tags:

Backport:

Regression:

Severity:

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

tziOm on IRC reported getting this 'corrupt' osdmap in dmesg when trying to map an rbd image with vanilla linux 3.6.1:

[32093.678524] libceph: client6003 fsid b0641d88-c825-48b1-9c45-8cd2041789d8
[32093.679221] libceph: osdc handle_map corrupt msg
[32093.682453] header: 00000000: 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682458] header: 00000010: 29 00 c4 00 01 00 44 08 00 00 00 00 00 00 00 00  ).....D.........
[32093.682462] header: 00000020: 00 00 00 00 01 00 00 00 00 00 00 00 00 01 00 00  ................
[32093.682466] header: 00000030: 00 d5 a3 1f 1a                                   .....
[32093.682470]  front: 00000000: b0 64 1d 88 c8 25 48 b1 9c 45 8c d2 04 17 89 d8  .d...%H..E......
[32093.682474]  front: 00000010: 00 00 00 00 01 00 00 00 7f 00 00 00 24 08 00 00  ............$...
[32093.682477]  front: 00000020: 05 00 b0 64 1d 88 c8 25 48 b1 9c 45 8c d2 04 17  ...d...%H..E....
[32093.682483]  front: 00000030: 89 d8 7f 00 00 00 28 98 74 50 88 3d 3a 0a e3 a7  ......(.tP.=:...
[32093.682488]  front: 00000040: 76 50 f0 1d f7 09 05 00 00 00 00 00 00 00 02 01  vP..............
[32093.682498]  front: 00000050: 02 00 02 c0 00 00 00 c0 00 00 00 00 00 00 00 00  ................
[32093.682508]  front: 00000060: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682515]  front: 00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682519]  front: 00000080: 00 00 00 01 00 00 00 02 01 02 01 02 c0 00 00 00  ................
[32093.682523]  front: 00000090: c0 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00  ................
[32093.682527]  front: 000000a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682530]  front: 000000b0: 00 00 00 00 00 00 00 00 00 00 00 00 02 00 00 00  ................
[32093.682534]  front: 000000c0: 02 01 02 02 02 c0 00 00 00 c0 00 00 00 00 00 00  ................
[32093.682538]  front: 000000d0: 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00  ................
[32093.682541]  front: 000000e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682545]  front: 000000f0: 00 00 00 00 00 03 00 00 00 02 01 02 00 02 00 03  ................
[32093.682549]  front: 00000100: 00 00 00 03 00 00 00 00 00 00 00 00 00 00 12 00  ................
[32093.682552]  front: 00000110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682556]  front: 00000120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 04 00  ................
[32093.682560]  front: 00000130: 00 00 02 01 02 00 02 08 00 00 00 08 00 00 00 00  ................
[32093.682563]  front: 00000140: 00 00 00 00 00 00 00 6d 00 00 00 02 00 00 00 00  .......m........
[32093.682570]  front: 00000150: 00 00 00 6d 00 00 00 02 00 00 00 00 00 00 00 ff  ...m............
[32093.682579]  front: 00000160: ff ff ff ff ff ff ff 01 00 00 00 00 00 00 00 02  ................
[32093.682588]  front: 00000170: 02 17 00 00 00 01 00 00 00 00 00 00 00 fe 96 76  ...............v
[32093.682597]  front: 00000180: 50 a8 5a de 32 03 00 00 00 66 6f 6f 02 00 00 00  P.Z.2....foo....
[32093.682607]  front: 00000190: 00 00 00 00 02 02 17 00 00 00 02 00 00 00 00 00  ................
[32093.682612]  front: 000001a0: 00 00 46 9c 76 50 70 90 b1 2f 03 00 00 00 64 65  ..F.vPp../....de
[32093.682616]  front: 000001b0: 62 05 00 00 00 00 00 00 00 04 00 00 00 64 61 74  b............dat
[32093.682619]  front: 000001c0: 61 01 00 00 00 08 00 00 00 6d 65 74 61 64 61 74  a........metadat
[32093.682623]  front: 000001d0: 61 02 00 00 00 03 00 00 00 72 62 64 03 00 00 00  a........rbd....
[32093.682627]  front: 000001e0: 06 00 00 00 70 62 65 6e 63 68 04 00 00 00 04 00  ....pbench......
[32093.682631]  front: 000001f0: 00 00 74 65 73 74 04 00 00 00 00 00 00 00 06 00  ..test..........
[32093.682634]  front: 00000200: 00 00 06 00 00 00 03 03 03 03 09 09 06 00 00 00  ................
[32093.682638]  front: 00000210: 00 00 01 00 00 00 01 00 00 00 01 00 00 00 01 00  ................
[32093.682641]  front: 00000220: 00 00 00 00 00 00 00 00 06 00 00 00 00 00 00 00  ................
[32093.682645]  front: 00000230: 17 09 00 00 00 02 1a 91 0a 05 00 10 00 00 00 00  ................
[32093.682649]  front: 00000240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682652]  front: 00000250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682656]  front: 00000260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682660]  front: 00000270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682664]  front: 00000280: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682667]  front: 00000290: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682671]  front: 000002a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682674]  front: 000002b0: 00 00 00 00 00 00 00 00 75 08 00 00 00 02 1a 90  ........u.......
[32093.682678]  front: 000002c0: 0a 05 00 11 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682682]  front: 000002d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682691]  front: 000002e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682724]  front: 000002f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682733]  front: 00000300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682742]  front: 00000310: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682752]  front: 00000320: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682786]  front: 00000330: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682792]  front: 00000340: 9d 0c 00 00 00 02 1a 90 0a 05 00 13 00 00 00 00  ................
[32093.682796]  front: 00000350: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682800]  front: 00000360: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682803]  front: 00000370: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682807]  front: 00000380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682811]  front: 00000390: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682814]  front: 000003a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682818]  front: 000003b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682822]  front: 000003c0: 00 00 00 00 00 00 00 00 ed 07 00 00 00 02 1a 90  ................
[32093.682825]  front: 000003d0: 0a 05 00 17 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682829]  front: 000003e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682833]  front: 000003f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682836]  front: 00000400: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682840]  front: 00000410: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682844]  front: 00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682847]  front: 00000430: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682851]  front: 00000440: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682854]  front: 00000450: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682858]  front: 00000460: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682862]  front: 00000470: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682865]  front: 00000480: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682869]  front: 00000490: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682873]  front: 000004a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682876]  front: 000004b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682880]  front: 000004c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682884]  front: 000004d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682887]  front: 000004e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682891]  front: 000004f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682895]  front: 00000500: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682898]  front: 00000510: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682902]  front: 00000520: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682910]  front: 00000530: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682920]  front: 00000540: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682929]  front: 00000550: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[32093.682939]  front: 00000560: e0 02 00 00 00 00 01 00 08 00 00 00 03 00 00 00  ................
[32093.682950]  front: 00000570: 04 00 00 00 04 00 00 00 ff ff ff ff 06 00 04 00  ................
[32093.682960]  front: 00000580: 00 00 04 00 01 00 00 00 fb ff ff ff 00 00 04 00  ................
[32093.682970]  front: 00000590: 00 00 01 00 04 00 00 00 fe ff ff ff 01 00 04 00  ................
[32093.683003]  front: 000005a0: 00 00 01 00 01 00 00 00 00 00 00 00 00 00 01 00  ................
[32093.683010]  front: 000005b0: 00 00 01 00 04 00 00 00 fd ff ff ff 02 00 04 00  ................
[32093.683014]  front: 000005c0: 00 00 04 00 04 00 00 00 fe ff ff ff fc ff ff ff  ................
[32093.683018]  front: 000005d0: fa ff ff ff f9 ff ff ff 00 00 01 00 00 00 01 00  ................
[32093.683022]  front: 000005e0: 00 00 01 00 00 00 01 00 00 00 01 00 00 00 01 00  ................
[32093.683026]  front: 000005f0: 00 00 01 00 00 00 01 00 04 00 00 00 fc ff ff ff  ................
[32093.683029]  front: 00000600: 01 00 04 00 00 00 01 00 01 00 00 00 01 00 00 00  ................
[32093.683033]  front: 00000610: 00 00 01 00 00 00 01 00 04 00 00 00 fb ff ff ff  ................
[32093.683037]  front: 00000620: 05 00 04 00 00 00 04 00 01 00 00 00 fd ff ff ff  ................
[32093.683041]  front: 00000630: 00 00 04 00 00 00 01 00 04 00 00 00 fa ff ff ff  ................
[32093.683044]  front: 00000640: 01 00 04 00 00 00 01 00 01 00 00 00 02 00 00 00  ................
[32093.683048]  front: 00000650: 00 00 01 00 00 00 01 00 04 00 00 00 f9 ff ff ff  ................
[32093.683052]  front: 00000660: 01 00 04 00 00 00 01 00 01 00 00 00 03 00 00 00  ................
[32093.683055]  front: 00000670: 00 00 01 00 00 00 01 00 00 00 00 00 01 00 00 00  ................
[32093.683059]  front: 00000680: 03 00 00 00 00 01 01 0a 01 00 00 00 ff ff ff ff  ................
[32093.683062]  front: 00000690: 00 00 00 00 06 00 00 00 00 00 00 00 01 00 00 00  ................
[32093.683066]  front: 000006a0: 04 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00  ................
[32093.683070]  front: 000006b0: 03 00 00 00 01 01 01 0a 01 00 00 00 ff ff ff ff  ................
[32093.683073]  front: 000006c0: 00 00 00 00 06 00 00 00 00 00 00 00 01 00 00 00  ................
[32093.683077]  front: 000006d0: 04 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00  ................
[32093.683080]  front: 000006e0: 03 00 00 00 02 01 01 0a 01 00 00 00 ff ff ff ff  ................
[32093.683084]  front: 000006f0: 00 00 00 00 06 00 00 00 00 00 00 00 01 00 00 00  ................
[32093.683088]  front: 00000700: 04 00 00 00 00 00 00 00 00 00 00 00 07 00 00 00  ................
[32093.683091]  front: 00000710: 00 00 00 00 03 00 00 00 6f 73 64 01 00 00 00 04  ........osd.....
[32093.683095]  front: 00000720: 00 00 00 68 6f 73 74 02 00 00 00 04 00 00 00 72  ...host........r
[32093.683099]  front: 00000730: 61 63 6b 03 00 00 00 03 00 00 00 72 6f 77 04 00  ack........row..
[32093.683102]  front: 00000740: 00 00 04 00 00 00 72 6f 6f 6d 05 00 00 00 0a 00  ......room......
[32093.683106]  front: 00000750: 00 00 64 61 74 61 63 65 6e 74 65 72 06 00 00 00  ..datacenter....
[32093.683110]  front: 00000760: 04 00 00 00 72 6f 6f 74 0b 00 00 00 f9 ff ff ff  ....root........
[32093.683114]  front: 00000770: 09 00 00 00 73 74 6f 72 61 67 65 30 31 fa ff ff  ....storage01...
[32093.683117]  front: 00000780: ff 09 00 00 00 73 74 6f 72 61 67 65 30 34 fb ff  .....storage04..
[32093.683121]  front: 00000790: ff ff 03 00 00 00 61 64 72 fc ff ff ff 09 00 00  ......adr.......
[32093.683125]  front: 000007a0: 00 73 74 6f 72 61 67 65 30 33 fd ff ff ff 07 00  .storage03......
[32093.683128]  front: 000007b0: 00 00 72 61 63 6b 30 30 31 fe ff ff ff 09 00 00  ..rack001.......
[32093.683132]  front: 000007c0: 00 73 74 6f 72 61 67 65 30 32 ff ff ff ff 07 00  .storage02......
[32093.683136]  front: 000007d0: 00 00 64 65 66 61 75 6c 74 00 00 00 00 05 00 00  ..default.......
[32093.683139]  front: 000007e0: 00 6f 73 64 2e 30 01 00 00 00 05 00 00 00 6f 73  .osd.0........os
[32093.683143]  front: 000007f0: 64 2e 31 02 00 00 00 05 00 00 00 6f 73 64 2e 32  d.1........osd.2
[32093.683147]  front: 00000800: 03 00 00 00 05 00 00 00 6f 73 64 2e 33 03 00 00  ........osd.3...
[32093.683150]  front: 00000810: 00 00 00 00 00 04 00 00 00 64 61 74 61 01 00 00  .........data...
[32093.683154]  front: 00000820: 00 08 00 00 00 6d 65 74 61 64 61 74 61 02 00 00  .....metadata...
[32093.683158]  front: 00000830: 00 03 00 00 00 72 62 64 02 00 00 00 05 00 00 00  .....rbd........
[32093.683162]  front: 00000840: 13 00 00 00                                      ....
[32093.683166] footer: 00000000: ef 46 46 b1 00 00 00 00 00 00 00 00 01           .FF..........
[32093.683181] libceph: mon0 10.5.0.16:6789 session established

After tziOm deleted the pools test, swimmingpool and pbench, the osdmap was able to be processed and 'rbd map' succeeded.