Project

General

Profile

Actions
Copy link

Bug #2091

closed

corrupt v5 inc osdmap

Added by Sage Weil about 12 years ago. Updated about 12 years ago.

Status:
Can't reproduce
Priority:
Normal
Assignee:
-
Category:
-
Target version:
v0.43
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description


Feb 20 17:54:54 fcmsnode0 kernel: libceph: corrupt inc osdmap epoch 880 off 102 (ffffc9001db8990a of ffffc9001db898a4-ffffc9001db89dae)
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000000: 05 00 34 b3 0e 21 99 f8 43 ff be f2 a3 c0 04 2b  ..4..!..C......+
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000010: 07 e1 70 03 00 00 10 79 42 4f 30 16 47 1d ff ff  ..p....yBO0.G...
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000020: ff ff ff ff ff ff 00 00 00 00 00 00 00 00 ff ff  ................
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000030: ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000040: 00 00 00 00 00 00 00 00 00 00 52 00 00 00 ff ff  ..........R.....
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000050: 06 00 07 00 00 00 03 00 00 00 00 00 00 00 02 00  ................
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000060: 00 00 03 00 00 00 ff ff 03 00 0a 00 00 00 03 00  ................
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000070: 00 00 00 00 00 00 02 00 00 00 03 00 00 00 ff ff  ................
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000080: 02 00 0b 00 00 00 03 00 00 00 00 00 00 00 02 00  ................
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000090: 00 00 03 00 00 00 ff ff 07 00 0f 00 00 00 00 00  ................
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000000a0: 00 00 ff ff 06 00 10 00 00 00 00 00 00 00 ff ff  ................
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000000b0: 05 00 11 00 00 00 00 00 00 00 ff ff 04 00 12 00  ................
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000000c0: 00 00 00 00 00 00 ff ff 02 00 14 00 00 00 00 00  ................
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000000d0: 00 00 ff ff 07 00 14 00 00 00 00 00 00 00 ff ff  ................
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000000e0: 01 00 15 00 00 00 00 00 00 00 ff ff 06 00 15 00  ................
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000000f0: 00 00 00 00 00 00 ff ff 00 00 16 00 00 00 00 00  ................
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000100: 00 00 ff ff 05 00 16 00 00 00 00 00 00 00 ff ff  ................
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000110: 04 00 17 00 00 00 00 00 00 00 ff ff 03 00 18 00  ................
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000120: 00 00 00 00 00 00 ff ff 02 00 19 00 00 00 00 00  ................
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000130: 00 00 ff ff 01 00 1a 00 00 00 00 00 00 00 ff ff  ................
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000140: 00 00 1b 00 00 00 00 00 00 00 ff ff 07 00 23 00  ..............#.
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000150: 00 00 00 00 00 00 ff ff 06 00 24 00 00 00 00 00  ..........$.....
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000160: 00 00 ff ff 05 00 25 00 00 00 00 00 00 00 ff ff  ......%.........
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000170: 04 00 26 00 00 00 00 00 00 00 ff ff 02 00 28 00  ..&...........(.
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000180: 00 00 00 00 00 00 ff ff 01 00 29 00 00 00 00 00  ..........).....
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000190: 00 00 ff ff 00 00 2a 00 00 00 00 00 00 00 ff ff  ......*.........
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000001a0: 07 00 33 00 00 00 00 00 00 00 ff ff 06 00 34 00  ..3...........4.
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000001b0: 00 00 00 00 00 00 ff ff 05 00 35 00 00 00 00 00  ..........5.....
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000001c0: 00 00 ff ff 04 00 36 00 00 00 00 00 00 00 ff ff  ......6.........
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000001d0: 03 00 37 00 00 00 00 00 00 00 ff ff 02 00 38 00  ..7...........8.
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000001e0: 00 00 00 00 00 00 ff ff 01 00 39 00 00 00 00 00  ..........9.....
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000001f0: 00 00 ff ff 06 00 3b 00 00 00 03 00 00 00 00 00  ......;.........
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000200: 00 00 02 00 00 00 03 00 00 00 ff ff 04 00 3d 00  ..............=.
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000210: 00 00 03 00 00 00 00 00 00 00 02 00 00 00 03 00  ................
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000220: 00 00 ff ff 03 00 3e 00 00 00 03 00 00 00 00 00  ......>.........
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000230: 00 00 02 00 00 00 03 00 00 00 ff ff 02 00 3f 00  ..............?.
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000240: 00 00 03 00 00 00 00 00 00 00 02 00 00 00 03 00  ................
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000250: 00 00 ff ff 01 00 40 00 00 00 03 00 00 00 00 00  ......@.........
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000260: 00 00 02 00 00 00 03 00 00 00 ff ff 00 00 41 00  ..............A.
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000270: 00 00 03 00 00 00 00 00 00 00 02 00 00 00 03 00  ................
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000280: 00 00 ff ff 03 00 42 00 00 00 03 00 00 00 03 00  ......B.........
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000290: 00 00 00 00 00 00 04 00 00 00 ff ff 07 00 42 00  ..............B.
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000002a0: 00 00 00 00 00 00 ff ff 06 00 43 00 00 00 00 00  ..........C.....
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000002b0: 00 00 ff ff 01 00 44 00 00 00 03 00 00 00 03 00  ......D.........
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000002c0: 00 00 00 00 00 00 04 00 00 00 ff ff 05 00 44 00  ..............D.
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000002d0: 00 00 00 00 00 00 ff ff 03 00 46 00 00 00 00 00  ..........F.....
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000002e0: 00 00 ff ff 02 00 47 00 00 00 00 00 00 00 ff ff  ......G.........
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000002f0: 01 00 48 00 00 00 00 00 00 00 ff ff 00 00 49 00  ..H...........I.
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000300: 00 00 00 00 00 00 ff ff 06 00 59 00 00 00 00 00  ..........Y.....
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000310: 00 00 ff ff 05 00 5a 00 00 00 00 00 00 00 ff ff  ......Z.........
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000320: 04 00 5b 00 00 00 00 00 00 00 ff ff 02 00 5d 00  ..[...........].
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000330: 00 00 00 00 00 00 ff ff 01 00 5e 00 00 00 00 00  ..........^.....
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000340: 00 00 ff ff 00 00 5f 00 00 00 00 00 00 00 ff ff  ......_.........
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000350: 07 00 60 00 00 00 03 00 00 00 00 00 00 00 02 00  ..`.............
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000360: 00 00 03 00 00 00 ff ff 06 00 61 00 00 00 03 00  ..........a.....
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000370: 00 00 00 00 00 00 02 00 00 00 03 00 00 00 ff ff  ................
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000380: 05 00 62 00 00 00 03 00 00 00 00 00 00 00 02 00  ..b.............
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000390: 00 00 03 00 00 00 ff ff 04 00 63 00 00 00 03 00  ..........c.....
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000003a0: 00 00 00 00 00 00 02 00 00 00 03 00 00 00 ff ff  ................
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000003b0: 07 00 63 00 00 00 00 00 00 00 ff ff 03 00 64 00  ..c...........d.
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000003c0: 00 00 03 00 00 00 00 00 00 00 02 00 00 00 03 00  ................
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000003d0: 00 00 ff ff 06 00 64 00 00 00 00 00 00 00 ff ff  ......d.........
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000003e0: 07 00 64 00 00 00 00 00 00 00 ff ff 02 00 65 00  ..d...........e.
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000003f0: 00 00 03 00 00 00 00 00 00 00 02 00 00 00 03 00  ................
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000400: 00 00 ff ff 05 00 65 00 00 00 00 00 00 00 ff ff  ......e.........
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000410: 06 00 65 00 00 00 00 00 00 00 ff ff 01 00 66 00  ..e...........f.
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000420: 00 00 03 00 00 00 00 00 00 00 02 00 00 00 03 00  ................
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000430: 00 00 ff ff 04 00 66 00 00 00 00 00 00 00 ff ff  ......f.........
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000440: 05 00 66 00 00 00 00 00 00 00 ff ff 00 00 67 00  ..f...........g.
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000450: 00 00 03 00 00 00 00 00 00 00 02 00 00 00 03 00  ................
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000460: 00 00 ff ff 03 00 67 00 00 00 00 00 00 00 ff ff  ......g.........
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000470: 04 00 67 00 00 00 00 00 00 00 ff ff 02 00 68 00  ..g...........h.
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000480: 00 00 00 00 00 00 ff ff 03 00 68 00 00 00 00 00  ..........h.....
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000490: 00 00 ff ff 01 00 69 00 00 00 00 00 00 00 ff ff  ......i.........
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000004a0: 02 00 69 00 00 00 00 00 00 00 ff ff 07 00 73 00  ..i...........s.
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000004b0: 00 00 00 00 00 00 ff ff 06 00 74 00 00 00 00 00  ..........t.....
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000004c0: 00 00 ff ff 05 00 75 00 00 00 00 00 00 00 ff ff  ......u.........
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000004d0: 04 00 76 00 00 00 00 00 00 00 ff ff 03 00 77 00  ..v...........w.
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000004e0: 00 00 00 00 00 00 ff ff 02 00 78 00 00 00 00 00  ..........x.....
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000004f0: 00 00 ff ff 01 00 79 00 00 00 00 00 00 00 ff ff  ......y.........
Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000500: 00 00 7a 00 00 00 00 00 00 00                    ..z.......

the original 880 inc, from the monitor:

fatty:~ 09:18 AM $ hexdump -C /tmp/880
00000000  06 00 34 b3 0e 21 99 f8  43 ff be f2 a3 c0 04 2b  |..4..!..C......+|
00000010  07 e1 70 03 00 00 10 79  42 4f 30 16 47 1d ff ff  |..p....yBO0.G...|
00000020  ff ff ff ff ff ff ff ff  ff ff 00 00 00 00 00 00  |................|
00000030  00 00 ff ff ff ff 00 00  00 00 00 00 00 00 00 00  |................|
00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 52 00  |..............R.|
00000050  00 00 01 07 00 00 00 00  00 00 00 06 00 00 00 ff  |................|
00000060  ff ff ff 03 00 00 00 00  00 00 00 02 00 00 00 03  |................|
00000070  00 00 00 01 0a 00 00 00  00 00 00 00 03 00 00 00  |................|
00000080  ff ff ff ff 03 00 00 00  00 00 00 00 02 00 00 00  |................|
00000090  03 00 00 00 01 0b 00 00  00 00 00 00 00 02 00 00  |................|
000000a0  00 ff ff ff ff 03 00 00  00 00 00 00 00 02 00 00  |................|
000000b0  00 03 00 00 00 01 0f 00  00 00 00 00 00 00 07 00  |................|
000000c0  00 00 ff ff ff ff 00 00  00 00 01 10 00 00 00 00  |................|
000000d0  00 00 00 06 00 00 00 ff  ff ff ff 00 00 00 00 01  |................|
000000e0  11 00 00 00 00 00 00 00  05 00 00 00 ff ff ff ff  |................|
000000f0  00 00 00 00 01 12 00 00  00 00 00 00 00 04 00 00  |................|
00000100  00 ff ff ff ff 00 00 00  00 01 14 00 00 00 00 00  |................|
00000110  00 00 02 00 00 00 ff ff  ff ff 00 00 00 00 01 14  |................|
00000120  00 00 00 00 00 00 00 07  00 00 00 ff ff ff ff 00  |................|
00000130  00 00 00 01 15 00 00 00  00 00 00 00 01 00 00 00  |................|
00000140  ff ff ff ff 00 00 00 00  01 15 00 00 00 00 00 00  |................|
00000150  00 06 00 00 00 ff ff ff  ff 00 00 00 00 01 16 00  |................|
00000160  00 00 00 00 00 00 00 00  00 00 ff ff ff ff 00 00  |................|
00000170  00 00 01 16 00 00 00 00  00 00 00 05 00 00 00 ff  |................|
00000180  ff ff ff 00 00 00 00 01  17 00 00 00 00 00 00 00  |................|
00000190  04 00 00 00 ff ff ff ff  00 00 00 00 01 18 00 00  |................|
000001a0  00 00 00 00 00 03 00 00  00 ff ff ff ff 00 00 00  |................|
000001b0  00 01 19 00 00 00 00 00  00 00 02 00 00 00 ff ff  |................|
000001c0  ff ff 00 00 00 00 01 1a  00 00 00 00 00 00 00 01  |................|
000001d0  00 00 00 ff ff ff ff 00  00 00 00 01 1b 00 00 00  |................|
000001e0  00 00 00 00 00 00 00 00  ff ff ff ff 00 00 00 00  |................|
000001f0  01 23 00 00 00 00 00 00  00 07 00 00 00 ff ff ff  |.#..............|
00000200  ff 00 00 00 00 01 24 00  00 00 00 00 00 00 06 00  |......$.........|
00000210  00 00 ff ff ff ff 00 00  00 00 01 25 00 00 00 00  |...........%....|
00000220  00 00 00 05 00 00 00 ff  ff ff ff 00 00 00 00 01  |................|
00000230  26 00 00 00 00 00 00 00  04 00 00 00 ff ff ff ff  |&...............|
00000240  00 00 00 00 01 28 00 00  00 00 00 00 00 02 00 00  |.....(..........|
00000250  00 ff ff ff ff 00 00 00  00 01 29 00 00 00 00 00  |..........).....|
00000260  00 00 01 00 00 00 ff ff  ff ff 00 00 00 00 01 2a  |...............*|
00000270  00 00 00 00 00 00 00 00  00 00 00 ff ff ff ff 00  |................|
00000280  00 00 00 01 33 00 00 00  00 00 00 00 07 00 00 00  |....3...........|
00000290  ff ff ff ff 00 00 00 00  01 34 00 00 00 00 00 00  |.........4......|
000002a0  00 06 00 00 00 ff ff ff  ff 00 00 00 00 01 35 00  |..............5.|
000002b0  00 00 00 00 00 00 05 00  00 00 ff ff ff ff 00 00  |................|
000002c0  00 00 01 36 00 00 00 00  00 00 00 04 00 00 00 ff  |...6............|
000002d0  ff ff ff 00 00 00 00 01  37 00 00 00 00 00 00 00  |........7.......|
000002e0  03 00 00 00 ff ff ff ff  00 00 00 00 01 38 00 00  |.............8..|
000002f0  00 00 00 00 00 02 00 00  00 ff ff ff ff 00 00 00  |................|
00000300  00 01 39 00 00 00 00 00  00 00 01 00 00 00 ff ff  |..9.............|
00000310  ff ff 00 00 00 00 01 3b  00 00 00 00 00 00 00 06  |.......;........|
00000320  00 00 00 ff ff ff ff 03  00 00 00 00 00 00 00 02  |................|
00000330  00 00 00 03 00 00 00 01  3d 00 00 00 00 00 00 00  |........=.......|
00000340  04 00 00 00 ff ff ff ff  03 00 00 00 00 00 00 00  |................|
00000350  02 00 00 00 03 00 00 00  01 3e 00 00 00 00 00 00  |.........>......|
00000360  00 03 00 00 00 ff ff ff  ff 03 00 00 00 00 00 00  |................|
00000370  00 02 00 00 00 03 00 00  00 01 3f 00 00 00 00 00  |..........?.....|
00000380  00 00 02 00 00 00 ff ff  ff ff 03 00 00 00 00 00  |................|
00000390  00 00 02 00 00 00 03 00  00 00 01 40 00 00 00 00  |...........@....|
000003a0  00 00 00 01 00 00 00 ff  ff ff ff 03 00 00 00 00  |................|
000003b0  00 00 00 02 00 00 00 03  00 00 00 01 41 00 00 00  |............A...|
000003c0  00 00 00 00 00 00 00 00  ff ff ff ff 03 00 00 00  |................|
000003d0  00 00 00 00 02 00 00 00  03 00 00 00 01 42 00 00  |.............B..|
000003e0  00 00 00 00 00 03 00 00  00 ff ff ff ff 03 00 00  |................|
000003f0  00 03 00 00 00 00 00 00  00 04 00 00 00 01 42 00  |..............B.|
00000400  00 00 00 00 00 00 07 00  00 00 ff ff ff ff 00 00  |................|
00000410  00 00 01 43 00 00 00 00  00 00 00 06 00 00 00 ff  |...C............|
00000420  ff ff ff 00 00 00 00 01  44 00 00 00 00 00 00 00  |........D.......|
00000430  01 00 00 00 ff ff ff ff  03 00 00 00 03 00 00 00  |................|
00000440  00 00 00 00 04 00 00 00  01 44 00 00 00 00 00 00  |.........D......|
00000450  00 05 00 00 00 ff ff ff  ff 00 00 00 00 01 46 00  |..............F.|
00000460  00 00 00 00 00 00 03 00  00 00 ff ff ff ff 00 00  |................|
00000470  00 00 01 47 00 00 00 00  00 00 00 02 00 00 00 ff  |...G............|
00000480  ff ff ff 00 00 00 00 01  48 00 00 00 00 00 00 00  |........H.......|
00000490  01 00 00 00 ff ff ff ff  00 00 00 00 01 49 00 00  |.............I..|
000004a0  00 00 00 00 00 00 00 00  00 ff ff ff ff 00 00 00  |................|
000004b0  00 01 59 00 00 00 00 00  00 00 06 00 00 00 ff ff  |..Y.............|
000004c0  ff ff 00 00 00 00 01 5a  00 00 00 00 00 00 00 05  |.......Z........|
000004d0  00 00 00 ff ff ff ff 00  00 00 00 01 5b 00 00 00  |............[...|
000004e0  00 00 00 00 04 00 00 00  ff ff ff ff 00 00 00 00  |................|
000004f0  01 5d 00 00 00 00 00 00  00 02 00 00 00 ff ff ff  |.]..............|
00000500  ff 00 00 00 00 01 5e 00  00 00 00 00 00 00 01 00  |......^.........|
00000510  00 00 ff ff ff ff 00 00  00 00 01 5f 00 00 00 00  |..........._....|
00000520  00 00 00 00 00 00 00 ff  ff ff ff 00 00 00 00 01  |................|
00000530  60 00 00 00 00 00 00 00  07 00 00 00 ff ff ff ff  |`...............|
00000540  03 00 00 00 00 00 00 00  02 00 00 00 03 00 00 00  |................|
00000550  01 61 00 00 00 00 00 00  00 06 00 00 00 ff ff ff  |.a..............|
00000560  ff 03 00 00 00 00 00 00  00 02 00 00 00 03 00 00  |................|
00000570  00 01 62 00 00 00 00 00  00 00 05 00 00 00 ff ff  |..b.............|
00000580  ff ff 03 00 00 00 00 00  00 00 02 00 00 00 03 00  |................|
00000590  00 00 01 63 00 00 00 00  00 00 00 04 00 00 00 ff  |...c............|
000005a0  ff ff ff 03 00 00 00 00  00 00 00 02 00 00 00 03  |................|
000005b0  00 00 00 01 63 00 00 00  00 00 00 00 07 00 00 00  |....c...........|
000005c0  ff ff ff ff 00 00 00 00  01 64 00 00 00 00 00 00  |.........d......|
000005d0  00 03 00 00 00 ff ff ff  ff 03 00 00 00 00 00 00  |................|
000005e0  00 02 00 00 00 03 00 00  00 01 64 00 00 00 00 00  |..........d.....|
000005f0  00 00 06 00 00 00 ff ff  ff ff 00 00 00 00 01 64  |...............d|
00000600  00 00 00 00 00 00 00 07  00 00 00 ff ff ff ff 00  |................|
00000610  00 00 00 01 65 00 00 00  00 00 00 00 02 00 00 00  |....e...........|
00000620  ff ff ff ff 03 00 00 00  00 00 00 00 02 00 00 00  |................|
00000630  03 00 00 00 01 65 00 00  00 00 00 00 00 05 00 00  |.....e..........|
00000640  00 ff ff ff ff 00 00 00  00 01 65 00 00 00 00 00  |..........e.....|
00000650  00 00 06 00 00 00 ff ff  ff ff 00 00 00 00 01 66  |...............f|
00000660  00 00 00 00 00 00 00 01  00 00 00 ff ff ff ff 03  |................|
00000670  00 00 00 00 00 00 00 02  00 00 00 03 00 00 00 01  |................|
00000680  66 00 00 00 00 00 00 00  04 00 00 00 ff ff ff ff  |f...............|
00000690  00 00 00 00 01 66 00 00  00 00 00 00 00 05 00 00  |.....f..........|
000006a0  00 ff ff ff ff 00 00 00  00 01 67 00 00 00 00 00  |..........g.....|
000006b0  00 00 00 00 00 00 ff ff  ff ff 03 00 00 00 00 00  |................|
000006c0  00 00 02 00 00 00 03 00  00 00 01 67 00 00 00 00  |...........g....|
000006d0  00 00 00 03 00 00 00 ff  ff ff ff 00 00 00 00 01  |................|
000006e0  67 00 00 00 00 00 00 00  04 00 00 00 ff ff ff ff  |g...............|
000006f0  00 00 00 00 01 68 00 00  00 00 00 00 00 02 00 00  |.....h..........|
00000700  00 ff ff ff ff 00 00 00  00 01 68 00 00 00 00 00  |..........h.....|
00000710  00 00 03 00 00 00 ff ff  ff ff 00 00 00 00 01 69  |...............i|
00000720  00 00 00 00 00 00 00 01  00 00 00 ff ff ff ff 00  |................|
00000730  00 00 00 01 69 00 00 00  00 00 00 00 02 00 00 00  |....i...........|
00000740  ff ff ff ff 00 00 00 00  01 73 00 00 00 00 00 00  |.........s......|
00000750  00 07 00 00 00 ff ff ff  ff 00 00 00 00 01 74 00  |..............t.|
00000760  00 00 00 00 00 00 06 00  00 00 ff ff ff ff 00 00  |................|
00000770  00 00 01 75 00 00 00 00  00 00 00 05 00 00 00 ff  |...u............|
00000780  ff ff ff 00 00 00 00 01  76 00 00 00 00 00 00 00  |........v.......|
00000790  04 00 00 00 ff ff ff ff  00 00 00 00 01 77 00 00  |.............w..|
000007a0  00 00 00 00 00 03 00 00  00 ff ff ff ff 00 00 00  |................|
000007b0  00 01 78 00 00 00 00 00  00 00 02 00 00 00 ff ff  |..x.............|
000007c0  ff ff 00 00 00 00 01 79  00 00 00 00 00 00 00 01  |.......y........|
000007d0  00 00 00 ff ff ff ff 00  00 00 00 01 7a 00 00 00  |............z...|
000007e0  00 00 00 00 00 00 00 00  ff ff ff ff 00 00 00 00  |................|
000007f0  07 00 00 00 00 00 01 00  00 00 03 00 00 00 6f 03  |..............o.|
00000800  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
0000081a

Related issues 1 (0 open — 1 closed)

Has duplicate Ceph - Bug #2446: libceph: corrupt inc osdmap epoch 24630 off 702 (ffff88001e5d876c of ffff88001e5d84ae-ffff88001e5d888c)Resolved05/18/2012

Actions
Actions
Copy link
 #1

Updated by Sage Weil about 12 years ago

  • Source changed from Development to Community (user)

reencoded to old format (using latest ceph-dencoder) gives us


$ hexdump  -C 880.old 
00000000  05 00 34 b3 0e 21 99 f8  43 ff be f2 a3 c0 04 2b  |..4..!..C......+|
00000010  07 e1 70 03 00 00 10 79  42 4f 30 16 47 1d ff ff  |..p....yBO0.G...|
00000020  ff ff ff ff ff ff 00 00  00 00 00 00 00 00 ff ff  |................|
00000030  ff ff 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000040  00 00 00 00 00 00 00 00  00 00 52 00 00 00 ff ff  |..........R.....|
00000050  06 00 07 00 00 00 03 00  00 00 00 00 00 00 02 00  |................|
00000060  00 00 03 00 00 00 ff ff  03 00 0a 00 00 00 03 00  |................|
00000070  00 00 00 00 00 00 02 00  00 00 03 00 00 00 ff ff  |................|
00000080  02 00 0b 00 00 00 03 00  00 00 00 00 00 00 02 00  |................|
00000090  00 00 03 00 00 00 ff ff  07 00 0f 00 00 00 00 00  |................|
000000a0  00 00 ff ff 06 00 10 00  00 00 00 00 00 00 ff ff  |................|
000000b0  05 00 11 00 00 00 00 00  00 00 ff ff 04 00 12 00  |................|
000000c0  00 00 00 00 00 00 ff ff  02 00 14 00 00 00 00 00  |................|
000000d0  00 00 ff ff 07 00 14 00  00 00 00 00 00 00 ff ff  |................|
000000e0  01 00 15 00 00 00 00 00  00 00 ff ff 06 00 15 00  |................|
000000f0  00 00 00 00 00 00 ff ff  00 00 16 00 00 00 00 00  |................|
00000100  00 00 ff ff 05 00 16 00  00 00 00 00 00 00 ff ff  |................|
00000110  04 00 17 00 00 00 00 00  00 00 ff ff 03 00 18 00  |................|
00000120  00 00 00 00 00 00 ff ff  02 00 19 00 00 00 00 00  |................|
00000130  00 00 ff ff 01 00 1a 00  00 00 00 00 00 00 ff ff  |................|
00000140  00 00 1b 00 00 00 00 00  00 00 ff ff 07 00 23 00  |..............#.|
00000150  00 00 00 00 00 00 ff ff  06 00 24 00 00 00 00 00  |..........$.....|
00000160  00 00 ff ff 05 00 25 00  00 00 00 00 00 00 ff ff  |......%.........|
00000170  04 00 26 00 00 00 00 00  00 00 ff ff 02 00 28 00  |..&...........(.|
00000180  00 00 00 00 00 00 ff ff  01 00 29 00 00 00 00 00  |..........).....|
00000190  00 00 ff ff 00 00 2a 00  00 00 00 00 00 00 ff ff  |......*.........|
000001a0  07 00 33 00 00 00 00 00  00 00 ff ff 06 00 34 00  |..3...........4.|
000001b0  00 00 00 00 00 00 ff ff  05 00 35 00 00 00 00 00  |..........5.....|
000001c0  00 00 ff ff 04 00 36 00  00 00 00 00 00 00 ff ff  |......6.........|
000001d0  03 00 37 00 00 00 00 00  00 00 ff ff 02 00 38 00  |..7...........8.|
000001e0  00 00 00 00 00 00 ff ff  01 00 39 00 00 00 00 00  |..........9.....|
000001f0  00 00 ff ff 06 00 3b 00  00 00 03 00 00 00 00 00  |......;.........|
00000200  00 00 02 00 00 00 03 00  00 00 ff ff 04 00 3d 00  |..............=.|
00000210  00 00 03 00 00 00 00 00  00 00 02 00 00 00 03 00  |................|
00000220  00 00 ff ff 03 00 3e 00  00 00 03 00 00 00 00 00  |......>.........|
00000230  00 00 02 00 00 00 03 00  00 00 ff ff 02 00 3f 00  |..............?.|
00000240  00 00 03 00 00 00 00 00  00 00 02 00 00 00 03 00  |................|
00000250  00 00 ff ff 01 00 40 00  00 00 03 00 00 00 00 00  |......@.........|
00000260  00 00 02 00 00 00 03 00  00 00 ff ff 00 00 41 00  |..............A.|
00000270  00 00 03 00 00 00 00 00  00 00 02 00 00 00 03 00  |................|
00000280  00 00 ff ff 03 00 42 00  00 00 03 00 00 00 03 00  |......B.........|
00000290  00 00 00 00 00 00 04 00  00 00 ff ff 07 00 42 00  |..............B.|
000002a0  00 00 00 00 00 00 ff ff  06 00 43 00 00 00 00 00  |..........C.....|
000002b0  00 00 ff ff 01 00 44 00  00 00 03 00 00 00 03 00  |......D.........|
000002c0  00 00 00 00 00 00 04 00  00 00 ff ff 05 00 44 00  |..............D.|
000002d0  00 00 00 00 00 00 ff ff  03 00 46 00 00 00 00 00  |..........F.....|
000002e0  00 00 ff ff 02 00 47 00  00 00 00 00 00 00 ff ff  |......G.........|
000002f0  01 00 48 00 00 00 00 00  00 00 ff ff 00 00 49 00  |..H...........I.|
00000300  00 00 00 00 00 00 ff ff  06 00 59 00 00 00 00 00  |..........Y.....|
00000310  00 00 ff ff 05 00 5a 00  00 00 00 00 00 00 ff ff  |......Z.........|
00000320  04 00 5b 00 00 00 00 00  00 00 ff ff 02 00 5d 00  |..[...........].|
00000330  00 00 00 00 00 00 ff ff  01 00 5e 00 00 00 00 00  |..........^.....|
00000340  00 00 ff ff 00 00 5f 00  00 00 00 00 00 00 ff ff  |......_.........|
00000350  07 00 60 00 00 00 03 00  00 00 00 00 00 00 02 00  |..`.............|
00000360  00 00 03 00 00 00 ff ff  06 00 61 00 00 00 03 00  |..........a.....|
00000370  00 00 00 00 00 00 02 00  00 00 03 00 00 00 ff ff  |................|
00000380  05 00 62 00 00 00 03 00  00 00 00 00 00 00 02 00  |..b.............|
00000390  00 00 03 00 00 00 ff ff  04 00 63 00 00 00 03 00  |..........c.....|
000003a0  00 00 00 00 00 00 02 00  00 00 03 00 00 00 ff ff  |................|
000003b0  07 00 63 00 00 00 00 00  00 00 ff ff 03 00 64 00  |..c...........d.|
000003c0  00 00 03 00 00 00 00 00  00 00 02 00 00 00 03 00  |................|
000003d0  00 00 ff ff 06 00 64 00  00 00 00 00 00 00 ff ff  |......d.........|
000003e0  07 00 64 00 00 00 00 00  00 00 ff ff 02 00 65 00  |..d...........e.|
000003f0  00 00 03 00 00 00 00 00  00 00 02 00 00 00 03 00  |................|
00000400  00 00 ff ff 05 00 65 00  00 00 00 00 00 00 ff ff  |......e.........|
00000410  06 00 65 00 00 00 00 00  00 00 ff ff 01 00 66 00  |..e...........f.|
00000420  00 00 03 00 00 00 00 00  00 00 02 00 00 00 03 00  |................|
00000430  00 00 ff ff 04 00 66 00  00 00 00 00 00 00 ff ff  |......f.........|
00000440  05 00 66 00 00 00 00 00  00 00 ff ff 00 00 67 00  |..f...........g.|
00000450  00 00 03 00 00 00 00 00  00 00 02 00 00 00 03 00  |................|
00000460  00 00 ff ff 03 00 67 00  00 00 00 00 00 00 ff ff  |......g.........|
00000470  04 00 67 00 00 00 00 00  00 00 ff ff 02 00 68 00  |..g...........h.|
00000480  00 00 00 00 00 00 ff ff  03 00 68 00 00 00 00 00  |..........h.....|
00000490  00 00 ff ff 01 00 69 00  00 00 00 00 00 00 ff ff  |......i.........|
000004a0  02 00 69 00 00 00 00 00  00 00 ff ff 07 00 73 00  |..i...........s.|
000004b0  00 00 00 00 00 00 ff ff  06 00 74 00 00 00 00 00  |..........t.....|
000004c0  00 00 ff ff 05 00 75 00  00 00 00 00 00 00 ff ff  |......u.........|
000004d0  04 00 76 00 00 00 00 00  00 00 ff ff 03 00 77 00  |..v...........w.|
000004e0  00 00 00 00 00 00 ff ff  02 00 78 00 00 00 00 00  |..........x.....|
000004f0  00 00 ff ff 01 00 79 00  00 00 00 00 00 00 ff ff  |......y.........|
00000500  00 00 7a 00 00 00 00 00  00 00                    |..z.......|
0000050a

Actions
Copy link
 #2

Updated by Sage Weil about 12 years ago

  • Assignee set to Alex Elder
Actions
Copy link
 #3

Updated by Alex Elder about 12 years ago

I'm starting to look at this in detail but haven't concluded what went wrong yet.

Does it matter whether it was the monitor or the OSD that hit the problem?
Is it possible to tell?

Actions
Copy link
 #4

Updated by Alex Elder about 12 years ago

I've manually decoded the entire ceph_osdmap dumped in the log and everything
therein looks fine. (This was overkill, but I automated the process a bit.)

The error message indicates the buffer pointer was at offset 102 = 0x66 into the
message buffer. That corresponds to the point at which the last (the third)
osd id has been decoded for the second ceph_pg_mapping in the message. I.e.,
it is here, in osdmap_apply_incremental(), around line 904 in the current code:

pg->pgid = pgid;
                        pg->len = pglen;
                        for (j = 0; j < pglen; j++)
                                pg->osds[j] = ceph_decode_32(p);
       ---> here --->
                        err = __insert_pg_mapping(pg, &map->pg_temp);
                        if (err) {
                                kfree(pg);
                                goto bad;
                        }

What this means is that the call to __insert_pg_mapping() must have
returned an error. The only error that function returns is -EEXIST,
meaning that the pgid we're trying to insert at this point was found
to already be present in the red-black tree.

But that doesn't appear possible, based on the content of the message
I've decoded.

I'm going to have to take another look at what I've done tomorrow to
validate my findings. I think it will also be useful to look at the
precise version of the code that hit this problem. According to Sage,
there is a tar file full of logs (at least) here: fatty:~sage/tmp/oliver

Actions
Copy link
 #5

Updated by Sage Weil about 12 years ago

Oh.. that means the pg_temp mapping was inserted by a previous inc map, probably. we need to find the first instance of weirdness.

Actions
Copy link
 #6

Updated by Sage Weil about 12 years ago

the first badness in the log is below. once it missed one incremental, things probably got out of sync and the pg_temp updates all seemed incorrect (entries already there, etc.).

this is the one to look at, i think. unfortunately it's way bigger.. :(

Feb 20 14:15:11 fcmsnode0 kernel: libceph: corrupt inc osdmap epoch 626 off 1898 (ffffc900265de82e of ffffc900265de0c4-ffffc900265dee0a)
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000000: 05 00 34 b3 0e 21 99 f8 43 ff be f2 a3 c0 04 2b  ..4..!..C......+
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000010: 07 e1 72 02 00 00 d2 45 42 4f 30 6c b5 2b ff ff  ..r....EBO0l.+..
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000020: ff ff ff ff ff ff 00 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000030: ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000040: 00 00 00 00 00 00 00 00 00 00 a6 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000050: 08 00 00 00 00 00 02 00 00 00 00 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000060: 00 00 ff ff 0d 00 00 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000070: 00 00 04 00 00 00 ff ff 1c 00 00 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000080: 00 00 00 00 00 00 04 00 00 00 ff ff 24 00 00 00  ............$...
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000090: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000000a0: 41 00 00 00 00 00 02 00 00 00 00 00 00 00 03 00  A...............
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000000b0: 00 00 ff ff 45 00 00 00 00 00 02 00 00 00 00 00  ....E...........
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000000c0: 00 00 04 00 00 00 ff ff 60 00 00 00 00 00 02 00  ........`.......
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000000d0: 00 00 00 00 00 00 04 00 00 00 ff ff 66 00 00 00  ............f...
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000000e0: 00 00 02 00 00 00 00 00 00 00 04 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000000f0: 67 00 00 00 00 00 02 00 00 00 00 00 00 00 03 00  g...............
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000100: 00 00 ff ff 7b 00 00 00 00 00 02 00 00 00 00 00  ....{...........
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000110: 00 00 03 00 00 00 ff ff 7c 00 00 00 00 00 02 00  ........|.......
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000120: 00 00 00 00 00 00 03 00 00 00 ff ff 81 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000130: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000140: 8b 00 00 00 00 00 02 00 00 00 00 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000150: 00 00 ff ff 8c 00 00 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000160: 00 00 04 00 00 00 ff ff 91 00 00 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000170: 00 00 00 00 00 00 03 00 00 00 ff ff 95 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000180: 00 00 02 00 00 00 00 00 00 00 04 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000190: a1 00 00 00 00 00 02 00 00 00 00 00 00 00 04 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000001a0: 00 00 ff ff ad 00 00 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000001b0: 00 00 04 00 00 00 ff ff b3 00 00 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000001c0: 00 00 00 00 00 00 03 00 00 00 ff ff b4 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000001d0: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000001e0: c0 00 00 00 00 00 02 00 00 00 00 00 00 00 04 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000001f0: 00 00 ff ff c1 00 00 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000200: 00 00 04 00 00 00 ff ff c4 00 00 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000210: 00 00 00 00 00 00 03 00 00 00 ff ff ca 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000220: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000230: cf 00 00 00 00 00 02 00 00 00 00 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000240: 00 00 ff ff d2 00 00 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000250: 00 00 03 00 00 00 ff ff db 00 00 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000260: 00 00 00 00 00 00 03 00 00 00 ff ff dd 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000270: 00 00 02 00 00 00 00 00 00 00 04 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000280: e1 00 00 00 00 00 02 00 00 00 00 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000290: 00 00 ff ff e4 00 00 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000002a0: 00 00 04 00 00 00 ff ff e8 00 00 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000002b0: 00 00 00 00 00 00 04 00 00 00 ff ff ea 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000002c0: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000002d0: f3 00 00 00 00 00 02 00 00 00 00 00 00 00 04 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000002e0: 00 00 ff ff f6 00 00 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000002f0: 00 00 03 00 00 00 ff ff fd 00 00 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000300: 00 00 00 00 00 00 03 00 00 00 ff ff 08 01 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000310: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000320: 09 01 00 00 00 00 02 00 00 00 00 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000330: 00 00 ff ff 0b 01 00 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000340: 00 00 03 00 00 00 ff ff 0c 01 00 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000350: 00 00 00 00 00 00 04 00 00 00 ff ff 10 01 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000360: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000370: 11 01 00 00 00 00 02 00 00 00 00 00 00 00 04 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000380: 00 00 ff ff 12 01 00 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000390: 00 00 03 00 00 00 ff ff 13 01 00 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000003a0: 00 00 00 00 00 00 04 00 00 00 ff ff 1b 01 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000003b0: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000003c0: 22 01 00 00 00 00 02 00 00 00 00 00 00 00 03 00  "...............
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000003d0: 00 00 ff ff 2f 01 00 00 00 00 02 00 00 00 00 00  ..../...........
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000003e0: 00 00 03 00 00 00 ff ff 32 01 00 00 00 00 02 00  ........2.......
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000003f0: 00 00 00 00 00 00 03 00 00 00 ff ff 3a 01 00 00  ............:...
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000400: 00 00 02 00 00 00 00 00 00 00 04 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000410: 3d 01 00 00 00 00 02 00 00 00 00 00 00 00 03 00  =...............
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000420: 00 00 ff ff 3f 01 00 00 00 00 02 00 00 00 00 00  ....?...........
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000430: 00 00 04 00 00 00 ff ff 07 00 01 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000440: 00 00 00 00 00 00 03 00 00 00 ff ff 7a 00 01 00  ............z...
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000450: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................

Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000460: 7b 00 01 00 00 00 02 00 00 00 00 00 00 00 03 00  {...............
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000470: 00 00 ff ff 80 00 01 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000480: 00 00 03 00 00 00 ff ff 8a 00 01 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000490: 00 00 00 00 00 00 03 00 00 00 ff ff 8b 00 01 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000004a0: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000004b0: 90 00 01 00 00 00 02 00 00 00 00 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000004c0: 00 00 ff ff a0 00 01 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000004d0: 00 00 03 00 00 00 ff ff b3 00 01 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000004e0: 00 00 00 00 00 00 03 00 00 00 ff ff c3 00 01 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000004f0: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000500: d1 00 01 00 00 00 02 00 00 00 00 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000510: 00 00 ff ff e7 00 01 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000520: 00 00 03 00 00 00 ff ff e9 00 01 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000530: 00 00 00 00 00 00 03 00 00 00 ff ff f5 00 01 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000540: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000550: 07 01 01 00 00 00 02 00 00 00 00 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000560: 00 00 ff ff 0f 01 01 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000570: 00 00 03 00 00 00 ff ff 11 01 01 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000580: 00 00 00 00 00 00 03 00 00 00 ff ff 12 01 01 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000590: 00 00 02 00 00 00 00 00 00 00 04 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000005a0: 1a 01 01 00 00 00 02 00 00 00 00 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000005b0: 00 00 ff ff 21 01 01 00 00 00 02 00 00 00 00 00  ....!...........
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000005c0: 00 00 03 00 00 00 ff ff 2e 01 01 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000005d0: 00 00 00 00 00 00 03 00 00 00 ff ff 06 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000005e0: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000005f0: 79 00 02 00 00 00 02 00 00 00 00 00 00 00 03 00  y...............
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000600: 00 00 ff ff 7a 00 02 00 00 00 02 00 00 00 00 00  ....z...........
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000610: 00 00 03 00 00 00 ff ff 7f 00 02 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000620: 00 00 00 00 00 00 03 00 00 00 ff ff 89 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000630: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000640: 8a 00 02 00 00 00 02 00 00 00 00 00 00 00 04 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000650: 00 00 ff ff 8f 00 02 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000660: 00 00 03 00 00 00 ff ff 9f 00 02 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000670: 00 00 00 00 00 00 04 00 00 00 ff ff c2 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000680: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000690: d0 00 02 00 00 00 02 00 00 00 00 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000006a0: 00 00 ff ff e6 00 02 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000006b0: 00 00 04 00 00 00 ff ff e8 00 02 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000006c0: 00 00 00 00 00 00 03 00 00 00 ff ff f4 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000006d0: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000006e0: 0e 01 02 00 00 00 02 00 00 00 00 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000006f0: 00 00 ff ff 10 01 02 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000700: 00 00 03 00 00 00 ff ff 11 01 02 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000710: 00 00 00 00 00 00 04 00 00 00 ff ff 19 01 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000720: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000730: 20 01 02 00 00 00 02 00 00 00 00 00 00 00 03 00   ...............
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000740: 00 00 ff ff 2d 01 02 00 00 00 02 00 00 00 00 00  ....-...........
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000750: 00 00 03 00 00 00 ff ff 01 00 03 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000760: 00 00 04 00 00 00 00 00 00 00 ff ff 00 00 04 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000770: 00 00 02 00 00 00 04 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000780: 07 00 04 00 00 00 02 00 00 00 04 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000790: 00 00 ff ff 06 00 05 00 00 00 02 00 00 00 04 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000007a0: 00 00 00 00 00 00 ff ff 02 00 06 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000007b0: 00 00 00 00 00 00 03 00 00 00 ff ff 05 00 06 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000007c0: 00 00 02 00 00 00 04 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000007d0: 04 00 07 00 00 00 02 00 00 00 04 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000007e0: 00 00 ff ff 03 00 08 00 00 00 02 00 00 00 04 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000007f0: 00 00 00 00 00 00 ff ff 02 00 09 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000800: 00 00 04 00 00 00 00 00 00 00 ff ff 01 00 0a 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000810: 00 00 02 00 00 00 04 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000820: 07 00 0a 00 00 00 02 00 00 00 04 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000830: 00 00 ff ff 00 00 0b 00 00 00 02 00 00 00 04 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000840: 00 00 00 00 00 00 ff ff 06 00 0b 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000850: 00 00 04 00 00 00 00 00 00 00 ff ff 07 00 0b 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000860: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000870: 05 00 0c 00 00 00 02 00 00 00 04 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000880: 00 00 ff ff 06 00 0c 00 00 00 02 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000890: 00 00 00 00 00 00 ff ff 04 00 0d 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000008a0: 00 00 04 00 00 00 00 00 00 00 ff ff 05 00 0d 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000008b0: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000008c0: 03 00 0e 00 00 00 02 00 00 00 04 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000008d0: 00 00 ff ff 04 00 0e 00 00 00 02 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000008e0: 00 00 00 00 00 00 ff ff 02 00 0f 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000008f0: 00 00 04 00 00 00 00 00 00 00 ff ff 03 00 0f 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000900: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff  ................

Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000910: 01 00 10 00 00 00 02 00 00 00 04 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000920: 00 00 ff ff 02 00 10 00 00 00 02 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000930: 00 00 00 00 00 00 ff ff 00 00 11 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000940: 00 00 04 00 00 00 00 00 00 00 ff ff 01 00 11 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000950: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000960: 00 00 12 00 00 00 02 00 00 00 03 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000970: 00 00 ff ff 06 00 14 00 00 00 02 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000980: 00 00 00 00 00 00 ff ff 05 00 15 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000990: 00 00 03 00 00 00 00 00 00 00 ff ff 04 00 16 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000009a0: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000009b0: 03 00 17 00 00 00 02 00 00 00 03 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000009c0: 00 00 ff ff 02 00 18 00 00 00 02 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000009d0: 00 00 00 00 00 00 ff ff 01 00 19 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000009e0: 00 00 03 00 00 00 00 00 00 00 ff ff 00 00 1a 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000009f0: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a00: 07 00 1b 00 00 00 02 00 00 00 04 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a10: 00 00 ff ff 00 00 1c 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a20: 00 00 04 00 00 00 ff ff 05 00 1d 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a30: 00 00 04 00 00 00 00 00 00 00 ff ff 04 00 1e 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a40: 00 00 02 00 00 00 04 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a50: 03 00 1f 00 00 00 02 00 00 00 04 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a60: 00 00 ff ff 02 00 20 00 00 00 02 00 00 00 04 00  ...... .........
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a70: 00 00 00 00 00 00 ff ff 01 00 21 00 00 00 02 00  ..........!.....
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a80: 00 00 04 00 00 00 00 00 00 00 ff ff 00 00 22 00  ..............".
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a90: 00 00 02 00 00 00 04 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000aa0: 07 00 25 00 00 00 02 00 00 00 04 00 00 00 00 00  ..%.............
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000ab0: 00 00 ff ff 06 00 26 00 00 00 02 00 00 00 04 00  ......&.........
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000ac0: 00 00 00 00 00 00 ff ff 04 00 28 00 00 00 02 00  ..........(.....
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000ad0: 00 00 04 00 00 00 00 00 00 00 ff ff 03 00 29 00  ..............).
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000ae0: 00 00 02 00 00 00 04 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000af0: 02 00 2a 00 00 00 02 00 00 00 04 00 00 00 00 00  ..*.............
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b00: 00 00 ff ff 01 00 2b 00 00 00 02 00 00 00 04 00  ......+.........
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b10: 00 00 00 00 00 00 ff ff 00 00 2c 00 00 00 02 00  ..........,.....
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b20: 00 00 04 00 00 00 00 00 00 00 ff ff 07 00 3a 00  ..............:.
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b30: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b40: 05 00 3c 00 00 00 02 00 00 00 00 00 00 00 03 00  ..<.............
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b50: 00 00 ff ff 06 00 3f 00 00 00 02 00 00 00 00 00  ......?.........
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b60: 00 00 04 00 00 00 ff ff 04 00 5c 00 00 00 02 00  ..........\.....
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b70: 00 00 00 00 00 00 04 00 00 00 ff ff 07 00 65 00  ..............e.
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b80: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b90: 06 00 66 00 00 00 02 00 00 00 03 00 00 00 00 00  ..f.............
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000ba0: 00 00 ff ff 05 00 67 00 00 00 02 00 00 00 03 00  ......g.........
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000bb0: 00 00 00 00 00 00 ff ff 04 00 68 00 00 00 02 00  ..........h.....
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000bc0: 00 00 03 00 00 00 00 00 00 00 ff ff 03 00 69 00  ..............i.
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000bd0: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000be0: 00 00 6c 00 00 00 02 00 00 00 03 00 00 00 00 00  ..l.............
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000bf0: 00 00 ff ff 07 00 6d 00 00 00 02 00 00 00 04 00  ......m.........
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c00: 00 00 00 00 00 00 ff ff 06 00 6e 00 00 00 02 00  ..........n.....
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c10: 00 00 04 00 00 00 00 00 00 00 ff ff 05 00 6f 00  ..............o.
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c20: 00 00 02 00 00 00 04 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c30: 04 00 70 00 00 00 02 00 00 00 04 00 00 00 00 00  ..p.............
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c40: 00 00 ff ff 03 00 71 00 00 00 02 00 00 00 04 00  ......q.........
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c50: 00 00 00 00 00 00 ff ff 02 00 72 00 00 00 02 00  ..........r.....
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c60: 00 00 04 00 00 00 00 00 00 00 ff ff 01 00 73 00  ..............s.
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c70: 00 00 02 00 00 00 04 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c80: 07 00 74 00 00 00 02 00 00 00 00 00 00 00 03 00  ..t.............
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c90: 00 00 ff ff 07 00 7f 00 00 00 02 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000ca0: 00 00 00 00 00 00 ff ff 06 00 80 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000cb0: 00 00 03 00 00 00 00 00 00 00 ff ff 05 00 81 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000cc0: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000cd0: 04 00 82 00 00 00 02 00 00 00 03 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000ce0: 00 00 ff ff 03 00 83 00 00 00 02 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000cf0: 00 00 00 00 00 00 ff ff 02 00 84 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000d00: 00 00 03 00 00 00 00 00 00 00 ff ff 01 00 85 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000d10: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000d20: 05 00 86 00 00 00 02 00 00 00 00 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000d30: 00 00 ff ff 06 00 86 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000d40: 00 00 04 00 00 00                                ......
Feb 20 14:15:11 fcmsnode0 kernel: libceph: osdc handle_map corrupt msg

Actions
Copy link
 #7

Updated by Dan Mick about 12 years ago

  • Subject changed from currupt v5 inc osdmap to corrupt v5 inc osdmap
Actions
Copy link
 #8

Updated by Alex Elder about 12 years ago

OK, picking a few things out of the original corruption report.

The basic header stuff is the same as before, as expected:

version = 0x0005 = CEPH_OSDMAP_INC_VERSION
fsid = {
34, b3, 0e, 21, 99, f8, 43, ff,
be, f2, a3, c0, 04, 2b, 07, e1
}
epoch = 0x00000272 = 626
modified = {
tv_sec = 0x4f4245d2,
tv_nsec = 0x2bb56c30
} => 1329743314.733310000 = Mon Feb 20 07:08:34 2012
new_pool_max = 0xffffffff => not new
new_flags = 0xffffffff => not new
(map) len = 0
(crush map) len = 0
max = 0xffffffff = -1 => not new

Only the pg mappings got updated:

(number of pools) len = 0
(number of pool names) len = 0
(number of pool id's, old pool) len = 0
(number of newly-up osd indexes) len = 0
(number of osd indexes + state changes) len = 0
(number of osd indexes + weights) len = 0
(number of new pg mappings) len = 0x000000a6 = 166

The message indicates that the corruption was detected when the offset had reached
1898 decimal, or 0x76a.

Here's what the first ceph_pg_mapping looks like:
pg0.pgid = {
preferred = 0xffff = -1
ps = 0x0008 = 8
pool = 0x00000000 = 0
}
pg0 pglen = 0x00000002 = 2
pg0.osds0 = 0x00000000 = 0
pg0.osds0 = 0x00000003 = 3
...

Offset 0x76a is just at the end of another one, meaning we must
have failed (as reported before) in the call to __insert_pg_mapping()
for the one just consumed. Here's what that one looks like:

pg91.pgid = {
preferred = 0xffff = -1
ps = 0x0001 = 1
pool = 0x00000003 = 3
}
pg91 pglen = 0x00000002 = 2
pg91.osds0 = 0x00000004 = 4
pg91.osds0 = 0x00000000 = 0

Evidently there is another entry in the pg_temp red-black tree in the
passed-in ceph_osdmap having a pgid that matches the one above.

None of the ceph_pg_mapping structures earlier in this map update
have that same pgid, it must have existed before the incremental
update started.

At this point I can't tell much more about this, and I'm not sure how
best to proceed.

Actions
Copy link
 #9

Updated by Sage Weil about 12 years ago

ok.. yeah, it looks like the monitor may have published a bad inc update or something? unclear. i'll check with them and see if they have this map history..

Actions
Copy link
 #10

Updated by Sage Weil about 12 years ago

  • Status changed from New to Can't reproduce
  • Assignee deleted (Alex Elder)

logs don't go far enough back. :(

moral of the story: next time grab the full mon data dir immediately in case it is useful later.

i don't think there's anything else we can do here, aside from blindly reading code and looking for possible bugs.

Actions
Copy link

Also available in: Atom PDF