Bug #2091
closedcorrupt v5 inc osdmap
0%
Description
Feb 20 17:54:54 fcmsnode0 kernel: libceph: corrupt inc osdmap epoch 880 off 102 (ffffc9001db8990a of ffffc9001db898a4-ffffc9001db89dae) Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000000: 05 00 34 b3 0e 21 99 f8 43 ff be f2 a3 c0 04 2b ..4..!..C......+ Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000010: 07 e1 70 03 00 00 10 79 42 4f 30 16 47 1d ff ff ..p....yBO0.G... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000020: ff ff ff ff ff ff 00 00 00 00 00 00 00 00 ff ff ................ Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000030: ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000040: 00 00 00 00 00 00 00 00 00 00 52 00 00 00 ff ff ..........R..... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000050: 06 00 07 00 00 00 03 00 00 00 00 00 00 00 02 00 ................ Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000060: 00 00 03 00 00 00 ff ff 03 00 0a 00 00 00 03 00 ................ Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000070: 00 00 00 00 00 00 02 00 00 00 03 00 00 00 ff ff ................ Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000080: 02 00 0b 00 00 00 03 00 00 00 00 00 00 00 02 00 ................ Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000090: 00 00 03 00 00 00 ff ff 07 00 0f 00 00 00 00 00 ................ Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000000a0: 00 00 ff ff 06 00 10 00 00 00 00 00 00 00 ff ff ................ Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000000b0: 05 00 11 00 00 00 00 00 00 00 ff ff 04 00 12 00 ................ Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000000c0: 00 00 00 00 00 00 ff ff 02 00 14 00 00 00 00 00 ................ Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000000d0: 00 00 ff ff 07 00 14 00 00 00 00 00 00 00 ff ff ................ Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000000e0: 01 00 15 00 00 00 00 00 00 00 ff ff 06 00 15 00 ................ Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000000f0: 00 00 00 00 00 00 ff ff 00 00 16 00 00 00 00 00 ................ Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000100: 00 00 ff ff 05 00 16 00 00 00 00 00 00 00 ff ff ................ Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000110: 04 00 17 00 00 00 00 00 00 00 ff ff 03 00 18 00 ................ Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000120: 00 00 00 00 00 00 ff ff 02 00 19 00 00 00 00 00 ................ Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000130: 00 00 ff ff 01 00 1a 00 00 00 00 00 00 00 ff ff ................ Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000140: 00 00 1b 00 00 00 00 00 00 00 ff ff 07 00 23 00 ..............#. Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000150: 00 00 00 00 00 00 ff ff 06 00 24 00 00 00 00 00 ..........$..... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000160: 00 00 ff ff 05 00 25 00 00 00 00 00 00 00 ff ff ......%......... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000170: 04 00 26 00 00 00 00 00 00 00 ff ff 02 00 28 00 ..&...........(. Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000180: 00 00 00 00 00 00 ff ff 01 00 29 00 00 00 00 00 ..........)..... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000190: 00 00 ff ff 00 00 2a 00 00 00 00 00 00 00 ff ff ......*......... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000001a0: 07 00 33 00 00 00 00 00 00 00 ff ff 06 00 34 00 ..3...........4. Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000001b0: 00 00 00 00 00 00 ff ff 05 00 35 00 00 00 00 00 ..........5..... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000001c0: 00 00 ff ff 04 00 36 00 00 00 00 00 00 00 ff ff ......6......... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000001d0: 03 00 37 00 00 00 00 00 00 00 ff ff 02 00 38 00 ..7...........8. Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000001e0: 00 00 00 00 00 00 ff ff 01 00 39 00 00 00 00 00 ..........9..... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000001f0: 00 00 ff ff 06 00 3b 00 00 00 03 00 00 00 00 00 ......;......... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000200: 00 00 02 00 00 00 03 00 00 00 ff ff 04 00 3d 00 ..............=. Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000210: 00 00 03 00 00 00 00 00 00 00 02 00 00 00 03 00 ................ Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000220: 00 00 ff ff 03 00 3e 00 00 00 03 00 00 00 00 00 ......>......... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000230: 00 00 02 00 00 00 03 00 00 00 ff ff 02 00 3f 00 ..............?. Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000240: 00 00 03 00 00 00 00 00 00 00 02 00 00 00 03 00 ................ Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000250: 00 00 ff ff 01 00 40 00 00 00 03 00 00 00 00 00 ......@......... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000260: 00 00 02 00 00 00 03 00 00 00 ff ff 00 00 41 00 ..............A. Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000270: 00 00 03 00 00 00 00 00 00 00 02 00 00 00 03 00 ................ Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000280: 00 00 ff ff 03 00 42 00 00 00 03 00 00 00 03 00 ......B......... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000290: 00 00 00 00 00 00 04 00 00 00 ff ff 07 00 42 00 ..............B. Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000002a0: 00 00 00 00 00 00 ff ff 06 00 43 00 00 00 00 00 ..........C..... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000002b0: 00 00 ff ff 01 00 44 00 00 00 03 00 00 00 03 00 ......D......... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000002c0: 00 00 00 00 00 00 04 00 00 00 ff ff 05 00 44 00 ..............D. Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000002d0: 00 00 00 00 00 00 ff ff 03 00 46 00 00 00 00 00 ..........F..... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000002e0: 00 00 ff ff 02 00 47 00 00 00 00 00 00 00 ff ff ......G......... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000002f0: 01 00 48 00 00 00 00 00 00 00 ff ff 00 00 49 00 ..H...........I. Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000300: 00 00 00 00 00 00 ff ff 06 00 59 00 00 00 00 00 ..........Y..... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000310: 00 00 ff ff 05 00 5a 00 00 00 00 00 00 00 ff ff ......Z......... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000320: 04 00 5b 00 00 00 00 00 00 00 ff ff 02 00 5d 00 ..[...........]. Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000330: 00 00 00 00 00 00 ff ff 01 00 5e 00 00 00 00 00 ..........^..... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000340: 00 00 ff ff 00 00 5f 00 00 00 00 00 00 00 ff ff ......_......... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000350: 07 00 60 00 00 00 03 00 00 00 00 00 00 00 02 00 ..`............. Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000360: 00 00 03 00 00 00 ff ff 06 00 61 00 00 00 03 00 ..........a..... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000370: 00 00 00 00 00 00 02 00 00 00 03 00 00 00 ff ff ................ Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000380: 05 00 62 00 00 00 03 00 00 00 00 00 00 00 02 00 ..b............. Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000390: 00 00 03 00 00 00 ff ff 04 00 63 00 00 00 03 00 ..........c..... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000003a0: 00 00 00 00 00 00 02 00 00 00 03 00 00 00 ff ff ................ Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000003b0: 07 00 63 00 00 00 00 00 00 00 ff ff 03 00 64 00 ..c...........d. Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000003c0: 00 00 03 00 00 00 00 00 00 00 02 00 00 00 03 00 ................ Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000003d0: 00 00 ff ff 06 00 64 00 00 00 00 00 00 00 ff ff ......d......... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000003e0: 07 00 64 00 00 00 00 00 00 00 ff ff 02 00 65 00 ..d...........e. Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000003f0: 00 00 03 00 00 00 00 00 00 00 02 00 00 00 03 00 ................ Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000400: 00 00 ff ff 05 00 65 00 00 00 00 00 00 00 ff ff ......e......... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000410: 06 00 65 00 00 00 00 00 00 00 ff ff 01 00 66 00 ..e...........f. Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000420: 00 00 03 00 00 00 00 00 00 00 02 00 00 00 03 00 ................ Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000430: 00 00 ff ff 04 00 66 00 00 00 00 00 00 00 ff ff ......f......... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000440: 05 00 66 00 00 00 00 00 00 00 ff ff 00 00 67 00 ..f...........g. Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000450: 00 00 03 00 00 00 00 00 00 00 02 00 00 00 03 00 ................ Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000460: 00 00 ff ff 03 00 67 00 00 00 00 00 00 00 ff ff ......g......... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000470: 04 00 67 00 00 00 00 00 00 00 ff ff 02 00 68 00 ..g...........h. Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000480: 00 00 00 00 00 00 ff ff 03 00 68 00 00 00 00 00 ..........h..... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000490: 00 00 ff ff 01 00 69 00 00 00 00 00 00 00 ff ff ......i......... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000004a0: 02 00 69 00 00 00 00 00 00 00 ff ff 07 00 73 00 ..i...........s. Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000004b0: 00 00 00 00 00 00 ff ff 06 00 74 00 00 00 00 00 ..........t..... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000004c0: 00 00 ff ff 05 00 75 00 00 00 00 00 00 00 ff ff ......u......... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000004d0: 04 00 76 00 00 00 00 00 00 00 ff ff 03 00 77 00 ..v...........w. Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000004e0: 00 00 00 00 00 00 ff ff 02 00 78 00 00 00 00 00 ..........x..... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 000004f0: 00 00 ff ff 01 00 79 00 00 00 00 00 00 00 ff ff ......y......... Feb 20 17:54:54 fcmsnode0 kernel: osdmap: 00000500: 00 00 7a 00 00 00 00 00 00 00 ..z....... the original 880 inc, from the monitor: fatty:~ 09:18 AM $ hexdump -C /tmp/880 00000000 06 00 34 b3 0e 21 99 f8 43 ff be f2 a3 c0 04 2b |..4..!..C......+| 00000010 07 e1 70 03 00 00 10 79 42 4f 30 16 47 1d ff ff |..p....yBO0.G...| 00000020 ff ff ff ff ff ff ff ff ff ff 00 00 00 00 00 00 |................| 00000030 00 00 ff ff ff ff 00 00 00 00 00 00 00 00 00 00 |................| 00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 52 00 |..............R.| 00000050 00 00 01 07 00 00 00 00 00 00 00 06 00 00 00 ff |................| 00000060 ff ff ff 03 00 00 00 00 00 00 00 02 00 00 00 03 |................| 00000070 00 00 00 01 0a 00 00 00 00 00 00 00 03 00 00 00 |................| 00000080 ff ff ff ff 03 00 00 00 00 00 00 00 02 00 00 00 |................| 00000090 03 00 00 00 01 0b 00 00 00 00 00 00 00 02 00 00 |................| 000000a0 00 ff ff ff ff 03 00 00 00 00 00 00 00 02 00 00 |................| 000000b0 00 03 00 00 00 01 0f 00 00 00 00 00 00 00 07 00 |................| 000000c0 00 00 ff ff ff ff 00 00 00 00 01 10 00 00 00 00 |................| 000000d0 00 00 00 06 00 00 00 ff ff ff ff 00 00 00 00 01 |................| 000000e0 11 00 00 00 00 00 00 00 05 00 00 00 ff ff ff ff |................| 000000f0 00 00 00 00 01 12 00 00 00 00 00 00 00 04 00 00 |................| 00000100 00 ff ff ff ff 00 00 00 00 01 14 00 00 00 00 00 |................| 00000110 00 00 02 00 00 00 ff ff ff ff 00 00 00 00 01 14 |................| 00000120 00 00 00 00 00 00 00 07 00 00 00 ff ff ff ff 00 |................| 00000130 00 00 00 01 15 00 00 00 00 00 00 00 01 00 00 00 |................| 00000140 ff ff ff ff 00 00 00 00 01 15 00 00 00 00 00 00 |................| 00000150 00 06 00 00 00 ff ff ff ff 00 00 00 00 01 16 00 |................| 00000160 00 00 00 00 00 00 00 00 00 00 ff ff ff ff 00 00 |................| 00000170 00 00 01 16 00 00 00 00 00 00 00 05 00 00 00 ff |................| 00000180 ff ff ff 00 00 00 00 01 17 00 00 00 00 00 00 00 |................| 00000190 04 00 00 00 ff ff ff ff 00 00 00 00 01 18 00 00 |................| 000001a0 00 00 00 00 00 03 00 00 00 ff ff ff ff 00 00 00 |................| 000001b0 00 01 19 00 00 00 00 00 00 00 02 00 00 00 ff ff |................| 000001c0 ff ff 00 00 00 00 01 1a 00 00 00 00 00 00 00 01 |................| 000001d0 00 00 00 ff ff ff ff 00 00 00 00 01 1b 00 00 00 |................| 000001e0 00 00 00 00 00 00 00 00 ff ff ff ff 00 00 00 00 |................| 000001f0 01 23 00 00 00 00 00 00 00 07 00 00 00 ff ff ff |.#..............| 00000200 ff 00 00 00 00 01 24 00 00 00 00 00 00 00 06 00 |......$.........| 00000210 00 00 ff ff ff ff 00 00 00 00 01 25 00 00 00 00 |...........%....| 00000220 00 00 00 05 00 00 00 ff ff ff ff 00 00 00 00 01 |................| 00000230 26 00 00 00 00 00 00 00 04 00 00 00 ff ff ff ff |&...............| 00000240 00 00 00 00 01 28 00 00 00 00 00 00 00 02 00 00 |.....(..........| 00000250 00 ff ff ff ff 00 00 00 00 01 29 00 00 00 00 00 |..........).....| 00000260 00 00 01 00 00 00 ff ff ff ff 00 00 00 00 01 2a |...............*| 00000270 00 00 00 00 00 00 00 00 00 00 00 ff ff ff ff 00 |................| 00000280 00 00 00 01 33 00 00 00 00 00 00 00 07 00 00 00 |....3...........| 00000290 ff ff ff ff 00 00 00 00 01 34 00 00 00 00 00 00 |.........4......| 000002a0 00 06 00 00 00 ff ff ff ff 00 00 00 00 01 35 00 |..............5.| 000002b0 00 00 00 00 00 00 05 00 00 00 ff ff ff ff 00 00 |................| 000002c0 00 00 01 36 00 00 00 00 00 00 00 04 00 00 00 ff |...6............| 000002d0 ff ff ff 00 00 00 00 01 37 00 00 00 00 00 00 00 |........7.......| 000002e0 03 00 00 00 ff ff ff ff 00 00 00 00 01 38 00 00 |.............8..| 000002f0 00 00 00 00 00 02 00 00 00 ff ff ff ff 00 00 00 |................| 00000300 00 01 39 00 00 00 00 00 00 00 01 00 00 00 ff ff |..9.............| 00000310 ff ff 00 00 00 00 01 3b 00 00 00 00 00 00 00 06 |.......;........| 00000320 00 00 00 ff ff ff ff 03 00 00 00 00 00 00 00 02 |................| 00000330 00 00 00 03 00 00 00 01 3d 00 00 00 00 00 00 00 |........=.......| 00000340 04 00 00 00 ff ff ff ff 03 00 00 00 00 00 00 00 |................| 00000350 02 00 00 00 03 00 00 00 01 3e 00 00 00 00 00 00 |.........>......| 00000360 00 03 00 00 00 ff ff ff ff 03 00 00 00 00 00 00 |................| 00000370 00 02 00 00 00 03 00 00 00 01 3f 00 00 00 00 00 |..........?.....| 00000380 00 00 02 00 00 00 ff ff ff ff 03 00 00 00 00 00 |................| 00000390 00 00 02 00 00 00 03 00 00 00 01 40 00 00 00 00 |...........@....| 000003a0 00 00 00 01 00 00 00 ff ff ff ff 03 00 00 00 00 |................| 000003b0 00 00 00 02 00 00 00 03 00 00 00 01 41 00 00 00 |............A...| 000003c0 00 00 00 00 00 00 00 00 ff ff ff ff 03 00 00 00 |................| 000003d0 00 00 00 00 02 00 00 00 03 00 00 00 01 42 00 00 |.............B..| 000003e0 00 00 00 00 00 03 00 00 00 ff ff ff ff 03 00 00 |................| 000003f0 00 03 00 00 00 00 00 00 00 04 00 00 00 01 42 00 |..............B.| 00000400 00 00 00 00 00 00 07 00 00 00 ff ff ff ff 00 00 |................| 00000410 00 00 01 43 00 00 00 00 00 00 00 06 00 00 00 ff |...C............| 00000420 ff ff ff 00 00 00 00 01 44 00 00 00 00 00 00 00 |........D.......| 00000430 01 00 00 00 ff ff ff ff 03 00 00 00 03 00 00 00 |................| 00000440 00 00 00 00 04 00 00 00 01 44 00 00 00 00 00 00 |.........D......| 00000450 00 05 00 00 00 ff ff ff ff 00 00 00 00 01 46 00 |..............F.| 00000460 00 00 00 00 00 00 03 00 00 00 ff ff ff ff 00 00 |................| 00000470 00 00 01 47 00 00 00 00 00 00 00 02 00 00 00 ff |...G............| 00000480 ff ff ff 00 00 00 00 01 48 00 00 00 00 00 00 00 |........H.......| 00000490 01 00 00 00 ff ff ff ff 00 00 00 00 01 49 00 00 |.............I..| 000004a0 00 00 00 00 00 00 00 00 00 ff ff ff ff 00 00 00 |................| 000004b0 00 01 59 00 00 00 00 00 00 00 06 00 00 00 ff ff |..Y.............| 000004c0 ff ff 00 00 00 00 01 5a 00 00 00 00 00 00 00 05 |.......Z........| 000004d0 00 00 00 ff ff ff ff 00 00 00 00 01 5b 00 00 00 |............[...| 000004e0 00 00 00 00 04 00 00 00 ff ff ff ff 00 00 00 00 |................| 000004f0 01 5d 00 00 00 00 00 00 00 02 00 00 00 ff ff ff |.]..............| 00000500 ff 00 00 00 00 01 5e 00 00 00 00 00 00 00 01 00 |......^.........| 00000510 00 00 ff ff ff ff 00 00 00 00 01 5f 00 00 00 00 |..........._....| 00000520 00 00 00 00 00 00 00 ff ff ff ff 00 00 00 00 01 |................| 00000530 60 00 00 00 00 00 00 00 07 00 00 00 ff ff ff ff |`...............| 00000540 03 00 00 00 00 00 00 00 02 00 00 00 03 00 00 00 |................| 00000550 01 61 00 00 00 00 00 00 00 06 00 00 00 ff ff ff |.a..............| 00000560 ff 03 00 00 00 00 00 00 00 02 00 00 00 03 00 00 |................| 00000570 00 01 62 00 00 00 00 00 00 00 05 00 00 00 ff ff |..b.............| 00000580 ff ff 03 00 00 00 00 00 00 00 02 00 00 00 03 00 |................| 00000590 00 00 01 63 00 00 00 00 00 00 00 04 00 00 00 ff |...c............| 000005a0 ff ff ff 03 00 00 00 00 00 00 00 02 00 00 00 03 |................| 000005b0 00 00 00 01 63 00 00 00 00 00 00 00 07 00 00 00 |....c...........| 000005c0 ff ff ff ff 00 00 00 00 01 64 00 00 00 00 00 00 |.........d......| 000005d0 00 03 00 00 00 ff ff ff ff 03 00 00 00 00 00 00 |................| 000005e0 00 02 00 00 00 03 00 00 00 01 64 00 00 00 00 00 |..........d.....| 000005f0 00 00 06 00 00 00 ff ff ff ff 00 00 00 00 01 64 |...............d| 00000600 00 00 00 00 00 00 00 07 00 00 00 ff ff ff ff 00 |................| 00000610 00 00 00 01 65 00 00 00 00 00 00 00 02 00 00 00 |....e...........| 00000620 ff ff ff ff 03 00 00 00 00 00 00 00 02 00 00 00 |................| 00000630 03 00 00 00 01 65 00 00 00 00 00 00 00 05 00 00 |.....e..........| 00000640 00 ff ff ff ff 00 00 00 00 01 65 00 00 00 00 00 |..........e.....| 00000650 00 00 06 00 00 00 ff ff ff ff 00 00 00 00 01 66 |...............f| 00000660 00 00 00 00 00 00 00 01 00 00 00 ff ff ff ff 03 |................| 00000670 00 00 00 00 00 00 00 02 00 00 00 03 00 00 00 01 |................| 00000680 66 00 00 00 00 00 00 00 04 00 00 00 ff ff ff ff |f...............| 00000690 00 00 00 00 01 66 00 00 00 00 00 00 00 05 00 00 |.....f..........| 000006a0 00 ff ff ff ff 00 00 00 00 01 67 00 00 00 00 00 |..........g.....| 000006b0 00 00 00 00 00 00 ff ff ff ff 03 00 00 00 00 00 |................| 000006c0 00 00 02 00 00 00 03 00 00 00 01 67 00 00 00 00 |...........g....| 000006d0 00 00 00 03 00 00 00 ff ff ff ff 00 00 00 00 01 |................| 000006e0 67 00 00 00 00 00 00 00 04 00 00 00 ff ff ff ff |g...............| 000006f0 00 00 00 00 01 68 00 00 00 00 00 00 00 02 00 00 |.....h..........| 00000700 00 ff ff ff ff 00 00 00 00 01 68 00 00 00 00 00 |..........h.....| 00000710 00 00 03 00 00 00 ff ff ff ff 00 00 00 00 01 69 |...............i| 00000720 00 00 00 00 00 00 00 01 00 00 00 ff ff ff ff 00 |................| 00000730 00 00 00 01 69 00 00 00 00 00 00 00 02 00 00 00 |....i...........| 00000740 ff ff ff ff 00 00 00 00 01 73 00 00 00 00 00 00 |.........s......| 00000750 00 07 00 00 00 ff ff ff ff 00 00 00 00 01 74 00 |..............t.| 00000760 00 00 00 00 00 00 06 00 00 00 ff ff ff ff 00 00 |................| 00000770 00 00 01 75 00 00 00 00 00 00 00 05 00 00 00 ff |...u............| 00000780 ff ff ff 00 00 00 00 01 76 00 00 00 00 00 00 00 |........v.......| 00000790 04 00 00 00 ff ff ff ff 00 00 00 00 01 77 00 00 |.............w..| 000007a0 00 00 00 00 00 03 00 00 00 ff ff ff ff 00 00 00 |................| 000007b0 00 01 78 00 00 00 00 00 00 00 02 00 00 00 ff ff |..x.............| 000007c0 ff ff 00 00 00 00 01 79 00 00 00 00 00 00 00 01 |.......y........| 000007d0 00 00 00 ff ff ff ff 00 00 00 00 01 7a 00 00 00 |............z...| 000007e0 00 00 00 00 00 00 00 00 ff ff ff ff 00 00 00 00 |................| 000007f0 07 00 00 00 00 00 01 00 00 00 03 00 00 00 6f 03 |..............o.| 00000800 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 0000081a
Updated by Sage Weil about 12 years ago
- Source changed from Development to Community (user)
reencoded to old format (using latest ceph-dencoder) gives us
$ hexdump -C 880.old 00000000 05 00 34 b3 0e 21 99 f8 43 ff be f2 a3 c0 04 2b |..4..!..C......+| 00000010 07 e1 70 03 00 00 10 79 42 4f 30 16 47 1d ff ff |..p....yBO0.G...| 00000020 ff ff ff ff ff ff 00 00 00 00 00 00 00 00 ff ff |................| 00000030 ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000040 00 00 00 00 00 00 00 00 00 00 52 00 00 00 ff ff |..........R.....| 00000050 06 00 07 00 00 00 03 00 00 00 00 00 00 00 02 00 |................| 00000060 00 00 03 00 00 00 ff ff 03 00 0a 00 00 00 03 00 |................| 00000070 00 00 00 00 00 00 02 00 00 00 03 00 00 00 ff ff |................| 00000080 02 00 0b 00 00 00 03 00 00 00 00 00 00 00 02 00 |................| 00000090 00 00 03 00 00 00 ff ff 07 00 0f 00 00 00 00 00 |................| 000000a0 00 00 ff ff 06 00 10 00 00 00 00 00 00 00 ff ff |................| 000000b0 05 00 11 00 00 00 00 00 00 00 ff ff 04 00 12 00 |................| 000000c0 00 00 00 00 00 00 ff ff 02 00 14 00 00 00 00 00 |................| 000000d0 00 00 ff ff 07 00 14 00 00 00 00 00 00 00 ff ff |................| 000000e0 01 00 15 00 00 00 00 00 00 00 ff ff 06 00 15 00 |................| 000000f0 00 00 00 00 00 00 ff ff 00 00 16 00 00 00 00 00 |................| 00000100 00 00 ff ff 05 00 16 00 00 00 00 00 00 00 ff ff |................| 00000110 04 00 17 00 00 00 00 00 00 00 ff ff 03 00 18 00 |................| 00000120 00 00 00 00 00 00 ff ff 02 00 19 00 00 00 00 00 |................| 00000130 00 00 ff ff 01 00 1a 00 00 00 00 00 00 00 ff ff |................| 00000140 00 00 1b 00 00 00 00 00 00 00 ff ff 07 00 23 00 |..............#.| 00000150 00 00 00 00 00 00 ff ff 06 00 24 00 00 00 00 00 |..........$.....| 00000160 00 00 ff ff 05 00 25 00 00 00 00 00 00 00 ff ff |......%.........| 00000170 04 00 26 00 00 00 00 00 00 00 ff ff 02 00 28 00 |..&...........(.| 00000180 00 00 00 00 00 00 ff ff 01 00 29 00 00 00 00 00 |..........).....| 00000190 00 00 ff ff 00 00 2a 00 00 00 00 00 00 00 ff ff |......*.........| 000001a0 07 00 33 00 00 00 00 00 00 00 ff ff 06 00 34 00 |..3...........4.| 000001b0 00 00 00 00 00 00 ff ff 05 00 35 00 00 00 00 00 |..........5.....| 000001c0 00 00 ff ff 04 00 36 00 00 00 00 00 00 00 ff ff |......6.........| 000001d0 03 00 37 00 00 00 00 00 00 00 ff ff 02 00 38 00 |..7...........8.| 000001e0 00 00 00 00 00 00 ff ff 01 00 39 00 00 00 00 00 |..........9.....| 000001f0 00 00 ff ff 06 00 3b 00 00 00 03 00 00 00 00 00 |......;.........| 00000200 00 00 02 00 00 00 03 00 00 00 ff ff 04 00 3d 00 |..............=.| 00000210 00 00 03 00 00 00 00 00 00 00 02 00 00 00 03 00 |................| 00000220 00 00 ff ff 03 00 3e 00 00 00 03 00 00 00 00 00 |......>.........| 00000230 00 00 02 00 00 00 03 00 00 00 ff ff 02 00 3f 00 |..............?.| 00000240 00 00 03 00 00 00 00 00 00 00 02 00 00 00 03 00 |................| 00000250 00 00 ff ff 01 00 40 00 00 00 03 00 00 00 00 00 |......@.........| 00000260 00 00 02 00 00 00 03 00 00 00 ff ff 00 00 41 00 |..............A.| 00000270 00 00 03 00 00 00 00 00 00 00 02 00 00 00 03 00 |................| 00000280 00 00 ff ff 03 00 42 00 00 00 03 00 00 00 03 00 |......B.........| 00000290 00 00 00 00 00 00 04 00 00 00 ff ff 07 00 42 00 |..............B.| 000002a0 00 00 00 00 00 00 ff ff 06 00 43 00 00 00 00 00 |..........C.....| 000002b0 00 00 ff ff 01 00 44 00 00 00 03 00 00 00 03 00 |......D.........| 000002c0 00 00 00 00 00 00 04 00 00 00 ff ff 05 00 44 00 |..............D.| 000002d0 00 00 00 00 00 00 ff ff 03 00 46 00 00 00 00 00 |..........F.....| 000002e0 00 00 ff ff 02 00 47 00 00 00 00 00 00 00 ff ff |......G.........| 000002f0 01 00 48 00 00 00 00 00 00 00 ff ff 00 00 49 00 |..H...........I.| 00000300 00 00 00 00 00 00 ff ff 06 00 59 00 00 00 00 00 |..........Y.....| 00000310 00 00 ff ff 05 00 5a 00 00 00 00 00 00 00 ff ff |......Z.........| 00000320 04 00 5b 00 00 00 00 00 00 00 ff ff 02 00 5d 00 |..[...........].| 00000330 00 00 00 00 00 00 ff ff 01 00 5e 00 00 00 00 00 |..........^.....| 00000340 00 00 ff ff 00 00 5f 00 00 00 00 00 00 00 ff ff |......_.........| 00000350 07 00 60 00 00 00 03 00 00 00 00 00 00 00 02 00 |..`.............| 00000360 00 00 03 00 00 00 ff ff 06 00 61 00 00 00 03 00 |..........a.....| 00000370 00 00 00 00 00 00 02 00 00 00 03 00 00 00 ff ff |................| 00000380 05 00 62 00 00 00 03 00 00 00 00 00 00 00 02 00 |..b.............| 00000390 00 00 03 00 00 00 ff ff 04 00 63 00 00 00 03 00 |..........c.....| 000003a0 00 00 00 00 00 00 02 00 00 00 03 00 00 00 ff ff |................| 000003b0 07 00 63 00 00 00 00 00 00 00 ff ff 03 00 64 00 |..c...........d.| 000003c0 00 00 03 00 00 00 00 00 00 00 02 00 00 00 03 00 |................| 000003d0 00 00 ff ff 06 00 64 00 00 00 00 00 00 00 ff ff |......d.........| 000003e0 07 00 64 00 00 00 00 00 00 00 ff ff 02 00 65 00 |..d...........e.| 000003f0 00 00 03 00 00 00 00 00 00 00 02 00 00 00 03 00 |................| 00000400 00 00 ff ff 05 00 65 00 00 00 00 00 00 00 ff ff |......e.........| 00000410 06 00 65 00 00 00 00 00 00 00 ff ff 01 00 66 00 |..e...........f.| 00000420 00 00 03 00 00 00 00 00 00 00 02 00 00 00 03 00 |................| 00000430 00 00 ff ff 04 00 66 00 00 00 00 00 00 00 ff ff |......f.........| 00000440 05 00 66 00 00 00 00 00 00 00 ff ff 00 00 67 00 |..f...........g.| 00000450 00 00 03 00 00 00 00 00 00 00 02 00 00 00 03 00 |................| 00000460 00 00 ff ff 03 00 67 00 00 00 00 00 00 00 ff ff |......g.........| 00000470 04 00 67 00 00 00 00 00 00 00 ff ff 02 00 68 00 |..g...........h.| 00000480 00 00 00 00 00 00 ff ff 03 00 68 00 00 00 00 00 |..........h.....| 00000490 00 00 ff ff 01 00 69 00 00 00 00 00 00 00 ff ff |......i.........| 000004a0 02 00 69 00 00 00 00 00 00 00 ff ff 07 00 73 00 |..i...........s.| 000004b0 00 00 00 00 00 00 ff ff 06 00 74 00 00 00 00 00 |..........t.....| 000004c0 00 00 ff ff 05 00 75 00 00 00 00 00 00 00 ff ff |......u.........| 000004d0 04 00 76 00 00 00 00 00 00 00 ff ff 03 00 77 00 |..v...........w.| 000004e0 00 00 00 00 00 00 ff ff 02 00 78 00 00 00 00 00 |..........x.....| 000004f0 00 00 ff ff 01 00 79 00 00 00 00 00 00 00 ff ff |......y.........| 00000500 00 00 7a 00 00 00 00 00 00 00 |..z.......| 0000050a
Updated by Alex Elder about 12 years ago
I'm starting to look at this in detail but haven't concluded what went wrong yet.
Does it matter whether it was the monitor or the OSD that hit the problem?
Is it possible to tell?
Updated by Alex Elder about 12 years ago
I've manually decoded the entire ceph_osdmap dumped in the log and everything
therein looks fine. (This was overkill, but I automated the process a bit.)
The error message indicates the buffer pointer was at offset 102 = 0x66 into the
message buffer. That corresponds to the point at which the last (the third)
osd id has been decoded for the second ceph_pg_mapping in the message. I.e.,
it is here, in osdmap_apply_incremental(), around line 904 in the current code:
pg->pgid = pgid;
pg->len = pglen;
for (j = 0; j < pglen; j++)
pg->osds[j] = ceph_decode_32(p);
---> here --->
err = __insert_pg_mapping(pg, &map->pg_temp);
if (err) {
kfree(pg);
goto bad;
}
What this means is that the call to __insert_pg_mapping() must have
returned an error. The only error that function returns is -EEXIST,
meaning that the pgid we're trying to insert at this point was found
to already be present in the red-black tree.
But that doesn't appear possible, based on the content of the message
I've decoded.
I'm going to have to take another look at what I've done tomorrow to
validate my findings. I think it will also be useful to look at the
precise version of the code that hit this problem. According to Sage,
there is a tar file full of logs (at least) here: fatty:~sage/tmp/oliver
Updated by Sage Weil about 12 years ago
Oh.. that means the pg_temp mapping was inserted by a previous inc map, probably. we need to find the first instance of weirdness.
Updated by Sage Weil about 12 years ago
the first badness in the log is below. once it missed one incremental, things probably got out of sync and the pg_temp updates all seemed incorrect (entries already there, etc.).
this is the one to look at, i think. unfortunately it's way bigger.. :(
Feb 20 14:15:11 fcmsnode0 kernel: libceph: corrupt inc osdmap epoch 626 off 1898 (ffffc900265de82e of ffffc900265de0c4-ffffc900265dee0a) Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000000: 05 00 34 b3 0e 21 99 f8 43 ff be f2 a3 c0 04 2b ..4..!..C......+ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000010: 07 e1 72 02 00 00 d2 45 42 4f 30 6c b5 2b ff ff ..r....EBO0l.+.. Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000020: ff ff ff ff ff ff 00 00 00 00 00 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000030: ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000040: 00 00 00 00 00 00 00 00 00 00 a6 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000050: 08 00 00 00 00 00 02 00 00 00 00 00 00 00 03 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000060: 00 00 ff ff 0d 00 00 00 00 00 02 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000070: 00 00 04 00 00 00 ff ff 1c 00 00 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000080: 00 00 00 00 00 00 04 00 00 00 ff ff 24 00 00 00 ............$... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000090: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000000a0: 41 00 00 00 00 00 02 00 00 00 00 00 00 00 03 00 A............... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000000b0: 00 00 ff ff 45 00 00 00 00 00 02 00 00 00 00 00 ....E........... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000000c0: 00 00 04 00 00 00 ff ff 60 00 00 00 00 00 02 00 ........`....... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000000d0: 00 00 00 00 00 00 04 00 00 00 ff ff 66 00 00 00 ............f... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000000e0: 00 00 02 00 00 00 00 00 00 00 04 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000000f0: 67 00 00 00 00 00 02 00 00 00 00 00 00 00 03 00 g............... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000100: 00 00 ff ff 7b 00 00 00 00 00 02 00 00 00 00 00 ....{........... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000110: 00 00 03 00 00 00 ff ff 7c 00 00 00 00 00 02 00 ........|....... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000120: 00 00 00 00 00 00 03 00 00 00 ff ff 81 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000130: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000140: 8b 00 00 00 00 00 02 00 00 00 00 00 00 00 03 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000150: 00 00 ff ff 8c 00 00 00 00 00 02 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000160: 00 00 04 00 00 00 ff ff 91 00 00 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000170: 00 00 00 00 00 00 03 00 00 00 ff ff 95 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000180: 00 00 02 00 00 00 00 00 00 00 04 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000190: a1 00 00 00 00 00 02 00 00 00 00 00 00 00 04 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000001a0: 00 00 ff ff ad 00 00 00 00 00 02 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000001b0: 00 00 04 00 00 00 ff ff b3 00 00 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000001c0: 00 00 00 00 00 00 03 00 00 00 ff ff b4 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000001d0: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000001e0: c0 00 00 00 00 00 02 00 00 00 00 00 00 00 04 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000001f0: 00 00 ff ff c1 00 00 00 00 00 02 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000200: 00 00 04 00 00 00 ff ff c4 00 00 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000210: 00 00 00 00 00 00 03 00 00 00 ff ff ca 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000220: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000230: cf 00 00 00 00 00 02 00 00 00 00 00 00 00 03 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000240: 00 00 ff ff d2 00 00 00 00 00 02 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000250: 00 00 03 00 00 00 ff ff db 00 00 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000260: 00 00 00 00 00 00 03 00 00 00 ff ff dd 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000270: 00 00 02 00 00 00 00 00 00 00 04 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000280: e1 00 00 00 00 00 02 00 00 00 00 00 00 00 03 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000290: 00 00 ff ff e4 00 00 00 00 00 02 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000002a0: 00 00 04 00 00 00 ff ff e8 00 00 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000002b0: 00 00 00 00 00 00 04 00 00 00 ff ff ea 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000002c0: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000002d0: f3 00 00 00 00 00 02 00 00 00 00 00 00 00 04 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000002e0: 00 00 ff ff f6 00 00 00 00 00 02 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000002f0: 00 00 03 00 00 00 ff ff fd 00 00 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000300: 00 00 00 00 00 00 03 00 00 00 ff ff 08 01 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000310: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000320: 09 01 00 00 00 00 02 00 00 00 00 00 00 00 03 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000330: 00 00 ff ff 0b 01 00 00 00 00 02 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000340: 00 00 03 00 00 00 ff ff 0c 01 00 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000350: 00 00 00 00 00 00 04 00 00 00 ff ff 10 01 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000360: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000370: 11 01 00 00 00 00 02 00 00 00 00 00 00 00 04 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000380: 00 00 ff ff 12 01 00 00 00 00 02 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000390: 00 00 03 00 00 00 ff ff 13 01 00 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000003a0: 00 00 00 00 00 00 04 00 00 00 ff ff 1b 01 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000003b0: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000003c0: 22 01 00 00 00 00 02 00 00 00 00 00 00 00 03 00 "............... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000003d0: 00 00 ff ff 2f 01 00 00 00 00 02 00 00 00 00 00 ..../........... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000003e0: 00 00 03 00 00 00 ff ff 32 01 00 00 00 00 02 00 ........2....... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000003f0: 00 00 00 00 00 00 03 00 00 00 ff ff 3a 01 00 00 ............:... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000400: 00 00 02 00 00 00 00 00 00 00 04 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000410: 3d 01 00 00 00 00 02 00 00 00 00 00 00 00 03 00 =............... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000420: 00 00 ff ff 3f 01 00 00 00 00 02 00 00 00 00 00 ....?........... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000430: 00 00 04 00 00 00 ff ff 07 00 01 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000440: 00 00 00 00 00 00 03 00 00 00 ff ff 7a 00 01 00 ............z... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000450: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000460: 7b 00 01 00 00 00 02 00 00 00 00 00 00 00 03 00 {............... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000470: 00 00 ff ff 80 00 01 00 00 00 02 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000480: 00 00 03 00 00 00 ff ff 8a 00 01 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000490: 00 00 00 00 00 00 03 00 00 00 ff ff 8b 00 01 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000004a0: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000004b0: 90 00 01 00 00 00 02 00 00 00 00 00 00 00 03 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000004c0: 00 00 ff ff a0 00 01 00 00 00 02 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000004d0: 00 00 03 00 00 00 ff ff b3 00 01 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000004e0: 00 00 00 00 00 00 03 00 00 00 ff ff c3 00 01 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000004f0: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000500: d1 00 01 00 00 00 02 00 00 00 00 00 00 00 03 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000510: 00 00 ff ff e7 00 01 00 00 00 02 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000520: 00 00 03 00 00 00 ff ff e9 00 01 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000530: 00 00 00 00 00 00 03 00 00 00 ff ff f5 00 01 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000540: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000550: 07 01 01 00 00 00 02 00 00 00 00 00 00 00 03 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000560: 00 00 ff ff 0f 01 01 00 00 00 02 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000570: 00 00 03 00 00 00 ff ff 11 01 01 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000580: 00 00 00 00 00 00 03 00 00 00 ff ff 12 01 01 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000590: 00 00 02 00 00 00 00 00 00 00 04 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000005a0: 1a 01 01 00 00 00 02 00 00 00 00 00 00 00 03 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000005b0: 00 00 ff ff 21 01 01 00 00 00 02 00 00 00 00 00 ....!........... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000005c0: 00 00 03 00 00 00 ff ff 2e 01 01 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000005d0: 00 00 00 00 00 00 03 00 00 00 ff ff 06 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000005e0: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000005f0: 79 00 02 00 00 00 02 00 00 00 00 00 00 00 03 00 y............... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000600: 00 00 ff ff 7a 00 02 00 00 00 02 00 00 00 00 00 ....z........... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000610: 00 00 03 00 00 00 ff ff 7f 00 02 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000620: 00 00 00 00 00 00 03 00 00 00 ff ff 89 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000630: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000640: 8a 00 02 00 00 00 02 00 00 00 00 00 00 00 04 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000650: 00 00 ff ff 8f 00 02 00 00 00 02 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000660: 00 00 03 00 00 00 ff ff 9f 00 02 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000670: 00 00 00 00 00 00 04 00 00 00 ff ff c2 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000680: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000690: d0 00 02 00 00 00 02 00 00 00 00 00 00 00 03 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000006a0: 00 00 ff ff e6 00 02 00 00 00 02 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000006b0: 00 00 04 00 00 00 ff ff e8 00 02 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000006c0: 00 00 00 00 00 00 03 00 00 00 ff ff f4 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000006d0: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000006e0: 0e 01 02 00 00 00 02 00 00 00 00 00 00 00 03 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000006f0: 00 00 ff ff 10 01 02 00 00 00 02 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000700: 00 00 03 00 00 00 ff ff 11 01 02 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000710: 00 00 00 00 00 00 04 00 00 00 ff ff 19 01 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000720: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000730: 20 01 02 00 00 00 02 00 00 00 00 00 00 00 03 00 ............... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000740: 00 00 ff ff 2d 01 02 00 00 00 02 00 00 00 00 00 ....-........... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000750: 00 00 03 00 00 00 ff ff 01 00 03 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000760: 00 00 04 00 00 00 00 00 00 00 ff ff 00 00 04 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000770: 00 00 02 00 00 00 04 00 00 00 00 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000780: 07 00 04 00 00 00 02 00 00 00 04 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000790: 00 00 ff ff 06 00 05 00 00 00 02 00 00 00 04 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000007a0: 00 00 00 00 00 00 ff ff 02 00 06 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000007b0: 00 00 00 00 00 00 03 00 00 00 ff ff 05 00 06 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000007c0: 00 00 02 00 00 00 04 00 00 00 00 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000007d0: 04 00 07 00 00 00 02 00 00 00 04 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000007e0: 00 00 ff ff 03 00 08 00 00 00 02 00 00 00 04 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000007f0: 00 00 00 00 00 00 ff ff 02 00 09 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000800: 00 00 04 00 00 00 00 00 00 00 ff ff 01 00 0a 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000810: 00 00 02 00 00 00 04 00 00 00 00 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000820: 07 00 0a 00 00 00 02 00 00 00 04 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000830: 00 00 ff ff 00 00 0b 00 00 00 02 00 00 00 04 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000840: 00 00 00 00 00 00 ff ff 06 00 0b 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000850: 00 00 04 00 00 00 00 00 00 00 ff ff 07 00 0b 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000860: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000870: 05 00 0c 00 00 00 02 00 00 00 04 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000880: 00 00 ff ff 06 00 0c 00 00 00 02 00 00 00 03 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000890: 00 00 00 00 00 00 ff ff 04 00 0d 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000008a0: 00 00 04 00 00 00 00 00 00 00 ff ff 05 00 0d 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000008b0: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000008c0: 03 00 0e 00 00 00 02 00 00 00 04 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000008d0: 00 00 ff ff 04 00 0e 00 00 00 02 00 00 00 03 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000008e0: 00 00 00 00 00 00 ff ff 02 00 0f 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000008f0: 00 00 04 00 00 00 00 00 00 00 ff ff 03 00 0f 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000900: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000910: 01 00 10 00 00 00 02 00 00 00 04 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000920: 00 00 ff ff 02 00 10 00 00 00 02 00 00 00 03 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000930: 00 00 00 00 00 00 ff ff 00 00 11 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000940: 00 00 04 00 00 00 00 00 00 00 ff ff 01 00 11 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000950: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000960: 00 00 12 00 00 00 02 00 00 00 03 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000970: 00 00 ff ff 06 00 14 00 00 00 02 00 00 00 03 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000980: 00 00 00 00 00 00 ff ff 05 00 15 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000990: 00 00 03 00 00 00 00 00 00 00 ff ff 04 00 16 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000009a0: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000009b0: 03 00 17 00 00 00 02 00 00 00 03 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000009c0: 00 00 ff ff 02 00 18 00 00 00 02 00 00 00 03 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000009d0: 00 00 00 00 00 00 ff ff 01 00 19 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000009e0: 00 00 03 00 00 00 00 00 00 00 ff ff 00 00 1a 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000009f0: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a00: 07 00 1b 00 00 00 02 00 00 00 04 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a10: 00 00 ff ff 00 00 1c 00 00 00 02 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a20: 00 00 04 00 00 00 ff ff 05 00 1d 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a30: 00 00 04 00 00 00 00 00 00 00 ff ff 04 00 1e 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a40: 00 00 02 00 00 00 04 00 00 00 00 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a50: 03 00 1f 00 00 00 02 00 00 00 04 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a60: 00 00 ff ff 02 00 20 00 00 00 02 00 00 00 04 00 ...... ......... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a70: 00 00 00 00 00 00 ff ff 01 00 21 00 00 00 02 00 ..........!..... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a80: 00 00 04 00 00 00 00 00 00 00 ff ff 00 00 22 00 ..............". Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a90: 00 00 02 00 00 00 04 00 00 00 00 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000aa0: 07 00 25 00 00 00 02 00 00 00 04 00 00 00 00 00 ..%............. Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000ab0: 00 00 ff ff 06 00 26 00 00 00 02 00 00 00 04 00 ......&......... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000ac0: 00 00 00 00 00 00 ff ff 04 00 28 00 00 00 02 00 ..........(..... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000ad0: 00 00 04 00 00 00 00 00 00 00 ff ff 03 00 29 00 ..............). Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000ae0: 00 00 02 00 00 00 04 00 00 00 00 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000af0: 02 00 2a 00 00 00 02 00 00 00 04 00 00 00 00 00 ..*............. Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b00: 00 00 ff ff 01 00 2b 00 00 00 02 00 00 00 04 00 ......+......... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b10: 00 00 00 00 00 00 ff ff 00 00 2c 00 00 00 02 00 ..........,..... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b20: 00 00 04 00 00 00 00 00 00 00 ff ff 07 00 3a 00 ..............:. Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b30: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b40: 05 00 3c 00 00 00 02 00 00 00 00 00 00 00 03 00 ..<............. Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b50: 00 00 ff ff 06 00 3f 00 00 00 02 00 00 00 00 00 ......?......... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b60: 00 00 04 00 00 00 ff ff 04 00 5c 00 00 00 02 00 ..........\..... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b70: 00 00 00 00 00 00 04 00 00 00 ff ff 07 00 65 00 ..............e. Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b80: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b90: 06 00 66 00 00 00 02 00 00 00 03 00 00 00 00 00 ..f............. Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000ba0: 00 00 ff ff 05 00 67 00 00 00 02 00 00 00 03 00 ......g......... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000bb0: 00 00 00 00 00 00 ff ff 04 00 68 00 00 00 02 00 ..........h..... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000bc0: 00 00 03 00 00 00 00 00 00 00 ff ff 03 00 69 00 ..............i. Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000bd0: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000be0: 00 00 6c 00 00 00 02 00 00 00 03 00 00 00 00 00 ..l............. Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000bf0: 00 00 ff ff 07 00 6d 00 00 00 02 00 00 00 04 00 ......m......... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c00: 00 00 00 00 00 00 ff ff 06 00 6e 00 00 00 02 00 ..........n..... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c10: 00 00 04 00 00 00 00 00 00 00 ff ff 05 00 6f 00 ..............o. Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c20: 00 00 02 00 00 00 04 00 00 00 00 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c30: 04 00 70 00 00 00 02 00 00 00 04 00 00 00 00 00 ..p............. Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c40: 00 00 ff ff 03 00 71 00 00 00 02 00 00 00 04 00 ......q......... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c50: 00 00 00 00 00 00 ff ff 02 00 72 00 00 00 02 00 ..........r..... Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c60: 00 00 04 00 00 00 00 00 00 00 ff ff 01 00 73 00 ..............s. Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c70: 00 00 02 00 00 00 04 00 00 00 00 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c80: 07 00 74 00 00 00 02 00 00 00 00 00 00 00 03 00 ..t............. Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c90: 00 00 ff ff 07 00 7f 00 00 00 02 00 00 00 03 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000ca0: 00 00 00 00 00 00 ff ff 06 00 80 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000cb0: 00 00 03 00 00 00 00 00 00 00 ff ff 05 00 81 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000cc0: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000cd0: 04 00 82 00 00 00 02 00 00 00 03 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000ce0: 00 00 ff ff 03 00 83 00 00 00 02 00 00 00 03 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000cf0: 00 00 00 00 00 00 ff ff 02 00 84 00 00 00 02 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000d00: 00 00 03 00 00 00 00 00 00 00 ff ff 01 00 85 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000d10: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000d20: 05 00 86 00 00 00 02 00 00 00 00 00 00 00 03 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000d30: 00 00 ff ff 06 00 86 00 00 00 02 00 00 00 00 00 ................ Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000d40: 00 00 04 00 00 00 ...... Feb 20 14:15:11 fcmsnode0 kernel: libceph: osdc handle_map corrupt msg
Updated by Dan Mick about 12 years ago
- Subject changed from currupt v5 inc osdmap to corrupt v5 inc osdmap
Updated by Alex Elder about 12 years ago
OK, picking a few things out of the original corruption report.
The basic header stuff is the same as before, as expected:
version = 0x0005 = CEPH_OSDMAP_INC_VERSION
fsid = {
34, b3, 0e, 21, 99, f8, 43, ff,
be, f2, a3, c0, 04, 2b, 07, e1
}
epoch = 0x00000272 = 626
modified = {
tv_sec = 0x4f4245d2,
tv_nsec = 0x2bb56c30
} => 1329743314.733310000 = Mon Feb 20 07:08:34 2012
new_pool_max = 0xffffffff => not new
new_flags = 0xffffffff => not new
(map) len = 0
(crush map) len = 0
max = 0xffffffff = -1 => not new
Only the pg mappings got updated:
(number of pools) len = 0
(number of pool names) len = 0
(number of pool id's, old pool) len = 0
(number of newly-up osd indexes) len = 0
(number of osd indexes + state changes) len = 0
(number of osd indexes + weights) len = 0
(number of new pg mappings) len = 0x000000a6 = 166
The message indicates that the corruption was detected when the offset had reached
1898 decimal, or 0x76a.
Here's what the first ceph_pg_mapping looks like:
pg0.pgid = {
preferred = 0xffff = -1
ps = 0x0008 = 8
pool = 0x00000000 = 0
}
pg0 pglen = 0x00000002 = 2
pg0.osds0 = 0x00000000 = 0
pg0.osds0 = 0x00000003 = 3
...
Offset 0x76a is just at the end of another one, meaning we must
have failed (as reported before) in the call to __insert_pg_mapping()
for the one just consumed. Here's what that one looks like:
pg91.pgid = {
preferred = 0xffff = -1
ps = 0x0001 = 1
pool = 0x00000003 = 3
}
pg91 pglen = 0x00000002 = 2
pg91.osds0 = 0x00000004 = 4
pg91.osds0 = 0x00000000 = 0
Evidently there is another entry in the pg_temp red-black tree in the
passed-in ceph_osdmap having a pgid that matches the one above.
None of the ceph_pg_mapping structures earlier in this map update
have that same pgid, it must have existed before the incremental
update started.
At this point I can't tell much more about this, and I'm not sure how
best to proceed.
Updated by Sage Weil about 12 years ago
ok.. yeah, it looks like the monitor may have published a bad inc update or something? unclear. i'll check with them and see if they have this map history..
Updated by Sage Weil about 12 years ago
- Status changed from New to Can't reproduce
- Assignee deleted (
Alex Elder)
logs don't go far enough back. :(
moral of the story: next time grab the full mon data dir immediately in case it is useful later.
i don't think there's anything else we can do here, aside from blindly reading code and looking for possible bugs.