Bug #2091: corrupt v5 inc osdmap - Ceph - Ceph

Actions

#1

Updated by Sage Weil about 12 years ago

Source changed from Development to Community (user)

reencoded to old format (using latest ceph-dencoder) gives us


$ hexdump  -C 880.old 
00000000  05 00 34 b3 0e 21 99 f8  43 ff be f2 a3 c0 04 2b  |..4..!..C......+|
00000010  07 e1 70 03 00 00 10 79  42 4f 30 16 47 1d ff ff  |..p....yBO0.G...|
00000020  ff ff ff ff ff ff 00 00  00 00 00 00 00 00 ff ff  |................|
00000030  ff ff 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000040  00 00 00 00 00 00 00 00  00 00 52 00 00 00 ff ff  |..........R.....|
00000050  06 00 07 00 00 00 03 00  00 00 00 00 00 00 02 00  |................|
00000060  00 00 03 00 00 00 ff ff  03 00 0a 00 00 00 03 00  |................|
00000070  00 00 00 00 00 00 02 00  00 00 03 00 00 00 ff ff  |................|
00000080  02 00 0b 00 00 00 03 00  00 00 00 00 00 00 02 00  |................|
00000090  00 00 03 00 00 00 ff ff  07 00 0f 00 00 00 00 00  |................|
000000a0  00 00 ff ff 06 00 10 00  00 00 00 00 00 00 ff ff  |................|
000000b0  05 00 11 00 00 00 00 00  00 00 ff ff 04 00 12 00  |................|
000000c0  00 00 00 00 00 00 ff ff  02 00 14 00 00 00 00 00  |................|
000000d0  00 00 ff ff 07 00 14 00  00 00 00 00 00 00 ff ff  |................|
000000e0  01 00 15 00 00 00 00 00  00 00 ff ff 06 00 15 00  |................|
000000f0  00 00 00 00 00 00 ff ff  00 00 16 00 00 00 00 00  |................|
00000100  00 00 ff ff 05 00 16 00  00 00 00 00 00 00 ff ff  |................|
00000110  04 00 17 00 00 00 00 00  00 00 ff ff 03 00 18 00  |................|
00000120  00 00 00 00 00 00 ff ff  02 00 19 00 00 00 00 00  |................|
00000130  00 00 ff ff 01 00 1a 00  00 00 00 00 00 00 ff ff  |................|
00000140  00 00 1b 00 00 00 00 00  00 00 ff ff 07 00 23 00  |..............#.|
00000150  00 00 00 00 00 00 ff ff  06 00 24 00 00 00 00 00  |..........$.....|
00000160  00 00 ff ff 05 00 25 00  00 00 00 00 00 00 ff ff  |......%.........|
00000170  04 00 26 00 00 00 00 00  00 00 ff ff 02 00 28 00  |..&...........(.|
00000180  00 00 00 00 00 00 ff ff  01 00 29 00 00 00 00 00  |..........).....|
00000190  00 00 ff ff 00 00 2a 00  00 00 00 00 00 00 ff ff  |......*.........|
000001a0  07 00 33 00 00 00 00 00  00 00 ff ff 06 00 34 00  |..3...........4.|
000001b0  00 00 00 00 00 00 ff ff  05 00 35 00 00 00 00 00  |..........5.....|
000001c0  00 00 ff ff 04 00 36 00  00 00 00 00 00 00 ff ff  |......6.........|
000001d0  03 00 37 00 00 00 00 00  00 00 ff ff 02 00 38 00  |..7...........8.|
000001e0  00 00 00 00 00 00 ff ff  01 00 39 00 00 00 00 00  |..........9.....|
000001f0  00 00 ff ff 06 00 3b 00  00 00 03 00 00 00 00 00  |......;.........|
00000200  00 00 02 00 00 00 03 00  00 00 ff ff 04 00 3d 00  |..............=.|
00000210  00 00 03 00 00 00 00 00  00 00 02 00 00 00 03 00  |................|
00000220  00 00 ff ff 03 00 3e 00  00 00 03 00 00 00 00 00  |......>.........|
00000230  00 00 02 00 00 00 03 00  00 00 ff ff 02 00 3f 00  |..............?.|
00000240  00 00 03 00 00 00 00 00  00 00 02 00 00 00 03 00  |................|
00000250  00 00 ff ff 01 00 40 00  00 00 03 00 00 00 00 00  |......@.........|
00000260  00 00 02 00 00 00 03 00  00 00 ff ff 00 00 41 00  |..............A.|
00000270  00 00 03 00 00 00 00 00  00 00 02 00 00 00 03 00  |................|
00000280  00 00 ff ff 03 00 42 00  00 00 03 00 00 00 03 00  |......B.........|
00000290  00 00 00 00 00 00 04 00  00 00 ff ff 07 00 42 00  |..............B.|
000002a0  00 00 00 00 00 00 ff ff  06 00 43 00 00 00 00 00  |..........C.....|
000002b0  00 00 ff ff 01 00 44 00  00 00 03 00 00 00 03 00  |......D.........|
000002c0  00 00 00 00 00 00 04 00  00 00 ff ff 05 00 44 00  |..............D.|
000002d0  00 00 00 00 00 00 ff ff  03 00 46 00 00 00 00 00  |..........F.....|
000002e0  00 00 ff ff 02 00 47 00  00 00 00 00 00 00 ff ff  |......G.........|
000002f0  01 00 48 00 00 00 00 00  00 00 ff ff 00 00 49 00  |..H...........I.|
00000300  00 00 00 00 00 00 ff ff  06 00 59 00 00 00 00 00  |..........Y.....|
00000310  00 00 ff ff 05 00 5a 00  00 00 00 00 00 00 ff ff  |......Z.........|
00000320  04 00 5b 00 00 00 00 00  00 00 ff ff 02 00 5d 00  |..[...........].|
00000330  00 00 00 00 00 00 ff ff  01 00 5e 00 00 00 00 00  |..........^.....|
00000340  00 00 ff ff 00 00 5f 00  00 00 00 00 00 00 ff ff  |......_.........|
00000350  07 00 60 00 00 00 03 00  00 00 00 00 00 00 02 00  |..`.............|
00000360  00 00 03 00 00 00 ff ff  06 00 61 00 00 00 03 00  |..........a.....|
00000370  00 00 00 00 00 00 02 00  00 00 03 00 00 00 ff ff  |................|
00000380  05 00 62 00 00 00 03 00  00 00 00 00 00 00 02 00  |..b.............|
00000390  00 00 03 00 00 00 ff ff  04 00 63 00 00 00 03 00  |..........c.....|
000003a0  00 00 00 00 00 00 02 00  00 00 03 00 00 00 ff ff  |................|
000003b0  07 00 63 00 00 00 00 00  00 00 ff ff 03 00 64 00  |..c...........d.|
000003c0  00 00 03 00 00 00 00 00  00 00 02 00 00 00 03 00  |................|
000003d0  00 00 ff ff 06 00 64 00  00 00 00 00 00 00 ff ff  |......d.........|
000003e0  07 00 64 00 00 00 00 00  00 00 ff ff 02 00 65 00  |..d...........e.|
000003f0  00 00 03 00 00 00 00 00  00 00 02 00 00 00 03 00  |................|
00000400  00 00 ff ff 05 00 65 00  00 00 00 00 00 00 ff ff  |......e.........|
00000410  06 00 65 00 00 00 00 00  00 00 ff ff 01 00 66 00  |..e...........f.|
00000420  00 00 03 00 00 00 00 00  00 00 02 00 00 00 03 00  |................|
00000430  00 00 ff ff 04 00 66 00  00 00 00 00 00 00 ff ff  |......f.........|
00000440  05 00 66 00 00 00 00 00  00 00 ff ff 00 00 67 00  |..f...........g.|
00000450  00 00 03 00 00 00 00 00  00 00 02 00 00 00 03 00  |................|
00000460  00 00 ff ff 03 00 67 00  00 00 00 00 00 00 ff ff  |......g.........|
00000470  04 00 67 00 00 00 00 00  00 00 ff ff 02 00 68 00  |..g...........h.|
00000480  00 00 00 00 00 00 ff ff  03 00 68 00 00 00 00 00  |..........h.....|
00000490  00 00 ff ff 01 00 69 00  00 00 00 00 00 00 ff ff  |......i.........|
000004a0  02 00 69 00 00 00 00 00  00 00 ff ff 07 00 73 00  |..i...........s.|
000004b0  00 00 00 00 00 00 ff ff  06 00 74 00 00 00 00 00  |..........t.....|
000004c0  00 00 ff ff 05 00 75 00  00 00 00 00 00 00 ff ff  |......u.........|
000004d0  04 00 76 00 00 00 00 00  00 00 ff ff 03 00 77 00  |..v...........w.|
000004e0  00 00 00 00 00 00 ff ff  02 00 78 00 00 00 00 00  |..........x.....|
000004f0  00 00 ff ff 01 00 79 00  00 00 00 00 00 00 ff ff  |......y.........|
00000500  00 00 7a 00 00 00 00 00  00 00                    |..z.......|
0000050a

Actions

Copy link

#2

Updated by Sage Weil about 12 years ago

Assignee set to Alex Elder

Actions

Copy link

#3

Updated by Alex Elder about 12 years ago

I'm starting to look at this in detail but haven't concluded what went wrong yet.

Does it matter whether it was the monitor or the OSD that hit the problem?
Is it possible to tell?

Actions

Copy link

#4

Updated by Alex Elder about 12 years ago

I've manually decoded the entire ceph_osdmap dumped in the log and everything
therein looks fine. (This was overkill, but I automated the process a bit.)

The error message indicates the buffer pointer was at offset 102 = 0x66 into the
message buffer. That corresponds to the point at which the last (the third)
osd id has been decoded for the second ceph_pg_mapping in the message. I.e.,
it is here, in osdmap_apply_incremental(), around line 904 in the current code:

pg->pgid = pgid;
                        pg->len = pglen;
                        for (j = 0; j < pglen; j++)
                                pg->osds[j] = ceph_decode_32(p);
       ---> here --->
                        err = __insert_pg_mapping(pg, &map->pg_temp);
                        if (err) {
                                kfree(pg);
                                goto bad;
                        }

What this means is that the call to __insert_pg_mapping() must have
returned an error. The only error that function returns is -EEXIST,
meaning that the pgid we're trying to insert at this point was found
to already be present in the red-black tree.

But that doesn't appear possible, based on the content of the message
I've decoded.

I'm going to have to take another look at what I've done tomorrow to
validate my findings. I think it will also be useful to look at the
precise version of the code that hit this problem. According to Sage,
there is a tar file full of logs (at least) here: fatty:~sage/tmp/oliver

Actions

Copy link

#5

Updated by Sage Weil about 12 years ago

Oh.. that means the pg_temp mapping was inserted by a previous inc map, probably. we need to find the first instance of weirdness.

Actions

Copy link

#6

Updated by Sage Weil about 12 years ago

the first badness in the log is below. once it missed one incremental, things probably got out of sync and the pg_temp updates all seemed incorrect (entries already there, etc.).

this is the one to look at, i think. unfortunately it's way bigger.. :(

Feb 20 14:15:11 fcmsnode0 kernel: libceph: corrupt inc osdmap epoch 626 off 1898 (ffffc900265de82e of ffffc900265de0c4-ffffc900265dee0a)
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000000: 05 00 34 b3 0e 21 99 f8 43 ff be f2 a3 c0 04 2b  ..4..!..C......+
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000010: 07 e1 72 02 00 00 d2 45 42 4f 30 6c b5 2b ff ff  ..r....EBO0l.+..
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000020: ff ff ff ff ff ff 00 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000030: ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000040: 00 00 00 00 00 00 00 00 00 00 a6 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000050: 08 00 00 00 00 00 02 00 00 00 00 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000060: 00 00 ff ff 0d 00 00 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000070: 00 00 04 00 00 00 ff ff 1c 00 00 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000080: 00 00 00 00 00 00 04 00 00 00 ff ff 24 00 00 00  ............$...
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000090: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000000a0: 41 00 00 00 00 00 02 00 00 00 00 00 00 00 03 00  A...............
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000000b0: 00 00 ff ff 45 00 00 00 00 00 02 00 00 00 00 00  ....E...........
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000000c0: 00 00 04 00 00 00 ff ff 60 00 00 00 00 00 02 00  ........`.......
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000000d0: 00 00 00 00 00 00 04 00 00 00 ff ff 66 00 00 00  ............f...
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000000e0: 00 00 02 00 00 00 00 00 00 00 04 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000000f0: 67 00 00 00 00 00 02 00 00 00 00 00 00 00 03 00  g...............
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000100: 00 00 ff ff 7b 00 00 00 00 00 02 00 00 00 00 00  ....{...........
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000110: 00 00 03 00 00 00 ff ff 7c 00 00 00 00 00 02 00  ........|.......
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000120: 00 00 00 00 00 00 03 00 00 00 ff ff 81 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000130: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000140: 8b 00 00 00 00 00 02 00 00 00 00 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000150: 00 00 ff ff 8c 00 00 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000160: 00 00 04 00 00 00 ff ff 91 00 00 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000170: 00 00 00 00 00 00 03 00 00 00 ff ff 95 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000180: 00 00 02 00 00 00 00 00 00 00 04 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000190: a1 00 00 00 00 00 02 00 00 00 00 00 00 00 04 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000001a0: 00 00 ff ff ad 00 00 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000001b0: 00 00 04 00 00 00 ff ff b3 00 00 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000001c0: 00 00 00 00 00 00 03 00 00 00 ff ff b4 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000001d0: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000001e0: c0 00 00 00 00 00 02 00 00 00 00 00 00 00 04 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000001f0: 00 00 ff ff c1 00 00 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000200: 00 00 04 00 00 00 ff ff c4 00 00 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000210: 00 00 00 00 00 00 03 00 00 00 ff ff ca 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000220: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000230: cf 00 00 00 00 00 02 00 00 00 00 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000240: 00 00 ff ff d2 00 00 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000250: 00 00 03 00 00 00 ff ff db 00 00 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000260: 00 00 00 00 00 00 03 00 00 00 ff ff dd 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000270: 00 00 02 00 00 00 00 00 00 00 04 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000280: e1 00 00 00 00 00 02 00 00 00 00 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000290: 00 00 ff ff e4 00 00 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000002a0: 00 00 04 00 00 00 ff ff e8 00 00 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000002b0: 00 00 00 00 00 00 04 00 00 00 ff ff ea 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000002c0: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000002d0: f3 00 00 00 00 00 02 00 00 00 00 00 00 00 04 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000002e0: 00 00 ff ff f6 00 00 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000002f0: 00 00 03 00 00 00 ff ff fd 00 00 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000300: 00 00 00 00 00 00 03 00 00 00 ff ff 08 01 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000310: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000320: 09 01 00 00 00 00 02 00 00 00 00 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000330: 00 00 ff ff 0b 01 00 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000340: 00 00 03 00 00 00 ff ff 0c 01 00 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000350: 00 00 00 00 00 00 04 00 00 00 ff ff 10 01 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000360: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000370: 11 01 00 00 00 00 02 00 00 00 00 00 00 00 04 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000380: 00 00 ff ff 12 01 00 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000390: 00 00 03 00 00 00 ff ff 13 01 00 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000003a0: 00 00 00 00 00 00 04 00 00 00 ff ff 1b 01 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000003b0: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000003c0: 22 01 00 00 00 00 02 00 00 00 00 00 00 00 03 00  "...............
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000003d0: 00 00 ff ff 2f 01 00 00 00 00 02 00 00 00 00 00  ..../...........
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000003e0: 00 00 03 00 00 00 ff ff 32 01 00 00 00 00 02 00  ........2.......
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000003f0: 00 00 00 00 00 00 03 00 00 00 ff ff 3a 01 00 00  ............:...
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000400: 00 00 02 00 00 00 00 00 00 00 04 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000410: 3d 01 00 00 00 00 02 00 00 00 00 00 00 00 03 00  =...............
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000420: 00 00 ff ff 3f 01 00 00 00 00 02 00 00 00 00 00  ....?...........
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000430: 00 00 04 00 00 00 ff ff 07 00 01 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000440: 00 00 00 00 00 00 03 00 00 00 ff ff 7a 00 01 00  ............z...
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000450: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................

Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000460: 7b 00 01 00 00 00 02 00 00 00 00 00 00 00 03 00  {...............
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000470: 00 00 ff ff 80 00 01 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000480: 00 00 03 00 00 00 ff ff 8a 00 01 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000490: 00 00 00 00 00 00 03 00 00 00 ff ff 8b 00 01 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000004a0: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000004b0: 90 00 01 00 00 00 02 00 00 00 00 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000004c0: 00 00 ff ff a0 00 01 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000004d0: 00 00 03 00 00 00 ff ff b3 00 01 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000004e0: 00 00 00 00 00 00 03 00 00 00 ff ff c3 00 01 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000004f0: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000500: d1 00 01 00 00 00 02 00 00 00 00 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000510: 00 00 ff ff e7 00 01 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000520: 00 00 03 00 00 00 ff ff e9 00 01 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000530: 00 00 00 00 00 00 03 00 00 00 ff ff f5 00 01 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000540: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000550: 07 01 01 00 00 00 02 00 00 00 00 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000560: 00 00 ff ff 0f 01 01 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000570: 00 00 03 00 00 00 ff ff 11 01 01 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000580: 00 00 00 00 00 00 03 00 00 00 ff ff 12 01 01 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000590: 00 00 02 00 00 00 00 00 00 00 04 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000005a0: 1a 01 01 00 00 00 02 00 00 00 00 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000005b0: 00 00 ff ff 21 01 01 00 00 00 02 00 00 00 00 00  ....!...........
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000005c0: 00 00 03 00 00 00 ff ff 2e 01 01 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000005d0: 00 00 00 00 00 00 03 00 00 00 ff ff 06 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000005e0: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000005f0: 79 00 02 00 00 00 02 00 00 00 00 00 00 00 03 00  y...............
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000600: 00 00 ff ff 7a 00 02 00 00 00 02 00 00 00 00 00  ....z...........
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000610: 00 00 03 00 00 00 ff ff 7f 00 02 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000620: 00 00 00 00 00 00 03 00 00 00 ff ff 89 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000630: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000640: 8a 00 02 00 00 00 02 00 00 00 00 00 00 00 04 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000650: 00 00 ff ff 8f 00 02 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000660: 00 00 03 00 00 00 ff ff 9f 00 02 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000670: 00 00 00 00 00 00 04 00 00 00 ff ff c2 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000680: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000690: d0 00 02 00 00 00 02 00 00 00 00 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000006a0: 00 00 ff ff e6 00 02 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000006b0: 00 00 04 00 00 00 ff ff e8 00 02 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000006c0: 00 00 00 00 00 00 03 00 00 00 ff ff f4 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000006d0: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000006e0: 0e 01 02 00 00 00 02 00 00 00 00 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000006f0: 00 00 ff ff 10 01 02 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000700: 00 00 03 00 00 00 ff ff 11 01 02 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000710: 00 00 00 00 00 00 04 00 00 00 ff ff 19 01 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000720: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000730: 20 01 02 00 00 00 02 00 00 00 00 00 00 00 03 00   ...............
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000740: 00 00 ff ff 2d 01 02 00 00 00 02 00 00 00 00 00  ....-...........
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000750: 00 00 03 00 00 00 ff ff 01 00 03 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000760: 00 00 04 00 00 00 00 00 00 00 ff ff 00 00 04 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000770: 00 00 02 00 00 00 04 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000780: 07 00 04 00 00 00 02 00 00 00 04 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000790: 00 00 ff ff 06 00 05 00 00 00 02 00 00 00 04 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000007a0: 00 00 00 00 00 00 ff ff 02 00 06 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000007b0: 00 00 00 00 00 00 03 00 00 00 ff ff 05 00 06 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000007c0: 00 00 02 00 00 00 04 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000007d0: 04 00 07 00 00 00 02 00 00 00 04 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000007e0: 00 00 ff ff 03 00 08 00 00 00 02 00 00 00 04 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000007f0: 00 00 00 00 00 00 ff ff 02 00 09 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000800: 00 00 04 00 00 00 00 00 00 00 ff ff 01 00 0a 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000810: 00 00 02 00 00 00 04 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000820: 07 00 0a 00 00 00 02 00 00 00 04 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000830: 00 00 ff ff 00 00 0b 00 00 00 02 00 00 00 04 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000840: 00 00 00 00 00 00 ff ff 06 00 0b 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000850: 00 00 04 00 00 00 00 00 00 00 ff ff 07 00 0b 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000860: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000870: 05 00 0c 00 00 00 02 00 00 00 04 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000880: 00 00 ff ff 06 00 0c 00 00 00 02 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000890: 00 00 00 00 00 00 ff ff 04 00 0d 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000008a0: 00 00 04 00 00 00 00 00 00 00 ff ff 05 00 0d 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000008b0: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000008c0: 03 00 0e 00 00 00 02 00 00 00 04 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000008d0: 00 00 ff ff 04 00 0e 00 00 00 02 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000008e0: 00 00 00 00 00 00 ff ff 02 00 0f 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000008f0: 00 00 04 00 00 00 00 00 00 00 ff ff 03 00 0f 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000900: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff  ................

Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000910: 01 00 10 00 00 00 02 00 00 00 04 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000920: 00 00 ff ff 02 00 10 00 00 00 02 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000930: 00 00 00 00 00 00 ff ff 00 00 11 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000940: 00 00 04 00 00 00 00 00 00 00 ff ff 01 00 11 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000950: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000960: 00 00 12 00 00 00 02 00 00 00 03 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000970: 00 00 ff ff 06 00 14 00 00 00 02 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000980: 00 00 00 00 00 00 ff ff 05 00 15 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000990: 00 00 03 00 00 00 00 00 00 00 ff ff 04 00 16 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000009a0: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000009b0: 03 00 17 00 00 00 02 00 00 00 03 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000009c0: 00 00 ff ff 02 00 18 00 00 00 02 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000009d0: 00 00 00 00 00 00 ff ff 01 00 19 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000009e0: 00 00 03 00 00 00 00 00 00 00 ff ff 00 00 1a 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 000009f0: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a00: 07 00 1b 00 00 00 02 00 00 00 04 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a10: 00 00 ff ff 00 00 1c 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a20: 00 00 04 00 00 00 ff ff 05 00 1d 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a30: 00 00 04 00 00 00 00 00 00 00 ff ff 04 00 1e 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a40: 00 00 02 00 00 00 04 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a50: 03 00 1f 00 00 00 02 00 00 00 04 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a60: 00 00 ff ff 02 00 20 00 00 00 02 00 00 00 04 00  ...... .........
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a70: 00 00 00 00 00 00 ff ff 01 00 21 00 00 00 02 00  ..........!.....
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a80: 00 00 04 00 00 00 00 00 00 00 ff ff 00 00 22 00  ..............".
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000a90: 00 00 02 00 00 00 04 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000aa0: 07 00 25 00 00 00 02 00 00 00 04 00 00 00 00 00  ..%.............
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000ab0: 00 00 ff ff 06 00 26 00 00 00 02 00 00 00 04 00  ......&.........
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000ac0: 00 00 00 00 00 00 ff ff 04 00 28 00 00 00 02 00  ..........(.....
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000ad0: 00 00 04 00 00 00 00 00 00 00 ff ff 03 00 29 00  ..............).
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000ae0: 00 00 02 00 00 00 04 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000af0: 02 00 2a 00 00 00 02 00 00 00 04 00 00 00 00 00  ..*.............
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b00: 00 00 ff ff 01 00 2b 00 00 00 02 00 00 00 04 00  ......+.........
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b10: 00 00 00 00 00 00 ff ff 00 00 2c 00 00 00 02 00  ..........,.....
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b20: 00 00 04 00 00 00 00 00 00 00 ff ff 07 00 3a 00  ..............:.
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b30: 00 00 02 00 00 00 00 00 00 00 03 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b40: 05 00 3c 00 00 00 02 00 00 00 00 00 00 00 03 00  ..<.............
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b50: 00 00 ff ff 06 00 3f 00 00 00 02 00 00 00 00 00  ......?.........
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b60: 00 00 04 00 00 00 ff ff 04 00 5c 00 00 00 02 00  ..........\.....
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b70: 00 00 00 00 00 00 04 00 00 00 ff ff 07 00 65 00  ..............e.
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b80: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000b90: 06 00 66 00 00 00 02 00 00 00 03 00 00 00 00 00  ..f.............
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000ba0: 00 00 ff ff 05 00 67 00 00 00 02 00 00 00 03 00  ......g.........
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000bb0: 00 00 00 00 00 00 ff ff 04 00 68 00 00 00 02 00  ..........h.....
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000bc0: 00 00 03 00 00 00 00 00 00 00 ff ff 03 00 69 00  ..............i.
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000bd0: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000be0: 00 00 6c 00 00 00 02 00 00 00 03 00 00 00 00 00  ..l.............
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000bf0: 00 00 ff ff 07 00 6d 00 00 00 02 00 00 00 04 00  ......m.........
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c00: 00 00 00 00 00 00 ff ff 06 00 6e 00 00 00 02 00  ..........n.....
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c10: 00 00 04 00 00 00 00 00 00 00 ff ff 05 00 6f 00  ..............o.
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c20: 00 00 02 00 00 00 04 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c30: 04 00 70 00 00 00 02 00 00 00 04 00 00 00 00 00  ..p.............
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c40: 00 00 ff ff 03 00 71 00 00 00 02 00 00 00 04 00  ......q.........
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c50: 00 00 00 00 00 00 ff ff 02 00 72 00 00 00 02 00  ..........r.....
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c60: 00 00 04 00 00 00 00 00 00 00 ff ff 01 00 73 00  ..............s.
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c70: 00 00 02 00 00 00 04 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c80: 07 00 74 00 00 00 02 00 00 00 00 00 00 00 03 00  ..t.............
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000c90: 00 00 ff ff 07 00 7f 00 00 00 02 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000ca0: 00 00 00 00 00 00 ff ff 06 00 80 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000cb0: 00 00 03 00 00 00 00 00 00 00 ff ff 05 00 81 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000cc0: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000cd0: 04 00 82 00 00 00 02 00 00 00 03 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000ce0: 00 00 ff ff 03 00 83 00 00 00 02 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000cf0: 00 00 00 00 00 00 ff ff 02 00 84 00 00 00 02 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000d00: 00 00 03 00 00 00 00 00 00 00 ff ff 01 00 85 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000d10: 00 00 02 00 00 00 03 00 00 00 00 00 00 00 ff ff  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000d20: 05 00 86 00 00 00 02 00 00 00 00 00 00 00 03 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000d30: 00 00 ff ff 06 00 86 00 00 00 02 00 00 00 00 00  ................
Feb 20 14:15:11 fcmsnode0 kernel: osdmap: 00000d40: 00 00 04 00 00 00                                ......
Feb 20 14:15:11 fcmsnode0 kernel: libceph: osdc handle_map corrupt msg

Actions

Copy link

#7

Updated by Dan Mick about 12 years ago

Subject changed from currupt v5 inc osdmap to corrupt v5 inc osdmap

Actions

Copy link

#8

Updated by Alex Elder about 12 years ago

OK, picking a few things out of the original corruption report.

The basic header stuff is the same as before, as expected:

version = 0x0005 = CEPH_OSDMAP_INC_VERSION
fsid = {
34, b3, 0e, 21, 99, f8, 43, ff,
be, f2, a3, c0, 04, 2b, 07, e1
}
epoch = 0x00000272 = 626
modified = {
tv_sec = 0x4f4245d2,
tv_nsec = 0x2bb56c30
} => 1329743314.733310000 = Mon Feb 20 07:08:34 2012
new_pool_max = 0xffffffff => not new
new_flags = 0xffffffff => not new
(map) len = 0
(crush map) len = 0
max = 0xffffffff = -1 => not new

Only the pg mappings got updated:

(number of pools) len = 0
(number of pool names) len = 0
(number of pool id's, old pool) len = 0
(number of newly-up osd indexes) len = 0
(number of osd indexes + state changes) len = 0
(number of osd indexes + weights) len = 0
(number of new pg mappings) len = 0x000000a6 = 166

The message indicates that the corruption was detected when the offset had reached
1898 decimal, or 0x76a.

Here's what the first ceph_pg_mapping looks like:
pg⁰.pgid = {
preferred = 0xffff = -1
ps = 0x0008 = 8
pool = 0x00000000 = 0
}
pg⁰ pglen = 0x00000002 = 2
pg⁰.osds⁰ = 0x00000000 = 0
pg⁰.osds⁰ = 0x00000003 = 3
...

Offset 0x76a is just at the end of another one, meaning we must
have failed (as reported before) in the call to __insert_pg_mapping()
for the one just consumed. Here's what that one looks like:

pg⁹¹.pgid = {
preferred = 0xffff = -1
ps = 0x0001 = 1
pool = 0x00000003 = 3
}
pg⁹¹ pglen = 0x00000002 = 2
pg⁹¹.osds⁰ = 0x00000004 = 4
pg⁹¹.osds⁰ = 0x00000000 = 0

Evidently there is another entry in the pg_temp red-black tree in the
passed-in ceph_osdmap having a pgid that matches the one above.

None of the ceph_pg_mapping structures earlier in this map update
have that same pgid, it must have existed before the incremental
update started.

At this point I can't tell much more about this, and I'm not sure how
best to proceed.

Actions

Copy link

#9

Updated by Sage Weil about 12 years ago

ok.. yeah, it looks like the monitor may have published a bad inc update or something? unclear. i'll check with them and see if they have this map history..

Actions

Copy link

#10

Updated by Sage Weil about 12 years ago

Status changed from New to Can't reproduce
Assignee deleted (~~Alex Elder~~)

logs don't go far enough back. :(

moral of the story: next time grab the full mon data dir immediately in case it is useful later.

i don't think there's anything else we can do here, aside from blindly reading code and looking for possible bugs.

Project

General

Profile

Ceph

Custom queries

Bug #2091

corrupt v5 inc osdmap

Updated by Sage Weil about 12 years ago

Updated by Sage Weil about 12 years ago

Updated by Alex Elder about 12 years ago

Updated by Alex Elder about 12 years ago

Updated by Sage Weil about 12 years ago

Updated by Sage Weil about 12 years ago

Updated by Dan Mick about 12 years ago

Updated by Alex Elder about 12 years ago

Updated by Sage Weil about 12 years ago

Updated by Sage Weil about 12 years ago