Bug #19119
Updated by Ilya Dryomov about 7 years ago
I have a bunch of misdirected requests from a recent kernel client to a hammer cluster, triggered by osd rm:
<pre>
2017-02-27 15:37:56.976845 osd.190 10.115.1.133:6808/3914 97 : cluster [WRN] client.9450549 10.115.1.35:0/1493770383 misdirected client.9450549.1:1379645861 pg 2.ec640804 to osd.190 in e241865, client e241865 pg 2.804 features 288863570635346
</pre>
e241864 -> e241865 incremental:
<pre>
{
"epoch": 241865,
"fsid": "9e3e9015-f626-4a44-83f7-0a939ef7ec02",
"modified": "2017-02-27 11:07:56.497658",
"new_pool_max": -1,
"new_flags": -1,
"new_max_osd": -1,
"new_pools": [],
"new_pool_names": [],
"old_pools": [],
"new_up_osds": [],
"new_weight": [],
"osd_state_xor": [
{
"osd": 204,
"state_xor": [
"autoout",
"exists"
]
},
[],
[],
[],
[],
[],
[],
[],
[],
[
{
"osd": 204,
"uuid": "00000000-0000-0000-0000-000000000000"
}
],
{},
[]
]
</pre>
On master:
<pre>
$ bin/osdmaptool --test-map-pg 2.ec640804 /tmp/map-241864.bin
bin/osdmaptool: osdmap file '/tmp/map-241864.bin'
parsed '2.ec640804' -> 2.ec640804
2.ec640804 raw ([197,201,1], p197) up ([197,201,1], p197) acting ([197,201,1], p197)
$ bin/osdmaptool --test-map-pg 2.ec640804 /tmp/map-241865.bin
bin/osdmaptool: osdmap file '/tmp/map-241865.bin'
parsed '2.ec640804' -> 2.ec640804
2.ec640804 raw ([197,201,1], p197) up ([197,201,1], p197) acting ([197,201,1], p197)
</pre>
but (with osdmaptool patched to accept and apply incrementals):
<pre>
$ bin/osdmaptool --test-map-pg 2.ec640804 /tmp/map-241864.bin /tmp/inc-241865.bin
bin/osdmaptool: osdmap file '/tmp/map-241864.bin'
bin/osdmaptool: incremental file '/tmp/inc-241865.bin'
parsed '2.ec640804' -> 2.ec640804
2.ec640804 raw ([190,1], p190) up ([190,1], p190) acting ([190,1], p190)
</pre>
which is where the misdirected request was sent.
On hammer:
<pre>
$ ./osdmaptool --test-map-pg 2.ec640804 /tmp/map-241864.bin
./osdmaptool: osdmap file '/tmp/map-241864.bin'
parsed '2.ec640804' -> 2.ec640804
2.ec640804 raw ([197,201,1], p197) up ([197,201,1], p197) acting ([197,201,1], p197)
$ ./osdmaptool --test-map-pg 2.ec640804 /tmp/map-241865.bin
./osdmaptool: osdmap file '/tmp/map-241865.bin'
parsed '2.ec640804' -> 2.ec640804
2.ec640804 raw ([197,201,1], p197) up ([197,201,1], p197) acting ([197,201,1], p197)
</pre>
and (same osdmaptool patch): but
<pre>
$ ./osdmaptool --test-map-pg 2.ec640804 /tmp/map-241864.bin /tmp/inc-241865.bin
./osdmaptool: osdmap file '/tmp/map-241864.bin'
./osdmaptool: incremental file '/tmp/inc-241865.bin'
parsed '2.ec640804' -> 2.ec640804
2.ec640804 raw ([197,201,1], p197) up ([197,201,1], p197) acting ([197,201,1], p197)
</pre>
<pre>
2017-02-27 15:37:56.976845 osd.190 10.115.1.133:6808/3914 97 : cluster [WRN] client.9450549 10.115.1.35:0/1493770383 misdirected client.9450549.1:1379645861 pg 2.ec640804 to osd.190 in e241865, client e241865 pg 2.804 features 288863570635346
</pre>
e241864 -> e241865 incremental:
<pre>
{
"epoch": 241865,
"fsid": "9e3e9015-f626-4a44-83f7-0a939ef7ec02",
"modified": "2017-02-27 11:07:56.497658",
"new_pool_max": -1,
"new_flags": -1,
"new_max_osd": -1,
"new_pools": [],
"new_pool_names": [],
"old_pools": [],
"new_up_osds": [],
"new_weight": [],
"osd_state_xor": [
{
"osd": 204,
"state_xor": [
"autoout",
"exists"
]
},
[],
[],
[],
[],
[],
[],
[],
[],
[
{
"osd": 204,
"uuid": "00000000-0000-0000-0000-000000000000"
}
],
{},
[]
]
</pre>
On master:
<pre>
$ bin/osdmaptool --test-map-pg 2.ec640804 /tmp/map-241864.bin
bin/osdmaptool: osdmap file '/tmp/map-241864.bin'
parsed '2.ec640804' -> 2.ec640804
2.ec640804 raw ([197,201,1], p197) up ([197,201,1], p197) acting ([197,201,1], p197)
$ bin/osdmaptool --test-map-pg 2.ec640804 /tmp/map-241865.bin
bin/osdmaptool: osdmap file '/tmp/map-241865.bin'
parsed '2.ec640804' -> 2.ec640804
2.ec640804 raw ([197,201,1], p197) up ([197,201,1], p197) acting ([197,201,1], p197)
</pre>
but (with osdmaptool patched to accept and apply incrementals):
<pre>
$ bin/osdmaptool --test-map-pg 2.ec640804 /tmp/map-241864.bin /tmp/inc-241865.bin
bin/osdmaptool: osdmap file '/tmp/map-241864.bin'
bin/osdmaptool: incremental file '/tmp/inc-241865.bin'
parsed '2.ec640804' -> 2.ec640804
2.ec640804 raw ([190,1], p190) up ([190,1], p190) acting ([190,1], p190)
</pre>
which is where the misdirected request was sent.
On hammer:
<pre>
$ ./osdmaptool --test-map-pg 2.ec640804 /tmp/map-241864.bin
./osdmaptool: osdmap file '/tmp/map-241864.bin'
parsed '2.ec640804' -> 2.ec640804
2.ec640804 raw ([197,201,1], p197) up ([197,201,1], p197) acting ([197,201,1], p197)
$ ./osdmaptool --test-map-pg 2.ec640804 /tmp/map-241865.bin
./osdmaptool: osdmap file '/tmp/map-241865.bin'
parsed '2.ec640804' -> 2.ec640804
2.ec640804 raw ([197,201,1], p197) up ([197,201,1], p197) acting ([197,201,1], p197)
</pre>
and (same osdmaptool patch): but
<pre>
$ ./osdmaptool --test-map-pg 2.ec640804 /tmp/map-241864.bin /tmp/inc-241865.bin
./osdmaptool: osdmap file '/tmp/map-241864.bin'
./osdmaptool: incremental file '/tmp/inc-241865.bin'
parsed '2.ec640804' -> 2.ec640804
2.ec640804 raw ([197,201,1], p197) up ([197,201,1], p197) acting ([197,201,1], p197)
</pre>