Ceph

Bug #19119

Updated by Ilya Dryomov about 7 years ago

I have a bunch of misdirected requests from a recent kernel client to a hammer cluster, triggered by osd rm: 

 <pre> 
 2017-02-27 15:37:56.976845 osd.190 10.115.1.133:6808/3914 97 : cluster [WRN] client.9450549 10.115.1.35:0/1493770383 misdirected client.9450549.1:1379645861 pg 2.ec640804 to osd.190 in e241865, client e241865 pg 2.804 features 288863570635346 
 </pre> 

 e241864 -> e241865 incremental: 

 <pre> 
 { 
     "epoch": 241865, 
     "fsid": "9e3e9015-f626-4a44-83f7-0a939ef7ec02", 
     "modified": "2017-02-27 11:07:56.497658", 
     "new_pool_max": -1, 
     "new_flags": -1, 
     "new_max_osd": -1, 
     "new_pools": [], 
     "new_pool_names": [], 
     "old_pools": [], 
     "new_up_osds": [], 
     "new_weight": [], 
     "osd_state_xor": [ 
         { 
             "osd": 204, 
             "state_xor": [ 
                 "autoout", 
                 "exists" 
             ] 
         }, 
         [], 
         [], 
         [], 
         [], 
         [], 
         [], 
         [], 
         [], 
         [ 
             { 
                 "osd": 204, 
                 "uuid": "00000000-0000-0000-0000-000000000000" 
             } 
         ], 
         {}, 
         [] 
     ] 
 </pre> 

 On master: 

 <pre> 
 $ bin/osdmaptool --test-map-pg 2.ec640804 /tmp/map-241864.bin  
 bin/osdmaptool: osdmap file '/tmp/map-241864.bin' 
  parsed '2.ec640804' -> 2.ec640804 
 2.ec640804 raw ([197,201,1], p197) up ([197,201,1], p197) acting ([197,201,1], p197) 
 $ bin/osdmaptool --test-map-pg 2.ec640804 /tmp/map-241865.bin  
 bin/osdmaptool: osdmap file '/tmp/map-241865.bin' 
  parsed '2.ec640804' -> 2.ec640804 
 2.ec640804 raw ([197,201,1], p197) up ([197,201,1], p197) acting ([197,201,1], p197) 
 </pre> 

 but (with osdmaptool patched to accept and apply incrementals): 

 <pre> 
 $ bin/osdmaptool --test-map-pg 2.ec640804 /tmp/map-241864.bin /tmp/inc-241865.bin  
 bin/osdmaptool: osdmap file '/tmp/map-241864.bin' 
 bin/osdmaptool: incremental file '/tmp/inc-241865.bin' 
  parsed '2.ec640804' -> 2.ec640804 
 2.ec640804 raw ([190,1], p190) up ([190,1], p190) acting ([190,1], p190) 
 </pre> 

 which is where the misdirected request was sent. 

 On hammer: 

 <pre> 
 $ ./osdmaptool --test-map-pg 2.ec640804 /tmp/map-241864.bin 
 ./osdmaptool: osdmap file '/tmp/map-241864.bin' 
  parsed '2.ec640804' -> 2.ec640804 
 2.ec640804 raw ([197,201,1], p197) up ([197,201,1], p197) acting ([197,201,1], p197) 
 $ ./osdmaptool --test-map-pg 2.ec640804 /tmp/map-241865.bin 
 ./osdmaptool: osdmap file '/tmp/map-241865.bin' 
  parsed '2.ec640804' -> 2.ec640804 
 2.ec640804 raw ([197,201,1], p197) up ([197,201,1], p197) acting ([197,201,1], p197) 
 </pre> 

 and (same osdmaptool patch): but 

 <pre> 
 $ ./osdmaptool --test-map-pg 2.ec640804 /tmp/map-241864.bin /tmp/inc-241865.bin 
 ./osdmaptool: osdmap file '/tmp/map-241864.bin' 
 ./osdmaptool: incremental file '/tmp/inc-241865.bin' 
  parsed '2.ec640804' -> 2.ec640804 
 2.ec640804 raw ([197,201,1], p197) up ([197,201,1], p197) acting ([197,201,1], p197) 
 </pre>

Back