Ceph

Bug #18533

Updated by Dan Mick about 7 years ago

Twice, I've noticed this sort of error pop on the long-running cluster: one of the metadata objects reports one replica has a different omap_digest: 

 <pre> 
 list-inconsistent-obj 1.3c metadata 
 { 
   "epoch": 771290, 
   "inconsistents": [ 
     { 
       "object": { 
         "name": "607.00000000", 
         "nspace": "", 
         "locator": "", 
         "snap": "head", 
         "version": 8962591 
       }, 
       "errors": [ 
         "omap_digest_mismatch" 
       ], 
       "union_shard_errors": [], 
       "selected_object_info": "1:3ed09add:::607.00000000:head(769238'8962591 mds.0.95185:3872723 dirty|omap|data_digest s 0 uv 8962591 dd ffffffff alloc_hint [0 0 0])", 
       "shards": [ 
         { 
           "osd": 31, 
           "errors": [], 
           "size": 0, 
           "omap_digest": "0xa99faf1c", 
           "data_digest": "0xffffffff" 
         }, 
         { 
           "osd": 48, 
           "errors": [], 
           "size": 0, 
           "omap_digest": "0x0b59d114", 
           "data_digest": "0xffffffff" 
         }, 
         { 
           "osd": 61, 
           "errors": [], 
           "size": 0, 
           "omap_digest": "0xa99faf1c", 
           "data_digest": "0xffffffff" 
         }, 
         { 
           "osd": 69, 
           "errors": [], 
           "size": 0, 
           "omap_digest": "0xa99faf1c", 
           "data_digest": "0xffffffff" 
         } 
       ] 
     } 
   ] 
 } 
 </pre> 


 Edit: removed note about xattr; not This object's omap has one key, 'parent', whose value is: 
 <pre> 
 00000000    05 04 38 00 00 00 07 06    00 00 00 00 00 00 01 00    |..8.............| 
 00000010    00 00 02 02 1a 00 00 00    00 01 00 00 00 00 00 00    |................| 
 00000020    06 00 00 00 73 74 72 61    79 37 47 4d 0e 00 00 00    |....stray7GM....| 
 00000030    00 00 01 00 00 00 00 00    00 00 00 00 00 00          |..............| 
 0000003e 
 </pre> 

 On the point here, I differing osd, the value is: 

 <pre> 
 ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-48 '["1.3c",{"oid":"607.00000000","key":"","snapid":-2,"hash":3143175036,"max":0,"pool":1,"namespace":"","max":0}]' get-attr _parent | hd 

 00000000    05 04 38 00 00 00 07 06    00 00 00 00 00 00 01 00    |..8.............| 
 00000010    00 00 02 02 1a 00 00 00    00 01 00 00 00 00 00 00    |................| 
 00000020    06 00 00 00 73 74 72 61    79 37 47 4d 0e 00 00 00    |....stray7GM....| 
 00000030    00 00 01 00 00 00 00 00    00 00 00 00 00 00          |..............| 
 0000003e 

 </pre> 

 which looks correct to me, but for some reason the omap_digest was confused 
  calculated differently during the scrub, maybe?.... 

 I've left the object as is. 

 

Back