Ceph

Bug #17972

Updated by Nathan Cutler over 7 years ago

This happened in hammer 0.94.10 integration testing. See http://tracker.ceph.com/issues/17151#note-11 

 The same error happened twice test in the question (in fs suite: 
 * suite) is http://pulpito.front.sepia.ceph.com/smithfarm-2016-11-18_09:59:00-fs-hammer-backports---basic-smithi/558529/ 
 * http://pulpito.front.sepia.ceph.com/smithfarm-2016-11-18_09:59:00-fs-hammer-backports---basic-smithi/558561/ 

 Full OSD logs log can be found here: 
 * /a/smithfarm-2016-11-18_09:59:00-fs-hammer-backports---basic-smithi/558529/remote/smithi083/log/ceph-osd.3.log.gz 
 * /a/smithfarm-2016-11-18_09:59:00-fs-hammer-backports---basic-smithi/558561/remote/smithi016/log/ceph-osd.3.log.gz 

 Sample Relevant log excerpt: 

 <pre> 
 2016-11-19 15:40:29.121713 7ff72fd65700 -1 filestore(/var/lib/ceph/osd/ceph-3) FileStore::_setattrs: chain_setxattr returned -28 
 2016-11-19 15:40:29.121715 7ff720bda700 10 osd.3 6 dequeue_op 0x88f41c0 finish 
 2016-11-19 15:40:29.121717 7ff72fd65700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 2.3_head/2/c072a833/1000000017b.00000009/head = -28 
 2016-11-19 15:40:29.121726 7ff72fd65700    0 filestore(/var/lib/ceph/osd/ceph-3)    ENOSPC on setxattr on 2.3_head/2/c072a833/1000000017b.00000009/head name snapset size 31 
 2016-11-19 15:40:29.121730 7ff72fd65700    0 filestore(/var/lib/ceph/osd/ceph-3)    error (28) No space left on device not handled on operation 0x64aa628 (277824.0.5, or op 5, counting from 0) 
 2016-11-19 15:40:29.121738 7ff72fd65700    0 filestore(/var/lib/ceph/osd/ceph-3) ENOSPC handling not implemented 
 2016-11-19 15:40:29.121740 7ff72fd65700    0 filestore(/var/lib/ceph/osd/ceph-3)    transaction dump: 
 { 
     "ops": [ 
         { 
             "op_num": 0, 
             "op_name": "omap_setkeys", 
             "collection": "2.3_head", 
             "oid": "2\/3\/\/head", 
             "attr_lens": { 
                 "0000000006.00000000000000045327": 164 
             } 
         }, 
         { 
             "op_num": 1, 
             "op_name": "omap_setkeys", 
             "collection": "2.3_head", 
             "oid": "2\/3\/\/head", 
             "attr_lens": { 
                 "_epoch": 4, 
                 "_info": 738 
             } 
         }, 
         {    
             "op_num": 2, 
             "op_name": "omap_setkeys", 
             "collection": "2.3_head", 
             "oid": "2\/3\/\/head", 
             "attr_lens": { 
                 "0000000006.00000000000000045327": 164, 
                 "can_rollback_to": 12, 
                 "rollback_info_trimmed_to": 12 
             } 
         }, 
         { 
             "op_num": 3, 
             "op_name": "write", 
             "collection": "2.3_head", 
             "oid": "2\/c072a833\/1000000017b.00000009\/head", 
             "length": 4096, 
             "offset": 2764800, 
             "bufferlist length": 4096 
         }, 
         { 
             "op_num": 4, 
             "op_name": "setattr", 
             "collection": "2.3_head", 
             "oid": "2\/c072a833\/1000000017b.00000009\/head", 
             "name": "_", 
             "length": 239 
         }, 
         { 
             "op_num": 5, 
             "op_name": "setattr", 
             "collection": "2.3_head", 
             "oid": "2\/c072a833\/1000000017b.00000009\/head", 
             "name": "snapset", 
             "length": 31 
         } 
     ] 
 } 
 </pre>

Back