Bug #17972
Updated by Nathan Cutler over 7 years ago
This happened in hammer 0.94.10 integration testing. See http://tracker.ceph.com/issues/17151#note-11 The same error happened twice in the fs suite: * http://pulpito.front.sepia.ceph.com/smithfarm-2016-11-18_09:59:00-fs-hammer-backports---basic-smithi/558529/ * http://pulpito.front.sepia.ceph.com/smithfarm-2016-11-18_09:59:00-fs-hammer-backports---basic-smithi/558561/ * http://pulpito.front.sepia.ceph.com/smithfarm-2016-11-20_22:39:17-fs-hammer-backports---basic-smithi/565545/ Full OSD logs can be found here: * /a/smithfarm-2016-11-18_09:59:00-fs-hammer-backports---basic-smithi/558529/remote/smithi083/log/ceph-osd.3.log.gz * /a/smithfarm-2016-11-18_09:59:00-fs-hammer-backports---basic-smithi/558561/remote/smithi016/log/ceph-osd.3.log.gz * /a/smithfarm-2016-11-20_22:39:17-fs-hammer-backports---basic-smithi/565545/remote/smithi090/log/ceph-osd.3.log.gz The fact that it occurred on osd.3 in all three tests piques the curiosity. Sample log excerpt: <pre> 2016-11-19 15:40:29.121713 7ff72fd65700 -1 filestore(/var/lib/ceph/osd/ceph-3) FileStore::_setattrs: chain_setxattr returned -28 2016-11-19 15:40:29.121715 7ff720bda700 10 osd.3 6 dequeue_op 0x88f41c0 finish 2016-11-19 15:40:29.121717 7ff72fd65700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 2.3_head/2/c072a833/1000000017b.00000009/head = -28 2016-11-19 15:40:29.121726 7ff72fd65700 0 filestore(/var/lib/ceph/osd/ceph-3) ENOSPC on setxattr on 2.3_head/2/c072a833/1000000017b.00000009/head name snapset size 31 2016-11-19 15:40:29.121730 7ff72fd65700 0 filestore(/var/lib/ceph/osd/ceph-3) error (28) No space left on device not handled on operation 0x64aa628 (277824.0.5, or op 5, counting from 0) 2016-11-19 15:40:29.121738 7ff72fd65700 0 filestore(/var/lib/ceph/osd/ceph-3) ENOSPC handling not implemented 2016-11-19 15:40:29.121740 7ff72fd65700 0 filestore(/var/lib/ceph/osd/ceph-3) transaction dump: { "ops": [ { "op_num": 0, "op_name": "omap_setkeys", "collection": "2.3_head", "oid": "2\/3\/\/head", "attr_lens": { "0000000006.00000000000000045327": 164 } }, { "op_num": 1, "op_name": "omap_setkeys", "collection": "2.3_head", "oid": "2\/3\/\/head", "attr_lens": { "_epoch": 4, "_info": 738 } }, { "op_num": 2, "op_name": "omap_setkeys", "collection": "2.3_head", "oid": "2\/3\/\/head", "attr_lens": { "0000000006.00000000000000045327": 164, "can_rollback_to": 12, "rollback_info_trimmed_to": 12 } }, { "op_num": 3, "op_name": "write", "collection": "2.3_head", "oid": "2\/c072a833\/1000000017b.00000009\/head", "length": 4096, "offset": 2764800, "bufferlist length": 4096 }, { "op_num": 4, "op_name": "setattr", "collection": "2.3_head", "oid": "2\/c072a833\/1000000017b.00000009\/head", "name": "_", "length": 239 }, { "op_num": 5, "op_name": "setattr", "collection": "2.3_head", "oid": "2\/c072a833\/1000000017b.00000009\/head", "name": "snapset", "length": 31 } ] } </pre>