Bug #17972
hammer: enospc on setxattr on btrfs
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
This happened in hammer 0.94.10 integration testing. See http://tracker.ceph.com/issues/17151#note-11
The same error happened twice in the fs suite:- http://pulpito.front.sepia.ceph.com/smithfarm-2016-11-18_09:59:00-fs-hammer-backports---basic-smithi/558529/
- http://pulpito.front.sepia.ceph.com/smithfarm-2016-11-18_09:59:00-fs-hammer-backports---basic-smithi/558561/
- http://pulpito.front.sepia.ceph.com/smithfarm-2016-11-20_22:39:17-fs-hammer-backports---basic-smithi/565545/
- /a/smithfarm-2016-11-18_09:59:00-fs-hammer-backports---basic-smithi/558529/remote/smithi083/log/ceph-osd.3.log.gz
- /a/smithfarm-2016-11-18_09:59:00-fs-hammer-backports---basic-smithi/558561/remote/smithi016/log/ceph-osd.3.log.gz
- /a/smithfarm-2016-11-20_22:39:17-fs-hammer-backports---basic-smithi/565545/remote/smithi090/log/ceph-osd.3.log.gz
The fact that it occurred on osd.3 in all three tests piques the curiosity.
Sample log excerpt:
2016-11-19 15:40:29.121713 7ff72fd65700 -1 filestore(/var/lib/ceph/osd/ceph-3) FileStore::_setattrs: chain_setxattr returned -28 2016-11-19 15:40:29.121715 7ff720bda700 10 osd.3 6 dequeue_op 0x88f41c0 finish 2016-11-19 15:40:29.121717 7ff72fd65700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 2.3_head/2/c072a833/1000000017b.00000009/head = -28 2016-11-19 15:40:29.121726 7ff72fd65700 0 filestore(/var/lib/ceph/osd/ceph-3) ENOSPC on setxattr on 2.3_head/2/c072a833/1000000017b.00000009/head name snapset size 31 2016-11-19 15:40:29.121730 7ff72fd65700 0 filestore(/var/lib/ceph/osd/ceph-3) error (28) No space left on device not handled on operation 0x64aa628 (277824.0.5, or op 5, counting from 0) 2016-11-19 15:40:29.121738 7ff72fd65700 0 filestore(/var/lib/ceph/osd/ceph-3) ENOSPC handling not implemented 2016-11-19 15:40:29.121740 7ff72fd65700 0 filestore(/var/lib/ceph/osd/ceph-3) transaction dump: { "ops": [ { "op_num": 0, "op_name": "omap_setkeys", "collection": "2.3_head", "oid": "2\/3\/\/head", "attr_lens": { "0000000006.00000000000000045327": 164 } }, { "op_num": 1, "op_name": "omap_setkeys", "collection": "2.3_head", "oid": "2\/3\/\/head", "attr_lens": { "_epoch": 4, "_info": 738 } }, { "op_num": 2, "op_name": "omap_setkeys", "collection": "2.3_head", "oid": "2\/3\/\/head", "attr_lens": { "0000000006.00000000000000045327": 164, "can_rollback_to": 12, "rollback_info_trimmed_to": 12 } }, { "op_num": 3, "op_name": "write", "collection": "2.3_head", "oid": "2\/c072a833\/1000000017b.00000009\/head", "length": 4096, "offset": 2764800, "bufferlist length": 4096 }, { "op_num": 4, "op_name": "setattr", "collection": "2.3_head", "oid": "2\/c072a833\/1000000017b.00000009\/head", "name": "_", "length": 239 }, { "op_num": 5, "op_name": "setattr", "collection": "2.3_head", "oid": "2\/c072a833\/1000000017b.00000009\/head", "name": "snapset", "length": 31 } ] }
Related issues
History
#1 Updated by Nathan Cutler over 7 years ago
- Description updated (diff)
#2 Updated by Nathan Cutler over 7 years ago
- Description updated (diff)
#3 Updated by Nathan Cutler over 7 years ago
- Related to Bug #16257: Ceph random bug is killing osds (os/filestore/FileStore.cc: 2912: FAILED assert(0 == "unexpected error") added
#4 Updated by Nathan Cutler over 7 years ago
- Description updated (diff)
#5 Updated by Sage Weil over 7 years ago
Can you see whether the disk actually filled up? We generally shouldn't see ENOSPC from setxattr.... I'm guessing there is a bug where we're setting a large xattr and shouldn't be.
#6 Updated by Sage Weil over 7 years ago
- Status changed from New to Won't Fix
Disk was not full. They were all btrfs. wontfix!
#7 Updated by Sage Weil over 7 years ago
#8 Updated by Sage Weil over 7 years ago
- Subject changed from hammer filestore does not implement ENOSPC handling? to hammer: enospc on setxattr on btrfs