Project

General

Profile

Bug #17972

hammer: enospc on setxattr on btrfs

Added by Nathan Cutler over 7 years ago. Updated over 7 years ago.

Status:
Won't Fix
Priority:
High
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

This happened in hammer 0.94.10 integration testing. See http://tracker.ceph.com/issues/17151#note-11

The same error happened twice in the fs suite: Full OSD logs can be found here:
  • /a/smithfarm-2016-11-18_09:59:00-fs-hammer-backports---basic-smithi/558529/remote/smithi083/log/ceph-osd.3.log.gz
  • /a/smithfarm-2016-11-18_09:59:00-fs-hammer-backports---basic-smithi/558561/remote/smithi016/log/ceph-osd.3.log.gz
  • /a/smithfarm-2016-11-20_22:39:17-fs-hammer-backports---basic-smithi/565545/remote/smithi090/log/ceph-osd.3.log.gz

The fact that it occurred on osd.3 in all three tests piques the curiosity.

Sample log excerpt:

2016-11-19 15:40:29.121713 7ff72fd65700 -1 filestore(/var/lib/ceph/osd/ceph-3) FileStore::_setattrs: chain_setxattr returned -28
2016-11-19 15:40:29.121715 7ff720bda700 10 osd.3 6 dequeue_op 0x88f41c0 finish
2016-11-19 15:40:29.121717 7ff72fd65700 10 filestore(/var/lib/ceph/osd/ceph-3) setattrs 2.3_head/2/c072a833/1000000017b.00000009/head = -28
2016-11-19 15:40:29.121726 7ff72fd65700  0 filestore(/var/lib/ceph/osd/ceph-3)  ENOSPC on setxattr on 2.3_head/2/c072a833/1000000017b.00000009/head name snapset size 31
2016-11-19 15:40:29.121730 7ff72fd65700  0 filestore(/var/lib/ceph/osd/ceph-3)  error (28) No space left on device not handled on operation 0x64aa628 (277824.0.5, or op 5, counting from 0)
2016-11-19 15:40:29.121738 7ff72fd65700  0 filestore(/var/lib/ceph/osd/ceph-3) ENOSPC handling not implemented
2016-11-19 15:40:29.121740 7ff72fd65700  0 filestore(/var/lib/ceph/osd/ceph-3)  transaction dump:
{
    "ops": [
        {
            "op_num": 0,
            "op_name": "omap_setkeys",
            "collection": "2.3_head",
            "oid": "2\/3\/\/head",
            "attr_lens": {
                "0000000006.00000000000000045327": 164
            }
        },
        {
            "op_num": 1,
            "op_name": "omap_setkeys",
            "collection": "2.3_head",
            "oid": "2\/3\/\/head",
            "attr_lens": {
                "_epoch": 4,
                "_info": 738
            }
        },
        {   
            "op_num": 2,
            "op_name": "omap_setkeys",
            "collection": "2.3_head",
            "oid": "2\/3\/\/head",
            "attr_lens": {
                "0000000006.00000000000000045327": 164,
                "can_rollback_to": 12,
                "rollback_info_trimmed_to": 12
            }
        },
        {
            "op_num": 3,
            "op_name": "write",
            "collection": "2.3_head",
            "oid": "2\/c072a833\/1000000017b.00000009\/head",
            "length": 4096,
            "offset": 2764800,
            "bufferlist length": 4096
        },
        {
            "op_num": 4,
            "op_name": "setattr",
            "collection": "2.3_head",
            "oid": "2\/c072a833\/1000000017b.00000009\/head",
            "name": "_",
            "length": 239
        },
        {
            "op_num": 5,
            "op_name": "setattr",
            "collection": "2.3_head",
            "oid": "2\/c072a833\/1000000017b.00000009\/head",
            "name": "snapset",
            "length": 31
        }
    ]
}

Related issues

Related to Ceph - Bug #16257: Ceph random bug is killing osds (os/filestore/FileStore.cc: 2912: FAILED assert(0 == "unexpected error") Closed 06/13/2016

History

#1 Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)

#2 Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)

#3 Updated by Nathan Cutler over 7 years ago

  • Related to Bug #16257: Ceph random bug is killing osds (os/filestore/FileStore.cc: 2912: FAILED assert(0 == "unexpected error") added

#4 Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)

#5 Updated by Sage Weil over 7 years ago

Can you see whether the disk actually filled up? We generally shouldn't see ENOSPC from setxattr.... I'm guessing there is a bug where we're setting a large xattr and shouldn't be.

#6 Updated by Sage Weil over 7 years ago

  • Status changed from New to Won't Fix

Disk was not full. They were all btrfs. wontfix!

#8 Updated by Sage Weil over 7 years ago

  • Subject changed from hammer filestore does not implement ENOSPC handling? to hammer: enospc on setxattr on btrfs

Also available in: Atom PDF