Ceph » RADOS

/a/sage-2017-06-06_21:54:14-rados-wip-sage-testing-distro-basic-smithi/1265435

rados/thrash/{0-size-min-size-overrides/2-size-1-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-require-luminous/at-end.yaml msgr-failures/fastclose.yaml msgr/random.yaml objectstore/filestore-btrfs.yaml rados.yaml rocksdb.yaml thrashers/morepggrow.yaml workloads/radosbench.yaml}

/a/sage-2017-06-06_21:54:14-rados-wip-sage-testing-distro-basic-smithi/1265467
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-require-luminous/at-end.yaml msgr-failures/osd-delay.yaml msgr/async.yaml objectstore/filestore-btrfs.yaml rados.yaml rocksdb.yaml thrashers/morepggrow.yaml workloads/cache-pool-snaps-readproxy.yaml}

/a/kchai-2017-06-15_17:39:27-rados-wip-kefu-testing---basic-smithi/1291475 also with rocksdb + btrfs

/a/sage-2017-06-16_00:46:50-rados-wip-sage-testing-distro-basic-smithi/1292433

rados/thrash-erasure-code/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-require-luminous/at-mkfs.yaml fast/fast.yaml leveldb.yaml msgr-failures/fastclose.yaml objectstore/filestore-btrfs.yaml rados.yaml thrashers/mapgap.yaml workloads/ec-radosbench.yaml}

Do we have any idea why it hasn't popped up in leveldb? Is the multi-threading stuff less conducive to being snapshotted somehow?

http://pulpito.ceph.com/jdillaman-2017-05-25_16:48:38-rbd-wip-jd-testing-distro-basic-smithi/1229611

/a/sage-2017-06-20_16:21:45-rados-wip-sage-testing2-distro-basic-smithi/1305525
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-require-luminous/at-end.yaml msgr-failures/few.yaml msgr/random.yaml objectstore/filestore-btrfs.yaml rados.yaml rocksdb.yaml thrashers/morepggrow.yaml workloads/radosbench.yaml}

2017-06-20 18:18:22.913720 7fd59ba16700 10 filestore(/var/lib/ceph/osd/ceph-3) error opening file /var/lib/ceph/osd/ceph-3/current/0.5c_head/benchmark\udata\usmithi107\u8090\uobject14919__head_B9E1125C__0 with flags=66: (28) No space left on device

  r = ::open((*path)->path(), flags, 0644);
  if (r < 0) {
    r = -errno;
    dout(10) << "error opening file " << (*path)->path() << " with flags=" 
      << flags << ": " << cpp_strerror(-r) << dendl;
    goto fail;
  }

2017-06-20 18:18:21.871308 7fd5861c3700 20 osd.3 505 update_osd_stat osd_stat(1322 MB used, 94065 MB avail, 95388 MB total, peers [0,1,2,4,5] op hist [])

2017-06-20T18:03:07.093 INFO:tasks.ceph:['mkfs.btrfs', '-m', 'single', '-l', '32768', '-n', '32768', '-f'] on /dev/nvme0n1p1 on ubuntu@smithi085.front.sepia.ceph.com
2017-06-20T18:03:07.093 INFO:teuthology.orchestra.run.smithi085:Running: 'yes | sudo mkfs.btrfs -m single -l 32768 -n 32768 -f /dev/nvme0n1p1'
2017-06-20T18:03:07.164 INFO:teuthology.orchestra.run.smithi085.stderr:WARNING: --leafsize is deprecated, use --nodesize
2017-06-20T18:03:07.595 INFO:teuthology.orchestra.run.smithi085.stdout:btrfs-progs v4.4
2017-06-20T18:03:07.596 INFO:teuthology.orchestra.run.smithi085.stdout:See http://btrfs.wiki.kernel.org for more information.
2017-06-20T18:03:07.596 INFO:teuthology.orchestra.run.smithi085.stdout:
2017-06-20T18:03:07.596 INFO:teuthology.orchestra.run.smithi085.stdout:Performing full device TRIM (93.15GiB) ...
2017-06-20T18:03:07.596 INFO:teuthology.orchestra.run.smithi085.stdout:Label:              (null)
2017-06-20T18:03:07.596 INFO:teuthology.orchestra.run.smithi085.stdout:UUID:               914843f1-ebf3-426d-948a-d89b720f6a8f
2017-06-20T18:03:07.596 INFO:teuthology.orchestra.run.smithi085.stdout:Node size:          32768
2017-06-20T18:03:07.596 INFO:teuthology.orchestra.run.smithi085.stdout:Sector size:        4096
2017-06-20T18:03:07.597 INFO:teuthology.orchestra.run.smithi085.stdout:Filesystem size:    93.15GiB
2017-06-20T18:03:07.597 INFO:teuthology.orchestra.run.smithi085.stdout:Block group profiles:
2017-06-20T18:03:07.597 INFO:teuthology.orchestra.run.smithi085.stdout:  Data:             single            8.00MiB
2017-06-20T18:03:07.597 INFO:teuthology.orchestra.run.smithi085.stdout:  Metadata:         single            8.00MiB
2017-06-20T18:03:07.597 INFO:teuthology.orchestra.run.smithi085.stdout:  System:           single            4.00MiB
2017-06-20T18:03:07.597 INFO:teuthology.orchestra.run.smithi085.stdout:SSD detected:       yes
2017-06-20T18:03:07.597 INFO:teuthology.orchestra.run.smithi085.stdout:Incompat features:  extref, skinny-metadata
2017-06-20T18:03:07.597 INFO:teuthology.orchestra.run.smithi085.stdout:Number of devices:  1
2017-06-20T18:03:07.598 INFO:teuthology.orchestra.run.smithi085.stdout:Devices:
2017-06-20T18:03:07.598 INFO:teuthology.orchestra.run.smithi085.stdout:   ID        SIZE  PATH
2017-06-20T18:03:07.598 INFO:teuthology.orchestra.run.smithi085.stdout:    1    93.15GiB  /dev/nvme0n1p1

I've seen xenial and centos failures now, no trusty yet.

Has this been reproduced with the following kernel fix applied?

commit 70e7af244f24c94604ef6eca32ad297632018583
Author: Omar Sandoval <osandov@fb.com>
Date: Fri Jun 2 01:20:01 2017 -0700

Btrfs: fix delalloc accounting leak caused by u32 overflow

btrfs_calc_trans_metadata_size() does an unsigned 32-bit multiplication,
    which can overflow if num_items >= 4 GB / (nodesize * BTRFS_MAX_LEVEL * 2).
    For a nodesize of 16kB, this overflow happens at 16k items. Usually,
    num_items is a small constant passed to btrfs_start_transaction(), but
    we also use btrfs_calc_trans_metadata_size() for metadata reservations
    for extent items in btrfs_delalloc_{reserve,release}_metadata().

In drop_outstanding_extents(), num_items is calculated as
    inode->reserved_extents - inode->outstanding_extents. The difference
    between these two counters is usually small, but if many delalloc
    extents are reserved and then the outstanding extents are merged in
    btrfs_merge_extent_hook(), the difference can become large enough to
    overflow in btrfs_calc_trans_metadata_size().

The overflow manifests itself as a leak of a multiple of 4 GB in
    delalloc_block_rsv and the metadata bytes_may_use counter. This in turn
    can cause early ENOSPC errors. Additionally, these WARN_ONs in
    extent-tree.c will be hit when unmounting:

WARN_ON(fs_info->delalloc_block_rsv.size > 0);
        WARN_ON(fs_info->delalloc_block_rsv.reserved > 0);
        WARN_ON(space_info->bytes_pinned > 0 ||
                space_info->bytes_reserved > 0 ||
                space_info->bytes_may_use > 0);

Fix it by casting nodesize to a u64 so that
    btrfs_calc_trans_metadata_size() does a full 64-bit multiplication.
    While we're here, do the same in btrfs_calc_trunc_metadata_size(); this
    can't overflow with any existing uses, but it's better to be safe here
    than have another hard-to-debug problem later on.

Cc: stable@vger.kernel.org
    Signed-off-by: Omar Sandoval <osandov@fb.com>
    Reviewed-by: David Sterba <dsterba@suse.com>
    Signed-off-by: Chris Mason <clm@fb.com>