Project

General

Profile

Bug #22656

scrub mismatch on bytes (cache pools)

Added by Sage Weil 9 months ago. Updated 5 months ago.

Status:
Verified
Priority:
High
Assignee:
-
Category:
Tiering
Target version:
-
Start date:
01/10/2018
Due date:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
OSD

Description

"2018-01-10 05:23:16.696161 osd.4 osd.4 172.21.15.43:6805/26441 370 : cluster [ERR] 3.2 scrub stat mismatch, got 46/46 objects, 1/1 clones, 21/21 dirty, 0/0 omap, 0/0 pinned, 1/1 hit_set_archive, 7/7 whiteouts, 110208381/111851830 bytes, 262/262 hit_set_archive bytes." in cluster log

/a/yuriw-2018-01-09_21:50:35-rados-wip-yuri2-testing-2018-01-09-1813-distro-basic-smithi/2050969

rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/fastclose.yaml msgr/async.yaml objectstore/bluestore.yaml rados.yaml rocksdb.yaml thrashers/morepggrow.yaml thrashosds-health.yaml workloads/cache-pool-snaps.yaml}

(Saw this yesterday on another run, too)


Related issues

Related to RADOS - Bug #23228: scrub mismatch on objects Closed 03/05/2018

History

#1 Updated by Sage Weil 9 months ago

/a/sage-2018-01-17_14:40:55-rados-wip-sage-testing-2018-01-16-2156-distro-basic-smithi/2082959

description: rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml deep-scrub/sleep.yaml msgr-failures/osd-delay.yaml msgr/simple.yaml objectstore/bluestore-comp.yaml rados.yaml rocksdb.yaml thrashers/morepggrow.yaml thrashosds-health.yaml workloads/cache-pool-snaps-readproxy.yaml}

#2 Updated by Sage Weil 9 months ago

  • Subject changed from scrub mismatch on bytes to scrub mismatch on bytes (cache pools)

#3 Updated by Nathan Cutler 9 months ago

Happened here as well: http://pulpito.ceph.com/smithfarm-2018-01-24_19:46:55-rados-wip-smithfarm-testing-distro-basic-smithi/2106400/

description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/few.yaml msgr/random.yaml objectstore/bluestore.yaml rados.yaml rocksdb.yaml thrashers/morepggrow.yaml thrashosds-health.yaml workloads/cache-snaps.yaml}

wip-smithfarm-testing is current master plus some build/ops PRs

#4 Updated by Sage Weil 9 months ago

/a/sage-2018-01-29_18:07:24-rados-wip-sage-testing-2018-01-29-0927-distro-basic-smithi/2122957
description: rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/fastclose.yaml msgr/random.yaml objectstore/bluestore-bitmap.yaml rados.yaml rocksdb.yaml thrashers/morepggrow.yaml thrashosds-health.yaml workloads/cache-pool-snaps-readproxy.yaml}

#5 Updated by Greg Farnum 9 months ago

  • Category set to Tiering
  • Priority changed from Urgent to High
  • Component(RADOS) OSD added

We just aren't assigning that much priority to cache tiering.

#8 Updated by Sage Weil 7 months ago

/a/sage-2018-03-11_23:03:25-rados-wip-sage2-testing-2018-03-10-1616-distro-basic-smithi/2280391

description: rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/osd-delay.yaml msgr/async.yaml objectstore/bluestore.yaml rados.yaml rocksdb.yaml thrashers/pggrow.yaml thrashosds-health.yaml workloads/cache-pool-snaps.yaml}

#9 Updated by Greg Farnum 7 months ago

  • Related to Bug #23228: scrub mismatch on objects added

#10 Updated by Sage Weil 6 months ago

Just bytes

dzafman-2018-03-28_18:21:29-rados-wip-zafman-testing-distro-basic-smithi/2332093

[ERR] 3.0 scrub stat mismatch, got
51/51 objects, 4/4 clones, 24/24 dirty, 0/0 omap, 0/0 pinned, 1/1 hit_set_archive,
7/7 whiteouts, 119951885/120679921 bytes, 0/0 manifest objects, 269/269 hit_set_archive
bytes

(originally posted on #23228)

#11 Updated by Sage Weil 6 months ago

/a/sage-2018-05-02_22:22:16-rados-wip-sage3-testing-2018-05-02-1448-distro-basic-smithi/2468046

description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/few.yaml msgr/random.yaml objectstore/bluestore.yaml rados.yaml rocksdb.yaml thrashers/morepggrow.yaml thrashosds-health.yaml workloads/cache-snaps.yaml}

#12 Updated by David Zafman 5 months ago

http://qa-proxy.ceph.com/teuthology/dzafman-2018-05-18_11:33:31-rados-wip-zafman-testing-mimic-distro-basic-smithi/2548845

description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/osd-delay.yaml msgr/simple.yaml objectstore/bluestore-bitmap.yaml rados.yaml rocksdb.yaml thrashers/default.yaml thrashosds-health.yaml workloads/cache-pool-snaps.yaml}

Also available in: Atom PDF