Bug #22656
openscrub mismatch on bytes (cache pools)
0%
Description
"2018-01-10 05:23:16.696161 osd.4 osd.4 172.21.15.43:6805/26441 370 : cluster [ERR] 3.2 scrub stat mismatch, got 46/46 objects, 1/1 clones, 21/21 dirty, 0/0 omap, 0/0 pinned, 1/1 hit_set_archive, 7/7 whiteouts, 110208381/111851830 bytes, 262/262 hit_set_archive bytes." in cluster log
/a/yuriw-2018-01-09_21:50:35-rados-wip-yuri2-testing-2018-01-09-1813-distro-basic-smithi/2050969
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/fastclose.yaml msgr/async.yaml objectstore/bluestore.yaml rados.yaml rocksdb.yaml thrashers/morepggrow.yaml thrashosds-health.yaml workloads/cache-pool-snaps.yaml}
(Saw this yesterday on another run, too)
Updated by Sage Weil over 6 years ago
/a/sage-2018-01-17_14:40:55-rados-wip-sage-testing-2018-01-16-2156-distro-basic-smithi/2082959
description: rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml deep-scrub/sleep.yaml msgr-failures/osd-delay.yaml msgr/simple.yaml objectstore/bluestore-comp.yaml rados.yaml rocksdb.yaml thrashers/morepggrow.yaml thrashosds-health.yaml workloads/cache-pool-snaps-readproxy.yaml}
Updated by Sage Weil over 6 years ago
- Subject changed from scrub mismatch on bytes to scrub mismatch on bytes (cache pools)
Updated by Nathan Cutler about 6 years ago
Happened here as well: http://pulpito.ceph.com/smithfarm-2018-01-24_19:46:55-rados-wip-smithfarm-testing-distro-basic-smithi/2106400/
description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/few.yaml msgr/random.yaml objectstore/bluestore.yaml rados.yaml rocksdb.yaml thrashers/morepggrow.yaml thrashosds-health.yaml workloads/cache-snaps.yaml}
wip-smithfarm-testing is current master plus some build/ops PRs
Updated by Sage Weil about 6 years ago
/a/sage-2018-01-29_18:07:24-rados-wip-sage-testing-2018-01-29-0927-distro-basic-smithi/2122957
description: rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/fastclose.yaml msgr/random.yaml objectstore/bluestore-bitmap.yaml rados.yaml rocksdb.yaml thrashers/morepggrow.yaml thrashosds-health.yaml workloads/cache-pool-snaps-readproxy.yaml}
Updated by Greg Farnum about 6 years ago
- Category set to Tiering
- Priority changed from Urgent to High
- Component(RADOS) OSD added
We just aren't assigning that much priority to cache tiering.
Updated by Sage Weil about 6 years ago
aah, just popped up on luminous: http://pulpito.ceph.com/yuriw-2018-02-05_23:07:16-rados-wip-yuri-testing-2018-02-05-2032-luminous-distro-basic-smithi/2156188
Updated by Kefu Chai about 6 years ago
please note, this test does not involve any cache pool.
Updated by Sage Weil about 6 years ago
/a/sage-2018-03-11_23:03:25-rados-wip-sage2-testing-2018-03-10-1616-distro-basic-smithi/2280391
description: rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/osd-delay.yaml msgr/async.yaml objectstore/bluestore.yaml rados.yaml rocksdb.yaml thrashers/pggrow.yaml thrashosds-health.yaml workloads/cache-pool-snaps.yaml}
Updated by Greg Farnum about 6 years ago
- Related to Bug #23228: scrub mismatch on objects added
Updated by Sage Weil about 6 years ago
Just bytes
dzafman-2018-03-28_18:21:29-rados-wip-zafman-testing-distro-basic-smithi/2332093
[ERR] 3.0 scrub stat mismatch, got
51/51 objects, 4/4 clones, 24/24 dirty, 0/0 omap, 0/0 pinned, 1/1 hit_set_archive,
7/7 whiteouts, 119951885/120679921 bytes, 0/0 manifest objects, 269/269 hit_set_archive
bytes
(originally posted on #23228)
Updated by Sage Weil almost 6 years ago
/a/sage-2018-05-02_22:22:16-rados-wip-sage3-testing-2018-05-02-1448-distro-basic-smithi/2468046
description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/few.yaml msgr/random.yaml objectstore/bluestore.yaml rados.yaml rocksdb.yaml thrashers/morepggrow.yaml thrashosds-health.yaml workloads/cache-snaps.yaml}
Updated by David Zafman almost 6 years ago
description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/osd-delay.yaml msgr/simple.yaml objectstore/bluestore-bitmap.yaml rados.yaml rocksdb.yaml thrashers/default.yaml thrashosds-health.yaml workloads/cache-pool-snaps.yaml}
Updated by Neha Ojha over 4 years ago
- Status changed from 12 to Can't reproduce
- Priority changed from High to Normal
Updated by Neha Ojha over 4 years ago
- Status changed from Can't reproduce to 12
mimic with cache pools: /a/yuriw-2019-11-09_18:55:35-rados-wip-yuri-mimic_13.2.7_RC2-distro-basic-smithi/4489809/
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-bitmap.yaml rados.yaml rocksdb.yaml supported-random-distro$/{centos_latest.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/cache-pool-snaps-readproxy.yaml}