Project

General

Profile

Bug #58256

ObjectStore/StoreTestSpecificAUSize.SpilloverTest/2: Expected: (logger->get(l_bluefs_slow_used_bytes)) >= (16 * 1024 * 1024), actual: 0 vs 16777216

Added by Laura Flores over 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
High
Assignee:
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

/a/yuriw-2022-12-07_15:48:38-rados-wip-yuri3-testing-2022-12-06-1211-distro-default-smithi/7106884

2022-12-08T05:16:19.183 INFO:teuthology.orchestra.run.smithi181.stdout:[ RUN      ] ObjectStore/StoreTestSpecificAUSize.SpilloverTest/2
2022-12-08T05:16:23.353 INFO:teuthology.orchestra.run.smithi181.stderr:seeding object 0
2022-12-08T05:16:23.354 INFO:teuthology.orchestra.run.smithi181.stderr:seeding object 10
2022-12-08T05:16:23.382 INFO:teuthology.orchestra.run.smithi181.stderr:seeding object 20
2022-12-08T05:16:23.383 INFO:teuthology.orchestra.run.smithi181.stderr:seeding object 30
2022-12-08T05:16:23.383 INFO:teuthology.orchestra.run.smithi181.stderr:seeding object 40
2022-12-08T05:16:23.384 INFO:teuthology.orchestra.run.smithi181.stderr:seeding object 50
2022-12-08T05:16:23.384 INFO:teuthology.orchestra.run.smithi181.stderr:seeding object 60
2022-12-08T05:16:23.385 INFO:teuthology.orchestra.run.smithi181.stderr:seeding object 70
2022-12-08T05:16:23.385 INFO:teuthology.orchestra.run.smithi181.stderr:seeding object 80
2022-12-08T05:16:23.386 INFO:teuthology.orchestra.run.smithi181.stderr:seeding object 90
2022-12-08T05:16:23.386 INFO:teuthology.orchestra.run.smithi181.stderr:seeding object 100
2022-12-08T05:16:23.420 INFO:teuthology.orchestra.run.smithi181.stderr:seeding object 110
2022-12-08T05:16:23.421 INFO:teuthology.orchestra.run.smithi181.stderr:seeding object 120
2022-12-08T05:16:23.421 INFO:teuthology.orchestra.run.smithi181.stderr:seeding object 130
2022-12-08T05:16:23.422 INFO:teuthology.orchestra.run.smithi181.stderr:seeding object 140
2022-12-08T05:16:23.422 INFO:teuthology.orchestra.run.smithi181.stderr:seeding object 150
2022-12-08T05:16:23.423 INFO:teuthology.orchestra.run.smithi181.stderr:seeding object 160
2022-12-08T05:16:23.423 INFO:teuthology.orchestra.run.smithi181.stderr:seeding object 170
2022-12-08T05:16:23.424 INFO:teuthology.orchestra.run.smithi181.stderr:seeding object 180
2022-12-08T05:16:23.425 INFO:teuthology.orchestra.run.smithi181.stderr:seeding object 190
2022-12-08T05:16:23.425 INFO:teuthology.orchestra.run.smithi181.stderr:seeding object 200
2022-12-08T05:16:23.471 INFO:teuthology.orchestra.run.smithi181.stderr:seeding object 210
2022-12-08T05:16:23.472 INFO:teuthology.orchestra.run.smithi181.stderr:seeding object 220
2022-12-08T05:16:23.472 INFO:teuthology.orchestra.run.smithi181.stderr:seeding object 230
2022-12-08T05:16:23.472 INFO:teuthology.orchestra.run.smithi181.stderr:seeding object 240
2022-12-08T05:16:23.473 INFO:teuthology.orchestra.run.smithi181.stderr:seeding object 250
2022-12-08T05:16:23.473 INFO:teuthology.orchestra.run.smithi181.stderr:Op 0
2022-12-08T05:16:23.474 INFO:teuthology.orchestra.run.smithi181.stderr:available_objects: 251 in_flight_objects: 5 total objects: 256 in_flight 5
2022-12-08T05:16:31.768 INFO:teuthology.orchestra.run.smithi181.stderr:Op 100
2022-12-08T05:16:31.768 INFO:teuthology.orchestra.run.smithi181.stderr:available_objects: 247 in_flight_objects: 9 total objects: 256 in_flight 9
2022-12-08T05:16:41.961 INFO:teuthology.orchestra.run.smithi181.stderr:Op 200
2022-12-08T05:16:41.961 INFO:teuthology.orchestra.run.smithi181.stderr:available_objects: 256 in_flight_objects: 0 total objects: 256 in_flight 0
2022-12-08T05:16:48.560 INFO:teuthology.orchestra.run.smithi181.stdout:done
2022-12-08T05:17:05.936 INFO:teuthology.orchestra.run.smithi181.stdout:db_used:1132462080
2022-12-08T05:17:05.936 INFO:teuthology.orchestra.run.smithi181.stdout:slow_used:0
2022-12-08T05:17:05.936 INFO:teuthology.orchestra.run.smithi181.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-1241-g0a1e06a0/rpm/el8/BUILD/ceph-18.0.0-1241-g0a1e06a0/src/test/objectstore/store_test.cc:10411: Failure
2022-12-08T05:17:05.937 INFO:teuthology.orchestra.run.smithi181.stdout:Expected: (logger->get(l_bluefs_slow_used_bytes)) >= (16 * 1024 * 1024), actual: 0 vs 16777216
2022-12-08T05:17:05.937 INFO:teuthology.orchestra.run.smithi181.stdout:1 : device size 0xbfffe000 : using 0x43800000(1.1 GiB)
2022-12-08T05:17:05.937 INFO:teuthology.orchestra.run.smithi181.stdout:2 : device size 0x2625a0000 : using 0x10000(64 KiB)
2022-12-08T05:17:05.937 INFO:teuthology.orchestra.run.smithi181.stdout:RocksDBBlueFSVolumeSelector: wal_total:0, db_total:3060164198, slow_total:9728000000, db_avail:0
2022-12-08T05:17:05.938 INFO:teuthology.orchestra.run.smithi181.stdout:Usage matrix:
2022-12-08T05:17:05.938 INFO:teuthology.orchestra.run.smithi181.stdout:DEV/LEV     WAL         DB          SLOW        *           *           REAL        FILES
2022-12-08T05:17:05.938 INFO:teuthology.orchestra.run.smithi181.stdout:LOG         0 B         4 MiB       0 B         0 B         0 B         1.7 MiB     1
2022-12-08T05:17:05.939 INFO:teuthology.orchestra.run.smithi181.stdout:WAL         0 B         15 MiB      0 B         0 B         0 B         12 MiB      3
2022-12-08T05:17:05.939 INFO:teuthology.orchestra.run.smithi181.stdout:DB          0 B         1.0 GiB     0 B         0 B         0 B         1.0 GiB     24
2022-12-08T05:17:05.939 INFO:teuthology.orchestra.run.smithi181.stdout:SLOW        0 B         0 B         0 B         0 B         0 B         0 B         0
2022-12-08T05:17:05.939 INFO:teuthology.orchestra.run.smithi181.stdout:TOTALS      0 B         1.1 GiB     0 B         0 B         0 B         0 B         28
2022-12-08T05:17:05.940 INFO:teuthology.orchestra.run.smithi181.stdout:MAXIMUMS:
2022-12-08T05:17:05.940 INFO:teuthology.orchestra.run.smithi181.stdout:LOG         0 B         4 MiB       0 B         0 B         0 B         1.7 MiB
2022-12-08T05:17:05.940 INFO:teuthology.orchestra.run.smithi181.stdout:WAL         0 B         1.1 GiB     0 B         0 B         0 B         1.0 GiB
2022-12-08T05:17:05.940 INFO:teuthology.orchestra.run.smithi181.stdout:DB          0 B         1.4 GiB     0 B         0 B         0 B         1.3 GiB
2022-12-08T05:17:05.941 INFO:teuthology.orchestra.run.smithi181.stdout:SLOW        0 B         0 B         0 B         0 B         0 B         0 B
2022-12-08T05:17:05.941 INFO:teuthology.orchestra.run.smithi181.stdout:TOTALS      0 B         1.9 GiB     0 B         0 B         0 B         0 B
2022-12-08T05:17:05.941 INFO:teuthology.orchestra.run.smithi181.stdout:
2022-12-08T05:17:08.370 INFO:teuthology.orchestra.run.smithi181.stdout:==> rm -r bluestore.test_temp_dir
2022-12-08T05:17:08.665 INFO:teuthology.orchestra.run.smithi181.stdout:[  FAILED  ] ObjectStore/StoreTestSpecificAUSize.SpilloverTest/2, where GetParam() = "bluestore" (49483 ms)

Also:
/a/yuriw-2022-12-09_22:27:10-rados-main-distro-default-smithi/7110638

History

#1 Updated by Laura Flores over 1 year ago

Reproduced twice in the yuri3 run: /a/lflores-2022-12-08_19:37:51-rados-wip-yuri3-testing-2022-12-06-1211-distro-default-smithi/7108938

#2 Updated by Laura Flores over 1 year ago

  • Priority changed from Normal to High

#3 Updated by Laura Flores over 1 year ago

This test failed on the main baseline. I scheduled 10 more tests to run on the same main SHA: http://pulpito.front.sepia.ceph.com/lflores-2022-12-12_18:02:31-rados-main-distro-default-smithi/

#4 Updated by Laura Flores over 1 year ago

/a/yuriw-2022-12-07_15:47:33-rados-wip-yuri-testing-2022-12-06-1204-distro-default-smithi/7106783
/a/lflores-2022-12-08_19:26:32-rados-wip-yuri-testing-2022-12-06-1204-distro-default-smithi/7108918

#5 Updated by Igor Fedotov over 1 year ago

  • Status changed from New to In Progress

Looks like https://github.com/ceph/ceph/pull/47221 is a culprit. Not a big deal though - this is just a test case behaving differently with new settings.

#6 Updated by Laura Flores over 1 year ago

Thanks for taking care of it Igor!

#7 Updated by Laura Flores over 1 year ago

  • Assignee set to Igor Fedotov

#8 Updated by Igor Fedotov over 1 year ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 49392

#10 Updated by Igor Fedotov about 1 year ago

  • Status changed from Fix Under Review to Resolved

Also available in: Atom PDF