Project

General

Profile

Bug #20236

bluestore: ObjectStore/StoreTestSpecificAUSize.Many4KWritesNoCSumTest/2 failure

Added by Sage Weil over 4 years ago. Updated about 1 year ago.

Status:
New
Priority:
Low
Assignee:
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
nautilus,octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2017-06-09T03:37:28.314 INFO:teuthology.orchestra.run.smithi068.stderr:seeding object 0
2017-06-09T03:37:28.314 INFO:teuthology.orchestra.run.smithi068.stderr:Op 0
2017-06-09T03:37:28.314 INFO:teuthology.orchestra.run.smithi068.stderr:available_objects: 0 in_flight_objects: 1 total objects: 1 in_flight 1
2017-06-09T03:37:41.290 INFO:teuthology.orchestra.run.smithi068.stderr:Op 200
2017-06-09T03:37:41.291 INFO:teuthology.orchestra.run.smithi068.stderr:available_objects: 0 in_flight_objects: 1 total objects: 1 in_flight 1
2017-06-09T03:37:55.988 INFO:teuthology.orchestra.run.smithi068.stderr:Op 400
2017-06-09T03:37:55.989 INFO:teuthology.orchestra.run.smithi068.stderr:available_objects: 0 in_flight_objects: 1 total objects: 1 in_flight 1
2017-06-09T03:38:12.470 INFO:teuthology.orchestra.run.smithi068.stderr:Op 600
2017-06-09T03:38:12.470 INFO:teuthology.orchestra.run.smithi068.stderr:available_objects: 0 in_flight_objects: 1 total objects: 1 in_flight 1
2017-06-09T03:38:31.486 INFO:teuthology.orchestra.run.smithi068.stderr:Op 800
2017-06-09T03:38:31.486 INFO:teuthology.orchestra.run.smithi068.stderr:available_objects: 0 in_flight_objects: 1 total objects: 1 in_flight 1
2017-06-09T03:38:52.763 INFO:teuthology.orchestra.run.smithi068.stdout:/build/ceph-12.0.2-2533-g080b00b/src/test/objectstore/store_test.cc:5729: Failure
2017-06-09T03:38:52.764 INFO:teuthology.orchestra.run.smithi068.stdout:      Expected: res_stat.allocated
2017-06-09T03:38:52.764 INFO:teuthology.orchestra.run.smithi068.stdout:      Which is: 4259840
2017-06-09T03:38:52.764 INFO:teuthology.orchestra.run.smithi068.stdout:To be equal to: max_object
2017-06-09T03:38:52.764 INFO:teuthology.orchestra.run.smithi068.stdout:      Which is: 4194304

/a/yuriw-2017-06-08_19:40:51-rados-wip-yuri-testing2_2017_7_10-distro-basic-smithi/1271584

Related issues

Duplicated by bluestore - Bug #36422: ObjectStore/StoreTestSpecificAUSize.Many4KWritesTest/2 failure Duplicate 10/14/2018

History

#1 Updated by Greg Farnum over 4 years ago

  • Project changed from Ceph to RADOS
  • Component(RADOS) BlueStore added

#2 Updated by Sage Weil over 4 years ago

  • Status changed from 12 to Need More Info
  • Priority changed from Immediate to Urgent

hmm, passed 1000 iterations for me. Not sure what this might be.. Igor?

#3 Updated by Sage Weil over 4 years ago

  • Priority changed from Urgent to High

#4 Updated by Sage Weil over 4 years ago

  • Priority changed from High to Normal

#5 Updated by Greg Farnum almost 4 years ago

  • Project changed from RADOS to bluestore

#6 Updated by Kefu Chai almost 4 years ago

/a//kchai-2018-01-11_06:11:31-rados-wip-kefu-testing-2018-01-11-1036-distro-basic-mira/2058374/teuthology.log

2018-01-11T10:36:08.666 INFO:teuthology.orchestra.run.mira057.stdout:/build/ceph-13.0.0-4698-g955c807/src/test/objectstore/store_test.cc:5828: Failure
2018-01-11T10:36:08.666 INFO:teuthology.orchestra.run.mira057.stdout:      Expected: res_stat.allocated
2018-01-11T10:36:08.666 INFO:teuthology.orchestra.run.mira057.stdout:      Which is: 4259840
2018-01-11T10:36:08.666 INFO:teuthology.orchestra.run.mira057.stdout:To be equal to: max_object
2018-01-11T10:36:08.666 INFO:teuthology.orchestra.run.mira057.stdout:      Which is: 4194304
2018-01-11T10:36:09.742 INFO:teuthology.orchestra.run.mira057.stdout:==> rm -r bluestore.test_temp_dir
2018-01-11T10:36:09.761 INFO:teuthology.orchestra.run.mira057.stdout:[  FAILED  ] ObjectStore/StoreTestSpecificAUSize.Many4KWritesNoCSumTest/2, where GetParam() = "bluestore" (11688 ms)

exactly the same failure.

#7 Updated by Igor Fedotov over 3 years ago

  • Status changed from Need More Info to In Progress
  • Assignee set to Igor Fedotov

It looks like I have some insight on the root cause, got that much more frequently once having some blobs at DB only (which is my another task intended to optimize space overhead while keeping tiny objects).
So the general idea is that under some extent/blob layout BlueStore is unable to reuse existing allocated pextent within a blob and hence creates a new blob. Which leads to space usage higher than expected.

#8 Updated by Sage Weil almost 3 years ago

  • Duplicated by Bug #36422: ObjectStore/StoreTestSpecificAUSize.Many4KWritesTest/2 failure added

#9 Updated by Sage Weil almost 3 years ago

I haven't seen this in a while.. have you?

#10 Updated by Igor Fedotov almost 3 years ago

Sage Weil wrote:

I haven't seen this in a while.. have you?

Me too. Just gave this another try for both master and tiny write PR which I mentioned in comment #7.
Both are fine.
So closing for now.

#11 Updated by Igor Fedotov almost 3 years ago

  • Status changed from In Progress to Can't reproduce

#12 Updated by Sage Weil over 2 years ago

  • Status changed from Can't reproduce to 12

/a/sage-2019-04-24_22:14:24-rados-wip-sage2-testing-2019-04-24-0829-distro-basic-smithi/3890129

2019-04-25T03:45:42.427 INFO:teuthology.orchestra.run.smithi083.stderr:Op 800
2019-04-25T03:45:42.428 INFO:teuthology.orchestra.run.smithi083.stderr:available_objects: 0 in_flight_objects: 1 total objects: 1 in_flight 1
2019-04-25T03:45:45.922 INFO:teuthology.orchestra.run.smithi083.stdout:/build/ceph-15.0.0-654-g97653e1/src/test/objectstore/store_test.cc:6267: Failure
2019-04-25T03:45:45.922 INFO:teuthology.orchestra.run.smithi083.stdout:Expected: (res_stat.allocated) <= (max_object_size), actual: 4259840 vs 4194304
2019-04-25T03:45:46.436 INFO:teuthology.orchestra.run.smithi083.stdout:==> rm -r bluestore.test_temp_dir
2019-04-25T03:45:46.508 INFO:teuthology.orchestra.run.smithi083.stdout:[  FAILED  ] ObjectStore/StoreTestSpecificAUSize.Many4KWritesNoCSumTest/2, where GetParam() = "bluestore" (9806 ms)

#13 Updated by Neha Ojha about 2 years ago

  • Priority changed from Normal to Low

#14 Updated by Neha Ojha almost 2 years ago

  • Priority changed from Low to Normal

This appeared in nautilus

/a/yuriw-2019-11-12_15:30:19-rados-wip-yuri5-testing-2019-11-11-1520-nautilus-distro-basic-smithi/4499742/

#15 Updated by Neha Ojha almost 2 years ago

  • Priority changed from Normal to High

/a/yuriw-2019-11-08_20:50:44-rados-wip-yuri3-testing-2019-11-08-1221-nautilus-distro-basic-smithi/4485359/

#16 Updated by Neha Ojha almost 2 years ago

  • Backport set to nautilus

/a/nojha-2019-11-22_22:10:13-rados-wip-31657-nautilus-distro-basic-smithi/4534580/

#17 Updated by Patrick Donnelly almost 2 years ago

  • Status changed from 12 to New

#18 Updated by Sage Weil almost 2 years ago

  • Subject changed from bluestore: ObjectStore/StoreTestSpecificAUSize.Many4KWritesNoCSumTest/2 failure to nautilus: bluestore: ObjectStore/StoreTestSpecificAUSize.Many4KWritesNoCSumTest/2 failure

#19 Updated by Brad Hubbard over 1 year ago

  • Subject changed from nautilus: bluestore: ObjectStore/StoreTestSpecificAUSize.Many4KWritesNoCSumTest/2 failure to bluestore: ObjectStore/StoreTestSpecificAUSize.Many4KWritesNoCSumTest/2 failure

/a/yuriw-2020-04-28_21:58:13-rados-wip-yuri-testing-2020-04-24-1941-master-distro-basic-smithi/4995289

#20 Updated by Neha Ojha about 1 year ago

  • Priority changed from High to Low
  • Backport changed from nautilus to nautilus,octopus

Not critical from user standpoint.

Also available in: Atom PDF