Project

General

Profile

Bug #21312

occaionsal ObjectStore/StoreTestSpecificAUSize.Many4KWritesTest/2 failure

Added by Sage Weil almost 4 years ago. Updated about 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
luminous,mimic,nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2017-09-08T01:24:09.540 INFO:teuthology.orchestra.run.smithi057.stderr:seeding object 0
2017-09-08T01:24:09.540 INFO:teuthology.orchestra.run.smithi057.stderr:Op 0
2017-09-08T01:24:09.540 INFO:teuthology.orchestra.run.smithi057.stderr:available_objects: 0 in_flight_objects: 1 total objects: 1 in_flight 1
2017-09-08T01:24:21.989 INFO:teuthology.orchestra.run.smithi057.stderr:Op 200
2017-09-08T01:24:21.989 INFO:teuthology.orchestra.run.smithi057.stderr:available_objects: 0 in_flight_objects: 1 total objects: 1 in_flight 1
2017-09-08T01:24:36.791 INFO:teuthology.orchestra.run.smithi057.stderr:Op 400
2017-09-08T01:24:36.791 INFO:teuthology.orchestra.run.smithi057.stderr:available_objects: 0 in_flight_objects: 1 total objects: 1 in_flight 1
2017-09-08T01:24:54.123 INFO:teuthology.orchestra.run.smithi057.stderr:Op 600
2017-09-08T01:24:54.124 INFO:teuthology.orchestra.run.smithi057.stderr:available_objects: 0 in_flight_objects: 1 total objects: 1 in_flight 1
2017-09-08T01:25:14.965 INFO:teuthology.orchestra.run.smithi057.stderr:Op 800
2017-09-08T01:25:14.965 INFO:teuthology.orchestra.run.smithi057.stderr:available_objects: 0 in_flight_objects: 1 total objects: 1 in_flight 1
2017-09-08T01:25:40.504 INFO:teuthology.orchestra.run.smithi057.stdout:/build/ceph-12.2.0-80-gded7118/src/test/objectstore/store_test.cc:5714: Failure
2017-09-08T01:25:40.505 INFO:teuthology.orchestra.run.smithi057.stdout:      Expected: res_stat.allocated
2017-09-08T01:25:40.505 INFO:teuthology.orchestra.run.smithi057.stdout:      Which is: 4259840
2017-09-08T01:25:40.505 INFO:teuthology.orchestra.run.smithi057.stdout:To be equal to: max_object
2017-09-08T01:25:40.505 INFO:teuthology.orchestra.run.smithi057.stdout:      Which is: 4194304

/a/yuriw-2017-09-07_19:30:56-rados-wip-yuri-testing4-2017-09-07-1811-distro-basic-smithi/1607591

issue-log-presentation.bad_full - log snippet showing the issue (22.1 KB) Igor Fedotov, 04/08/2019 05:20 PM


Related issues

Copied to bluestore - Backport #39254: luminous: occaionsal ObjectStore/StoreTestSpecificAUSize.Many4KWritesTest/2 failure Resolved
Copied to bluestore - Backport #39255: mimic: occaionsal ObjectStore/StoreTestSpecificAUSize.Many4KWritesTest/2 failure Resolved
Copied to bluestore - Backport #39256: nautilus: occaionsal ObjectStore/StoreTestSpecificAUSize.Many4KWritesTest/2 failure Resolved

History

#1 Updated by Sage Weil almost 4 years ago

/a/sage-2017-09-10_02:44:16-rados-wip-yuri-testing4-2017-09-08-1631-distro-basic-smithi/1614995

#2 Updated by Kefu Chai over 3 years ago

http://pulpito.ceph.com/yuriw-2018-01-26_18:13:44-rados-wip_yuri_master_1.26.18-distro-basic-smithi/2112995/

2018-01-27T01:37:23.890 INFO:teuthology.orchestra.run.smithi182.stdout:/build/ceph-13.0.0-5148-g7fe6917/src/test/objectstore/store_test.cc:5813: Failure
2018-01-27T01:37:23.890 INFO:teuthology.orchestra.run.smithi182.stdout:      Expected: res_stat.allocated
2018-01-27T01:37:23.890 INFO:teuthology.orchestra.run.smithi182.stdout:      Which is: 4259840
2018-01-27T01:37:23.890 INFO:teuthology.orchestra.run.smithi182.stdout:To be equal to: max_object
2018-01-27T01:37:23.891 INFO:teuthology.orchestra.run.smithi182.stdout:      Which is: 4194304
2018-01-27T01:37:26.453 INFO:teuthology.orchestra.run.smithi182.stdout:==> rm -r bluestore.test_temp_dir
2018-01-27T01:37:26.469 INFO:teuthology.orchestra.run.smithi182.stdout:[  FAILED  ] ObjectStore/StoreTestSpecificAUSize.Many4KWritesTest/2, where GetParam() = "bluestore" (17468 ms)

#3 Updated by Kefu Chai over 3 years ago

  • Category set to Correctness/Safety
  • Component(RADOS) BlueStore added

#4 Updated by Kefu Chai over 3 years ago

actual_allocated_size - expected_allocated_size = 4259840 - 4194304 = 0x10000

#6 Updated by Nathan Cutler over 3 years ago

  • Status changed from 12 to 7
  • Assignee set to Igor Fedotov

#7 Updated by Kefu Chai over 3 years ago

  • Status changed from 7 to 12
2018-02-11T15:41:30.493 INFO:teuthology.orchestra.run.mira112.stdout:/build/ceph-13.0.1-1854-gca7168c/src/test/objectstore/store_test.cc:5826: Failure
2018-02-11T15:41:30.493 INFO:teuthology.orchestra.run.mira112.stdout:Expected: (res_stat.allocated) <= (max_object_size), actual: 4259840 vs 4194304
2018-02-11T15:41:31.495 INFO:teuthology.orchestra.run.mira112.stdout:==> rm -r bluestore.test_temp_dir
2018-02-11T15:41:31.513 INFO:teuthology.orchestra.run.mira112.stdout:[  FAILED  ] ObjectStore/StoreTestSpecificAUSize.Many4KWritesTest/2, where GetParam() = "bluestore" (13327 ms)

/a//kchai-2018-02-11_04:16:47-rados-wip-kefu-testing-2018-02-11-0959-distro-basic-mira/2181880

and the tested branch contains https://github.com/ceph/ceph/pull/20230

#8 Updated by Igor Fedotov over 2 years ago

  • Status changed from 12 to Fix Under Review
  • Pull request ID set to 27444

The root cause seem to be the lack of proper fault_range() call in do_write_small(). It might happen that write request that should reuse existing blob is unable to locate such a blob as the latter is present in unloaded shard.
This happens when write request in question has loffset lower than the one for the the extent containing blob-to-reuse. fault_range() bounds don't include desired extent and hence corresponding shard might remain unloaded.
See attached log snippet with some inline comments for example

#10 Updated by Kefu Chai over 2 years ago

  • Status changed from Fix Under Review to Pending Backport
  • Backport changed from luminous to luminous,mimic,nautilus

#11 Updated by Igor Fedotov over 2 years ago

  • Project changed from RADOS to bluestore
  • Category deleted (Correctness/Safety)

#12 Updated by Nathan Cutler over 2 years ago

  • Copied to Backport #39254: luminous: occaionsal ObjectStore/StoreTestSpecificAUSize.Many4KWritesTest/2 failure added

#13 Updated by Nathan Cutler over 2 years ago

  • Copied to Backport #39255: mimic: occaionsal ObjectStore/StoreTestSpecificAUSize.Many4KWritesTest/2 failure added

#14 Updated by Nathan Cutler over 2 years ago

  • Copied to Backport #39256: nautilus: occaionsal ObjectStore/StoreTestSpecificAUSize.Many4KWritesTest/2 failure added

#15 Updated by Igor Fedotov about 2 years ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF