Bug #36412
closedceph-objectstore-tool import after pg splits which will lost objects
0%
Description
Hi, i have a test cluster, doing the follow steps, the pool is erasure k:m=3:1
step 1: export pg 2.f from osd.2, origin pg 2.f stored on [0,5,2,1]
2018-10-09T18:35:49.560 INFO:teuthology.orchestra.run.mira036.stderr:Exporting 2.fs2 info 2.fs2( v 211'52 (32'0,211'52] local-lis/les=77/78 n=9 ec=28/28 lis/c 77/28 les/c/f 78/29/0 77/77/28)
2018-10-09T18:35:49.564 INFO:teuthology.orchestra.run.mira036.stderr:Read 2#2:f79c6653:::rb.radosmodel.mira036.xsky.com23787-19 ooo:4d#
2018-10-09T18:35:49.569 INFO:teuthology.orchestra.run.mira036.stderr:Read 2#2:f79c6653:::rb.radosmodel.mira036.xsky.com23787-19 ooo:6d#
2018-10-09T18:35:49.574 INFO:teuthology.orchestra.run.mira036.stderr:Read 2#2:f79c6653:::rb.radosmodel.mira036.xsky.com23787-19 ooo:head#
2018-10-09T18:35:49.583 INFO:teuthology.orchestra.run.mira036.stderr:Read 2#2:f9c3f4f0:::rb.radosmodel.mira036.xsky.com23787-13 ooo:1c#
2018-10-09T18:35:49.588 INFO:teuthology.orchestra.run.mira036.stderr:Read 2#2:f9c3f4f0:::rb.radosmodel.mira036.xsky.com23787-13 ooo:47#
2018-10-09T18:35:49.592 INFO:teuthology.orchestra.run.mira036.stderr:Read 2#2:f9c3f4f0:::rb.radosmodel.mira036.xsky.com23787-13 ooo:69#
2018-10-09T18:35:49.605 INFO:teuthology.orchestra.run.mira036.stderr:Read 2#2:f9c3f4f0:::rb.radosmodel.mira036.xsky.com23787-13 ooo:6d#
2018-10-09T18:35:49.610 INFO:teuthology.orchestra.run.mira036.stderr:Read 2#2:f9c3f4f0:::rb.radosmodel.mira036.xsky.com23787-13 ooo:82#
2018-10-09T18:35:49.620 INFO:teuthology.orchestra.run.mira036.stderr:Read 2#2:f9c3f4f0:::rb.radosmodel.mira036.xsky.com23787-13 ooo:head#
2018-10-09T18:35:49.625 INFO:teuthology.orchestra.run.mira036.stderr:Export successful
step 2: set pool pg_num from 16 to 36, which will casue pg 2.f split to 2.1f
step 3: stop osd.3 and import is to osd.3
2018-10-09T18:36:43.470 INFO:teuthology.orchestra.run.mira036.stdout:Importing pgid 2.fs2
2018-10-09T18:36:43.522 INFO:tasks.ceph.osd.6.mira031.stderr:2018-10-09 18:38:11.001 7f6a10212700 -1 received signal: Hangup from PID: 22213 task name: /usr/bin/python /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 UID: 0
2018-10-09T18:36:43.566 INFO:teuthology.orchestra.run.mira036.stdout:Write 2#2:f79c6653:::rb.radosmodel.mira036.xsky.com23787-19 ooo:4d#
2018-10-09T18:36:43.574 INFO:teuthology.orchestra.run.mira031:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops'
2018-10-09T18:36:43.576 INFO:teuthology.orchestra.run.mira036.stdout:Write 2#2:f79c6653:::rb.radosmodel.mira036.xsky.com23787-19 ooo:6d#
2018-10-09T18:36:43.584 INFO:teuthology.orchestra.run.mira036.stdout:Write 2#2:f79c6653:::rb.radosmodel.mira036.xsky.com23787-19 ooo:head#
2018-10-09T18:36:43.584 INFO:teuthology.orchestra.run.mira036.stdout:snapset 6d=[6d,69,66,64,62,60,58,55,50,4d,4a]:
{4d=[4d,4b,4a,49,47,46,43],6d=[6d,69,66,64,62,60,58,55,50]}
2018-10-09T18:36:43.606 INFO:teuthology.orchestra.run.mira036.stdout:write_pg epoch 311 info 2.fs2( v 211'52 (32'0,211'52] local-lis/les=77/78 n=9 ec=28/28 lis/c 77/28 les/c/f 78/29/0 219/295/28)
2018-10-09T18:36:43.614 INFO:teuthology.orchestra.run.mira036.stdout:Import successful
2018-10-09T18:36:52.835 INFO:teuthology.orchestra.run.mira036.stderr:WARNING: Split occurred, some objects may be ignored
2018-10-09T18:36:52.835 INFO:teuthology.orchestra.run.mira036.stderr:check_new_interval interval(77-218 up [0,5,2,2147483647](0) acting [0,5,2,2147483647](0)) up_thru 209 up_from 15 last_epoch_clean 29 interval(77-218 up [0,5,2,2147483647](0) acting [0,5,2,2147483647](0) maybe_went_rw) : primary up 15-209 includes interval
2018-10-09T18:36:52.835 INFO:teuthology.orchestra.run.mira036.stderr:check_new_interval interval(219-294 up [0,5,2147483647,2147483647](0) acting [0,5,2147483647,2147483647](0)) : acting set is too small
2018-10-09T18:36:52.836 INFO:teuthology.orchestra.run.mira036.stderr:Skipping object '2#2:f9c3f4f0:::rb.radosmodel.mira036.xsky.com23787-13 ooo:1c#' which belongs in pg 2.1f
2018-10-09T18:36:52.836 INFO:teuthology.orchestra.run.mira036.stderr:Skipping object '2#2:f9c3f4f0:::rb.radosmodel.mira036.xsky.com23787-13 ooo:47#' which belongs in pg 2.1f
2018-10-09T18:36:52.836 INFO:teuthology.orchestra.run.mira036.stderr:Skipping object '2#2:f9c3f4f0:::rb.radosmodel.mira036.xsky.com23787-13 ooo:69#' which belongs in pg 2.1f
2018-10-09T18:36:52.836 INFO:teuthology.orchestra.run.mira036.stderr:Skipping object '2#2:f9c3f4f0:::rb.radosmodel.mira036.xsky.com23787-13 ooo:6d#' which belongs in pg 2.1f
2018-10-09T18:36:52.837 INFO:teuthology.orchestra.run.mira036.stderr:Skipping object '2#2:f9c3f4f0:::rb.radosmodel.mira036.xsky.com23787-13 ooo:82#' which belongs in pg 2.1f
2018-10-09T18:36:52.837 INFO:teuthology.orchestra.run.mira036.stderr:Skipping object '2#2:f9c3f4f0:::rb.radosmodel.mira036.xsky.com23787-13 ooo:head#' which belongs in pg 2.1f
2018-10-09T18:36:52.837 INFO:teuthology.orchestra.run.mira036:Running: 'rm -f /home/ubuntu/cephtest/ceph.data/exp.2.fs2.2'
when pg 2.1f peering, it lost objects prefix with "rb.radosmodel.mira036.xsky.com23787-13 ooo "
I think the problem is, when we import pg 2.f to osd.3, we should also import these objects belongs to 2.1f bc there isn't enough acting set in the interval [219-294].
If we just skip these objects, it will not found on osd.3, osd.2, and osd.1.
Updated by huang jun over 5 years ago
@David Zafman Do you have time to take a look?
Updated by Greg Farnum over 5 years ago
- Project changed from Ceph to RADOS
- Category deleted (
OSD)
Updated by Greg Farnum over 5 years ago
Did you try importing 2.1f with that original 2.f PG dump?
Updated by David Zafman over 5 years ago
As Greg pointed out you would use the --pgid 2.1f option with --op import to get the objects that split into that pg.
In the latest master branch you don't have to do that. After a straight import when the OSD joins the cluster it will perform the split because it will recognize the pg as a pre-split one.
Updated by David Zafman over 5 years ago
- Related to Bug #35955: ceph-objectstore-tool past_intervals broken added
Updated by David Zafman over 5 years ago
- Status changed from New to Closed
The original code is operating as intended when issuing this warning:
WARNING: Split occurred, some objects may be ignored
Already fixed by http://tracker.ceph.com/issues/35955
Updated by huang jun over 5 years ago
Greg Farnum wrote:
Did you try importing 2.1f with that original 2.f PG dump?
No, the test script doesn't import the child pg.
Updated by huang jun over 5 years ago
David Zafman wrote:
The original code is operating as intended when issuing this warning:
WARNING: Split occurred, some objects may be ignored
Already fixed by http://tracker.ceph.com/issues/35955
We don't have the assert backtrace described in #35955.
As you noted before, the newest master will not to import child pg explictly,
but in the case i showed, the objects belong to child pg 2.1f will lost because
osd.1 and osd.2(export osd.3) doesn't have these objects, and our erasure code profile is k:m=3:1,
so it can not recovered from other two osds (osd.5 osd.0).