Project

General

Profile

Feature #9420

erasure-code: tools and archive to check for non regression of encoding

Added by Loic Dachary almost 6 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
% Done:

100%

Source:
other
Tags:
Backport:
firefly, giant
Reviewed:
Affected Versions:
Pull request ID:

Description

to ensure the encoding of the data does not change over time. This must be made to check that a given plugin with a given set of parameters is able to read/write the same data after a patch and avoid regressions that could corrupt the content of a pool.


Related issues

Related to Ceph - Feature #9720: erasure-code: non regression should test jerasure variants Resolved 10/10/2014

Associated revisions

Revision f5901303 (diff)
Added by Loic Dachary over 5 years ago

erasure-code: store and compare encoded contents

Introduce ceph_erasure_code_non_regression to check and compare how an
erasure code plugin encodes and decodes content with a given set of
parameters. For instance:

./ceph_erasure_code_non_regression \
--plugin jerasure \
--parameter technique=reed_sol_van \
--parameter k=2 \
--parameter m=2 \
--stripe-width 3181 \
--create \
--check

Will create an encoded object (--create) and store it into a directory
along with the chunks, one chunk per file. The directory name is derived
from the parameters. The content of the object is a random pattern of 31
bytes repeated to fill the object size specified with --stripe-width.

The check function (--check) reads the object back from the file,
encodes it and compares the result with the content of the chunks read
from the files. It also attempts recover from one or two erasures.

Chunks encoded by a given version of Ceph are expected to be encoded
exactly in the same way by all Ceph versions going forward.

http://tracker.ceph.com/issues/9420 Refs: #9420

Signed-off-by: Loic Dachary <>

Revision 7638b15f (diff)
Added by Loic Dachary over 5 years ago

erasure-code: workunit to check for encoding regression

Clone the archive of encoded objects and decode all archived objects, up
to and including the current ceph version.

http://tracker.ceph.com/issues/9420 Refs: #9420

Signed-off-by: Loic Dachary <>

Revision 1b7fc7e4 (diff)
Added by Loic Dachary over 5 years ago

erasure-code: store and compare encoded contents

Introduce ceph_erasure_code_non_regression to check and compare how an
erasure code plugin encodes and decodes content with a given set of
parameters. For instance:

./ceph_erasure_code_non_regression \
--plugin jerasure \
--parameter technique=reed_sol_van \
--parameter k=2 \
--parameter m=2 \
--stripe-width 3181 \
--create \
--check

Will create an encoded object (--create) and store it into a directory
along with the chunks, one chunk per file. The directory name is derived
from the parameters. The content of the object is a random pattern of 31
bytes repeated to fill the object size specified with --stripe-width.

The check function (--check) reads the object back from the file,
encodes it and compares the result with the content of the chunks read
from the files. It also attempts recover from one or two erasures.

Chunks encoded by a given version of Ceph are expected to be encoded
exactly in the same way by all Ceph versions going forward.

http://tracker.ceph.com/issues/9420 Refs: #9420

Signed-off-by: Loic Dachary <>
(cherry picked from commit f5901303dbf50e9d08f2f1e510a1936a20037909)

Revision 1def82d5 (diff)
Added by Loic Dachary over 5 years ago

erasure-code: workunit to check for encoding regression

Clone the archive of encoded objects and decode all archived objects, up
to and including the current ceph version.

http://tracker.ceph.com/issues/9420 Refs: #9420

Signed-off-by: Loic Dachary <>
(cherry picked from commit 7638b15f23976c3265cf766e16cf93af1a7e0091)

Revision 4f435870 (diff)
Added by Loic Dachary over 5 years ago

erasure-code: store and compare encoded contents

Introduce ceph_erasure_code_non_regression to check and compare how an
erasure code plugin encodes and decodes content with a given set of
parameters. For instance:

./ceph_erasure_code_non_regression \
--plugin jerasure \
--parameter technique=reed_sol_van \
--parameter k=2 \
--parameter m=2 \
--stripe-width 3181 \
--create \
--check

Will create an encoded object (--create) and store it into a directory
along with the chunks, one chunk per file. The directory name is derived
from the parameters. The content of the object is a random pattern of 31
bytes repeated to fill the object size specified with --stripe-width.

The check function (--check) reads the object back from the file,
encodes it and compares the result with the content of the chunks read
from the files. It also attempts recover from one or two erasures.

Chunks encoded by a given version of Ceph are expected to be encoded
exactly in the same way by all Ceph versions going forward.

http://tracker.ceph.com/issues/9420 Refs: #9420

Signed-off-by: Loic Dachary <>

Revision 8d3d6bf5 (diff)
Added by Loic Dachary over 5 years ago

erasure-code: workunit to check for encoding regression

Clone the archive of encoded objects and decode all archived objects, up
to and including the current ceph version.

http://tracker.ceph.com/issues/9420 Refs: #9420

Signed-off-by: Loic Dachary <>

History

#1 Updated by Loic Dachary almost 6 years ago

  • Description updated (diff)

#2 Updated by Loic Dachary almost 6 years ago

  • Status changed from 12 to In Progress

#3 Updated by Loic Dachary almost 6 years ago

  • Description updated (diff)
  • Status changed from In Progress to Fix Under Review

#4 Updated by Loic Dachary almost 6 years ago

  • % Done changed from 0 to 50

#5 Updated by Loic Dachary almost 6 years ago

  • Description updated (diff)

#6 Updated by Loic Dachary almost 6 years ago

  • Subject changed from erasure-code: create a data corpus to erasure-code: tools and archive to check for non regression of encoding

#7 Updated by Loic Dachary almost 6 years ago

  • Asked Sandon if having such a repository checked out every make check by gitbuilder would be a problem but he said it should be fine

#8 Updated by Loic Dachary almost 6 years ago

  • Description updated (diff)
  • % Done changed from 50 to 80

#9 Updated by Loic Dachary almost 6 years ago

  • Description updated (diff)
  • Status changed from Fix Under Review to 7

#10 Updated by Loic Dachary almost 6 years ago

  • Status changed from 7 to Fix Under Review
  • % Done changed from 80 to 90

The gitbuilders have been updated, it is ready for review.

#11 Updated by Loic Dachary almost 6 years ago

  • Description updated (diff)

#12 Updated by Loic Dachary almost 6 years ago

  • Backport set to giant, firefly

#13 Updated by Loic Dachary almost 6 years ago

  • Tracker changed from Fix to Feature
  • Target version set to 0.89

#14 Updated by Samuel Just over 5 years ago

  • Status changed from Fix Under Review to Resolved
  • Backport deleted (giant, firefly)

I don't think this needs to be backported.

#15 Updated by Loic Dachary over 5 years ago

  • Status changed from Resolved to Pending Backport
  • Backport set to firefly, giant

#16 Updated by Loic Dachary over 5 years ago

  • Description updated (diff)

#18 Updated by Loic Dachary over 5 years ago

  • Description updated (diff)

#19 Updated by Loic Dachary over 5 years ago

  • Description updated (diff)

#20 Updated by Loic Dachary over 5 years ago

  • Status changed from Pending Backport to Fix Under Review

the non regression tests pass on an armv7 with the jerasure neon variant.

#21 Updated by Loic Dachary over 5 years ago

  • Description updated (diff)
  • Status changed from Fix Under Review to Resolved
  • % Done changed from 90 to 100

Also available in: Atom PDF