Feature #9420
erasure-code: tools and archive to check for non regression of encoding
100%
Description
to ensure the encoding of the data does not change over time. This must be made to check that a given plugin with a given set of parameters is able to read/write the same data after a patch and avoid regressions that could corrupt the content of a pool.
Related issues
Associated revisions
erasure-code: store and compare encoded contents
Introduce ceph_erasure_code_non_regression to check and compare how an
erasure code plugin encodes and decodes content with a given set of
parameters. For instance:
./ceph_erasure_code_non_regression \
--plugin jerasure \
--parameter technique=reed_sol_van \
--parameter k=2 \
--parameter m=2 \
--stripe-width 3181 \
--create \
--check
Will create an encoded object (--create) and store it into a directory
along with the chunks, one chunk per file. The directory name is derived
from the parameters. The content of the object is a random pattern of 31
bytes repeated to fill the object size specified with --stripe-width.
The check function (--check) reads the object back from the file,
encodes it and compares the result with the content of the chunks read
from the files. It also attempts recover from one or two erasures.
Chunks encoded by a given version of Ceph are expected to be encoded
exactly in the same way by all Ceph versions going forward.
http://tracker.ceph.com/issues/9420 Refs: #9420
Signed-off-by: Loic Dachary <loic-201408@dachary.org>
erasure-code: workunit to check for encoding regression
Clone the archive of encoded objects and decode all archived objects, up
to and including the current ceph version.
http://tracker.ceph.com/issues/9420 Refs: #9420
Signed-off-by: Loic Dachary <loic-201408@dachary.org>
erasure-code: store and compare encoded contents
Introduce ceph_erasure_code_non_regression to check and compare how an
erasure code plugin encodes and decodes content with a given set of
parameters. For instance:
./ceph_erasure_code_non_regression \
--plugin jerasure \
--parameter technique=reed_sol_van \
--parameter k=2 \
--parameter m=2 \
--stripe-width 3181 \
--create \
--check
Will create an encoded object (--create) and store it into a directory
along with the chunks, one chunk per file. The directory name is derived
from the parameters. The content of the object is a random pattern of 31
bytes repeated to fill the object size specified with --stripe-width.
The check function (--check) reads the object back from the file,
encodes it and compares the result with the content of the chunks read
from the files. It also attempts recover from one or two erasures.
Chunks encoded by a given version of Ceph are expected to be encoded
exactly in the same way by all Ceph versions going forward.
http://tracker.ceph.com/issues/9420 Refs: #9420
Signed-off-by: Loic Dachary <loic-201408@dachary.org>
(cherry picked from commit f5901303dbf50e9d08f2f1e510a1936a20037909)
erasure-code: workunit to check for encoding regression
Clone the archive of encoded objects and decode all archived objects, up
to and including the current ceph version.
http://tracker.ceph.com/issues/9420 Refs: #9420
Signed-off-by: Loic Dachary <loic-201408@dachary.org>
(cherry picked from commit 7638b15f23976c3265cf766e16cf93af1a7e0091)
erasure-code: store and compare encoded contents
Introduce ceph_erasure_code_non_regression to check and compare how an
erasure code plugin encodes and decodes content with a given set of
parameters. For instance:
./ceph_erasure_code_non_regression \
--plugin jerasure \
--parameter technique=reed_sol_van \
--parameter k=2 \
--parameter m=2 \
--stripe-width 3181 \
--create \
--check
Will create an encoded object (--create) and store it into a directory
along with the chunks, one chunk per file. The directory name is derived
from the parameters. The content of the object is a random pattern of 31
bytes repeated to fill the object size specified with --stripe-width.
The check function (--check) reads the object back from the file,
encodes it and compares the result with the content of the chunks read
from the files. It also attempts recover from one or two erasures.
Chunks encoded by a given version of Ceph are expected to be encoded
exactly in the same way by all Ceph versions going forward.
http://tracker.ceph.com/issues/9420 Refs: #9420
Signed-off-by: Loic Dachary <loic-201408@dachary.org>
erasure-code: workunit to check for encoding regression
Clone the archive of encoded objects and decode all archived objects, up
to and including the current ceph version.
http://tracker.ceph.com/issues/9420 Refs: #9420
Signed-off-by: Loic Dachary <loic-201408@dachary.org>
History
#1 Updated by Loïc Dachary over 8 years ago
- Description updated (diff)
#2 Updated by Loïc Dachary over 8 years ago
- Status changed from 12 to In Progress
#3 Updated by Loïc Dachary over 8 years ago
- Description updated (diff)
- Status changed from In Progress to Fix Under Review
#4 Updated by Loïc Dachary over 8 years ago
- % Done changed from 0 to 50
#5 Updated by Loïc Dachary over 8 years ago
- Description updated (diff)
#6 Updated by Loïc Dachary over 8 years ago
- Subject changed from erasure-code: create a data corpus to erasure-code: tools and archive to check for non regression of encoding
#7 Updated by Loïc Dachary over 8 years ago
- Created the repository https://github.com/ceph/ceph-erasure-code-corpus
- Asked Sandon if having such a repository checked out every make check by gitbuilder would be a problem but he said it should be fine
#8 Updated by Loïc Dachary over 8 years ago
- Description updated (diff)
- % Done changed from 50 to 80
#9 Updated by Loïc Dachary over 8 years ago
- Description updated (diff)
- Status changed from Fix Under Review to 7
#10 Updated by Loïc Dachary over 8 years ago
- Status changed from 7 to Fix Under Review
- % Done changed from 80 to 90
The gitbuilders have been updated, it is ready for review.
#11 Updated by Loïc Dachary over 8 years ago
- Description updated (diff)
#12 Updated by Loïc Dachary over 8 years ago
- Backport set to giant, firefly
#13 Updated by Loïc Dachary over 8 years ago
- Tracker changed from Fix to Feature
- Target version set to 0.89
#14 Updated by Samuel Just over 8 years ago
- Status changed from Fix Under Review to Resolved
- Backport deleted (
giant, firefly)
I don't think this needs to be backported.
#15 Updated by Loïc Dachary over 8 years ago
- Status changed from Resolved to Pending Backport
- Backport set to firefly, giant
#16 Updated by Loïc Dachary over 8 years ago
- Description updated (diff)
#17 Updated by Loïc Dachary over 8 years ago
the backport is needed to generate the content of https://github.com/ceph/ceph-erasure-code-corpus/tree/master/v0.80.5-231-g9f7aa29
#18 Updated by Loïc Dachary over 8 years ago
- Description updated (diff)
#19 Updated by Loïc Dachary over 8 years ago
- Description updated (diff)
#20 Updated by Loïc Dachary over 8 years ago
- Status changed from Pending Backport to Fix Under Review
the non regression tests pass on an armv7 with the jerasure neon variant.
#21 Updated by Loïc Dachary over 8 years ago
- Description updated (diff)
- Status changed from Fix Under Review to Resolved
- % Done changed from 90 to 100