Bug #11814
implicit erasure code crush ruleset is not validated
0%
Description
Context¶
- RHEL6
- Hammer 0.94.1
- 3 Mons
- 315 OSDs
Steps to reproduce¶
$ rm -fr out dev ; MON=1 OSD=3 ./vstart.sh -X -n -l mon osd $ ceph osd erasure-code-profile set myprofile plugin=lrc mapping=__DD__DD layers='[[ "_cDD_cDD", "" ],[ "cDDD____", "" ],[ "____cDDD", "" ],]' ruleset-steps='[ [ "choose", "datacenter", 3 ], [ "chooseleaf", "osd", 0] ]' $ ceph osd crush rule create-erasure myrule myprofile created ruleset myrule at 1 $ ceph osd getcrushmap > /tmp/c got crush map from osdmap epoch 13 $ ceph osd setcrushmap -i /tmp/c *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** Error EINVAL: Failed to parse crushmap: *** Caught signal (Segmentation fault) **
Related issues
Associated revisions
mon: test the crush ruleset when creating a pool
We want to fix the following scenario:
- an erasure code plugin (or another part of the code) creates a
ruleset - the ruleset crashes during mapping (for whatever reason)
- ceph osd pool create uses the bugous ruleset
- the monitors try to do mapping a crash
Having a bugous ruleset in the crush map is very difficult prevent. The
catastrophic event of using it with a newly created pool can however be
prevented by calling the CrushTester just before creating the pool and
after all implicit or explicit crush ruleset creation happened.
http://tracker.ceph.com/issues/11814 Fixes: #11814
Signed-off-by: Loic Dachary <ldachary@redhat.com>
mon: test the crush ruleset when creating a pool
We want to fix the following scenario:
- an erasure code plugin (or another part of the code) creates a
ruleset - the ruleset crashes during mapping (for whatever reason)
- ceph osd pool create uses the bugous ruleset
- the monitors try to do mapping a crash
Having a bugous ruleset in the crush map is very difficult prevent. The
catastrophic event of using it with a newly created pool can however be
prevented by calling the CrushTester just before creating the pool and
after all implicit or explicit crush ruleset creation happened.
http://tracker.ceph.com/issues/11814 Fixes: #11814
Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit f1e86be589803596e86acc964ac5c5c03b4038d8)
Conflicts:
src/test/mon/osd-crush.sh
removed the run_mon because in hammer it is shared
between all tests
src/mon/OSDMonitor.cc
prepare_new_pool changed stringstream to *ostream
History
#1 Updated by Loïc Dachary almost 9 years ago
- Subject changed from New EC pool crashed the mons to implicit erasure code crush ruleset is not validated
- Status changed from New to 12
- Assignee set to Loïc Dachary
- Priority changed from Normal to High
- Backport set to hammer
The crush ruleset created as a side effect of an erasure coded pool creation is not validated via crushtool, but it should be. In the same fashion a new crushmap being injected via ceph osd crush currently is.
#2 Updated by Loïc Dachary almost 9 years ago
- Description updated (diff)
#3 Updated by Loïc Dachary almost 9 years ago
- Description updated (diff)
#4 Updated by Loïc Dachary almost 9 years ago
- Status changed from 12 to Fix Under Review
#5 Updated by Loïc Dachary almost 9 years ago
- Priority changed from High to Urgent
#6 Updated by Loïc Dachary over 8 years ago
I believe this is fixed in hammer v0.94.2 with https://github.com/ceph/ceph/pull/4936 and various other patches that make it impossible to run into this specific situation. There were a few window of opportunities prior to v0.94.2.
The steps to reproduce the issue listed in the description do not actually work on v0.94.1:
loic@fold:~/software/ceph/ceph/src$ profile=k8m4isa loic@fold:~/software/ceph/ceph/src$ ceph osd erasure-code-profile set k8m4isa plugin=isa k=8 m=4 technique=reed_sol_van ruleset-root=bigbang ruleset-failure-domain=host loic@fold:~/software/ceph/ceph/src$ pool=castor-ec-isa loic@fold:~/software/ceph/ceph/src$ ceph osd pool create $pool 4096 4096 erasure k8m4isa castor-ec-isa Error ENOENT: specified ruleset castor-ec-isa doesn't exist loic@fold:~/software/ceph/ceph/src$ ceph --version ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
I suspect the situation was created with a different combo but it's difficult to figure it out.
#7 Updated by Dan van der Ster over 8 years ago
Hi Loic,
The reproducing steps were something like:
ceph osd crush add-bucket bigbang ceph osd erasure-code-profile set k8m4isa plugin=isa k=8 m=4 technique=reed_sol_van ruleset-root=bigbang ruleset-failure-domain=host ceph osd crush rm bigbang ceph osd pool create castor-ec-isa 4096 4096 erasure k8m4isa
The mon should crash after that last pool create.
Cheers, Dan
#8 Updated by Loïc Dachary over 8 years ago
- Description updated (diff)
#9 Updated by Loïc Dachary over 8 years ago
That can't happen (on master). But I updated with the description with another scenario that fails and I think the right fix is to verify the ruleset right before associating it with the pool.
loic@fold:~/software/ceph/ceph/src$ ceph osd crush add-bucket bigbang datacenter added bucket bigbang type datacenter to crush map loic@fold:~/software/ceph/ceph/src$ ceph osd erasure-code-profile set k8m4isa plugin=isa k=8 m=4 technique=reed_sol_van ruleset-root=bigbang ruleset-failure-domain=host loic@fold:~/software/ceph/ceph/src$ ceph osd crush rm bigbang removed item id -3 name 'bigbang' from crush map loic@fold:~/software/ceph/ceph/src$ ceph osd pool create castor-ec-isa 4096 4096 erasure k8m4isa Error ENOENT: root item bigbang does not exist loic@fold:~/software/ceph/ceph/src$ ceph --version ceph version 9.0.1-1494-g8fc0496 (8fc049664bc798432e1750da86b1f216f85a842d)
#10 Updated by Loïc Dachary over 8 years ago
- Description updated (diff)
#11 Updated by Loïc Dachary over 8 years ago
- Description updated (diff)
#12 Updated by Kefu Chai over 8 years ago
- Status changed from Fix Under Review to Pending Backport
#13 Updated by Loïc Dachary over 8 years ago
- Status changed from Pending Backport to Resolved
#14 Updated by Loïc Dachary over 8 years ago
Kefu added the script src/tools/ceph-monstore-update-crush.sh which is packaged with ceph-test to recover a monitor with a bugous crushmap.
#15 Updated by Ken Dreyer over 8 years ago
Is src/tools/ceph-monstore-update-crush.sh
something that only developers would run, or something that we ever expect users to run?
#16 Updated by Kefu Chai about 8 years ago
ken, i expect that users to use this tool,