Bug #13878
closed
test/librados/tier.cc doesn't completely clean up EC pools
Added by Dan Mick over 8 years ago.
Updated almost 8 years ago.
Backport:
hammer,infernalis
Description
Creating an EC pool also creates a custom crush rule for it; tier.cc creates and destroys the pool, but leaves behind the crush rule; with repeated runs, this can get to overflowing the crush rule table (256 entries) fairly quickly and cause a confusing failure, including a backtrace on a Lock of a null mutex (probably a second bug).
Make destroy_one_ec_pool* also remove the crush rule.
Another side effect of having too many ruleset is that injecting the crushmap into the mon will timeout. The crushmap validation happens in the mon by calling crushtool. If it does not return within mon_lease seconds, it is interrupted. If it was not interrupted the mon would be perceived as unresponsive by the other mons and that would trigger an election, which is not the desired outcome.
- Status changed from New to Fix Under Review
- Assignee set to Dan Mick
- Assignee changed from Dan Mick to Loïc Dachary
- Status changed from Fix Under Review to Resolved
- Status changed from Resolved to Pending Backport
- Backport set to infernalis
- Copied to Backport #15040: infernalis: test/librados/tier.cc doesn't completely clean up EC pools added
- Description updated (diff)
from dmick: mon calls crushtool with a timeout of 5s (mon_lease), and kills it if it takes longer. Because of accumulated EC-pool crush rules due to #13878, crushtool takes longer than it normally would to run; long enough that it violates the 5s timeout, and so the mon kills it; ultimately this shows up as the pool create failure.
- Backport changed from infernalis to hammer,infernalis
Added hammer backport based on IRC discussion with Yuri and Dan:
(10:53:33 PM) smithfarm: yuriw: loicd set the backport to infernalis (only) here: http://tracker.ceph.com/issues/13878#note-5
(10:54:46 PM) yuriw: smithfarm - i saw and wondering if hammer needs to be set too
(11:00:13 PM) yuriw: smithfarm - i thought that dupes should be fixed by it:
(11:00:13 PM) yuriw: http://tracker.ceph.com/issues/14927
(11:00:13 PM) yuriw: http://tracker.ceph.com/issues/14935
(11:05:44 PM) smithfarm: yuri: I don't know. dmick, do you think https://github.com/dachary/ceph/commit/04b4795f81c15bfcb62ba5807745470ce0e5e949 needs to be backported to hammer as well as infernalis? The issue is triggered by the hammer-x-jewel as well as the infernalis-x-jewel upgrade suites.
(11:06:05 PM) smithfarm: dmick: loicd staged the infernalis backport, but not hammer
(11:06:21 PM) dmick: seems to me it ought to go everywhere we're running tests, yes
- Copied to Backport #15051: hammer: test/librados/tier.cc doesn't completely clean up EC pools added
- Status changed from Pending Backport to Resolved
Also available in: Atom
PDF