Actions
Bug #16127
closedOSDMonitor: drop pg temps from not the current primary
% Done:
0%
Source:
other
Tags:
Backport:
jewel,hammer
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Otherwise, the vagaries of pg->osd->mon->other mon message ordering could result in a previous interval pg temp request being processed after a current interval request causing the pg to get stuck.
sjust@teuthology:/a/samuelj-2016-06-01_11:28:31-rados-wip-sam-testing-distro-basic-smithi/228382/remote
<sjusthm> sage: hmm<sjusthm> pg temps from two osds raced
<sjusthm> on the same osd
<sjusthm> osd.3 sent a request for [0,3] at pg_epoch 16
<sjusthm> which went out at osd epoch 16
<sjusthm> and then again at 17
<sjusthm> (before the pg processed the map)
<sjusthm> however
<sjusthm> 17 also changed the acting set to [0,3]
- kefu has quit (Quit: My Mac has gone to sleep. ZZZzzz…)
<sjusthm> and the new primary requested an empty temp mapping
<sjusthm> also at pg/osd epoch 17
<sjusthm> the mons processed the one from the new primary
<sjusthm> and then the stale one from the old primary
<sjusthm> resulting in the acting set remaining at [0,3] and the pg being stuck
<sjusthm> the osd epoch part seems to bge a red herring since it's not used in the OSDMonitor
<sjusthm> I think we need to include with each mapping the interval start epoch
<sjusthm> and remember that in the OSDMonitor
<sjusthm> that would allow us to dicard pg temp mappings based on previous intervals
<sjusthm> hmm
<sjusthm> wouldn't actually help here
<sjusthm> since the empty mapping wouldn't be remembered explicitely - sahid has quit (Quit: Lost terminal)
<sjusthm> maybe we can just ignore if it comes from not the current primary from the mon's point of view? - rzarzynski has quit (Quit: This computer has gone to sleep)
- dgurtner_ has quit (Ping timeout: 480 seconds)
- swami2 has quit (Ping timeout: 480 seconds)
- rzarzynski (~rzarzynsk@80.87.33.6) has joined #ceph-devel
<sjusthm> I guess that should be safe
<sjusthm> can I do that in preprocess?
Updated by John Coyle almost 8 years ago
Maybe unrelated but I'm curious if this issue would cause cephtool-test-mon.sh to get stuck at:
../qa/workunits/cephtool/test.sh:1401: test_mon_pg: ceph osd primary-affinity osd.0 1- DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2016-06-10 17:36:58.182222 7fbb90cfeab0 -1 WARNING: the following dangerous and experimental features are enabled: *
2016-06-10 17:36:58.182267 7fbb90cfeab0 0 lockdep start
2016-06-10 17:36:58.187814 7fbb90cfeab0 -1 WARNING: the following dangerous and experimental features are enabled: *
set osd.0 primary-affinity to 1 (8655362)
2016-06-10 17:36:58.370347 7fbb90cfeab0 0 lockdep stop
../qa/workunits/cephtool/test.sh:1403: test_mon_pg: ceph osd pg-temp 0.0 0 1 2 - DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2016-06-10 17:36:58.454619 7fc3e7279ab0 -1 WARNING: the following dangerous and experimental features are enabled: *
2016-06-10 17:36:58.454625 7fc3e7279ab0 0 lockdep start
2016-06-10 17:36:58.464076 7fc3e7279ab0 -1 WARNING: the following dangerous and experimental features are enabled: *
2016-06-10 17:37:13.384954 7fc3d78c2ab0 0 monclient: hunting for new mon
2016-06-10 17:37:33.464059 7fc3d78c2ab0 0 monclient: hunting for new mon
Thanks!
Updated by Samuel Just almost 8 years ago
sjust@teuthology:/a/teuthology-2016-06-12_18:15:02-upgrade:hammer-x-jewel-distro-basic-vps/255160/remote as well I think.
Updated by Samuel Just almost 8 years ago
- Status changed from New to Pending Backport
Updated by Nathan Cutler almost 8 years ago
- Copied to Backport #16429: jewel: OSDMonitor: drop pg temps from not the current primary added
Updated by Nathan Cutler almost 8 years ago
- Copied to Backport #16430: hammer: OSDMonitor: drop pg temps from not the current primary added
Updated by Loïc Dachary over 7 years ago
- Status changed from Pending Backport to Resolved
Actions