Project

General

Profile

Bug #8811

Journal corruption during upgrade to 0.82 with standby-replay daemons

Added by Greg Farnum about 5 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
Start date:
07/10/2014
Due date:
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS
Labels (FS):
Pull request ID:

Description

Two different ceph-users reports of hitting this issue on v0.82:

0> 2014-07-09 23:21:43.385274 7fb7f7b83700 -1 mds/MDLog.cc: In function 'void MDLog::_replay_thread()' thread 7fb7f7b83700 time 2014-07-09 23:21:43.383304                                                        
mds/MDLog.cc: 815: FAILED assert(journaler->is_readable())                                                                                                                                                             

 ceph version 0.82 (14085f42ddd0fef4e7e1dc99402d07a8df82c04e)                                                                                                                                                          
 1: (MDLog::_replay_thread()+0x197b) [0x85a3cb]                                                                                                                                                                        
 2: (MDLog::ReplayThread::entry()+0xd) [0x66466d]                                                                                                                                                                      
 3: (()+0x8062) [0x7fb7ffda1062]                                                                                                                                                                                       
 4: (clone()+0x6d) [0x7fb7feb35a3d]                                                                                                                                                                                    
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 

     0> 2014-07-10 11:35:36.107022 7f45f7c57700 -1 mds/MDLog.cc: In function 'void MDLog::_replay_thread()' thread 7f45f7c57700 time 2014-07-10 11:35:36.103147
mds/MDLog.cc: 815: FAILED assert(journaler->is_readable())

 ceph version 0.82 (14085f42ddd0fef4e7e1dc99402d07a8df82c04e)
 1: (MDLog::_replay_thread()+0x197b) [0x85a3cb]
 2: (MDLog::ReplayThread::entry()+0xd) [0x66466d]
 3: (()+0x6b50) [0x7f45ffdd7b50]
 4: (clone()+0x6d) [0x7f45fec000ed]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

I went over the code a little bit and it looks good to me, but we just made the JournalStream changes so I'm sure that's the issue. For context, this MDLog assert follows a loop that waits until the Journaler is readable, so it appears to be changing its mind...presumably we're incorrectly manipulating the read_bug in some way?

Associated revisions

Revision 5438500a (diff)
Added by John Spray about 5 years ago

mds: fix journal reformat failure in standbyreplay

In the 0.82 release, standbyreplay MDS daemons would try
to reformat the jouranl if they saw an older version on
disk, where this should have only been done by the active
MDS for the rank. Depending on timing, this could cause
fatal corruption of the journal.

This change handles the following cases:
  • only do reformat if not in standbyreplay (else raise EAGAIN
    to keep trying til an active mds reformats it)
  • if journal header goes away while in standbyreplay then raise
    EAGAIN (handle rewrite happening in background)
  • if journal version is greater than the max supported, suicide

Fixes: #8811

Signed-off-by: John Spray <>

Revision d3de69f8 (diff)
Added by John Spray about 5 years ago

mds: fix journal reformat failure in standbyreplay

In the 0.82 release, standbyreplay MDS daemons would try
to reformat the jouranl if they saw an older version on
disk, where this should have only been done by the active
MDS for the rank. Depending on timing, this could cause
fatal corruption of the journal.

This change handles the following cases:
  • only do reformat if not in standbyreplay (else raise EAGAIN
    to keep trying til an active mds reformats it)
  • if journal header goes away while in standbyreplay then raise
    EAGAIN (handle rewrite happening in background)
  • if journal version is greater than the max supported, suicide

Fixes: #8811

Signed-off-by: John Spray <>
(cherry picked from commit 5438500af8979fda32e61714ae40b71c7ffdfd15)

History

#1 Updated by John Spray about 5 years ago

Hmmm. Aside from is_readable() giving inconsistent results, seems like this could happen if there was a bug that caused read_pos to get ahead of write_pos, because the check at the top of the _replay_thread loop is for get_read_pos() < get_write_pos(), but the check right before the assertion is for ==.

#2 Updated by John Spray about 5 years ago

  • Status changed from New to In Progress

This may be the result of a bug in the journal reformatting that occurs during upgrade, affecting systems using standby-replay MDS daemons. Journal corruption can occur when both an active and a standby-replay daemon attempt to do the rewrite at the same time.

#3 Updated by John Spray about 5 years ago

  • Status changed from In Progress to Need Review

#5 Updated by John Spray about 5 years ago

  • Subject changed from MDLog::is_readable() assert to Journal corruption during upgrade to 0.82 with standby-replay daemons

#6 Updated by Greg Farnum about 5 years ago

  • Status changed from Need Review to Resolved

This got fixed 11 days ago, but was never marked closed. Merged in commit:b9463e3497cc1f2a1bab0838430a4402d8c88af0

#7 Updated by Greg Farnum about 3 years ago

  • Component(FS) MDS added

Also available in: Atom PDF