Project

General

Profile

Actions
Copy link

Bug #10883

closed

"Assertion: os/FileJournal.cc: 1769: FAILED assert(0)" in upgrade:dumpling-firefly-x-hammer-distro-basic-multi run

Added by Yuri Weinstein about 9 years ago. Updated about 9 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
David Zafman
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
giant,firefly,dumpling
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Run: http://pulpito.ceph.com/teuthology-2015-02-12_17:10:01-upgrade:dumpling-firefly-x-hammer-distro-basic-multi/

Job: ['753922']
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-02-12_17:10:01-upgrade:dumpling-firefly-x-hammer-distro-basic-multi/753922/teuthology.log

2015-02-12T20:52:36.720 INFO:tasks.ceph.osd.4.plana17.stdout:starting osd.4 at :/0 osd_data /var/lib/ceph/osd/ceph-4 /var/lib/ceph/osd/ceph-4/journal
2015-02-12T20:52:36.732 INFO:tasks.ceph.osd.4.plana17.stderr:2015-02-12 20:52:36.731565 7f97c261c780 -1 filestore(/var/lib/ceph/osd/ceph-4) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade.
2015-02-12T20:52:36.986 INFO:tasks.ceph.osd.4.plana17.stderr:2015-02-12 20:52:36.985359 7f97c261c780 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
2015-02-12T20:52:37.002 INFO:tasks.ceph.osd.4.plana17.stderr:2015-02-12 20:52:37.001415 7f97c261c780 -1 journal Unable to read past sequence 45776 but header indicates the journal has committed up through 45794, journal is corrupt
2015-02-12T20:52:37.005 INFO:tasks.ceph.osd.4.plana17.stderr:os/FileJournal.cc: In function 'bool FileJournal::read_entry(ceph::bufferlist&, uint64_t&, bool*)' thread 7f97c261c780 time 2015-02-12 20:52:37.001436
2015-02-12T20:52:37.005 INFO:tasks.ceph.osd.4.plana17.stderr:os/FileJournal.cc: 1769: FAILED assert(0)
2015-02-12T20:52:37.021 INFO:tasks.ceph.osd.4.plana17.stderr: ceph version 0.92-1093-g26dfad7 (26dfad798d5193295cbe86902a6bbcf773b03974)
2015-02-12T20:52:37.021 INFO:tasks.ceph.osd.4.plana17.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x7f) [0xaebc9f]
2015-02-12T20:52:37.022 INFO:tasks.ceph.osd.4.plana17.stderr: 2: (FileJournal::read_entry(ceph::buffer::list&, unsigned long&, bool*)+0x877) [0x9d0fc7]
2015-02-12T20:52:37.022 INFO:tasks.ceph.osd.4.plana17.stderr: 3: (FileJournal::open(unsigned long)+0x662) [0x9cc6d2]
2015-02-12T20:52:37.022 INFO:tasks.ceph.osd.4.plana17.stderr: 4: (JournalingObjectStore::journal_replay(unsigned long)+0x14a) [0x90832a]
2015-02-12T20:52:37.022 INFO:tasks.ceph.osd.4.plana17.stderr: 5: (FileStore::mount()+0x35b1) [0x8dd2c1]
2015-02-12T20:52:37.022 INFO:tasks.ceph.osd.4.plana17.stderr: 6: (OSD::init()+0x253) [0x6887e3]
2015-02-12T20:52:37.023 INFO:tasks.ceph.osd.4.plana17.stderr: 7: (main()+0x26b9) [0x626cd9]
2015-02-12T20:52:37.023 INFO:tasks.ceph.osd.4.plana17.stderr: 8: (__libc_start_main()+0xed) [0x7f97c021776d]
2015-02-12T20:52:37.023 INFO:tasks.ceph.osd.4.plana17.stderr: 9: ceph-osd() [0x62c249]
2015-02-12T20:52:37.023 INFO:tasks.ceph.osd.4.plana17.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-02-12T20:52:37.023 INFO:tasks.ceph.osd.4.plana17.stderr:2015-02-12 20:52:37.020739 7f97c261c780 -1 os/FileJournal.cc: In function 'bool FileJournal::read_entry(ceph::bufferlist&, uint64_t&, bool*)' thread 7f97c261c780 time 2015-02-12 20:52:37.001436
Actions
Copy link
 #1

Updated by Samuel Just about 9 years ago

also: ubuntu@teuthology:/a/samuelj-2015-02-12_21:43:00-rados-wip-sam-testing-wip-testing-vanilla-fixes-basic-multi/755435

repushed that one as wip-sam-testing-10883

Actions
Copy link
 #2

Updated by David Zafman about 9 years ago

  • Assignee set to David Zafman
Actions
Copy link
 #3

Updated by David Zafman about 9 years ago

This was introduced by f46b1b47 to make journal be able to split items across the end of the file to the front of the file. It also changed the order of the header being written to keep the number of seeks the same. Now a write of the first chunk can overwrite what is at the old header.start before the updated header is on disk. The old code always wrote the header before everything else. The fix will require an extra seek when wrapping the end of the journal because we must write the header first.


commit f46b1b473fce0322a672b16c7739e569a45054b6
Author: Sage Weil <sage@newdream.net>
Date:   Mon Feb 1 15:44:26 2010 -0800

    journal: make wrapping simpler

    Take out weirdness that tries to keep journal items contiguous.  No reason
    not to split them across the end/beginning of the journal.  In the general
    case, this is the same # of seeks because we have to rewrite the header
    anyway.
Actions
Copy link
 #4

Updated by David Zafman about 9 years ago

  • Status changed from New to In Progress
Actions
Copy link
 #5

Updated by David Zafman about 9 years ago

  • Status changed from In Progress to 7
Actions
Copy link
 #6

Updated by Sage Weil about 9 years ago

  • Status changed from 7 to Pending Backport
  • Backport set to giant,firefly,dumpling
Actions
Copy link
 #7

Updated by Sage Weil about 9 years ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link

Also available in: Atom PDF