Project

General

Profile

Actions
Copy link

Bug #4910

closed

journal Unable to read past sequence 337 but header indicates the journal has committed up through 348, journal is corrupt

Added by Tamilarasi muthamizhan almost 11 years ago. Updated almost 11 years ago.

Status:
Duplicate
Priority:
High
Assignee:
Samuel Just
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

logs: ubuntu@teuthology:/a/teuthology-2013-05-04_01:00:03-rados-next-testing-basic/6997

2013-05-04 04:41:07.307284 7f547bdbd780 -1 journal Unable to read past sequence 337 but header indicates the journal has committed up through 348, journal is corrupt
2013-05-04 04:41:07.309442 7f547bdbd780 -1 os/FileJournal.cc: In function 'bool FileJournal::read_entry(ceph::bufferlist&, uint64_t&, bool*)' thread 7f547bdbd780 time 2013-05-04 04:41:07.307297
os/FileJournal.cc: 1689: FAILED assert(0)

 ceph version 0.60-805-g1a67f7b (1a67f7b3ac3c035d6e4b2181fbad903aa4b03711)
 1: (FileJournal::read_entry(ceph::buffer::list&, unsigned long&, bool*)+0x7d6) [0x719cd6]
 2: (JournalingObjectStore::journal_replay(unsigned long)+0x33d) [0x760a2d]
 3: (FileStore::mount()+0x3984) [0x745374]
 4: (OSD::do_convertfs(ObjectStore*)+0x1a) [0x60c07a]
 5: (OSD::convertfs(std::string const&, std::string const&)+0x47) [0x60cae7]
 6: (main()+0x2239) [0x57fa59]
 7: (__libc_start_main()+0xed) [0x7f547985876d]
 8: ceph-osd() [0x58221d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Actions
Copy link
 #1

Updated by Tamilarasi muthamizhan almost 11 years ago 

ubuntu@teuthology:/a/teuthology-2013-05-04_01:00:03-rados-next-testing-basic/7001

ubuntu@teuthology:/a/teuthology-2013-05-04_01:00:03-rados-next-testing-basic/7001$ cat config.yaml 
kernel: &id001
  kdb: true
  sha1: b5b09be30cf99f9c699e825629f02e3bce555d44
machine_type: plana
nuke-on-error: true
overrides:
  ceph:
    conf:
      mon:
        debug mon: 20
        debug ms: 20
        debug paxos: 20
    fs: xfs
    log-whitelist:
    - slow request
    sha1: 1a67f7b3ac3c035d6e4b2181fbad903aa4b03711
  s3tests:
    branch: next
  workunit:
    sha1: 1a67f7b3ac3c035d6e4b2181fbad903aa4b03711
roles:
- - mon.0
  - mon.1
  - mon.2
  - mds.0
  - client.0
- - osd.0
- - osd.1
- - osd.2
targets:
  ubuntu@plana17.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZmJJ7QfE1OhVkGq6eOCkmvnom5zClgFwfNEPGC+UWkPQzXALAAmf/eGf212lAPIWJr3xFLhmW42gdToPvSYWtr/x5rV4JcFF+FaPu5xgvmGqSfHY9S7L4quutVd2lUxLZddviDtwofJWW7JSqKQVtG5OnP5q9yj55zHwPgN1sFJUtPWIYjfgGMHmnf0QH6V7TpZCidSGNiNLbw5EZaKfK6vJyO1xBysYuP82AcmtGXnj9zmp7W9yaja9oOn9/gKdIrlwjySR2iOR3AGK/+m2mEh0acJA06GUaZPVPQ56l6u2qc46Hrqu485fYj2zEiXxKLybi1AaIqAoKT3byJZuT
  ubuntu@plana45.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDp3cwfZhOipCot6NiKX4cRMn4zx43QY0+5HdqzCQU2y7OrOJt3d0qvifnZPyeq8/d+aW2WL2OM8m4taz380JsP0SLmlpY8D0pGY/tN0pQDqIFd8EboMtKY6tR8unQrVzuczMqup/tkKSfdRp0zAeTiJ8qH7l9MaVcOw6WfRACb8f7APJE2gVRBrzPAdbqKzAphTRzZSz0cq722AX7XQDPT2dz7NoTp5Tk7xaQdDu2II+78B1H27IWdyYeonfy17yf9N+IA2Xzna/g5zu8apg7UvzyFmHunLyjr78dhPtR39201A0QJ5x5Qli9/UaB3LwiqnbCiGfx4xWFazdUFzxiD
  ubuntu@plana61.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOTCMIScDTmD9NkfsWU7xeyZ+WOXai5izYeliiXDSjJC3bT6r8Fp+rhPfcHCVHiw++VsbvKZtkhjCSnJTVPWCdpRDghzJ3nZUBImWRo3PmHo1etQpCeimaOrIJ2q0ChN5jmSOqy5B+Z4om2vXBtBY6nkdTxDOr2+MH3NrSPkQSFB0zO+VPuwKXsemeUC6urb2IZZpxY3cxNq4fafTF9PROpgOnIA+o3igyU4duKEjnCzTHZjw/PL7Eph/7p6+UQgrUwe7pgVzT+2MM0zcBtBSXNqs3dCGmpvUapOkBlDoIX02EkWRNpkM3vfeFt1EFC17B5vd61Kg40bYUG8qWGR0T
  ubuntu@plana63.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDy2BKPe+fe5jK0ziU8aKM0DzODSTaKWecQRwLjnZLbjDvTyHm8x8xX/JCts3bFfrc2ozFz7ILBIWU96JRZiFF2TtFZjtf1H19kyvR8PWCxiZ/lld+C7B6U8iiPSNiSlgo7mwkpk1JoSpHe4rK/Z7WQRWBMsCC7XJETu6rRX3i0ZYaKh8BoWWhpsBs1quSNxRXNUqJ6OKnDbB5Vuan1TK9b49RXmibx+oapXm8V0sHEVLYa+NTUs+wAEHAnjFgRe75Cik/rmgeE0m2Cff1rp9tFhEEDwZ5PUdnscOTY78BxImMRdkbZ8lJXOGcOOsD3Dj1jOr4pVrgxZqUdtWfJGkj
tasks:
- internal.lock_machines:
  - 4
  - plana
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- internal.check_ceph_data: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock.check: null
- install: null
- ceph: null
- thrashosds:
    chance_down: 1.0
    powercycle: true
- ceph-fuse: null
- workunit:
    clients:
      all:
      - suites/fsx.sh
Actions
Copy link
 #2

Updated by Tamilarasi muthamizhan almost 11 years ago

ubuntu@teuthology:/a/teuthology-2013-05-04_01:00:03-rados-next-testing-basic/7008

Actions
Copy link
 #3

Updated by Tamilarasi muthamizhan almost 11 years ago

  • Priority changed from Urgent to High

not sure about the priority of this bug.

Actions
Copy link
 #4

Updated by Samuel Just almost 11 years ago

This reproduces very quickly without journal logging, but doesn't reproduce at all with.

Actions
Copy link
 #5

Updated by Samuel Just almost 11 years ago

This also stops happening if I disable aio.

Actions
Copy link
 #6

Updated by Ian Colle almost 11 years ago

  • Priority changed from High to Urgent
Actions
Copy link
 #7

Updated by Ian Colle almost 11 years ago

wip-4910

Actions
Copy link
 #8

Updated by Samuel Just almost 11 years ago

  • Priority changed from Urgent to Normal
Actions
Copy link
 #9

Updated by Samuel Just almost 11 years ago

  • Priority changed from Normal to High
Actions
Copy link
 #10

Updated by Samuel Just almost 11 years ago

  • Status changed from New to Resolved
Actions
Copy link
 #11

Updated by Sage Weil almost 11 years ago

  • Status changed from Resolved to Duplicate
Actions
Copy link

Also available in: Atom PDF