Bug #4910
closed
journal Unable to read past sequence 337 but header indicates the journal has committed up through 348, journal is corrupt
Added by Tamilarasi muthamizhan almost 11 years ago.
Updated almost 11 years ago.
Description
logs: ubuntu@teuthology:/a/teuthology-2013-05-04_01:00:03-rados-next-testing-basic/6997
2013-05-04 04:41:07.307284 7f547bdbd780 -1 journal Unable to read past sequence 337 but header indicates the journal has committed up through 348, journal is corrupt
2013-05-04 04:41:07.309442 7f547bdbd780 -1 os/FileJournal.cc: In function 'bool FileJournal::read_entry(ceph::bufferlist&, uint64_t&, bool*)' thread 7f547bdbd780 time 2013-05-04 04:41:07.307297
os/FileJournal.cc: 1689: FAILED assert(0)
ceph version 0.60-805-g1a67f7b (1a67f7b3ac3c035d6e4b2181fbad903aa4b03711)
1: (FileJournal::read_entry(ceph::buffer::list&, unsigned long&, bool*)+0x7d6) [0x719cd6]
2: (JournalingObjectStore::journal_replay(unsigned long)+0x33d) [0x760a2d]
3: (FileStore::mount()+0x3984) [0x745374]
4: (OSD::do_convertfs(ObjectStore*)+0x1a) [0x60c07a]
5: (OSD::convertfs(std::string const&, std::string const&)+0x47) [0x60cae7]
6: (main()+0x2239) [0x57fa59]
7: (__libc_start_main()+0xed) [0x7f547985876d]
8: ceph-osd() [0x58221d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
ubuntu@teuthology:/a/teuthology-2013-05-04_01:00:03-rados-next-testing-basic/7001
ubuntu@teuthology:/a/teuthology-2013-05-04_01:00:03-rados-next-testing-basic/7001$ cat config.yaml
kernel: &id001
kdb: true
sha1: b5b09be30cf99f9c699e825629f02e3bce555d44
machine_type: plana
nuke-on-error: true
overrides:
ceph:
conf:
mon:
debug mon: 20
debug ms: 20
debug paxos: 20
fs: xfs
log-whitelist:
- slow request
sha1: 1a67f7b3ac3c035d6e4b2181fbad903aa4b03711
s3tests:
branch: next
workunit:
sha1: 1a67f7b3ac3c035d6e4b2181fbad903aa4b03711
roles:
- - mon.0
- mon.1
- mon.2
- mds.0
- client.0
- - osd.0
- - osd.1
- - osd.2
targets:
ubuntu@plana17.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZmJJ7QfE1OhVkGq6eOCkmvnom5zClgFwfNEPGC+UWkPQzXALAAmf/eGf212lAPIWJr3xFLhmW42gdToPvSYWtr/x5rV4JcFF+FaPu5xgvmGqSfHY9S7L4quutVd2lUxLZddviDtwofJWW7JSqKQVtG5OnP5q9yj55zHwPgN1sFJUtPWIYjfgGMHmnf0QH6V7TpZCidSGNiNLbw5EZaKfK6vJyO1xBysYuP82AcmtGXnj9zmp7W9yaja9oOn9/gKdIrlwjySR2iOR3AGK/+m2mEh0acJA06GUaZPVPQ56l6u2qc46Hrqu485fYj2zEiXxKLybi1AaIqAoKT3byJZuT
ubuntu@plana45.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDp3cwfZhOipCot6NiKX4cRMn4zx43QY0+5HdqzCQU2y7OrOJt3d0qvifnZPyeq8/d+aW2WL2OM8m4taz380JsP0SLmlpY8D0pGY/tN0pQDqIFd8EboMtKY6tR8unQrVzuczMqup/tkKSfdRp0zAeTiJ8qH7l9MaVcOw6WfRACb8f7APJE2gVRBrzPAdbqKzAphTRzZSz0cq722AX7XQDPT2dz7NoTp5Tk7xaQdDu2II+78B1H27IWdyYeonfy17yf9N+IA2Xzna/g5zu8apg7UvzyFmHunLyjr78dhPtR39201A0QJ5x5Qli9/UaB3LwiqnbCiGfx4xWFazdUFzxiD
ubuntu@plana61.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOTCMIScDTmD9NkfsWU7xeyZ+WOXai5izYeliiXDSjJC3bT6r8Fp+rhPfcHCVHiw++VsbvKZtkhjCSnJTVPWCdpRDghzJ3nZUBImWRo3PmHo1etQpCeimaOrIJ2q0ChN5jmSOqy5B+Z4om2vXBtBY6nkdTxDOr2+MH3NrSPkQSFB0zO+VPuwKXsemeUC6urb2IZZpxY3cxNq4fafTF9PROpgOnIA+o3igyU4duKEjnCzTHZjw/PL7Eph/7p6+UQgrUwe7pgVzT+2MM0zcBtBSXNqs3dCGmpvUapOkBlDoIX02EkWRNpkM3vfeFt1EFC17B5vd61Kg40bYUG8qWGR0T
ubuntu@plana63.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDy2BKPe+fe5jK0ziU8aKM0DzODSTaKWecQRwLjnZLbjDvTyHm8x8xX/JCts3bFfrc2ozFz7ILBIWU96JRZiFF2TtFZjtf1H19kyvR8PWCxiZ/lld+C7B6U8iiPSNiSlgo7mwkpk1JoSpHe4rK/Z7WQRWBMsCC7XJETu6rRX3i0ZYaKh8BoWWhpsBs1quSNxRXNUqJ6OKnDbB5Vuan1TK9b49RXmibx+oapXm8V0sHEVLYa+NTUs+wAEHAnjFgRe75Cik/rmgeE0m2Cff1rp9tFhEEDwZ5PUdnscOTY78BxImMRdkbZ8lJXOGcOOsD3Dj1jOr4pVrgxZqUdtWfJGkj
tasks:
- internal.lock_machines:
- 4
- plana
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- internal.check_ceph_data: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock.check: null
- install: null
- ceph: null
- thrashosds:
chance_down: 1.0
powercycle: true
- ceph-fuse: null
- workunit:
clients:
all:
- suites/fsx.sh
ubuntu@teuthology:/a/teuthology-2013-05-04_01:00:03-rados-next-testing-basic/7008
- Priority changed from Urgent to High
not sure about the priority of this bug.
This reproduces very quickly without journal logging, but doesn't reproduce at all with.
This also stops happening if I disable aio.
- Priority changed from High to Urgent
- Priority changed from Urgent to Normal
- Priority changed from Normal to High
- Status changed from New to Resolved
- Status changed from Resolved to Duplicate
Also available in: Atom
PDF