Project

General

Profile

Bug #4687

ceph_test_filejournal failing

Added by Samuel Just almost 6 years ago. Updated almost 6 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
OSDMap
Target version:
Start date:
04/08/2013
Due date:
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

/a/teuthology-2013-04-08_02:44:31-rados-next-testing-basic/10131

Easily reproduced on next... Almost certainly caused by the recent journal corruption fix.

2013-04-08T03:57:32.800 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.884607 7f2082bb2780 1 journal close /tmp/ceph_test_filejournal.tmp.269215514
2013-04-08T03:57:32.807 INFO:teuthology.orchestra.run.out:[ OK ] TestFileJournal.WriteTrimSmall (8510 ms)
2013-04-08T03:57:32.807 INFO:teuthology.orchestra.run.out:[ RUN ] TestFileJournal.ReplayDetectCorruptFooterMagic
2013-04-08T03:57:32.807 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.892136 7f2082bb2780 1 journal _open /tmp/ceph_test_filejournal.tmp.269215514 fd 17: 209715200 bytes, block size 4096 byt
es, directio = 1, aio = 1
2013-04-08T03:57:32.823 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.908489 7f2082bb2780 1 journal _open /tmp/ceph_test_filejournal.tmp.269215514 fd 17: 209715200 bytes, block size 4096 byt
es, directio = 1, aio = 1
2013-04-08T03:57:32.833 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.917460 7f2082bb2780 1 journal close /tmp/ceph_test_filejournal.tmp.269215514
2013-04-08T03:57:32.833 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.917750 7f2082bb2780 1 journal _open /tmp/ceph_test_filejournal.tmp.269215514 fd 18: 209715200 bytes, block size 4096 byt
es, directio = 1, aio = 1
2013-04-08T03:57:32.833 INFO:teuthology.orchestra.run.out:corrupting journal
2013-04-08T03:57:32.849 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.933783 7f2082bb2780 1 journal Entry at pos 12288 possibly corrupt due to: (bad footer magic, partial entry)
2013-04-08T03:57:32.849 INFO:teuthology.orchestra.run.err:
2013-04-08T03:57:32.849 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.933796 7f2082bb2780 -1 journal Entry at pos 20480 valid, there are missing sequence numbers prior to seq 3
2013-04-08T03:57:32.850 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.933828 7f2082bb2780 1 journal _open /tmp/ceph_test_filejournal.tmp.269215514 fd 18: 209715200 bytes, block size 4096 byt
es, directio = 1, aio = 1
2013-04-08T03:57:32.850 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.933878 7f2082bb2780 1 journal close /tmp/ceph_test_filejournal.tmp.269215514
2013-04-08T03:57:32.850 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.934150 7f2082bb2780 1 journal _open /tmp/ceph_test_filejournal.tmp.269215514 fd 17: 209715200 bytes, block size 4096 byt
es, directio = 1, aio = 1
2013-04-08T03:57:32.850 INFO:teuthology.orchestra.run.out:[ OK ] TestFileJournal.ReplayDetectCorruptFooterMagic (42 ms)
2013-04-08T03:57:32.851 INFO:teuthology.orchestra.run.out:[ RUN ] TestFileJournal.ReplayDetectCorruptPayload
2013-04-08T03:57:32.865 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.950156 7f2082bb2780 1 journal _open /tmp/ceph_test_filejournal.tmp.269215514 fd 17: 209715200 bytes, block size 4096 byt
es, directio = 1, aio = 1
2013-04-08T03:57:32.884 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.967209 7f2082bb2780 1 journal close /tmp/ceph_test_filejournal.tmp.269215514
2013-04-08T03:57:32.884 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.967481 7f2082bb2780 1 journal _open /tmp/ceph_test_filejournal.tmp.269215514 fd 18: 209715200 bytes, block size 4096 byt
es, directio = 1, aio = 1
2013-04-08T03:57:32.884 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.968052 7f2082bb2780 -1 journal Entry at pos 12288 possibly corrupt due to: (header crc (3278193258) doesn't match body cr
c (2450721477))
2013-04-08T03:57:32.884 INFO:teuthology.orchestra.run.err:
2013-04-08T03:57:32.885 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.968054 7f2082bb2780 -1 journal Entry at pos 20480 valid, there are missing sequence numbers prior to seq 3
2013-04-08T03:57:32.885 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.968084 7f2082bb2780 1 journal _open /tmp/ceph_test_filejournal.tmp.269215514 fd 18: 209715200 bytes, block size 4096 byt
es, directio = 1, aio = 1
2013-04-08T03:57:32.885 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.968133 7f2082bb2780 1 journal close /tmp/ceph_test_filejournal.tmp.269215514
2013-04-08T03:57:32.885 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.968426 7f2082bb2780 1 journal _open /tmp/ceph_test_filejournal.tmp.269215514 fd 17: 209715200 bytes, block size 4096 byt
es, directio = 1, aio = 1
2013-04-08T03:57:32.885 INFO:teuthology.orchestra.run.out:corrupting journal
2013-04-08T03:57:32.885 INFO:teuthology.orchestra.run.out:[ OK ] TestFileJournal.ReplayDetectCorruptPayload (34 ms)
2013-04-08T03:57:32.886 INFO:teuthology.orchestra.run.out:[ RUN ] TestFileJournal.ReplayDetectCorruptHeader
2013-04-08T03:57:32.898 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.983485 7f2082bb2780 1 journal _open /tmp/ceph_test_filejournal.tmp.269215514 fd 17: 209715200 bytes, block size 4096 byt
es, directio = 1, aio = 1
2013-04-08T03:57:32.907 INFO:teuthology.orchestra.run.out:corrupting journal
2013-04-08T03:57:32.908 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.992190 7f2082bb2780 1 journal close /tmp/ceph_test_filejournal.tmp.269215514
2013-04-08T03:57:32.908 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.992413 7f2082bb2780 1 journal _open /tmp/ceph_test_filejournal.tmp.269215514 fd 18: 209715200 bytes, block size 4096 byt
es, directio = 1, aio = 1
2013-04-08T03:57:32.909 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.993604 7f2082bb2780 -1 journal Entry at pos 12288 possibly corrupt due to: (bad header magic)
2013-04-08T03:57:32.909 INFO:teuthology.orchestra.run.err:Entry at pos 16384 possibly corrupt due to: (bad header magic)
2013-04-08T03:57:32.909 INFO:teuthology.orchestra.run.err:
2013-04-08T03:57:32.909 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.993607 7f2082bb2780 -1 journal Entry at pos 20480 valid, there are missing sequence numbers prior to seq 3
2013-04-08T03:57:32.909 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.993638 7f2082bb2780 1 journal _open /tmp/ceph_test_filejournal.tmp.269215514 fd 18: 209715200 bytes, block size 4096 bytes, directio = 1, aio = 1
2013-04-08T03:57:32.910 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.993688 7f2082bb2780 1 journal close /tmp/ceph_test_filejournal.tmp.269215514
2013-04-08T03:57:32.910 INFO:teuthology.orchestra.run.out:[ OK ] TestFileJournal.ReplayDetectCorruptHeader (25 ms)
2013-04-08T03:57:32.910 INFO:teuthology.orchestra.run.out:[---------
] 12 tests from TestFileJournal (17324 ms total)
2013-04-08T03:57:32.910 INFO:teuthology.orchestra.run.out:
2013-04-08T03:57:32.910 INFO:teuthology.orchestra.run.out:[----------] Global test environment tear-down
2013-04-08T03:57:32.910 INFO:teuthology.orchestra.run.out:[==========] 12 tests from 1 test case ran. (17324 ms total)
2013-04-08T03:57:32.911 INFO:teuthology.orchestra.run.out:[ PASSED ] 9 tests.
2013-04-08T03:57:32.911 INFO:teuthology.orchestra.run.out:[ FAILED ] 3 tests, listed below:
2013-04-08T03:57:32.912 INFO:teuthology.orchestra.run.out:[ FAILED ] TestFileJournal.WriteManyVecs
2013-04-08T03:57:32.912 INFO:teuthology.orchestra.run.out:[ FAILED ] TestFileJournal.ReplaySmall
2013-04-08T03:57:32.912 INFO:teuthology.orchestra.run.out:[ FAILED ] TestFileJournal.ReplayCorrupt
2013-04-08T03:57:32.912 INFO:teuthology.orchestra.run.out:
2013-04-08T03:57:32.913 INFO:teuthology.orchestra.run.out: 3 FAILED TESTS
2013-04-08T03:57:32.915 ERROR:teuthology.run_tasks:Saw exception from tasks
Traceback (most recent call last):
File "/var/lib/teuthworker/teuthology-next/teuthology/run_tasks.py", line 25, in run_tasks
manager = _run_one_task(taskname, ctx=ctx, config=config)
File "/var/lib/teuthworker/teuthology-next/teuthology/run_tasks.py", line 14, in _run_one_task
return fn(**kwargs)


Related issues

Duplicated by Ceph - Bug #4682: ceph_test_filejournal tests failed in the nightlies Duplicate 04/08/2013

Associated revisions

Revision a48739d9 (diff)
Added by Samuel Just almost 6 years ago

FileJournal: clarify meaning of start_seq and fix initialization

Second guessing the first sequence number from the FileStore
was silly and broke tests which had the temerity to start at
1 instead of 2...

Fixes: #4687
Signed-off-by: Samuel Just <>
Reviewed-by: Greg Farnum <>

History

#1 Updated by Samuel Just almost 6 years ago

  • Status changed from Verified to Resolved

a48739d9ab69b8372723139f652717709910e4d2

Also available in: Atom PDF