Ceph

/a/teuthology-2013-04-08_02:44:31-rados-next-testing-basic/10131

Easily reproduced on next... Almost certainly caused by the recent journal corruption fix.

2013-04-08T03:57:32.800 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.884607 7f2082bb2780 1 journal close /tmp/ceph_test_filejournal.tmp.269215514
2013-04-08T03:57:32.807 INFO:teuthology.orchestra.run.out:[ OK ] TestFileJournal.WriteTrimSmall (8510 ms)
2013-04-08T03:57:32.807 INFO:teuthology.orchestra.run.out:[ RUN ] TestFileJournal.ReplayDetectCorruptFooterMagic
2013-04-08T03:57:32.807 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.892136 7f2082bb2780 1 journal _open /tmp/ceph_test_filejournal.tmp.269215514 fd 17: 209715200 bytes, block size 4096 byt
es, directio = 1, aio = 1
2013-04-08T03:57:32.823 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.908489 7f2082bb2780 1 journal _open /tmp/ceph_test_filejournal.tmp.269215514 fd 17: 209715200 bytes, block size 4096 byt
es, directio = 1, aio = 1
2013-04-08T03:57:32.833 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.917460 7f2082bb2780 1 journal close /tmp/ceph_test_filejournal.tmp.269215514
2013-04-08T03:57:32.833 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.917750 7f2082bb2780 1 journal _open /tmp/ceph_test_filejournal.tmp.269215514 fd 18: 209715200 bytes, block size 4096 byt
es, directio = 1, aio = 1
2013-04-08T03:57:32.833 INFO:teuthology.orchestra.run.out:corrupting journal
2013-04-08T03:57:32.849 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.933783 7f2082bb2780 1 journal Entry at pos 12288 possibly corrupt due to: (bad footer magic, partial entry)
2013-04-08T03:57:32.849 INFO:teuthology.orchestra.run.err:
2013-04-08T03:57:32.849 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.933796 7f2082bb2780 -1 journal Entry at pos 20480 valid, there are missing sequence numbers prior to seq 3
2013-04-08T03:57:32.850 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.933828 7f2082bb2780 1 journal _open /tmp/ceph_test_filejournal.tmp.269215514 fd 18: 209715200 bytes, block size 4096 byt
es, directio = 1, aio = 1
2013-04-08T03:57:32.850 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.933878 7f2082bb2780 1 journal close /tmp/ceph_test_filejournal.tmp.269215514
2013-04-08T03:57:32.850 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.934150 7f2082bb2780 1 journal _open /tmp/ceph_test_filejournal.tmp.269215514 fd 17: 209715200 bytes, block size 4096 byt
es, directio = 1, aio = 1
2013-04-08T03:57:32.850 INFO:teuthology.orchestra.run.out:[ OK ] TestFileJournal.ReplayDetectCorruptFooterMagic (42 ms)
2013-04-08T03:57:32.851 INFO:teuthology.orchestra.run.out:[ RUN ] TestFileJournal.ReplayDetectCorruptPayload
2013-04-08T03:57:32.865 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.950156 7f2082bb2780 1 journal _open /tmp/ceph_test_filejournal.tmp.269215514 fd 17: 209715200 bytes, block size 4096 byt
es, directio = 1, aio = 1
2013-04-08T03:57:32.884 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.967209 7f2082bb2780 1 journal close /tmp/ceph_test_filejournal.tmp.269215514
2013-04-08T03:57:32.884 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.967481 7f2082bb2780 1 journal _open /tmp/ceph_test_filejournal.tmp.269215514 fd 18: 209715200 bytes, block size 4096 byt
es, directio = 1, aio = 1
2013-04-08T03:57:32.884 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.968052 7f2082bb2780 -1 journal Entry at pos 12288 possibly corrupt due to: (header crc (3278193258) doesn't match body cr
c (2450721477))
2013-04-08T03:57:32.884 INFO:teuthology.orchestra.run.err:
2013-04-08T03:57:32.885 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.968054 7f2082bb2780 -1 journal Entry at pos 20480 valid, there are missing sequence numbers prior to seq 3
2013-04-08T03:57:32.885 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.968084 7f2082bb2780 1 journal _open /tmp/ceph_test_filejournal.tmp.269215514 fd 18: 209715200 bytes, block size 4096 byt
es, directio = 1, aio = 1
2013-04-08T03:57:32.885 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.968133 7f2082bb2780 1 journal close /tmp/ceph_test_filejournal.tmp.269215514
2013-04-08T03:57:32.885 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.968426 7f2082bb2780 1 journal _open /tmp/ceph_test_filejournal.tmp.269215514 fd 17: 209715200 bytes, block size 4096 byt
es, directio = 1, aio = 1
2013-04-08T03:57:32.885 INFO:teuthology.orchestra.run.out:corrupting journal
2013-04-08T03:57:32.885 INFO:teuthology.orchestra.run.out:[ OK ] TestFileJournal.ReplayDetectCorruptPayload (34 ms)
2013-04-08T03:57:32.886 INFO:teuthology.orchestra.run.out:[ RUN ] TestFileJournal.ReplayDetectCorruptHeader
2013-04-08T03:57:32.898 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.983485 7f2082bb2780 1 journal _open /tmp/ceph_test_filejournal.tmp.269215514 fd 17: 209715200 bytes, block size 4096 byt
es, directio = 1, aio = 1
2013-04-08T03:57:32.907 INFO:teuthology.orchestra.run.out:corrupting journal
2013-04-08T03:57:32.908 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.992190 7f2082bb2780 1 journal close /tmp/ceph_test_filejournal.tmp.269215514
2013-04-08T03:57:32.908 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.992413 7f2082bb2780 1 journal _open /tmp/ceph_test_filejournal.tmp.269215514 fd 18: 209715200 bytes, block size 4096 byt
es, directio = 1, aio = 1
2013-04-08T03:57:32.909 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.993604 7f2082bb2780 -1 journal Entry at pos 12288 possibly corrupt due to: (bad header magic)
2013-04-08T03:57:32.909 INFO:teuthology.orchestra.run.err:Entry at pos 16384 possibly corrupt due to: (bad header magic)
2013-04-08T03:57:32.909 INFO:teuthology.orchestra.run.err:
2013-04-08T03:57:32.909 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.993607 7f2082bb2780 -1 journal Entry at pos 20480 valid, there are missing sequence numbers prior to seq 3
2013-04-08T03:57:32.909 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.993638 7f2082bb2780 1 journal _open /tmp/ceph_test_filejournal.tmp.269215514 fd 18: 209715200 bytes, block size 4096 bytes, directio = 1, aio = 1
2013-04-08T03:57:32.910 INFO:teuthology.orchestra.run.err:2013-04-08 03:57:12.993688 7f2082bb2780 1 journal close /tmp/ceph_test_filejournal.tmp.269215514
2013-04-08T03:57:32.910 INFO:teuthology.orchestra.run.out:[ OK ] TestFileJournal.ReplayDetectCorruptHeader (25 ms)
2013-04-08T03:57:32.910 INFO:teuthology.orchestra.run.out:[---------] 12 tests from TestFileJournal (17324 ms total)
2013-04-08T03:57:32.910 INFO:teuthology.orchestra.run.out:
2013-04-08T03:57:32.910 INFO:teuthology.orchestra.run.out:[----------] Global test environment tear-down
2013-04-08T03:57:32.910 INFO:teuthology.orchestra.run.out:[==========] 12 tests from 1 test case ran. (17324 ms total)
2013-04-08T03:57:32.911 INFO:teuthology.orchestra.run.out:[ PASSED ] 9 tests.
2013-04-08T03:57:32.911 INFO:teuthology.orchestra.run.out:[ FAILED ] 3 tests, listed below:
2013-04-08T03:57:32.912 INFO:teuthology.orchestra.run.out:[ FAILED ] TestFileJournal.WriteManyVecs
2013-04-08T03:57:32.912 INFO:teuthology.orchestra.run.out:[ FAILED ] TestFileJournal.ReplaySmall
2013-04-08T03:57:32.912 INFO:teuthology.orchestra.run.out:[ FAILED ] TestFileJournal.ReplayCorrupt
2013-04-08T03:57:32.912 INFO:teuthology.orchestra.run.out:
2013-04-08T03:57:32.913 INFO:teuthology.orchestra.run.out: 3 FAILED TESTS
2013-04-08T03:57:32.915 ERROR:teuthology.run_tasks:Saw exception from tasks
Traceback (most recent call last):
File "/var/lib/teuthworker/teuthology-next/teuthology/run_tasks.py", line 25, in run_tasks
manager = _run_one_task(taskname, ctx=ctx, config=config)
File "/var/lib/teuthworker/teuthology-next/teuthology/run_tasks.py", line 14, in _run_one_task
return fn(**kwargs)