Project

General

Profile

Bug #5240

run_seed_to_range failed, probably fdcache

Added by Samuel Just almost 11 years ago. Updated almost 11 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
OSD
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2013-06-03T04:26:53.232 INFO:teuthology.orchestra.run.err:2013-06-03 04:27:34.948984 7fa652ef5780 0 filestore_diff diff_objects content mismatch on 0.14_head/6808bf5/obj5/head//-1
2013-06-03T04:26:53.233 INFO:teuthology.orchestra.run.err:2013-06-03 04:27:34.949414 7fa652ef5780 0 filestore_diff diff_objects num objs mismatch (A: 2, B: 1)
2013-06-03T04:26:53.237 INFO:teuthology.orchestra.run.err:2013-06-03 04:27:34.953682 7fa652ef5780 0 test_idempotent_sequence diff found an difference
2013-06-03T04:26:53.237 INFO:teuthology.orchestra.run.err:2013-06-03 04:27:34.954077 7fa652ef5780 1 journal close a.10/journal
2013-06-03T04:26:53.238 INFO:teuthology.orchestra.run.err:2013-06-03 04:27:34.954799 7fa652ef5780 1 journal close b.10/journal
2013-06-03T04:26:53.240 INFO:teuthology.orchestra.run.out:FAIL
2013-06-03T04:26:53.240 INFO:teuthology.orchestra.run.out: see:
2013-06-03T04:26:53.240 INFO:teuthology.orchestra.run.out: a.10.fail -- leading up to failure
2013-06-03T04:26:53.240 INFO:teuthology.orchestra.run.out: a.10.recover -- journal replay
2013-06-03T04:26:53.240 INFO:teuthology.orchestra.run.out: b.10.clean -- the clean reference
2013-06-03T04:26:53.240 INFO:teuthology.orchestra.run.out:no dir provided for evidence disposal
2013-06-03T04:26:53.241 INFO:teuthology.orchestra.run.err:./run_seed_to_range.sh: 15: ./run_seed_to_range.sh: d: not found
2013-06-03T04:26:53.241 DEBUG:teuthology.orchestra.run:Running [10.214.131.21]: 'cp -a /home/ubuntu/cephtest/30018/data/test.client.0 /home/ubuntu/cephtest/30018/archive/idempotent_failure'
2013-06-03T04:26:55.426 DEBUG:teuthology.orchestra.run:Running [10.214.131.21]: 'rm -rf -
/home/ubuntu/cephtest/30018/data/test.client.0'
2013-06-03T04:26:55.652 ERROR:teuthology.run_tasks:Saw exception from tasks
Traceback (most recent call last):
File "/var/lib/teuthworker/teuthology-master/teuthology/run_tasks.py", line 25, in run_tasks
manager = _run_one_task(taskname, ctx=ctx, config=config)
File "/var/lib/teuthworker/teuthology-master/teuthology/run_tasks.py", line 14, in _run_one_task
return fn(*kwargs)
File "/var/lib/teuthworker/teuthology-master/teuthology/task/filestore_idempotent.py", line 69, in task
raise Exception("./run_seed_to_range.sh errored out")
Exception: ./run_seed_to_range.sh errored out
2013-06-03T04:26:55.666 DEBUG:teuthology.run_tasks:Unwinding manager <contextlib.GeneratorContextManager object at 0x27f1590>
2013-06-03T04:26:55.667 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
File "/var/lib/teuthworker/teuthology-master/teuthology/contextutil.py", line 27, in nested
yield vars
File "/var/lib/teuthworker/teuthology-master/teuthology/task/ceph.py", line 1096, in task
yield
File "/var/lib/teuthworker/teuthology-master/teuthology/run_tasks.py", line 25, in run_tasks
manager = _run_one_task(taskname, ctx=ctx, config=config)
File "/var/lib/teuthworker/teuthology-master/teuthology/run_tasks.py", line 14, in _run_one_task
return fn(
*kwargs)
File "/var/lib/teuthworker/teuthology-master/teuthology/task/filestore_idempotent.py", line 69, in task
raise Exception("./run_seed_to_range.sh errored out")
Exception: ./run_seed_to_range.sh errored out
2013-06-03T04:26:55.667 INFO:teuthology.misc:Shutting down mds daemons...

ubuntu@teuthology:/a/teuthology-2013-06-03_01:00:10-rados-master-testing-basic/30018


Related issues

Duplicated by Ceph - Bug #5237: filestore idempotent tester failure Duplicate 06/03/2013

Associated revisions

Revision cdf5785c (diff)
Added by Samuel Just almost 11 years ago

test_filestore_idempotent: make newly created objects globally unique

The filestore requires hobjects to be globally unique.

Fixes: #5240
Signed-off-by: Samuel Just <>
Reviewed-by: Sage Weil <>

Revision 0bc731ea (diff)
Added by Samuel Just almost 11 years ago

test_filestore_idempotent: use obj name from source coll add

Fixes: #5240
Reviewed-by: David Zafman <>
Signed-off-by: Samuel Just <>

Revision 5dd1d4b1 (diff)
Added by David Zafman over 10 years ago

test: idempotent filestore test failure

Remove obsolete use of collection_move()
Allow operations to be skipped if random selections don't make sense
Track total number of possible objects in m_num_objects
BUG: do_remove() was calling _do_touch() instead of _do_remove()
For ops that require an object, select from among existing objects in collection
Initialize m_num_objects unique objects across collections

touch: don't create an object that already exists in another collection
remove: Use remove_obj() to clear object from m_objects to have accurate tracking
clone/clone_range(): Select 2 existing objects in the collection
add: Skip operation if selected target object name exists in target collection
move: Removed this buggy operation that is only present for upgrades

Fixes: #5371
Fixes: #5240

Signed-off-by: David Zafman <>

History

#1 Updated by Samuel Just almost 11 years ago

Looks like the tester will place objects with the same name into different collections, fixing test.

#2 Updated by Samuel Just almost 11 years ago

  • Status changed from In Progress to Resolved

#3 Updated by Samuel Just almost 11 years ago

  • Status changed from Resolved to 12

2013-06-05T04:21:14.657 INFO:teuthology.orchestra.run.err:2013-06-05 04:21:58.389650 7faabeffd700 10 filestore(b) truncate meta/718608c/txn/head//-1 size 0 = 0
2013-06-05T04:21:14.657 INFO:teuthology.orchestra.run.err:2013-06-05 04:21:58.389660 7faabeffd700 15 filestore(b) write meta/718608c/txn/head//-1 0~4
2013-06-05T04:21:14.657 INFO:teuthology.orchestra.run.err:2013-06-05 04:21:58.389703 7faabeffd700 10 filestore(b) write meta/718608c/txn/head//-1 0~4 = 4
2013-06-05T04:21:14.657 INFO:teuthology.orchestra.run.err:2013-06-05 04:21:58.389711 7faabeffd700 15 filestore(b) remove 0.19_head/cbe4aa10/obj4-2/head//-1
2013-06-05T04:21:14.657 INFO:teuthology.orchestra.run.err:2013-06-05 04:21:58.389761 7faabeffd700 10 filestore(b) remove 0.19_head/cbe4aa10/obj4-2/head//-1 = -2
2013-06-05T04:21:14.657 INFO:teuthology.orchestra.run.err:2013-06-05 04:21:58.389768 7faabeffd700 15 filestore(b) collection_add 0.19_head/cbe4aa10/obj4-2/head//-1 from 0.3_head/cbe4aa10/obj4-2/head//-1
2013-06-05T04:21:14.657 INFO:teuthology.orchestra.run.err:2013-06-05 04:21:58.389809 7faabeffd700 10 filestore(b) error opening file b/current/0.3_head/obj4-2__head_CBE4AA10__none with flags=2: (2) No such file or directory
2013-06-05T04:21:14.658 INFO:teuthology.orchestra.run.err:os/FileStore.cc: In function 'int FileStore::_collection_add(coll_t, coll_t, const hobject_t&, const SequencerPosition&)' thread 7faabeffd700 time 2013-06-05 04:21:58.389833
2013-06-05T04:21:14.658 INFO:teuthology.orchestra.run.err:os/FileStore.cc: 4577: FAILED assert(replaying)
2013-06-05T04:21:14.658 INFO:teuthology.orchestra.run.err: ceph version 0.63-352-g8c23d8c (8c23d8cb46d432ca3c840336e02a9eed868d78f3)
2013-06-05T04:21:14.658 INFO:teuthology.orchestra.run.err: 1: (FileStore::_collection_add(coll_t, coll_t, hobject_t const&, SequencerPosition const&)+0x7b6) [0x493266]
2013-06-05T04:21:14.658 INFO:teuthology.orchestra.run.err: 2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int)+0x2639) [0x496b89]
2013-06-05T04:21:14.658 INFO:teuthology.orchestra.run.err: 3: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x71) [0x49b351]
2013-06-05T04:21:14.658 INFO:teuthology.orchestra.run.err: 4: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x274) [0x49b5e4]
2013-06-05T04:21:14.658 INFO:teuthology.orchestra.run.err: 5: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4e6) [0x5107a6]
2013-06-05T04:21:14.658 INFO:teuthology.orchestra.run.err: 6: (ThreadPool::WorkThread::entry()+0x10) [0x5125d0]
2013-06-05T04:21:14.658 INFO:teuthology.orchestra.run.err: 7: (()+0x7e9a) [0x7faac8fede9a]
2013-06-05T04:21:14.658 INFO:teuthology.orchestra.run.err: 8: (clone()+0x6d) [0x7faac780accd]
2013-06-05T04:21:14.658 INFO:teuthology.orchestra.run.err: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

#4 Updated by Samuel Just almost 11 years ago

  • Status changed from 12 to Resolved

Also available in: Atom PDF