https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2014-09-27T08:33:45ZCeph Ceph - Bug #9610: Crash "RadosModel.h: In function 'virtual void WriteOp::_finish(TestOp::CallbackInfo*)'" in multi-version-giant-testing-basic-multi run https://tracker.ceph.com/issues/9610?journal_id=419582014-09-27T08:33:45ZYuri Weinsteinyweinste@redhat.com
<ul></ul><p>Another similar crash in job <a class="external" href="http://pulpito.front.sepia.ceph.com/teuthology-2014-09-26_23:20:01-multi-version-giant-testing-basic-multi/514671/">http://pulpito.front.sepia.ceph.com/teuthology-2014-09-26_23:20:01-multi-version-giant-testing-basic-multi/514671/</a></p>
<pre>
Assertion: ./test/osd/RadosModel.h: 829: FAILED assert(0)
ceph version 0.85-998-g3f05fbf (3f05fbf55bc94f057e04ef5640057a1c29b0d1ba)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x7f) [0x7fb24a43aecf]
2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x318) [0x419ee8]
3: (write_callback(void*, void*)+0x21) [0x4278b1]
4: (librados::C_AioSafe::finish(int)+0x1d) [0x7fb24a39c11d]
5: (Context::complete(int)+0x9) [0x7fb24a3786e9]
6: (Finisher::finisher_thread_entry()+0x160) [0x7fb24a43a0c0]
7: (()+0x7e9a) [0x7fb249fdae9a]
8: (clone()+0x6d) [0x7fb2497f231d]
</pre> Ceph - Bug #9610: Crash "RadosModel.h: In function 'virtual void WriteOp::_finish(TestOp::CallbackInfo*)'" in multi-version-giant-testing-basic-multi run https://tracker.ceph.com/issues/9610?journal_id=419622014-09-27T11:39:10ZSage Weilsage@newdream.net
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Resolved</i></li></ul><p>pushed fix to dumpling branch, <a class="changeset" title="osd/ReplicatedPG: respect RWORDERED rados flag If this flag is set, we need to order reads as wr..." href="https://tracker.ceph.com/projects/ceph/repository/revisions/503f865d6432bead72aac0ffba0539d807f078c4">503f865d6432bead72aac0ffba0539d807f078c4</a></p> Ceph - Bug #9610: Crash "RadosModel.h: In function 'virtual void WriteOp::_finish(TestOp::CallbackInfo*)'" in multi-version-giant-testing-basic-multi run https://tracker.ceph.com/issues/9610?journal_id=420972014-09-30T08:40:54ZYuri Weinsteinyweinste@redhat.com
<ul></ul><p>9/30/14 update - Still waiting in queue <a class="external" href="http://pulpito.front.sepia.ceph.com/teuthology-2014-09-29_23:20:02-multi-version-giant-testing-basic-multi/">http://pulpito.front.sepia.ceph.com/teuthology-2014-09-29_23:20:02-multi-version-giant-testing-basic-multi/</a></p> Ceph - Bug #9610: Crash "RadosModel.h: In function 'virtual void WriteOp::_finish(TestOp::CallbackInfo*)'" in multi-version-giant-testing-basic-multi run https://tracker.ceph.com/issues/9610?journal_id=428332014-10-09T08:08:59ZYuri Weinsteinyweinste@redhat.com
<ul><li><strong>Project</strong> changed from <i>teuthology</i> to <i>Ceph</i></li></ul> Ceph - Bug #9610: Crash "RadosModel.h: In function 'virtual void WriteOp::_finish(TestOp::CallbackInfo*)'" in multi-version-giant-testing-basic-multi run https://tracker.ceph.com/issues/9610?journal_id=428352014-10-09T08:11:41ZYuri Weinsteinyweinste@redhat.com
<ul></ul><p>Still an issue: <a class="external" href="http://pulpito.front.sepia.ceph.com/teuthology-2014-10-08_23:20:03-multi-version-giant-distro-basic-multi/534751">http://pulpito.front.sepia.ceph.com/teuthology-2014-10-08_23:20:03-multi-version-giant-distro-basic-multi/534751</a></p>
<pre>
Assertion: ./test/osd/RadosModel.h: 829: FAILED assert(0)
ceph version 0.86-49-g3bfb5fa (3bfb5fab41b6247259183c3f52c786e35beb3b01)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x7f) [0x7f98623f358f]
2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x318) [0x419ee8]
3: (write_callback(void*, void*)+0x21) [0x4278c1]
4: (librados::C_AioSafe::finish(int)+0x1d) [0x7f98623541ed]
5: (Context::complete(int)+0x9) [0x7f9862330739]
6: (Finisher::finisher_thread_entry()+0x160) [0x7f98623f2780]
7: (()+0x7e9a) [0x7f9861f92e9a]
8: (clone()+0x6d) [0x7f98617a93fd]
</pre> Ceph - Bug #9610: Crash "RadosModel.h: In function 'virtual void WriteOp::_finish(TestOp::CallbackInfo*)'" in multi-version-giant-testing-basic-multi run https://tracker.ceph.com/issues/9610?journal_id=428402014-10-09T08:28:49ZIan Colleicolle@redhat.com
<ul><li><strong>Status</strong> changed from <i>Resolved</i> to <i>12</i></li><li><strong>Assignee</strong> set to <i>Samuel Just</i></li></ul> Ceph - Bug #9610: Crash "RadosModel.h: In function 'virtual void WriteOp::_finish(TestOp::CallbackInfo*)'" in multi-version-giant-testing-basic-multi run https://tracker.ceph.com/issues/9610?journal_id=429932014-10-13T08:17:26ZSage Weilsage@newdream.net
<ul><li><strong>Priority</strong> changed from <i>Normal</i> to <i>Urgent</i></li></ul> Ceph - Bug #9610: Crash "RadosModel.h: In function 'virtual void WriteOp::_finish(TestOp::CallbackInfo*)'" in multi-version-giant-testing-basic-multi run https://tracker.ceph.com/issues/9610?journal_id=434782014-10-21T12:53:49ZTamilarasi muthamizhantamil.muthamizhan@inktank.com
<ul></ul><p>seeing this on the upgrade test from v0.67.11 to firefly [v0.80.7]</p>
<pre>
logs: ubuntu@teuthology:/a/teuthology-2014-10-20_18:40:02-upgrade:firefly:older-firefly-distro-basic-vps/561549
</pre> Ceph - Bug #9610: Crash "RadosModel.h: In function 'virtual void WriteOp::_finish(TestOp::CallbackInfo*)'" in multi-version-giant-testing-basic-multi run https://tracker.ceph.com/issues/9610?journal_id=435002014-10-21T13:44:17ZSage Weilsage@newdream.net
<ul><li><strong>Priority</strong> changed from <i>Urgent</i> to <i>Immediate</i></li></ul> Ceph - Bug #9610: Crash "RadosModel.h: In function 'virtual void WriteOp::_finish(TestOp::CallbackInfo*)'" in multi-version-giant-testing-basic-multi run https://tracker.ceph.com/issues/9610?journal_id=435152014-10-21T14:06:10ZTamilarasi muthamizhantamil.muthamizhan@inktank.com
<ul></ul><p>also: ubuntu@teuthology:/a/teuthology-2014-10-20_18:40:02-upgrade:firefly:older-firefly-distro-basic-vps/561550</p>
<pre>
2014-10-21T02:57:02.430 INFO:tasks.rados.rados.0.vpm154.stderr:Error: finished tid 1 when last_acked_tid was 4
2014-10-21T02:57:02.431 INFO:tasks.rados.rados.0.vpm154.stderr:./test/osd/RadosModel.h: In function 'virtual void WriteOp::_finish(TestOp::CallbackInfo*)' thread 7f1790f46700 time 2014-10-21 09:57:02.428940
2014-10-21T02:57:02.431 INFO:tasks.rados.rados.0.vpm154.stderr:./test/osd/RadosModel.h: 828: FAILED assert(0)
2014-10-21T02:57:02.434 INFO:tasks.rados.rados.0.vpm154.stderr: ceph version 0.80.7-73-g5a10b95 (5a10b95f7968ecac1f2af4abf9fb91347a290544)
2014-10-21T02:57:02.435 INFO:tasks.rados.rados.0.vpm154.stderr: 1: (WriteOp::_finish(TestOp::CallbackInfo*)+0x318) [0x41a058]
2014-10-21T02:57:02.435 INFO:tasks.rados.rados.0.vpm154.stderr: 2: (write_callback(void*, void*)+0x21) [0x427c21]
2014-10-21T02:57:02.435 INFO:tasks.rados.rados.0.vpm154.stderr: 3: (librados::C_AioSafe::finish(int)+0x1d) [0x7f1797cd89ad]
2014-10-21T02:57:02.435 INFO:tasks.rados.rados.0.vpm154.stderr: 4: (Context::complete(int)+0x9) [0x7f1797cb5f69]
2014-10-21T02:57:02.436 INFO:tasks.rados.rados.0.vpm154.stderr: 5: (Finisher::finisher_thread_entry()+0x1c0) [0x7f1797d69f30]
2014-10-21T02:57:02.436 INFO:tasks.rados.rados.0.vpm154.stderr: 6: (()+0x7e9a) [0x7f179791de9a]
2014-10-21T02:57:02.436 INFO:tasks.rados.rados.0.vpm154.stderr: 7: (clone()+0x6d) [0x7f179713531d]
2014-10-21T02:57:02.437 INFO:tasks.rados.rados.0.vpm154.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2014-10-21T02:57:02.438 INFO:tasks.rados.rados.0.vpm154.stderr:terminate called after throwing an instance of 'ceph::FailedAssertion'
2014-10-21T02:57:02.508 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 82, in __exit__
for result in self:
File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
resurrect_traceback(result)
File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/teuthology_master/teuthology/task/parallel.py", line 50, in _run_spawned
mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 41, in run_one_task
return fn(**kwargs)
File "/home/teuthworker/src/teuthology_master/teuthology/task/sequential.py", line 55, in task
mgr.__exit__(*exc_info)
File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/var/lib/teuthworker/src/ceph-qa-suite_firefly/tasks/rados.py", line 170, in task
running.get()
File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 331, in get
raise self._exception
CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --op read 100 --op write 100 --op delete 50 --max-ops 2000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op snap_create 50 --op snap_remove 50 --op rollback 50 --pool unique_pool_0'
2014-10-21T02:57:02.510 ERROR:teuthology.run_tasks:Saw exception from tasks.
</pre> Ceph - Bug #9610: Crash "RadosModel.h: In function 'virtual void WriteOp::_finish(TestOp::CallbackInfo*)'" in multi-version-giant-testing-basic-multi run https://tracker.ceph.com/issues/9610?journal_id=435162014-10-21T14:06:45ZSamuel Justsjust@redhat.com
<ul><li><strong>Assignee</strong> changed from <i>Samuel Just</i> to <i>Tamilarasi muthamizhan</i></li><li><strong>Priority</strong> changed from <i>Immediate</i> to <i>High</i></li></ul><p>New ceph_test_rados is too picky for dumpling osds. We only want to use dumpling ceph_test_rados against clusters with dumpling osds.</p> Ceph - Bug #9610: Crash "RadosModel.h: In function 'virtual void WriteOp::_finish(TestOp::CallbackInfo*)'" in multi-version-giant-testing-basic-multi run https://tracker.ceph.com/issues/9610?journal_id=435432014-10-21T17:16:43ZTamilarasi muthamizhantamil.muthamizhan@inktank.com
<ul><li><strong>Status</strong> changed from <i>12</i> to <i>Resolved</i></li></ul><p>fixed in multi-version suite already - commit b966da7b71c8aee22ff8e58b3b0c105b1d7ca4bf</p>
<p>fixed in upgrade:firefly/older suite now: 2687b3d8996a4985fc790aec16af25fe09fbebb3</p> Ceph - Bug #9610: Crash "RadosModel.h: In function 'virtual void WriteOp::_finish(TestOp::CallbackInfo*)'" in multi-version-giant-testing-basic-multi run https://tracker.ceph.com/issues/9610?journal_id=623702015-12-02T01:22:15Zbo cai
<ul></ul><p>can you give us a github commit link about how you resolved this?<br />the commit sha1 you said can not found.</p>