https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2019-06-24T12:49:09Z
Ceph
rbd - Bug #40519: rbd_mirror/ImageSyncThrottler.cc: 61: FAILED ceph_assert(m_queue.empty())
https://tracker.ceph.com/issues/40519?journal_id=139523
2019-06-24T12:49:09Z
Mykola Golub
mgolub@suse.com
<ul></ul><p>The test case that reproduces the assertion failure:<br /><pre>
TEST_F(TestMockImageSyncThrottler, Duplicate) {
MockImageSyncThrottler throttler(g_ceph_context);
throttler.set_max_concurrent_syncs(2);
C_SaferCond on_start1;
throttler.start_op("id1", &on_start1); // req (1) in flight (in_flight.size() == 1)
C_SaferCond on_start2;
throttler.start_op("id2", &on_start2); // req (2) in flight (in_flight.size() == 2)
C_SaferCond on_start3;
throttler.start_op("id3", &on_start3); // req (3) queued (m_queue.size() == 1)
C_SaferCond on_start4;
throttler.start_op("id3", &on_start4); // req (4) (dup!) queued (m_queue.size() == 2)
C_SaferCond on_start5;
throttler.start_op("id4", &on_start5); // req (5) queued (m_queue.size() == 3)
throttler.finish_op("id1"); // req (3) in flight (in_flight.size() == 2, m_queue.size() == 2)
ASSERT_EQ(0, on_start1.wait());
throttler.finish_op("id2"); // req (4) in flight (in_flight.size() == 1 due to dublicate!, m_queue.size() == 1)
ASSERT_EQ(0, on_start1.wait());
C_SaferCond on_start6;
throttler.start_op("id6", &on_start6); // FAILED ceph_assert(m_queue.empty())
throttler.finish_op("id3");
ASSERT_EQ(0, on_start3.wait());
ASSERT_EQ(-ECANCELED, on_start4.wait());
throttler.finish_op("id4");
ASSERT_EQ(0, on_start5.wait());
throttler.finish_op("id5");
ASSERT_EQ(0, on_start6.wait());
}
</pre></p>
<p>And in the logs one can see many duplicates at that moment (e.g. 10c5c813f2d7), which I believe are due to the trasher:<br /><pre>
trociny@teuthology:/a/trociny-2019-06-21_06:39:59-rbd-wip-mgolub-testing-distro-basic-smithi/4055998/remote/smithi039/log$ zgrep '^20.*handle_sync_request: instance_id=4651, sync_id=' cluster2-client.mirror.2.38780.log.gz
2019-06-23T07:56:49.284+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c9ddb5e0a
2019-06-23T07:56:49.288+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=1107334ad7fb
2019-06-23T07:56:49.288+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c9b4dfcd59
2019-06-23T07:56:49.288+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10be37fff151
2019-06-23T07:56:49.288+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=122bc575f727
2019-06-23T07:56:49.288+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c5c813f2d7
2019-06-23T07:56:49.292+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=110746577dca
2019-06-23T07:56:49.292+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c517ed9396
2019-06-23T07:56:49.292+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c939a2aa83
2019-06-23T07:56:49.292+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c916a6fb2
2019-06-23T07:56:49.292+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10be13accbcd
2019-06-23T07:56:52.024+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c5c813f2d7
2019-06-23T07:56:52.024+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=110746577dca
2019-06-23T07:56:52.024+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c517ed9396
2019-06-23T07:56:52.024+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c939a2aa83
2019-06-23T07:56:52.024+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c916a6fb2
2019-06-23T07:56:52.024+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10be13accbcd
2019-06-23T07:56:52.024+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c5c813f2d7
2019-06-23T07:56:52.024+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=110746577dca
2019-06-23T07:56:52.024+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c517ed9396
2019-06-23T07:56:52.028+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c939a2aa83
2019-06-23T07:56:52.028+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c916a6fb2
2019-06-23T07:56:52.028+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c5c813f2d7
2019-06-23T07:56:52.028+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=110746577dca
2019-06-23T07:56:52.028+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c517ed9396
2019-06-23T07:56:52.028+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c939a2aa83
2019-06-23T07:56:52.028+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c916a6fb2
2019-06-23T07:56:52.028+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c5c813f2d7
2019-06-23T07:56:52.028+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=110746577dca
2019-06-23T07:56:52.028+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c517ed9396
2019-06-23T07:56:52.028+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c939a2aa83
2019-06-23T07:56:52.028+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c916a6fb2
2019-06-23T07:56:52.340+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10be13accbcd
2019-06-23T07:56:52.340+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10be13accbcd
2019-06-23T07:56:52.948+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=110746577dca
2019-06-23T07:56:52.960+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10be13accbcd
2019-06-23T07:56:52.968+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c916a6fb2
2019-06-23T07:56:52.992+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c517ed9396
2019-06-23T07:56:52.992+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c5c813f2d7
2019-06-23T07:56:53.028+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=122bc575f727
2019-06-23T07:56:53.036+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10be13accbcd
2019-06-23T07:56:53.036+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c939a2aa83
2019-06-23T07:56:53.452+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c9b4dfcd59
2019-06-23T07:56:53.464+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=1107334ad7fb
2019-06-23T07:56:53.484+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c9ddb5e0a
2019-06-23T07:56:53.520+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10be37fff151
2019-06-23T07:56:56.596+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c916a6fb2
2019-06-23T07:56:56.596+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c517ed9396
2019-06-23T07:56:56.596+0000 7f5c1aad7700 10 rbd::mirror::InstanceWatcher: 0x55c7ed562000 handle_sync_request: instance_id=4651, sync_id=10c5c813f2d7
</pre></p>
rbd - Bug #40519: rbd_mirror/ImageSyncThrottler.cc: 61: FAILED ceph_assert(m_queue.empty())
https://tracker.ceph.com/issues/40519?journal_id=139553
2019-06-25T04:49:19Z
Mykola Golub
mgolub@suse.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Fix Under Review</i></li><li><strong>Pull request ID</strong> set to <i>28730</i></li></ul>
rbd - Bug #40519: rbd_mirror/ImageSyncThrottler.cc: 61: FAILED ceph_assert(m_queue.empty())
https://tracker.ceph.com/issues/40519?journal_id=139756
2019-06-27T17:40:14Z
Jason Dillaman
dillaman@redhat.com
<ul><li><strong>Status</strong> changed from <i>Fix Under Review</i> to <i>Pending Backport</i></li><li><strong>Backport</strong> set to <i>luminous,mimic,nautilus</i></li></ul>
rbd - Bug #40519: rbd_mirror/ImageSyncThrottler.cc: 61: FAILED ceph_assert(m_queue.empty())
https://tracker.ceph.com/issues/40519?journal_id=139828
2019-06-30T08:43:27Z
Nathan Cutler
ncutler@suse.cz
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-4 priority-default closed" href="/issues/40592">Backport #40592</a>: luminous: rbd_mirror/ImageSyncThrottler.cc: 61: FAILED ceph_assert(m_queue.empty())</i> added</li></ul>
rbd - Bug #40519: rbd_mirror/ImageSyncThrottler.cc: 61: FAILED ceph_assert(m_queue.empty())
https://tracker.ceph.com/issues/40519?journal_id=139830
2019-06-30T08:43:34Z
Nathan Cutler
ncutler@suse.cz
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-4 priority-default closed" href="/issues/40593">Backport #40593</a>: mimic: rbd_mirror/ImageSyncThrottler.cc: 61: FAILED ceph_assert(m_queue.empty())</i> added</li></ul>
rbd - Bug #40519: rbd_mirror/ImageSyncThrottler.cc: 61: FAILED ceph_assert(m_queue.empty())
https://tracker.ceph.com/issues/40519?journal_id=139832
2019-06-30T08:43:41Z
Nathan Cutler
ncutler@suse.cz
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-4 priority-default closed" href="/issues/40594">Backport #40594</a>: nautilus: rbd_mirror/ImageSyncThrottler.cc: 61: FAILED ceph_assert(m_queue.empty())</i> added</li></ul>
rbd - Bug #40519: rbd_mirror/ImageSyncThrottler.cc: 61: FAILED ceph_assert(m_queue.empty())
https://tracker.ceph.com/issues/40519?journal_id=140829
2019-07-16T15:44:43Z
Yuri Weinstein
yweinste@redhat.com
<ul></ul><p>merged <a class="external" href="https://github.com/ceph/ceph/pull/28815">https://github.com/ceph/ceph/pull/28815</a></p>
rbd - Bug #40519: rbd_mirror/ImageSyncThrottler.cc: 61: FAILED ceph_assert(m_queue.empty())
https://tracker.ceph.com/issues/40519?journal_id=147497
2019-10-02T13:15:24Z
Nathan Cutler
ncutler@suse.cz
<ul><li><strong>Status</strong> changed from <i>Pending Backport</i> to <i>Resolved</i></li></ul><p>While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved".</p>