Ceph » rbd

Follow is my method to slove this problem, Wonder if there is a better already existed api to do this.
I have test this method and passed.

commit 9560f9744fa4b86a49f717fcbe85eeb976cc9a51
Author: Chen liuzhong <liuzhong.chen@easystack.cn>
Date:   Mon Nov 18 16:22:53 2019 +0800

    Add image flush when pre-release exclusive_lock

diff --git a/src/librbd/exclusive_lock/PreReleaseRequest.cc b/src/librbd/exclusive_lock/PreReleaseRequest.cc
index 504de0f..66863e8 100644
--- a/src/librbd/exclusive_lock/PreReleaseRequest.cc
+++ b/src/librbd/exclusive_lock/PreReleaseRequest.cc
@@ -219,6 +219,26 @@ void PreReleaseRequest<I>::handle_flush_notifies(int r) {
   ldout(cct, 10) << dendl;

   assert(r == 0);
+  send_image_flush();
+}
+
+template <typename I>
+void PreReleaseRequest<I>::send_image_flush(){
+  CephContext *cct = m_image_ctx.cct;
+  ldout(cct, 10) << dendl;
+
+  using klass = PreReleaseRequest<I>;
+  Context *ctx =
+    create_context_callback<klass, &klass::handle_image_flush>(this);
+  m_image_ctx.flush(ctx);
+}
+
+template <typename I>
+void PreReleaseRequest<I>::handle_image_flush(int r){
+  CephContext *cct = m_image_ctx.cct;
+  ldout(cct, 10) << dendl;
+
+  assert(r == 0);
   send_close_journal();
 }

diff --git a/src/librbd/exclusive_lock/PreReleaseRequest.h b/src/librbd/exclusive_lock/PreReleaseRequest.h
index 34ac07a..0c6051c 100644
--- a/src/librbd/exclusive_lock/PreReleaseRequest.h
+++ b/src/librbd/exclusive_lock/PreReleaseRequest.h
@@ -93,6 +93,9 @@ private:
   void send_flush_notifies();
   void handle_flush_notifies(int r);

+  void send_image_flush();
+  void handle_image_flush(int r);
+
   void send_close_journal();
   void handle_close_journal(int r);

@jesse please review,thank you!

@Jason Borden Dillaman review this please,thank you!

The reason that exclusive lock be released is two fio client libaio write the image together with, two client battle for exclusive_lock.
Pre-release state machine has block all IO in 'send_block_writes' method and io in 'io_work_queue' would not dequeue, but these ops that have been in "op_work_queue" and not be completed may cause this err.
I will test again with "debug_rbd = 20/20" and upload log latter, just a moment. Thank you.

@Jason Borden Dillaman Yes, I find the image_ctx flush in the block_writes and it works.
But journal flush_event method is async. The C_FlushJournalCommit may be put into op_work_queue after on_blocked context which cause this crash.

typename Journal<I>::Future Journal<I>::wait_event(Mutex &lock, uint64_t tid,
                                                   Context *on_safe) {
  .......
  if (event.safe) {
    // journal entry already safe
    ldout(cct, 20) << this << " " << __func__ << ": " 
                   << "journal entry already safe" << dendl;
    m_image_ctx.op_work_queue->queue(on_safe, event.ret_val);
    return Future();
  }

  event.on_safe_contexts.push_back(create_async_context_callback(m_image_ctx,
                                                                 on_safe));
  return event.futures.back();

So my code before won't work as well.

Can you retest this w/ the current luminous branch? I think this might have been fixed by commit ef4fdf78da2d2fb0a05e59409b26d5f9597187c0 [1]

[1] https://tracker.ceph.com/issues/40574

@Jason Borden Dillaman　I have understand and test the patch(https://github.com/ceph/ceph/pull/28773), it can slove this issue. Close this issue please, Thank you.