Ceph

Looking at initializations of CephContext, I don't see any explicit freeing, which begs the question which thread is running ~CephContext()? If it happens to be the context thread, then it will be locking itself via exit_thread(). I'm not sure exactly what guarantees (if any) there are on who runs destructors on exit.

martini:ceph nwatkins$ git grep delete | grep cct
src/include/Context.h:    mydout(cct,10) << "C_Gather " << this << ".delete" << dendl;
src/librbd/internal.cc:      ldout(cct, 2) << "trim_image objects " << delete_start << " to " 
src/msg/SimpleMessenger.cc:    ldout(cct,10) << "reaper deleted pipe " << p << dendl;
src/osdc/Objecter.cc:  ldout(cct, 10) << "delete_pool_snap; pool: " << pool << "; snap: " << snap_name << dendl;
src/osdc/Objecter.cc:  ldout(cct, 10) << "delete_selfmanaged_snap; pool: " << pool << "; snap: " 
src/osdc/Objecter.cc:  ldout(cct, 10) << "delete_pool " << pool << dendl;

I don't think the theory from the previous comment has any merit! I wonder if it's possible that lockdep is missing an unlock, especially since g_lockdep isn't read under any type of mutex, although i'm failing to come up with any type of interleaving that would cause this.

diff --git a/src/common/ceph_context.cc b/src/common/ceph_context.cc
index 7822407..d0b4bc8 100644
--- a/src/common/ceph_context.cc
+++ b/src/common/ceph_context.cc
@@ -296,12 +296,12 @@ CephContext::CephContext(uint32_t module_type_)

 CephContext::~CephContext()
 {
+  join_service_thread();
+
   if (_conf->lockdep) {
     lockdep_unregister_ceph_context(this);
   }

-  join_service_thread();
-
   _admin_socket->unregister_command("perfcounters_dump");
   _admin_socket->unregister_command("perf dump");
   _admin_socket->unregister_command("1");

c7d975aeadf908d11577c480fa0a2e831d069c55 is the one I was discussing.

Has this been triggered again since the first time we saw it?

I just saw it a couple more times going over fs nightlies from this week, and it broke several rados tests last night (see #6738).

This is my theory. When CephContext is being cleaned-up, we first disable lockdep, then wait for the service thread to exit.

293 CephContext::~CephContext()                                                                                                                                                                                                                                               
294 {                                                                                                                                                                                                                                                                         
295   if (_conf->lockdep) {                                                                                                                                                                                                                                                   
296     lockdep_unregister_ceph_context(this);                                                                                                                                                                                                                                
297   }                                                                                                                                                                                                                                                                       
298                                                                                                                                                                                                                                                                           
299   join_service_thread();

When the context is unregistered, it occurs under a mutex.

 67 void lockdep_unregister_ceph_context(CephContext *cct)                                                                                                                                                                                                                    
 68 {                                                                                                                                                                                                                                                                         
 69   pthread_mutex_lock(&lockdep_mutex);                                                                                                                                                                                                                                     
 70   if (cct == g_lockdep_ceph_ctx) {                                                                                                                                                                                                                                        
 71     // this cct is going away; shut it down!                                                                                                                                                                                                                              
 72     g_lockdep = false;                                                                                                                                                                                                                                                    
 73     g_lockdep_ceph_ctx = NULL;                                                                                                                                                                                                                                            
 74   }                                                                                                                                                                                                                                                                       
 75   pthread_mutex_unlock(&lockdep_mutex);                                                                                                                                                                                                                                   
 76 } 

But, the g_lockdep flag is not checked under the mutex.

 79 void Mutex::Lock(bool no_lockdep) {                                                                                                                                                                                                                                       
 80   if (lockdep && g_lockdep && !no_lockdep) _will_lock();

So, if the context thread is migrated between CPUs, I don't think there is any guarantee that x86 makes that the cache lines will be updated. So, at first glance this looks like it is entirely possible that the service thread could see multiple values for g_lockdep, corrupting the lockdep state?