Project

General

Profile

Bug #8045

osd: deadlock from osdmap feature update

Added by Sage Weil almost 10 years ago. Updated almost 10 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
OSD
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

ubuntu@teuthology:/var/lib/teuthworker/archive/teuthology-2014-04-08_02:30:14-rados-firefly-distro-basic-plana/178670

Thread 62 (Thread 0x7f7d49a02700 (LWP 18026)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000728e21 in Wait (mutex=..., this=0x7f7d499ffed0) at ./common/Cond.h:55
#2  ObjectStore::apply_transactions (this=<optimized out>, osr=0x0, tls=..., ondisk=<optimized out>) at os/ObjectStore.cc:92
#3  0x000000000067789e in ObjectStore::apply_transaction (this=0x3181000, t=..., ondisk=0x0) at ./os/ObjectStore.h:1084
#4  0x000000000063036c in OSD::check_osdmap_features (this=0x31b5000, fs=0x3181000) at osd/OSD.cc:5672
#5  0x000000000066b461 in OSD::handle_osd_map (this=0x31b5000, m=0x3302900) at osd/OSD.cc:5602
#6  0x000000000066e39b in OSD::_dispatch (this=0x31b5000, m=0x3302900) at osd/OSD.cc:4939
#7  0x000000000066ea9e in OSD::ms_dispatch (this=0x31b5000, m=0x3302900) at osd/OSD.cc:4726
#8  0x0000000000b0cd29 in ms_deliver_dispatch (m=0x3302900, this=0x3193000) at msg/Messenger.h:587
#9  DispatchQueue::entry (this=0x31930e8) at msg/DispatchQueue.cc:124
#10 0x0000000000a49b6d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104
#11 0x00007f7d57a9be9a in start_thread (arg=0x7f7d49a02700) at pthread_create.c:308
#12 0x00007f7d5605c3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#13 0x0000000000000000 in ?? ()

vs

Thread 64 (Thread 0x7f7d4cb36700 (LWP 17971)):
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
#1  0x00007f7d57a9e065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f7d57a9deba in __pthread_mutex_lock (mutex=0x31b5028) at pthread_mutex_lock.c:61
#3  0x0000000000a1fcd3 in Mutex::Lock (this=0x31b5018, no_lockdep=<optimized out>) at common/Mutex.cc:89
#4  0x00000000006a4152 in Locker (m=..., this=<synthetic pointer>) at ./common/Mutex.h:120
#5  C_CompleteSplits::finish (this=0x320fa80, r=<optimized out>) at osd/OSD.cc:7654
#6  0x000000000066ec59 in Context::complete (this=0x320fa80, r=<optimized out>) at ./include/Context.h:64
#7  0x000000000067638d in finish_contexts (cct=0x316f230, finished=..., result=0) at ./include/Context.h:120
#8  0x000000000066ec59 in Context::complete (this=0x333fc80, r=<optimized out>) at ./include/Context.h:64
#9  0x00000000009a6c30 in Finisher::finisher_thread_entry (this=0x3181c60) at common/Finisher.cc:56
#10 0x00007f7d57a9be9a in start_thread (arg=0x7f7d4cb36700) at pthread_create.c:308
#11 0x00007f7d5605c3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#12 0x0000000000000000 in ?? ()

Associated revisions

Revision 22a0c1fd (diff)
Added by Sage Weil almost 10 years ago

osd: do not block when updating osdmap superblock features

We are holding osd_lock in check_osdmap_features, which means we cannot
block while waiting for filestore operations to flush/apply without
risking deadlock.

The important constraint is that we commit that the feature is enabled
before also commiting anything that utilizes sharded objects. The normal
commit sequencing does that already; there is no reason to block here.

Fixes: #8045
Signed-off-by: Sage Weil <>

History

#1 Updated by Sage Weil almost 10 years ago

  • Status changed from In Progress to Fix Under Review
  • Assignee deleted (Sage Weil)

#2 Updated by Sage Weil almost 10 years ago

ubuntu@teuthology:/var/lib/teuthworker/archive/teuthology-2014-04-08_02:30:14-rados-firefly-distro-basic-plana/178740

#3 Updated by Sage Weil almost 10 years ago

  • Status changed from Fix Under Review to Resolved

Also available in: Atom PDF