Actions
Bug #23891
closedunable to perform a "rbd-nbd map" without forgroud flag
Status:
Duplicate
Priority:
Normal
Assignee:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
Yes
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
rbd
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
I would like to map a rbd using rbd-nbd. Without adding the foreground flag it is not possible to map the device.
The command listed beyond, are executed on a mon server (same thing happens on other servers)
Not working scenario: running rbd-nbd as background proccess¶
Shell 1:
# rmmod nbd
rmmod: ERROR: Module nbd is not currently loaded
# rbd-nbd map --nbds_max 16 RBD_XenStorage-07449252-bf96-4daa-b0a6-687b7f1c369c/RBD-17efad10-6807-433f-879a-c59b511f895c
-> it blocks probably forever
I attached a strace of the problem situation
# strace -frvT -s128 -o /tmp/fail.strace rbd-nbd map --nbds_max 16 RBD_XenStorage-07449252-bf96-4daa-b0a6-687b7f1c369c/RBD-17efad10-6807-433f-879a-c59b511f895c
Shell 2:
# rbd-nbd list-mapped
-> no output
# lsmod |grep nbd
-> no output
Working scenario running rbd-nbd in foreground:¶
Shell 1:
# rbd-nbd map -d --nbds_max 16 RBD_XenStorage-07449252-bf96-4daa-b0a6-687b7f1c369c/RBD-17efad10-6807-433f-879a-c59b511f895c
2018-04-26 23:22:19.681903 7fde4b442e80 0 ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable), process (unknown), pid 326958
2018-04-26 23:22:19.681919 7fde4b442e80 0 pidfile_write: ignore empty --pid-file
/dev/nbd0
-> Blocking, because its running as foreeground proccess
Shell 2:
# rbd-nbd list-mapped
pid pool image snap device
326958 RBD_XenStorage-07449252-bf96-4daa-b0a6-687b7f1c369c RBD-17efad10-6807-433f-879a-c59b511f895c - /dev/nbd0
# blockdev --getsize64 /dev/nbd0
21474836480
# blockdev --setra 1024 /dev/nbd0
# blockdev --getra /dev/nbd0
1024
# rbd-nbd unmap /dev/nbd0
Shell 1:
-> terminates in unmap
2018-04-26 23:24:55.660478 7fdded7ba700 0 rbd-nbd: disconnect request received
Our System:¶
- Luminous/12.2.5
- Ubuntu 16.04
- 5 OSD Nodes (24*8 TB HDD OSDs, 48*1TB SSD OSDS, Bluestore, 6Gb Cache per OSD)
- Size per OSD, 192GB RAM, 56 HT CPUs)
- 3 Mons (64 GB RAM, 200GB SSD, 4 visible CPUs)
- 2 * 10 GBIT, SFP+, bonded xmit_hash_policy layer3+4
Files
Updated by Jason Dillaman about 6 years ago
- Status changed from New to Need More Info
@Marc: we run several NBD tests daily against the Xenial kernel (each involving mapping/unmapping dozens of images). Can you please install the debuginfo packages, run the daemon until it hangs, attach dbg, and run "thread apply all bt" and attach the results?
Updated by Marc Schöchlin about 6 years ago
# apt-get install rbd-nbd-dbg gdb -y
# rmmod nbd
rmmod: ERROR: Module nbd is not currently loaded
# pstree -p |grep -A 15 nbd
|-sshd(2362)-+-sshd(454585)---bash(454667)---rbd-nbd(459322)-+-rbd-nbd(459324)-+-{admin_socket}(459327)
| | | |-{flusher}(459341)
| | | |-{flusher}(459345)
| | | |-{flusher}(459347)
| | | |-{flusher}(459349)
| | | |-{flusher}(459351)
| | | |-{flusher}(459353)
| | | |-{flusher}(459355)
| | | |-{flusher}(459357)
| | | |-{flusher}(459359)
| | | |-{flusher}(459361)
| | | |-{flusher}(459363)
| | | |-{flusher}(459365)
| | | |-{flusher}(459367)
| | | |-{flusher}(459369)
| | | |-{flusher}(459371)
--
| | | |-{rbd-nbd}(459331)
| | | |-{safe_timer}(459334)
| | | |-{safe_timer}(459336)
| | | |-{safe_timer}(459337)
| | | |-{safe_timer}(459342)
| | | |-{service}(459326)
| | | |-{taskfin_librbd}(459343)
| | | `-{tp_librbd}(459339)
| | `-{log}(459325)
| `-sshd(458502)---bash(458540)-+-grep(459792)
| `-pstree(459791)
|-systemd(308611)---(sd-pam)(308612)
|-systemd-journal(453)
|-systemd-logind(1684)
|-systemd-udevd(492)
# gdb -p 459324
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185 ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or directory.
(gdb) thread apply all bt
Thread 59 (Thread 0x7f589f7d6700 (LWP 459383)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1 0x00007f58e5d110a5 in ?? () from /usr/lib/librbd.so.1
#2 0x00007f58e5d1e6cd in ?? () from /usr/lib/librbd.so.1
#3 0x00007f58dc86c6ba in start_thread (arg=0x7f589f7d6700) at pthread_create.c:333
#4 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 58 (Thread 0x7f589ffd7700 (LWP 459382)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd14469 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dc86c6ba in start_thread (arg=0x7f589ffd7700) at pthread_create.c:333
#3 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 57 (Thread 0x7f58a07d8700 (LWP 459381)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1 0x00007f58e5d110a5 in ?? () from /usr/lib/librbd.so.1
#2 0x00007f58e5d1e6cd in ?? () from /usr/lib/librbd.so.1
#3 0x00007f58dc86c6ba in start_thread (arg=0x7f58a07d8700) at pthread_create.c:333
#4 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 56 (Thread 0x7f58a0fd9700 (LWP 459380)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd14469 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dc86c6ba in start_thread (arg=0x7f58a0fd9700) at pthread_create.c:333
#3 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 55 (Thread 0x7f58a17da700 (LWP 459379)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1 0x00007f58e5d110a5 in ?? () from /usr/lib/librbd.so.1
#2 0x00007f58e5d1e6cd in ?? () from /usr/lib/librbd.so.1
#3 0x00007f58dc86c6ba in start_thread (arg=0x7f58a17da700) at pthread_create.c:333
#4 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 54 (Thread 0x7f58a1fdb700 (LWP 459378)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd14469 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dc86c6ba in start_thread (arg=0x7f58a1fdb700) at pthread_create.c:333
#3 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 53 (Thread 0x7f58a27dc700 (LWP 459377)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1 0x00007f58e5d110a5 in ?? () from /usr/lib/librbd.so.1
#2 0x00007f58e5d1e6cd in ?? () from /usr/lib/librbd.so.1
#3 0x00007f58dc86c6ba in start_thread (arg=0x7f58a27dc700) at pthread_create.c:333
#4 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 52 (Thread 0x7f58a2fdd700 (LWP 459376)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd14469 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dc86c6ba in start_thread (arg=0x7f58a2fdd700) at pthread_create.c:333
#3 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 51 (Thread 0x7f58a37de700 (LWP 459375)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1 0x00007f58e5d110a5 in ?? () from /usr/lib/librbd.so.1
---Type <return> to continue, or q <return> to quit---
#2 0x00007f58e5d1e6cd in ?? () from /usr/lib/librbd.so.1
#3 0x00007f58dc86c6ba in start_thread (arg=0x7f58a37de700) at pthread_create.c:333
#4 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 50 (Thread 0x7f58a3fdf700 (LWP 459374)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd14469 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dc86c6ba in start_thread (arg=0x7f58a3fdf700) at pthread_create.c:333
#3 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 49 (Thread 0x7f58a47e0700 (LWP 459373)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1 0x00007f58e5d110a5 in ?? () from /usr/lib/librbd.so.1
#2 0x00007f58e5d1e6cd in ?? () from /usr/lib/librbd.so.1
#3 0x00007f58dc86c6ba in start_thread (arg=0x7f58a47e0700) at pthread_create.c:333
#4 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 48 (Thread 0x7f58a4fe1700 (LWP 459372)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd14469 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dc86c6ba in start_thread (arg=0x7f58a4fe1700) at pthread_create.c:333
#3 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 47 (Thread 0x7f58a57e2700 (LWP 459371)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1 0x00007f58e5d110a5 in ?? () from /usr/lib/librbd.so.1
#2 0x00007f58e5d1e6cd in ?? () from /usr/lib/librbd.so.1
#3 0x00007f58dc86c6ba in start_thread (arg=0x7f58a57e2700) at pthread_create.c:333
#4 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 46 (Thread 0x7f58a5fe3700 (LWP 459370)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd14469 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dc86c6ba in start_thread (arg=0x7f58a5fe3700) at pthread_create.c:333
#3 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 45 (Thread 0x7f58a67e4700 (LWP 459369)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1 0x00007f58e5d110a5 in ?? () from /usr/lib/librbd.so.1
#2 0x00007f58e5d1e6cd in ?? () from /usr/lib/librbd.so.1
#3 0x00007f58dc86c6ba in start_thread (arg=0x7f58a67e4700) at pthread_create.c:333
#4 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 44 (Thread 0x7f58a6fe5700 (LWP 459368)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd14469 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dc86c6ba in start_thread (arg=0x7f58a6fe5700) at pthread_create.c:333
#3 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 43 (Thread 0x7f58a77e6700 (LWP 459367)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1 0x00007f58e5d110a5 in ?? () from /usr/lib/librbd.so.1
#2 0x00007f58e5d1e6cd in ?? () from /usr/lib/librbd.so.1
#3 0x00007f58dc86c6ba in start_thread (arg=0x7f58a77e6700) at pthread_create.c:333
#4 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
---Type <return> to continue, or q <return> to quit---
Thread 42 (Thread 0x7f58a7fe7700 (LWP 459366)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd14469 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dc86c6ba in start_thread (arg=0x7f58a7fe7700) at pthread_create.c:333
#3 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 41 (Thread 0x7f58a87e8700 (LWP 459365)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1 0x00007f58e5d110a5 in ?? () from /usr/lib/librbd.so.1
#2 0x00007f58e5d1e6cd in ?? () from /usr/lib/librbd.so.1
#3 0x00007f58dc86c6ba in start_thread (arg=0x7f58a87e8700) at pthread_create.c:333
#4 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 40 (Thread 0x7f58a8fe9700 (LWP 459364)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd14469 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dc86c6ba in start_thread (arg=0x7f58a8fe9700) at pthread_create.c:333
#3 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 39 (Thread 0x7f58a97ea700 (LWP 459363)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1 0x00007f58e5d110a5 in ?? () from /usr/lib/librbd.so.1
#2 0x00007f58e5d1e6cd in ?? () from /usr/lib/librbd.so.1
#3 0x00007f58dc86c6ba in start_thread (arg=0x7f58a97ea700) at pthread_create.c:333
#4 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 38 (Thread 0x7f58a9feb700 (LWP 459362)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd14469 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dc86c6ba in start_thread (arg=0x7f58a9feb700) at pthread_create.c:333
#3 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 37 (Thread 0x7f58aa7ec700 (LWP 459361)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1 0x00007f58e5d110a5 in ?? () from /usr/lib/librbd.so.1
#2 0x00007f58e5d1e6cd in ?? () from /usr/lib/librbd.so.1
#3 0x00007f58dc86c6ba in start_thread (arg=0x7f58aa7ec700) at pthread_create.c:333
#4 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 36 (Thread 0x7f58aafed700 (LWP 459360)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd14469 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dc86c6ba in start_thread (arg=0x7f58aafed700) at pthread_create.c:333
#3 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 35 (Thread 0x7f58ab7ee700 (LWP 459359)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1 0x00007f58e5d110a5 in ?? () from /usr/lib/librbd.so.1
#2 0x00007f58e5d1e6cd in ?? () from /usr/lib/librbd.so.1
#3 0x00007f58dc86c6ba in start_thread (arg=0x7f58ab7ee700) at pthread_create.c:333
#4 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 34 (Thread 0x7f58abfef700 (LWP 459358)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd14469 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dc86c6ba in start_thread (arg=0x7f58abfef700) at pthread_create.c:333
---Type <return> to continue, or q <return> to quit---
#3 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 33 (Thread 0x7f58ac7f0700 (LWP 459357)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1 0x00007f58e5d110a5 in ?? () from /usr/lib/librbd.so.1
#2 0x00007f58e5d1e6cd in ?? () from /usr/lib/librbd.so.1
#3 0x00007f58dc86c6ba in start_thread (arg=0x7f58ac7f0700) at pthread_create.c:333
#4 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 32 (Thread 0x7f58acff1700 (LWP 459356)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd14469 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dc86c6ba in start_thread (arg=0x7f58acff1700) at pthread_create.c:333
#3 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 31 (Thread 0x7f58ad7f2700 (LWP 459355)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1 0x00007f58e5d110a5 in ?? () from /usr/lib/librbd.so.1
#2 0x00007f58e5d1e6cd in ?? () from /usr/lib/librbd.so.1
#3 0x00007f58dc86c6ba in start_thread (arg=0x7f58ad7f2700) at pthread_create.c:333
#4 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 30 (Thread 0x7f58adff3700 (LWP 459354)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd14469 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dc86c6ba in start_thread (arg=0x7f58adff3700) at pthread_create.c:333
#3 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 29 (Thread 0x7f58ae7f4700 (LWP 459353)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1 0x00007f58e5d110a5 in ?? () from /usr/lib/librbd.so.1
#2 0x00007f58e5d1e6cd in ?? () from /usr/lib/librbd.so.1
#3 0x00007f58dc86c6ba in start_thread (arg=0x7f58ae7f4700) at pthread_create.c:333
#4 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 28 (Thread 0x7f58aeff5700 (LWP 459352)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd14469 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dc86c6ba in start_thread (arg=0x7f58aeff5700) at pthread_create.c:333
#3 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 27 (Thread 0x7f58af7f6700 (LWP 459351)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1 0x00007f58e5d110a5 in ?? () from /usr/lib/librbd.so.1
#2 0x00007f58e5d1e6cd in ?? () from /usr/lib/librbd.so.1
#3 0x00007f58dc86c6ba in start_thread (arg=0x7f58af7f6700) at pthread_create.c:333
#4 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 26 (Thread 0x7f58afff7700 (LWP 459350)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd14469 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dc86c6ba in start_thread (arg=0x7f58afff7700) at pthread_create.c:333
#3 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 25 (Thread 0x7f58b07f8700 (LWP 459349)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
---Type <return> to continue, or q <return> to quit---
#1 0x00007f58e5d110a5 in ?? () from /usr/lib/librbd.so.1
#2 0x00007f58e5d1e6cd in ?? () from /usr/lib/librbd.so.1
#3 0x00007f58dc86c6ba in start_thread (arg=0x7f58b07f8700) at pthread_create.c:333
#4 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 24 (Thread 0x7f58b0ff9700 (LWP 459348)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd14469 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dc86c6ba in start_thread (arg=0x7f58b0ff9700) at pthread_create.c:333
#3 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 23 (Thread 0x7f58b17fa700 (LWP 459347)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1 0x00007f58e5d110a5 in ?? () from /usr/lib/librbd.so.1
#2 0x00007f58e5d1e6cd in ?? () from /usr/lib/librbd.so.1
#3 0x00007f58dc86c6ba in start_thread (arg=0x7f58b17fa700) at pthread_create.c:333
#4 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 22 (Thread 0x7f58b1ffb700 (LWP 459346)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd14469 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dc86c6ba in start_thread (arg=0x7f58b1ffb700) at pthread_create.c:333
#3 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 21 (Thread 0x7f58b27fc700 (LWP 459345)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1 0x00007f58e5d110a5 in ?? () from /usr/lib/librbd.so.1
#2 0x00007f58e5d1e6cd in ?? () from /usr/lib/librbd.so.1
#3 0x00007f58dc86c6ba in start_thread (arg=0x7f58b27fc700) at pthread_create.c:333
#4 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 20 (Thread 0x7f58b2ffd700 (LWP 459344)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd14469 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dc86c6ba in start_thread (arg=0x7f58b2ffd700) at pthread_create.c:333
#3 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 19 (Thread 0x7f58b37fe700 (LWP 459343)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd14469 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dc86c6ba in start_thread (arg=0x7f58b37fe700) at pthread_create.c:333
#3 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 18 (Thread 0x7f58b3fff700 (LWP 459342)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd11c45 in SafeTimer::timer_thread() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dcd12d8d in SafeTimerThread::entry() () from /usr/lib/ceph/libceph-common.so.0
#3 0x00007f58dc86c6ba in start_thread (arg=0x7f58b3fff700) at pthread_create.c:333
#4 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 17 (Thread 0x7f58c8ff9700 (LWP 459341)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1 0x00007f58e5d110a5 in ?? () from /usr/lib/librbd.so.1
#2 0x00007f58e5d1e6cd in ?? () from /usr/lib/librbd.so.1
#3 0x00007f58dc86c6ba in start_thread (arg=0x7f58c8ff9700) at pthread_create.c:333
#4 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
---Type <return> to continue, or q <return> to quit---
Thread 16 (Thread 0x7f58c97fa700 (LWP 459340)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd14469 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dc86c6ba in start_thread (arg=0x7f58c97fa700) at pthread_create.c:333
#3 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 15 (Thread 0x7f58c9ffb700 (LWP 459339)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd395e3 in ceph::logging::Log::submit_entry(ceph::logging::Entry*) () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dcf8d18e in ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, long) () from /usr/lib/ceph/libceph-common.so.0
#3 0x00007f58dcf8d6ac in ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, long, long) () from /usr/lib/ceph/libceph-common.so.0
#4 0x00007f58dcd1dd1e in ThreadPool::worker(ThreadPool::WorkThread*) () from /usr/lib/ceph/libceph-common.so.0
#5 0x00007f58dcd1f4b0 in ThreadPool::WorkThread::entry() () from /usr/lib/ceph/libceph-common.so.0
#6 0x00007f58dc86c6ba in start_thread (arg=0x7f58c9ffb700) at pthread_create.c:333
#7 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 14 (Thread 0x7f58ca7fc700 (LWP 459338)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd395e3 in ceph::logging::Log::submit_entry(ceph::logging::Entry*) () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dccee820 in AdminSocket::register_command(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, AdminSocketHook*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) () from /usr/lib/ceph/libceph-common.so.0
#3 0x00007f58e5b9e402 in ?? () from /usr/lib/librbd.so.1
#4 0x00007f58e5b4ae56 in ?? () from /usr/lib/librbd.so.1
#5 0x00007f58e5bef40d in ?? () from /usr/lib/librbd.so.1
#6 0x00007f58e5bf4817 in ?? () from /usr/lib/librbd.so.1
#7 0x00007f58e5bf4c2f in ?? () from /usr/lib/librbd.so.1
#8 0x00007f58e58324ad in librados::C_AioComplete::finish(int) () from /usr/lib/librados.so.2
#9 0x00007f58e5811799 in Context::complete(int) () from /usr/lib/librados.so.2
#10 0x00007f58dcd13d80 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#11 0x00007f58dc86c6ba in start_thread (arg=0x7f58ca7fc700) at pthread_create.c:333
#12 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 13 (Thread 0x7f58caffd700 (LWP 459337)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd11c45 in SafeTimer::timer_thread() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dcd12d8d in SafeTimerThread::entry() () from /usr/lib/ceph/libceph-common.so.0
#3 0x00007f58dc86c6ba in start_thread (arg=0x7f58caffd700) at pthread_create.c:333
#4 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 12 (Thread 0x7f58cb7fe700 (LWP 459336)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd11c45 in SafeTimer::timer_thread() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dcd12d8d in SafeTimerThread::entry() () from /usr/lib/ceph/libceph-common.so.0
#3 0x00007f58dc86c6ba in start_thread (arg=0x7f58cb7fe700) at pthread_create.c:333
#4 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 11 (Thread 0x7f58cbfff700 (LWP 459335)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd14469 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dc86c6ba in start_thread (arg=0x7f58cbfff700) at pthread_create.c:333
#3 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 10 (Thread 0x7f58d0ff9700 (LWP 459334)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd395e3 in ceph::logging::Log::submit_entry(ceph::logging::Entry*) () from /usr/lib/ceph/libceph-common.so.0
---Type <return> to continue, or q <return> to quit---
#2 0x00007f58dcd66d5d in MonClient::tick() () from /usr/lib/ceph/libceph-common.so.0
#3 0x00007f58dccd01b9 in Context::complete(int) () from /usr/lib/ceph/libceph-common.so.0
#4 0x00007f58dcd11982 in SafeTimer::timer_thread() () from /usr/lib/ceph/libceph-common.so.0
#5 0x00007f58dcd12d8d in SafeTimerThread::entry() () from /usr/lib/ceph/libceph-common.so.0
#6 0x00007f58dc86c6ba in start_thread (arg=0x7f58d0ff9700) at pthread_create.c:333
#7 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 9 (Thread 0x7f58d17fa700 (LWP 459333)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd8f7e3 in DispatchQueue::run_local_delivery() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dce84a8d in DispatchQueue::LocalDeliveryThread::entry() () from /usr/lib/ceph/libceph-common.so.0
#3 0x00007f58dc86c6ba in start_thread (arg=0x7f58d17fa700) at pthread_create.c:333
#4 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 8 (Thread 0x7f58d1ffb700 (LWP 459332)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd8c9aa in DispatchQueue::entry() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dce849ad in DispatchQueue::DispatchThread::entry() () from /usr/lib/ceph/libceph-common.so.0
#3 0x00007f58dc86c6ba in start_thread (arg=0x7f58d1ffb700) at pthread_create.c:333
#4 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 7 (Thread 0x7f58d27fc700 (LWP 459331)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd395e3 in ceph::logging::Log::submit_entry(ceph::logging::Entry*) () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dce8f16d in AsyncConnection::send_message(Message*) () from /usr/lib/ceph/libceph-common.so.0
#3 0x00007f58e585fe2d in ?? () from /usr/lib/librados.so.2
#4 0x00007f58e586520d in ?? () from /usr/lib/librados.so.2
#5 0x00007f58e5874393 in ?? () from /usr/lib/librados.so.2
#6 0x00007f58e58434d5 in ceph::timer_detail::timer<ceph::time_detail::mono_clock>::timer_thread() () from /usr/lib/librados.so.2
#7 0x00007f58dc397c80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8 0x00007f58dc86c6ba in start_thread (arg=0x7f58d27fc700) at pthread_create.c:333
#9 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 6 (Thread 0x7f58d2ffd700 (LWP 459330)):
#0 0x00007f58dbe06a13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1 0x00007f58dd13bb49 in EpollDriver::event_wait(std::vector<FiredFileEvent, std::allocator<FiredFileEvent> >&, timeval*) () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dceb1131 in EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*) () from /usr/lib/ceph/libceph-common.so.0
#3 0x00007f58dceb6428 in ?? () from /usr/lib/ceph/libceph-common.so.0
#4 0x00007f58dc397c80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007f58dc86c6ba in start_thread (arg=0x7f58d2ffd700) at pthread_create.c:333
#6 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 5 (Thread 0x7f58d37fe700 (LWP 459329)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd395e3 in ceph::logging::Log::submit_entry(ceph::logging::Entry*) () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dcea1131 in AsyncConnection::process() () from /usr/lib/ceph/libceph-common.so.0
#3 0x00007f58dceb18a8 in EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*) () from /usr/lib/ceph/libceph-common.so.0
#4 0x00007f58dceb6428 in ?? () from /usr/lib/ceph/libceph-common.so.0
#5 0x00007f58dc397c80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x00007f58dc86c6ba in start_thread (arg=0x7f58d37fe700) at pthread_create.c:333
#7 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 4 (Thread 0x7f58d3fff700 (LWP 459328)):
#0 0x00007f58dbe06a13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1 0x00007f58dd13bb49 in EpollDriver::event_wait(std::vector<FiredFileEvent, std::allocator<FiredFileEvent> >&, timeval*) () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dceb1131 in EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*) () from /usr/lib/ceph/libceph-common.so.0
---Type <return> to continue, or q <return> to quit---
#3 0x00007f58dceb6428 in ?? () from /usr/lib/ceph/libceph-common.so.0
#4 0x00007f58dc397c80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007f58dc86c6ba in start_thread (arg=0x7f58d3fff700) at pthread_create.c:333
#6 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 3 (Thread 0x7f58d8bb9700 (LWP 459327)):
#0 0x00007f58dbdfa74d in poll () at ../sysdeps/unix/syscall-template.S:84
#1 0x00007f58dccf2555 in AdminSocket::entry() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dc86c6ba in start_thread (arg=0x7f58d8bb9700) at pthread_create.c:333
#3 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 2 (Thread 0x7f58d9b58700 (LWP 459326)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd395e3 in ceph::logging::Log::submit_entry(ceph::logging::Entry*) () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dcf8d18e in ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, long) () from /usr/lib/ceph/libceph-common.so.0
#3 0x00007f58dcf8da86 in ceph::HeartbeatMap::is_healthy() () from /usr/lib/ceph/libceph-common.so.0
#4 0x00007f58dcf8e21c in ceph::HeartbeatMap::check_touch_file() () from /usr/lib/ceph/libceph-common.so.0
#5 0x00007f58dcf77f57 in CephContextServiceThread::entry() () from /usr/lib/ceph/libceph-common.so.0
#6 0x00007f58dc86c6ba in start_thread (arg=0x7f58d9b58700) at pthread_create.c:333
#7 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 1 (Thread 0x7f58e6235e80 (LWP 459324)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58e5b60ad4 in ?? () from /usr/lib/librbd.so.1
#2 0x00007f58e5b3e460 in librbd::RBD::open(librados::IoCtx&, librbd::Image&, char const*, char const*) () from /usr/lib/librbd.so.1
#3 0x000055c7e874b979 in do_map (cfg=0x7fffd3ed3320, argv=<optimized out>, argc=<optimized out>) at /build/ceph-12.2.5/src/tools/rbd_nbd/rbd-nbd.cc:628
#4 rbd_nbd (argc=<optimized out>, argv=<optimized out>) at /build/ceph-12.2.5/src/tools/rbd_nbd/rbd-nbd.cc:1082
#5 0x000055c7e8748619 in main (argc=<optimized out>, argv=<optimized out>) at /build/ceph-12.2.5/src/tools/rbd_nbd/rbd-nbd.cc:1107
# gdb -p 459325
(gdb) thread apply all bt
Thread 1 (Thread 0x7f58d9b58700 (LWP 459325)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd3a1eb in ceph::logging::Log::entry() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dc86c6ba in start_thread (arg=0x7f58d9b58700) at pthread_create.c:333
#3 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
# gdb -p 459322
(gdb) thread apply all bt
Thread 2 (Thread 0x7f58d9b58700 (LWP 459325)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f58dcd3a1eb in ceph::logging::Log::entry() () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dc86c6ba in start_thread (arg=0x7f58d9b58700) at pthread_create.c:333
#3 0x00007f58dbe0641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 1 (Thread 0x7f58e6235e80 (LWP 459322)):
#0 0x00007f58dc87551d in read () at ../sysdeps/unix/syscall-template.S:84
#1 0x00007f58dd13d1ec in safe_read () from /usr/lib/ceph/libceph-common.so.0
#2 0x00007f58dd13d239 in safe_read_exact () from /usr/lib/ceph/libceph-common.so.0
#3 0x000055c7e874f071 in Preforker::parent_wait (this=0x7fffd3ed2fb0, err_msg="") at /build/ceph-12.2.5/src/common/Preforker.h:79
#4 0x000055c7e874c811 in do_map (cfg=0x7fffd3ed3320, argv=<optimized out>, argc=<optimized out>) at /build/ceph-12.2.5/src/tools/rbd_nbd/rbd-nbd.cc:601
#5 rbd_nbd (argc=<optimized out>, argv=<optimized out>) at /build/ceph-12.2.5/src/tools/rbd_nbd/rbd-nbd.cc:1082
#6 0x000055c7e8748619 in main (argc=<optimized out>, argv=<optimized out>) at /build/ceph-12.2.5/src/tools/rbd_nbd/rbd-nbd.cc:1107
Updated by Jason Dillaman about 6 years ago
- Status changed from Need More Info to Duplicate
Looks like the logging ring-buffer is full and not being flushed. Duplicate of #23143
Updated by Marc Schöchlin almost 6 years ago
Thanks :-)
We implememted a workaround: https://github.com/vico-research-and-consulting/RBDSR/blob/master/bins/ceph_plugin.py#L59
(function "nbd_map")
Actions