Project

General

Profile

Bug #43185

ceph -s not showing client activity

Added by super xor 4 months ago. Updated 27 days ago.

Status:
Need More Info
Priority:
High
Assignee:
-
Category:
-
Target version:
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature:

Description

Since Nautilus upgrade ceph -s often (2 out of 3 times) does not show any client or recovery activity. Right now it's not showing anything since minutes.
Must be some bug because we're having heavy I/O going.

Cant give much more information, there's no errors or anything in log that relates to this. Should be easy to find on other setups.

mgrdump2 (80.2 KB) super xor, 03/07/2020 08:09 AM

History

#1 Updated by super xor 4 months ago

ceph -s only looks like this:

ceph -s
cluster:
id: c4068f25-d46d-438d-af63-5679a2d56efb
health: HEALTH_WARN
1 nearfull osd(s)
2 pool(s) nearfull
Failed to send data to Zabbix
1 subtrees have overcommitted pool target_size_bytes
1 subtrees have overcommitted pool target_size_ratio

services:
mon: 3 daemons, quorum m1-1045558,m2-1045557,m3-1045556 (age 5w)
mgr: m1-1045558(active, since 5w), standbys: m3-1045556, m2-1045557
mds: xz:1 {0=m3-1045556=up:active} 2 up:standby
osd: 136 osds: 136 up (since 3w), 136 in (since 3w)
rgw: 1 daemon active (m1-1045558.rgw0)
data:
pools: 11 pools, 2657 pgs
objects: 549.18M objects, 353 TiB
usage: 752 TiB used, 350 TiB / 1.1 PiB avail
pgs: 2627 active+clean
29 active+clean+scrubbing+deep
1 active+clean+scrubbing

#2 Updated by Neha Ojha 4 months ago

  • Project changed from mgr to RADOS
  • Priority changed from Normal to Urgent

#4 Updated by Neha Ojha 3 months ago

  • Status changed from New to Need More Info

super xor wrote:

Possible relation to https://tracker.ceph.com/issues/43364 and https://tracker.ceph.com/issues/43317

Both these issues were reported in 14.2.5, are you also running 14.2.5? Affected versions indicates 14.2.4.

#5 Updated by super xor 3 months ago

We run 14.2.4. I see mgr process at 100% sometimes and I been told that the reason for lack of activity show might be overloaded mgr or something.
It's hard to debug I get it, but basically it just doesnt show anything as pasted above. Happens sporadically, usually when there is heavy IO due to recovery or similar 'changes'. We been adding OSDs, changing PGs etc lately, so we see it a lot.

#6 Updated by Sage Weil 2 months ago

  • Target version set to v15.0.0

#7 Updated by Sage Weil 2 months ago

  • Priority changed from Urgent to High

#8 Updated by Neha Ojha 2 months ago

Are you observing any client activity in the cluster logs when "ceph -s" isn't reporting them?
It is sometimes possible for client I/O to be intermittently missing during periods of aggressive recovery and scrub activity. You can increase the value of "mon_stat_smooth_intervals" from the default of 6 to maybe 9 or higher and see it helps. This will increase the number of PGMaps stats over which we calculate the average read/write throughput of the whole cluster before displaying them.

#9 Updated by super xor 2 months ago

There's almost no load apart from scrubbing, like this is pretty average io:
client: 20 MiB/s rd, 61 MiB/s wr, 815 op/s rd, 1.14k op/s wr
And we got like 150 OSDs.

But the mgr process is stuck at 100% and crashes every 24h or so. Furthermore I can't call certain function like balancer status, it'll just hang forever.

crashes with following error:

2020-01-30 14:54:56.872 7fc861370700  0 mgr[dashboard] [30/Jan/2020:14:54:56] ENGINE Error in HTTPServer.tick
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/__init__.py", line 2021, in start
    self.tick()
  File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/__init__.py", line 2090, in tick
    s, ssl_env = self.ssl_adapter.wrap(s)
  File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/ssl_builtin.py", line 67, in wrap
    server_side=True)
  File "/usr/lib/python2.7/ssl.py", line 369, in wrap_socket
    _context=self)
  File "/usr/lib/python2.7/ssl.py", line 617, in __init__
    self.do_handshake()
  File "/usr/lib/python2.7/ssl.py", line 846, in do_handshake
    self._sslobj.do_handshake()
error: [Errno 0] Error

#10 Updated by super xor 2 months ago

strace for the hanging mgr thread

...
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = -1 EAGAIN (Resource temporarily unavailable)
getpid()                                = 1360760
futex(0x56025eacff60, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x56025eacff60, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025e9d80e4, FUTEX_WAKE_PRIVATE, 2147483647) = 1
futex(0x56025e9d8038, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x56025eacff60, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x56025eacff60, FUTEX_WAKE_PRIVATE, 1) = 1
getpid()                                = 1360760
futex(0x56025eacff60, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x56025eacff60, FUTEX_WAKE_PRIVATE, 1) = 1
select(0, NULL, NULL, NULL, {tv_sec=0, tv_usec=1000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {tv_sec=0, tv_usec=2000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {tv_sec=0, tv_usec=4000}) = 0 (Timeout)
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x56025eacff60, FUTEX_WAKE_PRIVATE, 1) = 0
select(0, NULL, NULL, NULL, {tv_sec=0, tv_usec=8000}) = 0 (Timeout)
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = -1 EAGAIN (Resource temporarily unavailable)
...

#11 Updated by Neha Ojha 2 months ago

Can you grab a wallclock profiler dump from the mgr process when its usage goes to 100%?
Learn more about how to use it in https://github.com/markhpc/gdbpmp.
Grab the dump using something like - "gdbpmp.py -p`pidof ceph-mgr` -n 1000 -o mgr.gdbpmp"
View the results using - "gdbpmp.py -i mgr.gdbpmp -t 1"

You can also disable your mgr modules to see if the problem goes away. FWIW, we have fixed an issue with the balancer module, which made the mgr hang, the fix will be available in 14.2.8.

#12 Updated by super xor about 1 month ago

hread 6 "service" received signal SIGINT, Interrupt.
0x00007f1fe4d9cf85 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f1fde3f18a0, expected=0, futex_word=0x55a6cc0431e0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
205    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 2 "msgr-worker-0" received signal SIGINT, Interrupt.
0x00007f1fe3f7cbb7 in epoll_wait (epfd=3, events=0x55a6cc108000, maxevents=5000, timeout=1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 3 "msgr-worker-1" received signal SIGINT, Interrupt.
0x00007f1fe3f7cbb7 in epoll_wait (epfd=6, events=0x55a6ccc6a000, maxevents=5000, timeout=1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 2 "msgr-worker-0" received signal SIGINT, Interrupt.
0x00007f1fe3f7cbb7 in epoll_wait (epfd=3, events=0x55a6cc108000, maxevents=5000, timeout=1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 2 "msgr-worker-0" received signal SIGINT, Interrupt.
0x00007f1fe3f7cbb7 in epoll_wait (epfd=3, events=0x55a6cc108000, maxevents=5000, timeout=1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 2 "msgr-worker-0" received signal SIGINT, Interrupt.
0x00007f1fe3f7cbb7 in epoll_wait (epfd=3, events=0x55a6cc108000, maxevents=5000, timeout=1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 2 "msgr-worker-0" received signal SIGINT, Interrupt.
0x00007f1fe3f7cbb7 in epoll_wait (epfd=3, events=0x55a6cc108000, maxevents=5000, timeout=1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
.
Thread 2 "msgr-worker-0" received signal SIGINT, Interrupt.
0x00007f1fe3f7cbb7 in epoll_wait (epfd=3, events=0x55a6cc108000, maxevents=5000, timeout=1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
30    in ../sysdeps/unix/sysv/linux/epoll_wait.c
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
^C
Profiling complete with 141 samples.

#13 Updated by super xor about 1 month ago


Thread: 1 (ceph-mgr) - 100 samples 

+ 100.00% main
  + 100.00% MgrStandby::main(std::vector<char const*, std::allocator<char const*> >)
    + 100.00% AsyncMessenger::wait()
      + 100.00% __pthread_cond_wait
        + 100.00% __pthread_cond_wait_common
          + 100.00% futex_wait_cancelable

Thread: 2 (msgr-worker-0) - 100 samples 

+ 100.00% clone
  + 100.00% start_thread
    + 100.00% ???
      + 100.00% ???
        + 100.00% EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)
          + 93.00% EpollDriver::event_wait(std::vector<FiredFileEvent, std::allocator<FiredFileEvent> >&, timeval*)
          | + 93.00% epoll_wait
          + 6.00% AsyncConnection::process()
          | + 6.00% ProtocolV2::run_continuation(Ct<ProtocolV2>&)
          |   + 6.00% ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr<ceph::buffer::v14_2_0::ptr_node, ceph::buffer::v14_2_0::ptr_node::disposer>&&, int)
          |     + 6.00% ProtocolV2::handle_read_frame_dispatch()
          |       + 5.00% ProtocolV2::handle_message()
          |       | + 4.00% decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::v14_2_0::list&, ceph::buffer::v14_2_0::list&, ceph::buffer::v14_2_0::list&, Connection*)
          |       | | + 4.00% MPGStats::decode_payload()
          |       | |   + 4.00% pg_stat_t::decode(ceph::buffer::v14_2_0::list::iterator_impl<true>&)
          |       | |     + 2.00% std::enable_if<denc_traits<std::vector<int, std::allocator<int> >, void>::supported&&(!denc_traits<std::vector<int, std::allocator<int> >, void>::need_contiguous), void>::type ceph::decode<std::vector<int, std::allocator<int> >, denc_traits<std::vector<int, std::allocator<int> >, void> >(std::vector<int, std::allocator<int> >&, ceph::buffer::v14_2_0::list::iterator_impl<true>&)
          |       | |     | + 1.00% int& std::vector<int, std::allocator<int> >::emplace_back<int>(int&&)
          |       | |     | | + 1.00% tc_free
          |       | |     | + 1.00% ceph::buffer::v14_2_0::list::iterator_impl<true>::copy_shallow(unsigned int, ceph::buffer::v14_2_0::ptr&)
          |       | |     |   + 1.00% ceph::buffer::v14_2_0::ptr::ptr(ceph::buffer::v14_2_0::ptr const&, unsigned int, unsigned int)
          |       | |     + 1.00% std::enable_if<denc_traits<interval_set<snapid_t, std::map<snapid_t, snapid_t, std::less<snapid_t>, std::allocator<std::pair<snapid_t const, snapid_t> > > >, void>::supported&&denc_traits<interval_set<snapid_t, std::map<snapid_t, snapid_t, std::less<snapid_t>, std::allocator<std::pair<snapid_t const, snapid_t> > > >, void>::need_contiguous, void>::type ceph::decode<interval_set<snapid_t, std::map<snapid_t, snapid_t, std::less<snapid_t>, std::allocator<std::pair<snapid_t const, snapid_t> > > >, denc_traits<interval_set<snapid_t, std::map<snapid_t, snapid_t, std::less<snapid_t>, std::allocator<std::pair<snapid_t const, snapid_t> > > >, void> >(interval_set<snapid_t, std::map<snapid_t, snapid_t, std::less<snapid_t>, std::allocator<std::pair<snapid_t const, snapid_t> > > >&, ceph::buffer::v14_2_0::list::iterator_impl<true>&)
          |       | |     | + 1.00% ceph::buffer::v14_2_0::ptr::release()
          |       | |     + 1.00% ceph::buffer::v14_2_0::list::iterator_impl<true>::copy(unsigned int, char*)
          |       | + 1.00% DispatchQueue::enqueue(boost::intrusive_ptr<Message> const&, int, unsigned long)
          |       |   + 1.00% Mutex::lock(bool)
          |       |     + 1.00% __GI___pthread_mutex_lock
          |       |       + 1.00% __lll_lock_wait
          |       + 1.00% ProtocolV2::handle_frame_payload()
          |         + 1.00% ProtocolV2::handle_auth_request(ceph::buffer::v14_2_0::list&)
          |           + 1.00% ProtocolV2::_handle_auth_request(ceph::buffer::v14_2_0::list&, bool)
          |             + 1.00% MonClient::handle_auth_request(Connection*, AuthConnectionMeta*, bool, unsigned int, ceph::buffer::v14_2_0::list const&, ceph::buffer::v14_2_0::list*)
          |               + 1.00% CephxAuthorizeHandler::verify_authorizer(CephContext*, KeyStore*, ceph::buffer::v14_2_0::list const&, unsigned long, ceph::buffer::v14_2_0::list*, EntityName*, unsigned long*, AuthCapsInfo*, CryptoKey*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::unique_ptr<AuthAuthorizerChallenge, std::default_delete<AuthAuthorizerChallenge> >*)
          |                 + 1.00% cephx_verify_authorizer(CephContext*, KeyStore*, ceph::buffer::v14_2_0::list::iterator_impl<true>&, unsigned long, CephXServiceTicketInfo&, std::unique_ptr<AuthAuthorizerChallenge, std::default_delete<AuthAuthorizerChallenge> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, ceph::buffer::v14_2_0::list*)
          |                   + 1.00% void decode_decrypt_enc_bl<CephXServiceTicketInfo>(CephContext*, CephXServiceTicketInfo&, CryptoKey, ceph::buffer::v14_2_0::list const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)
          |                     + 1.00% CryptoAESKeyHandler::decrypt(ceph::buffer::v14_2_0::list const&, ceph::buffer::v14_2_0::list&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) const
          |                       + 1.00% AES_cbc_encrypt
          |                         + 1.00% ???
          + 1.00% EventCenter::process_time_events()
            + 1.00% AsyncConnection::wakeup_from(unsigned long)
              + 1.00% AsyncConnection::process()
                + 1.00% ProtocolV2::read_event()
                  + 1.00% ProtocolV2::run_continuation(Ct<ProtocolV2>&)
                    + 1.00% ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr<ceph::buffer::v14_2_0::ptr_node, ceph::buffer::v14_2_0::ptr_node::disposer>&&, int)
                      + 1.00% ProtocolV2::handle_read_frame_dispatch()
                        + 1.00% ProtocolV2::handle_message()
                          + 1.00% decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::v14_2_0::list&, ceph::buffer::v14_2_0::list&, ceph::buffer::v14_2_0::list&, Connection*)
                            + 1.00% MPGStats::decode_payload()
                              + 1.00% pg_stat_t::decode(ceph::buffer::v14_2_0::list::iterator_impl<true>&)
                                + 1.00% object_stat_collection_t::decode(ceph::buffer::v14_2_0::list::iterator_impl<true>&)
                                  + 1.00% object_stat_sum_t::decode(ceph::buffer::v14_2_0::list::iterator_impl<true>&)
                                    + 1.00% ceph::buffer::v14_2_0::list::iterator_impl<true>::copy(unsigned int, char*)

Thread: 3 (msgr-worker-1) - 100 samples 

+ 100.00% clone
  + 100.00% start_thread
    + 100.00% ???
      + 100.00% ???
        + 100.00% EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)
          + 88.00% EpollDriver::event_wait(std::vector<FiredFileEvent, std::allocator<FiredFileEvent> >&, timeval*)
          | + 88.00% epoll_wait
          + 9.00% AsyncConnection::process()
          | + 9.00% ProtocolV2::run_continuation(Ct<ProtocolV2>&)
          |   + 7.00% ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr<ceph::buffer::v14_2_0::ptr_node, ceph::buffer::v14_2_0::ptr_node::disposer>&&, int)
          |   | + 6.00% ProtocolV2::handle_read_frame_dispatch()
          |   | | + 5.00% ProtocolV2::handle_message()
          |   | | | + 5.00% decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::v14_2_0::list&, ceph::buffer::v14_2_0::list&, ceph::buffer::v14_2_0::list&, Connection*)
          |   | | |   + 5.00% MPGStats::decode_payload()
          |   | | |     + 5.00% pg_stat_t::decode(ceph::buffer::v14_2_0::list::iterator_impl<true>&)
          |   | | |       + 2.00% ceph::buffer::v14_2_0::list::iterator_impl<true>::copy(unsigned int, char*)
          |   | | |       + 1.00% std::enable_if<denc_traits<std::vector<int, std::allocator<int> >, void>::supported&&(!denc_traits<std::vector<int, std::allocator<int> >, void>::need_contiguous), void>::type ceph::decode<std::vector<int, std::allocator<int> >, denc_traits<std::vector<int, std::allocator<int> >, void> >(std::vector<int, std::allocator<int> >&, ceph::buffer::v14_2_0::list::iterator_impl<true>&)
          |   | | |       | + 1.00% int& std::vector<int, std::allocator<int> >::emplace_back<int>(int&&)
          |   | | |       |   + 1.00% tc_free
          |   | | |       + 1.00% std::enable_if<denc_traits<interval_set<snapid_t, std::map<snapid_t, snapid_t, std::less<snapid_t>, std::allocator<std::pair<snapid_t const, snapid_t> > > >, void>::supported&&denc_traits<interval_set<snapid_t, std::map<snapid_t, snapid_t, std::less<snapid_t>, std::allocator<std::pair<snapid_t const, snapid_t> > > >, void>::need_contiguous, void>::type ceph::decode<interval_set<snapid_t, std::map<snapid_t, snapid_t, std::less<snapid_t>, std::allocator<std::pair<snapid_t const, snapid_t> > > >, denc_traits<interval_set<snapid_t, std::map<snapid_t, snapid_t, std::less<snapid_t>, std::allocator<std::pair<snapid_t const, snapid_t> > > >, void> >(interval_set<snapid_t, std::map<snapid_t, snapid_t, std::less<snapid_t>, std::allocator<std::pair<snapid_t const, snapid_t> > > >&, ceph::buffer::v14_2_0::list::iterator_impl<true>&)
          |   | | |       | + 1.00% ceph::buffer::v14_2_0::ptr::c_str() const
          |   | | |       + 1.00% object_stat_collection_t::decode(ceph::buffer::v14_2_0::list::iterator_impl<true>&)
          |   | | |         + 1.00% ceph::buffer::v14_2_0::list::iterator_impl<true>::copy(unsigned int, char*)
          |   | | + 1.00% ProtocolV2::handle_frame_payload()
          |   | |   + 1.00% ProtocolV2::handle_auth_signature(ceph::buffer::v14_2_0::list&)
          |   | |     + 1.00% ProtocolV2::send_client_ident()
          |   | |       + 1.00% ProtocolV2::write(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, CtFun<ProtocolV2>&, ceph::buffer::v14_2_0::list&)
          |   | |         + 1.00% AsyncConnection::write(ceph::buffer::v14_2_0::list&, std::function<void (long)>, bool)
          |   | |           + 1.00% AsyncConnection::_try_send(bool)
          |   | |             + 1.00% PosixConnectedSocketImpl::send(ceph::buffer::v14_2_0::list&, bool)
          |   | |               + 1.00% __libc_sendmsg
          |   | + 1.00% ceph::buffer::v14_2_0::list::crc32c(unsigned int) const
          |   |   + 1.00% ceph_crc32c_intel_fast
          |   |     + 1.00% ???
          |   |       + 1.00% ???
          |   |         + 1.00% crc32_iscsi_00
          |   + 1.00% ProtocolV2::handle_read_frame_segment(std::unique_ptr<ceph::buffer::v14_2_0::ptr_node, ceph::buffer::v14_2_0::ptr_node::disposer>&&, int)
          |   | + 1.00% ProtocolV2::read_frame_segment()
          |   |   + 1.00% ProtocolV2::read(CtRxNode<ProtocolV2>&, std::unique_ptr<ceph::buffer::v14_2_0::ptr_node, ceph::buffer::v14_2_0::ptr_node::disposer>&&)
          |   |     + 1.00% AsyncConnection::read(unsigned int, char*, std::function<void (char*, long)>)
          |   |       + 1.00% AsyncConnection::read_until(unsigned int, char*)
          |   |         + 1.00% AsyncConnection::read_bulk(char*, unsigned int)
          |   |           + 1.00% PosixConnectedSocketImpl::read(char*, unsigned long)
          |   |             + 1.00% __libc_read
          |   + 1.00% ProtocolV2::_handle_peer_banner_payload(std::unique_ptr<ceph::buffer::v14_2_0::ptr_node, ceph::buffer::v14_2_0::ptr_node::disposer>&&, int)
          |     + 1.00% ProtocolV2::write(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, CtFun<ProtocolV2>&, ceph::buffer::v14_2_0::list&)
          |       + 1.00% AsyncConnection::write(ceph::buffer::v14_2_0::list&, std::function<void (long)>, bool)
          |         + 1.00% AsyncConnection::_try_send(bool)
          |           + 1.00% PosixConnectedSocketImpl::send(ceph::buffer::v14_2_0::list&, bool)
          |             + 1.00% __libc_sendmsg
          + 3.00% EventCenter::process_time_events()
            + 3.00% AsyncConnection::wakeup_from(unsigned long)
              + 3.00% AsyncConnection::process()
                + 3.00% ProtocolV2::read_event()
                  + 3.00% ProtocolV2::run_continuation(Ct<ProtocolV2>&)
                    + 2.00% ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr<ceph::buffer::v14_2_0::ptr_node, ceph::buffer::v14_2_0::ptr_node::disposer>&&, int)
                    | + 2.00% ProtocolV2::handle_read_frame_dispatch()
                    |   + 2.00% ProtocolV2::handle_message()
                    |     + 2.00% decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::v14_2_0::list&, ceph::buffer::v14_2_0::list&, ceph::buffer::v14_2_0::list&, Connection*)
                    |       + 2.00% MPGStats::decode_payload()
                    |         + 1.00% pg_t::decode(ceph::buffer::v14_2_0::list::iterator_impl<true>&)
                    |         + 1.00% pg_stat_t::decode(ceph::buffer::v14_2_0::list::iterator_impl<true>&)
                    |           + 1.00% ceph::buffer::v14_2_0::list::iterator_impl<true>::copy(unsigned int, char*)
                    |             + 1.00% ceph::buffer::v14_2_0::ptr::copy_out(unsigned int, unsigned int, char*) const
                    + 1.00% ProtocolV2::throttle_message()
                      + 1.00% EventCenter::create_time_event(unsigned long, EventCallback*)
                        + 1.00% std::_Rb_tree_insert_and_rebalance(bool, std::_Rb_tree_node_base*, std::_Rb_tree_node_base*, std::_Rb_tree_node_base&)

Thread: 4 (msgr-worker-2) - 100 samples 

+ 100.00% clone
  + 100.00% start_thread
    + 100.00% ???
      + 100.00% ???
        + 100.00% EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)
          + 90.00% EpollDriver::event_wait(std::vector<FiredFileEvent, std::allocator<FiredFileEvent> >&, timeval*)
          | + 90.00% epoll_wait
          + 8.00% AsyncConnection::process()
          | + 7.00% ProtocolV2::run_continuation(Ct<ProtocolV2>&)
          | | + 6.00% ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr<ceph::buffer::v14_2_0::ptr_node, ceph::buffer::v14_2_0::ptr_node::disposer>&&, int)
          | | | + 5.00% ProtocolV2::handle_read_frame_dispatch()
          | | | | + 4.00% ProtocolV2::handle_message()
          | | | | | + 4.00% decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::v14_2_0::list&, ceph::buffer::v14_2_0::list&, ceph::buffer::v14_2_0::list&, Connection*)
          | | | | |   + 4.00% MPGStats::decode_payload()
          | | | | |     + 4.00% pg_stat_t::decode(ceph::buffer::v14_2_0::list::iterator_impl<true>&)
          | | | | |       + 2.00% ceph::buffer::v14_2_0::list::iterator_impl<true>::copy(unsigned int, char*)
          | | | | |       | + 2.00% ceph::buffer::v14_2_0::ptr::copy_out(unsigned int, unsigned int, char*) const
          | | | | |       + 1.00% std::enable_if<denc_traits<std::vector<int, std::allocator<int> >, void>::supported&&(!denc_traits<std::vector<int, std::allocator<int> >, void>::need_contiguous), void>::type ceph::decode<std::vector<int, std::allocator<int> >, denc_traits<std::vector<int, std::allocator<int> >, void> >(std::vector<int, std::allocator<int> >&, ceph::buffer::v14_2_0::list::iterator_impl<true>&)
          | | | | |       | + 1.00% ceph::buffer::v14_2_0::ptr::release()
          | | | | |       + 1.00% object_stat_collection_t::decode(ceph::buffer::v14_2_0::list::iterator_impl<true>&)
          | | | | |         + 1.00% object_stat_sum_t::decode(ceph::buffer::v14_2_0::list::iterator_impl<true>&)
          | | | | |           + 1.00% ceph::buffer::v14_2_0::list::iterator_impl<true>::copy(unsigned int, char*)
          | | | | + 1.00% ProtocolV2::handle_frame_payload()
          | | | |   + 1.00% ProtocolV2::handle_client_ident(ceph::buffer::v14_2_0::list&)
          | | | |     + 1.00% std::_Hashtable<entity_addrvec_t, std::pair<entity_addrvec_t const, boost::intrusive_ptr<AsyncConnection> >, std::allocator<std::pair<entity_addrvec_t const, boost::intrusive_ptr<AsyncConnection> > >, std::__detail::_Select1st, std::equal_to<entity_addrvec_t>, std::hash<entity_addrvec_t>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::erase(std::__detail::_Node_const_iterator<std::pair<entity_addrvec_t const, boost::intrusive_ptr<AsyncConnection> >, false, true>)
          | | | |       + 1.00% RefCountedObject::put() const
          | | | |         + 1.00% AsyncConnection::~AsyncConnection()
          | | | |           + 1.00% AsyncConnection::~AsyncConnection()
          | | | |             + 1.00% RefCountedObject::put() const
          | | | |               + 1.00% MgrSession::~MgrSession()
          | | | + 1.00% ceph::buffer::v14_2_0::list::crc32c(unsigned int) const
          | | |   + 1.00% ceph_crc32c_intel_fast
          | | |     + 1.00% ???
          | | |       + 1.00% ???
          | | |         + 1.00% crc32_iscsi_00
          | | + 1.00% ProtocolV2::handle_read_frame_segment(std::unique_ptr<ceph::buffer::v14_2_0::ptr_node, ceph::buffer::v14_2_0::ptr_node::disposer>&&, int)
          | |   + 1.00% ProtocolV2::read_frame_segment()
          | |     + 1.00% ProtocolV2::read(CtRxNode<ProtocolV2>&, std::unique_ptr<ceph::buffer::v14_2_0::ptr_node, ceph::buffer::v14_2_0::ptr_node::disposer>&&)
          | |       + 1.00% AsyncConnection::read(unsigned int, char*, std::function<void (char*, long)>)
          | |         + 1.00% AsyncConnection::read_until(unsigned int, char*)
          | |           + 1.00% AsyncConnection::read_bulk(char*, unsigned int)
          | |             + 1.00% PosixConnectedSocketImpl::read(char*, unsigned long)
          | |               + 1.00% __libc_read
          | + 1.00% AsyncConnection::read(unsigned int, char*, std::function<void (char*, long)>)
          |   + 1.00% AsyncConnection::read_until(unsigned int, char*)
          |     + 1.00% AsyncConnection::read_bulk(char*, unsigned int)
          |       + 1.00% PosixConnectedSocketImpl::read(char*, unsigned long)
          |         + 1.00% __libc_read
          + 2.00% EventCenter::process_time_events()
            + 1.00% std::_Rb_tree_rebalance_for_erase(std::_Rb_tree_node_base*, std::_Rb_tree_node_base&)
            + 1.00% AsyncConnection::wakeup_from(unsigned long)
              + 1.00% AsyncConnection::process()
                + 1.00% ProtocolV2::read_event()
                  + 1.00% ProtocolV2::run_continuation(Ct<ProtocolV2>&)
                    + 1.00% ProtocolV2::handle_read_frame_segment(std::unique_ptr<ceph::buffer::v14_2_0::ptr_node, ceph::buffer::v14_2_0::ptr_node::disposer>&&, int)
                      + 1.00% ceph::buffer::v14_2_0::list::iterator_impl<false>::iterator_impl(ceph::buffer::v14_2_0::list*, unsigned int)

Thread: 5 (log) - 100 samples 

+ 100.00% clone
  + 100.00% start_thread
    + 100.00% ceph::logging::Log::entry()
      + 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
Traceback (most recent call last):
  File "gdbpmp.py", line 44, in <module>
    import tracer
  File "/root/gdbpmp/tracer.py", line 25, in <module>
    import gdb
ModuleNotFoundError: No module named 'gdb'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "gdbpmp.py", line 46, in <module>
    main()
  File "gdbpmp.py", line 25, in main
    common.print_callgraph(threads, ctx.threshold)
  File "/root/gdbpmp/common.py", line 34, in print_callgraph
    gdbth.function.print_percent("", samples, threshold, True)
  File "/root/gdbpmp/gdbtypes.py", line 85, in print_percent
    self.get_func(name).print_percent(prefix + new_prefix, total, threshold, include_sub)
  File "/root/gdbpmp/gdbtypes.py", line 85, in print_percent
    self.get_func(name).print_percent(prefix + new_prefix, total, threshold, include_sub)
  File "/root/gdbpmp/gdbtypes.py", line 85, in print_percent
    self.get_func(name).print_percent(prefix + new_prefix, total, threshold, include_sub)
  [Previous line repeated 8 more times]
  File "/root/gdbpmp/gdbtypes.py", line 79, in print_percent
    print ("%s%s%0.2f%% %s" % (prefix, "+ ", value, name))
BrokenPipeError: [Errno 32] Broken pipe

at the end

Thread: 62 (safe_timer) - 100 samples 

+ 100.00% clone
  + 100.00% start_thread
    + 100.00% SafeTimerThread::entry()
      + 100.00% SafeTimer::timer_thread()
        + 100.00% __pthread_cond_timedwait
          + 100.00% __pthread_cond_wait_common
            + 100.00% futex_abstimed_wait_cancelable

Thread: 63 (fn_anonymous) - 100 samples 

+ 100.00% clone
  + 100.00% start_thread
    + 100.00% Finisher::finisher_thread_entry()
      + 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
        + 100.00% __pthread_cond_wait
          + 100.00% __pthread_cond_wait_common
            + 100.00% futex_wait_cancelable

Thread: 64 (safe_timer) - 100 samples 

+ 100.00% clone
  + 100.00% start_thread
    + 100.00% SafeTimerThread::entry()
      + 100.00% SafeTimer::timer_thread()
        + 95.00% __pthread_cond_timedwait
        | + 95.00% __pthread_cond_wait_common
        |   + 95.00% futex_abstimed_wait_cancelable
        + 5.00% __pthread_cond_wait
          + 5.00% __pthread_cond_wait_common
            + 5.00% futex_wait_cancelable

Thread: 65 (safe_timer) - 100 samples 

+ 100.00% clone
  + 100.00% start_thread
    + 100.00% SafeTimerThread::entry()
      + 100.00% SafeTimer::timer_thread()
        + 100.00% __pthread_cond_wait
          + 100.00% __pthread_cond_wait_common
            + 100.00% futex_wait_cancelable

Thread: 66 (fn-radosclient) - 100 samples 

+ 100.00% clone
  + 100.00% start_thread
    + 100.00% Finisher::finisher_thread_entry()
      + 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
        + 100.00% __pthread_cond_wait
          + 100.00% __pthread_cond_wait_common
            + 100.00% futex_wait_cancelable

#14 Updated by super xor 29 days ago

new dump...after disabling almost all mgr modules

#15 Updated by super xor 27 days ago

fixed with 14.2.8

Also available in: Atom PDF