Project

General

Profile

Actions
Copy link

Bug #43185

closed

ceph -s not showing client activity

Added by Anonymous over 4 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
High
Assignee:
-
Category:
-
Target version:
Ceph - v15.0.0
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v14.2.4
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Since Nautilus upgrade ceph -s often (2 out of 3 times) does not show any client or recovery activity. Right now it's not showing anything since minutes.
Must be some bug because we're having heavy I/O going.

Cant give much more information, there's no errors or anything in log that relates to this. Should be easy to find on other setups.

Files

mgrdump2 (80.2 KB) mgrdump2 Anonymous, 03/07/2020 08:09 AM

Related issues 1 (0 open — 1 closed)

Related to mgr - Bug #39264: Ceph-mgr Hangup and _check_auth_rotating possible clock skew, rotating keys expired way too early ErrorsResolved

Actions
Actions
Copy link
 #1

Updated by Anonymous over 4 years ago

ceph -s only looks like this:

ceph -s
cluster:
id: c4068f25-d46d-438d-af63-5679a2d56efb
health: HEALTH_WARN
1 nearfull osd(s)
2 pool(s) nearfull
Failed to send data to Zabbix
1 subtrees have overcommitted pool target_size_bytes
1 subtrees have overcommitted pool target_size_ratio

services:
    mon: 3 daemons, quorum m1-1045558,m2-1045557,m3-1045556 (age 5w)
    mgr: m1-1045558(active, since 5w), standbys: m3-1045556, m2-1045557
    mds: xz:1 {0=m3-1045556=up:active} 2 up:standby
    osd: 136 osds: 136 up (since 3w), 136 in (since 3w)
    rgw: 1 daemon active (m1-1045558.rgw0)
data:
    pools:   11 pools, 2657 pgs
    objects: 549.18M objects, 353 TiB
    usage:   752 TiB used, 350 TiB / 1.1 PiB avail
    pgs:     2627 active+clean
             29   active+clean+scrubbing+deep
             1    active+clean+scrubbing
Actions
Copy link
 #2

Updated by Neha Ojha over 4 years ago

  • Project changed from mgr to RADOS
  • Priority changed from Normal to Urgent
Actions
Copy link
 #4

Updated by Neha Ojha over 4 years ago

  • Status changed from New to Need More Info

super xor wrote:

Possible relation to https://tracker.ceph.com/issues/43364 and https://tracker.ceph.com/issues/43317

Both these issues were reported in 14.2.5, are you also running 14.2.5? Affected versions indicates 14.2.4.

Actions
Copy link
 #5

Updated by Anonymous over 4 years ago

We run 14.2.4. I see mgr process at 100% sometimes and I been told that the reason for lack of activity show might be overloaded mgr or something.
It's hard to debug I get it, but basically it just doesnt show anything as pasted above. Happens sporadically, usually when there is heavy IO due to recovery or similar 'changes'. We been adding OSDs, changing PGs etc lately, so we see it a lot.

Actions
Copy link
 #6

Updated by Sage Weil over 4 years ago

  • Target version set to v15.0.0
Actions
Copy link
 #7

Updated by Sage Weil over 4 years ago

  • Priority changed from Urgent to High
Actions
Copy link
 #8

Updated by Neha Ojha over 4 years ago

Are you observing any client activity in the cluster logs when "ceph -s" isn't reporting them?
It is sometimes possible for client I/O to be intermittently missing during periods of aggressive recovery and scrub activity. You can increase the value of "mon_stat_smooth_intervals" from the default of 6 to maybe 9 or higher and see it helps. This will increase the number of PGMaps stats over which we calculate the average read/write throughput of the whole cluster before displaying them.

Actions
Copy link
 #9

Updated by Anonymous over 4 years ago

There's almost no load apart from scrubbing, like this is pretty average io:
client: 20 MiB/s rd, 61 MiB/s wr, 815 op/s rd, 1.14k op/s wr
And we got like 150 OSDs.

But the mgr process is stuck at 100% and crashes every 24h or so. Furthermore I can't call certain function like balancer status, it'll just hang forever.

crashes with following error:

2020-01-30 14:54:56.872 7fc861370700  0 mgr[dashboard] [30/Jan/2020:14:54:56] ENGINE Error in HTTPServer.tick
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/__init__.py", line 2021, in start
    self.tick()
  File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/__init__.py", line 2090, in tick
    s, ssl_env = self.ssl_adapter.wrap(s)
  File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/ssl_builtin.py", line 67, in wrap
    server_side=True)
  File "/usr/lib/python2.7/ssl.py", line 369, in wrap_socket
    _context=self)
  File "/usr/lib/python2.7/ssl.py", line 617, in __init__
    self.do_handshake()
  File "/usr/lib/python2.7/ssl.py", line 846, in do_handshake
    self._sslobj.do_handshake()
error: [Errno 0] Error

Actions
Copy link
 #10

Updated by Anonymous over 4 years ago

strace for the hanging mgr thread

...
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = -1 EAGAIN (Resource temporarily unavailable)
getpid()                                = 1360760
futex(0x56025eacff60, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x56025eacff60, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025e9d80e4, FUTEX_WAKE_PRIVATE, 2147483647) = 1
futex(0x56025e9d8038, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x56025eacff60, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x56025eacff60, FUTEX_WAKE_PRIVATE, 1) = 1
getpid()                                = 1360760
futex(0x56025eacff60, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x56025eacff60, FUTEX_WAKE_PRIVATE, 1) = 1
select(0, NULL, NULL, NULL, {tv_sec=0, tv_usec=1000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {tv_sec=0, tv_usec=2000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {tv_sec=0, tv_usec=4000}) = 0 (Timeout)
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x56025eacff60, FUTEX_WAKE_PRIVATE, 1) = 0
select(0, NULL, NULL, NULL, {tv_sec=0, tv_usec=8000}) = 0 (Timeout)
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = 0
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x56025eacff60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff) = -1 EAGAIN (Resource temporarily unavailable)
...
Actions
Copy link
 #11

Updated by Neha Ojha over 4 years ago

Can you grab a wallclock profiler dump from the mgr process when its usage goes to 100%?
Learn more about how to use it in https://github.com/markhpc/gdbpmp.
Grab the dump using something like - "gdbpmp.py -p`pidof ceph-mgr` -n 1000 -o mgr.gdbpmp"
View the results using - "gdbpmp.py -i mgr.gdbpmp -t 1"

You can also disable your mgr modules to see if the problem goes away. FWIW, we have fixed an issue with the balancer module, which made the mgr hang, the fix will be available in 14.2.8.

Actions
Copy link
 #12

Updated by Anonymous about 4 years ago 

hread 6 "service" received signal SIGINT, Interrupt.
0x00007f1fe4d9cf85 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f1fde3f18a0, expected=0, futex_word=0x55a6cc0431e0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
205    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 2 "msgr-worker-0" received signal SIGINT, Interrupt.
0x00007f1fe3f7cbb7 in epoll_wait (epfd=3, events=0x55a6cc108000, maxevents=5000, timeout=1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 3 "msgr-worker-1" received signal SIGINT, Interrupt.
0x00007f1fe3f7cbb7 in epoll_wait (epfd=6, events=0x55a6ccc6a000, maxevents=5000, timeout=1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 2 "msgr-worker-0" received signal SIGINT, Interrupt.
0x00007f1fe3f7cbb7 in epoll_wait (epfd=3, events=0x55a6cc108000, maxevents=5000, timeout=1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 2 "msgr-worker-0" received signal SIGINT, Interrupt.
0x00007f1fe3f7cbb7 in epoll_wait (epfd=3, events=0x55a6cc108000, maxevents=5000, timeout=1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 2 "msgr-worker-0" received signal SIGINT, Interrupt.
0x00007f1fe3f7cbb7 in epoll_wait (epfd=3, events=0x55a6cc108000, maxevents=5000, timeout=1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88    in ../sysdeps/unix/sysv/linux/futex-internal.h
.
Thread 2 "msgr-worker-0" received signal SIGINT, Interrupt.
0x00007f1fe3f7cbb7 in epoll_wait (epfd=3, events=0x55a6cc108000, maxevents=5000, timeout=1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
.
Thread 2 "msgr-worker-0" received signal SIGINT, Interrupt.
0x00007f1fe3f7cbb7 in epoll_wait (epfd=3, events=0x55a6cc108000, maxevents=5000, timeout=1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
30    in ../sysdeps/unix/sysv/linux/epoll_wait.c
.
Thread 1 "ceph-mgr" received signal SIGINT, Interrupt.
0x00007f1fe4d9c9f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55a6cc0c68d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
^C
Profiling complete with 141 samples.
Actions
Copy link
 #13

Updated by Anonymous about 4 years ago 


Thread: 1 (ceph-mgr) - 100 samples 

+ 100.00% main
  + 100.00% MgrStandby::main(std::vector<char const*, std::allocator<char const*> >)
    + 100.00% AsyncMessenger::wait()
      + 100.00% __pthread_cond_wait
        + 100.00% __pthread_cond_wait_common
          + 100.00% futex_wait_cancelable

Thread: 2 (msgr-worker-0) - 100 samples 

+ 100.00% clone
  + 100.00% start_thread
    + 100.00% ???
      + 100.00% ???
        + 100.00% EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)
          + 93.00% EpollDriver::event_wait(std::vector<FiredFileEvent, std::allocator<FiredFileEvent> >&, timeval*)
          | + 93.00% epoll_wait
          + 6.00% AsyncConnection::process()
          | + 6.00% ProtocolV2::run_continuation(Ct<ProtocolV2>&)
          |   + 6.00% ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr<ceph::buffer::v14_2_0::ptr_node, ceph::buffer::v14_2_0::ptr_node::disposer>&&, int)
          |     + 6.00% ProtocolV2::handle_read_frame_dispatch()
          |       + 5.00% ProtocolV2::handle_message()
          |       | + 4.00% decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::v14_2_0::list&, ceph::buffer::v14_2_0::list&, ceph::buffer::v14_2_0::list&, Connection*)
          |       | | + 4.00% MPGStats::decode_payload()
          |       | |   + 4.00% pg_stat_t::decode(ceph::buffer::v14_2_0::list::iterator_impl<true>&)
          |       | |     + 2.00% std::enable_if<denc_traits<std::vector<int, std::allocator<int> >, void>::supported&&(!denc_traits<std::vector<int, std::allocator<int> >, void>::need_contiguous), void>::type ceph::decode<std::vector<int, std::allocator<int> >, denc_traits<std::vector<int, std::allocator<int> >, void> >(std::vector<int, std::allocator<int> >&, ceph::buffer::v14_2_0::list::iterator_impl<true>&)
          |       | |     | + 1.00% int& std::vector<int, std::allocator<int> >::emplace_back<int>(int&&)
          |       | |     | | + 1.00% tc_free
          |       | |     | + 1.00% ceph::buffer::v14_2_0::list::iterator_impl<true>::copy_shallow(unsigned int, ceph::buffer::v14_2_0::ptr&)
          |       | |     |   + 1.00% ceph::buffer::v14_2_0::ptr::ptr(ceph::buffer::v14_2_0::ptr const&, unsigned int, unsigned int)
          |       | |     + 1.00% std::enable_if<denc_traits<interval_set<snapid_t, std::map<snapid_t, snapid_t, std::less<snapid_t>, std::allocator<std::pair<snapid_t const, snapid_t> > > >, void>::supported&&denc_traits<interval_set<snapid_t, std::map<snapid_t, snapid_t, std::less<snapid_t>, std::allocator<std::pair<snapid_t const, snapid_t> > > >, void>::need_contiguous, void>::type ceph::decode<interval_set<snapid_t, std::map<snapid_t, snapid_t, std::less<snapid_t>, std::allocator<std::pair<snapid_t const, snapid_t> > > >, denc_traits<interval_set<snapid_t, std::map<snapid_t, snapid_t, std::less<snapid_t>, std::allocator<std::pair<snapid_t const, snapid_t> > > >, void> >(interval_set<snapid_t, std::map<snapid_t, snapid_t, std::less<snapid_t>, std::allocator<std::pair<snapid_t const, snapid_t> > > >&, ceph::buffer::v14_2_0::list::iterator_impl<true>&)
          |       | |     | + 1.00% ceph::buffer::v14_2_0::ptr::release()
          |       | |     + 1.00% ceph::buffer::v14_2_0::list::iterator_impl<true>::copy(unsigned int, char*)
          |       | + 1.00% DispatchQueue::enqueue(boost::intrusive_ptr<Message> const&, int, unsigned long)
          |       |   + 1.00% Mutex::lock(bool)
          |       |     + 1.00% __GI___pthread_mutex_lock
          |       |       + 1.00% __lll_lock_wait
          |       + 1.00% ProtocolV2::handle_frame_payload()
          |         + 1.00% ProtocolV2::handle_auth_request(ceph::buffer::v14_2_0::list&)
          |           + 1.00% ProtocolV2::_handle_auth_request(ceph::buffer::v14_2_0::list&, bool)
          |             + 1.00% MonClient::handle_auth_request(Connection*, AuthConnectionMeta*, bool, unsigned int, ceph::buffer::v14_2_0::list const&, ceph::buffer::v14_2_0::list*)
          |               + 1.00% CephxAuthorizeHandler::verify_authorizer(CephContext*, KeyStore*, ceph::buffer::v14_2_0::list const&, unsigned long, ceph::buffer::v14_2_0::list*, EntityName*, unsigned long*, AuthCapsInfo*, CryptoKey*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::unique_ptr<AuthAuthorizerChallenge, std::default_delete<AuthAuthorizerChallenge> >*)
          |                 + 1.00% cephx_verify_authorizer(CephContext*, KeyStore*, ceph::buffer::v14_2_0::list::iterator_impl<true>&, unsigned long, CephXServiceTicketInfo&, std::unique_ptr<AuthAuthorizerChallenge, std::default_delete<AuthAuthorizerChallenge> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, ceph::buffer::v14_2_0::list*)
          |                   + 1.00% void decode_decrypt_enc_bl<CephXServiceTicketInfo>(CephContext*, CephXServiceTicketInfo&, CryptoKey, ceph::buffer::v14_2_0::list const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)
          |                     + 1.00% CryptoAESKeyHandler::decrypt(ceph::buffer::v14_2_0::list const&, ceph::buffer::v14_2_0::list&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) const
          |                       + 1.00% AES_cbc_encrypt
          |                         + 1.00% ???
          + 1.00% EventCenter::process_time_events()
            + 1.00% AsyncConnection::wakeup_from(unsigned long)
              + 1.00% AsyncConnection::process()
                + 1.00% ProtocolV2::read_event()
                  + 1.00% ProtocolV2::run_continuation(Ct<ProtocolV2>&)
                    + 1.00% ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr<ceph::buffer::v14_2_0::ptr_node, ceph::buffer::v14_2_0::ptr_node::disposer>&&, int)
                      + 1.00% ProtocolV2::handle_read_frame_dispatch()
                        + 1.00% ProtocolV2::handle_message()
                          + 1.00% decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::v14_2_0::list&, ceph::buffer::v14_2_0::list&, ceph::buffer::v14_2_0::list&, Connection*)
                            + 1.00% MPGStats::decode_payload()
                              + 1.00% pg_stat_t::decode(ceph::buffer::v14_2_0::list::iterator_impl<true>&)
                                + 1.00% object_stat_collection_t::decode(ceph::buffer::v14_2_0::list::iterator_impl<true>&)
                                  + 1.00% object_stat_sum_t::decode(ceph::buffer::v14_2_0::list::iterator_impl<true>&)
                                    + 1.00% ceph::buffer::v14_2_0::list::iterator_impl<true>::copy(unsigned int, char*)

Thread: 3 (msgr-worker-1) - 100 samples 

+ 100.00% clone
  + 100.00% start_thread
    + 100.00% ???
      + 100.00% ???
        + 100.00% EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)
          + 88.00% EpollDriver::event_wait(std::vector<FiredFileEvent, std::allocator<FiredFileEvent> >&, timeval*)
          | + 88.00% epoll_wait
          + 9.00% AsyncConnection::process()
          | + 9.00% ProtocolV2::run_continuation(Ct<ProtocolV2>&)
          |   + 7.00% ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr<ceph::buffer::v14_2_0::ptr_node, ceph::buffer::v14_2_0::ptr_node::disposer>&&, int)
          |   | + 6.00% ProtocolV2::handle_read_frame_dispatch()
          |   | | + 5.00% ProtocolV2::handle_message()
          |   | | | + 5.00% decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::v14_2_0::list&, ceph::buffer::v14_2_0::list&, ceph::buffer::v14_2_0::list&, Connection*)
          |   | | |   + 5.00% MPGStats::decode_payload()
          |   | | |     + 5.00% pg_stat_t::decode(ceph::buffer::v14_2_0::list::iterator_impl<true>&)
          |   | | |       + 2.00% ceph::buffer::v14_2_0::list::iterator_impl<true>::copy(unsigned int, char*)
          |   | | |       + 1.00% std::enable_if<denc_traits<std::vector<int, std::allocator<int> >, void>::supported&&(!denc_traits<std::vector<int, std::allocator<int> >, void>::need_contiguous), void>::type ceph::decode<std::vector<int, std::allocator<int> >, denc_traits<std::vector<int, std::allocator<int> >, void> >(std::vector<int, std::allocator<int> >&, ceph::buffer::v14_2_0::list::iterator_impl<true>&)
          |   | | |       | + 1.00% int& std::vector<int, std::allocator<int> >::emplace_back<int>(int&&)
          |   | | |       |   + 1.00% tc_free
          |   | | |       + 1.00% std::enable_if<denc_traits<interval_set<snapid_t, std::map<snapid_t, snapid_t, std::less<snapid_t>, std::allocator<std::pair<snapid_t const, snapid_t> > > >, void>::supported&&denc_traits<interval_set<snapid_t, std::map<snapid_t, snapid_t, std::less<snapid_t>, std::allocator<std::pair<snapid_t const, snapid_t> > > >, void>::need_contiguous, void>::type ceph::decode<interval_set<snapid_t, std::map<snapid_t, snapid_t, std::less<snapid_t>, std::allocator<std::pair<snapid_t const, snapid_t> > > >, denc_traits<interval_set<snapid_t, std::map<snapid_t, snapid_t, std::less<snapid_t>, std::allocator<std::pair<snapid_t const, snapid_t> > > >, void> >(interval_set<snapid_t, std::map<snapid_t, snapid_t, std::less<snapid_t>, std::allocator<std::pair<snapid_t const, snapid_t> > > >&, ceph::buffer::v14_2_0::list::iterator_impl<true>&)
          |   | | |       | + 1.00% ceph::buffer::v14_2_0::ptr::c_str() const
          |   | | |       + 1.00% object_stat_collection_t::decode(ceph::buffer::v14_2_0::list::iterator_impl<true>&)
          |   | | |         + 1.00% ceph::buffer::v14_2_0::list::iterator_impl<true>::copy(unsigned int, char*)
          |   | | + 1.00% ProtocolV2::handle_frame_payload()
          |   | |   + 1.00% ProtocolV2::handle_auth_signature(ceph::buffer::v14_2_0::list&)
          |   | |     + 1.00% ProtocolV2::send_client_ident()
          |   | |       + 1.00% ProtocolV2::write(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, CtFun<ProtocolV2>&, ceph::buffer::v14_2_0::list&)
          |   | |         + 1.00% AsyncConnection::write(ceph::buffer::v14_2_0::list&, std::function<void (long)>, bool)
          |   | |           + 1.00% AsyncConnection::_try_send(bool)
          |   | |             + 1.00% PosixConnectedSocketImpl::send(ceph::buffer::v14_2_0::list&, bool)
          |   | |               + 1.00% __libc_sendmsg
          |   | + 1.00% ceph::buffer::v14_2_0::list::crc32c(unsigned int) const
          |   |   + 1.00% ceph_crc32c_intel_fast
          |   |     + 1.00% ???
          |   |       + 1.00% ???
          |   |         + 1.00% crc32_iscsi_00
          |   + 1.00% ProtocolV2::handle_read_frame_segment(std::unique_ptr<ceph::buffer::v14_2_0::ptr_node, ceph::buffer::v14_2_0::ptr_node::disposer>&&, int)
          |   | + 1.00% ProtocolV2::read_frame_segment()
          |   |   + 1.00% ProtocolV2::read(CtRxNode<ProtocolV2>&, std::unique_ptr<ceph::buffer::v14_2_0::ptr_node, ceph::buffer::v14_2_0::ptr_node::disposer>&&)
          |   |     + 1.00% AsyncConnection::read(unsigned int, char*, std::function<void (char*, long)>)
          |   |       + 1.00% AsyncConnection::read_until(unsigned int, char*)
          |   |         + 1.00% AsyncConnection::read_bulk(char*, unsigned int)
          |   |           + 1.00% PosixConnectedSocketImpl::read(char*, unsigned long)
          |   |             + 1.00% __libc_read
          |   + 1.00% ProtocolV2::_handle_peer_banner_payload(std::unique_ptr<ceph::buffer::v14_2_0::ptr_node, ceph::buffer::v14_2_0::ptr_node::disposer>&&, int)
          |     + 1.00% ProtocolV2::write(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, CtFun<ProtocolV2>&, ceph::buffer::v14_2_0::list&)
          |       + 1.00% AsyncConnection::write(ceph::buffer::v14_2_0::list&, std::function<void (long)>, bool)
          |         + 1.00% AsyncConnection::_try_send(bool)
          |           + 1.00% PosixConnectedSocketImpl::send(ceph::buffer::v14_2_0::list&, bool)
          |             + 1.00% __libc_sendmsg
          + 3.00% EventCenter::process_time_events()
            + 3.00% AsyncConnection::wakeup_from(unsigned long)
              + 3.00% AsyncConnection::process()
                + 3.00% ProtocolV2::read_event()
                  + 3.00% ProtocolV2::run_continuation(Ct<ProtocolV2>&)
                    + 2.00% ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr<ceph::buffer::v14_2_0::ptr_node, ceph::buffer::v14_2_0::ptr_node::disposer>&&, int)
                    | + 2.00% ProtocolV2::handle_read_frame_dispatch()
                    |   + 2.00% ProtocolV2::handle_message()
                    |     + 2.00% decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::v14_2_0::list&, ceph::buffer::v14_2_0::list&, ceph::buffer::v14_2_0::list&, Connection*)
                    |       + 2.00% MPGStats::decode_payload()
                    |         + 1.00% pg_t::decode(ceph::buffer::v14_2_0::list::iterator_impl<true>&)
                    |         + 1.00% pg_stat_t::decode(ceph::buffer::v14_2_0::list::iterator_impl<true>&)
                    |           + 1.00% ceph::buffer::v14_2_0::list::iterator_impl<true>::copy(unsigned int, char*)
                    |             + 1.00% ceph::buffer::v14_2_0::ptr::copy_out(unsigned int, unsigned int, char*) const
                    + 1.00% ProtocolV2::throttle_message()
                      + 1.00% EventCenter::create_time_event(unsigned long, EventCallback*)
                        + 1.00% std::_Rb_tree_insert_and_rebalance(bool, std::_Rb_tree_node_base*, std::_Rb_tree_node_base*, std::_Rb_tree_node_base&)

Thread: 4 (msgr-worker-2) - 100 samples 

+ 100.00% clone
  + 100.00% start_thread
    + 100.00% ???
      + 100.00% ???
        + 100.00% EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)
          + 90.00% EpollDriver::event_wait(std::vector<FiredFileEvent, std::allocator<FiredFileEvent> >&, timeval*)
          | + 90.00% epoll_wait
          + 8.00% AsyncConnection::process()
          | + 7.00% ProtocolV2::run_continuation(Ct<ProtocolV2>&)
          | | + 6.00% ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr<ceph::buffer::v14_2_0::ptr_node, ceph::buffer::v14_2_0::ptr_node::disposer>&&, int)
          | | | + 5.00% ProtocolV2::handle_read_frame_dispatch()
          | | | | + 4.00% ProtocolV2::handle_message()
          | | | | | + 4.00% decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::v14_2_0::list&, ceph::buffer::v14_2_0::list&, ceph::buffer::v14_2_0::list&, Connection*)
          | | | | |   + 4.00% MPGStats::decode_payload()
          | | | | |     + 4.00% pg_stat_t::decode(ceph::buffer::v14_2_0::list::iterator_impl<true>&)
          | | | | |       + 2.00% ceph::buffer::v14_2_0::list::iterator_impl<true>::copy(unsigned int, char*)
          | | | | |       | + 2.00% ceph::buffer::v14_2_0::ptr::copy_out(unsigned int, unsigned int, char*) const
          | | | | |       + 1.00% std::enable_if<denc_traits<std::vector<int, std::allocator<int> >, void>::supported&&(!denc_traits<std::vector<int, std::allocator<int> >, void>::need_contiguous), void>::type ceph::decode<std::vector<int, std::allocator<int> >, denc_traits<std::vector<int, std::allocator<int> >, void> >(std::vector<int, std::allocator<int> >&, ceph::buffer::v14_2_0::list::iterator_impl<true>&)
          | | | | |       | + 1.00% ceph::buffer::v14_2_0::ptr::release()
          | | | | |       + 1.00% object_stat_collection_t::decode(ceph::buffer::v14_2_0::list::iterator_impl<true>&)
          | | | | |         + 1.00% object_stat_sum_t::decode(ceph::buffer::v14_2_0::list::iterator_impl<true>&)
          | | | | |           + 1.00% ceph::buffer::v14_2_0::list::iterator_impl<true>::copy(unsigned int, char*)
          | | | | + 1.00% ProtocolV2::handle_frame_payload()
          | | | |   + 1.00% ProtocolV2::handle_client_ident(ceph::buffer::v14_2_0::list&)
          | | | |     + 1.00% std::_Hashtable<entity_addrvec_t, std::pair<entity_addrvec_t const, boost::intrusive_ptr<AsyncConnection> >, std::allocator<std::pair<entity_addrvec_t const, boost::intrusive_ptr<AsyncConnection> > >, std::__detail::_Select1st, std::equal_to<entity_addrvec_t>, std::hash<entity_addrvec_t>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::erase(std::__detail::_Node_const_iterator<std::pair<entity_addrvec_t const, boost::intrusive_ptr<AsyncConnection> >, false, true>)
          | | | |       + 1.00% RefCountedObject::put() const
          | | | |         + 1.00% AsyncConnection::~AsyncConnection()
          | | | |           + 1.00% AsyncConnection::~AsyncConnection()
          | | | |             + 1.00% RefCountedObject::put() const
          | | | |               + 1.00% MgrSession::~MgrSession()
          | | | + 1.00% ceph::buffer::v14_2_0::list::crc32c(unsigned int) const
          | | |   + 1.00% ceph_crc32c_intel_fast
          | | |     + 1.00% ???
          | | |       + 1.00% ???
          | | |         + 1.00% crc32_iscsi_00
          | | + 1.00% ProtocolV2::handle_read_frame_segment(std::unique_ptr<ceph::buffer::v14_2_0::ptr_node, ceph::buffer::v14_2_0::ptr_node::disposer>&&, int)
          | |   + 1.00% ProtocolV2::read_frame_segment()
          | |     + 1.00% ProtocolV2::read(CtRxNode<ProtocolV2>&, std::unique_ptr<ceph::buffer::v14_2_0::ptr_node, ceph::buffer::v14_2_0::ptr_node::disposer>&&)
          | |       + 1.00% AsyncConnection::read(unsigned int, char*, std::function<void (char*, long)>)
          | |         + 1.00% AsyncConnection::read_until(unsigned int, char*)
          | |           + 1.00% AsyncConnection::read_bulk(char*, unsigned int)
          | |             + 1.00% PosixConnectedSocketImpl::read(char*, unsigned long)
          | |               + 1.00% __libc_read
          | + 1.00% AsyncConnection::read(unsigned int, char*, std::function<void (char*, long)>)
          |   + 1.00% AsyncConnection::read_until(unsigned int, char*)
          |     + 1.00% AsyncConnection::read_bulk(char*, unsigned int)
          |       + 1.00% PosixConnectedSocketImpl::read(char*, unsigned long)
          |         + 1.00% __libc_read
          + 2.00% EventCenter::process_time_events()
            + 1.00% std::_Rb_tree_rebalance_for_erase(std::_Rb_tree_node_base*, std::_Rb_tree_node_base&)
            + 1.00% AsyncConnection::wakeup_from(unsigned long)
              + 1.00% AsyncConnection::process()
                + 1.00% ProtocolV2::read_event()
                  + 1.00% ProtocolV2::run_continuation(Ct<ProtocolV2>&)
                    + 1.00% ProtocolV2::handle_read_frame_segment(std::unique_ptr<ceph::buffer::v14_2_0::ptr_node, ceph::buffer::v14_2_0::ptr_node::disposer>&&, int)
                      + 1.00% ceph::buffer::v14_2_0::list::iterator_impl<false>::iterator_impl(ceph::buffer::v14_2_0::list*, unsigned int)

Thread: 5 (log) - 100 samples 

+ 100.00% clone
  + 100.00% start_thread
    + 100.00% ceph::logging::Log::entry()
      + 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
Traceback (most recent call last):
  File "gdbpmp.py", line 44, in <module>
    import tracer
  File "/root/gdbpmp/tracer.py", line 25, in <module>
    import gdb
ModuleNotFoundError: No module named 'gdb'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "gdbpmp.py", line 46, in <module>
    main()
  File "gdbpmp.py", line 25, in main
    common.print_callgraph(threads, ctx.threshold)
  File "/root/gdbpmp/common.py", line 34, in print_callgraph
    gdbth.function.print_percent("", samples, threshold, True)
  File "/root/gdbpmp/gdbtypes.py", line 85, in print_percent
    self.get_func(name).print_percent(prefix + new_prefix, total, threshold, include_sub)
  File "/root/gdbpmp/gdbtypes.py", line 85, in print_percent
    self.get_func(name).print_percent(prefix + new_prefix, total, threshold, include_sub)
  File "/root/gdbpmp/gdbtypes.py", line 85, in print_percent
    self.get_func(name).print_percent(prefix + new_prefix, total, threshold, include_sub)
  [Previous line repeated 8 more times]
  File "/root/gdbpmp/gdbtypes.py", line 79, in print_percent
    print ("%s%s%0.2f%% %s" % (prefix, "+ ", value, name))
BrokenPipeError: [Errno 32] Broken pipe

at the end

Thread: 62 (safe_timer) - 100 samples 

+ 100.00% clone
  + 100.00% start_thread
    + 100.00% SafeTimerThread::entry()
      + 100.00% SafeTimer::timer_thread()
        + 100.00% __pthread_cond_timedwait
          + 100.00% __pthread_cond_wait_common
            + 100.00% futex_abstimed_wait_cancelable

Thread: 63 (fn_anonymous) - 100 samples 

+ 100.00% clone
  + 100.00% start_thread
    + 100.00% Finisher::finisher_thread_entry()
      + 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
        + 100.00% __pthread_cond_wait
          + 100.00% __pthread_cond_wait_common
            + 100.00% futex_wait_cancelable

Thread: 64 (safe_timer) - 100 samples 

+ 100.00% clone
  + 100.00% start_thread
    + 100.00% SafeTimerThread::entry()
      + 100.00% SafeTimer::timer_thread()
        + 95.00% __pthread_cond_timedwait
        | + 95.00% __pthread_cond_wait_common
        |   + 95.00% futex_abstimed_wait_cancelable
        + 5.00% __pthread_cond_wait
          + 5.00% __pthread_cond_wait_common
            + 5.00% futex_wait_cancelable

Thread: 65 (safe_timer) - 100 samples 

+ 100.00% clone
  + 100.00% start_thread
    + 100.00% SafeTimerThread::entry()
      + 100.00% SafeTimer::timer_thread()
        + 100.00% __pthread_cond_wait
          + 100.00% __pthread_cond_wait_common
            + 100.00% futex_wait_cancelable

Thread: 66 (fn-radosclient) - 100 samples 

+ 100.00% clone
  + 100.00% start_thread
    + 100.00% Finisher::finisher_thread_entry()
      + 100.00% std::condition_variable::wait(std::unique_lock<std::mutex>&)
        + 100.00% __pthread_cond_wait
          + 100.00% __pthread_cond_wait_common
            + 100.00% futex_wait_cancelable
Actions
Copy link
 #14

Updated by Anonymous about 4 years ago

new dump...after disabling almost all mgr modules

Actions
Copy link
 #15

Updated by Anonymous about 4 years ago

fixed with 14.2.8

Actions
Copy link
 #16

Updated by Lenz Grimmer almost 4 years ago

  • Related to Bug #39264: Ceph-mgr Hangup and _check_auth_rotating possible clock skew, rotating keys expired way too early Errors added
Actions
Copy link
 #17

Updated by Neha Ojha over 3 years ago

  • Status changed from Need More Info to Resolved
Actions
Copy link

Also available in: Atom PDF