Project

General

Profile

Bug #39713

"Ceph -s" execution consumes too much time

Added by Lei Liu 2 months ago. Updated 15 days ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
Start date:
05/13/2019
Due date:
% Done:

0%

Source:
Tags:
ceph cli
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

ceph version: 14.2.0 update from luminous

  1. strace -c ceph -s
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 99.91    0.016000          74       216           select
  0.09    0.000015           0       385           read
  0.00    0.000000           0        19           write
  0.00    0.000000           0      1151       948 open
  0.00    0.000000           0       207           close
  0.00    0.000000           0       301       250 stat
  0.00    0.000000           0       348           fstat
  0.00    0.000000           0         6           lstat
  0.00    0.000000           0        62        58 lseek
  0.00    0.000000           0       138           mmap
  0.00    0.000000           0        70           mprotect
  0.00    0.000000           0        30           munmap
  0.00    0.000000           0        57           brk
  0.00    0.000000           0        68           rt_sigaction
  0.00    0.000000           0         1           rt_sigprocmask
  0.00    0.000000           0         6         1 ioctl
  0.00    0.000000           0        28        28 access
  0.00    0.000000           0         2           pipe
  0.00    0.000000           0         1           mremap
  0.00    0.000000           0         7           clone
  0.00    0.000000           0         1           execve
  0.00    0.000000           0         1           wait4
  0.00    0.000000           0         1           uname
  0.00    0.000000           0        11           fcntl
  0.00    0.000000           0         4           getdents
  0.00    0.000000           0         2         2 readlink
  0.00    0.000000           0         2           getrlimit
  0.00    0.000000           0         1           sysinfo
  0.00    0.000000           0         1           getuid
  0.00    0.000000           0         1           getgid
  0.00    0.000000           0         1           geteuid
  0.00    0.000000           0         1           getegid
  0.00    0.000000           0         1         1 statfs
  0.00    0.000000           0         1           arch_prctl
  0.00    0.000000           0        25         8 futex
  0.00    0.000000           0         1           set_tid_address
  0.00    0.000000           0         1           set_robust_list
  0.00    0.000000           0         1           membarrier
------ ----------- ----------- --------- --------- ----------------
100.00    0.016015                  3160      1296 total
mmap(NULL, 8392704, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f2fd4de4000
mprotect(0x7f2fd4de4000, 4096, PROT_NONE) = 0
clone(child_stack=0x7f2fd55e3ff0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f2fd55e49d0, tls=0x7f2fd55e4700, child_tidptr=0x7f2fd55e49d0) = 3768757
futex(0x14e4870, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, ffffffff) = 0
futex(0x143e760, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, ffffffff) = 0
select(0, NULL, NULL, NULL, {0, 1000})  = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 2000})  = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 4000})  = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 8000})  = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 16000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 32000}) = 0 (Timeout)
clone(child_stack=0x7f2fd55e3ff0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f2fd55e49d0, tls=0x7f2fd55e4700, child_tidptr=0x7f2fd55e49d0) = 3768759
futex(0x143e760, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x14907c0, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x143e760, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x143e760, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, ffffffff) = 0
clone(child_stack=0x7f2fd55e3ff0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f2fd55e49d0, tls=0x7f2fd55e4700, child_tidptr=0x7f2fd55e49d0) = 3768760
futex(0x143e760, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x143e760, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, ffffffff) = -1 EAGAIN (Resource temporarily unavailable)
select(0, NULL, NULL, NULL, {0, 1000})  = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 2000})  = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 4000})  = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 8000})  = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 16000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 32000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)

lots of Timeout message

What system parameter settings are unreasonable for ceph 14.2.0 ?

History

#1 Updated by Casey Bodley 2 months ago

  • Project changed from rgw to Ceph

#2 Updated by Nathan Fish 18 days ago

Has there been any progress on this? If not, can it be moved to 14.2.3? This is one of 2 issues blocking 14.2.2 and there are a lot of important bugs fixed in 14.2.2.

#3 Updated by Lei Liu 18 days ago

Nathan Fish wrote:

Has there been any progress on this? If not, can it be moved to 14.2.3? This is one of 2 issues blocking 14.2.2 and there are a lot of important bugs fixed in 14.2.2.

Hi Nathan,

The problem still exists, cluster ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable), I checked the official website of ceph. The latest release version is 14.2.1. I have not found the updated release version. What am I missing?

Thanks for your reply.

#4 Updated by Nathan Fish 15 days ago

14.2.2 has indeed not been released yet, that's the problem. This is one of two issues which are listed as blocking the release of 14.2.2:

https://tracker.ceph.com/versions/574

14.2.2 has multiple important bug fixes which I would like to put into production soon. So I would like to know if this issue is nearly done, or if it can be moved to 14.2.3 so 14.2.2 can ship.

#5 Updated by Lei Liu 15 days ago

Of course, but I don't have no permission to modify, I think this problem may be caused by the operating system settings, currently not resolved, the administrator can close this issue.

Will test while v14.2.2 released.

Also available in: Atom PDF