Actions
Bug #5765
openkclient: High CPU due to raw_spin_lock in ceph_cap_string
Status:
New
Priority:
Normal
Assignee:
-
Category:
Performance/Resource Usage
Target version:
-
% Done:
0%
Source:
other
Tags:
perf
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
kceph
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
During CephFS testing, very high CPU usage was noticed during writes across lots of clients. perf showed that a significant amount of time was spent in ceph_cap_string doing raw_spin_lock. Sage believes this may be because CONFIG_DYNAMIC_DEBUG wasn't compiled into the kernel and we shouldn't expect that it always will be.
Here's the relevant part of the perf callgraph:
71.62% IOR [kernel.kallsyms] [k] _raw_spin_lock
|
--- _raw_spin_lock
|
|--98.19%-- ceph_cap_string
| |
| |--44.55%-- ceph_check_caps
| | |
| | |--100.00%-- ceph_put_cap_refs
| | | ceph_write_end
| | | generic_perform_write
| | | generic_file_buffered_write
| | | __generic_file_aio_write
| | | generic_file_aio_write
| | | ceph_aio_write
| | | do_sync_write
| | | vfs_write
| | | sys_write
| | | system_call_fastpath
| | | |
| | | |--13.77%-- 0x7f637afd848d
| | | | WriteOrRead
| | | | TestIoSys
| | | | main
| | | | __libc_start_main
| | | |
| | | |--12.69%-- 0x7f47a6bc648d
| | | | WriteOrRead
| | | | TestIoSys
| | | | main
| | | | __libc_start_main
| | | |
| | | |--12.56%-- 0x7f606569548d
| | | | WriteOrRead
| | | | TestIoSys
| | | | main
| | | | __libc_start_main
| | | |
| | | |--12.47%-- 0x7fc82977148d
| | | | WriteOrRead
| | | | TestIoSys
| | | | main
| | | | __libc_start_main
| | | |
| | | |--12.31%-- 0x7fb96d93048d
| | | | WriteOrRead
| | | | TestIoSys
| | | | main
| | | | __libc_start_main
| | | |
| | | |--12.20%-- 0x7f78f046a48d
| | | | WriteOrRead
| | | | TestIoSys
| | | | main
| | | | __libc_start_main
| | | |
| | | |--12.00%-- 0x7f1739c0948d
| | | | WriteOrRead
| | | | TestIoSys
| | | | main
| | | | __libc_start_main
| | | |
| | | --11.99%-- 0x7fd22ec7f48d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | --0.00%-- [...]
| |
| |--23.78%-- try_get_cap_refs
| | ceph_get_caps
| | ceph_write_begin
| | generic_perform_write
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ceph_aio_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | |
| | |--13.76%-- 0x7f637afd848d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.74%-- 0x7f47a6bc648d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.67%-- 0x7fc82977148d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.54%-- 0x7f606569548d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.22%-- 0x7f78f046a48d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.16%-- 0x7fb96d93048d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.13%-- 0x7fd22ec7f48d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | --11.78%-- 0x7f1739c0948d
| | WriteOrRead
| | TestIoSys
| | main
| | __libc_start_main
| |
| |--12.73%-- __ceph_mark_dirty_caps
| | ceph_write_end
| | generic_perform_write
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ceph_aio_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | |
| | |--14.75%-- 0x7f637afd848d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.68%-- 0x7f47a6bc648d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.63%-- 0x7f606569548d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.41%-- 0x7fc82977148d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.25%-- 0x7f78f046a48d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.05%-- 0x7fd22ec7f48d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--11.79%-- 0x7fb96d93048d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | --11.44%-- 0x7f1739c0948d
| | WriteOrRead
| | TestIoSys
| | main
| | __libc_start_main
| |
| |--7.30%-- __ceph_caps_issued
| | |
| | |--50.04%-- try_get_cap_refs
| | | ceph_get_caps
| | | ceph_write_begin
| | | generic_perform_write
| | | generic_file_buffered_write
| | | __generic_file_aio_write
| | | generic_file_aio_write
| | | ceph_aio_write
| | | do_sync_write
| | | vfs_write
| | | sys_write
| | | system_call_fastpath
| | | |
| | | |--13.93%-- 0x7f637afd848d
| | | | WriteOrRead
| | | | TestIoSys
| | | | main
| | | | __libc_start_main
| | | |
| | | |--13.03%-- 0x7f47a6bc648d
| | | | WriteOrRead
| | | | TestIoSys
| | | | main
| | | | __libc_start_main
| | | |
| | | |--12.58%-- 0x7fc82977148d
| | | | WriteOrRead
| | | | TestIoSys
| | | | main
| | | | __libc_start_main
| | | |
| | | |--12.45%-- 0x7f606569548d
| | | | WriteOrRead
| | | | TestIoSys
| | | | main
| | | | __libc_start_main
| | | |
| | | |--12.05%-- 0x7f78f046a48d
| | | | WriteOrRead
| | | | TestIoSys
| | | | main
| | | | __libc_start_main
| | | |
| | | |--12.02%-- 0x7fd22ec7f48d
| | | | WriteOrRead
| | | | TestIoSys
| | | | main
| | | | __libc_start_main
| | | |
| | | |--11.99%-- 0x7fb96d93048d
| | | | WriteOrRead
| | | | TestIoSys
| | | | main
| | | | __libc_start_main
| | | |
| | | --11.95%-- 0x7f1739c0948d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | --49.96%-- ceph_check_caps
| | ceph_put_cap_refs
| | ceph_write_end
| | generic_perform_write
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ceph_aio_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | |
| | |--13.68%-- 0x7f637afd848d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--13.03%-- 0x7f47a6bc648d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.65%-- 0x7fc82977148d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.26%-- 0x7f606569548d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.18%-- 0x7f1739c0948d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.09%-- 0x7f78f046a48d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.08%-- 0x7fb96d93048d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | --12.02%-- 0x7fd22ec7f48d
| | WriteOrRead
| | TestIoSys
| | main
| | __libc_start_main
| |
| |--4.11%-- ceph_write_begin
| | generic_perform_write
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ceph_aio_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | |
| | |--13.63%-- 0x7f637afd848d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--13.49%-- 0x7f47a6bc648d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.39%-- 0x7fb96d93048d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.32%-- 0x7f606569548d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.10%-- 0x7f1739c0948d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.06%-- 0x7fc82977148d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.05%-- 0x7f78f046a48d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | --11.97%-- 0x7fd22ec7f48d
| | WriteOrRead
| | TestIoSys
| | main
| | __libc_start_main
| |
| |--3.84%-- ceph_put_cap_refs
| | ceph_write_end
| | generic_perform_write
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ceph_aio_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | |
| | |--13.55%-- 0x7f637afd848d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.61%-- 0x7f47a6bc648d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.59%-- 0x7f78f046a48d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.54%-- 0x7fc82977148d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.49%-- 0x7f606569548d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.19%-- 0x7f1739c0948d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.17%-- 0x7fb96d93048d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | --11.87%-- 0x7fd22ec7f48d
| | WriteOrRead
| | TestIoSys
| | main
| | __libc_start_main
| |
| |--3.68%-- ceph_write_end
| | generic_perform_write
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ceph_aio_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | |
| | |--13.67%-- 0x7f637afd848d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.63%-- 0x7f78f046a48d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.49%-- 0x7f47a6bc648d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.45%-- 0x7fc82977148d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.40%-- 0x7f606569548d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.17%-- 0x7f1739c0948d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | |--12.11%-- 0x7fd22ec7f48d
| | | WriteOrRead
| | | TestIoSys
| | | main
| | | __libc_start_main
| | |
| | --12.08%-- 0x7fb96d93048d
| | WriteOrRead
| | TestIoSys
| | main
| | __libc_start_main
| --0.01%-- [...]
--1.81%-- [...]Updated by Greg Farnum about 10 years ago
- Priority changed from High to Normal
Demoting due to performance, not correctness.
Updated by Greg Farnum almost 8 years ago
- Category changed from 53 to Performance/Resource Usage
Updated by Patrick Donnelly about 6 years ago
- Subject changed from High CPU due to raw_spin_lock in ceph_cap_string to kclient: High CPU due to raw_spin_lock in ceph_cap_string
- Tags set to perf
Actions