Actions
Bug #65851
openMDS Squid Metadata Performance Regression
Status:
New
Priority:
Urgent
Assignee:
Category:
Performance/Resource Usage
Target version:
% Done:
0%
Source:
Development
Tags:
Backport:
squid
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS, quiesce
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Found during 21 MDS IO500 runs comparing v18.2.2 to v19.0.0.
| Ceph Version | Meaurement | v18.2.2 | v18.2.2 | v19.0.0-2502 | v19.0.0-2502 |
| ---------------------------- | --------------- | --------------- | --------------- | --------------- | --------------- |
| mdtest-easy pinning strategy | n-1 round-robin | n-1 round-robin | n-1 round-robin | n-1 round-robin | |
| API | kernel | libcephfs | kernel | libcephfs | |
| LazyIO | no | yes | no | yes | |
| Msgr Threads | 3 | 3 | 3 | 3 | |
| Test | Result | Result | Result | Result | |
| ior-easy-write | GiB/s | 11.53 | 29.22 | 11.71 | 28.97 |
| mdtest-easy-write | kIOPS | 152.44 | 127.72 | 47.41 | 38.00 |
| timestamp | kIOPS | 0.00 | 0.00 | 0.00 | 0.00 |
| ior-hard-write | GiB/s | 0.39 | 7.49 | 0.40 | 8.82 |
| mdtest-hard-write | kIOPS | 9.69 | 22.09 | 7.05 | 6.09 |
| find | kIOPS | 478.83 | 344.13 | 232.55 | 278.90 |
| ior-easy-read | GiB/s | 27.53 | 33.65 | 31.84 | 33.40 |
| mdtest-easy-stat | kIOPS | 234.07 | 260.74 | 232.42 | 340.22 |
| ior-hard-read | GiB/s | 3.50 | 10.94 | 3.18 | 12.97 |
| mdtest-hard-stat | kIOPS | 112.25 | 119.67 | 36.51 | 35.68 |
| mdtest-easy-delete | kIOPS | 52.00 | 49.41 | 29.20 | 30.02 |
| mdtest-hard-read | kIOPS | 44.67 | 42.09 | 7.19 | 35.57 |
| mdtest-hard-delete | kIOPS | 10.04 | 36.11 | 5.56 | 5.58 |
| Bandwidth | GiB/s | 4.56 | 16.85 | 4.66 | 18.24 |
| IOPS | kIOPS | 67.55 | 83.11 | 30.61 | 38.34 |
| Total | Score | 17.55 | 37.42 | 11.95 | 26.45 |
A bisection of the mdtest-easy results indicates https://github.com/ceph/ceph/commit/a8cbcde03951adabe13442c2e8b244794ebd86dd as the likely culprit.
| Version | Step | mdtest-easy-write (kIOPS) |
| ----------------------- | ---- | ------------------------- |
| 18.2.2 | 1 | 127.72 |
| v19.0.0-898 (1112689d) | 2 | 130.89 |
| v19.0.0-1700 (ga499fbd) | 3 | 136.30 |
| v19.0.0-1762 (0618afe6) | 7 | 133.60 |
| v19.0.0-1794 (7bdc74ec) | 6 | stalled (fast?) |
| v19.0.0-1806 (7dacd283) | 11 | stalled (fast?) |
| v19.0.0-1807 (a8cbcde0) | 14 | 39.03 |
| v19.0.0-1809 (45c18be1) | 13 | 38.92 |
| v19.0.0-1811 (36d314c0) | 12 | 38.59 |
| v19.0.0-1815 (5fdf9c56) | 10 | 38.06 |
| v19.0.0-1816 (9d7f5b04) | 9 | 38.59 |
| v19.0.0-1825 (2f8b96de) | 8 | 37.94 |
| v19.0.0-1893 (cc7ac758) | 5 | 38.27 |
| v19.0.0-2099 (c531a773) | 4 | 38.92 |
| v19.0.0-2502 (742a2451) | 0 | 38.00 |
Updated by Patrick Donnelly 12 days ago
- Assignee set to Patrick Donnelly
- Priority changed from Normal to Urgent
- Target version changed from v19.1.0 to v20.0.0
- Source set to Development
- Backport set to squid
- Affected Versions v19.0.0 added
- Component(FS) MDS, quiesce added
Actions