Actions
Bug #57872
closed[pwl] inconsistent "rbd status" output (clean = true but dirty_bytes = 61440)
% Done:
0%
Source:
Tags:
backport_processed
Backport:
pacific,quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Description
This popped up in a quincy integration branch run, but the code in main is exactly the same:
2022-10-12T19:57:10.829 INFO:tasks.rbd_pwl_cache_recovery:start rbd bench
2022-10-12T19:57:10.829 DEBUG:teuthology.orchestra.run.smithi143:> rbd bench --io-type write --io-pattern full-seq --io-size 4K --io-total 100G testimage
2022-10-12T19:57:11.482 INFO:teuthology.orchestra.run.smithi143.stdout:bench type write io_size 4096 io_threads 16 bytes 107374182400 pattern full sequential
2022-10-12T19:57:12.873 INFO:teuthology.orchestra.run.smithi143.stdout: SEC OPS OPS/SEC BYTES/SEC
2022-10-12T19:57:12.874 INFO:teuthology.orchestra.run.smithi143.stdout: 1 48 84.2127 337 KiB/s
2022-10-12T19:57:13.616 INFO:teuthology.orchestra.run.smithi143.stdout: 2 80 63.8739 255 KiB/s
2022-10-12T19:57:14.770 INFO:teuthology.orchestra.run.smithi143.stdout: 3 128 54.1775 217 KiB/s
2022-10-12T19:57:15.666 INFO:teuthology.orchestra.run.smithi143.stdout: 4 176 54.0403 216 KiB/s
2022-10-12T19:57:16.601 INFO:teuthology.orchestra.run.smithi143.stdout: 5 224 53.4773 214 KiB/s
2022-10-12T19:57:17.587 INFO:teuthology.orchestra.run.smithi143.stdout: 6 256 44.1251 176 KiB/s
2022-10-12T19:57:18.537 INFO:teuthology.orchestra.run.smithi143.stdout: 7 304 45.5297 182 KiB/s
2022-10-12T19:57:19.518 INFO:teuthology.orchestra.run.smithi143.stdout: 8 352 47.1989 189 KiB/s
2022-10-12T19:57:20.570 INFO:teuthology.orchestra.run.smithi143.stdout: 9 400 45.6782 183 KiB/s
2022-10-12T19:57:20.872 INFO:tasks.rbd_pwl_cache_recovery:dump cache state when rbd bench running.
2022-10-12T19:57:20.873 DEBUG:teuthology.orchestra.run.smithi143:> rbd status testimage --format=json
2022-10-12T19:57:20.930 INFO:teuthology.orchestra.run.smithi143.stdout:{"watchers":[{"address":"172.21.15.143:0/1691094829","client":4303,"cookie":139634285850816}],"persistent_cache":{"host":"smithi143","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10b9d59933d9.pool","size":8589934592,"mode":"rwl","stats_timestamp":"Wed Oct 12 19:57:16 2022","present":true,"empty":false,"clean":false,"allocated_bytes":20955136,"cached_bytes":20955136,"dirty_bytes":917504,"free_bytes":5991999078,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}}
2022-10-12T19:57:20.932 INFO:tasks.rbd_pwl_cache_recovery:sleep...
2022-10-12T19:57:21.556 INFO:teuthology.orchestra.run.smithi143.stdout: 10 448 45.1989 181 KiB/s
2022-10-12T19:57:22.553 INFO:teuthology.orchestra.run.smithi143.stdout: 11 480 45.1079 180 KiB/s
2022-10-12T19:57:23.702 INFO:teuthology.orchestra.run.smithi143.stdout: 12 544 46.4498 186 KiB/s
2022-10-12T19:57:24.724 INFO:teuthology.orchestra.run.smithi143.stdout: 13 592 46.093 184 KiB/s
2022-10-12T19:57:25.752 INFO:teuthology.orchestra.run.smithi143.stdout: 14 640 46.3154 185 KiB/s
2022-10-12T19:57:26.587 INFO:teuthology.orchestra.run.smithi143.stdout: 15 672 44.5251 178 KiB/s
2022-10-12T19:57:27.760 INFO:teuthology.orchestra.run.smithi143.stdout: 16 720 46.0842 184 KiB/s
2022-10-12T19:57:28.576 INFO:teuthology.orchestra.run.smithi143.stdout: 17 768 45.9594 184 KiB/s
2022-10-12T19:57:29.487 INFO:teuthology.orchestra.run.smithi143.stdout: 18 816 47.0206 188 KiB/s
2022-10-12T19:57:30.548 INFO:teuthology.orchestra.run.smithi143.stdout: 19 864 46.6971 187 KiB/s
2022-10-12T19:57:31.493 INFO:teuthology.orchestra.run.smithi143.stdout: 20 912 48.931 196 KiB/s
2022-10-12T19:57:32.826 INFO:teuthology.orchestra.run.smithi143.stdout: 21 960 47.3853 190 KiB/s
2022-10-12T19:57:33.709 INFO:teuthology.orchestra.run.smithi143.stdout: 22 1008 46.7575 187 KiB/s
2022-10-12T19:57:34.942 INFO:teuthology.orchestra.run.smithi143.stdout: 23 1056 43.9975 176 KiB/s
2022-10-12T19:57:35.712 INFO:teuthology.orchestra.run.smithi143.stdout: 24 1088 43.3784 174 KiB/s
2022-10-12T19:57:36.635 INFO:teuthology.orchestra.run.smithi143.stdout: 25 1136 43.5894 174 KiB/s
2022-10-12T19:57:37.718 INFO:teuthology.orchestra.run.smithi143.stdout: 26 1184 45.7809 183 KiB/s
2022-10-12T19:57:38.754 INFO:teuthology.orchestra.run.smithi143.stdout: 27 1232 44.4016 178 KiB/s
2022-10-12T19:57:39.733 INFO:teuthology.orchestra.run.smithi143.stdout: 28 1280 46.7556 187 KiB/s
2022-10-12T19:57:40.708 INFO:teuthology.orchestra.run.smithi143.stdout: 29 1312 44.8371 179 KiB/s
2022-10-12T19:57:41.729 INFO:teuthology.orchestra.run.smithi143.stdout: 30 1360 43.9486 176 KiB/s
2022-10-12T19:57:42.815 INFO:teuthology.orchestra.run.smithi143.stdout: 31 1408 43.9486 176 KiB/s
2022-10-12T19:57:43.634 INFO:teuthology.orchestra.run.smithi143.stdout: 32 1440 42.6241 170 KiB/s
2022-10-12T19:57:44.712 INFO:teuthology.orchestra.run.smithi143.stdout: 33 1488 41.7766 167 KiB/s
2022-10-12T19:57:45.948 INFO:teuthology.orchestra.run.smithi143.stdout: 34 1536 42.7492 171 KiB/s
2022-10-12T19:57:46.734 INFO:teuthology.orchestra.run.smithi143.stdout: 35 1568 41.5512 166 KiB/s
2022-10-12T19:57:47.637 INFO:teuthology.orchestra.run.smithi143.stdout: 36 1616 43.1368 173 KiB/s
2022-10-12T19:57:48.798 INFO:teuthology.orchestra.run.smithi143.stdout: 37 1664 43.3784 174 KiB/s
2022-10-12T19:57:49.656 INFO:teuthology.orchestra.run.smithi143.stdout: 38 1696 42.0723 168 KiB/s
2022-10-12T19:57:50.611 INFO:teuthology.orchestra.run.smithi143.stdout: 39 1744 44.6077 178 KiB/s
2022-10-12T19:57:51.756 INFO:teuthology.orchestra.run.smithi143.stdout: 40 1792 44.6049 178 KiB/s
2022-10-12T19:57:52.715 INFO:teuthology.orchestra.run.smithi143.stdout: 41 1840 44.1043 176 KiB/s
2022-10-12T19:57:53.753 INFO:teuthology.orchestra.run.smithi143.stdout: 42 1888 45.2081 181 KiB/s
2022-10-12T19:57:54.739 INFO:teuthology.orchestra.run.smithi143.stdout: 43 1920 44.0696 176 KiB/s
2022-10-12T19:57:55.766 INFO:teuthology.orchestra.run.smithi143.stdout: 44 1968 43.4541 174 KiB/s
2022-10-12T19:57:56.877 INFO:teuthology.orchestra.run.smithi143.stdout: 45 2016 43.7341 175 KiB/s
2022-10-12T19:57:57.764 INFO:teuthology.orchestra.run.smithi143.stdout: 46 2048 41.2055 165 KiB/s
2022-10-12T19:57:58.497 INFO:teuthology.orchestra.run.smithi143.stdout: 47 2080 40.4732 162 KiB/s
2022-10-12T19:57:59.484 INFO:teuthology.orchestra.run.smithi143.stdout: 48 2128 43.8368 175 KiB/s
2022-10-12T19:58:00.679 INFO:teuthology.orchestra.run.smithi143.stdout: 49 2176 42.3378 169 KiB/s
2022-10-12T19:58:01.596 INFO:teuthology.orchestra.run.smithi143.stdout: 50 2224 44.0876 176 KiB/s
2022-10-12T19:58:02.571 INFO:teuthology.orchestra.run.smithi143.stdout: 51 2272 46.5999 186 KiB/s
2022-10-12T19:58:03.609 INFO:teuthology.orchestra.run.smithi143.stdout: 52 2320 46.9404 188 KiB/s
2022-10-12T19:58:04.514 INFO:teuthology.orchestra.run.smithi143.stdout: 53 2352 44.5251 178 KiB/s
2022-10-12T19:58:05.684 INFO:teuthology.orchestra.run.smithi143.stdout: 54 2400 44.7564 179 KiB/s
2022-10-12T19:58:06.686 INFO:teuthology.orchestra.run.smithi143.stdout: 55 2448 44.0004 176 KiB/s
2022-10-12T19:58:07.612 INFO:teuthology.orchestra.run.smithi143.stdout: 56 2496 44.4368 178 KiB/s
2022-10-12T19:58:08.703 INFO:teuthology.orchestra.run.smithi143.stdout: 57 2544 43.9745 176 KiB/s
2022-10-12T19:58:10.028 INFO:teuthology.orchestra.run.smithi143.stdout: 58 2592 43.5267 174 KiB/s
2022-10-12T19:58:10.688 INFO:teuthology.orchestra.run.smithi143.stdout: 59 2624 44.7564 179 KiB/s
2022-10-12T19:58:11.499 INFO:teuthology.orchestra.run.smithi143.stdout: 60 2672 46.5418 186 KiB/s
2022-10-12T19:58:12.669 INFO:teuthology.orchestra.run.smithi143.stdout: 61 2720 44.2874 177 KiB/s
2022-10-12T19:58:13.755 INFO:teuthology.orchestra.run.smithi143.stdout: 62 2768 44.34 177 KiB/s
2022-10-12T19:58:14.536 INFO:teuthology.orchestra.run.smithi143.stdout: 63 2800 46.1516 185 KiB/s
2022-10-12T19:58:15.661 INFO:teuthology.orchestra.run.smithi143.stdout: 64 2848 45.0535 180 KiB/s
2022-10-12T19:58:16.563 INFO:teuthology.orchestra.run.smithi143.stdout: 65 2896 44.235 177 KiB/s
2022-10-12T19:58:17.593 INFO:teuthology.orchestra.run.smithi143.stdout: 66 2944 45.4927 182 KiB/s
2022-10-12T19:58:18.510 INFO:teuthology.orchestra.run.smithi143.stdout: 67 2976 43.7538 175 KiB/s
2022-10-12T19:58:19.808 INFO:teuthology.orchestra.run.smithi143.stdout: 68 3040 45.5161 182 KiB/s
2022-10-12T19:58:20.658 INFO:teuthology.orchestra.run.smithi143.stdout: 69 3072 44.8191 179 KiB/s
2022-10-12T19:58:20.933 INFO:tasks.rbd_pwl_cache_recovery:rbd bench crash.
2022-10-12T19:58:20.934 DEBUG:teuthology.orchestra.run.smithi143:> killall -9 rbd
2022-10-12T19:58:20.962 INFO:tasks.rbd_pwl_cache_recovery:wait for watch timeout.
2022-10-12T19:59:00.964 INFO:tasks.rbd_pwl_cache_recovery:check cache state after crash.
2022-10-12T19:59:00.966 DEBUG:teuthology.orchestra.run.smithi143:> rbd status testimage --format=json
2022-10-12T19:59:01.025 INFO:teuthology.orchestra.run.smithi143.stdout:{"watchers":[],"persistent_cache":{"host":"smithi143","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10b9d59933d9.pool","size":8589934592,"mode":"rwl","stats_timestamp":"Wed Oct 12 19:58:20 2022","present":true,"empty":false,"clean":true,"allocated_bytes":32731136,"cached_bytes":32731136,"dirty_bytes":61440,"free_bytes":5980223078,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}}
2022-10-12T19:59:01.027 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/contextutil.py", line 31, in nested
vars.append(enter())
File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/home/teuthworker/src/github.com_ceph_ceph-c_18f1152e5abc61579d870e05168b42e885f4a242/qa/tasks/rbd_pwl_cache_recovery.py", line 69, in thrashes_rbd_bench_on_persistent_cache
assert rbd_status['persistent_cache']['clean'] == False
AssertionError
Updated by CONGMIN YIN over 1 year ago
- Status changed from New to Fix Under Review
- Pull request ID set to 48542
Updated by Ilya Dryomov over 1 year ago
- Assignee set to CONGMIN YIN
- Backport set to pacific,quincy
Updated by Ilya Dryomov over 1 year ago
- Status changed from Fix Under Review to Pending Backport
Updated by Backport Bot over 1 year ago
- Copied to Backport #58076: pacific: [pwl] inconsistent "rbd status" output (clean = true but dirty_bytes = 61440) added
Updated by Backport Bot over 1 year ago
- Copied to Backport #58077: quincy: [pwl] inconsistent "rbd status" output (clean = true but dirty_bytes = 61440) added
Updated by Backport Bot about 1 year ago
- Status changed from Pending Backport to Resolved
While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".
Actions