Project

General

Profile

Bug #57872

[pwl] inconsistent "rbd status" output (clean = true but dirty_bytes = 61440)

Added by Ilya Dryomov 4 months ago. Updated 2 months ago.

Status:
Pending Backport
Priority:
Normal
Assignee:
Target version:
-
% Done:

0%

Source:
Tags:
backport_processed
Backport:
pacific,quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

This popped up in a quincy integration branch run, but the code in main is exactly the same:

2022-10-12T19:57:10.829 INFO:tasks.rbd_pwl_cache_recovery:start rbd bench
2022-10-12T19:57:10.829 DEBUG:teuthology.orchestra.run.smithi143:> rbd bench --io-type write --io-pattern full-seq --io-size 4K --io-total 100G testimage
2022-10-12T19:57:11.482 INFO:teuthology.orchestra.run.smithi143.stdout:bench  type write io_size 4096 io_threads 16 bytes 107374182400 pattern full sequential
2022-10-12T19:57:12.873 INFO:teuthology.orchestra.run.smithi143.stdout:  SEC       OPS   OPS/SEC   BYTES/SEC
2022-10-12T19:57:12.874 INFO:teuthology.orchestra.run.smithi143.stdout:    1        48   84.2127   337 KiB/s
2022-10-12T19:57:13.616 INFO:teuthology.orchestra.run.smithi143.stdout:    2        80   63.8739   255 KiB/s
2022-10-12T19:57:14.770 INFO:teuthology.orchestra.run.smithi143.stdout:    3       128   54.1775   217 KiB/s
2022-10-12T19:57:15.666 INFO:teuthology.orchestra.run.smithi143.stdout:    4       176   54.0403   216 KiB/s
2022-10-12T19:57:16.601 INFO:teuthology.orchestra.run.smithi143.stdout:    5       224   53.4773   214 KiB/s
2022-10-12T19:57:17.587 INFO:teuthology.orchestra.run.smithi143.stdout:    6       256   44.1251   176 KiB/s
2022-10-12T19:57:18.537 INFO:teuthology.orchestra.run.smithi143.stdout:    7       304   45.5297   182 KiB/s
2022-10-12T19:57:19.518 INFO:teuthology.orchestra.run.smithi143.stdout:    8       352   47.1989   189 KiB/s
2022-10-12T19:57:20.570 INFO:teuthology.orchestra.run.smithi143.stdout:    9       400   45.6782   183 KiB/s
2022-10-12T19:57:20.872 INFO:tasks.rbd_pwl_cache_recovery:dump cache state when rbd bench running.
2022-10-12T19:57:20.873 DEBUG:teuthology.orchestra.run.smithi143:> rbd status testimage --format=json
2022-10-12T19:57:20.930 INFO:teuthology.orchestra.run.smithi143.stdout:{"watchers":[{"address":"172.21.15.143:0/1691094829","client":4303,"cookie":139634285850816}],"persistent_cache":{"host":"smithi143","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10b9d59933d9.pool","size":8589934592,"mode":"rwl","stats_timestamp":"Wed Oct 12 19:57:16 2022","present":true,"empty":false,"clean":false,"allocated_bytes":20955136,"cached_bytes":20955136,"dirty_bytes":917504,"free_bytes":5991999078,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}}
2022-10-12T19:57:20.932 INFO:tasks.rbd_pwl_cache_recovery:sleep...
2022-10-12T19:57:21.556 INFO:teuthology.orchestra.run.smithi143.stdout:   10       448   45.1989   181 KiB/s
2022-10-12T19:57:22.553 INFO:teuthology.orchestra.run.smithi143.stdout:   11       480   45.1079   180 KiB/s
2022-10-12T19:57:23.702 INFO:teuthology.orchestra.run.smithi143.stdout:   12       544   46.4498   186 KiB/s
2022-10-12T19:57:24.724 INFO:teuthology.orchestra.run.smithi143.stdout:   13       592    46.093   184 KiB/s
2022-10-12T19:57:25.752 INFO:teuthology.orchestra.run.smithi143.stdout:   14       640   46.3154   185 KiB/s
2022-10-12T19:57:26.587 INFO:teuthology.orchestra.run.smithi143.stdout:   15       672   44.5251   178 KiB/s
2022-10-12T19:57:27.760 INFO:teuthology.orchestra.run.smithi143.stdout:   16       720   46.0842   184 KiB/s
2022-10-12T19:57:28.576 INFO:teuthology.orchestra.run.smithi143.stdout:   17       768   45.9594   184 KiB/s
2022-10-12T19:57:29.487 INFO:teuthology.orchestra.run.smithi143.stdout:   18       816   47.0206   188 KiB/s
2022-10-12T19:57:30.548 INFO:teuthology.orchestra.run.smithi143.stdout:   19       864   46.6971   187 KiB/s
2022-10-12T19:57:31.493 INFO:teuthology.orchestra.run.smithi143.stdout:   20       912    48.931   196 KiB/s
2022-10-12T19:57:32.826 INFO:teuthology.orchestra.run.smithi143.stdout:   21       960   47.3853   190 KiB/s
2022-10-12T19:57:33.709 INFO:teuthology.orchestra.run.smithi143.stdout:   22      1008   46.7575   187 KiB/s
2022-10-12T19:57:34.942 INFO:teuthology.orchestra.run.smithi143.stdout:   23      1056   43.9975   176 KiB/s
2022-10-12T19:57:35.712 INFO:teuthology.orchestra.run.smithi143.stdout:   24      1088   43.3784   174 KiB/s
2022-10-12T19:57:36.635 INFO:teuthology.orchestra.run.smithi143.stdout:   25      1136   43.5894   174 KiB/s
2022-10-12T19:57:37.718 INFO:teuthology.orchestra.run.smithi143.stdout:   26      1184   45.7809   183 KiB/s
2022-10-12T19:57:38.754 INFO:teuthology.orchestra.run.smithi143.stdout:   27      1232   44.4016   178 KiB/s
2022-10-12T19:57:39.733 INFO:teuthology.orchestra.run.smithi143.stdout:   28      1280   46.7556   187 KiB/s
2022-10-12T19:57:40.708 INFO:teuthology.orchestra.run.smithi143.stdout:   29      1312   44.8371   179 KiB/s
2022-10-12T19:57:41.729 INFO:teuthology.orchestra.run.smithi143.stdout:   30      1360   43.9486   176 KiB/s
2022-10-12T19:57:42.815 INFO:teuthology.orchestra.run.smithi143.stdout:   31      1408   43.9486   176 KiB/s
2022-10-12T19:57:43.634 INFO:teuthology.orchestra.run.smithi143.stdout:   32      1440   42.6241   170 KiB/s
2022-10-12T19:57:44.712 INFO:teuthology.orchestra.run.smithi143.stdout:   33      1488   41.7766   167 KiB/s
2022-10-12T19:57:45.948 INFO:teuthology.orchestra.run.smithi143.stdout:   34      1536   42.7492   171 KiB/s
2022-10-12T19:57:46.734 INFO:teuthology.orchestra.run.smithi143.stdout:   35      1568   41.5512   166 KiB/s
2022-10-12T19:57:47.637 INFO:teuthology.orchestra.run.smithi143.stdout:   36      1616   43.1368   173 KiB/s
2022-10-12T19:57:48.798 INFO:teuthology.orchestra.run.smithi143.stdout:   37      1664   43.3784   174 KiB/s
2022-10-12T19:57:49.656 INFO:teuthology.orchestra.run.smithi143.stdout:   38      1696   42.0723   168 KiB/s
2022-10-12T19:57:50.611 INFO:teuthology.orchestra.run.smithi143.stdout:   39      1744   44.6077   178 KiB/s
2022-10-12T19:57:51.756 INFO:teuthology.orchestra.run.smithi143.stdout:   40      1792   44.6049   178 KiB/s
2022-10-12T19:57:52.715 INFO:teuthology.orchestra.run.smithi143.stdout:   41      1840   44.1043   176 KiB/s
2022-10-12T19:57:53.753 INFO:teuthology.orchestra.run.smithi143.stdout:   42      1888   45.2081   181 KiB/s
2022-10-12T19:57:54.739 INFO:teuthology.orchestra.run.smithi143.stdout:   43      1920   44.0696   176 KiB/s
2022-10-12T19:57:55.766 INFO:teuthology.orchestra.run.smithi143.stdout:   44      1968   43.4541   174 KiB/s
2022-10-12T19:57:56.877 INFO:teuthology.orchestra.run.smithi143.stdout:   45      2016   43.7341   175 KiB/s
2022-10-12T19:57:57.764 INFO:teuthology.orchestra.run.smithi143.stdout:   46      2048   41.2055   165 KiB/s
2022-10-12T19:57:58.497 INFO:teuthology.orchestra.run.smithi143.stdout:   47      2080   40.4732   162 KiB/s
2022-10-12T19:57:59.484 INFO:teuthology.orchestra.run.smithi143.stdout:   48      2128   43.8368   175 KiB/s
2022-10-12T19:58:00.679 INFO:teuthology.orchestra.run.smithi143.stdout:   49      2176   42.3378   169 KiB/s
2022-10-12T19:58:01.596 INFO:teuthology.orchestra.run.smithi143.stdout:   50      2224   44.0876   176 KiB/s
2022-10-12T19:58:02.571 INFO:teuthology.orchestra.run.smithi143.stdout:   51      2272   46.5999   186 KiB/s
2022-10-12T19:58:03.609 INFO:teuthology.orchestra.run.smithi143.stdout:   52      2320   46.9404   188 KiB/s
2022-10-12T19:58:04.514 INFO:teuthology.orchestra.run.smithi143.stdout:   53      2352   44.5251   178 KiB/s
2022-10-12T19:58:05.684 INFO:teuthology.orchestra.run.smithi143.stdout:   54      2400   44.7564   179 KiB/s
2022-10-12T19:58:06.686 INFO:teuthology.orchestra.run.smithi143.stdout:   55      2448   44.0004   176 KiB/s
2022-10-12T19:58:07.612 INFO:teuthology.orchestra.run.smithi143.stdout:   56      2496   44.4368   178 KiB/s
2022-10-12T19:58:08.703 INFO:teuthology.orchestra.run.smithi143.stdout:   57      2544   43.9745   176 KiB/s
2022-10-12T19:58:10.028 INFO:teuthology.orchestra.run.smithi143.stdout:   58      2592   43.5267   174 KiB/s
2022-10-12T19:58:10.688 INFO:teuthology.orchestra.run.smithi143.stdout:   59      2624   44.7564   179 KiB/s
2022-10-12T19:58:11.499 INFO:teuthology.orchestra.run.smithi143.stdout:   60      2672   46.5418   186 KiB/s
2022-10-12T19:58:12.669 INFO:teuthology.orchestra.run.smithi143.stdout:   61      2720   44.2874   177 KiB/s
2022-10-12T19:58:13.755 INFO:teuthology.orchestra.run.smithi143.stdout:   62      2768     44.34   177 KiB/s
2022-10-12T19:58:14.536 INFO:teuthology.orchestra.run.smithi143.stdout:   63      2800   46.1516   185 KiB/s
2022-10-12T19:58:15.661 INFO:teuthology.orchestra.run.smithi143.stdout:   64      2848   45.0535   180 KiB/s
2022-10-12T19:58:16.563 INFO:teuthology.orchestra.run.smithi143.stdout:   65      2896    44.235   177 KiB/s
2022-10-12T19:58:17.593 INFO:teuthology.orchestra.run.smithi143.stdout:   66      2944   45.4927   182 KiB/s
2022-10-12T19:58:18.510 INFO:teuthology.orchestra.run.smithi143.stdout:   67      2976   43.7538   175 KiB/s
2022-10-12T19:58:19.808 INFO:teuthology.orchestra.run.smithi143.stdout:   68      3040   45.5161   182 KiB/s
2022-10-12T19:58:20.658 INFO:teuthology.orchestra.run.smithi143.stdout:   69      3072   44.8191   179 KiB/s
2022-10-12T19:58:20.933 INFO:tasks.rbd_pwl_cache_recovery:rbd bench crash.
2022-10-12T19:58:20.934 DEBUG:teuthology.orchestra.run.smithi143:> killall -9 rbd
2022-10-12T19:58:20.962 INFO:tasks.rbd_pwl_cache_recovery:wait for watch timeout.
2022-10-12T19:59:00.964 INFO:tasks.rbd_pwl_cache_recovery:check cache state after crash.
2022-10-12T19:59:00.966 DEBUG:teuthology.orchestra.run.smithi143:> rbd status testimage --format=json
2022-10-12T19:59:01.025 INFO:teuthology.orchestra.run.smithi143.stdout:{"watchers":[],"persistent_cache":{"host":"smithi143","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10b9d59933d9.pool","size":8589934592,"mode":"rwl","stats_timestamp":"Wed Oct 12 19:58:20 2022","present":true,"empty":false,"clean":true,"allocated_bytes":32731136,"cached_bytes":32731136,"dirty_bytes":61440,"free_bytes":5980223078,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}}
2022-10-12T19:59:01.027 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/contextutil.py", line 31, in nested
    vars.append(enter())
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_18f1152e5abc61579d870e05168b42e885f4a242/qa/tasks/rbd_pwl_cache_recovery.py", line 69, in thrashes_rbd_bench_on_persistent_cache
    assert rbd_status['persistent_cache']['clean'] == False
AssertionError

http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-12_16:20:34-rbd-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063517/teuthology.log


Related issues

Copied to rbd - Backport #58076: pacific: [pwl] inconsistent "rbd status" output (clean = true but dirty_bytes = 61440) In Progress
Copied to rbd - Backport #58077: quincy: [pwl] inconsistent "rbd status" output (clean = true but dirty_bytes = 61440) In Progress

History

#1 Updated by CONGMIN YIN 3 months ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 48542

#2 Updated by Ilya Dryomov 3 months ago

  • Assignee set to CONGMIN YIN
  • Backport set to pacific,quincy

#3 Updated by Ilya Dryomov 2 months ago

  • Status changed from Fix Under Review to Pending Backport

#4 Updated by Backport Bot 2 months ago

  • Copied to Backport #58076: pacific: [pwl] inconsistent "rbd status" output (clean = true but dirty_bytes = 61440) added

#5 Updated by Backport Bot 2 months ago

  • Copied to Backport #58077: quincy: [pwl] inconsistent "rbd status" output (clean = true but dirty_bytes = 61440) added

#6 Updated by Backport Bot 2 months ago

  • Tags set to backport_processed

Also available in: Atom PDF