Actions
Bug #38703
closedlazy omap stats aren't incorportaed into pg_autoscaler size value
Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
on lab cluster,
root@mira070:~# ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO PG_NUM NEW PG_NUM AUTOSCALE device_health_metrics 0 3.0 431.3T 0.0000 1 4 warn default.rgw.buckets.non-ec 0 3.0 431.3T 0.0000 8 warn default.rgw.meta 1336 3.0 431.3T 0.0000 8 warn default.rgw.buckets.index 0 3.0 431.3T 0.0000 8 warn default.rgw.control 0 3.0 431.3T 0.0000 8 warn default.rgw.buckets.data 743.5G 3.0 431.3T 0.0050 32 on .rgw.root 1113 3.0 431.3T 0.0000 8 warn djf_tmp 8240 3.0 431.3T 0.0000 4096 4 warn libvirt-pool 2048M 3.0 431.3T 0.0000 4096 4 warn data 78796G 3.0 431.3T 0.5351 4096 warn default.rgw.log 0 3.0 431.3T 0.0000 8 warn metadata 863.3M 4.0 431.3T 0.0000 64 4 warn
i.e., 860 MB.
but the pg stats show lots more data in omap:
root@mira070:~# ceph pg ls-by-pool metadata PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP 1.0 37441 0 0 0 8388608 1004561096 2266056 3048 active+clean 7h 1084714'22593561 1084714:28756662 [149,9,105,96]p149 [149,9,105,96]p149 2019-03-12 09:11:08.387089 2019-03-08 23:41:12.690675 1.1 37570 0 0 0 16777216 985896696 2219788 3040 active+clean 31h 1084714'22792937 1084714:29404032 [91,71,69,37]p91 [91,71,69,37]p91 2019-03-11 09:32:46.943843 2019-03-07 18:57:40.432687 1.2 37636 0 0 0 4194304 980205936 2209344 3059 active+clean 32h 1084714'22913289 1084714:29626803 [129,83,49,104]p129 [129,83,49,104]p129 2019-03-11 08:07:24.635665 2019-03-06 06:37:33.634607 1.3 37363 0 0 0 8388608 0 0 3069 active+clean 22h 1084714'23108051 1084714:29464208 [125,3,54,65]p125 [125,3,54,65]p125 2019-03-11 18:11:20.705113 2019-03-05 09:53:49.106319 1.4 37178 0 0 0 16777216 1130328292 2531384 3033 active+clean 5h 1084714'23028327 1084714:30106978 [54,47,33,147]p54 [54,47,33,147]p54 2019-03-12 11:12:25.545303 2019-03-10 01:15:58.614116 1.5 37527 0 0 0 16777216 994207248 2252300 3004 active+clean 23h 1084714'22897470 1084714:29411024 [69,134,135,120]p69 [69,134,135,120]p69 2019-03-11 16:52:45.729981 2019-03-10 11:57:01.610544 1.6 37676 0 0 0 8388608 1037062984 2334220 3084 active+clean 11h 1084714'22915672 1084714:29427802 [26,82,146,114]p26 [26,82,146,114]p26 2019-03-12 05:28:54.416447 2019-03-09 11:48:17.664437 1.7 37701 0 0 0 12582912 1056816688 2371680 3054 active+clean 22h 1084714'22720924 1084714:30480200 [2,87,77,114]p2 [2,87,77,114]p2 2019-03-11 18:10:55.977174 2019-03-08 07:28:22.457829 1.8 37524 0 0 0 16777216 1088492424 2465496 3096 active+clean 39m 1084684'22656541 1084714:29144768 [17,41,10,47]p17 [17,41,10,47]p17 2019-03-12 16:02:05.362587 2019-03-08 16:46:00.830506 1.9 37325 0 0 0 8388608 929899340 2106088 3084 active+clean 9h 1084714'22869496 1084714:29631741 [138,103,121,133]p138 [138,103,121,133]p138 2019-03-12 07:36:34.117740 2019-03-10 20:12:44.318531 1.a 37726 0 0 0 16777216 1024256920 2350900 3008 active+clean 17h 1084714'23162183 1084714:29739146 [25,60,96,13]p25 [25,60,96,13]p25 2019-03-11 22:46:48.964533 2019-03-11 22:46:48.964533 1.b 37652 0 0 0 12582912 993524772 2240112 3049 active+clean 12h 1084702'23414081 1084713:29464174 [83,75,142,106]p83 [83,75,142,106]p83 2019-03-12 04:19:49.755534 2019-03-12 04:19:49.755534 1.c 37609 0 0 0 4194304 1068101576 2406348 3054 active+clean 29h 1084712'22827093 1084714:29329690 [65,147,132,121]p65 [65,147,132,121]p65 2019-03-11 10:54:09.504936 2019-03-08 12:38:26.731382 1.d 37917 0 0 0 16777216 1033104672 2326148 3084 active+clean 27h 1084714'22948846 1084714:28746352 [10,33,54,123]p10 [10,33,54,123]p10 2019-03-11 13:08:51.142121 2019-03-07 20:08:56.234449 1.e 37641 0 0 0 20971520 1062104348 2385888 3042 active+clean 15h 1084684'22761496 1084713:30317254 [99,87,5,51]p99 [99,87,5,51]p99 2019-03-12 00:58:47.715232 2019-03-07 03:53:34.700645 1.f 37627 0 0 0 29360150 1000324596 2247708 3083 active+clean 23h 1084714'22831186 1084714:30333260 [3,104,114,74]p3 [3,104,114,74]p3 2019-03-11 16:43:16.030285 2019-03-07 16:06:14.289076 1.10 37752 0 0 0 16777216 1020877468 2301192 3001 active+clean 17h 1084714'23034440 1084714:29803207 [46,79,128,62]p46 [46,79,128,62]p46 2019-03-11 23:39:38.449144 2019-03-11 23:39:38.449144 1.11 37685 0 0 0 12582912 986447832 2214376 3062 active+clean 15h 1084714'22786800 1084714:29331295 [88,112,14,91]p88 [88,112,14,91]p88 2019-03-12 00:46:28.825399 2019-03-10 22:19:08.845563 1.12 37900 0 0 0 25165824 1050285100 2364628 3097 active+clean 11h 1084702'22465490 1084713:29287250 [146,141,2,133]p146 [146,141,2,133]p146 2019-03-12 05:30:57.312170 2019-03-11 05:24:51.369990 1.13 37977 0 0 0 12582946 958213092 2170928 3076 active+clean 26h 1084714'23036476 1084714:30127653 [29,117,82,131]p29 [29,117,82,131]p29 2019-03-11 14:03:39.620316 2019-03-06 14:44:13.154698 1.14 37350 0 0 0 16777752 957445504 2158396 3089 active+clean 20h 1084714'23488663 1084714:29810957 [31,25,55,130]p31 [31,25,55,130]p31 2019-03-11 20:13:07.367415 2019-03-06 16:53:40.291771 1.15 37959 0 0 0 37748826 1172884792 2634804 3088 active+clean 31h 1084714'23673119 1084714:32069068 [75,128,126,56]p75 [75,128,126,56]p75 2019-03-11 09:31:53.568899 2019-03-10 02:11:53.020578 1.16 37727 0 0 0 8388608 1218193808 2727328 3082 active+clean 23h 1084714'22782413 1084714:29392305 [117,138,54,111]p117 [117,138,54,111]p117 2019-03-11 16:51:26.978984 2019-03-10 08:18:25.094591 1.17 37320 0 0 0 20971520 1055333080 2375108 3019 active+clean 69m 1084714'22794164 1084714:29228741 [1,76,33,36]p1 [1,76,33,36]p1 2019-03-12 15:31:55.832244 2019-03-12 15:31:55.832244 1.18 37489 0 0 0 12582912 1072608144 2418196 3071 active+clean 13m 1084714'22712346 1084714:29334553 [3,72,104,122]p3 [3,72,104,122]p3 2019-03-12 16:27:50.485740 2019-03-11 11:33:09.985266 1.19 37836 0 0 0 8388608 1080224556 2426688 3094 active+clean 22h 1084714'22689481 1084714:30118049 [34,17,25,81]p34 [34,17,25,81]p34 2019-03-11 18:23:05.486487 2019-03-09 02:52:22.238989 1.1a 37489 0 0 0 4194304 1087617244 2446916 3021 active+clean 18h 1084714'22953855 1084714:30105021 [111,112,36,125]p111 [111,112,36,125]p111 2019-03-11 22:39:42.235912 2019-03-09 03:55:25.638908 1.1b 37672 0 0 0 25165824 1060972872 2384244 3075 active+clean 5h 1084714'23293178 1084714:31018277 [42,94,84,90]p42 [42,94,84,90]p42 2019-03-12 10:50:23.851682 2019-03-08 15:34:07.410255 1.1c 37942 0 0 0 16777216 970694172 2187208 3082 active+clean 11h 1084702'22770376 1084713:29493410 [105,82,99,41]p105 [105,82,99,41]p105 2019-03-12 04:58:26.687344 2019-03-08 06:01:52.062667 1.1d 37698 0 0 0 4194304 1040358068 2395752 3089 active+clean 17h 1084714'22920879 1084714:29831066 [147,7,17,137]p147 [147,7,17,137]p147 2019-03-11 23:12:18.155537 2019-03-11 23:12:18.155537 1.1e 37540 0 0 0 12582912 1048299500 2354120 3049 active+clean 24h 1084714'22899603 1084714:28874114 [73,4,55,111]p73 [73,4,55,111]p73 2019-03-11 16:12:11.771140 2019-03-09 01:57:21.354640 1.1f 37681 0 0 0 12583002 1071175084 2436884 3034 active+clean 13h 1084714'34659708 1084714:39640459 [47,76,26,71]p47 [47,76,26,71]p47 2019-03-12 02:56:47.414893 2019-03-12 02:56:47.414893 1.20 37482 0 0 0 8388608 1007294092 2278844 3086 active+clean 12h 1084714'22666701 1084714:27833486 [27,47,144,112]p27 [27,47,144,112]p27 2019-03-12 04:21:24.662595 2019-03-07 01:45:37.893343 1.21 37396 0 0 0 12582912 903715736 2038668 3016 active+clean 18h 1084714'23174844 1084714:28556515 [41,13,9,31]p41 [41,13,9,31]p41 2019-03-11 22:40:02.812292 2019-03-06 12:22:24.401887 1.22 37519 0 0 0 12582912 1147825068 2584412 3023 active+clean 20h 1084714'22868407 1084714:29169783 [59,121,136,98]p59 [59,121,136,98]p59 2019-03-11 20:08:15.431712 2019-03-07 12:23:35.981059 1.23 37932 0 0 0 33554432 1039746440 2337804 3063 active+clean 15h 1084714'22761272 1084714:29840531 [96,125,13,141]p96 [96,125,13,141]p96 2019-03-12 01:36:49.607077 2019-03-08 14:40:10.041695 1.24 37399 0 0 0 4194304 953720100 2165256 3083 active+clean 15h 1084714'22750102 1084714:29131213 [47,74,57,31]p47 [47,74,57,31]p47 2019-03-12 01:05:35.672878 2019-03-07 01:38:04.681243 1.25 37453 0 0 0 12582912 1101494164 2497040 3017 active+clean 28h 1084714'22828721 1084714:30230519 [145,70,50,126]p145 [145,70,50,126]p145 2019-03-11 12:36:39.096380 2019-03-08 19:32:21.200444 1.26 37759 0 0 0 8388608 0 0 3049 active+clean 21h 1084714'27852272 1084714:33703446 [132,26,56,57]p132 [132,26,56,57]p132 2019-03-11 19:01:47.819748 2019-03-05 12:31:14.189398 1.27 37756 0 0 0 12583636 983941928 2212824 3087 active+clean 21h 1084714'22602755 1084714:30048376 [105,114,52,95]p105 [105,114,52,95]p105 2019-03-11 19:19:06.724417 2019-03-11 19:19:06.724417 1.28 37863 0 0 0 16777216 1040398984 2333324 3061 active+clean 14h 1084714'22738725 1084714:28887703 [128,44,124,60]p128 [128,44,124,60]p128 2019-03-12 01:50:35.616499 2019-03-12 01:50:35.616499 1.29 37594 0 0 0 25165824 1044094300 2354788 3042 active+clean 28h 1084714'23007656 1084714:28744937 [86,32,68,102]p86 [86,32,68,102]p86 2019-03-11 11:44:53.115346 2019-03-11 11:44:53.115346 1.2a 37454 0 0 0 8388608 1019231464 2347240 3070 active+clean 13h 1084714'22829286 1084714:29727280 [19,136,73,138]p19 [19,136,73,138]p19 2019-03-12 03:26:36.109351 2019-03-10 18:15:43.402194 1.2b 37678 0 0 0 25165824 998802724 2250000 3092 active+clean 13h 1084713'23022347 1084714:30309661 [96,46,78,146]p96 [96,46,78,146]p96 2019-03-12 03:39:46.705843 2019-03-06 17:10:50.083391 1.2c 37336 0 0 0 16777216 982482924 2216272 3040 active+clean 29h 1084714'22922958 1084714:30014900 [124,46,2,28]p124 [124,46,2,28]p124 2019-03-11 11:03:00.612032 2019-03-11 11:03:00.612032 1.2d 37566 0 0 0 12582982 971046340 2192748 3058 active+clean 27h 1084712'22673424 1084713:29072363 [74,32,38,67]p74 [74,32,38,67]p74 2019-03-11 13:27:46.167881 2019-03-10 12:05:06.730248 1.2e 37780 0 0 0 20971520 1000687408 2250964 3038 active+clean 10h 1084714'23139951 1084714:28553192 [123,100,20,91]p123 [123,100,20,91]p123 2019-03-12 06:04:11.170393 2019-03-11 04:22:46.409811 1.2f 37457 0 0 0 16777216 965642252 2191872 3090 active+clean 24h 1084712'22764879 1084713:29297566 [18,143,120,133]p18 [18,143,120,133]p18 2019-03-11 15:55:27.961537 2019-03-11 15:55:27.961537 1.30 37528 0 0 0 8388608 1016447000 2293400 3002 active+clean 24h 1084714'22792001 1084714:28839373 [35,47,78,17]p35 [35,47,78,17]p35 2019-03-11 16:03:05.879032 2019-03-11 16:03:05.879032 1.31 37355 0 0 0 16777216 1050670180 2372964 3002 active+clean 14m 1084714'22805199 1084714:29809613 [127,30,110,49]p127 [127,30,110,49]p127 2019-03-12 16:26:38.431907 2019-03-09 01:57:49.212009 1.32 37812 0 0 0 4194304 1049948284 2369772 3001 active+clean 13h 1084714'22844477 1084714:29391693 [41,123,15,125]p41 [41,123,15,125]p41 2019-03-12 02:53:12.021771 2019-03-12 02:53:12.021771 1.33 37797 0 0 0 4194304 984721348 2221216 3096 active+clean 26h 1084714'22774445 1084714:29153034 [69,86,30,138]p69 [69,86,30,138]p69 2019-03-11 14:16:17.521605 2019-03-07 18:52:19.437961 1.34 37586 0 0 0 10404158 0 0 3075 active+clean 19h 1084714'23009518 1084714:30283894 [74,130,0,54]p74 [74,130,0,54]p74 2019-03-11 21:04:10.873707 2019-03-05 15:42:45.011809 1.35 37810 0 0 0 0 1169703248 2615884 3074 active+clean 17h 1084714'22594608 1084714:28429289 [62,18,26,10]p62 [62,18,26,10]p62 2019-03-11 22:41:19.060206 2019-03-08 08:24:15.242867 1.36 37540 0 0 0 16777216 961795864 2168668 3027 active+clean 17h 1084714'22952227 1084714:30425595 [83,12,72,57]p83 [83,12,72,57]p83 2019-03-11 22:51:14.992833 2019-03-07 00:21:41.500736 1.37 37790 0 0 0 17306525 1062089800 2380568 3071 active+clean 4h 1084714'22874788 1084714:29226272 [99,50,149,67]p99 [99,50,149,67]p99 2019-03-12 11:41:38.902199 2019-03-11 10:59:43.906664 1.38 37763 0 0 0 20971520 983899612 2220456 3047 active+clean 89m 1084714'22723442 1084714:28894821 [125,102,136,82]p125 [125,102,136,82]p125 2019-03-12 15:12:05.666325 2019-03-11 03:26:37.105435 1.39 37237 0 0 0 20971520 1035812612 2328928 3090 active+clean 22h 1084714'22883822 1084714:29664564 [46,36,34,49]p46 [46,36,34,49]p46 2019-03-11 17:42:34.811056 2019-03-11 17:42:34.811056 1.3a 37617 0 0 0 4194304 1020080168 2291520 3020 active+clean 6h 1084702'22781891 1084714:29801197 [37,29,41,145]p37 [37,29,41,145]p37 2019-03-12 10:14:24.296445 2019-03-07 19:08:41.616166 1.3b 37507 0 0 0 12582912 1020676628 2296028 3025 active+clean 4h 1084714'22787832 1084714:30417133 [145,33,113,135]p145 [145,33,113,135]p145 2019-03-12 12:24:12.553458 2019-03-07 08:56:07.808949 1.3c 37783 0 0 0 10422738 1028500148 2326708 3021 active+clean 25h 1084702'22769694 1084713:29727735 [0,138,48,31]p0 [0,138,48,31]p0 2019-03-11 15:02:08.471586 2019-03-07 02:55:45.471143 1.3d 37571 0 0 0 12582912 1034089808 2323892 3088 active+clean 23h 1084714'22684357 1084714:28657194 [125,128,79,117]p125 [125,128,79,117]p125 2019-03-11 17:27:03.390717 2019-03-11 17:27:03.390717 1.3e 37610 0 0 0 16777216 1114097152 2497280 3082 active+clean 7h 1084714'22974468 1084714:29273496 [52,64,122,85]p52 [52,64,122,85]p52 2019-03-12 09:14:16.532618 2019-03-09 18:07:35.077896 1.3f 37394 0 0 0 12583456 1042435200 2343380 3024 active+clean 19h 1084714'22968009 1084714:28697534 [47,0,140,104]p47 [47,0,140,104]p47 2019-03-11 20:44:14.297409 2019-03-07 19:31:11.102469 * NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilisation. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details.
Actions