Bug #38703: lazy omap stats aren't incorportaed into pg_autoscaler size value - RADOS - Ceph

Actions

Copy link

Bug #38703

closed

lazy omap stats aren't incorportaed into pg_autoscaler size value

Added by Sage Weil about 5 years ago. Updated about 5 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Category:

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(RADOS):

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

on lab cluster,

root@mira070:~# ceph osd pool autoscale-status
 POOL                          SIZE  TARGET SIZE  RATE  RAW CAPACITY   RATIO  TARGET RATIO  PG_NUM  NEW PG_NUM  AUTOSCALE 
 device_health_metrics           0                 3.0        431.3T  0.0000                     1           4  warn      
 default.rgw.buckets.non-ec      0                 3.0        431.3T  0.0000                     8              warn      
 default.rgw.meta             1336                 3.0        431.3T  0.0000                     8              warn      
 default.rgw.buckets.index       0                 3.0        431.3T  0.0000                     8              warn      
 default.rgw.control             0                 3.0        431.3T  0.0000                     8              warn      
 default.rgw.buckets.data    743.5G                3.0        431.3T  0.0050                    32              on        
 .rgw.root                    1113                 3.0        431.3T  0.0000                     8              warn      
 djf_tmp                      8240                 3.0        431.3T  0.0000                  4096           4  warn      
 libvirt-pool                 2048M                3.0        431.3T  0.0000                  4096           4  warn      
 data                        78796G                3.0        431.3T  0.5351                  4096              warn      
 default.rgw.log                 0                 3.0        431.3T  0.0000                     8              warn      
 metadata                    863.3M                4.0        431.3T  0.0000                    64           4  warn

i.e., 860 MB.

but the pg stats root@mira070:~# PG OBJECTS 1.0 37441 0 1.1 37570 0 1.2 37636 0 1.3 37363 0 1.4 37178 0 1.5 37527 0 1.6 37676 0 1.7 37701 0 1.8 37524 0 1.9 37325 0 1.a 37726 0 1.b 37652 0 1.c 37609 0 1.d 37917 0 1.e 37641 0 1.f 37627 0 1.10 37752 0 1.11 37685 0 1.12 37900 0 1.13 37977 0 1.14 37350 0 1.15 37959 0 1.16 37727 0 1.17 37320 0 1.18 37489 0 1.19 37836 0 1.1a 37489 0 1.1b 37672 0 1.1c 37942 0 1.1d 37698 0 1.1e 37540 0 1.1f 37681 0 1.20 37482 0 1.21 37396 0 1.22 37519 0 1.23 37932 0 1.24 37399 0 1.25 37453 0 1.26 37759 0 1.27 37756 0 1.28 37863 0 1.29 37594 0 1.2a 37454 0 1.2b 37678 0 1.2c 37336 0 1.2d 37566 0 1.2e 37780 0 1.2f 37457 0 1.30 37528 0 1.31 37355 0 1.32 37812 0 1.33 37797 0 1.34 37586 0 1.35 37810 0 1.36 37540 0 1.37 37790 0 1.38 37763 0 1.39 37237 0 1.3a 37617 0 1.3b 37507 0 1.3c 37783 0 1.3d 37571 0 1.3e 37610 0 1.3f 37394 0 * NOTE: Omap show lots more data in omap:

ceph pg ls-by-pool metadata DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP 0 0 8388608 1004561096 2266056 3048 active+clean 7h 1084714'22593561 1084714:28756662 [149,9,105,96]p149 [149,9,105,96]p149 2019-03-12 09:11:08.387089 2019-03-08 23:41:12.690675 0 0 16777216 985896696 2219788 3040 active+clean 31h 1084714'22792937 1084714:29404032 [91,71,69,37]p91 [91,71,69,37]p91 2019-03-11 09:32:46.943843 2019-03-07 18:57:40.432687 0 0 4194304 980205936 2209344 3059 active+clean 32h 1084714'22913289 1084714:29626803 [129,83,49,104]p129 [129,83,49,104]p129 2019-03-11 08:07:24.635665 2019-03-06 06:37:33.634607 0 0 8388608 0 0 3069 active+clean 22h 1084714'23108051 1084714:29464208 [125,3,54,65]p125 [125,3,54,65]p125 2019-03-11 18:11:20.705113 2019-03-05 09:53:49.106319 0 0 16777216 1130328292 2531384 3033 active+clean 5h 1084714'23028327 1084714:30106978 [54,47,33,147]p54 [54,47,33,147]p54 2019-03-12 11:12:25.545303 2019-03-10 01:15:58.614116 0 0 16777216 994207248 2252300 3004 active+clean 23h 1084714'22897470 1084714:29411024 [69,134,135,120]p69 [69,134,135,120]p69 2019-03-11 16:52:45.729981 2019-03-10 11:57:01.610544 0 0 8388608 1037062984 2334220 3084 active+clean 11h 1084714'22915672 1084714:29427802 [26,82,146,114]p26 [26,82,146,114]p26 2019-03-12 05:28:54.416447 2019-03-09 11:48:17.664437 0 0 12582912 1056816688 2371680 3054 active+clean 22h 1084714'22720924 1084714:30480200 [2,87,77,114]p2 [2,87,77,114]p2 2019-03-11 18:10:55.977174 2019-03-08 07:28:22.457829 0 0 16777216 1088492424 2465496 3096 active+clean 39m 1084684'22656541 1084714:29144768 [17,41,10,47]p17 [17,41,10,47]p17 2019-03-12 16:02:05.362587 2019-03-08 16:46:00.830506 0 0 8388608 929899340 2106088 3084 active+clean 9h 1084714'22869496 1084714:29631741 [138,103,121,133]p138 [138,103,121,133]p138 2019-03-12 07:36:34.117740 2019-03-10 20:12:44.318531 0 0 16777216 1024256920 2350900 3008 active+clean 17h 1084714'23162183 1084714:29739146 [25,60,96,13]p25 [25,60,96,13]p25 2019-03-11 22:46:48.964533 2019-03-11 22:46:48.964533 0 0 12582912 993524772 2240112 3049 active+clean 12h 1084702'23414081 1084713:29464174 [83,75,142,106]p83 [83,75,142,106]p83 2019-03-12 04:19:49.755534 2019-03-12 04:19:49.755534 0 0 4194304 1068101576 2406348 3054 active+clean 29h 1084712'22827093 1084714:29329690 [65,147,132,121]p65 [65,147,132,121]p65 2019-03-11 10:54:09.504936 2019-03-08 12:38:26.731382 0 0 16777216 1033104672 2326148 3084 active+clean 27h 1084714'22948846 1084714:28746352 [10,33,54,123]p10 [10,33,54,123]p10 2019-03-11 13:08:51.142121 2019-03-07 20:08:56.234449 0 0 20971520 1062104348 2385888 3042 active+clean 15h 1084684'22761496 1084713:30317254 [99,87,5,51]p99 [99,87,5,51]p99 2019-03-12 00:58:47.715232 2019-03-07 03:53:34.700645 0 0 29360150 1000324596 2247708 3083 active+clean 23h 1084714'22831186 1084714:30333260 [3,104,114,74]p3 [3,104,114,74]p3 2019-03-11 16:43:16.030285 2019-03-07 16:06:14.289076 0 0 16777216 1020877468 2301192 3001 active+clean 17h 1084714'23034440 1084714:29803207 [46,79,128,62]p46 [46,79,128,62]p46 2019-03-11 23:39:38.449144 2019-03-11 23:39:38.449144 0 0 12582912 986447832 2214376 3062 active+clean 15h 1084714'22786800 1084714:29331295 [88,112,14,91]p88 [88,112,14,91]p88 2019-03-12 00:46:28.825399 2019-03-10 22:19:08.845563 0 0 25165824 1050285100 2364628 3097 active+clean 11h 1084702'22465490 1084713:29287250 [146,141,2,133]p146 [146,141,2,133]p146 2019-03-12 05:30:57.312170 2019-03-11 05:24:51.369990 0 0 12582946 958213092 2170928 3076 active+clean 26h 1084714'23036476 1084714:30127653 [29,117,82,131]p29 [29,117,82,131]p29 2019-03-11 14:03:39.620316 2019-03-06 14:44:13.154698 0 0 16777752 957445504 2158396 3089 active+clean 20h 1084714'23488663 1084714:29810957 [31,25,55,130]p31 [31,25,55,130]p31 2019-03-11 20:13:07.367415 2019-03-06 16:53:40.291771 0 0 37748826 1172884792 2634804 3088 active+clean 31h 1084714'23673119 1084714:32069068 [75,128,126,56]p75 [75,128,126,56]p75 2019-03-11 09:31:53.568899 2019-03-10 02:11:53.020578 0 0 8388608 1218193808 2727328 3082 active+clean 23h 1084714'22782413 1084714:29392305 [117,138,54,111]p117 [117,138,54,111]p117 2019-03-11 16:51:26.978984 2019-03-10 08:18:25.094591 0 0 20971520 1055333080 2375108 3019 active+clean 69m 1084714'22794164 1084714:29228741 [1,76,33,36]p1 [1,76,33,36]p1 2019-03-12 15:31:55.832244 2019-03-12 15:31:55.832244 0 0 12582912 1072608144 2418196 3071 active+clean 13m 1084714'22712346 1084714:29334553 [3,72,104,122]p3 [3,72,104,122]p3 2019-03-12 16:27:50.485740 2019-03-11 11:33:09.985266 0 0 8388608 1080224556 2426688 3094 active+clean 22h 1084714'22689481 1084714:30118049 [34,17,25,81]p34 [34,17,25,81]p34 2019-03-11 18:23:05.486487 2019-03-09 02:52:22.238989 0 0 4194304 1087617244 2446916 3021 active+clean 18h 1084714'22953855 1084714:30105021 [111,112,36,125]p111 [111,112,36,125]p111 2019-03-11 22:39:42.235912 2019-03-09 03:55:25.638908 0 0 25165824 1060972872 2384244 3075 active+clean 5h 1084714'23293178 1084714:31018277 [42,94,84,90]p42 [42,94,84,90]p42 2019-03-12 10:50:23.851682 2019-03-08 15:34:07.410255 0 0 16777216 970694172 2187208 3082 active+clean 11h 1084702'22770376 1084713:29493410 [105,82,99,41]p105 [105,82,99,41]p105 2019-03-12 04:58:26.687344 2019-03-08 06:01:52.062667 0 0 4194304 1040358068 2395752 3089 active+clean 17h 1084714'22920879 1084714:29831066 [147,7,17,137]p147 [147,7,17,137]p147 2019-03-11 23:12:18.155537 2019-03-11 23:12:18.155537 0 0 12582912 1048299500 2354120 3049 active+clean 24h 1084714'22899603 1084714:28874114 [73,4,55,111]p73 [73,4,55,111]p73 2019-03-11 16:12:11.771140 2019-03-09 01:57:21.354640 0 0 12583002 1071175084 2436884 3034 active+clean 13h 1084714'34659708 1084714:39640459 [47,76,26,71]p47 [47,76,26,71]p47 2019-03-12 02:56:47.414893 2019-03-12 02:56:47.414893 0 0 8388608 1007294092 2278844 3086 active+clean 12h 1084714'22666701 1084714:27833486 [27,47,144,112]p27 [27,47,144,112]p27 2019-03-12 04:21:24.662595 2019-03-07 01:45:37.893343 0 0 12582912 903715736 2038668 3016 active+clean 18h 1084714'23174844 1084714:28556515 [41,13,9,31]p41 [41,13,9,31]p41 2019-03-11 22:40:02.812292 2019-03-06 12:22:24.401887 0 0 12582912 1147825068 2584412 3023 active+clean 20h 1084714'22868407 1084714:29169783 [59,121,136,98]p59 [59,121,136,98]p59 2019-03-11 20:08:15.431712 2019-03-07 12:23:35.981059 0 0 33554432 1039746440 2337804 3063 active+clean 15h 1084714'22761272 1084714:29840531 [96,125,13,141]p96 [96,125,13,141]p96 2019-03-12 01:36:49.607077 2019-03-08 14:40:10.041695 0 0 4194304 953720100 2165256 3083 active+clean 15h 1084714'22750102 1084714:29131213 [47,74,57,31]p47 [47,74,57,31]p47 2019-03-12 01:05:35.672878 2019-03-07 01:38:04.681243 0 0 12582912 1101494164 2497040 3017 active+clean 28h 1084714'22828721 1084714:30230519 [145,70,50,126]p145 [145,70,50,126]p145 2019-03-11 12:36:39.096380 2019-03-08 19:32:21.200444 0 0 8388608 0 0 3049 active+clean 21h 1084714'27852272 1084714:33703446 [132,26,56,57]p132 [132,26,56,57]p132 2019-03-11 19:01:47.819748 2019-03-05 12:31:14.189398 0 0 12583636 983941928 2212824 3087 active+clean 21h 1084714'22602755 1084714:30048376 [105,114,52,95]p105 [105,114,52,95]p105 2019-03-11 19:19:06.724417 2019-03-11 19:19:06.724417 0 0 16777216 1040398984 2333324 3061 active+clean 14h 1084714'22738725 1084714:28887703 [128,44,124,60]p128 [128,44,124,60]p128 2019-03-12 01:50:35.616499 2019-03-12 01:50:35.616499 0 0 25165824 1044094300 2354788 3042 active+clean 28h 1084714'23007656 1084714:28744937 [86,32,68,102]p86 [86,32,68,102]p86 2019-03-11 11:44:53.115346 2019-03-11 11:44:53.115346 0 0 8388608 1019231464 2347240 3070 active+clean 13h 1084714'22829286 1084714:29727280 [19,136,73,138]p19 [19,136,73,138]p19 2019-03-12 03:26:36.109351 2019-03-10 18:15:43.402194 0 0 25165824 998802724 2250000 3092 active+clean 13h 1084713'23022347 1084714:30309661 [96,46,78,146]p96 [96,46,78,146]p96 2019-03-12 03:39:46.705843 2019-03-06 17:10:50.083391 0 0 16777216 982482924 2216272 3040 active+clean 29h 1084714'22922958 1084714:30014900 [124,46,2,28]p124 [124,46,2,28]p124 2019-03-11 11:03:00.612032 2019-03-11 11:03:00.612032 0 0 12582982 971046340 2192748 3058 active+clean 27h 1084712'22673424 1084713:29072363 [74,32,38,67]p74 [74,32,38,67]p74 2019-03-11 13:27:46.167881 2019-03-10 12:05:06.730248 0 0 20971520 1000687408 2250964 3038 active+clean 10h 1084714'23139951 1084714:28553192 [123,100,20,91]p123 [123,100,20,91]p123 2019-03-12 06:04:11.170393 2019-03-11 04:22:46.409811 0 0 16777216 965642252 2191872 3090 active+clean 24h 1084712'22764879 1084713:29297566 [18,143,120,133]p18 [18,143,120,133]p18 2019-03-11 15:55:27.961537 2019-03-11 15:55:27.961537 0 0 8388608 1016447000 2293400 3002 active+clean 24h 1084714'22792001 1084714:28839373 [35,47,78,17]p35 [35,47,78,17]p35 2019-03-11 16:03:05.879032 2019-03-11 16:03:05.879032 0 0 16777216 1050670180 2372964 3002 active+clean 14m 1084714'22805199 1084714:29809613 [127,30,110,49]p127 [127,30,110,49]p127 2019-03-12 16:26:38.431907 2019-03-09 01:57:49.212009 0 0 4194304 1049948284 2369772 3001 active+clean 13h 1084714'22844477 1084714:29391693 [41,123,15,125]p41 [41,123,15,125]p41 2019-03-12 02:53:12.021771 2019-03-12 02:53:12.021771 0 0 4194304 984721348 2221216 3096 active+clean 26h 1084714'22774445 1084714:29153034 [69,86,30,138]p69 [69,86,30,138]p69 2019-03-11 14:16:17.521605 2019-03-07 18:52:19.437961 0 0 10404158 0 0 3075 active+clean 19h 1084714'23009518 1084714:30283894 [74,130,0,54]p74 [74,130,0,54]p74 2019-03-11 21:04:10.873707 2019-03-05 15:42:45.011809 0 0 0 1169703248 2615884 3074 active+clean 17h 1084714'22594608 1084714:28429289 [62,18,26,10]p62 [62,18,26,10]p62 2019-03-11 22:41:19.060206 2019-03-08 08:24:15.242867 0 0 16777216 961795864 2168668 3027 active+clean 17h 1084714'22952227 1084714:30425595 [83,12,72,57]p83 [83,12,72,57]p83 2019-03-11 22:51:14.992833 2019-03-07 00:21:41.500736 0 0 17306525 1062089800 2380568 3071 active+clean 4h 1084714'22874788 1084714:29226272 [99,50,149,67]p99 [99,50,149,67]p99 2019-03-12 11:41:38.902199 2019-03-11 10:59:43.906664 0 0 20971520 983899612 2220456 3047 active+clean 89m 1084714'22723442 1084714:28894821 [125,102,136,82]p125 [125,102,136,82]p125 2019-03-12 15:12:05.666325 2019-03-11 03:26:37.105435 0 0 20971520 1035812612 2328928 3090 active+clean 22h 1084714'22883822 1084714:29664564 [46,36,34,49]p46 [46,36,34,49]p46 2019-03-11 17:42:34.811056 2019-03-11 17:42:34.811056 0 0 4194304 1020080168 2291520 3020 active+clean 6h 1084702'22781891 1084714:29801197 [37,29,41,145]p37 [37,29,41,145]p37 2019-03-12 10:14:24.296445 2019-03-07 19:08:41.616166 0 0 12582912 1020676628 2296028 3025 active+clean 4h 1084714'22787832 1084714:30417133 [145,33,113,135]p145 [145,33,113,135]p145 2019-03-12 12:24:12.553458 2019-03-07 08:56:07.808949 0 0 10422738 1028500148 2326708 3021 active+clean 25h 1084702'22769694 1084713:29727735 [0,138,48,31]p0 [0,138,48,31]p0 2019-03-11 15:02:08.471586 2019-03-07 02:55:45.471143 0 0 12582912 1034089808 2323892 3088 active+clean 23h 1084714'22684357 1084714:28657194 [125,128,79,117]p125 [125,128,79,117]p125 2019-03-11 17:27:03.390717 2019-03-11 17:27:03.390717 0 0 16777216 1114097152 2497280 3082 active+clean 7h 1084714'22974468 1084714:29273496 [52,64,122,85]p52 [52,64,122,85]p52 2019-03-12 09:14:16.532618 2019-03-09 18:07:35.077896 0 0 12583456 1042435200 2343380 3024 active+clean 19h 1084714'22968009 1084714:28697534 [47,0,140,104]p47 [47,0,140,104]p47 2019-03-11 20:44:14.297409 2019-03-07 19:31:11.102469 statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilisation. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details.