STDOUT of rados bench performed on OSD replicated pools having different compression algorithms - # ceph df 2023-04-13T04:29:21.933+0000 7ff5a3db0700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-04-13T04:29:21.934+0000 7ff5a3db0700 -1 WARNING: the following dangerous and experimental features are enabled: crimson --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED TOTAL 1.5 TiB 1.5 TiB 9.4 GiB 9.4 GiB 0.63 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 897 KiB 4 897 KiB 0 470 GiB cephfs.cephfs.meta 2 32 4.6 KiB 44 4.6 KiB 0 470 GiB cephfs.cephfs.data 3 32 0 B 0 0 B 0 470 GiB .rgw.root 4 32 368 B 8 368 B 0 470 GiB default.rgw.log 5 32 7.2 KiB 418 7.2 KiB 0 470 GiB default.rgw.control 6 32 0 B 16 0 B 0 470 GiB default.rgw.meta 7 32 738 B 2 738 B 0 470 GiB re_pool_compress_snappy 8 32 0 B 0 0 B 0 470 GiB re_pool_compress_zlib 9 32 0 B 0 0 B 0 470 GiB re_pool_compress_zstd 11 32 0 B 0 0 B 0 470 GiB re_pool_no_compress 12 32 0 B 0 0 B 0 470 GiB [ceph: root@dell-r640-056 /]# rados --no-log-to-stderr -b 40KB -p re_pool_no_compress bench 500 write --no-cleanup --max-objects 100000 hints = 1 Maintaining 16 concurrent writes of 40960 bytes to objects of size 40960 for up to 500 seconds or 100000 objects Object prefix: benchmark_data_dell-r640-056.dsal.lab.eng.rd_270 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 287 271 10.5847 10.5859 0.0683872 0.0575751 2 16 578 562 10.9752 11.3672 0.0434753 0.0561028 3 16 875 859 11.1833 11.6016 0.0423493 0.0551702 4 16 1150 1134 11.0727 10.7422 0.0388067 0.0560068 5 16 1438 1422 11.1079 11.25 0.0552649 0.0559756 6 16 1739 1723 11.2159 11.7578 0.0172517 0.0554418 7 16 2032 2016 11.2484 11.4453 0.0634214 0.0553534 8 16 2327 2311 11.2826 11.5234 0.0222442 0.0551975 9 16 2627 2611 11.3308 11.7188 0.0464365 0.0549118 10 15 2924 2909 11.3616 11.6406 0.0495665 0.0548811 11 16 3217 3201 11.3655 11.4062 0.0321529 0.054786 12 15 3468 3453 11.2385 9.84375 0.0623625 0.0554531 13 16 3715 3699 11.1131 9.60938 0.100359 0.0561318 14 16 3985 3969 11.0725 10.5469 0.0779134 0.056318 15 16 4216 4200 10.9358 9.02344 0.0221148 0.0570195 16 16 4456 4440 10.8382 9.375 0.0497517 0.0575287 17 16 4690 4674 10.7383 9.14062 0.059378 0.0580069 18 16 4923 4907 10.6472 9.10156 0.0790001 0.0585814 19 16 5152 5136 10.5576 8.94531 0.106847 0.0590694 2023-04-13T04:30:15.781292+0000 min lat: 0.00787493 max lat: 0.187179 avg lat: 0.0596175 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 20 16 5375 5359 10.4652 8.71094 0.0573091 0.0596175 21 16 5601 5585 10.3872 8.82812 0.0603609 0.0600644 22 16 5834 5818 10.3287 9.10156 0.0615958 0.0604168 23 16 6061 6045 10.2651 8.86719 0.125829 0.0607753 24 16 6296 6280 10.2198 9.17969 0.12045 0.0610495 25 16 6538 6522 10.189 9.45312 0.0842851 0.0612415 26 16 6769 6753 10.1442 9.02344 0.0493381 0.0615238 27 16 7017 7001 10.1272 9.6875 0.068832 0.0616427 28 16 7230 7214 10.0626 8.32031 0.0716816 0.062014 29 16 7462 7446 10.028 9.0625 0.0398883 0.062242 30 16 7700 7684 10.0036 9.29688 0.0512588 0.0623731 31 16 7945 7929 9.9896 9.57031 0.0906923 0.0624863 32 16 8206 8190 9.99598 10.1953 0.0582938 0.062434 33 16 8465 8449 9.99959 10.1172 0.05068 0.0624362 34 16 8723 8707 10.0018 10.0781 0.0509273 0.062405 35 16 8994 8978 10.0185 10.5859 0.0801977 0.0623172 36 16 9234 9218 10.0006 9.375 0.0483149 0.0623864 37 16 9493 9477 10.0037 10.1172 0.0805909 0.0624034 38 16 9752 9736 10.0066 10.1172 0.059335 0.06239 39 16 10011 9995 10.0094 10.1172 0.0790754 0.0623882 2023-04-13T04:30:35.784652+0000 min lat: 0.00770803 max lat: 0.191989 avg lat: 0.0623409 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 40 16 10275 10259 10.0169 10.3125 0.0680646 0.0623409 41 16 10526 10510 10.0117 9.80469 0.0690124 0.0623651 42 16 10764 10748 9.9947 9.29688 0.0563514 0.0624586 43 16 11021 11005 9.99568 10.0391 0.0524905 0.0624584 44 16 11248 11232 9.97 8.86719 0.0450503 0.0626299 45 16 11510 11494 9.97584 10.2344 0.0673725 0.062573 46 16 11790 11774 9.99671 10.9375 0.0527783 0.0624433 47 16 12071 12055 10.0175 10.9766 0.0584449 0.0623422 48 16 12338 12322 10.0261 10.4297 0.0151255 0.0622728 49 16 12606 12590 10.0351 10.4688 0.0523621 0.0622383 50 16 12876 12860 10.0453 10.5469 0.085393 0.0621534 51 16 13143 13127 10.0528 10.4297 0.0485874 0.0621096 52 16 13396 13380 10.0495 9.88281 0.120692 0.0621315 53 16 13652 13636 10.0485 10 0.0639508 0.0621354 54 16 13918 13902 10.0548 10.3906 0.0968875 0.0621035 55 16 14178 14162 10.0566 10.1562 0.0486418 0.0620842 56 16 14448 14432 10.0653 10.5469 0.048846 0.0620493 57 16 14764 14748 10.1053 12.3438 0.0595027 0.0618032 58 16 15074 15058 10.1398 12.1094 0.0664689 0.0615948 59 16 15403 15387 10.1857 12.8516 0.0766947 0.0613256 2023-04-13T04:30:55.787967+0000 min lat: 0.00694351 max lat: 0.233946 avg lat: 0.0612729 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 60 16 15679 15663 10.1956 10.7812 0.0584477 0.0612729 61 16 15965 15949 10.2116 11.1719 0.0450021 0.0611701 62 16 16263 16247 10.2346 11.6406 0.0588476 0.0610306 63 16 16557 16541 10.2544 11.4844 0.0475451 0.0609172 64 16 16857 16841 10.2773 11.7188 0.0496627 0.0607792 65 16 17156 17140 10.2988 11.6797 0.0275898 0.0606571 66 16 17443 17427 10.3126 11.2109 0.0277537 0.0605678 67 16 17737 17721 10.3301 11.4844 0.0296222 0.0604712 68 16 18005 17989 10.3321 10.4688 0.0881236 0.0604652 69 16 18298 18282 10.3482 11.4453 0.0239707 0.0603626 70 16 18595 18579 10.3661 11.6016 0.0300168 0.0602513 71 16 18883 18867 10.3785 11.25 0.0547226 0.0601932 72 16 19128 19112 10.3672 9.57031 0.0872738 0.0602446 73 16 19362 19346 10.3504 9.14062 0.0486792 0.0603437 74 16 19609 19593 10.3409 9.64844 0.0706281 0.0604003 75 16 19869 19853 10.3384 10.1562 0.0719124 0.0604122 76 16 20109 20093 10.3257 9.375 0.0598322 0.0604827 77 16 20357 20341 10.3174 9.6875 0.0757183 0.0605452 78 16 20603 20587 10.3083 9.60938 0.0795949 0.060581 79 16 20850 20834 10.2999 9.64844 0.0521018 0.0606538 2023-04-13T04:31:15.791223+0000 min lat: 0.00694351 max lat: 0.233946 avg lat: 0.0607083 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 80 16 21092 21076 10.2893 9.45312 0.077884 0.0607083 81 16 21336 21320 10.28 9.53125 0.0488792 0.0607624 82 16 21582 21566 10.2718 9.60938 0.0505721 0.0608182 83 16 21828 21812 10.2638 9.60938 0.0668564 0.0608611 84 16 22071 22055 10.2546 9.49219 0.0815923 0.060912 85 16 22322 22306 10.2493 9.80469 0.0454986 0.0609426 86 16 22565 22549 10.2404 9.49219 0.0276259 0.0609949 87 16 22825 22809 10.2395 10.1562 0.106878 0.0610087 88 16 23086 23070 10.2389 10.1953 0.0609279 0.0610106 89 16 23374 23358 10.2503 11.25 0.0409195 0.060947 90 16 23643 23627 10.2531 10.5078 0.0527102 0.0609126 91 16 23943 23927 10.2692 11.7188 0.0750178 0.060836 92 16 24216 24200 10.2735 10.6641 0.0549978 0.0608076 93 16 24504 24488 10.284 11.25 0.0599619 0.0607441 94 16 24807 24791 10.3004 11.8359 0.0706325 0.0606512 95 16 25093 25077 10.3096 11.1719 0.0607697 0.0605937 96 16 25371 25355 10.3153 10.8594 0.0564289 0.0605525 97 16 25665 25649 10.3273 11.4844 0.0540379 0.0604877 98 16 25958 25942 10.3387 11.4453 0.0382119 0.0604187 99 16 26252 26236 10.3503 11.4844 0.0460709 0.0603542 2023-04-13T04:31:35.794409+0000 min lat: 0.0067134 max lat: 0.233946 avg lat: 0.0603391 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 100 16 26525 26509 10.3534 10.6641 0.0459871 0.0603391 101 16 26816 26800 10.3634 11.3672 0.0516047 0.0602842 102 16 27121 27105 10.3786 11.9141 0.0288025 0.0601922 103 16 27430 27414 10.395 12.0703 0.0613306 0.0600954 104 16 27728 27712 10.407 11.6406 0.0621419 0.0600244 105 16 28003 27987 10.4101 10.7422 0.0572566 0.0600054 106 16 28261 28245 10.407 10.0781 0.0283223 0.0600278 107 16 28534 28518 10.4094 10.6641 0.0529175 0.0600203 108 16 28787 28771 10.4045 9.88281 0.0661167 0.0600448 109 16 29053 29037 10.4044 10.3906 0.0800588 0.0600439 110 16 29319 29303 10.4042 10.3906 0.041533 0.0600496 111 16 29582 29566 10.403 10.2734 0.0888175 0.060054 112 16 29844 29828 10.4015 10.2344 0.0551219 0.0600654 113 16 30114 30098 10.4028 10.5469 0.095835 0.0600486 114 16 30368 30352 10.3985 9.92188 0.0551143 0.0600765 115 16 30640 30624 10.4005 10.625 0.0615037 0.0600687 116 16 30893 30877 10.396 9.88281 0.0885447 0.0600941 117 16 31132 31116 10.3869 9.33594 0.047488 0.0601405 118 16 31392 31376 10.385 10.1562 0.0750483 0.0601534 119 16 31645 31629 10.3807 9.88281 0.0672809 0.0601761 2023-04-13T04:31:55.797762+0000 min lat: 0.0067134 max lat: 0.233946 avg lat: 0.0602283 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 120 16 31885 31869 10.3723 9.375 0.0429886 0.0602283 121 16 32123 32107 10.3634 9.29688 0.0935269 0.060279 122 16 32358 32342 10.3537 9.17969 0.0769058 0.0603418 123 16 32588 32572 10.3426 8.98438 0.0179038 0.0604043 124 16 32811 32795 10.3294 8.71094 0.0492311 0.060478 125 16 33045 33029 10.3199 9.14062 0.0635331 0.0605353 126 16 33280 33264 10.3108 9.17969 0.0620663 0.0605857 127 16 33516 33500 10.3022 9.21875 0.0691514 0.0606408 128 16 33750 33734 10.2931 9.14062 0.0493491 0.0606934 129 16 33991 33975 10.2863 9.41406 0.0704108 0.0607359 130 16 34222 34206 10.2766 9.02344 0.0521439 0.0607918 131 16 34463 34447 10.27 9.41406 0.0851605 0.0608288 132 16 34649 34633 10.2472 7.26562 0.00911847 0.0609636 133 16 34830 34814 10.2233 7.07031 0.0611074 0.0611036 134 16 35020 35004 10.2024 7.42188 0.178597 0.0612323 135 16 35216 35200 10.1835 7.65625 0.0674179 0.0613446 136 16 35427 35411 10.1692 8.24219 0.0996166 0.0614303 137 15 35644 35629 10.1572 8.51562 0.081284 0.0615001 138 16 35862 35846 10.145 8.47656 0.03464 0.0615814 139 16 36073 36057 10.1313 8.24219 0.0518543 0.0616589 2023-04-13T04:32:15.801171+0000 min lat: 0.0067134 max lat: 0.233946 avg lat: 0.0617785 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 140 16 36265 36249 10.1125 7.5 0.0842291 0.0617785 141 16 36474 36458 10.0986 8.16406 0.0527481 0.0618636 142 16 36698 36682 10.0891 8.75 0.0649487 0.0619235 143 16 36921 36905 10.0795 8.71094 0.0978577 0.0619808 144 16 37140 37124 10.0689 8.55469 0.0452806 0.0620417 145 16 37354 37338 10.0571 8.35938 0.0510779 0.0621247 146 16 37563 37547 10.0441 8.16406 0.0903234 0.0621978 147 16 37794 37778 10.0372 9.02344 0.06509 0.0622458 148 16 38003 37987 10.0245 8.16406 0.111246 0.0623178 149 16 38237 38221 10.0186 9.14062 0.107148 0.0623603 150 16 38477 38461 10.0143 9.375 0.067709 0.0623907 151 16 38715 38699 10.0095 9.29688 0.0191322 0.0624071 152 16 38952 38936 10.0045 9.25781 0.0538711 0.0624429 153 16 39178 39162 9.99683 8.82812 0.0841213 0.0624919 154 16 39419 39403 9.99304 9.41406 0.0488296 0.0625166 155 16 39662 39646 9.9898 9.49219 0.0558414 0.0625386 156 16 39888 39872 9.98235 8.82812 0.0544514 0.0625877 157 16 40115 40099 9.97523 8.86719 0.06045 0.0626281 158 16 40361 40345 9.97291 9.60938 0.0695494 0.062647 159 16 40611 40595 9.97159 9.76562 0.105889 0.0626557 2023-04-13T04:32:35.804283+0000 min lat: 0.0067134 max lat: 0.233946 avg lat: 0.0626791 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 160 16 40852 40836 9.9681 9.41406 0.0949545 0.0626791 161 16 41083 41067 9.96223 9.02344 0.064039 0.0627124 162 16 41348 41332 9.96462 10.3516 0.0612381 0.0627018 163 16 41623 41607 9.96938 10.7422 0.0320456 0.062666 164 16 41880 41864 9.9698 10.0391 0.0235704 0.0626692 165 16 42137 42121 9.97021 10.0391 0.0609901 0.0626645 166 16 42389 42373 9.96944 9.84375 0.0651547 0.0626661 167 16 42630 42614 9.96611 9.41406 0.0709919 0.0626897 168 16 42869 42853 9.96236 9.33594 0.0509809 0.0627113 169 16 43108 43092 9.95864 9.33594 0.046363 0.0627379 170 16 43349 43333 9.95543 9.41406 0.0805091 0.0627562 171 16 43594 43578 9.95316 9.57031 0.0872654 0.0627699 172 16 43829 43813 9.94866 9.17969 0.0781715 0.0627971 173 16 44067 44051 9.94488 9.29688 0.0524784 0.0628258 174 16 44323 44307 9.94519 10 0.0870153 0.0628232 175 16 44571 44555 9.94371 9.6875 0.0559403 0.0628311 176 16 44813 44797 9.94092 9.45312 0.111347 0.0628519 177 16 45031 45015 9.93286 8.51562 0.16219 0.0628952 178 16 45253 45237 9.92578 8.67188 0.0539801 0.0629429 179 16 45504 45488 9.92509 9.80469 0.0611745 0.0629502 2023-04-13T04:32:55.806968+0000 min lat: 0.0067134 max lat: 0.233946 avg lat: 0.0629999 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 180 16 45718 45702 9.91639 8.35938 0.0636252 0.0629999 181 16 45958 45942 9.91339 9.375 0.0622234 0.0630244 182 16 46205 46189 9.91192 9.64844 0.0468308 0.0630348 183 16 46450 46434 9.91005 9.57031 0.0501969 0.063044 184 16 46695 46679 9.90819 9.57031 0.055093 0.0630585 185 16 46947 46931 9.90783 9.84375 0.063371 0.0630619 186 16 47199 47183 9.90748 9.84375 0.0493386 0.0630628 187 16 47458 47442 9.90859 10.1172 0.0807961 0.0630542 188 16 47694 47678 9.90492 9.21875 0.032734 0.0630772 189 16 47929 47913 9.90107 9.17969 0.0560884 0.0630982 190 16 48191 48175 9.90282 10.2344 0.0889829 0.0630953 191 16 48442 48426 9.90229 9.80469 0.0635014 0.0630958 192 16 48704 48688 9.90402 10.2344 0.0503256 0.0630859 193 16 48964 48948 9.90531 10.1562 0.0854421 0.0630799 194 16 49222 49206 9.9062 10.0781 0.0231897 0.0630706 195 16 49501 49485 9.91128 10.8984 0.0459288 0.063042 196 16 49752 49736 9.91073 9.80469 0.0287034 0.0630464 197 16 50007 49991 9.91097 9.96094 0.0156564 0.0630418 198 16 50281 50265 9.91496 10.7031 0.0782293 0.063017 199 16 50537 50521 9.91539 10 0.0408209 0.0630176 2023-04-13T04:33:15.810095+0000 min lat: 0.0066822 max lat: 0.233946 avg lat: 0.0630029 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 200 16 50796 50780 9.91639 10.1172 0.0829141 0.0630029 201 16 51058 51042 9.91796 10.2344 0.0499998 0.0629932 202 16 51335 51319 9.92242 10.8203 0.0464602 0.0629718 203 16 51605 51589 9.92549 10.5469 0.0599267 0.0629497 204 16 51855 51839 9.9247 9.76562 0.0634712 0.062953 205 16 52117 52101 9.9262 10.2344 0.0680669 0.0629452 206 16 52385 52369 9.92883 10.4688 0.0778171 0.062929 207 16 52654 52638 9.93162 10.5078 0.0574906 0.062911 208 15 52938 52923 9.93738 11.1328 0.0379962 0.062876 209 16 53225 53209 9.94328 11.1719 0.052867 0.0628389 210 16 53473 53457 9.94205 9.6875 0.0190741 0.0628425 211 16 53710 53694 9.9388 9.25781 0.0614042 0.0628664 212 16 53924 53908 9.93134 8.35938 0.0858349 0.0629106 213 16 54166 54150 9.92909 9.45312 0.0283211 0.0629235 214 16 54410 54394 9.92723 9.53125 0.0458464 0.0629385 215 16 54647 54631 9.9241 9.25781 0.0947063 0.0629569 216 16 54885 54869 9.92119 9.29688 0.0413292 0.0629741 217 16 55130 55114 9.91957 9.57031 0.0477699 0.0629894 218 16 55356 55340 9.91455 8.82812 0.0187326 0.0630181 219 16 55594 55578 9.91173 9.29688 0.0466249 0.0630379 2023-04-13T04:33:35.813442+0000 min lat: 0.0066822 max lat: 0.237063 avg lat: 0.063093 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 220 16 55798 55782 9.90289 7.96875 0.0860161 0.063093 221 16 56029 56013 9.8989 9.02344 0.0587585 0.0631212 222 16 56289 56273 9.90005 10.1562 0.0322639 0.0631149 223 16 56531 56515 9.89804 9.45312 0.03688 0.0631261 224 16 56761 56745 9.89395 8.98438 0.0538373 0.0631514 225 16 56987 56971 9.88921 8.82812 0.033802 0.063183 226 16 57249 57233 9.89073 10.2344 0.0494237 0.0631729 227 16 57516 57500 9.8931 10.4297 0.0311167 0.0631559 228 16 57767 57751 9.89271 9.80469 0.0548087 0.0631573 229 16 58029 58013 9.89419 10.2344 0.0704255 0.0631467 230 16 58300 58284 9.89719 10.5859 0.0510825 0.0631309 231 16 58569 58553 9.89982 10.5078 0.0519224 0.0631137 232 16 58837 58821 9.90227 10.4688 0.0386986 0.0631014 233 16 59108 59092 9.9052 10.5859 0.0325558 0.0630807 234 16 59373 59357 9.9071 10.3516 0.0516129 0.0630674 235 16 59645 59629 9.91014 10.625 0.0590862 0.063049 236 16 59888 59872 9.90837 9.49219 0.0500432 0.0630592 237 16 60136 60120 9.90743 9.6875 0.0493265 0.0630641 238 16 60383 60367 9.90634 9.64844 0.0687673 0.0630736 239 16 60648 60632 9.90819 10.3516 0.0497189 0.063061 2023-04-13T04:33:55.816661+0000 min lat: 0.0066822 max lat: 0.237063 avg lat: 0.0630575 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 240 16 60907 60891 9.90906 10.1172 0.0738061 0.0630575 241 16 61172 61156 9.91089 10.3516 0.0825007 0.0630431 242 16 61430 61414 9.91157 10.0781 0.071449 0.0630384 243 16 61707 61691 9.9153 10.8203 0.0502643 0.0630133 244 16 61955 61939 9.91436 9.6875 0.016625 0.0630203 245 16 62218 62202 9.91582 10.2734 0.0463848 0.0630093 246 16 62496 62480 9.91965 10.8594 0.0502291 0.0629887 247 16 62748 62732 9.91933 9.84375 0.0604503 0.0629908 248 16 63005 62989 9.91981 10.0391 0.0730583 0.0629887 249 16 63274 63258 9.92217 10.5078 0.0763509 0.0629714 250 16 63542 63526 9.92434 10.4688 0.0473558 0.0629557 251 16 63814 63798 9.92713 10.625 0.0415787 0.062942 252 16 64058 64042 9.92555 9.53125 0.0178633 0.0629534 253 16 64316 64300 9.92615 10.0781 0.137866 0.0629464 254 16 64577 64561 9.9272 10.1953 0.115637 0.0629401 255 16 64833 64817 9.92748 10 0.0630622 0.0629398 256 16 65089 65073 9.92776 10 0.0532817 0.0629394 257 16 65351 65335 9.92895 10.2344 0.0966405 0.062931 258 16 65595 65579 9.9274 9.53125 0.0785797 0.0629408 259 16 65840 65824 9.92602 9.57031 0.0389289 0.0629473 2023-04-13T04:34:15.819815+0000 min lat: 0.0066822 max lat: 0.237063 avg lat: 0.0629678 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 260 16 66072 66056 9.92269 9.0625 0.0653287 0.0629678 261 16 66315 66299 9.92104 9.49219 0.0875313 0.0629761 262 16 66561 66545 9.91984 9.60938 0.0533952 0.062985 263 16 66807 66791 9.91865 9.60938 0.0433344 0.0629926 264 16 67052 67036 9.91733 9.57031 0.0471389 0.0630039 265 16 67298 67282 9.91616 9.60938 0.0282711 0.0630125 266 16 67551 67535 9.91603 9.88281 0.0279553 0.0630133 267 16 67797 67781 9.91488 9.60938 0.0569988 0.0630207 268 16 68046 68030 9.91417 9.72656 0.02916 0.0630255 269 16 68332 68316 9.91884 11.1719 0.0642025 0.0629942 270 16 68631 68615 9.92536 11.6797 0.0436197 0.0629549 271 16 68927 68911 9.93139 11.5625 0.0379381 0.062914 272 16 69239 69223 9.93968 12.1875 0.0583116 0.0628645 273 16 69557 69541 9.94877 12.4219 0.041011 0.0628071 274 16 69873 69857 9.9575 12.3438 0.0639172 0.0627517 275 16 70169 70153 9.96333 11.5625 0.0494705 0.0627125 276 16 70464 70448 9.96898 11.5234 0.0338115 0.0626793 277 16 70772 70756 9.97642 12.0312 0.0401009 0.0626328 278 16 71080 71064 9.9838 12.0312 0.0398965 0.0625856 279 15 71389 71374 9.99141 12.1094 0.0934818 0.0625374 2023-04-13T04:34:35.822698+0000 min lat: 0.0066822 max lat: 0.237063 avg lat: 0.0624895 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 280 16 71702 71686 9.99925 12.1875 0.0516263 0.0624895 281 16 72010 71994 10.0065 12.0312 0.0523098 0.0624424 282 16 72318 72302 10.0136 12.0312 0.101752 0.0623977 283 16 72643 72627 10.0231 12.6953 0.0463344 0.062341 284 16 72955 72939 10.0307 12.1875 0.0531815 0.0622941 285 16 73265 73249 10.038 12.1094 0.0437772 0.0622458 286 16 73564 73548 10.0438 11.6797 0.0552705 0.0622124 287 16 73863 73847 10.0494 11.6797 0.0755014 0.0621792 288 16 74154 74138 10.054 11.3672 0.045814 0.0621506 289 16 74448 74432 10.059 11.4844 0.0284454 0.0621169 290 16 74746 74730 10.0644 11.6406 0.052025 0.0620825 291 16 75059 75043 10.0718 12.2266 0.0616117 0.0620407 292 16 75325 75309 10.0729 10.3906 0.0563765 0.0620336 293 16 75616 75600 10.0773 11.3672 0.0573873 0.0620019 294 16 75914 75898 10.0826 11.6406 0.0938196 0.0619712 295 16 76212 76196 10.0879 11.6406 0.0451231 0.0619399 296 16 76500 76484 10.0918 11.25 0.0337673 0.0619153 297 16 76762 76746 10.0923 10.2344 0.0538929 0.0619118 298 16 77013 76997 10.0913 9.80469 0.060154 0.061916 299 16 77232 77216 10.0862 8.55469 0.0534335 0.0619495 2023-04-13T04:34:55.825747+0000 min lat: 0.0066822 max lat: 0.237063 avg lat: 0.0619842 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 300 16 77447 77431 10.0806 8.39844 0.0626546 0.0619842 301 16 77685 77669 10.078 9.29688 0.0679352 0.0619977 302 16 77939 77923 10.0774 9.92188 0.035612 0.0620025 303 16 78197 78181 10.0774 10.0781 0.0486818 0.0620036 304 16 78453 78437 10.0772 10 0.0699791 0.062003 305 16 78707 78691 10.0767 9.92188 0.051139 0.0620081 306 16 78955 78939 10.0754 9.6875 0.0636815 0.0620151 307 16 79208 79192 10.0748 9.88281 0.0832643 0.0620186 308 16 79450 79434 10.0727 9.45312 0.0557125 0.0620328 309 16 79703 79687 10.0721 9.88281 0.0543401 0.0620332 310 16 79958 79942 10.0717 9.96094 0.0827969 0.0620381 311 16 80212 80196 10.0713 9.92188 0.103057 0.0620414 312 16 80479 80463 10.0724 10.4297 0.0439203 0.0620372 313 16 80741 80725 10.0729 10.2344 0.0646419 0.062034 314 16 81034 81018 10.0773 11.4453 0.054165 0.0620042 315 16 81321 81305 10.0809 11.2109 0.0812169 0.0619845 316 16 81580 81564 10.081 10.1172 0.0988961 0.0619838 317 16 81848 81832 10.0822 10.4688 0.0572393 0.061975 318 16 82130 82114 10.0851 11.0156 0.0648129 0.0619583 319 16 82417 82401 10.0887 11.2109 0.0618122 0.0619363 2023-04-13T04:35:15.828882+0000 min lat: 0.0066822 max lat: 0.237063 avg lat: 0.0619283 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 320 16 82688 82672 10.0902 10.5859 0.021457 0.0619283 321 16 82966 82950 10.0926 10.8594 0.0462888 0.0619101 322 16 83238 83222 10.0942 10.625 0.0187315 0.0619013 323 16 83512 83496 10.0961 10.7031 0.0817472 0.0618893 324 16 83778 83762 10.097 10.3906 0.054698 0.0618831 325 16 84050 84034 10.0986 10.625 0.0592656 0.0618707 326 16 84325 84309 10.1006 10.7422 0.0684018 0.0618614 327 16 84609 84593 10.1036 11.0938 0.0541419 0.0618432 328 16 84897 84881 10.1071 11.25 0.0230326 0.0618237 329 16 85175 85159 10.1094 10.8594 0.0487811 0.0618066 330 16 85413 85397 10.1069 9.29688 0.0471349 0.0618246 331 16 85660 85644 10.1056 9.64844 0.050628 0.0618328 332 16 85881 85865 10.1011 8.63281 0.0207907 0.0618549 333 16 86139 86123 10.101 10.0781 0.054665 0.0618599 334 16 86389 86373 10.1 9.76562 0.0680543 0.0618661 335 16 86647 86631 10.1 10.0781 0.141393 0.0618672 336 16 86902 86886 10.0995 9.96094 0.079734 0.0618675 337 16 87154 87138 10.0988 9.84375 0.0983093 0.0618726 338 16 87397 87381 10.097 9.49219 0.0161421 0.0618824 339 16 87654 87638 10.0968 10.0391 0.0662832 0.0618858 2023-04-13T04:35:35.832063+0000 min lat: 0.0066822 max lat: 0.237063 avg lat: 0.0619078 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 340 16 87880 87864 10.0931 8.82812 0.0465942 0.0619078 341 16 88129 88113 10.092 9.72656 0.0815433 0.0619139 342 16 88352 88336 10.088 8.71094 0.0895559 0.061941 343 16 88555 88539 10.0817 7.92969 0.0494939 0.06198 344 16 88766 88750 10.0763 8.24219 0.0565766 0.0620086 345 16 88982 88966 10.0716 8.4375 0.0463733 0.0620396 346 16 89225 89209 10.0699 9.49219 0.0561288 0.0620549 347 16 89444 89428 10.0655 8.55469 0.0573374 0.0620768 348 16 89662 89646 10.061 8.51562 0.106141 0.0621009 349 16 89892 89876 10.058 8.98438 0.0580343 0.0621245 350 16 90118 90102 10.0544 8.82812 0.0577856 0.0621444 351 16 90349 90333 10.0515 9.02344 0.0859864 0.0621633 352 16 90582 90566 10.0488 9.10156 0.0530436 0.0621803 353 16 90804 90788 10.0449 8.67188 0.0268503 0.0622046 354 16 91034 91018 10.0419 8.98438 0.0948587 0.0622231 355 16 91264 91248 10.0389 8.98438 0.050689 0.0622393 356 16 91476 91460 10.034 8.28125 0.0645269 0.0622746 357 16 91711 91695 10.0316 9.17969 0.072997 0.062287 358 16 91976 91960 10.0325 10.3516 0.066533 0.0622827 359 16 92240 92224 10.0332 10.3125 0.0574625 0.0622774 2023-04-13T04:35:55.835142+0000 min lat: 0.0066822 max lat: 0.237063 avg lat: 0.0622773 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 360 16 92496 92480 10.0331 10 0.0447063 0.0622773 361 16 92734 92718 10.0311 9.29688 0.0524811 0.0622918 362 16 92996 92980 10.0317 10.2344 0.048341 0.0622883 363 16 93264 93248 10.0329 10.4688 0.0996151 0.062281 364 16 93498 93482 10.0304 9.14062 0.0447888 0.062293 365 16 93755 93739 10.0304 10.0391 0.0503456 0.0622958 366 16 94010 93994 10.0302 9.96094 0.0513725 0.0622972 367 16 94275 94259 10.0311 10.3516 0.0649333 0.06229 368 16 94536 94520 10.0315 10.1953 0.060286 0.0622894 369 16 94790 94774 10.0312 9.92188 0.04978 0.0622886 370 16 95041 95025 10.0306 9.80469 0.0550417 0.0622927 371 16 95305 95289 10.0314 10.3125 0.0213855 0.0622899 372 16 95561 95545 10.0313 10 0.0482672 0.0622884 373 16 95820 95804 10.0315 10.1172 0.0503317 0.0622873 374 16 96067 96051 10.0305 9.64844 0.0457981 0.0622953 375 16 96304 96288 10.0284 9.25781 0.0460983 0.0623058 376 16 96540 96524 10.0263 9.21875 0.0575584 0.0623222 377 16 96765 96749 10.023 8.78906 0.077206 0.0623421 378 16 96993 96977 10.02 8.90625 0.0693835 0.06236 379 16 97223 97207 10.0173 8.98438 0.0966295 0.0623748 2023-04-13T04:36:15.838299+0000 min lat: 0.00628277 max lat: 0.237063 avg lat: 0.0624066 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 380 16 97434 97418 10.0126 8.24219 0.0692607 0.0624066 381 16 97655 97639 10.009 8.63281 0.0908336 0.0624291 382 16 97885 97869 10.0063 8.98438 0.0563521 0.0624467 383 16 98119 98103 10.004 9.14062 0.0850483 0.0624585 384 16 98358 98342 10.0023 9.33594 0.0816665 0.0624705 385 16 98589 98573 9.99974 9.02344 0.0760355 0.0624872 386 16 98816 98800 9.9968 8.86719 0.0585332 0.0625028 387 16 99064 99048 9.996 9.6875 0.0873998 0.0625094 388 16 99275 99259 9.99147 8.24219 0.0750911 0.062537 389 16 99517 99501 9.99009 9.45312 0.0733724 0.0625466 390 16 99769 99753 9.98971 9.84375 0.0436203 0.0625503 391 16 99993 99977 9.98653 8.75 0.0567087 0.0625694 Total time run: 391.174 Total writes made: 100000 Write size: 40960 Object size: 40960 Bandwidth (MB/sec): 9.98597 Stddev Bandwidth: 1.02962 Max bandwidth (MB/sec): 12.8516 Min bandwidth (MB/sec): 7.07031 Average IOPS: 255 Stddev IOPS: 26.3582 Max IOPS: 329 Min IOPS: 181 Average Latency(s): 0.0625713 Stddev Latency(s): 0.0233159 Max latency(s): 0.237063 Min latency(s): 0.00628277 [ceph: root@dell-r640-056 /]# ceph df 2023-04-13T04:37:32.260+0000 7f98a5894700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-04-13T04:37:32.261+0000 7f98a5894700 -1 WARNING: the following dangerous and experimental features are enabled: crimson --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED TOTAL 1.5 TiB 1.4 TiB 19 GiB 19 GiB 1.25 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 897 KiB 4 897 KiB 0 465 GiB cephfs.cephfs.meta 2 32 4.6 KiB 44 4.6 KiB 0 465 GiB cephfs.cephfs.data 3 32 0 B 0 0 B 0 465 GiB .rgw.root 4 32 368 B 8 368 B 0 465 GiB default.rgw.log 5 32 7.2 KiB 418 7.2 KiB 0 465 GiB default.rgw.control 6 32 0 B 16 0 B 0 465 GiB default.rgw.meta 7 32 738 B 2 738 B 0 465 GiB re_pool_compress_snappy 8 32 0 B 0 0 B 0 465 GiB re_pool_compress_zlib 9 32 0 B 0 0 B 0 465 GiB re_pool_compress_zstd 11 32 0 B 0 0 B 0 465 GiB re_pool_no_compress 12 32 7.6 GiB 200.00k 7.6 GiB 0.54 465 GiB # rados --no-log-to-stderr -b 40KB -p re_pool_compress_snappy bench 500 write --no-cleanup --max-objects 100000 hints = 1 Maintaining 16 concurrent writes of 40960 bytes to objects of size 40960 for up to 500 seconds or 100000 objects Object prefix: benchmark_data_dell-r640-056.dsal.lab.eng.rd_314 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 307 291 11.3658 11.3672 0.0574587 0.0528637 2 16 587 571 11.1506 10.9375 0.0536615 0.055293 3 16 871 855 11.131 11.0938 0.0530275 0.0556187 4 16 1148 1132 11.0528 10.8203 0.0588173 0.056124 5 16 1444 1428 11.1544 11.5625 0.0440195 0.0557268 6 16 1737 1721 11.2026 11.4453 0.0299697 0.0554426 7 16 2027 2011 11.2202 11.3281 0.0472697 0.0554723 8 16 2314 2298 11.2188 11.2109 0.0639709 0.0554958 9 16 2607 2591 11.2437 11.4453 0.0780707 0.055343 10 16 2874 2858 11.1622 10.4297 0.0668183 0.0557896 11 16 3168 3152 11.1913 11.4844 0.0507199 0.0556421 12 16 3453 3437 11.1863 11.1328 0.0588823 0.0557396 13 16 3731 3715 11.1611 10.8594 0.0388154 0.0558615 14 16 4008 3992 11.1367 10.8203 0.0558481 0.0560106 15 16 4282 4266 11.1077 10.7031 0.0494121 0.0561426 16 16 4522 4506 10.9993 9.375 0.0539563 0.0566813 17 16 4763 4747 10.906 9.41406 0.0939461 0.0571875 18 16 5000 4984 10.8144 9.25781 0.0527782 0.0576787 19 16 5246 5230 10.7509 9.60938 0.113147 0.0580339 2023-04-13T04:41:13.939381+0000 min lat: 0.00732427 max lat: 0.168968 avg lat: 0.0582053 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 20 16 5507 5491 10.7231 10.1953 0.0742529 0.0582053 21 16 5759 5743 10.6811 9.84375 0.0205993 0.0584174 22 16 6000 5984 10.6234 9.41406 0.0431899 0.05872 23 16 6255 6239 10.5946 9.96094 0.0832311 0.0589103 24 16 6507 6491 10.5632 9.84375 0.049078 0.0590723 25 16 6754 6738 10.5266 9.64844 0.057048 0.0592781 26 16 6984 6968 10.4672 8.98438 0.030255 0.0596319 27 16 7232 7216 10.4383 9.6875 0.073647 0.0597735 28 16 7472 7456 10.4003 9.375 0.0447358 0.0599983 29 16 7728 7712 10.3864 10 0.0858013 0.0600709 30 16 8009 7993 10.406 10.9766 0.0472455 0.0600007 31 16 8302 8286 10.4395 11.4453 0.0584443 0.0597936 32 16 8583 8567 10.4562 10.9766 0.0557158 0.059692 33 16 8867 8851 10.4755 11.0938 0.0677574 0.0595941 34 16 9128 9112 10.4672 10.1953 0.0557886 0.0596194 35 16 9416 9400 10.4895 11.25 0.0516821 0.0595105 36 16 9707 9691 10.5139 11.3672 0.0361386 0.0594016 37 16 9981 9965 10.519 10.7031 0.034913 0.0593457 38 16 10252 10236 10.5207 10.5859 0.0672024 0.0593524 39 16 10529 10513 10.5283 10.8203 0.0495474 0.059311 2023-04-13T04:41:33.942409+0000 min lat: 0.00653868 max lat: 0.184556 avg lat: 0.0592654 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 40 16 10805 10789 10.5346 10.7812 0.0911341 0.0592654 41 16 11086 11070 10.5453 10.9766 0.0658177 0.0592165 42 16 11342 11326 10.5323 10 0.0863111 0.059291 43 16 11611 11595 10.5317 10.5078 0.04827 0.0592527 44 16 11884 11868 10.5347 10.6641 0.0597173 0.0592859 45 16 12125 12109 10.5097 9.41406 0.107223 0.0594106 46 16 12365 12349 10.485 9.375 0.0613916 0.0595587 47 16 12585 12569 10.4448 8.59375 0.103996 0.0597715 48 16 12820 12804 10.4184 9.17969 0.0966298 0.0599505 49 16 13058 13042 10.3955 9.29688 0.0931819 0.0600697 50 16 13277 13261 10.3586 8.55469 0.0629762 0.0602846 51 16 13511 13495 10.3347 9.14062 0.0888981 0.0604289 52 16 13741 13725 10.3087 8.98438 0.0521622 0.0605863 53 16 13968 13952 10.2815 8.86719 0.0532152 0.0607405 54 16 14199 14183 10.2581 9.02344 0.0563699 0.0608782 55 16 14424 14408 10.2314 8.78906 0.0546372 0.0610312 56 16 14656 14640 10.2105 9.0625 0.0724768 0.0611541 57 16 14871 14855 10.1787 8.39844 0.05905 0.0613482 58 16 15078 15062 10.1426 8.08594 0.0869856 0.0615732 59 16 15301 15285 10.1183 8.71094 0.105919 0.0616969 2023-04-13T04:41:53.945727+0000 min lat: 0.00653868 max lat: 0.209478 avg lat: 0.0618684 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 60 16 15521 15505 10.0929 8.59375 0.0500107 0.0618684 61 16 15725 15709 10.058 7.96875 0.0466885 0.0620847 62 16 15941 15925 10.0319 8.4375 0.0539646 0.0622544 63 16 16193 16177 10.0289 9.84375 0.0544166 0.0622723 64 16 16446 16430 10.0265 9.88281 0.0912195 0.0622917 65 16 16703 16687 10.0267 10.0391 0.0513326 0.0622912 66 16 16947 16931 10.0192 9.53125 0.0490937 0.0623457 67 16 17201 17185 10.0177 9.92188 0.0681913 0.0623464 68 16 17453 17437 10.0151 9.84375 0.0620186 0.0623605 69 16 17705 17689 10.0126 9.84375 0.0696956 0.062379 70 16 17956 17940 10.0096 9.80469 0.0778849 0.062408 71 16 18213 18197 10.01 10.0391 0.0824696 0.0623972 72 16 18468 18452 10.0093 9.96094 0.0822041 0.0623994 73 16 18732 18716 10.0135 10.3125 0.0655069 0.0623737 74 16 18997 18981 10.018 10.3516 0.0491589 0.0623592 75 16 19284 19268 10.0339 11.2109 0.0547634 0.0622502 76 16 19570 19554 10.0489 11.1719 0.0377353 0.0621618 77 16 19867 19851 10.069 11.6016 0.0616151 0.0620406 78 16 20150 20134 10.0816 11.0547 0.128128 0.0619635 79 16 20431 20415 10.0929 10.9766 0.0499245 0.0618877 2023-04-13T04:42:13.948633+0000 min lat: 0.00653868 max lat: 0.25169 avg lat: 0.0618586 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 80 16 20700 20684 10.0981 10.5078 0.0549242 0.0618586 81 16 20981 20965 10.1089 10.9766 0.0728879 0.0617951 82 16 21240 21224 10.109 10.1172 0.0369697 0.0617975 83 16 21513 21497 10.1157 10.6641 0.0603836 0.0617468 84 16 21791 21775 10.1245 10.8594 0.0848649 0.0617043 85 16 22058 22042 10.1281 10.4297 0.0535945 0.0616714 86 15 22336 22321 10.137 10.8984 0.0574718 0.0616271 87 16 22613 22597 10.1444 10.7812 0.0241732 0.0615822 88 16 22888 22872 10.1512 10.7422 0.0905408 0.0615408 89 16 23158 23142 10.1556 10.5469 0.0536937 0.0615083 90 16 23413 23397 10.1534 9.96094 0.0642597 0.0615182 91 16 23661 23645 10.1483 9.6875 0.0690572 0.0615528 92 16 23912 23896 10.1445 9.80469 0.0521785 0.0615756 93 16 24139 24123 10.1308 8.86719 0.0642338 0.0616546 94 16 24340 24324 10.1065 7.85156 0.0252187 0.0618016 95 16 24565 24549 10.0926 8.78906 0.086396 0.0618911 96 16 24782 24766 10.0758 8.47656 0.0876922 0.0619986 97 16 25007 24991 10.0625 8.78906 0.0520098 0.0620807 98 16 25214 25198 10.0423 8.08594 0.035514 0.0621881 99 16 25436 25420 10.0285 8.67188 0.0697511 0.0622893 2023-04-13T04:42:33.951782+0000 min lat: 0.00653868 max lat: 0.25169 avg lat: 0.0624156 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 100 16 25640 25624 10.0078 7.96875 0.0968589 0.0624156 101 16 25857 25841 9.99268 8.47656 0.0638954 0.0625197 102 16 26075 26059 9.97818 8.51562 0.0703838 0.0626106 103 16 26282 26266 9.95979 8.08594 0.0756721 0.0626962 104 16 26498 26482 9.94515 8.4375 0.104355 0.0628133 105 16 26707 26691 9.92817 8.16406 0.062172 0.0629233 106 16 26936 26920 9.91889 8.94531 0.0590071 0.0629818 107 16 27191 27175 9.91927 9.96094 0.100652 0.0629775 108 16 27453 27437 9.92218 10.2344 0.0466969 0.0629649 109 16 27720 27704 9.92683 10.4297 0.0526431 0.0629285 110 16 27997 27981 9.93494 10.8203 0.052642 0.0628732 111 16 28261 28245 9.93832 10.3125 0.0582324 0.0628591 112 16 28529 28513 9.94304 10.4688 0.0591 0.0628325 113 16 28806 28790 9.95079 10.8203 0.0738468 0.062778 114 16 29057 29041 9.9495 9.80469 0.0560033 0.0627907 115 16 29317 29301 9.95128 10.1562 0.0648513 0.0627667 116 15 29590 29575 9.95775 10.7031 0.042674 0.0627434 117 16 29860 29844 9.96244 10.5078 0.0842838 0.0627055 118 16 30126 30110 9.96606 10.3906 0.0684494 0.0626869 119 16 30379 30363 9.96535 9.88281 0.0874816 0.0626877 2023-04-13T04:42:53.954525+0000 min lat: 0.0064997 max lat: 0.264076 avg lat: 0.0626682 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 120 16 30646 30630 9.96921 10.4297 0.0680716 0.0626682 121 16 30860 30844 9.95589 8.35938 0.0668264 0.0627399 122 16 31078 31062 9.94407 8.51562 0.0860585 0.0628216 123 16 31319 31303 9.93975 9.41406 0.0523729 0.0628485 124 16 31562 31546 9.93613 9.49219 0.074899 0.0628713 125 16 31811 31795 9.93444 9.72656 0.0731229 0.0628858 126 16 32040 32024 9.92658 8.94531 0.106918 0.06293 127 16 32289 32273 9.925 9.72656 0.083493 0.0629501 128 16 32527 32511 9.92009 9.29688 0.0543291 0.0629776 129 15 32766 32751 9.91585 9.375 0.053345 0.0630067 130 16 32999 32983 9.90928 9.0625 0.0944766 0.0630395 131 16 33238 33222 9.90489 9.33594 0.0624035 0.0630722 132 16 33476 33460 9.90027 9.29688 0.080247 0.0631028 133 16 33717 33701 9.89661 9.41406 0.0456283 0.0631275 134 16 33971 33955 9.89678 9.92188 0.127787 0.0631311 135 16 34243 34227 9.90216 10.625 0.0338039 0.0630961 136 16 34546 34530 9.91637 11.8359 0.0579709 0.0630066 137 16 34856 34840 9.93236 12.1094 0.0529094 0.0629035 138 16 35128 35112 9.93736 10.625 0.0666495 0.0628668 139 16 35429 35413 9.95045 11.7578 0.0470977 0.0627872 2023-04-13T04:43:13.957550+0000 min lat: 0.00624755 max lat: 0.264076 avg lat: 0.0627373 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 140 16 35713 35697 9.95861 11.0938 0.0727729 0.0627373 141 16 35995 35979 9.96609 11.0156 0.0511951 0.06269 142 16 36290 36274 9.97704 11.5234 0.032265 0.0626172 143 16 36580 36564 9.98648 11.3281 0.0642103 0.062561 144 16 36856 36840 9.99199 10.7812 0.0558291 0.0625213 145 16 37144 37128 10.0007 11.25 0.0571381 0.0624766 146 16 37422 37406 10.0065 10.8594 0.0323395 0.0624394 147 16 37710 37694 10.015 11.25 0.102689 0.0623844 148 16 37982 37966 10.0191 10.625 0.0702717 0.0623554 149 16 38252 38236 10.0226 10.5469 0.0785171 0.0623322 150 16 38479 38463 10.0149 8.86719 0.0527645 0.0623846 151 16 38680 38664 10.0006 7.85156 0.0694127 0.0624609 152 16 38897 38881 9.99054 8.47656 0.0486419 0.0625331 153 16 39095 39079 9.97578 7.73438 0.089181 0.0626251 154 16 39277 39261 9.95716 7.10938 0.0977308 0.0627396 155 16 39470 39454 9.94155 7.53906 0.107489 0.0628387 156 16 39658 39642 9.92489 7.34375 0.106162 0.0629501 157 16 39841 39825 9.9072 7.14844 0.0832585 0.0630586 158 16 40030 40014 9.89122 7.38281 0.123184 0.0631572 159 16 40215 40199 9.87445 7.22656 0.114418 0.0632653 2023-04-13T04:43:33.960577+0000 min lat: 0.00624755 max lat: 0.264076 avg lat: 0.0633838 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 160 16 40391 40375 9.8557 6.875 0.098156 0.0633838 161 16 40577 40561 9.8396 7.26562 0.143923 0.0634903 162 16 40761 40745 9.82323 7.1875 0.0905407 0.0635962 163 16 40961 40945 9.81088 7.8125 0.0796331 0.0636724 164 16 41155 41139 9.79725 7.57812 0.0666428 0.0637663 165 16 41367 41351 9.78805 8.28125 0.0875172 0.0638262 166 16 41595 41579 9.78274 8.90625 0.055195 0.063861 167 16 41829 41813 9.77888 9.14062 0.0720849 0.0638905 168 16 42061 42045 9.77461 9.0625 0.0906455 0.0639179 169 16 42307 42291 9.77362 9.60938 0.0456371 0.0639202 170 16 42555 42539 9.7731 9.6875 0.0376911 0.063929 171 16 42797 42781 9.77122 9.45312 0.0468646 0.0639372 172 16 43036 43020 9.76869 9.33594 0.10876 0.0639588 173 16 43288 43272 9.76911 9.84375 0.0662404 0.0639559 174 16 43530 43514 9.76729 9.45312 0.0551162 0.0639684 175 16 43777 43761 9.7666 9.64844 0.0431009 0.0639735 176 16 44028 44012 9.76681 9.80469 0.0195891 0.0639729 177 16 44268 44252 9.76459 9.375 0.0568398 0.0639863 178 16 44498 44482 9.7602 8.98438 0.0808624 0.0640118 179 16 44733 44717 9.75695 9.17969 0.0428937 0.0640359 2023-04-13T04:43:53.963647+0000 min lat: 0.00624755 max lat: 0.264076 avg lat: 0.0640405 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 180 16 44979 44963 9.75613 9.60938 0.0472697 0.0640405 181 16 45218 45202 9.7538 9.33594 0.0870281 0.0640557 182 16 45466 45450 9.75342 9.6875 0.116243 0.0640606 183 16 45718 45702 9.75391 9.84375 0.0815813 0.064059 184 16 45950 45934 9.75015 9.0625 0.0711562 0.0640781 185 16 46196 46180 9.74938 9.60938 0.0973352 0.0640842 186 16 46434 46418 9.74694 9.29688 0.0679461 0.0641026 187 16 46683 46667 9.74682 9.72656 0.0682717 0.0641019 188 16 46923 46907 9.74483 9.375 0.0296864 0.0641117 189 16 47163 47147 9.74287 9.375 0.0505963 0.0641254 190 16 47399 47383 9.7401 9.21875 0.086111 0.0641479 191 16 47645 47629 9.73941 9.60938 0.0591314 0.0641471 192 16 47877 47861 9.73588 9.0625 0.0675228 0.0641712 193 16 48127 48111 9.73603 9.76562 0.0405405 0.0641722 194 16 48357 48341 9.73215 8.98438 0.0514322 0.0641947 195 16 48586 48570 9.7281 8.94531 0.072867 0.0642246 196 16 48839 48823 9.72889 9.88281 0.0171347 0.0642192 197 16 49090 49074 9.72927 9.80469 0.0500637 0.0642129 198 16 49356 49340 9.7326 10.3906 0.110436 0.0641966 199 16 49624 49608 9.73629 10.4688 0.0434785 0.0641724 2023-04-13T04:44:13.966545+0000 min lat: 0.00624755 max lat: 0.264076 avg lat: 0.0641421 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 200 16 49896 49880 9.74073 10.625 0.032717 0.0641421 201 16 50162 50146 9.74395 10.3906 0.0740198 0.0641233 202 16 50413 50397 9.74425 9.80469 0.0818806 0.0641188 203 16 50681 50665 9.74781 10.4688 0.0504578 0.0640956 204 16 50950 50934 9.75153 10.5078 0.0352748 0.0640757 205 16 51212 51196 9.75387 10.2344 0.0120801 0.0640574 206 16 51483 51467 9.75791 10.5859 0.058306 0.0640313 207 16 51740 51724 9.75926 10.0391 0.0659045 0.0640212 208 16 52000 51984 9.76116 10.1562 0.0781019 0.0640063 209 16 52272 52256 9.76528 10.625 0.0796731 0.0639822 210 16 52552 52536 9.77085 10.9375 0.0635316 0.0639463 211 16 52860 52844 9.78156 12.0312 0.0364383 0.0638774 212 16 53162 53146 9.79106 11.7969 0.0481811 0.0638135 213 16 53467 53451 9.80102 11.9141 0.0774414 0.0637494 214 16 53777 53761 9.81179 12.1094 0.0560928 0.0636821 215 16 54079 54063 9.82102 11.7969 0.0852196 0.0636227 216 16 54373 54357 9.82871 11.4844 0.0980532 0.063569 217 16 54679 54663 9.83849 11.9531 0.0551568 0.0635112 218 16 54980 54964 9.84729 11.7578 0.0196313 0.0634533 219 16 55281 55265 9.856 11.7578 0.0462865 0.0633969 2023-04-13T04:44:33.969619+0000 min lat: 0.00624755 max lat: 0.264076 avg lat: 0.0633448 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 220 16 55578 55562 9.86393 11.6016 0.0491219 0.0633448 221 16 55873 55857 9.87143 11.5234 0.0441072 0.0632951 222 16 56185 56169 9.88185 12.1875 0.0667003 0.0632304 223 16 56485 56469 9.89008 11.7188 0.0261063 0.0631748 224 16 56781 56765 9.89754 11.5625 0.0442076 0.0631262 225 16 57076 57060 9.90476 11.5234 0.0429922 0.0630804 226 16 57326 57310 9.90414 9.76562 0.0464857 0.0630823 227 16 57579 57563 9.90404 9.88281 0.0558088 0.0630839 228 16 57821 57805 9.90205 9.45312 0.0536385 0.0630988 229 16 58071 58055 9.90145 9.76562 0.0462671 0.0631051 230 16 58304 58288 9.89796 9.10156 0.102302 0.06313 231 16 58531 58515 9.8935 8.86719 0.0706839 0.0631543 232 16 58770 58754 9.89109 9.33594 0.0922224 0.0631671 233 16 59017 59001 9.89004 9.64844 0.133349 0.0631766 234 16 59250 59234 9.88666 9.10156 0.0614177 0.0631954 235 16 59496 59480 9.88547 9.60938 0.0865889 0.0632053 236 16 59742 59726 9.8843 9.60938 0.0595199 0.0632147 237 16 59985 59969 9.88264 9.49219 0.0541789 0.0632228 238 16 60226 60210 9.88066 9.41406 0.0557306 0.0632334 239 16 60486 60470 9.88181 10.1562 0.0785607 0.0632318 2023-04-13T04:44:53.972840+0000 min lat: 0.00624755 max lat: 0.264076 avg lat: 0.0632581 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 240 16 60713 60697 9.87757 8.86719 0.0769217 0.0632581 241 16 60943 60927 9.87387 8.98438 0.0363967 0.0632804 242 16 61192 61176 9.87325 9.72656 0.0660218 0.0632825 243 16 61449 61433 9.87393 10.0391 0.0630219 0.0632811 244 16 61703 61687 9.87412 9.92188 0.0716653 0.0632799 245 16 61961 61945 9.87495 10.0781 0.0806934 0.0632733 246 16 62226 62210 9.87688 10.3516 0.0823828 0.0632614 247 16 62490 62474 9.87864 10.3125 0.0491534 0.0632491 248 16 62752 62736 9.88007 10.2344 0.0944444 0.0632409 249 16 63020 63004 9.88242 10.4688 0.0613676 0.0632247 250 16 63275 63259 9.88273 9.96094 0.0482042 0.0632215 251 16 63546 63530 9.88553 10.5859 0.0863082 0.0632048 252 16 63809 63793 9.88706 10.2734 0.0527971 0.0631964 253 16 64070 64054 9.88827 10.1953 0.0671832 0.0631873 254 16 64327 64311 9.88886 10.0391 0.0848178 0.063181 255 16 64586 64570 9.88975 10.1172 0.0498954 0.0631802 256 15 64829 64814 9.88835 9.53125 0.0660369 0.0631864 257 16 65077 65061 9.88741 9.64844 0.0449425 0.0631927 258 16 65302 65286 9.88315 8.78906 0.0552475 0.0632202 259 16 65551 65535 9.88254 9.72656 0.0792835 0.0632258 2023-04-13T04:45:13.975476+0000 min lat: 0.00624755 max lat: 0.264076 avg lat: 0.0632484 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 260 16 65781 65765 9.87908 8.98438 0.0977995 0.0632484 261 16 66014 65998 9.8761 9.10156 0.104553 0.0632653 262 16 66243 66227 9.87254 8.94531 0.0948564 0.0632865 263 16 66489 66473 9.87153 9.60938 0.0662899 0.0632967 264 16 66719 66703 9.86816 8.98438 0.10416 0.0633153 265 16 66961 66945 9.86659 9.45312 0.0698017 0.0633255 266 16 67193 67177 9.86357 9.0625 0.119688 0.0633452 267 16 67437 67421 9.86232 9.53125 0.0725425 0.0633532 268 16 67672 67656 9.85977 9.17969 0.028594 0.0633701 269 16 67919 67903 9.85897 9.64844 0.0640698 0.0633772 270 16 68182 68166 9.8605 10.2734 0.0878324 0.063366 271 16 68411 68395 9.85712 8.94531 0.0585462 0.0633843 272 16 68645 68629 9.85448 9.14062 0.0764666 0.0634038 273 16 68897 68881 9.85444 9.84375 0.0659865 0.0634044 274 16 69106 69090 9.84826 8.16406 0.114751 0.0634429 275 16 69334 69318 9.84483 8.90625 0.0850696 0.063469 276 16 69549 69533 9.83958 8.39844 0.124704 0.0634968 277 16 69751 69735 9.83254 7.89062 0.0459833 0.0635422 278 16 69953 69937 9.82555 7.89062 0.0778083 0.0635896 279 16 70171 70155 9.82085 8.51562 0.087771 0.0636213 2023-04-13T04:45:33.978628+0000 min lat: 0.00624755 max lat: 0.264076 avg lat: 0.0636555 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 280 16 70384 70368 9.81549 8.32031 0.100843 0.0636555 281 16 70609 70593 9.81183 8.78906 0.101451 0.0636807 282 16 70822 70806 9.80654 8.32031 0.0471407 0.0637133 283 16 71047 71031 9.80294 8.78906 0.0437994 0.0637366 284 16 71265 71249 9.7984 8.51562 0.0820132 0.063766 285 16 71514 71498 9.79815 9.72656 0.0764992 0.0637706 286 16 71799 71783 9.80281 11.1328 0.0539942 0.0637425 287 16 72076 72060 9.80635 10.8203 0.0296461 0.0637166 288 16 72367 72351 9.81176 11.3672 0.0523744 0.0636842 289 16 72667 72651 9.81835 11.7188 0.0374744 0.0636415 290 15 72948 72933 9.82248 11.0156 0.0449695 0.0636133 291 16 73249 73233 9.82899 11.7188 0.0665039 0.063573 292 16 73548 73532 9.83532 11.6797 0.0671722 0.0635315 293 16 73849 73833 9.84187 11.7578 0.0338539 0.0634877 294 16 74154 74138 9.84892 11.9141 0.0335904 0.0634436 295 16 74460 74444 9.85605 11.9531 0.0528545 0.0633965 296 16 74766 74750 9.86312 11.9531 0.0362051 0.0633511 297 16 75068 75052 9.86963 11.7969 0.0512472 0.0633102 298 16 75360 75344 9.87478 11.4062 0.0800193 0.0632766 299 16 75670 75654 9.88225 12.1094 0.0716154 0.0632293 2023-04-13T04:45:53.981445+0000 min lat: 0.00624755 max lat: 0.264076 avg lat: 0.063202 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 300 16 75953 75937 9.88615 11.0547 0.0467762 0.063202 301 16 76230 76214 9.88925 10.8203 0.0504977 0.0631848 302 16 76494 76478 9.89064 10.3125 0.0540835 0.0631752 303 16 76773 76757 9.89397 10.8984 0.0348127 0.0631539 304 16 77037 77021 9.89534 10.3125 0.07904 0.0631473 305 16 77300 77284 9.89657 10.2734 0.0557756 0.0631365 306 16 77553 77537 9.89652 9.88281 0.0526972 0.0631361 307 16 77826 77810 9.89901 10.6641 0.120245 0.0631217 308 16 78090 78074 9.90035 10.3125 0.0714652 0.0631138 309 16 78344 78328 9.90041 9.92188 0.0886559 0.0631119 310 16 78600 78584 9.90073 10 0.0490466 0.0631083 311 16 78869 78853 9.90268 10.5078 0.0616911 0.0630973 312 16 79142 79126 9.90511 10.6641 0.0350741 0.0630827 313 16 79414 79398 9.90741 10.625 0.0598676 0.0630658 314 16 79671 79655 9.90782 10.0391 0.0760208 0.0630634 315 16 79958 79942 9.91195 11.2109 0.0523957 0.0630384 316 16 80250 80234 9.91667 11.4062 0.0523395 0.0630095 317 16 80557 80541 9.92321 11.9922 0.0590168 0.0629695 318 16 80833 80817 9.92591 10.7812 0.047837 0.0629509 319 16 81122 81106 9.93018 11.2891 0.0543067 0.0629218 2023-04-13T04:46:13.984633+0000 min lat: 0.00624755 max lat: 0.264076 avg lat: 0.0628906 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 320 16 81421 81405 9.93564 11.6797 0.0554743 0.0628906 321 16 81710 81694 9.93985 11.2891 0.0557154 0.0628621 322 16 81988 81972 9.9427 10.8594 0.054222 0.0628447 323 16 82275 82259 9.94662 11.2109 0.0636848 0.0628204 324 16 82576 82560 9.9522 11.7578 0.0257465 0.0627862 325 15 82866 82851 9.95655 11.3672 0.0598849 0.0627596 326 16 83150 83134 9.95991 11.0547 0.0556147 0.0627361 327 16 83443 83427 9.96445 11.4453 0.0632592 0.0627081 328 16 83729 83713 9.96813 11.1719 0.0464036 0.0626858 329 16 84001 83985 9.97012 10.625 0.0351641 0.0626699 330 16 84245 84229 9.96878 9.53125 0.0568977 0.0626775 331 16 84520 84504 9.97111 10.7422 0.0565855 0.062665 332 16 84776 84760 9.9712 10 0.0422305 0.0626666 333 16 85033 85017 9.9714 10.0391 0.0761761 0.0626649 334 16 85274 85258 9.96972 9.41406 0.0553696 0.0626717 335 16 85526 85510 9.96935 9.84375 0.0674597 0.0626757 336 16 85762 85746 9.96711 9.21875 0.0719888 0.0626888 337 16 86015 85999 9.96685 9.88281 0.0611129 0.0626912 338 16 86241 86225 9.96348 8.82812 0.092442 0.0627092 339 16 86496 86480 9.96347 9.96094 0.0705997 0.0627142 2023-04-13T04:46:33.987841+0000 min lat: 0.00624755 max lat: 0.264076 avg lat: 0.0627231 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 340 16 86739 86723 9.96208 9.49219 0.0593044 0.0627231 341 16 86980 86964 9.96047 9.41406 0.07033 0.0627306 342 16 87235 87219 9.96046 9.96094 0.0751722 0.0627328 343 16 87476 87460 9.95887 9.41406 0.02368 0.0627405 344 16 87724 87708 9.95807 9.6875 0.0886478 0.0627469 345 16 87984 87968 9.95864 10.1562 0.0445199 0.0627415 346 16 88206 88190 9.95492 8.67188 0.0815992 0.0627671 347 16 88427 88411 9.9511 8.63281 0.106183 0.0627878 348 16 88649 88633 9.94742 8.67188 0.058664 0.0628136 349 16 88894 88878 9.94634 9.57031 0.028643 0.0628207 350 16 89145 89129 9.94593 9.80469 0.0361643 0.0628247 351 16 89380 89364 9.94374 9.17969 0.0908233 0.0628355 352 16 89621 89605 9.94223 9.41406 0.0830529 0.0628481 353 16 89862 89846 9.94073 9.41406 0.058514 0.0628555 354 16 90103 90087 9.93924 9.41406 0.0444875 0.062866 355 16 90351 90335 9.93853 9.6875 0.0582593 0.0628699 356 16 90602 90586 9.93815 9.80469 0.0397087 0.0628738 357 16 90845 90829 9.9369 9.49219 0.0448686 0.0628812 358 16 91087 91071 9.93554 9.45312 0.11499 0.062889 359 16 91327 91311 9.93397 9.375 0.0471264 0.0628998 2023-04-13T04:46:53.991043+0000 min lat: 0.00624755 max lat: 0.264076 avg lat: 0.0629175 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 360 16 91551 91535 9.93068 8.75 0.0606573 0.0629175 361 16 91787 91771 9.92871 9.21875 0.0767404 0.0629334 362 16 92014 91998 9.92577 8.86719 0.0425213 0.0629509 363 16 92252 92236 9.92403 9.29688 0.118294 0.0629603 364 16 92472 92456 9.92038 8.59375 0.0952313 0.0629854 365 16 92697 92681 9.91727 8.78906 0.0966823 0.0630035 366 16 92926 92910 9.91461 8.94531 0.0518043 0.0630188 367 16 93159 93143 9.91239 9.10156 0.0877329 0.0630366 368 16 93392 93376 9.91019 9.10156 0.0980894 0.0630497 369 16 93626 93610 9.9081 9.14062 0.118602 0.0630624 370 16 93848 93832 9.90475 8.67188 0.0561881 0.0630853 371 16 94085 94069 9.90301 9.25781 0.0725164 0.0630968 372 16 94315 94299 9.90053 8.98438 0.0534841 0.0631128 373 16 94552 94536 9.89881 9.25781 0.0898539 0.0631245 374 16 94789 94773 9.89709 9.25781 0.0597903 0.0631358 375 16 95035 95019 9.89632 9.60938 0.0790532 0.0631391 376 16 95269 95253 9.89431 9.14062 0.065108 0.0631492 377 16 95498 95482 9.89178 8.94531 0.101835 0.0631664 378 16 95723 95707 9.88886 8.78906 0.047272 0.0631854 379 16 95977 95961 9.88895 9.92188 0.0594277 0.0631853 2023-04-13T04:47:13.993886+0000 min lat: 0.00624755 max lat: 0.264076 avg lat: 0.0631813 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 380 16 96235 96219 9.88944 10.0781 0.0667487 0.0631813 381 16 96494 96478 9.89003 10.1172 0.0589883 0.063181 382 16 96737 96721 9.88899 9.49219 0.0502008 0.0631865 383 16 96991 96975 9.88907 9.92188 0.0491254 0.0631865 384 16 97237 97221 9.88834 9.60938 0.0705035 0.0631897 385 16 97492 97476 9.88852 9.96094 0.0603205 0.0631885 386 16 97743 97727 9.8883 9.80469 0.0548527 0.0631915 387 16 97991 97975 9.88778 9.6875 0.0472599 0.0631924 388 16 98258 98242 9.88917 10.4297 0.0633717 0.0631877 389 16 98510 98494 9.88905 9.84375 0.0727465 0.0631859 390 16 98759 98743 9.88863 9.72656 0.0611736 0.0631881 391 16 99028 99012 9.89021 10.5078 0.0616892 0.0631786 392 16 99313 99297 9.89337 11.1328 0.072705 0.0631592 393 16 99571 99555 9.89384 10.0781 0.0552408 0.0631527 394 16 99816 99800 9.89302 9.57031 0.0697386 0.0631575 Total time run: 394.848 Total writes made: 100000 Write size: 40960 Object size: 40960 Bandwidth (MB/sec): 9.89306 Stddev Bandwidth: 1.07736 Max bandwidth (MB/sec): 12.1875 Min bandwidth (MB/sec): 6.875 Average IOPS: 253 Stddev IOPS: 27.5805 Max IOPS: 312 Min IOPS: 176 Average Latency(s): 0.0631611 Stddev Latency(s): 0.0230223 Max latency(s): 0.264076 Min latency(s): 0.00624755 [ceph: root@dell-r640-056 /]# ceph df 2023-04-13T04:47:42.844+0000 7f172770d700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-04-13T04:47:42.845+0000 7f172770d700 -1 WARNING: the following dangerous and experimental features are enabled: crimson --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED TOTAL 1.5 TiB 1.4 TiB 31 GiB 31 GiB 2.08 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 897 KiB 4 897 KiB 0 459 GiB cephfs.cephfs.meta 2 32 4.6 KiB 44 4.6 KiB 0 459 GiB cephfs.cephfs.data 3 32 0 B 0 0 B 0 459 GiB .rgw.root 4 32 368 B 8 368 B 0 459 GiB default.rgw.log 5 32 7.2 KiB 418 7.2 KiB 0 459 GiB default.rgw.control 6 32 0 B 16 0 B 0 459 GiB default.rgw.meta 7 32 738 B 2 738 B 0 459 GiB re_pool_compress_snappy 8 32 7.6 GiB 200.00k 7.6 GiB 0.55 459 GiB re_pool_compress_zlib 9 32 0 B 0 0 B 0 459 GiB re_pool_compress_zstd 11 32 0 B 0 0 B 0 459 GiB re_pool_no_compress 12 32 7.6 GiB 200.00k 7.6 GiB 0.55 459 GiB # rados --no-log-to-stderr -b 40KB -p re_pool_compress_zlib bench 500 write --no-cleanup --max-objects 100000 hints = 1 Maintaining 16 concurrent writes of 40960 bytes to objects of size 40960 for up to 500 seconds or 100000 objects Object prefix: benchmark_data_dell-r640-056.dsal.lab.eng.rd_358 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 226 210 8.2025 8.20312 0.0685649 0.0737091 2 16 450 434 8.47587 8.75 0.0216442 0.0722498 3 16 674 658 8.56674 8.75 0.0979723 0.0720738 4 16 912 896 8.74887 9.29688 0.0911455 0.0708235 5 16 1137 1121 8.75662 8.78906 0.0818079 0.0709547 6 16 1361 1345 8.75529 8.75 0.0783018 0.0709873 7 16 1621 1605 8.95518 10.1562 0.013252 0.0694632 8 16 1875 1859 9.07578 9.92188 0.0613364 0.0685333 9 16 2117 2101 9.11756 9.45312 0.0751735 0.0682693 10 16 2355 2339 9.13534 9.29688 0.0579234 0.0682349 11 16 2588 2572 9.13212 9.10156 0.101038 0.0681959 12 16 2842 2826 9.19777 9.92188 0.0751782 0.067719 13 16 3093 3077 9.24434 9.80469 0.0525305 0.0674911 14 16 3323 3307 9.22566 8.98438 0.0668989 0.0675546 15 16 3577 3561 9.27196 9.92188 0.0500397 0.0672477 16 16 3819 3803 9.28318 9.45312 0.0621786 0.0671626 17 16 4061 4045 9.29308 9.45312 0.0509133 0.0670827 18 16 4300 4284 9.29541 9.33594 0.0272248 0.0671216 19 15 4546 4531 9.3139 9.64844 0.0967184 0.0669904 2023-04-13T04:49:22.469186+0000 min lat: 0.00823295 max lat: 0.207886 avg lat: 0.0668146 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 20 16 4800 4784 9.34228 9.88281 0.0475042 0.0668146 21 16 5055 5039 9.37167 9.96094 0.0480189 0.0665356 22 16 5298 5282 9.37711 9.49219 0.0870943 0.0665113 23 16 5545 5529 9.38884 9.64844 0.0486589 0.0664828 24 16 5780 5764 9.38006 9.17969 0.046074 0.0665306 25 16 6020 6004 9.37979 9.375 0.0521704 0.0665168 26 16 6249 6233 9.36302 8.94531 0.062339 0.0666341 27 16 6488 6472 9.36196 9.33594 0.0574283 0.0666824 28 16 6714 6698 9.34284 8.82812 0.0934674 0.0667445 29 16 6958 6942 9.34928 9.53125 0.0464233 0.0667431 30 16 7194 7178 9.34488 9.21875 0.0539674 0.0667913 31 16 7427 7411 9.33699 9.10156 0.071703 0.0668524 32 16 7670 7654 9.34178 9.49219 0.0654345 0.0668453 33 16 7901 7885 9.3321 9.02344 0.0886101 0.0668619 34 16 8131 8115 9.32183 8.98438 0.0494188 0.0669454 35 16 8377 8361 9.33 9.60938 0.107267 0.0669234 36 16 8620 8604 9.33448 9.49219 0.0594993 0.066886 37 16 8870 8854 9.3461 9.76562 0.0567901 0.0668079 38 16 9132 9116 9.36944 10.2344 0.0746143 0.0666485 39 16 9396 9380 9.39358 10.3125 0.0727083 0.0664465 2023-04-13T04:49:42.472190+0000 min lat: 0.00609258 max lat: 0.208147 avg lat: 0.0664492 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 40 16 9634 9618 9.39114 9.29688 0.100724 0.0664492 41 16 9875 9859 9.39166 9.41406 0.0533716 0.0664728 42 16 10115 10099 9.39122 9.375 0.0747724 0.0664834 43 16 10361 10345 9.39625 9.60938 0.0821603 0.0664458 44 16 10602 10586 9.39661 9.41406 0.0545859 0.0664406 45 16 10838 10822 9.39262 9.21875 0.0592263 0.0664839 46 16 11073 11057 9.38797 9.17969 0.0841184 0.0665121 47 16 11320 11304 9.39348 9.64844 0.0573814 0.0664822 48 16 11556 11540 9.38981 9.21875 0.0264111 0.0664812 49 16 11798 11782 9.39107 9.45312 0.0403542 0.0665025 50 16 12018 12002 9.37509 8.59375 0.0755431 0.0666137 51 16 12261 12245 9.37736 9.49219 0.0885622 0.0665953 52 16 12491 12475 9.36977 8.98438 0.0735784 0.0666389 53 16 12697 12681 9.34479 8.04688 0.0852687 0.0668174 54 16 12902 12886 9.32001 8.00781 0.0900708 0.067011 55 16 13116 13100 9.30251 8.35938 0.0618945 0.0671149 56 16 13337 13321 9.29052 8.63281 0.0819431 0.067205 57 16 13559 13543 9.27965 8.67188 0.0608227 0.0672882 58 16 13785 13769 9.27184 8.82812 0.0625679 0.0673512 59 16 14018 14002 9.26893 9.10156 0.0275134 0.0673631 2023-04-13T04:50:02.475508+0000 min lat: 0.00609258 max lat: 0.208147 avg lat: 0.067398 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 60 16 14251 14235 9.26612 9.10156 0.0811621 0.067398 61 16 14479 14463 9.26019 8.90625 0.0731817 0.0674483 62 16 14703 14687 9.25193 8.75 0.0667234 0.0674991 63 16 14929 14913 9.24518 8.82812 0.0641485 0.0675519 64 16 15161 15145 9.2423 9.0625 0.0404973 0.0675668 65 16 15394 15378 9.24012 9.10156 0.0524184 0.0676017 66 16 15616 15600 9.2315 8.67188 0.0546193 0.0676597 67 16 15875 15859 9.24469 10.1172 0.0766365 0.0675644 68 16 16129 16113 9.25463 9.92188 0.057821 0.0674869 69 16 16390 16374 9.26823 10.1953 0.0305026 0.0673922 70 16 16650 16634 9.28089 10.1562 0.0929356 0.0672996 71 16 16907 16891 9.29155 10.0391 0.0178655 0.067228 72 16 17163 17147 9.30137 10 0.0363256 0.0671549 73 16 17409 17393 9.30558 9.60938 0.087245 0.067111 74 16 17656 17640 9.31019 9.64844 0.0156375 0.0670899 75 16 17899 17883 9.31259 9.49219 0.0733615 0.0670655 76 16 18156 18140 9.32212 10.0391 0.10065 0.0670066 77 16 18411 18395 9.3304 9.96094 0.0388152 0.0669489 78 16 18663 18647 9.33696 9.84375 0.0799738 0.0669045 79 16 18911 18895 9.34138 9.6875 0.0612723 0.0668682 2023-04-13T04:50:22.478597+0000 min lat: 0.00609258 max lat: 0.208147 avg lat: 0.0668551 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 80 16 19154 19138 9.34326 9.49219 0.0266544 0.0668551 81 16 19397 19381 9.34508 9.49219 0.103838 0.0668286 82 16 19631 19615 9.34257 9.14062 0.0703425 0.0668648 83 16 19872 19856 9.34341 9.41406 0.0583483 0.066862 84 16 20120 20104 9.34749 9.6875 0.0931678 0.0668342 85 16 20347 20331 9.34182 8.86719 0.0210845 0.0668538 86 16 20604 20588 9.34992 10.0391 0.0783883 0.0668144 87 16 20854 20838 9.35467 9.76562 0.0280881 0.0667775 88 16 21109 21093 9.36154 9.96094 0.103336 0.0667274 89 16 21358 21342 9.36563 9.72656 0.0516844 0.0666843 90 16 21611 21595 9.37136 9.88281 0.0660757 0.0666574 91 16 21859 21843 9.37481 9.6875 0.0908693 0.0666252 92 16 22111 22095 9.37989 9.84375 0.0671247 0.0665986 93 16 22373 22357 9.38907 10.2344 0.0588248 0.0665342 94 16 22626 22610 9.39431 9.88281 0.0873143 0.0664969 95 16 22880 22864 9.39985 9.92188 0.0609601 0.0664561 96 16 23138 23122 9.40691 10.0781 0.0761544 0.0664077 97 16 23390 23374 9.4114 9.84375 0.0458934 0.0663756 98 16 23634 23618 9.41261 9.53125 0.061633 0.0663658 99 16 23876 23860 9.413 9.45312 0.04028 0.0663676 2023-04-13T04:50:42.481518+0000 min lat: 0.00609258 max lat: 0.208147 avg lat: 0.0663327 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 100 16 24128 24112 9.41729 9.84375 0.0808062 0.0663327 101 16 24367 24351 9.41647 9.33594 0.038409 0.0663399 102 16 24603 24587 9.41453 9.21875 0.0724896 0.0663509 103 16 24856 24840 9.41906 9.88281 0.0467206 0.0663256 104 16 25085 25069 9.41449 8.94531 0.0787992 0.0663539 105 16 25312 25296 9.40927 8.86719 0.0525191 0.0663847 106 16 25538 25522 9.40378 8.82812 0.109406 0.0664315 107 16 25778 25762 9.40351 9.375 0.0353749 0.0664366 108 16 26014 25998 9.40179 9.21875 0.0600836 0.0664404 109 16 26258 26242 9.40297 9.53125 0.102428 0.0664378 110 16 26488 26472 9.39914 8.98438 0.0813448 0.0664664 111 16 26727 26711 9.39856 9.33594 0.0395832 0.066468 112 16 26973 26957 9.40043 9.60938 0.0679239 0.0664567 113 16 27207 27191 9.39812 9.14062 0.0712888 0.0664654 114 16 27437 27421 9.39448 8.98438 0.0828985 0.0664941 115 16 27669 27653 9.39158 9.0625 0.0623605 0.0665266 116 16 27901 27885 9.38873 9.0625 0.0548933 0.0665351 117 16 28142 28126 9.38893 9.41406 0.0960319 0.0665344 118 16 28381 28365 9.38848 9.33594 0.100887 0.066547 119 16 28622 28606 9.38868 9.41406 0.0488356 0.0665424 2023-04-13T04:51:02.484221+0000 min lat: 0.00609258 max lat: 0.208147 avg lat: 0.0665341 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 120 16 28865 28849 9.38953 9.49219 0.0563405 0.0665341 121 16 29110 29094 9.39101 9.57031 0.116839 0.0665181 122 16 29344 29328 9.38895 9.14062 0.0599572 0.0665309 123 16 29595 29579 9.39232 9.80469 0.027594 0.0665141 124 16 29827 29811 9.38964 9.0625 0.0561064 0.0665334 125 16 30068 30052 9.38983 9.41406 0.0243338 0.0665309 126 16 30306 30290 9.38908 9.29688 0.0614081 0.0665346 127 16 30527 30511 9.38311 8.63281 0.0837234 0.0665818 128 16 30772 30756 9.38456 9.57031 0.0638654 0.0665725 129 16 31024 31008 9.38811 9.84375 0.0512297 0.0665368 130 16 31276 31260 9.3916 9.84375 0.06107 0.0665105 131 16 31538 31522 9.39802 10.2344 0.0268654 0.0664738 132 16 31800 31784 9.40435 10.2344 0.0301548 0.0664378 133 16 32048 32032 9.40647 9.6875 0.069546 0.0664146 134 16 32300 32284 9.40972 9.84375 0.0541141 0.0663974 135 16 32551 32535 9.41263 9.80469 0.0673721 0.0663755 136 16 32801 32785 9.41521 9.76562 0.0765491 0.0663533 137 16 33056 33040 9.41918 9.96094 0.0330727 0.0663302 138 16 33306 33290 9.42168 9.76562 0.0791174 0.0663074 139 16 33559 33543 9.42498 9.88281 0.0294161 0.0662835 2023-04-13T04:51:22.487445+0000 min lat: 0.00609258 max lat: 0.208147 avg lat: 0.0662655 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 140 16 33814 33798 9.4288 9.96094 0.0526078 0.0662655 141 16 34071 34055 9.43312 10.0391 0.0493036 0.0662312 142 16 34322 34306 9.43573 9.80469 0.0576888 0.066213 143 16 34577 34561 9.43939 9.96094 0.0909741 0.0661853 144 16 34838 34822 9.44462 10.1953 0.0651171 0.0661507 145 16 35086 35070 9.44629 9.6875 0.0548243 0.0661392 146 16 35333 35317 9.44766 9.64844 0.0541606 0.0661319 147 16 35580 35564 9.44901 9.64844 0.106927 0.0661191 148 16 35836 35820 9.45272 10 0.122017 0.0660968 149 16 36099 36083 9.45822 10.2734 0.0494031 0.0660554 150 16 36360 36344 9.46312 10.1953 0.0785846 0.0660224 151 16 36614 36598 9.46615 9.92188 0.0689865 0.0659991 152 16 36864 36848 9.46811 9.76562 0.111302 0.065986 153 16 37116 37100 9.47056 9.84375 0.0556018 0.0659674 154 16 37366 37350 9.47246 9.76562 0.0734531 0.0659582 155 16 37625 37609 9.47661 10.1172 0.0490097 0.0659307 156 16 37879 37863 9.47946 9.92188 0.0517293 0.0659074 157 16 38139 38123 9.48376 10.1562 0.0635739 0.0658779 158 16 38377 38361 9.48256 9.29688 0.0297179 0.0658845 159 16 38624 38608 9.4836 9.64844 0.0612389 0.0658807 2023-04-13T04:51:42.490731+0000 min lat: 0.00609258 max lat: 0.208147 avg lat: 0.0658566 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 160 16 38878 38862 9.48633 9.92188 0.0628764 0.0658566 161 16 39135 39119 9.48975 10.0391 0.0646808 0.0658351 162 16 39384 39368 9.4912 9.72656 0.0252055 0.0658258 163 16 39641 39625 9.49456 10.0391 0.0657094 0.0658048 164 16 39889 39873 9.49572 9.6875 0.0538389 0.0657912 165 16 40151 40135 9.50019 10.2344 0.0542449 0.0657712 166 16 40404 40388 9.50249 9.88281 0.0708651 0.0657498 167 16 40647 40631 9.50242 9.49219 0.0694328 0.0657484 168 16 40902 40886 9.50514 9.96094 0.0116044 0.0657333 169 16 41152 41136 9.50667 9.76562 0.0944381 0.0657187 170 16 41409 41393 9.50979 10.0391 0.0381478 0.0657027 171 16 41659 41643 9.51128 9.76562 0.0765461 0.0656861 172 16 41887 41871 9.50775 8.90625 0.0556794 0.0657122 173 16 42139 42123 9.50968 9.84375 0.0825104 0.0656991 174 16 42396 42380 9.51272 10.0391 0.0548403 0.0656803 175 16 42616 42600 9.50746 8.59375 0.0655186 0.0657096 176 16 42817 42801 9.49804 7.85156 0.0430055 0.0657726 177 16 43029 43013 9.49116 8.28125 0.0559691 0.0658187 178 16 43235 43219 9.48304 8.04688 0.0543335 0.0658805 179 16 43446 43430 9.4761 8.24219 0.0559509 0.0659294 2023-04-13T04:52:02.493953+0000 min lat: 0.00609258 max lat: 0.208147 avg lat: 0.0659759 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 180 16 43651 43635 9.46793 8.00781 0.0931529 0.0659759 181 16 43866 43850 9.46201 8.39844 0.107103 0.0660278 182 16 44074 44058 9.45466 8.125 0.0995421 0.0660837 183 16 44283 44267 9.44761 8.16406 0.0816789 0.0661306 184 16 44492 44476 9.44062 8.16406 0.0960536 0.0661821 185 16 44708 44692 9.43519 8.4375 0.0607406 0.0662219 186 16 44911 44895 9.4271 7.92969 0.0656621 0.0662721 187 16 45133 45117 9.42305 8.67188 0.107077 0.0663025 188 16 45353 45337 9.41863 8.59375 0.085747 0.066338 189 16 45571 45555 9.41385 8.51562 0.0671118 0.0663715 190 16 45793 45777 9.40994 8.67188 0.101721 0.0663933 191 16 46028 46012 9.40872 9.17969 0.0562795 0.0664029 192 16 46275 46259 9.40997 9.64844 0.0697634 0.066399 193 16 46506 46490 9.40796 9.02344 0.128068 0.0664095 194 16 46732 46716 9.40496 8.82812 0.07176 0.0664311 195 16 46972 46956 9.4048 9.375 0.0659331 0.0664297 196 16 47212 47196 9.40464 9.375 0.0714921 0.0664363 197 16 47440 47424 9.4021 8.90625 0.0535439 0.0664546 198 16 47680 47664 9.40196 9.375 0.0641332 0.0664529 199 16 47912 47896 9.40025 9.0625 0.070708 0.0664654 2023-04-13T04:52:22.496802+0000 min lat: 0.00609258 max lat: 0.208147 avg lat: 0.0664782 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 200 16 48145 48129 9.39875 9.10156 0.0460042 0.0664782 201 16 48371 48355 9.3959 8.82812 0.0635713 0.0664982 202 16 48608 48592 9.39521 9.25781 0.0677377 0.0665019 203 16 48860 48844 9.39742 9.84375 0.0484828 0.066487 204 16 49087 49071 9.39481 8.86719 0.0703523 0.0665015 205 16 49339 49323 9.39699 9.84375 0.0627347 0.0664903 206 16 49568 49552 9.39479 8.94531 0.0812168 0.0665005 207 16 49808 49792 9.39469 9.375 0.0907508 0.0665048 208 16 50056 50040 9.39609 9.6875 0.0926987 0.0664943 209 16 50284 50268 9.39374 8.90625 0.0901168 0.0665096 210 16 50525 50509 9.39383 9.41406 0.0592358 0.0665137 211 16 50760 50744 9.3928 9.17969 0.0642166 0.0665152 212 16 50993 50977 9.39143 9.10156 0.0495543 0.0665284 213 16 51236 51220 9.39189 9.49219 0.0590005 0.0665266 214 16 51460 51444 9.38889 8.75 0.0741634 0.0665436 215 16 51697 51681 9.38827 9.25781 0.081925 0.0665503 216 16 51940 51924 9.38875 9.49219 0.0464893 0.0665497 217 16 52178 52162 9.38832 9.29688 0.0507034 0.0665527 218 16 52420 52404 9.38861 9.45312 0.10063 0.0665516 219 16 52649 52633 9.38658 8.94531 0.102275 0.0665615 2023-04-13T04:52:42.499876+0000 min lat: 0.00609258 max lat: 0.208147 avg lat: 0.066567 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 220 16 52891 52875 9.38687 9.45312 0.0492538 0.066567 221 16 53142 53126 9.38876 9.80469 0.0548719 0.0665505 222 16 53401 53385 9.39203 10.1172 0.0631548 0.066524 223 16 53661 53645 9.39545 10.1562 0.0501081 0.0665015 224 16 53919 53903 9.39849 10.0781 0.0851561 0.0664797 225 16 54173 54157 9.40081 9.92188 0.069015 0.0664632 226 16 54425 54409 9.40276 9.84375 0.057002 0.0664537 227 16 54685 54669 9.40608 10.1562 0.167822 0.0664264 228 16 54946 54930 9.40953 10.1953 0.0491825 0.0664028 229 16 55202 55186 9.41211 10 0.0561838 0.0663896 230 16 55451 55435 9.41347 9.72656 0.0395951 0.0663755 231 16 55688 55672 9.41279 9.25781 0.0546183 0.0663749 232 16 55926 55910 9.41228 9.29688 0.0800308 0.066386 233 16 56196 56180 9.41715 10.5469 0.0287504 0.0663494 234 16 56465 56449 9.4218 10.5078 0.0474444 0.0663158 235 16 56738 56722 9.42708 10.6641 0.0417754 0.0662785 236 16 56988 56972 9.42851 9.76562 0.0392204 0.0662692 237 16 57241 57225 9.43042 9.88281 0.0776224 0.0662587 238 16 57486 57470 9.431 9.57031 0.0758565 0.0662495 239 16 57742 57726 9.43338 10 0.102165 0.066235 2023-04-13T04:53:02.502725+0000 min lat: 0.00609258 max lat: 0.208147 avg lat: 0.0662173 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 240 16 57998 57982 9.43573 10 0.0475091 0.0662173 241 16 58257 58241 9.43855 10.1172 0.065022 0.0662008 242 16 58496 58480 9.43812 9.33594 0.051999 0.0662017 243 16 58753 58737 9.44059 10.0391 0.0581186 0.0661833 244 16 58998 58982 9.44111 9.57031 0.0636307 0.0661783 245 16 59252 59236 9.44307 9.92188 0.0549637 0.0661652 246 16 59506 59490 9.44501 9.92188 0.0680759 0.0661533 247 16 59750 59734 9.44535 9.53125 0.0667001 0.0661525 248 16 59977 59961 9.44302 8.86719 0.0904863 0.0661653 249 16 60204 60188 9.4407 8.86719 0.0493645 0.0661811 250 16 60431 60415 9.4384 8.86719 0.0593782 0.0661993 251 16 60685 60669 9.44032 9.92188 0.0756801 0.0661864 252 16 60947 60931 9.44346 10.2344 0.0588639 0.0661662 253 16 61194 61178 9.44427 9.64844 0.0585553 0.0661576 254 16 61446 61430 9.44584 9.84375 0.0673136 0.0661501 255 16 61695 61679 9.44693 9.72656 0.0908427 0.0661416 256 16 61936 61920 9.4468 9.41406 0.11988 0.0661412 257 16 62187 62171 9.44819 9.80469 0.0789619 0.0661315 258 16 62434 62418 9.44896 9.64844 0.02619 0.0661272 259 16 62673 62657 9.44852 9.33594 0.0482341 0.0661255 2023-04-13T04:53:22.505760+0000 min lat: 0.00609258 max lat: 0.208147 avg lat: 0.0661097 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 260 16 62929 62913 9.45063 10 0.0467921 0.0661097 261 16 63190 63174 9.45348 10.1953 0.0478673 0.0660932 262 16 63426 63410 9.45257 9.21875 0.0570092 0.066099 263 16 63676 63660 9.45376 9.76562 0.0412521 0.0660907 264 16 63941 63925 9.45716 10.3516 0.0242972 0.0660662 265 16 64200 64184 9.45964 10.1172 0.0642421 0.0660501 266 16 64463 64447 9.46269 10.2734 0.0511145 0.0660249 267 16 64730 64714 9.46631 10.4297 0.133584 0.0660052 268 16 64985 64969 9.46815 9.96094 0.0487482 0.0659924 269 16 65246 65230 9.47085 10.1953 0.0592777 0.0659766 270 16 65507 65491 9.47353 10.1953 0.0491554 0.0659558 271 16 65783 65767 9.47834 10.7812 0.0727778 0.0659239 272 16 66051 66035 9.48198 10.4688 0.0721882 0.0658988 273 16 66307 66291 9.48387 10 0.0502033 0.0658817 274 16 66566 66550 9.48618 10.1172 0.0359866 0.0658683 275 16 66827 66811 9.48875 10.1953 0.0546764 0.0658476 276 16 67096 67080 9.49244 10.5078 0.0561477 0.0658257 277 16 67376 67360 9.49765 10.9375 0.0543013 0.0657864 278 16 67639 67623 9.50043 10.2734 0.0559063 0.0657694 279 16 67887 67871 9.5011 9.6875 0.0542277 0.0657614 2023-04-13T04:53:42.508817+0000 min lat: 0.00609258 max lat: 0.208147 avg lat: 0.0657566 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 280 16 68137 68121 9.50204 9.76562 0.0587023 0.0657566 281 16 68378 68362 9.50172 9.41406 0.0445454 0.0657624 282 16 68625 68609 9.50224 9.64844 0.0580916 0.0657585 283 16 68873 68857 9.50289 9.6875 0.0607762 0.0657499 284 16 69128 69112 9.50449 9.96094 0.0508337 0.0657418 285 16 69377 69361 9.50527 9.72656 0.0828177 0.0657348 286 16 69623 69607 9.50563 9.60938 0.0473511 0.065733 287 16 69860 69844 9.50476 9.25781 0.0973725 0.0657397 288 16 70107 70091 9.50525 9.64844 0.0593015 0.0657362 289 16 70353 70337 9.50561 9.60938 0.0425485 0.0657355 290 16 70589 70573 9.50461 9.21875 0.0718065 0.065738 291 16 70834 70818 9.50483 9.57031 0.114939 0.0657359 292 16 71105 71089 9.50853 10.5859 0.066948 0.0657127 293 16 71400 71384 9.5154 11.5234 0.0512469 0.0656663 294 16 71703 71687 9.52329 11.8359 0.0471868 0.0656121 295 16 72012 71996 9.53192 12.0703 0.0549399 0.0655547 296 16 72327 72311 9.54128 12.3047 0.0512014 0.0654887 297 16 72637 72621 9.54992 12.1094 0.0458083 0.0654311 298 16 72940 72924 9.55758 11.8359 0.0498225 0.0653745 299 16 73264 73248 9.56794 12.6562 0.062813 0.0653071 2023-04-13T04:54:02.511681+0000 min lat: 0.00609258 max lat: 0.208147 avg lat: 0.0652559 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 300 16 73567 73551 9.5755 11.8359 0.0216545 0.0652559 301 16 73882 73866 9.58456 12.3047 0.0378149 0.0651944 302 16 74190 74174 9.59266 12.0312 0.0599039 0.0651385 303 16 74495 74479 9.60031 11.9141 0.0848404 0.0650868 304 16 74812 74796 9.60946 12.3828 0.0460561 0.0650224 305 16 75116 75100 9.61688 11.875 0.0388859 0.0649734 306 16 75416 75400 9.62374 11.7188 0.0506751 0.0649274 307 16 75696 75680 9.62802 10.9375 0.0407647 0.0648982 308 16 75947 75931 9.62859 9.80469 0.0689715 0.0648954 309 16 76206 76190 9.63016 10.1172 0.0978874 0.0648821 310 16 76465 76449 9.63173 10.1172 0.0341606 0.0648759 311 16 76720 76704 9.63278 9.96094 0.0599198 0.0648636 312 16 76983 76967 9.63483 10.2734 0.0544892 0.0648534 313 16 77235 77219 9.63549 9.84375 0.0735394 0.0648485 314 16 77489 77473 9.6364 9.92188 0.052359 0.0648425 315 16 77749 77733 9.63805 10.1562 0.0236616 0.0648287 316 16 78006 77990 9.63931 10.0391 0.030649 0.0648241 317 16 78259 78243 9.64008 9.88281 0.0684182 0.0648164 318 16 78511 78495 9.64071 9.84375 0.059446 0.0648107 319 16 78770 78754 9.6422 10.1172 0.0376627 0.064803 2023-04-13T04:54:22.514578+0000 min lat: 0.00609258 max lat: 0.208147 avg lat: 0.0648021 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 320 16 79019 79003 9.64246 9.72656 0.0421884 0.0648021 321 16 79272 79256 9.6432 9.88281 0.0675018 0.0647954 322 16 79532 79516 9.64479 10.1562 0.0695255 0.0647863 323 16 79780 79764 9.64491 9.6875 0.0659587 0.0647817 324 16 80024 80008 9.64456 9.53125 0.0353376 0.0647831 325 16 80263 80247 9.6436 9.33594 0.0599849 0.0647926 326 16 80505 80489 9.64302 9.45312 0.0828568 0.0647966 327 16 80750 80734 9.64279 9.57031 0.063849 0.0647962 328 16 81001 80985 9.64328 9.80469 0.0523267 0.0647958 329 16 81244 81228 9.64282 9.49219 0.0571026 0.0647992 330 16 81483 81467 9.64188 9.33594 0.0520064 0.0648029 331 16 81726 81710 9.64143 9.49219 0.0670454 0.064809 332 16 81961 81945 9.64003 9.17969 0.0562956 0.064815 333 16 82204 82188 9.63959 9.49219 0.105772 0.0648221 334 16 82447 82431 9.63914 9.49219 0.0682384 0.0648242 335 16 82687 82671 9.63835 9.375 0.0558389 0.0648296 336 16 82940 82924 9.63907 9.88281 0.0930512 0.064827 337 16 83185 83169 9.63886 9.57031 0.077192 0.0648246 338 16 83449 83433 9.64085 10.3125 0.0519897 0.0648144 339 16 83720 83704 9.64363 10.5859 0.0605243 0.0647925 2023-04-13T04:54:42.517581+0000 min lat: 0.00609258 max lat: 0.208147 avg lat: 0.064782 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 340 16 83984 83968 9.6456 10.3125 0.0515293 0.064782 341 16 84246 84230 9.64732 10.2344 0.077165 0.0647703 342 16 84504 84488 9.64857 10.0781 0.0729266 0.06476 343 16 84769 84753 9.65062 10.3516 0.0628382 0.0647486 344 16 85032 85016 9.65242 10.2734 0.0653638 0.064736 345 16 85304 85288 9.65524 10.625 0.0279609 0.0647177 346 16 85572 85556 9.65759 10.4688 0.0547784 0.0647022 347 16 85847 85831 9.66071 10.7422 0.0417702 0.064678 348 16 86119 86103 9.66348 10.625 0.0891804 0.0646627 349 16 86379 86363 9.66488 10.1562 0.0556997 0.0646502 350 16 86637 86621 9.66606 10.0781 0.0747106 0.0646437 351 16 86900 86884 9.66779 10.2734 0.0620478 0.0646279 352 16 87136 87120 9.66651 9.21875 0.0668434 0.0646401 353 16 87348 87332 9.66258 8.28125 0.042302 0.0646672 354 16 87554 87538 9.65801 8.04688 0.0459135 0.0646921 355 16 87777 87761 9.65534 8.71094 0.0646967 0.0647122 356 16 88000 87984 9.65268 8.71094 0.0979247 0.0647324 357 16 88219 88203 9.6496 8.55469 0.0562438 0.0647546 358 16 88435 88419 9.64621 8.4375 0.0754464 0.0647764 359 16 88658 88642 9.6436 8.71094 0.0672776 0.0647925 2023-04-13T04:55:02.520703+0000 min lat: 0.00609258 max lat: 0.240071 avg lat: 0.0648158 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 360 16 88870 88854 9.63981 8.28125 0.0778955 0.0648158 361 16 89085 89069 9.63637 8.39844 0.0515207 0.0648426 362 16 89292 89276 9.63208 8.08594 0.0327926 0.0648687 363 16 89525 89509 9.63061 9.10156 0.0648385 0.0648815 364 16 89750 89734 9.6283 8.78906 0.0525968 0.0648951 365 16 89967 89951 9.62514 8.47656 0.0791533 0.0649179 366 16 90182 90166 9.62178 8.39844 0.0706269 0.064939 367 16 90437 90421 9.6227 9.96094 0.0567639 0.0649375 368 16 90683 90667 9.62266 9.60938 0.0631361 0.0649363 369 16 90934 90918 9.62315 9.80469 0.0221434 0.0649311 370 16 91168 91152 9.62184 9.14062 0.0544424 0.0649388 371 16 91414 91398 9.62181 9.60938 0.0336262 0.0649413 372 16 91644 91628 9.62009 8.98438 0.044454 0.0649515 373 16 91888 91872 9.61985 9.53125 0.055974 0.0649531 374 16 92120 92104 9.61835 9.0625 0.0867904 0.0649635 375 16 92355 92339 9.61718 9.17969 0.0764242 0.0649723 376 16 92589 92573 9.61591 9.14062 0.0511875 0.0649789 377 16 92827 92811 9.61506 9.29688 0.0635383 0.0649858 378 16 93058 93042 9.61349 9.02344 0.0792253 0.0649945 379 16 93298 93282 9.61286 9.375 0.0740048 0.0650005 2023-04-13T04:55:22.523908+0000 min lat: 0.00609258 max lat: 0.240071 avg lat: 0.0650071 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 380 16 93536 93520 9.61202 9.29688 0.0616704 0.0650071 381 16 93775 93759 9.61129 9.33594 0.0548263 0.0650084 382 16 93996 93980 9.60873 8.63281 0.0247426 0.0650247 383 16 94216 94200 9.60608 8.59375 0.0819456 0.0650469 384 16 94433 94417 9.60313 8.47656 0.0440235 0.0650687 385 16 94654 94638 9.60061 8.63281 0.0748346 0.0650838 386 16 94888 94872 9.59941 9.14062 0.0617576 0.0650916 387 16 95136 95120 9.59964 9.6875 0.0849638 0.0650918 388 16 95368 95352 9.59825 9.0625 0.0637458 0.0650994 389 16 95605 95589 9.59737 9.25781 0.0595924 0.0651057 390 16 95839 95823 9.59619 9.14062 0.059571 0.0651125 391 16 96072 96056 9.59492 9.10156 0.0970147 0.0651231 392 16 96312 96296 9.59436 9.375 0.085516 0.0651276 393 16 96544 96528 9.593 9.0625 0.108883 0.0651363 394 16 96771 96755 9.59116 8.86719 0.0513423 0.0651488 395 16 97013 96997 9.5908 9.45312 0.0778986 0.0651503 396 16 97248 97232 9.58976 9.17969 0.0739628 0.0651563 397 16 97498 97482 9.5902 9.76562 0.0684651 0.0651564 398 16 97746 97730 9.59044 9.6875 0.0875853 0.0651538 399 16 97996 97980 9.59088 9.76562 0.049249 0.06515 2023-04-13T04:55:42.527126+0000 min lat: 0.00609258 max lat: 0.240071 avg lat: 0.0651496 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 400 16 98245 98229 9.59121 9.72656 0.070738 0.0651496 401 16 98498 98482 9.59193 9.88281 0.0483233 0.0651448 402 16 98745 98729 9.59207 9.64844 0.0324629 0.0651423 403 16 98998 98982 9.59279 9.88281 0.100296 0.0651372 404 16 99249 99233 9.59331 9.80469 0.0563201 0.065135 405 16 99498 99482 9.59364 9.72656 0.0666934 0.0651303 406 16 99752 99736 9.59444 9.92188 0.0656299 0.065126 407 14 100000 99986 9.59486 9.76562 0.121083 0.0651235 Total time run: 407.121 Total writes made: 100000 Write size: 40960 Object size: 40960 Bandwidth (MB/sec): 9.59481 Stddev Bandwidth: 0.734871 Max bandwidth (MB/sec): 12.6562 Min bandwidth (MB/sec): 7.85156 Average IOPS: 245 Stddev IOPS: 18.8127 Max IOPS: 324 Min IOPS: 201 Average Latency(s): 0.0651235 Stddev Latency(s): 0.0230983 Max latency(s): 0.240071 Min latency(s): 0.00609258 [ceph: root@dell-r640-056 /]# [ceph: root@dell-r640-056 /]# [ceph: root@dell-r640-056 /]# ceph df 2023-04-13T04:58:29.452+0000 7f1700c19700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-04-13T04:58:29.453+0000 7f1700c19700 -1 WARNING: the following dangerous and experimental features are enabled: crimson --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED TOTAL 1.5 TiB 1.4 TiB 44 GiB 44 GiB 2.92 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 897 KiB 4 897 KiB 0 457 GiB cephfs.cephfs.meta 2 32 4.6 KiB 44 4.6 KiB 0 457 GiB cephfs.cephfs.data 3 32 0 B 0 0 B 0 457 GiB .rgw.root 4 32 368 B 8 368 B 0 457 GiB default.rgw.log 5 32 7.2 KiB 418 7.2 KiB 0 457 GiB default.rgw.control 6 32 0 B 16 0 B 0 457 GiB default.rgw.meta 7 32 738 B 2 738 B 0 457 GiB re_pool_compress_snappy 8 32 7.6 GiB 200.00k 7.6 GiB 0.55 457 GiB re_pool_compress_zlib 9 32 7.6 GiB 200.00k 7.6 GiB 0.55 457 GiB re_pool_compress_zstd 11 32 0 B 0 0 B 0 457 GiB re_pool_no_compress 12 32 7.6 GiB 200.00k 7.6 GiB 0.55 457 GiB # rados --no-log-to-stderr -b 40KB -p re_pool_compress_zstd bench 500 write --no-cleanup --max-objects 100000 hints = 1 Maintaining 16 concurrent writes of 40960 bytes to objects of size 40960 for up to 500 seconds or 100000 objects Object prefix: benchmark_data_dell-r640-056.dsal.lab.eng.rd_402 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 246 230 8.98324 8.98438 0.0600373 0.0674067 2 16 510 494 9.6469 10.3125 0.0439033 0.0637052 3 16 789 773 10.0634 10.8984 0.0578939 0.0613752 4 16 1055 1039 10.1448 10.3906 0.0511392 0.0609975 5 16 1320 1304 10.1858 10.3516 0.0555896 0.0608579 6 16 1600 1584 10.3108 10.9375 0.0903209 0.0603261 7 16 1863 1847 10.3052 10.2734 0.0596241 0.0603386 8 16 2143 2127 10.3841 10.9375 0.0438589 0.0600137 9 16 2399 2383 10.3412 10 0.0557349 0.0602268 10 16 2668 2652 10.3577 10.5078 0.0691487 0.0601699 11 16 2931 2915 10.3499 10.2734 0.0250146 0.0602007 12 16 3212 3196 10.402 10.9766 0.0358875 0.0599715 13 16 3487 3471 10.428 10.7422 0.0514698 0.0597882 14 16 3764 3748 10.4559 10.8203 0.0296665 0.059666 15 16 4026 4010 10.441 10.2344 0.0520078 0.0597104 16 16 4309 4293 10.4793 11.0547 0.0363369 0.0595173 17 16 4588 4572 10.5038 10.8984 0.0909232 0.0593626 18 16 4875 4859 10.543 11.2109 0.097626 0.0591631 19 16 5169 5153 10.5925 11.4844 0.0642877 0.0589207 2023-04-13T05:03:46.736437+0000 min lat: 0.00667864 max lat: 0.185719 avg lat: 0.0588912 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 20 16 5439 5423 10.5901 10.5469 0.0564301 0.0588912 21 16 5738 5722 10.6419 11.6797 0.0387952 0.058652 22 16 6017 6001 10.6535 10.8984 0.0588147 0.0585536 23 16 6298 6282 10.6674 10.9766 0.0475009 0.0584733 24 16 6595 6579 10.7063 11.6016 0.0219641 0.0583107 25 16 6873 6857 10.7123 10.8594 0.060493 0.0582638 26 16 7143 7127 10.7059 10.5469 0.0476892 0.0582864 27 16 7440 7424 10.739 11.6016 0.0659161 0.0581262 28 16 7677 7661 10.6861 9.25781 0.0762396 0.0584138 29 16 7926 7910 10.6529 9.72656 0.0405787 0.0585591 30 16 8158 8142 10.5999 9.0625 0.0318263 0.0588898 31 16 8388 8372 10.5477 8.98438 0.0785698 0.0591597 32 16 8615 8599 10.4951 8.86719 0.0640042 0.0594684 33 16 8823 8807 10.4233 8.125 0.0973481 0.05987 34 16 9046 9030 10.3728 8.71094 0.0918376 0.0601701 35 16 9275 9259 10.332 8.94531 0.0582292 0.0604291 36 16 9477 9461 10.2642 7.89062 0.0578657 0.0607981 37 16 9709 9693 10.2316 9.0625 0.0638804 0.0610336 38 16 9919 9903 10.1782 8.20312 0.0794127 0.0613189 39 16 10141 10125 10.1396 8.67188 0.0645853 0.0615625 2023-04-13T05:04:06.739767+0000 min lat: 0.00667864 max lat: 0.224792 avg lat: 0.0617762 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 40 16 10364 10348 10.1038 8.71094 0.0621559 0.0617762 41 16 10562 10546 10.046 7.73438 0.0972722 0.0621445 42 16 10772 10756 10.0021 8.20312 0.111906 0.0624179 43 16 10991 10975 9.96839 8.55469 0.0902309 0.0626331 44 15 11224 11209 9.94954 9.14062 0.0858894 0.0627505 45 16 11457 11441 9.9298 9.0625 0.0478469 0.0628798 46 16 11696 11680 9.91686 9.33594 0.0910557 0.0629661 47 16 11935 11919 9.90446 9.33594 0.0620639 0.0630485 48 16 12182 12166 9.89909 9.64844 0.0328229 0.0630711 49 15 12420 12405 9.88756 9.33594 0.083491 0.0631589 50 16 12662 12646 9.87805 9.41406 0.0760639 0.0632139 51 16 12909 12893 9.87352 9.64844 0.0264998 0.0632585 52 16 13150 13134 9.86466 9.41406 0.112248 0.0633021 53 16 13396 13380 9.85981 9.60938 0.0951963 0.0633438 54 16 13635 13619 9.85009 9.33594 0.0614263 0.0633982 55 16 13877 13861 9.84284 9.45312 0.0499505 0.0634441 56 16 14121 14105 9.83725 9.53125 0.0507468 0.0634734 57 16 14363 14347 9.83049 9.45312 0.0657081 0.0635331 58 16 14624 14608 9.83675 10.1953 0.0905146 0.063491 59 16 14873 14857 9.83486 9.72656 0.0469819 0.0635037 2023-04-13T05:04:26.742956+0000 min lat: 0.00667864 max lat: 0.224792 avg lat: 0.0635364 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 60 16 15119 15103 9.83108 9.60938 0.0557167 0.0635364 61 16 15353 15337 9.81974 9.14062 0.0603674 0.0636026 62 16 15581 15565 9.80498 8.90625 0.0449469 0.0637039 63 16 15809 15793 9.79069 8.90625 0.124267 0.0637902 64 16 16045 16029 9.78173 9.21875 0.0428793 0.06383 65 16 16281 16265 9.77305 9.21875 0.0905538 0.0639139 66 16 16515 16499 9.76344 9.14062 0.0537279 0.0639677 67 16 16747 16731 9.75296 9.0625 0.0898372 0.0640364 68 16 16970 16954 9.73762 8.71094 0.0614205 0.0641217 69 16 17201 17185 9.72724 9.02344 0.100148 0.0642086 70 16 17428 17412 9.71494 8.86719 0.0908992 0.0642928 71 16 17665 17649 9.70848 9.25781 0.060991 0.0643297 72 16 17902 17886 9.70219 9.25781 0.052698 0.0643587 73 16 18134 18118 9.69342 9.0625 0.0460938 0.064434 74 16 18389 18373 9.69701 9.96094 0.0850629 0.0643937 75 16 18636 18620 9.69634 9.64844 0.0623747 0.0644019 76 16 18879 18863 9.69363 9.49219 0.0462191 0.0644403 77 16 19116 19100 9.68796 9.25781 0.0934112 0.0644811 78 16 19359 19343 9.68543 9.49219 0.0712317 0.0644864 79 16 19604 19588 9.68395 9.57031 0.0449236 0.0645074 2023-04-13T05:04:46.746200+0000 min lat: 0.00667864 max lat: 0.224792 avg lat: 0.0645136 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 80 16 19846 19830 9.68104 9.45312 0.0550302 0.0645136 81 16 20086 20070 9.67724 9.375 0.0539646 0.0645429 82 16 20336 20320 9.6783 9.76562 0.0635161 0.064539 83 16 20586 20570 9.67933 9.76562 0.0849871 0.0645439 84 16 20827 20811 9.67616 9.41406 0.0413172 0.0645572 85 16 21069 21053 9.67352 9.45312 0.0786252 0.0645734 86 16 21308 21292 9.66958 9.33594 0.13493 0.0645953 87 16 21550 21534 9.66707 9.45312 0.136871 0.0646164 88 16 21784 21768 9.66107 9.14062 0.130085 0.0646464 89 16 21989 21973 9.64248 8.00781 0.0652254 0.0647701 90 16 22240 22224 9.64427 9.80469 0.0796747 0.0647634 91 16 22476 22460 9.63957 9.21875 0.0608202 0.0647934 92 16 22730 22714 9.64263 9.92188 0.0288281 0.0647849 93 16 22984 22968 9.64561 9.92188 0.0664309 0.0647702 94 16 23231 23215 9.64563 9.64844 0.0772133 0.0647615 95 16 23492 23476 9.65139 10.1953 0.0453507 0.0647246 96 16 23747 23731 9.6546 9.96094 0.0639312 0.0647005 97 16 24006 23990 9.65935 10.1172 0.0545359 0.0646739 98 16 24270 24254 9.666 10.3125 0.0542092 0.0646215 99 16 24523 24507 9.66817 9.88281 0.0509716 0.064606 2023-04-13T05:05:06.749423+0000 min lat: 0.00667864 max lat: 0.224792 avg lat: 0.0645814 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 100 16 24783 24767 9.67304 10.1562 0.0529317 0.0645814 101 16 25037 25021 9.67549 9.92188 0.059723 0.0645621 102 16 25297 25281 9.68019 10.1562 0.0508106 0.0645349 103 16 25562 25546 9.68669 10.3516 0.107989 0.0644936 104 16 25839 25823 9.69758 10.8203 0.0636972 0.0644174 105 16 26086 26070 9.6971 9.64844 0.0731411 0.064419 106 16 26346 26330 9.70142 10.1562 0.0884742 0.0643943 107 16 26613 26597 9.70822 10.4297 0.0271443 0.0643466 108 16 26878 26862 9.71416 10.3516 0.0902865 0.0643062 109 16 27156 27140 9.72465 10.8594 0.0499879 0.0642417 110 16 27409 27393 9.72608 9.88281 0.0573942 0.0642245 111 16 27679 27663 9.73345 10.5469 0.0465214 0.0641859 112 16 27938 27922 9.73687 10.1172 0.0795246 0.0641541 113 16 28203 28187 9.7423 10.3516 0.0391268 0.064122 114 16 28458 28442 9.7442 9.96094 0.0474732 0.0641117 115 16 28723 28707 9.74947 10.3516 0.0452291 0.0640724 116 16 28995 28979 9.75701 10.625 0.0964019 0.0640267 117 16 29270 29254 9.76541 10.7422 0.0799 0.0639672 118 16 29537 29521 9.77102 10.4297 0.0651751 0.0639308 119 16 29791 29775 9.77228 9.92188 0.085791 0.0639359 2023-04-13T05:05:26.752330+0000 min lat: 0.00667864 max lat: 0.224792 avg lat: 0.0639076 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 120 16 30048 30032 9.77448 10.0391 0.0501469 0.0639076 121 16 30307 30291 9.7773 10.1172 0.0706417 0.0638992 122 16 30574 30558 9.78263 10.4297 0.0196671 0.0638619 123 16 30854 30838 9.79201 10.9375 0.0712609 0.0638073 124 16 31116 31100 9.79556 10.2344 0.0436002 0.0637803 125 16 31380 31364 9.79968 10.3125 0.0577835 0.0637513 126 16 31651 31635 9.80591 10.5859 0.117601 0.0637133 127 16 31912 31896 9.80897 10.1953 0.028266 0.0636881 128 16 32193 32177 9.81808 10.9766 0.0473209 0.0636302 129 16 32460 32444 9.82281 10.4297 0.0849002 0.0636026 130 16 32723 32707 9.82626 10.2734 0.0457651 0.0635767 131 16 33000 32984 9.83385 10.8203 0.0681242 0.0635352 132 16 33256 33240 9.83509 10 0.066326 0.0635253 133 16 33518 33502 9.83808 10.2344 0.0585911 0.0635002 134 16 33798 33782 9.84627 10.9375 0.0870283 0.0634414 135 16 34063 34047 9.85 10.3516 0.0415457 0.0634316 136 16 34340 34324 9.85713 10.8203 0.0604918 0.0633813 137 16 34592 34576 9.85702 9.84375 0.037175 0.0633816 138 16 34850 34834 9.85861 10.0781 0.0796612 0.0633717 139 16 35107 35091 9.8599 10.0391 0.0844975 0.0633672 2023-04-13T05:05:46.755223+0000 min lat: 0.00667864 max lat: 0.224792 avg lat: 0.0633873 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 140 16 35347 35331 9.85643 9.375 0.0641171 0.0633873 141 16 35594 35578 9.85494 9.64844 0.111485 0.0633939 142 16 35838 35822 9.85266 9.53125 0.0547389 0.0634051 143 16 36091 36075 9.85286 9.88281 0.0581925 0.0634116 144 16 36345 36329 9.85332 9.92188 0.0631138 0.0634024 145 16 36593 36577 9.85217 9.6875 0.0711022 0.0634158 146 16 36848 36832 9.8529 9.96094 0.0746745 0.06341 147 16 37106 37090 9.85442 10.0781 0.0519341 0.0634024 148 16 37351 37335 9.85249 9.57031 0.0784268 0.0634069 149 16 37612 37596 9.85478 10.1953 0.122777 0.0633925 150 16 37882 37866 9.85938 10.5469 0.138849 0.0633658 151 16 38153 38137 9.86418 10.5859 0.0203277 0.0633362 152 16 38415 38399 9.86661 10.2344 0.0589114 0.0633182 153 16 38665 38649 9.86594 9.76562 0.0292812 0.0633209 154 16 38919 38903 9.8663 9.92188 0.0621862 0.0633229 155 16 39182 39166 9.86891 10.2734 0.0560412 0.063308 156 16 39441 39425 9.87049 10.1172 0.0540725 0.063295 157 16 39700 39684 9.87205 10.1172 0.0597216 0.0632881 158 16 39954 39938 9.87236 9.92188 0.067886 0.0632823 159 16 40218 40202 9.87512 10.3125 0.0558274 0.0632667 2023-04-13T05:06:06.758350+0000 min lat: 0.00667864 max lat: 0.224792 avg lat: 0.0632613 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 160 16 40473 40457 9.87564 9.96094 0.0541502 0.0632613 161 16 40720 40704 9.87422 9.64844 0.0609531 0.0632706 162 16 40982 40966 9.87644 10.2344 0.0479343 0.0632568 163 16 41214 41198 9.87143 9.0625 0.0704905 0.0632918 164 16 41425 41409 9.86149 8.24219 0.0852202 0.0633561 165 16 41632 41616 9.85072 8.08594 0.0863793 0.0634238 166 16 41829 41813 9.83773 7.69531 0.113525 0.0634992 167 16 42048 42032 9.83004 8.55469 0.0583474 0.0635604 168 16 42282 42266 9.82593 9.14062 0.0903292 0.0635768 169 16 42519 42503 9.82256 9.25781 0.0780876 0.0636054 170 16 42746 42730 9.81693 8.86719 0.103735 0.0636452 171 16 42985 42969 9.81411 9.33594 0.0719402 0.0636648 172 16 43199 43183 9.80564 8.35938 0.0479421 0.0637133 173 16 43436 43420 9.80246 9.25781 0.0415538 0.0637348 174 16 43662 43646 9.79686 8.82812 0.0827463 0.0637697 175 16 43905 43889 9.79511 9.49219 0.0668389 0.0637785 176 16 44137 44121 9.79094 9.0625 0.0801236 0.063806 177 16 44360 44344 9.78483 8.71094 0.0728736 0.0638506 178 16 44603 44587 9.78318 9.49219 0.0659189 0.0638625 179 16 44829 44813 9.77784 8.82812 0.0827434 0.0638976 2023-04-13T05:06:26.761493+0000 min lat: 0.00667864 max lat: 0.224792 avg lat: 0.0639161 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 180 16 45067 45051 9.77516 9.29688 0.050262 0.0639161 181 16 45313 45297 9.77423 9.60938 0.0623425 0.063921 182 16 45541 45525 9.76945 8.90625 0.0828264 0.0639484 183 16 45764 45748 9.76366 8.71094 0.0228834 0.0639934 184 16 45981 45965 9.75666 8.47656 0.0833643 0.0640315 185 16 46195 46179 9.7491 8.35938 0.0354542 0.0640851 186 16 46415 46399 9.74288 8.59375 0.138937 0.0641221 187 16 46633 46617 9.73631 8.51562 0.0711016 0.0641738 188 16 46850 46834 9.72961 8.47656 0.0991698 0.0642167 189 16 47073 47057 9.72421 8.71094 0.0707563 0.0642514 190 16 47295 47279 9.71867 8.67188 0.0996738 0.0642835 191 16 47519 47503 9.71359 8.75 0.065211 0.0643239 192 16 47736 47720 9.70714 8.47656 0.0970251 0.0643631 193 16 47962 47946 9.70258 8.82812 0.025963 0.064392 194 16 48206 48190 9.70169 9.53125 0.0776154 0.0643918 195 16 48444 48428 9.69961 9.29688 0.0623114 0.0644144 196 16 48690 48674 9.69914 9.60938 0.0346621 0.0644178 197 16 48949 48933 9.70125 10.1172 0.0320813 0.0644029 198 16 49219 49203 9.70552 10.5469 0.0721697 0.0643731 199 16 49496 49480 9.71111 10.8203 0.0956345 0.0643397 2023-04-13T05:06:46.764482+0000 min lat: 0.00667864 max lat: 0.224792 avg lat: 0.064299 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 200 16 49775 49759 9.71703 10.8984 0.0555362 0.064299 201 16 50042 50026 9.72057 10.4297 0.0586437 0.0642783 202 16 50329 50313 9.72794 11.2109 0.0667378 0.064229 203 16 50603 50587 9.73274 10.7031 0.0644792 0.0641966 204 16 50868 50852 9.73576 10.3516 0.0473984 0.0641757 205 16 51150 51134 9.742 11.0156 0.0522043 0.0641355 206 16 51427 51411 9.74723 10.8203 0.106332 0.0640951 207 16 51705 51689 9.75259 10.8594 0.0562377 0.0640667 208 16 51994 51978 9.75997 11.2891 0.0782351 0.0640166 209 16 52256 52240 9.76224 10.2344 0.0727238 0.0639972 210 16 52530 52514 9.76671 10.7031 0.0509261 0.0639703 211 16 52805 52789 9.77132 10.7422 0.0719211 0.0639411 212 16 53069 53053 9.77387 10.3125 0.0977523 0.0639267 213 16 53332 53316 9.77621 10.2734 0.0533422 0.0639119 214 16 53589 53573 9.77743 10.0391 0.0774934 0.0639008 215 16 53854 53838 9.7801 10.3516 0.0765342 0.0638867 216 16 54120 54104 9.78292 10.3906 0.0922682 0.0638669 217 16 54373 54357 9.78337 9.88281 0.0736175 0.0638651 218 16 54637 54621 9.78579 10.3125 0.0608035 0.0638491 219 16 54901 54885 9.78819 10.3125 0.0884411 0.0638321 2023-04-13T05:07:06.767227+0000 min lat: 0.00667864 max lat: 0.224792 avg lat: 0.0638218 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 220 16 55159 55143 9.7895 10.0781 0.0558823 0.0638218 221 16 55431 55415 9.79328 10.625 0.0837393 0.0637995 222 15 55679 55664 9.79297 9.72656 0.156184 0.0637968 223 16 55922 55906 9.79144 9.45312 0.0520001 0.0638098 224 16 56186 56170 9.79376 10.3125 0.0877063 0.0637949 225 16 56434 56418 9.79328 9.6875 0.0593193 0.0637984 226 16 56675 56659 9.7916 9.41406 0.0204906 0.0638094 227 16 56907 56891 9.78838 9.0625 0.0487648 0.0638306 228 16 57138 57122 9.78502 9.02344 0.0694344 0.0638556 229 16 57367 57351 9.78135 8.94531 0.0484405 0.0638772 230 16 57612 57596 9.78042 9.57031 0.0653802 0.0638797 231 16 57827 57811 9.77444 8.39844 0.0450063 0.0639212 232 16 58067 58051 9.77271 9.375 0.0586233 0.0639318 233 16 58291 58275 9.76831 8.75 0.0892565 0.0639634 234 16 58514 58498 9.76379 8.71094 0.0281675 0.0639905 235 16 58743 58727 9.7603 8.94531 0.0562109 0.0640151 236 16 58964 58948 9.75552 8.63281 0.0735953 0.0640435 237 16 59202 59186 9.75358 9.29688 0.0461864 0.0640605 238 16 59420 59404 9.74837 8.51562 0.0741301 0.0640946 239 16 59635 59619 9.74272 8.39844 0.0600326 0.0641319 2023-04-13T05:07:26.770062+0000 min lat: 0.00667864 max lat: 0.271505 avg lat: 0.0641691 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 240 16 59846 59830 9.73646 8.24219 0.0858436 0.0641691 241 16 60069 60053 9.7322 8.71094 0.0827704 0.0642007 242 16 60310 60294 9.73088 9.41406 0.0706037 0.0642082 243 16 60546 60530 9.72876 9.21875 0.0438746 0.0642226 244 16 60776 60760 9.72571 8.98438 0.0822955 0.064244 245 16 61020 61004 9.72491 9.53125 0.101321 0.0642514 246 16 61257 61241 9.723 9.25781 0.055308 0.0642605 247 16 61492 61476 9.7208 9.17969 0.0711446 0.0642765 248 16 61737 61721 9.72018 9.57031 0.0788291 0.064278 249 16 61961 61945 9.71628 8.75 0.0217609 0.0643016 250 16 62212 62196 9.71663 9.80469 0.0717121 0.064302 251 16 62453 62437 9.71542 9.41406 0.057147 0.064312 252 16 62690 62674 9.7136 9.25781 0.096273 0.0643218 253 16 62940 62924 9.7138 9.76562 0.047011 0.0643224 254 16 63177 63161 9.71199 9.25781 0.0878912 0.0643311 255 16 63395 63379 9.7073 8.51562 0.075669 0.0643643 256 16 63620 63604 9.70371 8.78906 0.0688472 0.0643888 257 16 63846 63830 9.70029 8.82812 0.0964148 0.0644097 258 16 64083 64067 9.69857 9.25781 0.0373237 0.0644229 259 16 64318 64302 9.69657 9.17969 0.0497907 0.0644396 2023-04-13T05:07:46.773236+0000 min lat: 0.00663439 max lat: 0.271505 avg lat: 0.0644628 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 260 16 64540 64524 9.69262 8.67188 0.0641766 0.0644628 261 16 64770 64754 9.6899 8.98438 0.0901743 0.0644815 262 16 65000 64984 9.6872 8.98438 0.0530057 0.0644963 263 16 65225 65209 9.68378 8.78906 0.0550628 0.0645193 264 16 65467 65451 9.6829 9.45312 0.0478029 0.0645293 265 16 65687 65671 9.67879 8.59375 0.0704308 0.064556 266 16 65916 65900 9.67603 8.94531 0.0522666 0.0645725 267 16 66146 66130 9.67343 8.98438 0.0383689 0.064593 268 16 66360 66344 9.66852 8.35938 0.0925736 0.0646248 269 16 66583 66567 9.66495 8.71094 0.0582961 0.0646487 270 16 66809 66793 9.66185 8.82812 0.10157 0.0646691 271 16 67018 67002 9.65632 8.16406 0.0746992 0.0647064 272 16 67229 67213 9.65111 8.24219 0.0874332 0.0647387 273 16 67432 67416 9.6448 7.92969 0.086388 0.0647817 274 16 67634 67618 9.6384 7.89062 0.0974402 0.0648183 275 16 67851 67835 9.63416 8.47656 0.102395 0.0648534 276 16 68061 68045 9.62898 8.20312 0.0802328 0.0648879 277 16 68270 68254 9.62368 8.16406 0.0842581 0.064925 278 16 68473 68457 9.61758 7.92969 0.0697109 0.0649658 279 16 68683 68667 9.61251 8.20312 0.0674325 0.0650018 2023-04-13T05:08:06.776493+0000 min lat: 0.00605058 max lat: 0.271505 avg lat: 0.0650307 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 280 16 68897 68881 9.60803 8.35938 0.113342 0.0650307 281 16 69103 69087 9.60247 8.04688 0.0489676 0.0650701 282 16 69323 69307 9.59889 8.59375 0.072355 0.0650953 283 16 69555 69539 9.59698 9.0625 0.0702289 0.0651038 284 16 69792 69776 9.59579 9.25781 0.0322341 0.0651133 285 16 70030 70014 9.59473 9.29688 0.0591303 0.0651212 286 16 70275 70259 9.59464 9.57031 0.0684268 0.0651205 287 16 70511 70495 9.59333 9.21875 0.0806728 0.065132 288 16 70776 70760 9.59595 10.3516 0.0792249 0.0651119 289 16 71023 71007 9.59613 9.64844 0.0918503 0.0651116 290 16 71282 71266 9.59792 10.1172 0.0397073 0.0651015 291 16 71540 71524 9.59956 10.0781 0.0276064 0.0650907 292 16 71785 71769 9.59946 9.57031 0.0545044 0.0650898 293 16 72042 72026 9.60095 10.0391 0.0648292 0.0650771 294 16 72310 72294 9.6039 10.4688 0.0513896 0.0650598 295 16 72564 72548 9.60497 9.92188 0.0499123 0.0650549 296 16 72834 72818 9.60815 10.5469 0.064905 0.0650313 297 16 73081 73065 9.60828 9.64844 0.0538388 0.0650289 298 16 73329 73313 9.60854 9.6875 0.0892616 0.0650254 299 16 73549 73533 9.60514 8.59375 0.0508483 0.0650517 2023-04-13T05:08:26.779737+0000 min lat: 0.00605058 max lat: 0.271505 avg lat: 0.0650661 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 300 16 73777 73761 9.60281 8.90625 0.0688372 0.0650661 301 16 74008 73992 9.60087 9.02344 0.0813623 0.0650816 302 16 74221 74205 9.59663 8.32031 0.0569139 0.0651093 303 16 74439 74423 9.59306 8.51562 0.0836303 0.0651347 304 16 74655 74639 9.58925 8.4375 0.0662269 0.0651581 305 16 74858 74842 9.5838 7.92969 0.0565877 0.0651919 306 16 75083 75067 9.5812 8.78906 0.0864143 0.0652137 307 16 75301 75285 9.57773 8.51562 0.074016 0.0652341 308 16 75510 75494 9.57313 8.16406 0.0907225 0.0652668 309 16 75729 75713 9.56983 8.55469 0.0540173 0.0652864 310 16 75949 75933 9.56668 8.59375 0.0914949 0.0653152 311 16 76161 76145 9.56254 8.28125 0.0862239 0.0653399 312 16 76375 76359 9.55868 8.35938 0.120013 0.0653648 313 16 76599 76583 9.55609 8.75 0.0713196 0.0653823 314 16 76832 76816 9.55464 9.10156 0.0466855 0.0653977 315 16 77075 77059 9.55444 9.49219 0.0724518 0.0654006 316 16 77314 77298 9.55374 9.33594 0.0548728 0.0654018 317 16 77561 77545 9.55404 9.64844 0.0865801 0.0654 318 16 77778 77762 9.55064 8.47656 0.101307 0.0654227 319 16 78017 78001 9.54996 9.33594 0.0693834 0.065425 2023-04-13T05:08:46.783098+0000 min lat: 0.00605058 max lat: 0.271505 avg lat: 0.0654288 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 320 16 78260 78244 9.54978 9.49219 0.106634 0.0654288 321 16 78474 78458 9.54607 8.35938 0.149554 0.0654548 322 16 78726 78710 9.54699 9.84375 0.0587728 0.0654494 323 16 78970 78954 9.54693 9.53125 0.0175476 0.0654485 324 16 79199 79183 9.54507 8.94531 0.0610551 0.0654571 325 16 79443 79427 9.54503 9.53125 0.0584572 0.0654607 326 16 79689 79673 9.54522 9.60938 0.0679268 0.0654577 327 16 79930 79914 9.54481 9.41406 0.0529874 0.0654643 328 16 80163 80147 9.54346 9.10156 0.0643509 0.0654716 329 16 80403 80387 9.54294 9.375 0.0482653 0.0654759 330 16 80657 80641 9.54408 9.92188 0.0576842 0.0654678 331 16 80913 80897 9.54545 10 0.0584924 0.0654606 332 16 81139 81123 9.54329 8.82812 0.0782335 0.0654748 333 16 81381 81365 9.54301 9.45312 0.0493046 0.0654782 334 16 81611 81595 9.54133 8.98438 0.0852482 0.0654865 335 16 81856 81840 9.54142 9.57031 0.0243376 0.0654855 336 16 82114 82098 9.54301 10.0781 0.0550497 0.065476 337 16 82335 82319 9.5403 8.63281 0.154609 0.0654953 338 16 82563 82547 9.53842 8.90625 0.115017 0.0655047 339 16 82807 82791 9.5384 9.53125 0.0495607 0.0655061 2023-04-13T05:09:06.786536+0000 min lat: 0.00605058 max lat: 0.271505 avg lat: 0.0655108 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 340 16 83048 83032 9.53803 9.41406 0.0928415 0.0655108 341 16 83290 83274 9.53777 9.45312 0.057413 0.065511 342 16 83528 83512 9.53706 9.29688 0.0499899 0.0655159 343 16 83749 83733 9.53442 8.63281 0.0559793 0.0655286 344 16 83979 83963 9.53282 8.98438 0.0761341 0.0655446 345 16 84203 84187 9.53055 8.75 0.0336576 0.06556 346 16 84430 84414 9.52863 8.86719 0.0363203 0.065573 347 16 84661 84645 9.52716 9.02344 0.0756145 0.0655844 348 16 84889 84873 9.52538 8.90625 0.0875002 0.0655975 349 16 85126 85110 9.52461 9.25781 0.105544 0.0656015 350 16 85355 85339 9.52295 8.94531 0.0656413 0.0656138 351 16 85582 85566 9.52108 8.86719 0.149186 0.0656264 352 16 85807 85791 9.51899 8.78906 0.0380592 0.0656404 353 16 86026 86010 9.51626 8.55469 0.0685323 0.0656605 354 16 86261 86245 9.5153 9.17969 0.0602591 0.0656658 355 16 86492 86476 9.51391 9.02344 0.0584618 0.0656768 356 16 86710 86694 9.5111 8.51562 0.0855792 0.0656958 357 16 86941 86925 9.50973 9.02344 0.0802387 0.0657055 358 16 87164 87148 9.5075 8.71094 0.0551107 0.0657203 359 16 87370 87354 9.50343 8.04688 0.105618 0.0657468 2023-04-13T05:09:26.789738+0000 min lat: 0.00605058 max lat: 0.271505 avg lat: 0.0657572 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 360 16 87604 87588 9.50241 9.14062 0.0340762 0.0657572 361 16 87818 87802 9.49924 8.35938 0.0575179 0.0657783 362 16 88068 88052 9.49997 9.76562 0.0675018 0.0657753 363 16 88305 88289 9.4993 9.25781 0.0521042 0.0657787 364 16 88534 88518 9.49778 8.94531 0.119252 0.0657881 365 16 88779 88763 9.49797 9.57031 0.0913646 0.0657875 366 16 89025 89009 9.49827 9.60938 0.0488694 0.0657847 367 16 89256 89240 9.49698 9.02344 0.0666665 0.0657933 368 16 89494 89478 9.49643 9.29688 0.0330884 0.0657972 369 16 89723 89707 9.49493 8.94531 0.0800626 0.0658088 370 16 89961 89945 9.49439 9.29688 0.128946 0.065811 371 16 90205 90189 9.49449 9.53125 0.053807 0.0658115 372 16 90443 90427 9.49395 9.29688 0.0335141 0.0658143 373 16 90671 90655 9.49237 8.90625 0.0532628 0.0658242 374 16 90922 90906 9.4932 9.80469 0.0486844 0.0658207 375 16 91160 91144 9.49267 9.29688 0.0700027 0.065825 376 16 91405 91389 9.49288 9.57031 0.0566409 0.0658233 377 16 91631 91615 9.49111 8.82812 0.0800658 0.0658338 378 16 91863 91847 9.48997 9.0625 0.0410411 0.0658422 379 16 92090 92074 9.48832 8.86719 0.0596788 0.0658544 2023-04-13T05:09:46.793012+0000 min lat: 0.00605058 max lat: 0.271505 avg lat: 0.0658645 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 380 16 92316 92300 9.48658 8.82812 0.0177906 0.0658645 381 15 92548 92533 9.48557 9.10156 0.0553965 0.0658733 382 16 92772 92756 9.48354 8.71094 0.065066 0.0658867 383 16 92996 92980 9.48162 8.75 0.0473835 0.0658991 384 16 93236 93220 9.48134 9.375 0.111492 0.065904 385 16 93459 93443 9.47933 8.71094 0.0409527 0.0659167 386 16 93685 93669 9.47764 8.82812 0.0829271 0.0659281 387 15 93915 93900 9.47646 9.02344 0.0727319 0.0659381 388 16 94144 94128 9.47499 8.90625 0.0853927 0.0659469 389 16 94388 94372 9.47513 9.53125 0.096931 0.065946 390 16 94623 94607 9.47437 9.17969 0.0913951 0.0659493 391 16 94854 94838 9.47321 9.02344 0.0814616 0.0659601 392 16 95102 95086 9.47376 9.6875 0.0878519 0.0659567 393 16 95351 95335 9.4744 9.72656 0.0142497 0.0659526 394 16 95618 95602 9.47682 10.4297 0.0425142 0.0659363 395 16 95883 95867 9.47903 10.3516 0.04016 0.0659194 396 16 96147 96131 9.48113 10.3125 0.0379097 0.0659056 397 16 96411 96395 9.48321 10.3125 0.0698793 0.0658895 398 16 96684 96668 9.48618 10.6641 0.0469134 0.0658701 399 16 96940 96924 9.48746 10 0.0824558 0.0658616 2023-04-13T05:10:06.796231+0000 min lat: 0.00605058 max lat: 0.271505 avg lat: 0.0658396 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 400 16 97214 97198 9.4905 10.7031 0.0565129 0.0658396 401 16 97470 97454 9.49176 10 0.055572 0.0658322 402 16 97734 97718 9.4938 10.3125 0.0655177 0.0658175 403 16 98008 97992 9.4968 10.7031 0.0542325 0.0657966 404 16 98296 98280 9.50113 11.25 0.0462149 0.0657674 405 16 98596 98580 9.5066 11.7188 0.0618077 0.0657307 406 16 98882 98866 9.5107 11.1719 0.0523336 0.0657001 407 16 99157 99141 9.51372 10.7422 0.012724 0.0656793 408 16 99452 99436 9.51864 11.5234 0.0333666 0.0656463 409 16 99726 99710 9.52153 10.7031 0.0546391 0.0656268 410 9 100000 99991 9.52508 10.9766 0.0458565 0.0656015 Total time run: 410.092 Total writes made: 100000 Write size: 40960 Object size: 40960 Bandwidth (MB/sec): 9.52531 Stddev Bandwidth: 0.821681 Max bandwidth (MB/sec): 11.7188 Min bandwidth (MB/sec): 7.69531 Average IOPS: 243 Stddev IOPS: 21.035 Max IOPS: 300 Min IOPS: 197 Average Latency(s): 0.0656007 Stddev Latency(s): 0.0236665 Max latency(s): 0.271505 Min latency(s): 0.00605058 [ceph: root@dell-r640-056 /]# ceph df 2023-04-13T05:23:07.002+0000 7f9785ca0700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-04-13T05:23:07.003+0000 7f9785ca0700 -1 WARNING: the following dangerous and experimental features are enabled: crimson --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED TOTAL 1.5 TiB 1.4 TiB 55 GiB 55 GiB 3.69 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 897 KiB 4 897 KiB 0 450 GiB cephfs.cephfs.meta 2 32 4.6 KiB 44 4.6 KiB 0 450 GiB cephfs.cephfs.data 3 32 0 B 0 0 B 0 450 GiB .rgw.root 4 32 368 B 8 368 B 0 450 GiB default.rgw.log 5 32 7.2 KiB 418 7.2 KiB 0 450 GiB default.rgw.control 6 32 0 B 16 0 B 0 450 GiB default.rgw.meta 7 32 738 B 2 738 B 0 450 GiB re_pool_compress_snappy 8 32 7.6 GiB 200.00k 7.6 GiB 0.56 450 GiB re_pool_compress_zlib 9 32 7.6 GiB 200.00k 7.6 GiB 0.56 450 GiB re_pool_compress_zstd 11 32 7.6 GiB 200.00k 7.6 GiB 0.56 450 GiB re_pool_no_compress 12 32 7.6 GiB 200.00k 7.6 GiB 0.56 450 GiB # ceph df detail 2023-04-13T05:23:49.944+0000 7fa70a8bf700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-04-13T05:23:49.944+0000 7fa70a8bf700 -1 WARNING: the following dangerous and experimental features are enabled: crimson --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED TOTAL 1.5 TiB 1.4 TiB 55 GiB 55 GiB 3.69 --- POOLS --- POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR .mgr 1 1 897 KiB 897 KiB 0 B 4 897 KiB 897 KiB 0 B 0 450 GiB N/A N/A N/A 0 B 0 B cephfs.cephfs.meta 2 32 4.6 KiB 4.6 KiB 0 B 44 4.6 KiB 4.6 KiB 0 B 0 450 GiB N/A N/A N/A 0 B 0 B cephfs.cephfs.data 3 32 0 B 0 B 0 B 0 0 B 0 B 0 B 0 450 GiB N/A N/A N/A 0 B 0 B .rgw.root 4 32 368 B 368 B 0 B 8 368 B 368 B 0 B 0 450 GiB N/A N/A N/A 0 B 0 B default.rgw.log 5 32 7.2 KiB 7.2 KiB 0 B 418 7.2 KiB 7.2 KiB 0 B 0 450 GiB N/A N/A N/A 0 B 0 B default.rgw.control 6 32 0 B 0 B 0 B 16 0 B 0 B 0 B 0 450 GiB N/A N/A N/A 0 B 0 B default.rgw.meta 7 32 738 B 738 B 0 B 2 738 B 738 B 0 B 0 450 GiB N/A N/A N/A 0 B 0 B re_pool_compress_snappy 8 32 7.6 GiB 7.6 GiB 0 B 200.00k 7.6 GiB 7.6 GiB 0 B 0.56 450 GiB N/A N/A N/A 0 B 0 B re_pool_compress_zlib 9 32 7.6 GiB 7.6 GiB 0 B 200.00k 7.6 GiB 7.6 GiB 0 B 0.56 450 GiB N/A N/A N/A 0 B 0 B re_pool_compress_zstd 11 32 7.6 GiB 7.6 GiB 0 B 200.00k 7.6 GiB 7.6 GiB 0 B 0.56 450 GiB N/A N/A N/A 0 B 0 B re_pool_no_compress 12 32 7.6 GiB 7.6 GiB 0 B 200.00k 7.6 GiB 7.6 GiB 0 B 0.56 450 GiB N/A N/A N/A 0 B 0 B # cephadm shell -- ceph version Inferring fsid 0f746e28-d97c-11ed-81f2-78ac443b3a54 Inferring config /var/lib/ceph/0f746e28-d97c-11ed-81f2-78ac443b3a54/mon.dell-r640-056/config Using ceph image with id 'ad3ecf86ac85' and tag 'bf6db9f2c0b862c27e2a4db5e7a4a16b5fd297b5-crimson' created on 2023-04-12 01:13:07 +0000 UTC quay.ceph.io/ceph-ci/ceph@sha256:019bf5f6100b8a449c3bb9c23dd979d09323930df8824aaed818162decb4098d 2023-04-13T05:25:19.040+0000 7f28c0207700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-04-13T05:25:19.041+0000 7f28c0207700 -1 WARNING: the following dangerous and experimental features are enabled: crimson ceph version 18.0.0-3438-gbf6db9f2 (bf6db9f2c0b862c27e2a4db5e7a4a16b5fd297b5) reef (dev)