Project

General

Profile

Bug #47673

cephfs 4k randwrite + EC pool(2+1) + single node all OSDs OOM

Added by Stellar Wang over 3 years ago. Updated almost 3 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Performance/Resource Usage
Target version:
% Done:

0%

Source:
Community (dev)
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
rbd
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

A 4K random write scenario in a single-node full SSD cephfs will cause the OSD memory space to grow indefinitely and system OOM.

According to our analysis, it can be seen from the system that mempool reaches 8G, but actually cache_size: 1020054730 kv_alloc: 268435456 kV_used: 74407616 meta_alloc: 675282944 Meta_used: 107497683 Data_alloc: 67108864 data_used: 0
At present,it is not clear who USES the remaining 6G space.

【data loading】
8 clients, each client configuration as bellow:
fio -directory=/tmp/ec_cifs/ -direct=1 -iodepth=1 -ioengine=libaio -thread -numjobs=64 -rw=randwrite -size=100G -runtime=300 -fallocate=none -group_reporting -name=4k_write_random=1_1024threads_`hostname` -bs=4k

【Node physical config】
A single node and include 24 SSD disks, total mem 125G, 2G of space is allocated per disk(osd_memory_target=2147483648).

[root@node200 ~]# free -h
total used free shared buff/cache available
Mem: 125G 18G 87G 658M 19G 104G
Swap: 0B 0B 0B

【ceph config】
ceph config file:
bluestore_cache_autotune": "true",
"bluestore_cache_autotune_interval": "5.000000",
"bluestore_cache_kv_ratio": "0.200000",
"bluestore_cache_meta_ratio": "0.800000",
"bluestore_cache_size": "0",
"bluestore_cache_size_hdd": "1073741824",
"bluestore_cache_size_ssd": "3221225472",
"bluestore_cache_trim_interval": "0.050000",
"bluestore_cache_trim_max_skip_pinned": "64",
osd_memory_cache_resize_interval =1.0
"osd_memory_target": "2147483648"
"osd_memory_target_cgroup_limit_ratio": "0.800000",
"bluestore_rocksdb_options": "compression=kNoCompression,max_write_buffer_number=32,min_write_buffer_number_to_merge=2,recycle_log_file_num=4,write_buffer_size=536870912,writable_file_max_buffer_size=0,compaction_readahead_size=2097152",

【run before osd mem】 osd mem is up to 1G.
top - 19:24:20 up 2:45, 7 users, load average: 4.57, 5.16, 7.69
Tasks: 758 total, 4 running, 397 sleeping, 0 stopped, 3 zombie
%Cpu(s): 3.6 us, 4.6 sy, 0.0 ni, 90.7 id, 1.0 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 13173620+total, 91182504 free, 19820964 used, 20732744 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 10999477+avail Mem

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                          
3793 ceph 20 0 1250672 239248 13572 S 1.7 0.2 10:31.82 ceph-mgr
2613446 ceph 20 0 1225696 345268 34060 S 0.7 0.3 0:06.67 ceph-osd
2613586 ceph 20 0 1495700 613356 34320 S 0.7 0.5 0:22.83 ceph-osd
3625 ceph 20 0 752324 252512 15644 S 0.3 0.2 2:33.37 ceph-mon
2613029 ceph 20 0 2452828 1.503g 34356 S 0.3 1.2 1:57.93 ceph-osd
2613387 ceph 20 0 1244060 358188 33760 S 0.3 0.3 0:07.90 ceph-osd
2613414 ceph 20 0 1247464 366336 33480 S 0.3 0.3 0:08.59 ceph-osd
2613417 ceph 20 0 1220576 340036 33340 S 0.3 0.3 0:07.65 ceph-osd
2613430 ceph 20 0 1263524 377076 33204 S 0.3 0.3 0:08.01 ceph-osd

【run after osd mem】 osd mem is up to 8G.
top - 19:37:57 up 2:59, 7 users, load average: 68.90, 20.29, 11.01
Tasks: 754 total, 6 running, 393 sleeping, 0 stopped, 3 zombie
%Cpu(s): 52.2 us, 29.4 sy, 0.0 ni, 0.9 id, 7.9 wa, 0.0 hi, 9.6 si, 0.0 st
KiB Mem : 13173620+total, 33345848 free, 72141024 used, 26249340 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 54638116 avail Mem

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                          
2613029 ceph 20 0 7823372 7.479g 34612 S 342.8 5.2 3:28.60 ceph-osd
2613850 ceph 20 0 6522456 9.347g 34428 S 337.5 4.3 2:11.97 ceph-osd
2613808 ceph 20 0 7243444 8.882g 34712 S 329.6 4.7 2:13.12 ceph-osd
2613787 ceph 20 0 6654084 9.370g 34644 S 305.9 4.3 2:05.99 ceph-osd
2613858 ceph 20 0 5499624 7.271g 34388 S 276.6 3.4 2:07.11 ceph-osd
2613802 ceph 20 0 5949948 6.712g 34232 S 275.3 3.8 2:01.89 ceph-osd
2613856 ceph 20 0 7289896 8.449g 34484 S 269.4 4.3 1:58.32 ceph-osd
2613813 ceph 20 0 5985556 7.277g 34652 S 264.5 3.4 2:17.43 ceph-osd
2613717 ceph 20 0 5596584 9.222g 34584 S 264.1 3.4 2:10.22 ceph-osd
2613851 ceph 20 0 7628248 8.274g 34672 S 240.1 5.0 1:53.80 ceph-osd
2613741 ceph 20 0 6179744 8.043g 34508 S 199.0 4.0 1:44.97 ceph-osd

History

#1 Updated by Igor Fedotov over 3 years ago

This might be related to https://tracker.ceph.com/issues/46658

Could you please collect a mempool dump from an OSD suffering from high memory usage?

#2 Updated by Stellar Wang over 3 years ago

hi lgor,
Thank you for your quick feedback, Osd memory still exceeds the set threshold of 2G when i run again it,and I am confused as to why the OSD Memory Target(osd_memory_target=2G)is not working .
Looking forward to your reply.

mempool dump info as bellow:

{
"mempool": {
"by_pool": {
"bloom_filter": {
"items": 0,
"bytes": 0
},
"bluestore_alloc": {
"items": 1630421,
"bytes": 13043368
},
"bluestore_cache_data": {
"items": 50201,
"bytes": 622333952
},
"bluestore_cache_onode": {
"items": 7282,
"bytes": 4077920
},
"bluestore_cache_other": {
"items": 2233577,
"bytes": 61721151
},
"bluestore_fsck": {
"items": 0,
"bytes": 0
},
"bluestore_txc": {
"items": 57,
"bytes": 41040
},
"bluestore_writing_deferred": {
"items": 44,
"bytes": 176000
},
"bluestore_writing": {
"items": 10341,
"bytes": 7902257240
},
"bluefs": {
"items": 462,
"bytes": 14296
},
"buffer_anon": {
"items": 75272,
"bytes": 1645452384
},
"buffer_meta": {
"items": 84902,
"bytes": 7471376
},
"osd": {
"items": 179,
"bytes": 2122224
},
"osd_mapbl": {
"items": 43,
"bytes": 654588
},
"osd_pglog": {
"items": 643984,
"bytes": 265203561
},
"osdmap": {
"items": 7496,
"bytes": 345208
},
"osdmap_mapping": {
"items": 0,
"bytes": 0
},
"pgmap": {
"items": 0,
"bytes": 0
},
"mds_co": {
"items": 0,
"bytes": 0
},
"unittest_1": {
"items": 0,
"bytes": 0
},
"unittest_2": {
"items": 0,
"bytes": 0
}
},
"total": {
"items": 4744261,
"bytes": 10524914308
}
}
}

#3 Updated by Igor Fedotov over 3 years ago

Does memory consumption stay that high after benchmark is completed/terminated?

#4 Updated by Igor Fedotov over 3 years ago

And could you please try the same benchmark against replicated pool? Would this have the same problem?

#5 Updated by Stellar Wang over 3 years ago

Igor Fedotov wrote:

Does memory consumption stay that high after benchmark is completed/terminated?

Answer: Memory consumption will be reduced to 1G after the test is completed.

#6 Updated by Stellar Wang over 3 years ago

Igor Fedotov wrote:

And could you please try the same benchmark against replicated pool? Would this have the same problem?

Answer: I try to run the same benchmark in the replicated pool, In fact,Osd memory does not exceed the threshold. It's always going to be around 2G. it's normal.

detail as below:

top - 11:05:35 up 1:50, 7 users, load average: 16.45, 51.68, 45.59
Tasks: 769 total, 6 running, 368 sleeping, 0 stopped, 3 zombie
%Cpu(s): 4.2 us, 4.5 sy, 0.0 ni, 90.6 id, 0.6 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 13173620+total, 2841232 free, 44923536 used, 83971440 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 85068760 avail Mem

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                      
262543 ceph 20 0 1262008 261948 26548 S 1.7 0.2 5:24.98 ceph-mgr
263037 ceph 20 0 690800 203588 21504 S 1.7 0.2 1:36.70 ceph-mon
263158 ceph 20 0 1173048 293524 15236 S 0.7 0.2 0:19.62 ceph-osd
263183 ceph 20 0 9.970g 2.001g 16216 S 0.7 1.6 28:31.75 ceph-osd
263214 ceph 20 0 1248680 368536 15616 S 0.7 0.3 0:20.95 ceph-osd
263226 ceph 20 0 9560924 2.040g 16180 S 0.7 1.6 32:30.37 ceph-osd
263232 ceph 20 0 11.031g 2.003g 16028 S 0.7 1.6 32:32.75 ceph-osd
263235 ceph 20 0 11.650g 2.035g 16336 S 0.7 1.6 42:22.93 ceph-osd
263245 ceph 20 0 8648968 2.017g 16440 S 0.7 1.6 34:45.08 ceph-osd
263166 ceph 20 0 8234032 2.041g 16236 S 0.3 1.6 29:29.31 ceph-osd
263187 ceph 20 0 3877488 2.037g 16020 S 0.3 1.6 43:59.86 ceph-osd
263189 ceph 20 0 13.908g 2.045g 16160 S 0.3 1.6 34:59.97 ceph-osd
263195 ceph 20 0 14.053g 2.022g 16028 S 0.3 1.6 37:35.70 ceph-osd
263202 ceph 20 0 1168920 289740 15772 S 0.3 0.2 0:19.10 ceph-osd
263204 ceph 20 0 11.286g 2.037g 15764 S 0.3 1.6 23:30.76 ceph-osd

During the run, the mempool dump information as follows: {
"mempool": {
"by_pool": {
"bloom_filter": {
"items": 0,
"bytes": 0
},
"bluestore_alloc": {
"items": 1630421,
"bytes": 13043368
},
"bluestore_cache_data": {
"items": 0,
"bytes": 0
},
"bluestore_cache_onode": {
"items": 81529,
"bytes": 45656240
},
"bluestore_cache_other": {
"items": 18384298,
"bytes": 534364445
},
"bluestore_fsck": {
"items": 0,
"bytes": 0
},
"bluestore_txc": {
"items": 23,
"bytes": 16560
},
"bluestore_writing_deferred": {
"items": 32,
"bytes": 128000
},
"bluestore_writing": {
"items": 162,
"bytes": 2128537
},
"bluefs": {
"items": 705,
"bytes": 19408
},
"buffer_anon": {
"items": 48708,
"bytes": 4514317
},
"buffer_meta": {
"items": 635,
"bytes": 55880
},
"osd": {
"items": 225,
"bytes": 2667600
},
"osd_mapbl": {
"items": 54,
"bytes": 877903
},
"osd_pglog": {
"items": 759306,
"bytes": 307798844
},
"osdmap": {
"items": 9266,
"bytes": 448416
},
"osdmap_mapping": {
"items": 0,
"bytes": 0
},
"pgmap": {
"items": 0,
"bytes": 0
},
"mds_co": {
"items": 0,
"bytes": 0
},
"unittest_1": {
"items": 0,
"bytes": 0
},
"unittest_2": {
"items": 0,
"bytes": 0
}
},
"total": {
"items": 20915364,
"bytes": 911719518
}
}
}

#7 Updated by Stellar Wang over 3 years ago

hi Igor,

I do the following today.
  1. Adjust rocksDB parameters (max_write_buffer_number=4,write_buffer_size=128M,min_write_buffer_number_to_merge=1), then run FIO, OSD memory will still grow to 9G.
  2. Adjust BS =1m, then run FIO, osD memory is maintained at about 2G, and make sure that only 4K random write will show a rapid increase in OSD memory.

Question:
In mempool dump information, bluestore_writing takes up most of the memory. How is this part of memory allocated? How should I manage this memory?

#8 Updated by Igor Fedotov over 3 years ago

So a short summary for now is:
1) High memory consumption is just temporary and goes away on writing benchmark completion.
2) EC pools suffer from the issue only.

Is the above correct?

And intermediate root cause analysis is:
Looks like pending writes take plenty of RAM in cache (7.9 GB):
"bluestore_writing": {
"items": 10341,
"bytes": 7902257240
},

It's expected that they go away when writes are completed. And this happens indeed. But an open question is why there are that large amount of them cached during the benchmark. Still investigating..

#9 Updated by Igor Fedotov over 3 years ago

鑫 王 wrote:

Question:
In mempool dump information, bluestore_writing takes up most of the memory. How is this part of memory allocated? How should I manage this memory?

bluestore_writing tracks the data kept in cache while data writing is in progress (submitted to disk but hasn't yet got a commit ack). So the only way to control that is to throttle amount of write requests coming to BlueStore. But it's still not clear to me why there are that many data coming in at a time. Looks like this is related to partial and unaligned(!) data overwrites caused by 4K write block size.

#10 Updated by Igor Fedotov over 3 years ago

Would you please collect perf counter dumps for both running benchmark (e.g. in the middle of it) and on its completion.
Please reset perf counters prior to running the benchmark. Thanks!

#11 Updated by Stellar Wang over 3 years ago

Igor Fedotov wrote:

So a short summary for now is:
1) High memory consumption is just temporary and goes away on writing benchmark completion.
2) EC pools suffer from the issue only.

Is the above correct?

Answer:
yes,Everything is right.
If the benchmark takes a long time (2 minutes), all OSDS will be down due to memory exhaustion.
If the benchmark time is short (20 seconds), then all OSDS will rise to 8G and then gradually return to normal.

In addition, Can you tell me which fields you care about perf counters.

#12 Updated by Igor Fedotov over 3 years ago

鑫 王 wrote:

In addition, Can you tell me which fields you care about perf counters.

Everything under "bluestore", "mempool" and "osd" sections please

#13 Updated by Stellar Wang over 3 years ago

Igor Fedotov wrote:

Would you please collect perf counter dumps for both running benchmark (e.g. ) and on its completion.
Please reset perf counters prior to running the benchmark. Thanks!

Perf counter information as bellow:

  • in the middle of it

    {
    "AsyncMessenger::Worker-0": {
    "msgr_recv_messages": 14758,
    "msgr_send_messages": 22504,
    "msgr_recv_bytes": 4006382430,
    "msgr_send_bytes": 8011235283,
    "msgr_created_connections": 0,
    "msgr_active_connections": 0,
    "msgr_running_total_time": 35.326263657,
    "msgr_running_send_time": 17.459648872,
    "msgr_running_recv_time": 15.989741981,
    "msgr_running_fast_dispatch_time": 1.045755922
    },
    "AsyncMessenger::Worker-1": {
    "msgr_recv_messages": 1590,
    "msgr_send_messages": 1517,
    "msgr_recv_bytes": 880862608,
    "msgr_send_bytes": 34037602,
    "msgr_created_connections": 0,
    "msgr_active_connections": 0,
    "msgr_running_total_time": 4.108869601,
    "msgr_running_send_time": 0.172507468,
    "msgr_running_recv_time": 3.296442043,
    "msgr_running_fast_dispatch_time": 0.385720106
    },
    "AsyncMessenger::Worker-2": {
    "msgr_recv_messages": 5637,
    "msgr_send_messages": 8058,
    "msgr_recv_bytes": 2645586847,
    "msgr_send_bytes": 2882936769,
    "msgr_created_connections": 0,
    "msgr_active_connections": 0,
    "msgr_running_total_time": 20.693582115,
    "msgr_running_send_time": 6.579921501,
    "msgr_running_recv_time": 11.843648013,
    "msgr_running_fast_dispatch_time": 1.371034067
    },
    "AsyncMessenger::Worker-3": {
    "msgr_recv_messages": 45492,
    "msgr_send_messages": 12732,
    "msgr_recv_bytes": 1491482041,
    "msgr_send_bytes": 2857232853,
    "msgr_created_connections": 1,
    "msgr_active_connections": 1,
    "msgr_running_total_time": 21.590310825,
    "msgr_running_send_time": 7.769481173,
    "msgr_running_recv_time": 11.500933975,
    "msgr_running_fast_dispatch_time": 1.595340457
    },
    "AsyncMessenger::Worker-4": {
    "msgr_recv_messages": 2977,
    "msgr_send_messages": 3114,
    "msgr_recv_bytes": 31622474,
    "msgr_send_bytes": 1330616640,
    "msgr_created_connections": 0,
    "msgr_active_connections": 0,
    "msgr_running_total_time": 4.060999709,
    "msgr_running_send_time": 2.430382381,
    "msgr_running_recv_time": 0.575906881,
    "msgr_running_fast_dispatch_time": 0.131844898
    },
    "bluefs": {
    "gift_bytes": 0,
    "reclaim_bytes": 0,
    "db_total_bytes": 16003497984,
    "db_used_bytes": 1019543552,
    "wal_total_bytes": 0,
    "wal_used_bytes": 0,
    "slow_total_bytes": 0,
    "slow_used_bytes": 0,
    "num_files": 23,
    "log_bytes": 58757120,
    "log_compactions": 0,
    "logged_bytes": 3981312,
    "files_written_wal": 0,
    "files_written_sst": 0,
    "bytes_written_wal": 69475425,
    "bytes_written_sst": 0,
    "bytes_written_slow": 0,
    "max_bytes_wal": 0,
    "max_bytes_db": 1019805696,
    "max_bytes_slow": 0,
    "read_random_count": 124,
    "read_random_bytes": 73119933,
    "read_random_disk_count": 124,
    "read_random_disk_bytes": 73119933,
    "read_random_buffer_count": 0,
    "read_random_buffer_bytes": 0,
    "read_count": 0,
    "read_bytes": 0,
    "read_prefetch_count": 0,
    "read_prefetch_bytes": 0
    },
    "bluestore": {
    "kv_flush_lat": {
    "avgcount": 972,
    "sum": 7.212361922,
    "avgtime": 0.007420125
    },
    "kv_commit_lat": {
    "avgcount": 972,
    "sum": 34.007326339,
    "avgtime": 0.034986961
    },
    "kv_sync_lat": {
    "avgcount": 972,
    "sum": 41.219688261,
    "avgtime": 0.042407086
    },
    "kv_final_lat": {
    "avgcount": 964,
    "sum": 3.580052252,
    "avgtime": 0.003713747
    },
    "state_prepare_lat": {
    "avgcount": 21949,
    "sum": 106.935298334,
    "avgtime": 0.004871989
    },
    "state_aio_wait_lat": {
    "avgcount": 21941,
    "sum": 415.447180789,
    "avgtime": 0.018934742
    },
    "state_io_done_lat": {
    "avgcount": 21941,
    "sum": 7.263685675,
    "avgtime": 0.000331055
    },
    "state_kv_queued_lat": {
    "avgcount": 21932,
    "sum": 1191.686464177,
    "avgtime": 0.054335512
    },
    "state_kv_commiting_lat": {
    "avgcount": 21868,
    "sum": 1254.366584940,
    "avgtime": 0.057360827
    },
    "state_kv_done_lat": {
    "avgcount": 21868,
    "sum": 0.013641341,
    "avgtime": 0.000000623
    },
    "state_deferred_queued_lat": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "state_deferred_aio_wait_lat": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "state_deferred_cleanup_lat": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "state_finishing_lat": {
    "avgcount": 21866,
    "sum": 0.007742956,
    "avgtime": 0.000000354
    },
    "state_done_lat": {
    "avgcount": 21866,
    "sum": 2.859099240,
    "avgtime": 0.000130755
    },
    "throttle_lat": {
    "avgcount": 21949,
    "sum": 43.276581949,
    "avgtime": 0.001971688
    },
    "submit_lat": {
    "avgcount": 21949,
    "sum": 114.751779990,
    "avgtime": 0.005228109
    },
    "commit_lat": {
    "avgcount": 21868,
    "sum": 2971.108660658,
    "avgtime": 0.135865587
    },
    "read_lat": {
    "avgcount": 12119,
    "sum": 117.191786744,
    "avgtime": 0.009670087
    },
    "read_onode_meta_lat": {
    "avgcount": 24250,
    "sum": 0.564981240,
    "avgtime": 0.000023298
    },
    "read_wait_aio_lat": {
    "avgcount": 12119,
    "sum": 114.847844956,
    "avgtime": 0.009476676
    },
    "compress_lat": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "decompress_lat": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "csum_lat": {
    "avgcount": 12878,
    "sum": 0.951551251,
    "avgtime": 0.000073889
    },
    "compress_success_count": 0,
    "compress_rejected_count": 0,
    "write_pad_bytes": 23014,
    "deferred_write_ops": 0,
    "deferred_write_bytes": 0,
    "write_penalty_read_ops": 0,
    "bluestore_allocated": 206216097792,
    "bluestore_stored": 206637715913,
    "bluestore_compressed": 0,
    "bluestore_compressed_allocated": 0,
    "bluestore_compressed_original": 0,
    "bluestore_onodes": 12179,
    "bluestore_onode_hits": 53316,
    "bluestore_onode_misses": 31431,
    "bluestore_onode_shard_hits": 57796,
    "bluestore_onode_shard_misses": 5313,
    "bluestore_extents": 137858,
    "bluestore_blobs": 137904,
    "bluestore_buffers": 894,
    "bluestore_buffer_bytes": 58589184,
    "bluestore_buffer_hit_bytes": 32954,
    "bluestore_buffer_miss_bytes": 844365824,
    "bluestore_write_big": 34364,
    "bluestore_write_big_bytes": 15228100608,
    "bluestore_write_big_blobs": 232364,
    "bluestore_write_small": 8,
    "bluestore_write_small_bytes": 9754,
    "bluestore_write_small_unused": 0,
    "bluestore_write_small_deferred": 4,
    "bluestore_write_small_pre_read": 4,
    "bluestore_write_small_new": 232368,
    "bluestore_txc": 21949,
    "bluestore_onode_reshard": 7635,
    "bluestore_blob_split": 0,
    "bluestore_extent_compress": 0,
    "bluestore_gc_merged": 0,
    "bluestore_read_eio": 0,
    "bluestore_reads_with_retries": 0,
    "bluestore_fragmentation_micros": 95,
    "omap_seek_to_first_lat": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "omap_upper_bound_lat": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "omap_lower_bound_lat": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "omap_next_lat": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "clist_lat": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "cct": {
    "total_workers": 17,
    "unhealthy_workers": 0
    },
    "finisher-commit_finisher": {
    "queue_len": 0,
    "complete_latency": {
    "avgcount": 2,
    "sum": 0.002241235,
    "avgtime": 0.001120617
    }
    },
    "finisher-defered_finisher": {
    "queue_len": 0,
    "complete_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "finisher-objecter-finisher-0": {
    "queue_len": 0,
    "complete_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "mempool": {
    "bloom_filter_bytes": 0,
    "bloom_filter_items": 0,
    "bluestore_alloc_bytes": 13043368,
    "bluestore_alloc_items": 1630421,
    "bluestore_cache_data_bytes": 509865984,
    "bluestore_cache_data_items": 8379,
    "bluestore_cache_onode_bytes": 6781040,
    "bluestore_cache_onode_items": 12109,
    "bluestore_cache_other_bytes": 60716031,
    "bluestore_cache_other_items": 2415100,
    "bluestore_fsck_bytes": 0,
    "bluestore_fsck_items": 0,
    "bluestore_txc_bytes": 200160,
    "bluestore_txc_items": 278,
    "bluestore_writing_deferred_bytes": 64000,
    "bluestore_writing_deferred_items": 16,
    "bluestore_writing_bytes": 7426528761,
    "bluestore_writing_items": 8704,
    "bluefs_bytes": 14752,
    "bluefs_items": 500,
    "buffer_anon_bytes": 1627593673,
    "buffer_anon_items": 164359,
    "buffer_meta_bytes": 5088600,
    "buffer_meta_items": 57825,
    "osd_bytes": 2240784,
    "osd_items": 189,
    "osd_mapbl_bytes": 582061,
    "osd_mapbl_items": 40,
    "osd_pglog_bytes": 263463683,
    "osd_pglog_items": 628269,
    "osdmap_bytes": 303112,
    "osdmap_items": 6320,
    "osdmap_mapping_bytes": 0,
    "osdmap_mapping_items": 0,
    "pgmap_bytes": 0,
    "pgmap_items": 0,
    "mds_co_bytes": 0,
    "mds_co_items": 0,
    "unittest_1_bytes": 0,
    "unittest_1_items": 0,
    "unittest_2_bytes": 0,
    "unittest_2_items": 0
    },
    "objecter": {
    "op_active": 0,
    "op_laggy": 0,
    "op_send": 0,
    "op_send_bytes": 0,
    "op_resend": 0,
    "op_reply": 0,
    "op": 0,
    "op_r": 0,
    "op_w": 0,
    "op_rmw": 0,
    "op_pg": 0,
    "osdop_stat": 0,
    "osdop_create": 0,
    "osdop_read": 0,
    "osdop_write": 0,
    "osdop_writefull": 0,
    "osdop_writesame": 0,
    "osdop_append": 0,
    "osdop_zero": 0,
    "osdop_truncate": 0,
    "osdop_delete": 0,
    "osdop_mapext": 0,
    "osdop_sparse_read": 0,
    "osdop_clonerange": 0,
    "osdop_getxattr": 0,
    "osdop_setxattr": 0,
    "osdop_cmpxattr": 0,
    "osdop_rmxattr": 0,
    "osdop_resetxattrs": 0,
    "osdop_call": 0,
    "osdop_watch": 0,
    "osdop_notify": 0,
    "osdop_src_cmpxattr": 0,
    "osdop_pgls": 0,
    "osdop_pgls_filter": 0,
    "osdop_other": 0,
    "linger_active": 0,
    "linger_send": 0,
    "linger_resend": 0,
    "linger_ping": 0,
    "poolop_active": 0,
    "poolop_send": 0,
    "poolop_resend": 0,
    "poolstat_active": 0,
    "poolstat_send": 0,
    "poolstat_resend": 0,
    "statfs_active": 0,
    "statfs_send": 0,
    "statfs_resend": 0,
    "command_active": 0,
    "command_send": 0,
    "command_resend": 0,
    "map_epoch": 1591,
    "map_full": 0,
    "map_inc": 2,
    "osd_sessions": 0,
    "osd_session_open": 0,
    "osd_session_close": 0,
    "osd_laggy": 0,
    "omap_wr": 0,
    "omap_rd": 0,
    "omap_del": 0
    },
    "osd": {
    "op_wip": 16352,
    "op": 3969,
    "op_in_bytes": 31088640,
    "op_out_bytes": 0,
    "op_latency": {
    "avgcount": 3969,
    "sum": 88819.937374107,
    "avgtime": 22.378417075
    },
    "op_process_latency": {
    "avgcount": 3969,
    "sum": 48246.655352583,
    "avgtime": 12.155871844
    },
    "op_prepare_latency": {
    "avgcount": 20321,
    "sum": 103.296337888,
    "avgtime": 0.005083231
    },
    "op_r": 0,
    "op_r_out_bytes": 0,
    "op_r_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "op_r_process_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "op_r_prepare_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "op_w": 3969,
    "op_w_in_bytes": 31088640,
    "op_w_latency": {
    "avgcount": 3969,
    "sum": 88819.937374107,
    "avgtime": 22.378417075
    },
    "op_w_process_latency": {
    "avgcount": 3969,
    "sum": 48246.655352583,
    "avgtime": 12.155871844
    },
    "op_w_prepare_latency": {
    "avgcount": 20321,
    "sum": 103.296337888,
    "avgtime": 0.005083231
    },
    "op_rw": 0,
    "op_rw_in_bytes": 0,
    "op_rw_out_bytes": 0,
    "op_rw_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "op_rw_process_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "op_rw_prepare_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "op_before_queue_op_lat": {
    "avgcount": 81250,
    "sum": 21.512183017,
    "avgtime": 0.000264765
    },
    "op_before_dequeue_op_lat": {
    "avgcount": 61205,
    "sum": 166353.690807047,
    "avgtime": 2.717975505
    },
    "subop": 0,
    "subop_in_bytes": 0,
    "subop_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "subop_w": 0,
    "subop_w_in_bytes": 0,
    "subop_w_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "subop_pull": 0,
    "subop_pull_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "subop_push": 0,
    "subop_push_in_bytes": 0,
    "subop_push_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "pull": 0,
    "push": 0,
    "push_out_bytes": 0,
    "recovery_ops": 0,
    "recovery_bytes": 0,
    "loadavg": 7573,
    "cached_crc": 0,
    "cached_crc_adjusted": 0,
    "missed_crc": 0,
    "numpg": 189,
    "numpg_primary": 75,
    "numpg_replica": 114,
    "numpg_stray": 0,
    "numpg_removing": 0,
    "heartbeat_to_peers": 20,
    "map_messages": 13,
    "map_message_epochs": 20,
    "map_message_epoch_dups": 18,
    "messages_delayed_for_map": 0,
    "osd_map_cache_hit": 382,
    "osd_map_cache_miss": 0,
    "osd_map_cache_miss_low": 0,
    "osd_map_cache_miss_low_avg": {
    "avgcount": 0,
    "sum": 0
    },
    "osd_map_bl_cache_hit": 19,
    "osd_map_bl_cache_miss": 4,
    "stat_bytes": 400086269952,
    "stat_bytes_used": 207091470336,
    "stat_bytes_avail": 192994799616,
    "copyfrom": 0,
    "tier_promote": 0,
    "tier_flush": 0,
    "tier_flush_fail": 0,
    "tier_try_flush": 0,
    "tier_try_flush_fail": 0,
    "tier_evict": 0,
    "tier_whiteout": 0,
    "tier_dirty": 3640,
    "tier_clean": 0,
    "tier_delay": 0,
    "tier_proxy_read": 0,
    "tier_proxy_write": 0,
    "agent_wake": 0,
    "agent_skip": 0,
    "agent_flush": 0,
    "agent_evict": 0,
    "object_ctx_cache_hit": 15280,
    "object_ctx_cache_total": 20321,
    "op_cache_hit": 0,
    "osd_tier_flush_lat": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "osd_tier_promote_lat": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "osd_tier_r_lat": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "osd_pg_info": 21947,
    "osd_pg_fastinfo": 20730,
    "osd_pg_biginfo": 0
    },
    "prioritycache": {
    "target_bytes": 2147483648,
    "mapped_bytes": 11144265728,
    "unmapped_bytes": 120446976,
    "heap_bytes": 11264712704,
    "cache_bytes": 134217728
    },
    "prioritycache:data": {
    "pri0_bytes": 0,
    "pri1_bytes": 0,
    "pri2_bytes": 0,
    "pri3_bytes": 0,
    "pri4_bytes": 0,
    "pri5_bytes": 0,
    "pri6_bytes": 0,
    "pri7_bytes": 0,
    "pri8_bytes": 0,
    "pri9_bytes": 0,
    "pri10_bytes": 0,
    "pri11_bytes": 0,
    "reserved_bytes": 67108864,
    "committed_bytes": 67108864
    },
    "prioritycache:kv": {
    "pri0_bytes": 0,
    "pri1_bytes": 0,
    "pri2_bytes": 0,
    "pri3_bytes": 0,
    "pri4_bytes": 0,
    "pri5_bytes": 0,
    "pri6_bytes": 0,
    "pri7_bytes": 0,
    "pri8_bytes": 0,
    "pri9_bytes": 0,
    "pri10_bytes": 0,
    "pri11_bytes": 0,
    "reserved_bytes": 67108864,
    "committed_bytes": 67108864
    },
    "prioritycache:meta": {
    "pri0_bytes": 0,
    "pri1_bytes": 0,
    "pri2_bytes": 0,
    "pri3_bytes": 0,
    "pri4_bytes": 0,
    "pri5_bytes": 0,
    "pri6_bytes": 0,
    "pri7_bytes": 0,
    "pri8_bytes": 0,
    "pri9_bytes": 0,
    "pri10_bytes": 0,
    "pri11_bytes": 0,
    "reserved_bytes": 67108864,
    "committed_bytes": 67108864
    },
    "recoverystate_perf": {
    "initial_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "started_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "reset_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "start_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "primary_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "peering_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "backfilling_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "waitremotebackfillreserved_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "waitlocalbackfillreserved_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "notbackfilling_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "repnotrecovering_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "repwaitrecoveryreserved_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "repwaitbackfillreserved_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "reprecovering_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "activating_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "waitlocalrecoveryreserved_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "waitremoterecoveryreserved_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "recovering_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "recovered_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "clean_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "active_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "replicaactive_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "stray_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "getinfo_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "getlog_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "waitactingchange_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "incomplete_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "down_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "getmissing_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "waitupthru_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "notrecovering_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "rocksdb": {
    "get": 37534,
    "submit_transaction": 21932,
    "submit_transaction_sync": 972,
    "get_latency": {
    "avgcount": 37534,
    "sum": 4.097644331,
    "avgtime": 0.000109171
    },
    "submit_latency": {
    "avgcount": 21932,
    "sum": 3.275299821,
    "avgtime": 0.000149338
    },
    "submit_sync_latency": {
    "avgcount": 972,
    "sum": 30.205914873,
    "avgtime": 0.031076044
    },
    "compact": 0,
    "compact_range": 0,
    "compact_queue_merge": 0,
    "compact_queue_len": 0,
    "rocksdb_write_wal_time": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "rocksdb_write_memtable_time": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "rocksdb_write_delay_time": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "rocksdb_write_pre_and_post_time": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "throttle-bluestore_throttle_bytes": {
    "val": 9506665,
    "max": 67108864,
    "get_started": 21949,
    "get": 21949,
    "get_sum": 16356110333,
    "get_or_fail_fail": 0,
    "get_or_fail_success": 0,
    "take": 0,
    "take_sum": 0,
    "put": 973,
    "put_sum": 16346603668,
    "wait": {
    "avgcount": 1796,
    "sum": 42.779978509,
    "avgtime": 0.023819587
    }
    },
    "throttle-bluestore_throttle_deferred_bytes": {
    "val": 135220,
    "max": 201326592,
    "get_started": 0,
    "get": 2,
    "get_sum": 67610,
    "get_or_fail_fail": 0,
    "get_or_fail_success": 2,
    "take": 0,
    "take_sum": 0,
    "put": 0,
    "put_sum": 0,
    "wait": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "throttle-msgr_dispatch_throttler-client": {
    "val": 0,
    "max": 104857600,
    "get_started": 0,
    "get": 37509,
    "get_sum": 421801498,
    "get_or_fail_fail": 0,
    "get_or_fail_success": 37509,
    "take": 0,
    "take_sum": 0,
    "put": 37509,
    "put_sum": 421801498,
    "wait": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "throttle-msgr_dispatch_throttler-cluster": {
    "val": 1773931,
    "max": 104857600,
    "get_started": 0,
    "get": 31675,
    "get_sum": 8628158173,
    "get_or_fail_fail": 0,
    "get_or_fail_success": 31675,
    "take": 0,
    "take_sum": 0,
    "put": 31673,
    "put_sum": 8626384242,
    "wait": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "throttle-msgr_dispatch_throttler-hb_back_client": {
    "val": 0,
    "max": 104857600,
    "get_started": 0,
    "get": 320,
    "get_sum": 641280,
    "get_or_fail_fail": 0,
    "get_or_fail_success": 320,
    "take": 0,
    "take_sum": 0,
    "put": 320,
    "put_sum": 641280,
    "wait": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "throttle-msgr_dispatch_throttler-hb_back_server": {
    "val": 0,
    "max": 104857600,
    "get_started": 0,
    "get": 313,
    "get_sum": 627252,
    "get_or_fail_fail": 0,
    "get_or_fail_success": 313,
    "take": 0,
    "take_sum": 0,
    "put": 313,
    "put_sum": 627252,
    "wait": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "throttle-msgr_dispatch_throttler-hb_front_client": {
    "val": 0,
    "max": 104857600,
    "get_started": 0,
    "get": 320,
    "get_sum": 641280,
    "get_or_fail_fail": 0,
    "get_or_fail_success": 320,
    "take": 0,
    "take_sum": 0,
    "put": 320,
    "put_sum": 641280,
    "wait": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "throttle-msgr_dispatch_throttler-hb_front_server": {
    "val": 0,
    "max": 104857600,
    "get_started": 0,
    "get": 313,
    "get_sum": 627252,
    "get_or_fail_fail": 0,
    "get_or_fail_success": 313,
    "take": 0,
    "take_sum": 0,
    "put": 313,
    "put_sum": 627252,
    "wait": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "throttle-msgr_dispatch_throttler-ms_objecter": {
    "val": 0,
    "max": 104857600,
    "get_started": 0,
    "get": 0,
    "get_sum": 0,
    "get_or_fail_fail": 0,
    "get_or_fail_success": 0,
    "take": 0,
    "take_sum": 0,
    "put": 0,
    "put_sum": 0,
    "wait": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "throttle-objecter_bytes": {
    "val": 0,
    "max": 104857600,
    "get_started": 0,
    "get": 0,
    "get_sum": 0,
    "get_or_fail_fail": 0,
    "get_or_fail_success": 0,
    "take": 0,
    "take_sum": 0,
    "put": 0,
    "put_sum": 0,
    "wait": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "throttle-objecter_ops": {
    "val": 0,
    "max": 1024,
    "get_started": 0,
    "get": 0,
    "get_sum": 0,
    "get_or_fail_fail": 0,
    "get_or_fail_success": 0,
    "take": 0,
    "take_sum": 0,
    "put": 0,
    "put_sum": 0,
    "wait": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "throttle-osd_client_bytes": {
    "val": 385065705,
    "max": 524288000,
    "get_started": 0,
    "get": 37508,
    "get_sum": 421801218,
    "get_or_fail_fail": 0,
    "get_or_fail_success": 37508,
    "take": 0,
    "take_sum": 0,
    "put": 24290,
    "put_sum": 36735513,
    "wait": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    }
    }

  • After the benchmark is complete

    {
    "AsyncMessenger::Worker-0": {
    "msgr_recv_messages": 93709,
    "msgr_send_messages": 93700,
    "msgr_recv_bytes": 11068431529,
    "msgr_send_bytes": 14100625759,
    "msgr_created_connections": 1,
    "msgr_active_connections": 1,
    "msgr_running_total_time": 52.242010513,
    "msgr_running_send_time": 23.224253302,
    "msgr_running_recv_time": 24.047067771,
    "msgr_running_fast_dispatch_time": 2.298904292
    },
    "AsyncMessenger::Worker-1": {
    "msgr_recv_messages": 9739,
    "msgr_send_messages": 9740,
    "msgr_recv_bytes": 2129161753,
    "msgr_send_bytes": 318287910,
    "msgr_created_connections": 0,
    "msgr_active_connections": 0,
    "msgr_running_total_time": 6.867824457,
    "msgr_running_send_time": 0.672000304,
    "msgr_running_recv_time": 5.186834303,
    "msgr_running_fast_dispatch_time": 0.528367866
    },
    "AsyncMessenger::Worker-2": {
    "msgr_recv_messages": 58835,
    "msgr_send_messages": 58836,
    "msgr_recv_bytes": 8046397329,
    "msgr_send_bytes": 7627727787,
    "msgr_created_connections": 0,
    "msgr_active_connections": 0,
    "msgr_running_total_time": 33.727499297,
    "msgr_running_send_time": 10.984159180,
    "msgr_running_recv_time": 18.187015550,
    "msgr_running_fast_dispatch_time": 2.292816960
    },
    "AsyncMessenger::Worker-3": {
    "msgr_recv_messages": 64084,
    "msgr_send_messages": 64153,
    "msgr_recv_bytes": 2547263233,
    "msgr_send_bytes": 4934369949,
    "msgr_created_connections": 1,
    "msgr_active_connections": 0,
    "msgr_running_total_time": 27.719357373,
    "msgr_running_send_time": 11.284186561,
    "msgr_running_recv_time": 13.054635493,
    "msgr_running_fast_dispatch_time": 1.876878212
    },
    "AsyncMessenger::Worker-4": {
    "msgr_recv_messages": 9950,
    "msgr_send_messages": 9950,
    "msgr_recv_bytes": 326211849,
    "msgr_send_bytes": 2195991170,
    "msgr_created_connections": 0,
    "msgr_active_connections": 0,
    "msgr_running_total_time": 5.727147232,
    "msgr_running_send_time": 3.258781141,
    "msgr_running_recv_time": 1.043309708,
    "msgr_running_fast_dispatch_time": 0.213028922
    },
    "bluefs": {
    "gift_bytes": 0,
    "reclaim_bytes": 0,
    "db_total_bytes": 16003497984,
    "db_used_bytes": 1339031552,
    "wal_total_bytes": 0,
    "wal_used_bytes": 0,
    "slow_total_bytes": 0,
    "slow_used_bytes": 0,
    "num_files": 24,
    "log_bytes": 48410624,
    "log_compactions": 1,
    "logged_bytes": 76066816,
    "files_written_wal": 1,
    "files_written_sst": 0,
    "bytes_written_wal": 437130742,
    "bytes_written_sst": 0,
    "bytes_written_slow": 0,
    "max_bytes_wal": 0,
    "max_bytes_db": 1339031552,
    "max_bytes_slow": 0,
    "read_random_count": 495,
    "read_random_bytes": 340714457,
    "read_random_disk_count": 496,
    "read_random_disk_bytes": 340714457,
    "read_random_buffer_count": 0,
    "read_random_buffer_bytes": 0,
    "read_count": 0,
    "read_bytes": 0,
    "read_prefetch_count": 0,
    "read_prefetch_bytes": 0
    },
    "bluestore": {
    "kv_flush_lat": {
    "avgcount": 18569,
    "sum": 23.946921628,
    "avgtime": 0.001289618
    },
    "kv_commit_lat": {
    "avgcount": 18569,
    "sum": 95.027560921,
    "avgtime": 0.005117537
    },
    "kv_sync_lat": {
    "avgcount": 18569,
    "sum": 118.974482549,
    "avgtime": 0.006407156
    },
    "kv_final_lat": {
    "avgcount": 18477,
    "sum": 7.116110377,
    "avgtime": 0.000385133
    },
    "state_prepare_lat": {
    "avgcount": 111606,
    "sum": 143.020081704,
    "avgtime": 0.001281473
    },
    "state_aio_wait_lat": {
    "avgcount": 111606,
    "sum": 658.037411255,
    "avgtime": 0.005896075
    },
    "state_io_done_lat": {
    "avgcount": 111606,
    "sum": 9.176486553,
    "avgtime": 0.000082222
    },
    "state_kv_queued_lat": {
    "avgcount": 111606,
    "sum": 2075.907554734,
    "avgtime": 0.018600322
    },
    "state_kv_commiting_lat": {
    "avgcount": 111606,
    "sum": 2023.806329050,
    "avgtime": 0.018133490
    },
    "state_kv_done_lat": {
    "avgcount": 111606,
    "sum": 0.047261486,
    "avgtime": 0.000000423
    },
    "state_deferred_queued_lat": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "state_deferred_aio_wait_lat": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "state_deferred_cleanup_lat": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "state_finishing_lat": {
    "avgcount": 111604,
    "sum": 0.023028613,
    "avgtime": 0.000000206
    },
    "state_done_lat": {
    "avgcount": 111604,
    "sum": 4.955921038,
    "avgtime": 0.000044406
    },
    "throttle_lat": {
    "avgcount": 111606,
    "sum": 43.804460676,
    "avgtime": 0.000392491
    },
    "submit_lat": {
    "avgcount": 111606,
    "sum": 155.268066084,
    "avgtime": 0.001391216
    },
    "commit_lat": {
    "avgcount": 111606,
    "sum": 4909.975184211,
    "avgtime": 0.043993828
    },
    "read_lat": {
    "avgcount": 73485,
    "sum": 304.253162428,
    "avgtime": 0.004140343
    },
    "read_onode_meta_lat": {
    "avgcount": 146970,
    "sum": 2.574883082,
    "avgtime": 0.000017519
    },
    "read_wait_aio_lat": {
    "avgcount": 73485,
    "sum": 296.737060324,
    "avgtime": 0.004038063
    },
    "compress_lat": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "decompress_lat": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "csum_lat": {
    "avgcount": 79180,
    "sum": 2.505528845,
    "avgtime": 0.000031643
    },
    "compress_success_count": 0,
    "compress_rejected_count": 0,
    "write_pad_bytes": 23014,
    "deferred_write_ops": 0,
    "deferred_write_bytes": 0,
    "write_penalty_read_ops": 0,
    "bluestore_allocated": 192451899392,
    "bluestore_stored": 192873517513,
    "bluestore_compressed": 0,
    "bluestore_compressed_allocated": 0,
    "bluestore_compressed_original": 0,
    "bluestore_onodes": 42523,
    "bluestore_onode_hits": 343277,
    "bluestore_onode_misses": 253905,
    "bluestore_onode_shard_hits": 440651,
    "bluestore_onode_shard_misses": 173892,
    "bluestore_extents": 120654,
    "bluestore_blobs": 120654,
    "bluestore_buffers": 0,
    "bluestore_buffer_bytes": 0,
    "bluestore_buffer_hit_bytes": 32954,
    "bluestore_buffer_miss_bytes": 5189140480,
    "bluestore_write_big": 213888,
    "bluestore_write_big_bytes": 34151686144,
    "bluestore_write_big_blobs": 521115,
    "bluestore_write_small": 8,
    "bluestore_write_small_bytes": 9754,
    "bluestore_write_small_unused": 0,
    "bluestore_write_small_deferred": 4,
    "bluestore_write_small_pre_read": 4,
    "bluestore_write_small_new": 521119,
    "bluestore_txc": 111606,
    "bluestore_onode_reshard": 11336,
    "bluestore_blob_split": 0,
    "bluestore_extent_compress": 0,
    "bluestore_gc_merged": 0,
    "bluestore_read_eio": 0,
    "bluestore_reads_with_retries": 0,
    "bluestore_fragmentation_micros": 137,
    "omap_seek_to_first_lat": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "omap_upper_bound_lat": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "omap_lower_bound_lat": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "omap_next_lat": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "clist_lat": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "cct": {
    "total_workers": 17,
    "unhealthy_workers": 0
    },
    "finisher-commit_finisher": {
    "queue_len": 0,
    "complete_latency": {
    "avgcount": 2,
    "sum": 0.002241235,
    "avgtime": 0.001120617
    }
    },
    "finisher-defered_finisher": {
    "queue_len": 0,
    "complete_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "finisher-objecter-finisher-0": {
    "queue_len": 0,
    "complete_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "mempool": {
    "bloom_filter_bytes": 0,
    "bloom_filter_items": 0,
    "bluestore_alloc_bytes": 13043368,
    "bluestore_alloc_items": 1630421,
    "bluestore_cache_data_bytes": 0,
    "bluestore_cache_data_items": 0,
    "bluestore_cache_onode_bytes": 23812880,
    "bluestore_cache_onode_items": 42523,
    "bluestore_cache_other_bytes": 54818945,
    "bluestore_cache_other_items": 5165340,
    "bluestore_fsck_bytes": 0,
    "bluestore_fsck_items": 0,
    "bluestore_txc_bytes": 2880,
    "bluestore_txc_items": 4,
    "bluestore_writing_deferred_bytes": 64000,
    "bluestore_writing_deferred_items": 16,
    "bluestore_writing_bytes": 125440,
    "bluestore_writing_items": 31,
    "bluefs_bytes": 17632,
    "bluefs_items": 696,
    "buffer_anon_bytes": 46543588,
    "buffer_anon_items": 111133,
    "buffer_meta_bytes": 1203312,
    "buffer_meta_items": 13674,
    "osd_bytes": 2240784,
    "osd_items": 189,
    "osd_mapbl_bytes": 582061,
    "osd_mapbl_items": 40,
    "osd_pglog_bytes": 256468652,
    "osd_pglog_items": 612338,
    "osdmap_bytes": 303112,
    "osdmap_items": 6320,
    "osdmap_mapping_bytes": 0,
    "osdmap_mapping_items": 0,
    "pgmap_bytes": 0,
    "pgmap_items": 0,
    "mds_co_bytes": 0,
    "mds_co_items": 0,
    "unittest_1_bytes": 0,
    "unittest_1_items": 0,
    "unittest_2_bytes": 0,
    "unittest_2_items": 0
    },
    "objecter": {
    "op_active": 0,
    "op_laggy": 0,
    "op_send": 0,
    "op_send_bytes": 0,
    "op_resend": 0,
    "op_reply": 0,
    "op": 0,
    "op_r": 0,
    "op_w": 0,
    "op_rmw": 0,
    "op_pg": 0,
    "osdop_stat": 0,
    "osdop_create": 0,
    "osdop_read": 0,
    "osdop_write": 0,
    "osdop_writefull": 0,
    "osdop_writesame": 0,
    "osdop_append": 0,
    "osdop_zero": 0,
    "osdop_truncate": 0,
    "osdop_delete": 0,
    "osdop_mapext": 0,
    "osdop_sparse_read": 0,
    "osdop_clonerange": 0,
    "osdop_getxattr": 0,
    "osdop_setxattr": 0,
    "osdop_cmpxattr": 0,
    "osdop_rmxattr": 0,
    "osdop_resetxattrs": 0,
    "osdop_call": 0,
    "osdop_watch": 0,
    "osdop_notify": 0,
    "osdop_src_cmpxattr": 0,
    "osdop_pgls": 0,
    "osdop_pgls_filter": 0,
    "osdop_other": 0,
    "linger_active": 0,
    "linger_send": 0,
    "linger_resend": 0,
    "linger_ping": 0,
    "poolop_active": 0,
    "poolop_send": 0,
    "poolop_resend": 0,
    "poolstat_active": 0,
    "poolstat_send": 0,
    "poolstat_resend": 0,
    "statfs_active": 0,
    "statfs_send": 0,
    "statfs_resend": 0,
    "command_active": 0,
    "command_send": 0,
    "command_resend": 0,
    "map_epoch": 1591,
    "map_full": 0,
    "map_inc": 2,
    "osd_sessions": 0,
    "osd_session_open": 0,
    "osd_session_close": 0,
    "osd_laggy": 0,
    "omap_wr": 0,
    "omap_rd": 0,
    "omap_del": 0
    },
    "osd": {
    "op_wip": 0,
    "op": 41187,
    "op_in_bytes": 411136000,
    "op_out_bytes": 0,
    "op_latency": {
    "avgcount": 41187,
    "sum": 2233365.952145376,
    "avgtime": 54.225021296
    },
    "op_process_latency": {
    "avgcount": 41187,
    "sum": 1323734.727810710,
    "avgtime": 32.139624828
    },
    "op_prepare_latency": {
    "avgcount": 41188,
    "sum": 115.582287926,
    "avgtime": 0.002806212
    },
    "op_r": 0,
    "op_r_out_bytes": 0,
    "op_r_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "op_r_process_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "op_r_prepare_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "op_w": 41187,
    "op_w_in_bytes": 411136000,
    "op_w_latency": {
    "avgcount": 41187,
    "sum": 2233365.952145376,
    "avgtime": 54.225021296
    },
    "op_w_process_latency": {
    "avgcount": 41187,
    "sum": 1323734.727810710,
    "avgtime": 32.139624828
    },
    "op_w_prepare_latency": {
    "avgcount": 41188,
    "sum": 115.582287926,
    "avgtime": 0.002806212
    },
    "op_rw": 0,
    "op_rw_in_bytes": 0,
    "op_rw_out_bytes": 0,
    "op_rw_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "op_rw_process_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "op_rw_prepare_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "op_before_queue_op_lat": {
    "avgcount": 270554,
    "sum": 27.370902651,
    "avgtime": 0.000101166
    },
    "op_before_dequeue_op_lat": {
    "avgcount": 270564,
    "sum": 1105671.624604655,
    "avgtime": 4.086543755
    },
    "subop": 0,
    "subop_in_bytes": 0,
    "subop_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "subop_w": 0,
    "subop_w_in_bytes": 0,
    "subop_w_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "subop_pull": 0,
    "subop_pull_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "subop_push": 0,
    "subop_push_in_bytes": 0,
    "subop_push_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "pull": 0,
    "push": 0,
    "push_out_bytes": 0,
    "recovery_ops": 0,
    "recovery_bytes": 0,
    "loadavg": 4348,
    "cached_crc": 0,
    "cached_crc_adjusted": 0,
    "missed_crc": 0,
    "numpg": 189,
    "numpg_primary": 75,
    "numpg_replica": 114,
    "numpg_stray": 0,
    "numpg_removing": 0,
    "heartbeat_to_peers": 20,
    "map_messages": 13,
    "map_message_epochs": 20,
    "map_message_epoch_dups": 18,
    "messages_delayed_for_map": 0,
    "osd_map_cache_hit": 382,
    "osd_map_cache_miss": 0,
    "osd_map_cache_miss_low": 0,
    "osd_map_cache_miss_low_avg": {
    "avgcount": 0,
    "sum": 0
    },
    "osd_map_bl_cache_hit": 19,
    "osd_map_bl_cache_miss": 4,
    "stat_bytes": 400086269952,
    "stat_bytes_used": 193790939136,
    "stat_bytes_avail": 206295330816,
    "copyfrom": 0,
    "tier_promote": 0,
    "tier_flush": 0,
    "tier_flush_fail": 0,
    "tier_try_flush": 0,
    "tier_try_flush_fail": 0,
    "tier_evict": 0,
    "tier_whiteout": 0,
    "tier_dirty": 3932,
    "tier_clean": 0,
    "tier_delay": 0,
    "tier_proxy_read": 0,
    "tier_proxy_write": 0,
    "agent_wake": 0,
    "agent_skip": 0,
    "agent_flush": 0,
    "agent_evict": 0,
    "object_ctx_cache_hit": 31065,
    "object_ctx_cache_total": 41188,
    "op_cache_hit": 0,
    "osd_tier_flush_lat": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "osd_tier_promote_lat": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "osd_tier_r_lat": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "osd_pg_info": 111604,
    "osd_pg_fastinfo": 77210,
    "osd_pg_biginfo": 0
    },
    "prioritycache": {
    "target_bytes": 2147483648,
    "mapped_bytes": 1780252672,
    "unmapped_bytes": 9910681600,
    "heap_bytes": 11690934272,
    "cache_bytes": 1020028139
    },
    "prioritycache:data": {
    "pri0_bytes": 0,
    "pri1_bytes": 0,
    "pri2_bytes": 0,
    "pri3_bytes": 0,
    "pri4_bytes": 0,
    "pri5_bytes": 0,
    "pri6_bytes": 0,
    "pri7_bytes": 0,
    "pri8_bytes": 0,
    "pri9_bytes": 0,
    "pri10_bytes": 0,
    "pri11_bytes": 0,
    "reserved_bytes": 67108864,
    "committed_bytes": 67108864
    },
    "prioritycache:kv": {
    "pri0_bytes": 2196672,
    "pri1_bytes": 65095968,
    "pri2_bytes": 0,
    "pri3_bytes": 0,
    "pri4_bytes": 0,
    "pri5_bytes": 0,
    "pri6_bytes": 0,
    "pri7_bytes": 0,
    "pri8_bytes": 0,
    "pri9_bytes": 0,
    "pri10_bytes": 0,
    "pri11_bytes": 132036413,
    "reserved_bytes": 69106403,
    "committed_bytes": 268435456
    },
    "prioritycache:meta": {
    "pri0_bytes": 0,
    "pri1_bytes": 78631825,
    "pri2_bytes": 0,
    "pri3_bytes": 0,
    "pri4_bytes": 0,
    "pri5_bytes": 0,
    "pri6_bytes": 0,
    "pri7_bytes": 0,
    "pri8_bytes": 0,
    "pri9_bytes": 0,
    "pri10_bytes": 0,
    "pri11_bytes": 528145654,
    "reserved_bytes": 68505465,
    "committed_bytes": 675282944
    },
    "recoverystate_perf": {
    "initial_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "started_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "reset_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "start_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "primary_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "peering_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "backfilling_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "waitremotebackfillreserved_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "waitlocalbackfillreserved_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "notbackfilling_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "repnotrecovering_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "repwaitrecoveryreserved_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "repwaitbackfillreserved_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "reprecovering_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "activating_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "waitlocalrecoveryreserved_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "waitremoterecoveryreserved_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "recovering_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "recovered_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "clean_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "active_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "replicaactive_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "stray_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "getinfo_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "getlog_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "waitactingchange_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "incomplete_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "down_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "getmissing_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "waitupthru_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "notrecovering_latency": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "rocksdb": {
    "get": 569053,
    "submit_transaction": 111606,
    "submit_transaction_sync": 18569,
    "get_latency": {
    "avgcount": 569053,
    "sum": 10.117198420,
    "avgtime": 0.000017779
    },
    "submit_latency": {
    "avgcount": 111606,
    "sum": 10.534594650,
    "avgtime": 0.000094390
    },
    "submit_sync_latency": {
    "avgcount": 18569,
    "sum": 83.365767450,
    "avgtime": 0.004489513
    },
    "compact": 0,
    "compact_range": 0,
    "compact_queue_merge": 0,
    "compact_queue_len": 0,
    "rocksdb_write_wal_time": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "rocksdb_write_memtable_time": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "rocksdb_write_delay_time": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    },
    "rocksdb_write_pre_and_post_time": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "throttle-bluestore_throttle_bytes": {
    "val": 0,
    "max": 67108864,
    "get_started": 111606,
    "get": 111606,
    "get_sum": 37081999434,
    "get_or_fail_fail": 0,
    "get_or_fail_success": 0,
    "take": 0,
    "take_sum": 0,
    "put": 18569,
    "put_sum": 37081999434,
    "wait": {
    "avgcount": 1826,
    "sum": 43.245669379,
    "avgtime": 0.023683280
    }
    },
    "throttle-bluestore_throttle_deferred_bytes": {
    "val": 135220,
    "max": 201326592,
    "get_started": 0,
    "get": 2,
    "get_sum": 67610,
    "get_or_fail_fail": 0,
    "get_or_fail_success": 2,
    "take": 0,
    "take_sum": 0,
    "put": 0,
    "put_sum": 0,
    "wait": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "throttle-msgr_dispatch_throttler-client": {
    "val": 0,
    "max": 104857600,
    "get_started": 0,
    "get": 41208,
    "get_sum": 422648569,
    "get_or_fail_fail": 0,
    "get_or_fail_success": 41208,
    "take": 0,
    "take_sum": 0,
    "put": 41208,
    "put_sum": 422648569,
    "wait": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "throttle-msgr_dispatch_throttler-cluster": {
    "val": 0,
    "max": 104857600,
    "get_started": 0,
    "get": 191867,
    "get_sum": 23670872778,
    "get_or_fail_fail": 0,
    "get_or_fail_success": 191867,
    "take": 0,
    "take_sum": 0,
    "put": 191867,
    "put_sum": 23670872778,
    "wait": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "throttle-msgr_dispatch_throttler-hb_back_client": {
    "val": 0,
    "max": 104857600,
    "get_started": 0,
    "get": 760,
    "get_sum": 1523040,
    "get_or_fail_fail": 0,
    "get_or_fail_success": 760,
    "take": 0,
    "take_sum": 0,
    "put": 760,
    "put_sum": 1523040,
    "wait": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "throttle-msgr_dispatch_throttler-hb_back_server": {
    "val": 0,
    "max": 104857600,
    "get_started": 0,
    "get": 851,
    "get_sum": 1705404,
    "get_or_fail_fail": 0,
    "get_or_fail_success": 851,
    "take": 0,
    "take_sum": 0,
    "put": 851,
    "put_sum": 1705404,
    "wait": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "throttle-msgr_dispatch_throttler-hb_front_client": {
    "val": 0,
    "max": 104857600,
    "get_started": 0,
    "get": 760,
    "get_sum": 1523040,
    "get_or_fail_fail": 0,
    "get_or_fail_success": 760,
    "take": 0,
    "take_sum": 0,
    "put": 760,
    "put_sum": 1523040,
    "wait": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "throttle-msgr_dispatch_throttler-hb_front_server": {
    "val": 0,
    "max": 104857600,
    "get_started": 0,
    "get": 851,
    "get_sum": 1705404,
    "get_or_fail_fail": 0,
    "get_or_fail_success": 851,
    "take": 0,
    "take_sum": 0,
    "put": 851,
    "put_sum": 1705404,
    "wait": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "throttle-msgr_dispatch_throttler-ms_objecter": {
    "val": 0,
    "max": 104857600,
    "get_started": 0,
    "get": 0,
    "get_sum": 0,
    "get_or_fail_fail": 0,
    "get_or_fail_success": 0,
    "take": 0,
    "take_sum": 0,
    "put": 0,
    "put_sum": 0,
    "wait": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "throttle-objecter_bytes": {
    "val": 0,
    "max": 104857600,
    "get_started": 0,
    "get": 0,
    "get_sum": 0,
    "get_or_fail_fail": 0,
    "get_or_fail_success": 0,
    "take": 0,
    "take_sum": 0,
    "put": 0,
    "put_sum": 0,
    "wait": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "throttle-objecter_ops": {
    "val": 0,
    "max": 1024,
    "get_started": 0,
    "get": 0,
    "get_sum": 0,
    "get_or_fail_fail": 0,
    "get_or_fail_success": 0,
    "take": 0,
    "take_sum": 0,
    "put": 0,
    "put_sum": 0,
    "wait": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    },
    "throttle-osd_client_bytes": {
    "val": 0,
    "max": 524288000,
    "get_started": 0,
    "get": 41207,
    "get_sum": 422648289,
    "get_or_fail_fail": 0,
    "get_or_fail_success": 41207,
    "take": 0,
    "take_sum": 0,
    "put": 78696,
    "put_sum": 422648289,
    "wait": {
    "avgcount": 0,
    "sum": 0.000000000,
    "avgtime": 0.000000000
    }
    }
    }

#14 Updated by Stellar Wang over 3 years ago

hi Igor,
Is there any new progress?

#15 Updated by Igor Fedotov over 3 years ago

鑫 王 wrote:

hi Igor,
Is there any new progress?

Hi!
I haven't managed to reproduce this locally for master brunch.
I could see interim flukes (up to approx. 40000-50000 items) in "bluestore_writing_items" in my tests but since they deal with 4K writing blocks this didn't cause that large RAM usage as you have:
"bluestore_writing_bytes": 7426528761,
"bluestore_writing_items": 8704,

And the above numbers mean your OSD is dealing with 850K (writing 7426528761/8704) blocks which I can't explain. Can you?

Additionally you might have worse experience due to the lack of backport for https://github.com/ceph/ceph/pull/28597 in Nautilus. Can't say if this is the case for sure though...

#16 Updated by Stellar Wang over 3 years ago

hi Igor,
I can't explain why OSD handles 850K(writing 7426528761/8704), but when load is very low (a client iodepth=1 numjobs=1) OSD handles about 4k.When I mount through CIFS, however, The load is high (6 clients iodepth=1 numjobs=64) num=1) Bluestore_writing_bytes is not 4K.
A slow IO will occur during execution.
I have another question why is the field buffer_anon also growing so much?Can you explain how buffer_anon is used?
"buffer_anon" : {
"Items" : 317088,
"Bytes" : 3024777598
},

#17 Updated by Igor Fedotov over 3 years ago

鑫 王 wrote:

A slow IO will occur during execution.
I have another question why is the field buffer_anon also growing so much?Can you explain how buffer_anon is used?
"buffer_anon" : {
"Items" : 317088,
"Bytes" : 3024777598
},

These are general-purpose buffers allocated by OSD. Where "general-purpose buffers" rather means data block which Ceph doesn't care to assign to a specific component. This can be for multiple reasons, e.g. developers expect such data to be short-living, minor or insignificant. Looks like such an expectation in your case wasn't proper. Unfortunately it's hard to backtrace the origin of such buffers with reproduction in the lab...

#18 Updated by Greg Farnum almost 3 years ago

  • Project changed from Ceph to RADOS
  • Category changed from OSD to Performance/Resource Usage

Also available in: Atom PDF