Project

General

Profile

Bug #13882

The monitor always election

Added by ceph zte over 8 years ago. Updated over 8 years ago.

Status:
Rejected
Priority:
High
Assignee:
Joao Eduardo Luis
Category:
Monitor
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Hello.My ceph version is 0.87.2.

The ceph info is below

[root@ceph243 ~]# rbd showmapped
id pool image snap device
3 cc im - /dev/rbd3
4 11 im - /dev/rbd4
5 11 im1 - /dev/rbd5
6 ld2 ld2 - /dev/rbd6
7 zhupool20151124 image20151124 - /dev/rbd7

[root@ceph243 ~]# rbd info zhupool20151124/image20151124
rbd image 'image20151124':
size 2048 GB in 524288 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.33c2d542ae8944a
format: 2
features: layering

I use the command dd if=/dev/sda of=/dev/rbd7 bs=64M count=32768

The mon election log:

2015-11-25 13:03:02.087855 7f7828708700 7 mon.ceph243@1(peon).log v9958288 update_from_paxos applying incremental log 9958288 2015-11-25 13:02:54.254709 mon.2 100.100.100.244:6789/0 1999563 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[json]: finished
2015-11-25 13:03:02.087861 7f7828708700 20 mon.ceph243@1(peon).log v9958288 update_from_paxos logging for channel 'audit' to file '/var/log/ceph/ceph.audit.log'
2015-11-25 13:03:02.087866 7f7828708700 7 mon.ceph243@1(peon).log v9958288 update_from_paxos applying incremental log 9958288 2015-11-25 13:02:55.261661 mon.2 100.100.100.244:6789/0 1999564 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[json]: dispatch
2015-11-25 13:03:02.087873 7f7828708700 20 mon.ceph243@1(peon).log v9958288 update_from_paxos logging for channel 'audit' to file '/var/log/ceph/ceph.audit.log'
2015-11-25 13:03:02.087879 7f7828708700 7 mon.ceph243@1(peon).log v9958288 update_from_paxos applying incremental log 9958288 2015-11-25 13:02:55.261804 mon.2 100.100.100.244:6789/0 1999565 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[json]: finished
2015-11-25 13:03:02.087886 7f7828708700 20 mon.ceph243@1(peon).log v9958288 update_from_paxos logging for channel 'audit' to file '/var/log/ceph/ceph.audit.log'
2015-11-25 13:03:02.087891 7f7828708700 7 mon.ceph243@1(peon).log v9958288 update_from_paxos applying incremental log 9958288 2015-11-25 13:03:00.006738 mon.2 100.100.100.244:6789/0 1999566 : cluster [INF] mon.ceph244 calling new monitor election
2015-11-25 13:03:02.087898 7f7828708700 20 mon.ceph243@1(peon).log v9958288 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log'
2015-11-25 13:03:02.087909 7f7828708700 7 mon.ceph243@1(peon).log v9958288 update_from_paxos applying incremental log 9958288 2015-11-25 13:03:00.008501 mon.1 100.100.100.243:6789/0 397684 : cluster [INF] mon.ceph243 calling new monitor election
2015-11-25 13:03:02.087922 7f7828708700 20 mon.ceph243@1(peon).log v9958288 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log'
2015-11-25 13:03:02.087934 7f7828708700 7 mon.ceph243@1(peon).log v9958288 update_from_paxos applying incremental log 9958288 2015-11-25 13:03:00.025120 mon.0 100.100.100.240:6789/0 2524041 : cluster [INF] mon.ceph240 calling new monitor election
2015-11-25 13:03:02.087942 7f7828708700 20 mon.ceph243@1(peon).log v9958288 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log'
2015-11-25 13:03:02.087953 7f7828708700 7 mon.ceph243@1(peon).log v9958288 update_from_paxos applying incremental log 9958288 2015-11-25 13:03:00.509767 mon.0 100.100.100.240:6789/0 2524042 : cluster [INF] mon.ceph240@0 won leader election with quorum 0,1,2
2015-11-25 13:03:02.087959 7f7828708700 20 mon.ceph243@1(peon).log v9958288 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log'
2015-11-25 13:03:02.087970 7f7828708700 7 mon.ceph243@1(peon).log v9958288 update_from_paxos applying incremental log 9958288 2015-11-25 13:03:00.719834 mon.1 100.100.100.243:6789/0 397685 : cluster [WRN] reached concerning levels of available space on local monitor storage (30% free)
2015-11-25 13:03:02.087980 7f7828708700 20 mon.ceph243@1(peon).log v9958288 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log'
2015-11-25 13:03:02.087992 7f7828708700 7 mon.ceph243@1(peon).log v9958288 update_from_paxos applying incremental log 9958288 2015-11-25 13:03:00.990518 mon.0 100.100.100.240:6789/0 2524043 : cluster [INF] monmap e26: 3 mons at {ceph240=100.100.100.240:6789/0,ceph243=100.100.100.243:6789/0,ceph244=100.100.100.244:6789/0}
2015-11-25 13:03:02.088007 7f7828708700 20 mon.ceph243@1(peon).log v9958288 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log'
2015-11-25 13:03:02.088019 7f7828708700 7 mon.ceph243@1(peon).log v9958288 update_from_paxos applying incremental log 9958288 2015-11-25 13:03:00.993741 mon.0 100.100.100.240:6789/0 2524044 : cluster [INF] pgmap v7868594: 14352 pgs: 2 active+undersized+remapped, 13346 active+clean, 56 active+undersized+degraded, 11 active+clean+inconsistent, 1 peering, 7 active+remapped+wait_backfill, 660 active+undersized+degraded+remapped+wait_backfill, 3 active+undersized+degraded+remapped+inconsistent+wait_backfill, 204 active+remapped, 62 active+undersized+degraded+remapped+backfilling; 3087 GB data, 8706 GB used, 40639 GB / 49345 GB avail; 12656 kB/s rd, 5660 kB/s wr, 1413 op/s; 180798/2609684 objects degraded (6.928%); 587434/2609684 objects misplaced (22.510%); 406 MB/s, 103 objects/s recovering
2015-11-25 13:03:02.088031 7f7828708700 20 mon.ceph243@1(peon).log v9958288 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log'
2015-11-25 13:03:02.088043 7f7828708700 7 mon.ceph243@1(peon).log v9958288 update_from_paxos applying incremental log 9958288 2015-11-25 13:03:00.993852 mon.0 100.100.100.240:6789/0 2524045 : cluster [INF] mdsmap e1: 0/0/0 up
2015-11-25 13:03:02.088055 7f7828708700 20 mon.ceph243@1(peon).log v9958288 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log'
2015-11-25 13:03:02.088066 7f7828708700 7 mon.ceph243@1(peon).log v9958288 update_from_paxos applying incremental log 9958288 2015-11-25 13:03:00.998824 mon.1 100.100.100.243:6789/0 397686 : audit [DBG] from='client.? 100.100.100.253:0/3917384350' entity='client.cinder' cmd=[{,",p,r,e,f,i,x,",:,",d,f,",,, ,",f,o,r,m,a,t,",:,",j,s,o,n,",}]: dispatch
2015-11-25 13:03:02.088073 7f7828708700 20 mon.ceph243@1(peon).log v9958288 update_from_paxos logging for channel 'audit' to file '/var/log/ceph/ceph.audit.log'
2015-11-25 13:03:02.088078 7f7828708700 7 mon.ceph243@1(peon).log v9958288 update_from_paxos applying incremental log 9958288 2015-11-25 13:03:01.007513 mon.0 100.100.100.240:6789/0 2524046 : cluster [INF] osdmap e520979: 42 osds: 32 up, 32 in

2015-11-25 13:03:19.627070 7f7828f09700 1 mon.ceph243@1(peon).paxos(paxos updating c 18842322..18842990) lease_timeout -- calling new election
2015-11-25 13:03:19.627087 7f7828f09700 10 mon.ceph243@1(peon) e26 bootstrap
2015-11-25 13:03:19.627092 7f7828f09700 10 mon.ceph243@1(peon) e26 sync_reset_requester
2015-11-25 13:03:19.627095 7f7828f09700 10 mon.ceph243@1(peon) e26 unregister_cluster_logger
2015-11-25 13:03:19.627101 7f7828f09700 10 mon.ceph243@1(peon) e26 cancel_probe_timeout (none scheduled)
2015-11-25 13:03:19.627106 7f7828f09700 10 mon.ceph243@1(probing) e26 _reset
2015-11-25 13:03:19.627108 7f7828f09700 10 mon.ceph243@1(probing) e26 cancel_probe_timeout (none scheduled)
2015-11-25 13:03:19.627111 7f7828f09700 10 mon.ceph243@1(probing) e26 timecheck_finish
2015-11-25 13:03:19.627116 7f7828f09700 10 mon.ceph243@1(probing) e26 scrub_reset
2015-11-25 13:03:19.627126 7f7828f09700 10 mon.ceph243@1(probing) e26 cancel_probe_timeout (none scheduled)
2015-11-25 13:03:19.627131 7f7828f09700 10 mon.ceph243@1(probing) e26 reset_probe_timeout 0x6dca970 after 2 seconds
2015-11-25 13:03:19.627145 7f7828f09700 10 mon.ceph243@1(probing) e26 probing other monitors
2015-11-25 13:03:19.627824 7f7828708700 20 mon.ceph243@1(probing) e26 have connection
2015-11-25 13:03:19.627826 7f7828708700 20 mon.ceph243@1(probing) e26 ms_dispatch existing session MonSession: mon.0 100.100.100.240:6789/0 is openallow * for mon.0 100.100.100.240:6789/0
2015-11-25 13:03:19.627832 7f7828708700 20 mon.ceph243@1(probing) e26 caps allow *
2015-11-25 13:03:19.627834 7f7828708700 20 is_capable service=mon command= read on cap allow *
2015-11-25 13:03:19.627836 7f7828708700 20 allow so far , doing grant allow *
2015-11-25 13:03:19.627837 7f7828708700 20 allow all
2015-11-25 13:03:19.627838 7f7828708700 10 mon.ceph243@1(probing) e26 handle_probe mon_probe(reply bd9e23c5-da96-4908-a335-455445480e54 name ceph240 quorum 0,1,2 paxos( fc 18842322 lc 18842990 )) v6
2015-11-25 13:03:19.627843 7f7828708700 10 mon.ceph243@1(probing) e26 handle_probe_reply mon.0 100.100.100.240:6789/0mon_probe(reply bd9e23c5-da96-4908-a335-455445480e54 name ceph240 quorum 0,1,2 paxos( fc 18842322 lc 18842990 )) v6
2015-11-25 13:03:19.627850 7f7828708700 10 mon.ceph243@1(probing) e26 monmap is e26: 3 mons at {ceph240=100.100.100.240:6789/0,ceph243=100.100.100.243:6789/0,ceph244=100.100.100.244:6789/0}
2015-11-25 13:03:19.627862 7f7828708700 10 mon.ceph243@1(probing) e26 peer name is ceph240
2015-11-25 13:03:19.627865 7f7828708700 10 mon.ceph243@1(probing) e26 existing quorum 0,1,2
2015-11-25 13:03:19.627868 7f7828708700 10 mon.ceph243@1(probing) e26 peer paxos version 18842990 vs my version 18842990 (ok)
2015-11-25 13:03:19.627874 7f7828708700 10 mon.ceph243@1(probing) e26 start_election
2015-11-25 13:03:19.627876 7f7828708700 10 mon.ceph243@1(probing) e26 _reset
2015-11-25 13:03:19.627877 7f7828708700 10 mon.ceph243@1(probing) e26 cancel_probe_timeout 0x6dca970
2015-11-25 13:03:19.627880 7f7828708700 10 mon.ceph243@1(probing) e26 timecheck_finish
2015-11-25 13:03:19.627882 7f7828708700 10 mon.ceph243@1(probing) e26 scrub_reset
2015-11-25 13:03:19.627885 7f7828708700 10 mon.ceph243@1(electing) e26 cancel_probe_timeout (none scheduled)
2015-11-25 13:03:19.627893 7f7828708700 0 log_channel(cluster) log [INF] : mon.ceph243 calling new monitor election
2015-11-25 13:03:19.627953 7f7828708700 5 mon.ceph243@1(electing).elector(5730) start -- can i be leader?
2015-11-25 13:03:19.628054 7f7828708700 1 mon.ceph243@1(electing).elector(5730) init, last seen epoch 5730
2015-11-25 13:03:19.628062 7f7828708700 10 mon.ceph243@1(electing).elector(5730) bump_epoch 5730 to 5731
2015-11-25 13:03:19.628632 7f7828708700 10 mon.ceph243@1(electing) e26 join_election

2015-11-25 13:03:21.247554 7f7828708700 7 mon.ceph243@1(peon).log v9958296 update_from_paxos applying incremental log 9958296 2015-11-25 13:03:17.404980 mon.0 100.100.100.240:6789/0 2524071 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[json]: finished
2015-11-25 13:03:21.247561 7f7828708700 20 mon.ceph243@1(peon).log v9958296 update_from_paxos logging for channel 'audit' to file '/var/log/ceph/ceph.audit.log'
2015-11-25 13:03:21.247566 7f7828708700 7 mon.ceph243@1(peon).log v9958296 update_from_paxos applying incremental log 9958296 2015-11-25 13:03:17.458261 mon.0 100.100.100.240:6789/0 2524072 : audit [DBG] from='admin socket' entity='admin socket' cmd='mon_status' args=[json]: dispatch
2015-11-25 13:03:21.247574 7f7828708700 20 mon.ceph243@1(peon).log v9958296 update_from_paxos logging for channel 'audit' to file '/var/log/ceph/ceph.audit.log'
2015-11-25 13:03:21.247580 7f7828708700 7 mon.ceph243@1(peon).log v9958296 update_from_paxos applying incremental log 9958296 2015-11-25 13:03:17.458381 mon.0 100.100.100.240:6789/0 2524073 : audit [DBG] from='admin socket' entity='admin socket' cmd=mon_status args=[json]: finished
2015-11-25 13:03:21.247586 7f7828708700 20 mon.ceph243@1(peon).log v9958296 update_from_paxos logging for channel 'audit' to file '/var/log/ceph/ceph.audit.log'
2015-11-25 13:03:21.247601 7f7828708700 7 mon.ceph243@1(peon).log v9958296 update_from_paxos applying incremental log 9958296 2015-11-25 13:03:19.627507 mon.2 100.100.100.244:6789/0 1999589 : cluster [INF] mon.ceph244 calling new monitor election
2015-11-25 13:03:21.247616 7f7828708700 20 mon.ceph243@1(peon).log v9958296 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log'
2015-11-25 13:03:21.247632 7f7828708700 7 mon.ceph243@1(peon).log v9958296 update_from_paxos applying incremental log 9958296 2015-11-25 13:03:19.627895 mon.1 100.100.100.243:6789/0 397759 : cluster [INF] mon.ceph243 calling new monitor election
2015-11-25 13:03:21.247639 7f7828708700 20 mon.ceph243@1(peon).log v9958296 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log'
2015-11-25 13:03:21.247650 7f7828708700 7 mon.ceph243@1(peon).log v9958296 update_from_paxos applying incremental log 9958296 2015-11-25 13:03:19.635632 mon.0 100.100.100.240:6789/0 2524074 : cluster [INF] mon.ceph240 calling new monitor election
2015-11-25 13:03:21.247657 7f7828708700 20 mon.ceph243@1(peon).log v9958296 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log'
2015-11-25 13:03:21.247667 7f7828708700 7 mon.ceph243@1(peon).log v9958296 update_from_paxos applying incremental log 9958296 2015-11-25 13:03:19.908441 mon.0 100.100.100.240:6789/0 2524075 : cluster [INF] mon.ceph240@0 won leader election with quorum 0,1,2
2015-11-25 13:03:21.247674 7f7828708700 20 mon.ceph243@1(peon).log v9958296 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log'
2015-11-25 13:03:21.247685 7f7828708700 7 mon.ceph243@1(peon).log v9958296 update_from_paxos applying incremental log 9958296 2015-11-25 13:03:20.243652 mon.0 100.100.100.240:6789/0 2524076 : cluster [INF] monmap e26: 3 mons at {ceph240=100.100.100.240:6789/0,ceph243=100.100.100.243:6789/0,ceph244=100.100.100.244:6789/0}

The system info:

ernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
enp130s0 1500 2655292865 0 30092 0 2346105503 0 0 0 BMRU
enp130s0 1500 6866040602 0 30092 0 6923061412 0 0 0 BMRU
enp4s0f0 1500 7018724 0 6742 0 4893981 0 0 0 BMRU
lo 65536 24844352 0 0 0 24844352 0 0 0 LRU

procs -----------memory---------- ---swap-- -----io---- system- ------cpu----- -----timestamp-----
r b swpd free buff cache si so bi bo in cs us sy id wa st CST
9 46 652200 229044 10948068 35933220 5 26 449 5679 5 3 4 2 85 9 0 2015-11-25 15:15:57

Linux 3.10.0-123.el7.x86_64 (ceph243) 11/25/2015 x86_64 (24 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
3.98 0.00 1.82 8.84 0.00 85.36

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sdm 40.59 722.50 2555.03 138446089 489594694
sdh 22.16 1012.08 6076.07 193935689 1164295740
sdn 23.55 964.20 6863.94 184759201 1315267185
sdp 22.25 596.43 6893.80 114288622 1320989714
sdb 26.62 754.18 6879.62 144516557 1318273295
sdr 21.05 698.51 6101.10 133848534 1169092018
sds 21.89 1103.94 5931.56 211537342 1136605200
rbd3 0.00 0.01 0.07 2307 12787
rbd4 0.00 0.01 0.00 1190 0
rbd5 0.00 0.01 0.00 1245 4
rbd6 0.76 0.03 388.05 5356 74358784
sdc 18.61 750.45 4233.62 143801275 811246422
rbd7 1.74 0.01 758.88 2192 145416816
sdf 13.71 486.14 3301.68 93154396 632667624
sdd 14.86 660.32 3079.37 126530236 590067891
sdi 0.01 0.13 0.05 25396 10278
sda 83.70 204.04 30636.90 39098841 5870643248
sdj 17.41 987.40 2944.21 189205091 564169853

Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
enp130s0 1500 2655336350 0 30093 0 2346184382 0 0 0 BMRU
enp130s0 1500 6866343564 0 30093 0 6923324043 0 0 0 BMRU
enp4s0f0 1500 7023343 0 6742 0 4902862 0 0 0 BMRU
lo 65536 24849395 0 0 0 24849395 0 0 0 LRU

procs -----------memory---------- ---swap-- -----io---- system- ------cpu----- -----timestamp-----
r b swpd free buff cache si so bi bo in cs us sy id wa st CST
32 42 652200 228232 11014036 35868864 5 26 449 5680 5 3 4 2 85 9 0 2015-11-25 15:16:05

Linux 3.10.0-123.el7.x86_64 (ceph243) 11/25/2015 x86_64 (24 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
3.98 0.00 1.82 8.84 0.00 85.36

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sdm 40.59 722.57 2555.48 138465325 489700778
sdh 22.16 1012.09 6076.55 193944121 1164436567
sdn 23.56 964.81 6864.08 184884425 1315349735
sdp 22.25 596.62 6893.93 114329614 1321069416
sdb 26.63 754.50 6880.05 144583181 1318411254
sdr 21.06 699.18 6101.10 133981582 1169141323
sds 21.89 1104.30 5931.42 211615654 1136625697
rbd3 0.00 0.01 0.07 2307 12787
rbd4 0.00 0.01 0.00 1190 0
rbd5 0.00 0.01 0.00 1245 4
rbd6 0.76 0.03 388.04 5356 74358784
sdc 18.61 750.52 4234.31 143821419 811411840
rbd7 1.74 0.01 759.40 2192 145521448
sdf 13.72 486.36 3302.28 93200900 632810100
sdd 14.87 660.95 3079.31 126656716 590082847
sdi 0.01 0.13 0.05 25396 10278
sda 83.72 204.69 30639.55 39224797 5871397704
sdj 17.41 987.90 2944.30 189309271 564209994

Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
enp130s0 1500 2655391539 0 30094 0 2346249138 0 0 0 BMRU
enp130s0 1500 6866783643 0 30094 0 6923834760 0 0 0 BMRU
enp4s0f0 1500 7025190 0 6772 0 4906311 0 0 0 BMRU
lo 65536 24857447 0 0 0 24857447 0 0 0 LRU

procs -----------memory---------- ---swap-- -----io---- system- ------cpu----- -----timestamp-----
r b swpd free buff cache si so bi bo in cs us sy id wa st CST
8 35 652200 226536 10925824 35945628 5 26 449 5680 5 4 4 2 85 9 0 2015-11-25 15:16:11

Linux 3.10.0-123.el7.x86_64 (ceph243) 11/25/2015 x86_64 (24 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
3.98 0.00 1.82 8.84 0.00 85.35

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sdm 40.59 722.69 2555.94 138491973 489803838
sdh 22.16 1012.08 6076.92 193948961 1164541038
sdn 23.56 965.47 6864.30 185017069 1315430456
sdp 22.25 596.86 6893.99 114378874 1321118787
sdb 26.63 754.72 6880.23 144630173 1318483052
sdr 21.06 699.72 6101.21 134090202 1169197060
sds 21.89 1104.70 5931.26 211697102 1136627213
rbd3 0.00 0.01 0.07 2307 12787
rbd4 0.00 0.01 0.00 1190 0
rbd5 0.00 0.01 0.00 1245 4
rbd6 0.76 0.03 388.03 5356 74358784
sdc 18.61 750.71 4234.73 143861291 811516589
rbd7 1.75 0.01 759.74 2192 145591796
sdf 13.72 486.46 3302.91 93222048 632948872
sdd 14.87 661.57 3079.29 126779248 590095357
sdi 0.01 0.13 0.05 25396 10278
sda 83.72 204.76 30642.33 39238105 5872096264
sdj 17.42 988.31 2944.55 189393035 564273884

Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
enp130s0 1500 2655416076 0 30094 0 2346323274 0 0 0 BMRU
enp130s0 1500 6867112750 0 30094 0 6924398557 0 0 0 BMRU
enp4s0f0 1500 7025290 0 6772 0 4906311 0 0 0 BMRU
lo 65536 24865190 0 0 0 24865190 0 0 0 LRU

procs -----------memory---------- ---swap-- -----io---- system- ------cpu----- -----timestamp-----
r b swpd free buff cache si so bi bo in cs us sy id wa st CST
20 32 652200 405576 10811980 35885728 5 26 450 5680 5 4 4 2 85 9 0 2015-11-25 15:16:20

Linux 3.10.0-123.el7.x86_64 (ceph243) 11/25/2015 x86_64 (24 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
3.98 0.00 1.82 8.84 0.00 85.35

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sdm 40.59 722.66 2555.91 138491985 489822006
sdh 22.17 1012.12 6077.59 193965581 1164725557
sdn 23.56 965.87 6864.80 185101225 1315586942
sdp 22.25 597.22 6893.91 114452830 1321167315
sdb 26.63 754.93 6880.51 144676021 1318599159
sdr 21.06 700.39 6101.27 134224638 1169262638
sds 21.90 1105.23 5931.25 211809106 1136680634
rbd3 0.00 0.01 0.07 2307 12787
rbd4 0.00 0.01 0.00 1190 0
rbd5 0.00 0.01 0.00 1245 4
rbd6 0.76 0.03 388.01 5356 74358784
sdc 18.62 750.76 4235.70 143878063 811741026
rbd7 1.75 0.01 759.98 2192 145644352
sdf 13.73 486.53 3303.70 93240400 633129153
sdd 14.88 662.04 3079.24 126875368 590112609
sdi 0.01 0.13 0.05 25396 10278
sda 83.73 204.97 30645.94 39281657 5873066304
sdj 17.42 988.84 2944.98 189503359 564384286

Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
enp130s0 1500 2655460932 0 30096 0 2346354551 0 0 0 BMRU
enp130s0 1500 6867583112 0 30096 0 6924808706 0 0 0 BMRU
enp4s0f0 1500 7025362 0 6773 0 4906324 0 0 0 BMRU
lo 65536 24870208 0 0 0 24870208 0 0 0 LRU

procs -----------memory---------- ---swap-- -----io---- system- ------cpu----- -----timestamp-----
r b swpd free buff cache si so bi bo in cs us sy id wa st CST
0 27 652200 335604 10761432 35983232 5 26 450 5680 5 4 4 2 85 9 0 2015-11-25 15:16:25

Linux 3.10.0-123.el7.x86_64 (ceph243) 11/25/2015 x86_64 (24 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
3.98 0.00 1.82 8.85 0.00 85.35

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sdm 40.59 722.74 2556.71 138510637 489988262
sdh 22.17 1012.25 6077.77 193995277 1164791706
sdn 23.57 966.48 6864.93 185224333 1315647621
sdp 22.25 597.57 6894.09 114522682 1321236470
sdb 26.63 755.10 6880.63 144712649 1318657208
sdr 21.06 700.86 6101.35 134318746 1169309774
sds 21.90 1105.55 5931.47 211875558 1136752070
rbd3 0.00 0.01 0.07 2307 12787
rbd4 0.00 0.01 0.00 1190 0
rbd5 0.00 0.01 0.00 1245 4
rbd6 0.76 0.03 388.00 5356 74358784
sdc 18.62 750.83 4236.75 143895747 811962579
rbd7 1.75 0.01 760.29 2192 145708096
sdf 13.73 486.69 3304.06 93273280 633215286
sdd 14.88 662.44 3079.56 126954576 590191235
sdi 0.01 0.13 0.05 25396 10278
sda 83.74 205.07 30649.09 39300345 5873828648
sdj 17.43 989.26 2944.95 189589207 564392599

Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
enp130s0 1500 2655507115 0 30097 0 2346379326 0 0 0 BMRU
enp130s0 1500 6868119551 0 30097 0 6925301205 0 0 0 BMRU
enp4s0f0 1500 7030460 0 6791 0 4915920 0 0 0 BMRU
lo 65536 24878875 0 0 0 24878875 0 0 0 LRU

procs -----------memory---------- ---swap-- -----io---- system- ------cpu----- -----timestamp-----
r b swpd free buff cache si so bi bo in cs us sy id wa st CST
57 22 652200 355992 10699124 35979736 5 26 450 5680 5 4 4 2 85 9 0 2015-11-25 15:16:30

Linux 3.10.0-123.el7.x86_64 (ceph243) 11/25/2015 x86_64 (24 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
3.98 0.00 1.82 8.85 0.00 85.35

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sdm 40.59 722.76 2556.66 138518829 489991870
sdh 22.17 1012.48 6078.29 194044641 1164921081
sdn 23.57 966.91 6865.09 185310573 1315714563
sdp 22.26 597.90 6894.11 114588910 1321275880
sdb 26.64 755.17 6880.79 144730169 1318723313
sdr 21.07 701.28 6101.38 134402718 1169348019
sds 21.90 1105.77 5931.70 211924222 1136827885
rbd3 0.00 0.01 0.07 2307 12787
rbd4 0.00 0.01 0.00 1190 0
rbd5 0.00 0.01 0.00 1245 4
rbd6 0.76 0.03 387.99 5356 74358784
sdc 18.63 751.06 4237.43 143943155 812116350
rbd7 1.75 0.01 760.40 2192 145732704
sdf 13.73 486.72 3304.61 93281528 633338239
sdd 14.89 662.79 3079.76 127024668 590245032
sdi 0.01 0.13 0.05 25396 10278
sda 83.75 205.15 30652.24 39317497 5874590552
sdj 17.43 989.70 2944.88 189678755 564394203

Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
enp130s0 1500 2655519221 0 30098 0 2346392279 0 0 0 BMRU
enp130s0 1500 6868506167 0 30098 0 6925583681 0 0 0 BMRU
enp4s0f0 1500 7035359 0 6791 0 4924088 0 0 0 BMRU
lo 65536 24884549 0 0 0 24884549 0 0 0 LRU

procs -----------memory---------- ---swap-- -----io---- system- ------cpu----- -----timestamp-----
r b swpd free buff cache si so bi bo in cs us sy id wa st CST
24 49 652200 308744 10845712 35900660 5 26 450 5680 5 4 4 2 85 9 0 2015-11-25 15:16:38

Linux 3.10.0-123.el7.x86_64 (ceph243) 11/25/2015 x86_64 (24 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
3.98 0.00 1.82 8.85 0.00 85.34

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sdm 40.59 722.79 2556.58 138531117 489994686
sdh 22.17 1012.76 6078.16 194105397 1164942412
sdn 23.57 967.12 6864.83 185357817 1315715577
sdp 22.26 598.14 6893.91 114639002 1321289910
sdb 26.64 755.29 6880.68 144758657 1318752844
sdr 21.07 701.47 6101.20 134443890 1169358519
sds 21.90 1105.96 5931.49 211969526 1136831328
rbd3 0.00 0.01 0.07 2307 12787
rbd4 0.00 0.01 0.00 1190 0
rbd5 0.00 0.01 0.00 1245 4
rbd6 0.76 0.03 387.97 5356 74358784
sdc 18.63 751.18 4237.93 143971119 812242691
rbd7 1.75 0.01 760.46 2192 145749880
sdf 13.74 486.75 3304.84 93290284 633406331
sdd 14.89 662.94 3079.77 127059888 590269269
sdi 0.01 0.13 0.05 25396 10278
sda 83.75 205.63 30653.15 39411077 5874993276
sdj 17.43 989.99 2944.94 189742735 564428654

Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
enp130s0 1500 2655530355 0 30099 0 2346407446 0 0 0 BMRU
enp130s0 1500 6868620226 0 30099 0 6925869691 0 0 0 BMRU
enp4s0f0 1500 7039151 0 6791 0 4931421 0 0 0 BMRU
lo 65536 24893019 0 0 0 24893019 0 0 0 LRU

procs -----------memory---------- ---swap-- -----io---- system- ------cpu----- -----timestamp-----
r b swpd free buff cache si so bi bo in cs us sy id wa st CST
3 20 652200 230444 11102336 35685680 5 26 450 5681 5 4 4 2 85 9 0 2015-11-25 15:16:46

Linux 3.10.0-123.el7.x86_64 (ceph243) 11/25/2015 x86_64 (24 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
3.98 0.00 1.82 8.85 0.00 85.34

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sdm 40.59 722.83 2556.61 138543881 490022706
sdh 22.18 1012.82 6077.94 194127073 1164952475
sdn 23.57 967.14 6864.82 185370581 1315771796
sdp 22.26 598.41 6894.10 114696654 1321384735
sdb 26.65 755.43 6881.06 144792161 1318885750
sdr 21.07 701.69 6101.26 134492446 1169422743
sds 21.91 1106.03 5931.47 211991426 1136878520
rbd3 0.00 0.01 0.07 2307 12787
rbd4 0.00 0.01 0.00 1190 0
rbd5 0.00 0.01 0.00 1245 4
rbd6 0.76 0.03 387.95 5356 74358784
sdc 18.63 751.22 4238.22 143985207 812334250
rbd7 1.75 0.01 760.52 2192 145768672
sdf 13.74 486.75 3305.92 93294548 633642886
sdd 14.89 663.38 3079.88 127149992 590316491
sdi 0.01 0.13 0.05 25396 10278
sda 83.77 206.36 30655.74 39552005 5875752036
sdj 17.44 990.52 2944.90 189852403 564446286

Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
enp130s0 1500 2655553430 0 30100 0 2346429464 0 0 0 BMRU
enp130s0 1500 6869171946 0 30100 0 6926295806 0 0 0 BMRU
enp4s0f0 1500 7044241 0 6791 0 4941247 0 0 0 BMRU
lo 65536 24905015 0 0 0 24905015 0 0 0 LRU

procs -----------memory---------- ---swap-- -----io---- system- ------cpu----- -----timestamp-----
r b swpd free buff cache si so bi bo in cs us sy id wa st CST
25 26 652200 230348 11098940 35723240 5 26 450 5681 5 4 4 2 85 9 0 2015-11-25 15:16:52

Linux 3.10.0-123.el7.x86_64 (ceph243) 11/25/2015 x86_64 (24 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
3.98 0.00 1.82 8.85 0.00 85.34

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sdm 40.59 722.93 2556.74 138566729 490061986
sdh 22.18 1013.04 6077.86 194173245 1164970073
sdn 23.57 967.28 6864.67 185403813 1315782819
sdp 22.26 598.67 6894.03 114749118 1321410453
sdb 26.65 755.60 6881.03 144828673 1318917578
sdr 21.07 701.99 6101.24 134553054 1169452331
sds 21.91 1106.38 5931.45 212065330 1136907716
rbd3 0.00 0.01 0.07 2307 12787
rbd4 0.00 0.01 0.00 1190 0
rbd5 0.00 0.01 0.00 1245 4
rbd6 0.76 0.03 387.94 5356 74358784
sdc 18.64 751.29 4238.79 144003207 812468694
rbd7 1.75 0.01 760.54 2192 145776252
sdf 13.74 486.77 3306.17 93302156 633708964
sdd 14.90 663.57 3079.91 127188816 590339807
sdi 0.01 0.13 0.05 25396 10278
sda 83.78 206.35 30657.65 39552005 5876291440
sdj 17.44 990.80 2944.99 189911479 564480213

Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
enp130s0 1500 2655565050 0 30101 0 2346434616 0 0 0 BMRU
enp130s0 1500 6869658084 0 30101 0 6926680056 0 0 0 BMRU
enp4s0f0 1500 7055468 0 6792 0 4962768 0 0 0 BMRU
lo 65536 24917564 0 0 0 24917564 0 0 0 LRU

procs -----------memory---------- ---swap-- -----io---- system- ------cpu----- -----timestamp-----
r b swpd free buff cache si so bi bo in cs us sy id wa st CST
0 45 652200 234464 11089268 35708720 5 26 450 5680 5 4 4 2 85 9 0 2015-11-25 15:17:11

Linux 3.10.0-123.el7.x86_64 (ceph243) 11/25/2015 x86_64 (24 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
3.98 0.00 1.82 8.86 0.00 85.33

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sdm 40.59 722.91 2556.50 138577317 490066286
sdh 22.18 1013.09 6077.32 194203261 1164985446
sdn 23.57 967.32 6864.33 185428541 1315850653
sdp 22.27 598.92 6893.63 114808894 1321467834
sdb 26.65 755.69 6880.61 144860489 1318971640
sdr 21.07 702.17 6100.73 134601462 1169472950
sds 21.91 1106.51 5930.90 212110578 1136916629
rbd3 0.00 0.01 0.07 2307 12787
rbd4 0.00 0.01 0.00 1190 0
rbd5 0.00 0.01 0.00 1245 4
rbd6 0.76 0.03 387.90 5356 74358784
sdc 18.64 751.29 4238.52 144018387 812498978
rbd7 1.75 0.01 760.52 2192 145787324
sdf 13.74 486.74 3306.03 93304892 633746824
sdd 14.90 663.62 3079.63 127212224 590346301
sdi 0.01 0.13 0.05 25396 10278
sda 83.78 206.33 30656.05 39552005 5876577856
sdj 17.44 990.81 2944.82 189933039 564504632

Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
enp130s0 1500 2655571520 0 30104 0 2346447772 0 0 0 BMRU
enp130s0 1500 6869859400 0 30104 0 6926905767 0 0 0 BMRU
enp4s0f0 1500 7057799 0 6800 0 4967149 0 0 0 BMRU
lo 65536 24923508 0 0 0 24923508 0 0 0 LRU

procs -----------memory---------- ---swap-- -----io---- system- ------cpu----- -----timestamp-----
r b swpd free buff cache si so bi bo in cs us sy id wa st CST
10 25 652196 230796 11006808 35790484 5 26 450 5680 5 4 4 2 85 9 0 2015-11-25 15:17:21

Linux 3.10.0-123.el7.x86_64 (ceph243) 11/25/2015 x86_64 (24 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
3.98 0.00 1.82 8.86 0.00 85.33

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sdm 40.59 723.12 2556.70 138625469 490128926
sdh 22.18 1013.32 6077.38 194256109 1165054763
sdn 23.58 967.47 6864.66 185466841 1315979232
sdp 22.27 599.25 6893.40 114878862 1321487762
sdb 26.65 755.87 6880.60 144902957 1319034325
sdr 21.07 702.40 6100.47 134652730 1169479951
sds 21.91 1106.63 5930.85 212144726 1136963609
rbd3 0.00 0.01 0.07 2307 12787
rbd4 0.00 0.01 0.00 1190 0
rbd5 0.00 0.01 0.00 1245 4
rbd6 0.76 0.03 387.88 5356 74358784
sdc 18.64 751.45 4238.85 144054699 812602042
rbd7 1.75 0.01 760.52 2192 145794200
sdf 13.75 486.76 3306.57 93314316 633879787
sdd 14.90 663.91 3079.55 127274636 590359302
sdi 0.01 0.13 0.05 25396 10278
sda 83.78 206.32 30656.98 39552005 5877046684
sdj 17.45 991.07 2944.88 189991491 564544256

Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
enp130s0 1500 2655583802 0 30106 0 2346510220 0 0 0 BMRU
enp130s0 1500 6869891521 0 30106 0 6927228896 0 0 0 BMRU
enp4s0f0 1500 7065855 0 6813 0 4982840 0 0 0 BMRU
lo 65536 24934717 0 0 0 24934717 0 0 0 LRU

procs -----------memory---------- ---swap-- -----io---- system- ------cpu----- -----timestamp-----
r b swpd free buff cache si so bi bo in cs us sy id wa st CST
16 19 652196 493540 10930808 35541784 5 26 450 5680 5 4 4 2 85 9 0 2015-11-25 15:17:27

Linux 3.10.0-123.el7.x86_64 (ceph243) 11/25/2015 x86_64 (24 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
3.99 0.00 1.82 8.86 0.00 85.32

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sdm 40.59 723.12 2556.65 138629569 490133578
sdh 22.19 1013.43 6077.24 194283657 1165063706
sdn 23.58 967.61 6864.61 185499317 1316010128
sdp 22.27 599.38 6893.37 114905882 1321524020
sdb 26.66 756.09 6880.67 144950309 1319089626
sdr 21.08 702.65 6100.36 134704190 1169497106
sds 21.91 1106.79 5930.94 212182222 1137017242
rbd3 0.00 0.01 0.07 2307 12787
rbd4 0.00 0.01 0.00 1190 0
rbd5 0.00 0.01 0.00 1245 4
rbd6 0.76 0.03 387.87 5356 74358784
sdc 18.64 751.56 4239.12 144080583 812679074
rbd7 1.75 0.01 760.54 2192 145802800
sdf 13.75 486.76 3306.90 93316512 633963165
sdd 14.91 664.11 3079.51 127317044 590370582
sdi 0.01 0.13 0.05 25396 10278
sda 83.79 206.31 30657.23 39552005 5877279740
sdj 17.45 991.53 2944.83 190086175 564552442

Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
enp130s0 1500 2655600194 0 30107 0 2346577287 0 0 0 BMRU
enp130s0 1500 6870255571 0 30107 0 6927424292 0 0 0 BMRU
enp4s0f0 1500 7068007 0 6813 0 4986634 0 0 0 BMRU
lo 65536 24944796 0 0 0 24944796 0 0 0 LRU

procs -----------memory---------- ---swap-- -----io---- system- ------cpu----- -----timestamp-----
r b swpd free buff cache si so bi bo in cs us sy id wa st CST
99 18 652196 281916 10909880 35735000 5 26 450 5680 5 4 4 2 85 9 0 2015-11-25 15:17:32

Linux 3.10.0-123.el7.x86_64 (ceph243) 11/25/2015 x86_64 (24 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
3.99 0.00 1.82 8.87 0.00 85.32

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sdm 40.59 723.21 2556.67 138650081 490151682
sdh 22.19 1013.81 6077.55 194363021 1165156051
sdn 23.58 967.86 6865.08 185553113 1316136143
sdp 22.27 599.64 6893.50 114960382 1321585536
sdb 26.66 756.50 6880.88 145032125 1319165678
sdr 21.08 702.99 6100.56 134773870 1169567242
sds 21.92 1107.09 5931.09 212244674 1137076602
rbd3 0.00 0.01 0.07 2307 12787
rbd4 0.00 0.01 0.00 1190 0
rbd5 0.00 0.01 0.00 1245 4
rbd6 0.76 0.03 387.86 5356 74358784
sdc 18.65 751.56 4239.51 144084759 812775819
rbd7 1.75 0.01 760.60 2192 145817360
sdf 13.75 486.81 3307.10 93328200 634018873
sdd 14.91 664.40 3079.74 127375896 590432289
sdi 0.01 0.13 0.05 25396 10278
sda 83.79 206.31 30660.44 39552005 5878057820
sdj 17.45 991.70 2944.83 190124167 564567715

Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
enp130s0 1500 2655623010 0 30107 0 2346642153 0 0 0 BMRU
enp130s0 1500 6870567718 0 30107 0 6927721556 0 0 0 BMRU
enp4s0f0 1500 7076198 0 6813 0 5001661 0 0 0 BMRU
lo 65536 24949329 0 0 0 24949329 0 0 0 LRU

procs -----------memory---------- ---swap-- -----io---- system- ------cpu----- -----timestamp-----
r b swpd free buff cache si so bi bo in cs us sy id wa st CST
32 37 652196 234444 10903408 35764852 5 26 450 5680 6 4 4 2 85 9 0 2015-11-25 15:17:37

Linux 3.10.0-123.el7.x86_64 (ceph243) 11/25/2015 x86_64 (24 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
3.99 0.00 1.82 8.87 0.00 85.32

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sdm 40.59 723.24 2556.65 138660433 490160070
sdh 22.19 1013.85 6077.49 194375261 1165176833
sdn 23.58 967.88 6865.31 185561357 1316216396
sdp 22.27 599.76 6893.42 114985078 1321605524
sdb 26.66 756.56 6880.81 145048609 1319188167
sdr 21.08 702.99 6100.40 134778070 1169568313
sds 21.92 1107.08 5930.97 212249454 1137085462
rbd3 0.00 0.01 0.07 2307 12787
rbd4 0.00 0.01 0.00 1190 0
rbd5 0.00 0.01 0.00 1245 4
rbd6 0.76 0.03 387.85 5356 74358784
sdc 18.65 751.61 4239.78 144099243 812851374
rbd7 1.75 0.01 760.59 2192 145821040
sdf 13.76 486.80 3307.47 93329704 634109036
sdd 14.91 664.50 3079.73 127397868 590445192
sdi 0.01 0.13 0.05 25396 10278
sda 83.80 206.30 30661.33 39552005 5878388940
sdj 17.46 991.76 2944.77 190140559 564571542

Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
enp130s0 1500 2655629465 0 30108 0 2346683867 0 0 0 BMRU
enp130s0 1500 6870901251 0 30108 0 6927964471 0 0 0 BMRU
enp4s0f0 1500 7079507 0 6813 0 5008007 0 0 0 BMRU
lo 65536 24953513 0 0 0 24953513 0 0 0 LRU

procs -----------memory---------- ---swap-- -----io---- system- ------cpu----- -----timestamp-----
r b swpd free buff cache si so bi bo in cs us sy id wa st CST
48 32 652196 357376 10834964 35699472 5 26 451 5680 6 4 4 2 85 9 0 2015-11-25 15:17:43

Linux 3.10.0-123.el7.x86_64 (ceph243) 11/25/2015 x86_64 (24 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
3.99 0.00 1.83 8.87 0.00 85.32

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sdm 40.59 723.28 2556.59 138670869 490164174
sdh 22.19 1014.13 6077.92 194435037 1165293005
sdn 23.59 968.21 6865.62 185631305 1316314626
sdp 22.28 600.39 6893.79 115111018 1321715234
sdb 26.67 756.76 6881.14 145090477 1319290697
sdr 21.08 703.50 6100.57 134878306 1169635440
sds 21.92 1107.85 5931.01 212402958 1137125815
rbd3 0.00 0.01 0.07 2307 12787
rbd4 0.00 0.01 0.00 1190 0
rbd5 0.00 0.01 0.00 1245 4
rbd6 0.76 0.03 387.84 5356 74358784
sdc 18.65 752.00 4240.48 144176871 813008079
rbd7 1.75 0.01 760.57 2192 145821292
sdf 13.76 486.96 3308.35 93362988 634294805
sdd 14.92 665.01 3079.98 127500096 590511200
sdi 0.01 0.13 0.05 25396 10278
sda 83.81 206.29 30663.70 39552005 5879015820
sdj 17.46 992.09 2944.80 190209827 564594169

Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
enp130s0 1500 2655636467 0 30109 0 2346695453 0 0 0 BMRU
enp130s0 1500 6871015322 0 30109 0 6928289094 0 0 0 BMRU
enp4s0f0 1500 7083631 0 6813 0 5015986 0 0 0 BMRU
lo 65536 24960990 0 0 0 24960990 0 0 0 LRU

procs -----------memory---------- ---swap-- -----io---- system- ------cpu----- -----timestamp-----
r b swpd free buff cache si so bi bo in cs us sy id wa st CST
0 51 652196 260972 10793840 35835164 5 26 451 5681 6 4 4 2 85 9 0 2015-11-25 15:17:48

Linux 3.10.0-123.el7.x86_64 (ceph243) 11/25/2015 x86_64 (24 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
3.99 0.00 1.83 8.87 0.00 85.32

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sdm 40.59 723.37 2556.66 138691369 490191210
sdh 22.20 1014.34 6078.14 194480217 1165366244
sdn 23.59 968.58 6865.86 185707397 1316397294
sdp 22.28 601.09 6893.98 115246834 1321787661
sdb 26.67 756.98 6881.59 145135461 1319411612
sdr 21.09 703.97 6100.92 134973138 1169734514
sds 21.93 1108.30 5931.13 212495058 1137180561
rbd3 0.00 0.01 0.07 2307 12787
rbd4 0.00 0.01 0.00 1190 0
rbd5 0.00 0.01 0.00 1245 4
rbd6 0.76 0.03 387.83 5356 74358784
sdc 18.66 752.26 4240.93 144230447 813117139
rbd7 1.75 0.01 760.66 2192 145841260
sdf 13.77 487.03 3308.68 93379340 634375924
sdd 14.92 665.39 3080.34 127576384 590595917
sdi 0.01 0.13 0.05 25396 10278
sda 83.81 206.29 30665.95 39552005 5879605832
sdj 17.46 992.39 2944.82 190272311 564612852

Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
enp130s0 1500 2655671466 0 30110 0 2346724314 0 0 0 BMRU
enp130s0 1500 6871443171 0 30109 0 6928728476 0 0 0 BMRU
enp4s0f0 1500 7089764 0 6814 0 5027957 0 0 0 BMRU
lo 65536 24966794 0 0 0 24966794 0 0 0 LRU

procs -----------memory---------- ---swap-- -----io---- system- ------cpu----- -----timestamp-----
r b swpd free buff cache si so bi bo in cs us sy id wa st CST
4 15 652196 450168 10764968 35662476 5 26 451 5681 6 4 4 2 85 9 0 2015-11-25 15:17:53

Linux 3.10.0-123.el7.x86_64 (ceph243) 11/25/2015 x86_64 (24 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
3.99 0.00 1.83 8.87 0.00 85.31

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sdm 40.59 723.41 2556.99 138703409 490267262
sdh 22.20 1014.57 6078.25 194529441 1165418983
sdn 23.59 969.02 6866.16 185795457 1316489992
sdp 22.28 601.28 6894.13 115287714 1321852925
sdb 26.68 757.20 6881.94 145182641 1319514255
sdr 21.09 704.30 6101.00 135040494 1169779836
sds 21.93 1108.46 5931.21 212531670 1137225773
rbd3 0.00 0.01 0.07 2307 12787
rbd4 0.00 0.01 0.00 1190 0
rbd5 0.00 0.01 0.00 1245 4
rbd6 0.76 0.03 387.82 5356 74358784
sdc 18.67 752.57 4241.29 144293987 813208233
rbd7 1.75 0.01 760.68 2192 145850580
sdf 13.77 487.14 3309.32 93402628 634515853
sdd 14.93 665.75 3080.53 127647468 590648411
sdi 0.01 0.13 0.05 25396 10278
sda 83.82 206.29 30668.17 39552261 5880188848
sdj 17.47 992.50 2944.86 190296967 564636090

Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
enp130s0 1500 2655709320 0 30111 0 2346748541 0 0 0 BMRU
enp130s0 1500 6871924978 0 30111 0 6929243759 0 0 0 BMRU
enp4s0f0 1500 7094558 0 6814 0 5037288 0 0 0 BMRU
lo 65536 24973081 0 0 0 24973081 0 0 0 LRU

procs -----------memory---------- ---swap-- -----io---- system- ------cpu----- -----timestamp-----
r b swpd free buff cache si so bi bo in cs us sy id wa st CST
49 37 652196 254496 10752504 35860492 5 26 451 5681 6 4 4 2 85 9 0 2015-11-25 15:17:59

Linux 3.10.0-123.el7.x86_64 (ceph243) 11/25/2015 x86_64 (24 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
3.99 0.00 1.83 8.87 0.00 85.31

History

#1 Updated by ceph zte over 8 years ago

In ceph the leader will keep talk to the quorum every 3s.If all the quorum answer the lease message ,

it is ok. But if there is only one does not repley the the leader query after 10S.The leader will lead to

new election.In other words if the front network of the leander and quorum is not stable, the monitor will

keep no election.But in my environment i donot find the front network is very bad as above to show.

#2 Updated by Joao Eduardo Luis over 8 years ago

  • Category set to Monitor
  • Status changed from New to Rejected
  • Assignee set to Joao Eduardo Luis

I don't think this is a bug. The problem should be the monitors being unable to reply to leases in time. This is also an older cluster, so there's a chance that's better handled in newer versions. Further discussions of this, should the problem persist, should be had on ceph-users.

As a suggestion, try increasing the lease timeouts on your cluster; e.g.,

mon lease = 10 # (default: 5)
mon lease ack timeout = 20 # (default: 10)

Adjust as needed.

Also available in: Atom PDF