2017-03-02 15:08:00.454185 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13538) propose_pending 2017-03-02 15:08:00.454222 7f4f743f0700 10 mon.0@0(leader).log v13538 encode_full log v 13538 2017-03-02 15:08:00.454384 7f4f743f0700 10 mon.0@0(leader).log v13538 encode_pending v13539 2017-03-02 15:08:00.454413 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27314) queue_pending_finisher 0x558a1412df30 2017-03-02 15:08:00.454422 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27314) trigger_propose active, proposing now 2017-03-02 15:08:00.454437 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27314) propose_pending 27315 10306 bytes 2017-03-02 15:08:00.454443 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27314) begin for 27315 10306 bytes 2017-03-02 15:08:00.487221 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27314) commit_start 27315 2017-03-02 15:08:00.512367 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27314) commit_finish 27315 2017-03-02 15:08:00.512429 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:00.512463 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12961) refresh 2017-03-02 15:08:00.512484 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..732) refresh 2017-03-02 15:08:00.512508 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:00.512528 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13539) refresh 2017-03-02 15:08:00.512535 7f4f763f4700 10 mon.0@0(leader).log v13539 update_from_paxos 2017-03-02 15:08:00.512538 7f4f763f4700 10 mon.0@0(leader).log v13539 update_from_paxos version 13539 summary v 13538 2017-03-02 15:08:00.512545 7f4f763f4700 10 mon.0@0(leader).log v13539 update_from_paxos latest full 13538 2017-03-02 15:08:00.512559 7f4f763f4700 7 mon.0@0(leader).log v13539 update_from_paxos applying incremental log 13539 2017-03-02 15:07:59.454730 mon.0 172.21.15.17:6789/0 14098 : cluster [INF] pgmap v12961: 32 pgs: 32 active+clean; 88231 MB data, 165 GB used, 206 GB / 372 GB avail; 2582 kB/s rd, 17978 kB/s wr, 540 op/s 2017-03-02 15:08:00.512583 7f4f763f4700 20 mon.0@0(leader).log v13539 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:00.512606 7f4f763f4700 15 mon.0@0(leader).log v13539 update_from_paxos logging for 1 channels 2017-03-02 15:08:00.512608 7f4f763f4700 15 mon.0@0(leader).log v13539 update_from_paxos channel 'cluster' logging 203 bytes 2017-03-02 15:08:00.512728 7f4f763f4700 10 mon.0@0(leader).log v13539 check_subs 2017-03-02 15:08:00.512788 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:00.512866 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:00.512871 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:00.512895 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:00.512898 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12961) post_refresh 2017-03-02 15:08:00.512901 7f4f763f4700 10 mon.0@0(leader).pg v12961 post_paxos_update 2017-03-02 15:08:00.512904 7f4f763f4700 10 mon.0@0(leader).pg v12961 check_subs 2017-03-02 15:08:00.512908 7f4f763f4700 20 mon.0@0(leader).pg v12961 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:00.512916 7f4f763f4700 20 mon.0@0(leader).pg v12961 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:00.512920 7f4f763f4700 20 mon.0@0(leader).pg v12961 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:00.512924 7f4f763f4700 20 mon.0@0(leader).pg v12961 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:00.512928 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..732) post_refresh 2017-03-02 15:08:00.512931 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:00.512932 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13539) post_refresh 2017-03-02 15:08:00.512934 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:00.512936 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:00.512938 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:00.512942 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27315) commit_proposal 2017-03-02 15:08:00.512946 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13539) _active - not active 2017-03-02 15:08:00.512950 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27315) finish_round 2017-03-02 15:08:00.512953 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27315) finish_round waiting_for_acting 2017-03-02 15:08:00.512956 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13539) _active 2017-03-02 15:08:00.512959 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13539) _active creating new pending 2017-03-02 15:08:00.512969 7f4f763f4700 10 mon.0@0(leader).log v13539 create_pending v 13540 2017-03-02 15:08:00.512973 7f4f763f4700 7 mon.0@0(leader).log v13539 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:00.512980 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a145dc500 0x558a15c34ee0 log(last 14098) v1 2017-03-02 15:08:00.512986 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14098) v1 -- 0x558a15c34ee0 con 0 2017-03-02 15:08:00.513016 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27315) finish_round waiting_for_readable 2017-03-02 15:08:00.513021 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27315) finish_round waiting_for_writeable 2017-03-02 15:08:00.513023 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27315) finish_round done w/ waiters, state active 2017-03-02 15:08:00.513073 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14098) v1 ==== 0+0+0 (0 0 0) 0x558a15c34ee0 con 0x558a14482000 2017-03-02 15:08:00.513104 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:00.513112 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:00.513116 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:00.513120 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:00.513122 7f4f71beb700 20 allow all 2017-03-02 15:08:01.010562 7f4f743f0700 11 mon.0@0(leader) e2 tick 2017-03-02 15:08:01.010587 7f4f743f0700 10 mon.0@0(leader).pg v12961 v12961: 32 pgs: 32 active+clean; 88231 MB data, 165 GB used, 206 GB / 372 GB avail; 2582 kB/s rd, 17978 kB/s wr, 540 op/s 2017-03-02 15:08:01.010607 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12961) maybe_trim trim_to 12461 would only trim 192 < paxos_service_trim_min 250 2017-03-02 15:08:01.010612 7f4f743f0700 10 mon.0@0(leader).mds e732 e732: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:01.010651 7f4f743f0700 10 mon.0@0(leader).paxosservice(mdsmap 1..732) maybe_trim trim_to 232 would only trim 231 < paxos_service_trim_min 250 2017-03-02 15:08:01.010654 7f4f743f0700 10 mon.0@0(leader).osd e7 e7: 4 osds: 4 up, 4 in 2017-03-02 15:08:01.010660 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27315) is_readable = 1 - now=2017-03-02 15:08:01.010661 lease_expire=0.000000 has v0 lc 27315 2017-03-02 15:08:01.010670 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27315) is_readable = 1 - now=2017-03-02 15:08:01.010671 lease_expire=0.000000 has v0 lc 27315 2017-03-02 15:08:01.010679 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27315) is_readable = 1 - now=2017-03-02 15:08:01.010679 lease_expire=0.000000 has v0 lc 27315 2017-03-02 15:08:01.010683 7f4f743f0700 10 mon.0@0(leader).osd e7 min_last_epoch_clean 6 2017-03-02 15:08:01.010686 7f4f743f0700 10 mon.0@0(leader).log v13539 log 2017-03-02 15:08:01.010688 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13539) maybe_trim trim_to 13039 would only trim 12 < paxos_service_trim_min 250 2017-03-02 15:08:01.010692 7f4f743f0700 10 mon.0@0(leader).auth v10 auth 2017-03-02 15:08:01.010696 7f4f743f0700 10 cephx keyserver: _check_rotating_secrets 2017-03-02 15:08:01.010706 7f4f743f0700 20 mon.0@0(leader) e2 sync_trim_providers 2017-03-02 15:08:02.869987 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.5 172.21.15.11:6804/1637448123 8779 ==== mdsbeacon(4116/e up:active seq 8492 v732) v7 ==== 365+0+0 (2283204953 0 0) 0x558a148d8080 con 0x558a1471d800 2017-03-02 15:08:02.870031 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b0f20 for mds.? 172.21.15.11:6804/1637448123 2017-03-02 15:08:02.870039 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:02.870043 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..732) dispatch 0x558a148d8080 mdsbeacon(4116/e up:active seq 8492 v732) v7 from mds.5 172.21.15.11:6804/1637448123 con 0x558a1471d800 2017-03-02 15:08:02.870052 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27315) is_readable = 1 - now=2017-03-02 15:08:02.870052 lease_expire=0.000000 has v0 lc 27315 2017-03-02 15:08:02.870064 7f4f71beb700 10 mon.0@0(leader).mds e732 preprocess_query mdsbeacon(4116/e up:active seq 8492 v732) v7 from mds.5 172.21.15.11:6804/1637448123 2017-03-02 15:08:02.870074 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:02.870078 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:02.870079 7f4f71beb700 20 allow all 2017-03-02 15:08:02.870081 7f4f71beb700 12 mon.0@0(leader).mds e732 preprocess_beacon mdsbeacon(4116/e up:active seq 8492 v732) v7 from mds.5 172.21.15.11:6804/1637448123 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:02.870092 7f4f71beb700 10 mon.0@0(leader).mds e732 preprocess_beacon: GID exists in map: 4116 2017-03-02 15:08:02.870099 7f4f71beb700 15 mon.0@0(leader).mds e732 _note_beacon mdsbeacon(4116/e up:active seq 8492 v732) v7 noting time 2017-03-02 15:08:02.870103 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a1498f800 0x558a160c0d80 mdsbeacon(4116/e up:active seq 8492 v732) v7 2017-03-02 15:08:02.870108 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6804/1637448123 -- mdsbeacon(4116/e up:active seq 8492 v732) v7 -- 0x558a160c0d80 con 0 2017-03-02 15:08:02.883625 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.3 172.21.15.11:6805/1926402648 8555 ==== mdsbeacon(4118/f up:active seq 8492 v732) v7 ==== 406+0+0 (1167370377 0 0) 0x558a15204000 con 0x558a14905800 2017-03-02 15:08:02.883671 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b2260 for mds.? 172.21.15.11:6805/1926402648 2017-03-02 15:08:02.883680 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:02.883684 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..732) dispatch 0x558a15204000 mdsbeacon(4118/f up:active seq 8492 v732) v7 from mds.3 172.21.15.11:6805/1926402648 con 0x558a14905800 2017-03-02 15:08:02.883692 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27315) is_readable = 1 - now=2017-03-02 15:08:02.883693 lease_expire=0.000000 has v0 lc 27315 2017-03-02 15:08:02.883705 7f4f71beb700 10 mon.0@0(leader).mds e732 preprocess_query mdsbeacon(4118/f up:active seq 8492 v732) v7 from mds.3 172.21.15.11:6805/1926402648 2017-03-02 15:08:02.883714 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:02.883717 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:02.883719 7f4f71beb700 20 allow all 2017-03-02 15:08:02.883720 7f4f71beb700 12 mon.0@0(leader).mds e732 preprocess_beacon mdsbeacon(4118/f up:active seq 8492 v732) v7 from mds.3 172.21.15.11:6805/1926402648 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:02.883730 7f4f71beb700 10 mon.0@0(leader).mds e732 preprocess_beacon: GID exists in map: 4118 2017-03-02 15:08:02.883736 7f4f71beb700 15 mon.0@0(leader).mds e732 _note_beacon mdsbeacon(4118/f up:active seq 8492 v732) v7 noting time 2017-03-02 15:08:02.883741 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a1498f800 0x558a148d8080 mdsbeacon(4118/f up:active seq 8492 v732) v7 2017-03-02 15:08:02.883746 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6805/1926402648 -- mdsbeacon(4118/f up:active seq 8492 v732) v7 -- 0x558a148d8080 con 0 2017-03-02 15:08:02.884462 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.2 172.21.15.22:6804/673088617 8554 ==== mdsbeacon(4119/c up:active seq 8492 v732) v7 ==== 365+0+0 (2819618055 0 0) 0x558a160c0d80 con 0x558a14907000 2017-03-02 15:08:02.884504 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14916f20 for mds.? 172.21.15.22:6804/673088617 2017-03-02 15:08:02.884512 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:02.884516 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..732) dispatch 0x558a160c0d80 mdsbeacon(4119/c up:active seq 8492 v732) v7 from mds.2 172.21.15.22:6804/673088617 con 0x558a14907000 2017-03-02 15:08:02.884525 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27315) is_readable = 1 - now=2017-03-02 15:08:02.884526 lease_expire=0.000000 has v0 lc 27315 2017-03-02 15:08:02.884537 7f4f71beb700 10 mon.0@0(leader).mds e732 preprocess_query mdsbeacon(4119/c up:active seq 8492 v732) v7 from mds.2 172.21.15.22:6804/673088617 2017-03-02 15:08:02.884546 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:02.884549 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:02.884552 7f4f71beb700 20 allow all 2017-03-02 15:08:02.884554 7f4f71beb700 12 mon.0@0(leader).mds e732 preprocess_beacon mdsbeacon(4119/c up:active seq 8492 v732) v7 from mds.2 172.21.15.22:6804/673088617 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:02.884571 7f4f71beb700 10 mon.0@0(leader).mds e732 preprocess_beacon: GID exists in map: 4119 2017-03-02 15:08:02.884579 7f4f71beb700 15 mon.0@0(leader).mds e732 _note_beacon mdsbeacon(4119/c up:active seq 8492 v732) v7 noting time 2017-03-02 15:08:02.884587 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a1498f800 0x558a15204000 mdsbeacon(4119/c up:active seq 8492 v732) v7 2017-03-02 15:08:02.884594 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6804/673088617 -- mdsbeacon(4119/c up:active seq 8492 v732) v7 -- 0x558a15204000 con 0 2017-03-02 15:08:02.884728 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.0 172.21.15.22:6805/2203629154 8604 ==== mdsbeacon(4121/d up:active seq 8488 v732) v7 ==== 454+0+0 (3304994173 0 0) 0x558a148d8080 con 0x558a1492a000 2017-03-02 15:08:02.884773 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a149195a0 for mds.? 172.21.15.22:6805/2203629154 2017-03-02 15:08:02.884784 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:02.884789 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..732) dispatch 0x558a148d8080 mdsbeacon(4121/d up:active seq 8488 v732) v7 from mds.0 172.21.15.22:6805/2203629154 con 0x558a1492a000 2017-03-02 15:08:02.884802 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27315) is_readable = 1 - now=2017-03-02 15:08:02.884803 lease_expire=0.000000 has v0 lc 27315 2017-03-02 15:08:02.884818 7f4f71beb700 10 mon.0@0(leader).mds e732 preprocess_query mdsbeacon(4121/d up:active seq 8488 v732) v7 from mds.0 172.21.15.22:6805/2203629154 2017-03-02 15:08:02.884867 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:02.884872 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:02.884875 7f4f71beb700 20 allow all 2017-03-02 15:08:02.884877 7f4f71beb700 12 mon.0@0(leader).mds e732 preprocess_beacon mdsbeacon(4121/d up:active seq 8488 v732) v7 from mds.0 172.21.15.22:6805/2203629154 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:02.884887 7f4f71beb700 10 mon.0@0(leader).mds e732 preprocess_beacon: GID exists in map: 4121 2017-03-02 15:08:02.884894 7f4f71beb700 15 mon.0@0(leader).mds e732 _note_beacon mdsbeacon(4121/d up:active seq 8488 v732) v7 noting time 2017-03-02 15:08:02.884899 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a1498f800 0x558a160c0d80 mdsbeacon(4121/d up:active seq 8488 v732) v7 2017-03-02 15:08:02.884903 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6805/2203629154 -- mdsbeacon(4121/d up:active seq 8488 v732) v7 -- 0x558a160c0d80 con 0 2017-03-02 15:08:02.923872 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.1 172.21.15.17:6804/2576824570 8551 ==== mdsbeacon(4120/g up:active seq 8492 v732) v7 ==== 365+0+0 (1610659868 0 0) 0x558a154b8d00 con 0x558a14908800 2017-03-02 15:08:02.923912 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14917600 for mds.? 172.21.15.17:6804/2576824570 2017-03-02 15:08:02.923920 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:02.923924 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..732) dispatch 0x558a154b8d00 mdsbeacon(4120/g up:active seq 8492 v732) v7 from mds.1 172.21.15.17:6804/2576824570 con 0x558a14908800 2017-03-02 15:08:02.923932 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27315) is_readable = 1 - now=2017-03-02 15:08:02.923933 lease_expire=0.000000 has v0 lc 27315 2017-03-02 15:08:02.923944 7f4f71beb700 10 mon.0@0(leader).mds e732 preprocess_query mdsbeacon(4120/g up:active seq 8492 v732) v7 from mds.1 172.21.15.17:6804/2576824570 2017-03-02 15:08:02.923953 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:02.923957 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:02.923959 7f4f71beb700 20 allow all 2017-03-02 15:08:02.923961 7f4f71beb700 12 mon.0@0(leader).mds e732 preprocess_beacon mdsbeacon(4120/g up:active seq 8492 v732) v7 from mds.1 172.21.15.17:6804/2576824570 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:02.923971 7f4f71beb700 10 mon.0@0(leader).mds e732 preprocess_beacon: GID exists in map: 4120 2017-03-02 15:08:02.923976 7f4f71beb700 15 mon.0@0(leader).mds e732 _note_beacon mdsbeacon(4120/g up:active seq 8492 v732) v7 noting time 2017-03-02 15:08:02.923981 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a1498f800 0x558a148d8080 mdsbeacon(4120/g up:active seq 8492 v732) v7 2017-03-02 15:08:02.923986 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6804/2576824570 -- mdsbeacon(4120/g up:active seq 8492 v732) v7 -- 0x558a148d8080 con 0 2017-03-02 15:08:02.968764 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.4 172.21.15.26:6805/2033388711 8549 ==== mdsbeacon(4117/b up:active seq 8492 v732) v7 ==== 365+0+0 (2661918431 0 0) 0x558a148d8080 con 0x558a14904000 2017-03-02 15:08:02.968809 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b1600 for mds.? 172.21.15.26:6805/2033388711 2017-03-02 15:08:02.968816 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:02.968865 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..732) dispatch 0x558a148d8080 mdsbeacon(4117/b up:active seq 8492 v732) v7 from mds.4 172.21.15.26:6805/2033388711 con 0x558a14904000 2017-03-02 15:08:02.968876 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27315) is_readable = 1 - now=2017-03-02 15:08:02.968877 lease_expire=0.000000 has v0 lc 27315 2017-03-02 15:08:02.968889 7f4f71beb700 10 mon.0@0(leader).mds e732 preprocess_query mdsbeacon(4117/b up:active seq 8492 v732) v7 from mds.4 172.21.15.26:6805/2033388711 2017-03-02 15:08:02.968899 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:02.968902 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:02.968903 7f4f71beb700 20 allow all 2017-03-02 15:08:02.968905 7f4f71beb700 12 mon.0@0(leader).mds e732 preprocess_beacon mdsbeacon(4117/b up:active seq 8492 v732) v7 from mds.4 172.21.15.26:6805/2033388711 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:02.968916 7f4f71beb700 10 mon.0@0(leader).mds e732 preprocess_beacon: GID exists in map: 4117 2017-03-02 15:08:02.968921 7f4f71beb700 15 mon.0@0(leader).mds e732 _note_beacon mdsbeacon(4117/b up:active seq 8492 v732) v7 noting time 2017-03-02 15:08:02.968926 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a1498f800 0x558a154b8d00 mdsbeacon(4117/b up:active seq 8492 v732) v7 2017-03-02 15:08:02.968930 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6805/2033388711 -- mdsbeacon(4117/b up:active seq 8492 v732) v7 -- 0x558a154b8d00 con 0 2017-03-02 15:08:02.978003 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.6 172.21.15.26:6804/2954782670 8741 ==== mdsbeacon(4115/a up:active seq 8492 v732) v7 ==== 365+0+0 (3309523639 0 0) 0x558a160c0d80 con 0x558a1471f000 2017-03-02 15:08:02.978048 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14868d60 for mds.? 172.21.15.26:6804/2954782670 2017-03-02 15:08:02.978056 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:02.978059 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..732) dispatch 0x558a160c0d80 mdsbeacon(4115/a up:active seq 8492 v732) v7 from mds.6 172.21.15.26:6804/2954782670 con 0x558a1471f000 2017-03-02 15:08:02.978068 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27315) is_readable = 1 - now=2017-03-02 15:08:02.978068 lease_expire=0.000000 has v0 lc 27315 2017-03-02 15:08:02.978080 7f4f71beb700 10 mon.0@0(leader).mds e732 preprocess_query mdsbeacon(4115/a up:active seq 8492 v732) v7 from mds.6 172.21.15.26:6804/2954782670 2017-03-02 15:08:02.978089 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:02.978092 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:02.978094 7f4f71beb700 20 allow all 2017-03-02 15:08:02.978096 7f4f71beb700 12 mon.0@0(leader).mds e732 preprocess_beacon mdsbeacon(4115/a up:active seq 8492 v732) v7 from mds.6 172.21.15.26:6804/2954782670 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:02.978106 7f4f71beb700 10 mon.0@0(leader).mds e732 preprocess_beacon: GID exists in map: 4115 2017-03-02 15:08:02.978112 7f4f71beb700 15 mon.0@0(leader).mds e732 _note_beacon mdsbeacon(4115/a up:active seq 8492 v732) v7 noting time 2017-03-02 15:08:02.978117 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a1498f800 0x558a148d8080 mdsbeacon(4115/a up:active seq 8492 v732) v7 2017-03-02 15:08:02.978121 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6804/2954782670 -- mdsbeacon(4115/a up:active seq 8492 v732) v7 -- 0x558a148d8080 con 0 2017-03-02 15:08:03.258815 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.3 172.21.15.17:6800/27100 6829 ==== pg_stats(1 pgs tid 6793 v 0) v1 ==== 698+0+0 (3062676894 0 0) 0x558a15c60080 con 0x558a145d7000 2017-03-02 15:08:03.258871 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415f440 for osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:03.258881 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:03.258885 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12961) dispatch 0x558a15c60080 pg_stats(1 pgs tid 6793 v 0) v1 from osd.3 172.21.15.17:6800/27100 con 0x558a145d7000 2017-03-02 15:08:03.258893 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27315) is_readable = 1 - now=2017-03-02 15:08:03.258894 lease_expire=0.000000 has v0 lc 27315 2017-03-02 15:08:03.258905 7f4f71beb700 10 mon.0@0(leader).pg v12961 preprocess_query pg_stats(1 pgs tid 6793 v 0) v1 from osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:03.258914 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:03.258919 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:03.258920 7f4f71beb700 20 allow all 2017-03-02 15:08:03.258923 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27315) is_readable = 1 - now=2017-03-02 15:08:03.258923 lease_expire=0.000000 has v0 lc 27315 2017-03-02 15:08:03.258930 7f4f71beb700 10 mon.0@0(leader).pg v12961 prepare_update pg_stats(1 pgs tid 6793 v 0) v1 from osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:03.258938 7f4f71beb700 10 mon.0@0(leader).pg v12961 prepare_pg_stats pg_stats(1 pgs tid 6793 v 0) v1 from osd.3 2017-03-02 15:08:03.258947 7f4f71beb700 10 mon.0@0(leader).pg v12961 got osd.3 osd_stat(42049 MB used, 53292 MB avail, 95341 MB total, peers [0,1,2] op hist []) (was osd_stat(42060 MB used, 53281 MB avail, 95341 MB total, peers [0,1,2] op hist [])) 2017-03-02 15:08:03.258960 7f4f71beb700 15 mon.0@0(leader).pg v12961 got 2.6 reported at 7:963678 state active+clean -> active+clean 2017-03-02 15:08:03.258975 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12961) setting proposal_timer 0x558a1412e3f0 with delay of 0.05 2017-03-02 15:08:03.309061 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12961) propose_pending 2017-03-02 15:08:03.309076 7f4f743f0700 10 mon.0@0(leader).pg v12961 encode_pending v 12962 2017-03-02 15:08:03.309103 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27315) queue_pending_finisher 0x558a1412d050 2017-03-02 15:08:03.309106 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27315) trigger_propose active, proposing now 2017-03-02 15:08:03.309114 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27315) propose_pending 27316 1090 bytes 2017-03-02 15:08:03.309116 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27315) begin for 27316 1090 bytes 2017-03-02 15:08:03.320594 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27315) commit_start 27316 2017-03-02 15:08:03.328957 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27315) commit_finish 27316 2017-03-02 15:08:03.329014 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:03.329054 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12962) refresh 2017-03-02 15:08:03.329065 7f4f763f4700 10 mon.0@0(leader).pg v12961 update_from_paxos read_incremental 2017-03-02 15:08:03.329098 7f4f763f4700 20 mon.0@0(leader).pg v12961 refreshing pg 2.6 7:963678 active+clean 2017-03-02 15:08:03.329112 7f4f763f4700 20 mon.0@0(leader).pg v12961 refreshing osd.3 2017-03-02 15:08:03.329130 7f4f763f4700 10 mon.0@0(leader).pg v12962 read_pgmap_meta 2017-03-02 15:08:03.329169 7f4f763f4700 10 mon.0@0(leader).pg v12962 update_logger 2017-03-02 15:08:03.329196 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..732) refresh 2017-03-02 15:08:03.329227 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:03.329257 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13539) refresh 2017-03-02 15:08:03.329266 7f4f763f4700 10 mon.0@0(leader).log v13539 update_from_paxos 2017-03-02 15:08:03.329276 7f4f763f4700 10 mon.0@0(leader).log v13539 update_from_paxos version 13539 summary v 13539 2017-03-02 15:08:03.329296 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:03.329318 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:03.329327 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:03.329349 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:03.329359 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12962) post_refresh 2017-03-02 15:08:03.329368 7f4f763f4700 10 mon.0@0(leader).pg v12962 post_paxos_update 2017-03-02 15:08:03.329371 7f4f763f4700 10 mon.0@0(leader).pg v12962 check_subs 2017-03-02 15:08:03.329375 7f4f763f4700 20 mon.0@0(leader).pg v12962 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:03.329385 7f4f763f4700 20 mon.0@0(leader).pg v12962 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:03.329389 7f4f763f4700 20 mon.0@0(leader).pg v12962 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:03.329393 7f4f763f4700 20 mon.0@0(leader).pg v12962 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:03.329398 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..732) post_refresh 2017-03-02 15:08:03.329400 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:03.329402 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13539) post_refresh 2017-03-02 15:08:03.329404 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:03.329406 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:03.329408 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:03.329411 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27316) commit_proposal 2017-03-02 15:08:03.329416 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12962) _active - not active 2017-03-02 15:08:03.329419 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27316) finish_round 2017-03-02 15:08:03.329422 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27316) finish_round waiting_for_acting 2017-03-02 15:08:03.329425 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12962) _active 2017-03-02 15:08:03.329428 7f4f763f4700 7 mon.0@0(leader).paxosservice(pgmap 12269..12962) _active creating new pending 2017-03-02 15:08:03.329432 7f4f763f4700 10 mon.0@0(leader).pg v12962 create_pending v 12963 2017-03-02 15:08:03.329439 7f4f763f4700 7 mon.0@0(leader).pg v12962 _updated_stats for osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:03.329445 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a1498f800 0x558a15298200 pg_stats_ack(1 pgs tid 6793) v1 2017-03-02 15:08:03.329451 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6800/27100 -- pg_stats_ack(1 pgs tid 6793) v1 -- 0x558a15298200 con 0 2017-03-02 15:08:03.329518 7f4f763f4700 10 mon.0@0(leader).pg v12962 check_osd_map already seen 7 >= 7 2017-03-02 15:08:03.329523 7f4f763f4700 10 mon.0@0(leader).pg v12962 update_logger 2017-03-02 15:08:03.329545 7f4f763f4700 0 log_channel(cluster) log [INF] : pgmap v12962: 32 pgs: 32 active+clean; 88231 MB data, 165 GB used, 206 GB / 372 GB avail; 1963 kB/s rd, 11622 kB/s wr, 334 op/s 2017-03-02 15:08:03.329567 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14099 at 2017-03-02 15:08:03.329548) v1 -- 0x558a15a10f40 con 0 2017-03-02 15:08:03.329591 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27316) finish_round waiting_for_readable 2017-03-02 15:08:03.329595 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27316) finish_round waiting_for_writeable 2017-03-02 15:08:03.329598 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27316) finish_round done w/ waiters, state active 2017-03-02 15:08:03.329619 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14099 at 2017-03-02 15:08:03.329548) v1 ==== 0+0+0 (0 0 0) 0x558a15a10f40 con 0x558a14482000 2017-03-02 15:08:03.329642 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:03.329649 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:03.329652 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13539) dispatch 0x558a15a10f40 log(1 entries from seq 14099 at 2017-03-02 15:08:03.329548) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:03.329660 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27316) is_readable = 1 - now=2017-03-02 15:08:03.329661 lease_expire=0.000000 has v0 lc 27316 2017-03-02 15:08:03.329668 7f4f71beb700 10 mon.0@0(leader).log v13539 preprocess_query log(1 entries from seq 14099 at 2017-03-02 15:08:03.329548) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:03.329677 7f4f71beb700 10 mon.0@0(leader).log v13539 preprocess_log log(1 entries from seq 14099 at 2017-03-02 15:08:03.329548) v1 from mon.0 2017-03-02 15:08:03.329682 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:03.329685 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:03.329687 7f4f71beb700 20 allow all 2017-03-02 15:08:03.329695 7f4f71beb700 10 mon.0@0(leader).log v13539 prepare_update log(1 entries from seq 14099 at 2017-03-02 15:08:03.329548) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:03.329703 7f4f71beb700 10 mon.0@0(leader).log v13539 prepare_log log(1 entries from seq 14099 at 2017-03-02 15:08:03.329548) v1 from mon.0 2017-03-02 15:08:03.329708 7f4f71beb700 10 mon.0@0(leader).log v13539 logging 2017-03-02 15:08:03.329548 mon.0 172.21.15.17:6789/0 14099 : cluster [INF] pgmap v12962: 32 pgs: 32 active+clean; 88231 MB data, 165 GB used, 206 GB / 372 GB avail; 1963 kB/s rd, 11622 kB/s wr, 334 op/s 2017-03-02 15:08:03.329719 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13539) setting proposal_timer 0x558a1412c4a0 with delay of 0.999262 2017-03-02 15:08:03.461946 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.1 172.21.15.22:6800/26531 6829 ==== pg_stats(4 pgs tid 6793 v 0) v1 ==== 2396+0+0 (2126690717 0 0) 0x558a15c08c80 con 0x558a14643800 2017-03-02 15:08:03.461969 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1463b1e0 for osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:03.461975 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:03.461978 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12962) dispatch 0x558a15c08c80 pg_stats(4 pgs tid 6793 v 0) v1 from osd.1 172.21.15.22:6800/26531 con 0x558a14643800 2017-03-02 15:08:03.461984 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27316) is_readable = 1 - now=2017-03-02 15:08:03.461985 lease_expire=0.000000 has v0 lc 27316 2017-03-02 15:08:03.461992 7f4f71beb700 10 mon.0@0(leader).pg v12962 preprocess_query pg_stats(4 pgs tid 6793 v 0) v1 from osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:03.461999 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:03.462002 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:03.462003 7f4f71beb700 20 allow all 2017-03-02 15:08:03.462005 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27316) is_readable = 1 - now=2017-03-02 15:08:03.462006 lease_expire=0.000000 has v0 lc 27316 2017-03-02 15:08:03.462012 7f4f71beb700 10 mon.0@0(leader).pg v12962 prepare_update pg_stats(4 pgs tid 6793 v 0) v1 from osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:03.462018 7f4f71beb700 10 mon.0@0(leader).pg v12962 prepare_pg_stats pg_stats(4 pgs tid 6793 v 0) v1 from osd.1 2017-03-02 15:08:03.462025 7f4f71beb700 10 mon.0@0(leader).pg v12962 got osd.1 osd_stat(52989 MB used, 42352 MB avail, 95341 MB total, peers [0,2,3] op hist []) (was osd_stat(53005 MB used, 42335 MB avail, 95341 MB total, peers [0,2,3] op hist [])) 2017-03-02 15:08:03.462037 7f4f71beb700 15 mon.0@0(leader).pg v12962 got 1.1 reported at 7:586127 state active+clean -> active+clean 2017-03-02 15:08:03.462047 7f4f71beb700 15 mon.0@0(leader).pg v12962 got 1.5 reported at 7:567368 state active+clean -> active+clean 2017-03-02 15:08:03.462054 7f4f71beb700 15 mon.0@0(leader).pg v12962 got 2.4 reported at 7:1025358 state active+clean -> active+clean 2017-03-02 15:08:03.462061 7f4f71beb700 15 mon.0@0(leader).pg v12962 got 2.5 reported at 7:1128642 state active+clean -> active+clean 2017-03-02 15:08:03.462068 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12962) setting proposal_timer 0x558a1412e830 with delay of 0.866913 2017-03-02 15:08:03.554758 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.2 172.21.15.11:6800/26536 6829 ==== pg_stats(7 pgs tid 6793 v 0) v1 ==== 4094+0+0 (3063722831 0 0) 0x558a15aea300 con 0x558a145d5800 2017-03-02 15:08:03.554783 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a145c3700 for osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:03.554789 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:03.554792 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12962) dispatch 0x558a15aea300 pg_stats(7 pgs tid 6793 v 0) v1 from osd.2 172.21.15.11:6800/26536 con 0x558a145d5800 2017-03-02 15:08:03.554798 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27316) is_readable = 1 - now=2017-03-02 15:08:03.554799 lease_expire=0.000000 has v0 lc 27316 2017-03-02 15:08:03.554807 7f4f71beb700 10 mon.0@0(leader).pg v12962 preprocess_query pg_stats(7 pgs tid 6793 v 0) v1 from osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:03.554813 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:03.554816 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:03.554817 7f4f71beb700 20 allow all 2017-03-02 15:08:03.554818 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27316) is_readable = 1 - now=2017-03-02 15:08:03.554847 lease_expire=0.000000 has v0 lc 27316 2017-03-02 15:08:03.554855 7f4f71beb700 10 mon.0@0(leader).pg v12962 prepare_update pg_stats(7 pgs tid 6793 v 0) v1 from osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:03.554862 7f4f71beb700 10 mon.0@0(leader).pg v12962 prepare_pg_stats pg_stats(7 pgs tid 6793 v 0) v1 from osd.2 2017-03-02 15:08:03.554867 7f4f71beb700 10 mon.0@0(leader).pg v12962 got osd.2 osd_stat(63682 MB used, 31659 MB avail, 95341 MB total, peers [0,1,3] op hist []) (was osd_stat(63687 MB used, 31654 MB avail, 95341 MB total, peers [0,1,3] op hist [])) 2017-03-02 15:08:03.554877 7f4f71beb700 15 mon.0@0(leader).pg v12962 got 1.0 reported at 7:495990 state active+clean -> active+clean 2017-03-02 15:08:03.554887 7f4f71beb700 15 mon.0@0(leader).pg v12962 got 1.7 reported at 7:529024 state active+clean -> active+clean 2017-03-02 15:08:03.554893 7f4f71beb700 15 mon.0@0(leader).pg v12962 got 2.0 reported at 7:1055951 state active+clean -> active+clean 2017-03-02 15:08:03.554898 7f4f71beb700 15 mon.0@0(leader).pg v12962 got 2.1 reported at 7:1088667 state active+clean -> active+clean 2017-03-02 15:08:03.554904 7f4f71beb700 15 mon.0@0(leader).pg v12962 got 2.2 reported at 7:1162398 state active+clean -> active+clean 2017-03-02 15:08:03.554909 7f4f71beb700 15 mon.0@0(leader).pg v12962 got 2.3 reported at 7:1094551 state active+clean -> active+clean 2017-03-02 15:08:03.554915 7f4f71beb700 15 mon.0@0(leader).pg v12962 got 2.7 reported at 7:1134585 state active+clean -> active+clean 2017-03-02 15:08:03.554922 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12962) proposal_timer already set 2017-03-02 15:08:03.708409 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.0 172.21.15.26:6800/26515 6829 ==== pg_stats(2 pgs tid 6793 v 0) v1 ==== 1264+0+0 (2923149425 0 0) 0x558a158d3480 con 0x558a145d8800 2017-03-02 15:08:03.708454 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1463af20 for osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:03.708471 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:03.708477 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12962) dispatch 0x558a158d3480 pg_stats(2 pgs tid 6793 v 0) v1 from osd.0 172.21.15.26:6800/26515 con 0x558a145d8800 2017-03-02 15:08:03.708490 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27316) is_readable = 1 - now=2017-03-02 15:08:03.708491 lease_expire=0.000000 has v0 lc 27316 2017-03-02 15:08:03.708506 7f4f71beb700 10 mon.0@0(leader).pg v12962 preprocess_query pg_stats(2 pgs tid 6793 v 0) v1 from osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:03.708519 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:03.708524 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:03.708527 7f4f71beb700 20 allow all 2017-03-02 15:08:03.708530 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27316) is_readable = 1 - now=2017-03-02 15:08:03.708531 lease_expire=0.000000 has v0 lc 27316 2017-03-02 15:08:03.708542 7f4f71beb700 10 mon.0@0(leader).pg v12962 prepare_update pg_stats(2 pgs tid 6793 v 0) v1 from osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:03.708554 7f4f71beb700 10 mon.0@0(leader).pg v12962 prepare_pg_stats pg_stats(2 pgs tid 6793 v 0) v1 from osd.0 2017-03-02 15:08:03.708564 7f4f71beb700 10 mon.0@0(leader).pg v12962 got osd.0 osd_stat(10935 MB used, 84405 MB avail, 95341 MB total, peers [1,2,3] op hist []) (was osd_stat(10941 MB used, 84400 MB avail, 95341 MB total, peers [1,2,3] op hist [])) 2017-03-02 15:08:03.708583 7f4f71beb700 15 mon.0@0(leader).pg v12962 got 1.3 reported at 7:499529 state active+clean -> active+clean 2017-03-02 15:08:03.708601 7f4f71beb700 15 mon.0@0(leader).pg v12962 got 1.4 reported at 7:485981 state active+clean -> active+clean 2017-03-02 15:08:03.708615 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12962) proposal_timer already set 2017-03-02 15:08:04.329110 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12962) propose_pending 2017-03-02 15:08:04.329132 7f4f743f0700 10 mon.0@0(leader).pg v12962 encode_pending v 12963 2017-03-02 15:08:04.329233 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27316) queue_pending_finisher 0x558a1412e3f0 2017-03-02 15:08:04.329238 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27316) trigger_propose active, proposing now 2017-03-02 15:08:04.329263 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27316) propose_pending 27317 8536 bytes 2017-03-02 15:08:04.329267 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27316) begin for 27317 8536 bytes 2017-03-02 15:08:04.353734 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27316) commit_start 27317 2017-03-02 15:08:04.353876 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13539) propose_pending 2017-03-02 15:08:04.353883 7f4f743f0700 10 mon.0@0(leader).log v13539 encode_full log v 13539 2017-03-02 15:08:04.353974 7f4f743f0700 10 mon.0@0(leader).log v13539 encode_pending v13540 2017-03-02 15:08:04.353993 7f4f743f0700 5 mon.0@0(leader).paxos(paxos writing c 26607..27316) queue_pending_finisher 0x558a1412e830 2017-03-02 15:08:04.353998 7f4f743f0700 10 mon.0@0(leader).paxos(paxos writing c 26607..27316) trigger_propose not active, will propose later 2017-03-02 15:08:04.378937 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27316) commit_finish 27317 2017-03-02 15:08:04.379001 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:04.379043 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12963) refresh 2017-03-02 15:08:04.379048 7f4f763f4700 10 mon.0@0(leader).pg v12962 update_from_paxos read_incremental 2017-03-02 15:08:04.379086 7f4f763f4700 20 mon.0@0(leader).pg v12962 refreshing pg 1.0 7:495990 active+clean 2017-03-02 15:08:04.379111 7f4f763f4700 20 mon.0@0(leader).pg v12962 refreshing pg 1.1 7:586127 active+clean 2017-03-02 15:08:04.379131 7f4f763f4700 20 mon.0@0(leader).pg v12962 refreshing pg 1.3 7:499529 active+clean 2017-03-02 15:08:04.379149 7f4f763f4700 20 mon.0@0(leader).pg v12962 refreshing pg 1.4 7:485981 active+clean 2017-03-02 15:08:04.379168 7f4f763f4700 20 mon.0@0(leader).pg v12962 refreshing pg 1.5 7:567368 active+clean 2017-03-02 15:08:04.379185 7f4f763f4700 20 mon.0@0(leader).pg v12962 refreshing pg 1.7 7:529024 active+clean 2017-03-02 15:08:04.379204 7f4f763f4700 20 mon.0@0(leader).pg v12962 refreshing pg 2.0 7:1055951 active+clean 2017-03-02 15:08:04.379223 7f4f763f4700 20 mon.0@0(leader).pg v12962 refreshing pg 2.1 7:1088667 active+clean 2017-03-02 15:08:04.379241 7f4f763f4700 20 mon.0@0(leader).pg v12962 refreshing pg 2.2 7:1162398 active+clean 2017-03-02 15:08:04.379258 7f4f763f4700 20 mon.0@0(leader).pg v12962 refreshing pg 2.3 7:1094551 active+clean 2017-03-02 15:08:04.379275 7f4f763f4700 20 mon.0@0(leader).pg v12962 refreshing pg 2.4 7:1025358 active+clean 2017-03-02 15:08:04.379292 7f4f763f4700 20 mon.0@0(leader).pg v12962 refreshing pg 2.5 7:1128642 active+clean 2017-03-02 15:08:04.379309 7f4f763f4700 20 mon.0@0(leader).pg v12962 refreshing pg 2.7 7:1134585 active+clean 2017-03-02 15:08:04.379316 7f4f763f4700 20 mon.0@0(leader).pg v12962 refreshing osd.0 2017-03-02 15:08:04.379330 7f4f763f4700 20 mon.0@0(leader).pg v12962 refreshing osd.1 2017-03-02 15:08:04.379341 7f4f763f4700 20 mon.0@0(leader).pg v12962 refreshing osd.2 2017-03-02 15:08:04.379358 7f4f763f4700 10 mon.0@0(leader).pg v12963 read_pgmap_meta 2017-03-02 15:08:04.379409 7f4f763f4700 10 mon.0@0(leader).pg v12963 update_logger 2017-03-02 15:08:04.379440 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..732) refresh 2017-03-02 15:08:04.379475 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:04.379503 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13539) refresh 2017-03-02 15:08:04.379507 7f4f763f4700 10 mon.0@0(leader).log v13539 update_from_paxos 2017-03-02 15:08:04.379511 7f4f763f4700 10 mon.0@0(leader).log v13539 update_from_paxos version 13539 summary v 13539 2017-03-02 15:08:04.379540 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:04.379573 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:04.379577 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:04.379611 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:04.379616 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12963) post_refresh 2017-03-02 15:08:04.379620 7f4f763f4700 10 mon.0@0(leader).pg v12963 post_paxos_update 2017-03-02 15:08:04.379623 7f4f763f4700 10 mon.0@0(leader).pg v12963 check_subs 2017-03-02 15:08:04.379628 7f4f763f4700 20 mon.0@0(leader).pg v12963 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:04.379640 7f4f763f4700 20 mon.0@0(leader).pg v12963 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:04.379646 7f4f763f4700 20 mon.0@0(leader).pg v12963 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:04.379654 7f4f763f4700 20 mon.0@0(leader).pg v12963 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:04.379661 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..732) post_refresh 2017-03-02 15:08:04.379665 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:04.379668 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13539) post_refresh 2017-03-02 15:08:04.379672 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:04.379675 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:04.379678 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:04.379682 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27317) commit_proposal 2017-03-02 15:08:04.379688 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12963) _active - not active 2017-03-02 15:08:04.379692 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27317) finish_round 2017-03-02 15:08:04.379697 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27317) finish_round waiting_for_acting 2017-03-02 15:08:04.379701 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12963) _active 2017-03-02 15:08:04.379705 7f4f763f4700 7 mon.0@0(leader).paxosservice(pgmap 12269..12963) _active creating new pending 2017-03-02 15:08:04.379714 7f4f763f4700 10 mon.0@0(leader).pg v12963 create_pending v 12964 2017-03-02 15:08:04.379724 7f4f763f4700 7 mon.0@0(leader).pg v12963 _updated_stats for osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:04.379733 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a16467a00 0x558a153fc200 pg_stats_ack(4 pgs tid 6793) v1 2017-03-02 15:08:04.379739 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6800/26531 -- pg_stats_ack(4 pgs tid 6793) v1 -- 0x558a153fc200 con 0 2017-03-02 15:08:04.379801 7f4f763f4700 7 mon.0@0(leader).pg v12963 _updated_stats for osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:04.379812 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a14c93000 0x558a1547ae00 pg_stats_ack(7 pgs tid 6793) v1 2017-03-02 15:08:04.379818 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6800/26536 -- pg_stats_ack(7 pgs tid 6793) v1 -- 0x558a1547ae00 con 0 2017-03-02 15:08:04.379870 7f4f763f4700 7 mon.0@0(leader).pg v12963 _updated_stats for osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:04.379878 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a163bdc00 0x558a154e8200 pg_stats_ack(2 pgs tid 6793) v1 2017-03-02 15:08:04.379884 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6800/26515 -- pg_stats_ack(2 pgs tid 6793) v1 -- 0x558a154e8200 con 0 2017-03-02 15:08:04.379941 7f4f763f4700 10 mon.0@0(leader).pg v12963 check_osd_map already seen 7 >= 7 2017-03-02 15:08:04.379947 7f4f763f4700 10 mon.0@0(leader).pg v12963 update_logger 2017-03-02 15:08:04.379977 7f4f763f4700 0 log_channel(cluster) log [INF] : pgmap v12963: 32 pgs: 32 active+clean; 88215 MB data, 165 GB used, 206 GB / 372 GB avail; 840 kB/s rd, 7660 kB/s wr, 276 op/s 2017-03-02 15:08:04.380005 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14100 at 2017-03-02 15:08:04.379980) v1 -- 0x558a15c00240 con 0 2017-03-02 15:08:04.380037 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27317) finish_round waiting_for_readable 2017-03-02 15:08:04.380043 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27317) finish_round waiting_for_writeable 2017-03-02 15:08:04.380047 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27317) finish_round done w/ waiters, state active 2017-03-02 15:08:04.380063 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27317) propose_pending 27318 10389 bytes 2017-03-02 15:08:04.380071 7f4f763f4700 10 mon.0@0(leader).paxos(paxos updating c 26607..27317) begin for 27318 10389 bytes 2017-03-02 15:08:04.380120 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14100 at 2017-03-02 15:08:04.379980) v1 ==== 0+0+0 (0 0 0) 0x558a15c00240 con 0x558a14482000 2017-03-02 15:08:04.403946 7f4f763f4700 10 mon.0@0(leader).paxos(paxos updating c 26607..27317) commit_start 27318 2017-03-02 15:08:04.404035 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:04.404071 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:04.404078 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13539) dispatch 0x558a15c00240 log(1 entries from seq 14100 at 2017-03-02 15:08:04.379980) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:04.404099 7f4f71beb700 5 mon.0@0(leader).paxos(paxos writing c 26607..27317) is_readable = 1 - now=2017-03-02 15:08:04.404100 lease_expire=0.000000 has v0 lc 27317 2017-03-02 15:08:04.404114 7f4f71beb700 10 mon.0@0(leader).log v13539 preprocess_query log(1 entries from seq 14100 at 2017-03-02 15:08:04.379980) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:04.404129 7f4f71beb700 10 mon.0@0(leader).log v13539 preprocess_log log(1 entries from seq 14100 at 2017-03-02 15:08:04.379980) v1 from mon.0 2017-03-02 15:08:04.404139 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:04.404145 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:04.404148 7f4f71beb700 20 allow all 2017-03-02 15:08:04.404156 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13539) waiting for paxos -> writeable 2017-03-02 15:08:04.429070 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27317) commit_finish 27318 2017-03-02 15:08:04.429129 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:04.429161 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12963) refresh 2017-03-02 15:08:04.429199 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..732) refresh 2017-03-02 15:08:04.429219 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:04.429243 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13540) refresh 2017-03-02 15:08:04.429248 7f4f763f4700 10 mon.0@0(leader).log v13540 update_from_paxos 2017-03-02 15:08:04.429251 7f4f763f4700 10 mon.0@0(leader).log v13540 update_from_paxos version 13540 summary v 13539 2017-03-02 15:08:04.429257 7f4f763f4700 10 mon.0@0(leader).log v13540 update_from_paxos latest full 13539 2017-03-02 15:08:04.429268 7f4f763f4700 7 mon.0@0(leader).log v13540 update_from_paxos applying incremental log 13540 2017-03-02 15:08:03.329548 mon.0 172.21.15.17:6789/0 14099 : cluster [INF] pgmap v12962: 32 pgs: 32 active+clean; 88231 MB data, 165 GB used, 206 GB / 372 GB avail; 1963 kB/s rd, 11622 kB/s wr, 334 op/s 2017-03-02 15:08:04.429288 7f4f763f4700 20 mon.0@0(leader).log v13540 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:04.429306 7f4f763f4700 15 mon.0@0(leader).log v13540 update_from_paxos logging for 1 channels 2017-03-02 15:08:04.429308 7f4f763f4700 15 mon.0@0(leader).log v13540 update_from_paxos channel 'cluster' logging 203 bytes 2017-03-02 15:08:04.429405 7f4f763f4700 10 mon.0@0(leader).log v13540 check_subs 2017-03-02 15:08:04.429461 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:04.429503 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:04.429514 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:04.429534 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:04.429537 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12963) post_refresh 2017-03-02 15:08:04.429539 7f4f763f4700 10 mon.0@0(leader).pg v12963 post_paxos_update 2017-03-02 15:08:04.429542 7f4f763f4700 10 mon.0@0(leader).pg v12963 check_subs 2017-03-02 15:08:04.429545 7f4f763f4700 20 mon.0@0(leader).pg v12963 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:04.429552 7f4f763f4700 20 mon.0@0(leader).pg v12963 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:04.429556 7f4f763f4700 20 mon.0@0(leader).pg v12963 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:04.429559 7f4f763f4700 20 mon.0@0(leader).pg v12963 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:04.429562 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..732) post_refresh 2017-03-02 15:08:04.429564 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:04.429566 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13540) post_refresh 2017-03-02 15:08:04.429568 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:04.429569 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:04.429571 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:04.429574 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27318) commit_proposal 2017-03-02 15:08:04.429578 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13540) _active - not active 2017-03-02 15:08:04.429581 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27318) finish_round 2017-03-02 15:08:04.429583 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27318) finish_round waiting_for_acting 2017-03-02 15:08:04.429586 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13540) _active 2017-03-02 15:08:04.429588 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13540) _active creating new pending 2017-03-02 15:08:04.429596 7f4f763f4700 10 mon.0@0(leader).log v13540 create_pending v 13541 2017-03-02 15:08:04.429599 7f4f763f4700 7 mon.0@0(leader).log v13540 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:04.429605 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a1543ad00 0x558a152ff0e0 log(last 14099) v1 2017-03-02 15:08:04.429609 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14099) v1 -- 0x558a152ff0e0 con 0 2017-03-02 15:08:04.429643 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13540) dispatch 0x558a15c00240 log(1 entries from seq 14100 at 2017-03-02 15:08:04.379980) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:04.429659 7f4f763f4700 5 mon.0@0(leader).paxos(paxos active c 26607..27318) is_readable = 1 - now=2017-03-02 15:08:04.429660 lease_expire=0.000000 has v0 lc 27318 2017-03-02 15:08:04.429668 7f4f763f4700 10 mon.0@0(leader).log v13540 preprocess_query log(1 entries from seq 14100 at 2017-03-02 15:08:04.379980) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:04.429676 7f4f763f4700 10 mon.0@0(leader).log v13540 preprocess_log log(1 entries from seq 14100 at 2017-03-02 15:08:04.379980) v1 from mon.0 2017-03-02 15:08:04.429681 7f4f763f4700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:04.429684 7f4f763f4700 20 allow so far , doing grant allow * 2017-03-02 15:08:04.429686 7f4f763f4700 20 allow all 2017-03-02 15:08:04.429690 7f4f763f4700 10 mon.0@0(leader).log v13540 prepare_update log(1 entries from seq 14100 at 2017-03-02 15:08:04.379980) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:04.429697 7f4f763f4700 10 mon.0@0(leader).log v13540 prepare_log log(1 entries from seq 14100 at 2017-03-02 15:08:04.379980) v1 from mon.0 2017-03-02 15:08:04.429701 7f4f763f4700 10 mon.0@0(leader).log v13540 logging 2017-03-02 15:08:04.379980 mon.0 172.21.15.17:6789/0 14100 : cluster [INF] pgmap v12963: 32 pgs: 32 active+clean; 88215 MB data, 165 GB used, 206 GB / 372 GB avail; 840 kB/s rd, 7660 kB/s wr, 276 op/s 2017-03-02 15:08:04.429684 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14099) v1 ==== 0+0+0 (0 0 0) 0x558a152ff0e0 con 0x558a14482000 2017-03-02 15:08:04.429711 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13540) setting proposal_timer 0x558a1412e830 with delay of 0.999388 2017-03-02 15:08:04.429733 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27318) finish_round waiting_for_readable 2017-03-02 15:08:04.429736 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27318) finish_round waiting_for_writeable 2017-03-02 15:08:04.429739 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27318) finish_round done w/ waiters, state active 2017-03-02 15:08:04.429770 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:04.429784 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:04.429788 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:04.429791 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:04.429792 7f4f71beb700 20 allow all 2017-03-02 15:08:05.429263 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13540) propose_pending 2017-03-02 15:08:05.429293 7f4f743f0700 10 mon.0@0(leader).log v13540 encode_full log v 13540 2017-03-02 15:08:05.429395 7f4f743f0700 10 mon.0@0(leader).log v13540 encode_pending v13541 2017-03-02 15:08:05.429406 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27318) queue_pending_finisher 0x558a1412c4a0 2017-03-02 15:08:05.429412 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27318) trigger_propose active, proposing now 2017-03-02 15:08:05.429422 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27318) propose_pending 27319 10394 bytes 2017-03-02 15:08:05.429426 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27318) begin for 27319 10394 bytes 2017-03-02 15:08:05.462693 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27318) commit_start 27319 2017-03-02 15:08:05.487815 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27318) commit_finish 27319 2017-03-02 15:08:05.487922 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:05.487958 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12963) refresh 2017-03-02 15:08:05.487979 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..732) refresh 2017-03-02 15:08:05.488004 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:05.488029 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13541) refresh 2017-03-02 15:08:05.488041 7f4f763f4700 10 mon.0@0(leader).log v13541 update_from_paxos 2017-03-02 15:08:05.488044 7f4f763f4700 10 mon.0@0(leader).log v13541 update_from_paxos version 13541 summary v 13540 2017-03-02 15:08:05.488050 7f4f763f4700 10 mon.0@0(leader).log v13541 update_from_paxos latest full 13540 2017-03-02 15:08:05.488064 7f4f763f4700 7 mon.0@0(leader).log v13541 update_from_paxos applying incremental log 13541 2017-03-02 15:08:04.379980 mon.0 172.21.15.17:6789/0 14100 : cluster [INF] pgmap v12963: 32 pgs: 32 active+clean; 88215 MB data, 165 GB used, 206 GB / 372 GB avail; 840 kB/s rd, 7660 kB/s wr, 276 op/s 2017-03-02 15:08:05.488087 7f4f763f4700 20 mon.0@0(leader).log v13541 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:05.488108 7f4f763f4700 15 mon.0@0(leader).log v13541 update_from_paxos logging for 1 channels 2017-03-02 15:08:05.488111 7f4f763f4700 15 mon.0@0(leader).log v13541 update_from_paxos channel 'cluster' logging 201 bytes 2017-03-02 15:08:05.488211 7f4f763f4700 10 mon.0@0(leader).log v13541 check_subs 2017-03-02 15:08:05.488275 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:05.488300 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:05.488311 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:05.488335 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:05.488338 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12963) post_refresh 2017-03-02 15:08:05.488341 7f4f763f4700 10 mon.0@0(leader).pg v12963 post_paxos_update 2017-03-02 15:08:05.488344 7f4f763f4700 10 mon.0@0(leader).pg v12963 check_subs 2017-03-02 15:08:05.488348 7f4f763f4700 20 mon.0@0(leader).pg v12963 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:05.488356 7f4f763f4700 20 mon.0@0(leader).pg v12963 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:05.488360 7f4f763f4700 20 mon.0@0(leader).pg v12963 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:05.488364 7f4f763f4700 20 mon.0@0(leader).pg v12963 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:05.488368 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..732) post_refresh 2017-03-02 15:08:05.488370 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:05.488372 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13541) post_refresh 2017-03-02 15:08:05.488374 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:05.488376 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:05.488378 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:05.488381 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27319) commit_proposal 2017-03-02 15:08:05.488386 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13541) _active - not active 2017-03-02 15:08:05.488389 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27319) finish_round 2017-03-02 15:08:05.488392 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27319) finish_round waiting_for_acting 2017-03-02 15:08:05.488395 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13541) _active 2017-03-02 15:08:05.488398 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13541) _active creating new pending 2017-03-02 15:08:05.488407 7f4f763f4700 10 mon.0@0(leader).log v13541 create_pending v 13542 2017-03-02 15:08:05.488412 7f4f763f4700 7 mon.0@0(leader).log v13541 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:05.488419 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a14dbf500 0x558a1592e000 log(last 14100) v1 2017-03-02 15:08:05.488425 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14100) v1 -- 0x558a1592e000 con 0 2017-03-02 15:08:05.488464 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27319) finish_round waiting_for_readable 2017-03-02 15:08:05.488471 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27319) finish_round waiting_for_writeable 2017-03-02 15:08:05.488475 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27319) finish_round done w/ waiters, state active 2017-03-02 15:08:05.488553 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14100) v1 ==== 0+0+0 (0 0 0) 0x558a1592e000 con 0x558a14482000 2017-03-02 15:08:05.488583 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:05.488591 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:05.488595 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:05.488599 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:05.488601 7f4f71beb700 20 allow all 2017-03-02 15:08:06.010797 7f4f743f0700 11 mon.0@0(leader) e2 tick 2017-03-02 15:08:06.010838 7f4f743f0700 10 mon.0@0(leader).pg v12963 v12963: 32 pgs: 32 active+clean; 88215 MB data, 165 GB used, 206 GB / 372 GB avail; 840 kB/s rd, 7660 kB/s wr, 276 op/s 2017-03-02 15:08:06.010900 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12963) maybe_trim trim_to 12463 would only trim 194 < paxos_service_trim_min 250 2017-03-02 15:08:06.010908 7f4f743f0700 10 mon.0@0(leader).mds e732 e732: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:06.010962 7f4f743f0700 10 mon.0@0(leader).paxosservice(mdsmap 1..732) maybe_trim trim_to 232 would only trim 231 < paxos_service_trim_min 250 2017-03-02 15:08:06.010968 7f4f743f0700 10 mon.0@0(leader).osd e7 e7: 4 osds: 4 up, 4 in 2017-03-02 15:08:06.010974 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27319) is_readable = 1 - now=2017-03-02 15:08:06.010975 lease_expire=0.000000 has v0 lc 27319 2017-03-02 15:08:06.010985 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27319) is_readable = 1 - now=2017-03-02 15:08:06.010986 lease_expire=0.000000 has v0 lc 27319 2017-03-02 15:08:06.010996 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27319) is_readable = 1 - now=2017-03-02 15:08:06.010997 lease_expire=0.000000 has v0 lc 27319 2017-03-02 15:08:06.011002 7f4f743f0700 10 mon.0@0(leader).osd e7 min_last_epoch_clean 6 2017-03-02 15:08:06.011006 7f4f743f0700 10 mon.0@0(leader).log v13541 log 2017-03-02 15:08:06.011009 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13541) maybe_trim trim_to 13041 would only trim 14 < paxos_service_trim_min 250 2017-03-02 15:08:06.011014 7f4f743f0700 10 mon.0@0(leader).auth v10 auth 2017-03-02 15:08:06.011019 7f4f743f0700 10 cephx keyserver: _check_rotating_secrets 2017-03-02 15:08:06.011030 7f4f743f0700 20 mon.0@0(leader) e2 sync_trim_providers 2017-03-02 15:08:06.869952 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.5 172.21.15.11:6804/1637448123 8780 ==== mdsbeacon(4116/e up:active seq 8493 v732) v7 ==== 365+0+0 (3282945416 0 0) 0x558a154b9d40 con 0x558a1471d800 2017-03-02 15:08:06.870013 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b0f20 for mds.? 172.21.15.11:6804/1637448123 2017-03-02 15:08:06.870026 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:06.870032 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..732) dispatch 0x558a154b9d40 mdsbeacon(4116/e up:active seq 8493 v732) v7 from mds.5 172.21.15.11:6804/1637448123 con 0x558a1471d800 2017-03-02 15:08:06.870045 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27319) is_readable = 1 - now=2017-03-02 15:08:06.870046 lease_expire=0.000000 has v0 lc 27319 2017-03-02 15:08:06.870061 7f4f71beb700 10 mon.0@0(leader).mds e732 preprocess_query mdsbeacon(4116/e up:active seq 8493 v732) v7 from mds.5 172.21.15.11:6804/1637448123 2017-03-02 15:08:06.870075 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:06.870080 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:06.870082 7f4f71beb700 20 allow all 2017-03-02 15:08:06.870085 7f4f71beb700 12 mon.0@0(leader).mds e732 preprocess_beacon mdsbeacon(4116/e up:active seq 8493 v732) v7 from mds.5 172.21.15.11:6804/1637448123 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:06.870101 7f4f71beb700 10 mon.0@0(leader).mds e732 preprocess_beacon: GID exists in map: 4116 2017-03-02 15:08:06.870110 7f4f71beb700 15 mon.0@0(leader).mds e732 _note_beacon mdsbeacon(4116/e up:active seq 8493 v732) v7 noting time 2017-03-02 15:08:06.870117 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15f71300 0x558a160c0d80 mdsbeacon(4116/e up:active seq 8493 v732) v7 2017-03-02 15:08:06.870124 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6804/1637448123 -- mdsbeacon(4116/e up:active seq 8493 v732) v7 -- 0x558a160c0d80 con 0 2017-03-02 15:08:06.883771 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.3 172.21.15.11:6805/1926402648 8556 ==== mdsbeacon(4118/f up:active seq 8493 v732) v7 ==== 406+0+0 (608995039 0 0) 0x558a148d8080 con 0x558a14905800 2017-03-02 15:08:06.883815 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b2260 for mds.? 172.21.15.11:6805/1926402648 2017-03-02 15:08:06.883866 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:06.883871 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..732) dispatch 0x558a148d8080 mdsbeacon(4118/f up:active seq 8493 v732) v7 from mds.3 172.21.15.11:6805/1926402648 con 0x558a14905800 2017-03-02 15:08:06.883881 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27319) is_readable = 1 - now=2017-03-02 15:08:06.883881 lease_expire=0.000000 has v0 lc 27319 2017-03-02 15:08:06.883893 7f4f71beb700 10 mon.0@0(leader).mds e732 preprocess_query mdsbeacon(4118/f up:active seq 8493 v732) v7 from mds.3 172.21.15.11:6805/1926402648 2017-03-02 15:08:06.883902 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:06.883906 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:06.883907 7f4f71beb700 20 allow all 2017-03-02 15:08:06.883909 7f4f71beb700 12 mon.0@0(leader).mds e732 preprocess_beacon mdsbeacon(4118/f up:active seq 8493 v732) v7 from mds.3 172.21.15.11:6805/1926402648 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:06.883919 7f4f71beb700 10 mon.0@0(leader).mds e732 preprocess_beacon: GID exists in map: 4118 2017-03-02 15:08:06.883925 7f4f71beb700 15 mon.0@0(leader).mds e732 _note_beacon mdsbeacon(4118/f up:active seq 8493 v732) v7 noting time 2017-03-02 15:08:06.883931 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15f71300 0x558a154b9d40 mdsbeacon(4118/f up:active seq 8493 v732) v7 2017-03-02 15:08:06.883935 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6805/1926402648 -- mdsbeacon(4118/f up:active seq 8493 v732) v7 -- 0x558a154b9d40 con 0 2017-03-02 15:08:06.884475 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.2 172.21.15.22:6804/673088617 8555 ==== mdsbeacon(4119/c up:active seq 8493 v732) v7 ==== 365+0+0 (3820274134 0 0) 0x558a160c0d80 con 0x558a14907000 2017-03-02 15:08:06.884512 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14916f20 for mds.? 172.21.15.22:6804/673088617 2017-03-02 15:08:06.884519 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:06.884522 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..732) dispatch 0x558a160c0d80 mdsbeacon(4119/c up:active seq 8493 v732) v7 from mds.2 172.21.15.22:6804/673088617 con 0x558a14907000 2017-03-02 15:08:06.884529 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27319) is_readable = 1 - now=2017-03-02 15:08:06.884530 lease_expire=0.000000 has v0 lc 27319 2017-03-02 15:08:06.884539 7f4f71beb700 10 mon.0@0(leader).mds e732 preprocess_query mdsbeacon(4119/c up:active seq 8493 v732) v7 from mds.2 172.21.15.22:6804/673088617 2017-03-02 15:08:06.884546 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:06.884549 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:06.884550 7f4f71beb700 20 allow all 2017-03-02 15:08:06.884552 7f4f71beb700 12 mon.0@0(leader).mds e732 preprocess_beacon mdsbeacon(4119/c up:active seq 8493 v732) v7 from mds.2 172.21.15.22:6804/673088617 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:06.884561 7f4f71beb700 10 mon.0@0(leader).mds e732 preprocess_beacon: GID exists in map: 4119 2017-03-02 15:08:06.884566 7f4f71beb700 15 mon.0@0(leader).mds e732 _note_beacon mdsbeacon(4119/c up:active seq 8493 v732) v7 noting time 2017-03-02 15:08:06.884571 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15f71300 0x558a148d8080 mdsbeacon(4119/c up:active seq 8493 v732) v7 2017-03-02 15:08:06.884574 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6804/673088617 -- mdsbeacon(4119/c up:active seq 8493 v732) v7 -- 0x558a148d8080 con 0 2017-03-02 15:08:06.884647 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.0 172.21.15.22:6805/2203629154 8605 ==== mdsbeacon(4121/d up:active seq 8489 v732) v7 ==== 452+0+0 (313242073 0 0) 0x558a154b9d40 con 0x558a1492a000 2017-03-02 15:08:06.884662 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a149195a0 for mds.? 172.21.15.22:6805/2203629154 2017-03-02 15:08:06.884666 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:06.884669 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..732) dispatch 0x558a154b9d40 mdsbeacon(4121/d up:active seq 8489 v732) v7 from mds.0 172.21.15.22:6805/2203629154 con 0x558a1492a000 2017-03-02 15:08:06.884678 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27319) is_readable = 1 - now=2017-03-02 15:08:06.884679 lease_expire=0.000000 has v0 lc 27319 2017-03-02 15:08:06.884690 7f4f71beb700 10 mon.0@0(leader).mds e732 preprocess_query mdsbeacon(4121/d up:active seq 8489 v732) v7 from mds.0 172.21.15.22:6805/2203629154 2017-03-02 15:08:06.884697 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:06.884699 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:06.884701 7f4f71beb700 20 allow all 2017-03-02 15:08:06.884702 7f4f71beb700 12 mon.0@0(leader).mds e732 preprocess_beacon mdsbeacon(4121/d up:active seq 8489 v732) v7 from mds.0 172.21.15.22:6805/2203629154 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:06.884709 7f4f71beb700 10 mon.0@0(leader).mds e732 preprocess_beacon: GID exists in map: 4121 2017-03-02 15:08:06.884714 7f4f71beb700 20 mon.0@0(leader).mds e732 preprocess_beacon health metrics for gid 4121 were updated 2017-03-02 15:08:06.884716 7f4f71beb700 15 mon.0@0(leader).mds e732 _note_beacon mdsbeacon(4121/d up:active seq 8489 v732) v7 noting time 2017-03-02 15:08:06.884721 7f4f71beb700 7 mon.0@0(leader).mds e732 prepare_update mdsbeacon(4121/d up:active seq 8489 v732) v7 2017-03-02 15:08:06.884724 7f4f71beb700 12 mon.0@0(leader).mds e732 prepare_beacon mdsbeacon(4121/d up:active seq 8489 v732) v7 from mds.0 172.21.15.22:6805/2203629154 2017-03-02 15:08:06.884729 7f4f71beb700 20 mon.0@0(leader).mds e732 prepare_beacon got health from gid 4121 with 1 metrics. 2017-03-02 15:08:06.884734 7f4f71beb700 10 mon.0@0(leader).mds e732 prepare_beacon mds.0 up:active -> up:active standby_for_rank=-1 2017-03-02 15:08:06.884738 7f4f71beb700 7 mon.0@0(leader).mds e732 prepare_beacon pending map now: 2017-03-02 15:08:06.884740 7f4f71beb700 7 mon.0@0(leader).mds e732 print_map 2017-03-02 15:08:06.884791 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..732) setting proposal_timer 0x558a1412fcb0 with delay of 0.05 2017-03-02 15:08:06.923917 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.1 172.21.15.17:6804/2576824570 8552 ==== mdsbeacon(4120/g up:active seq 8493 v732) v7 ==== 365+0+0 (733711565 0 0) 0x558a154b8d00 con 0x558a14908800 2017-03-02 15:08:06.923978 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14917600 for mds.? 172.21.15.17:6804/2576824570 2017-03-02 15:08:06.923990 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:06.923996 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..732) dispatch 0x558a154b8d00 mdsbeacon(4120/g up:active seq 8493 v732) v7 from mds.1 172.21.15.17:6804/2576824570 con 0x558a14908800 2017-03-02 15:08:06.924008 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27319) is_readable = 1 - now=2017-03-02 15:08:06.924009 lease_expire=0.000000 has v0 lc 27319 2017-03-02 15:08:06.924024 7f4f71beb700 10 mon.0@0(leader).mds e732 preprocess_query mdsbeacon(4120/g up:active seq 8493 v732) v7 from mds.1 172.21.15.17:6804/2576824570 2017-03-02 15:08:06.924038 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:06.924043 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:06.924045 7f4f71beb700 20 allow all 2017-03-02 15:08:06.924048 7f4f71beb700 12 mon.0@0(leader).mds e732 preprocess_beacon mdsbeacon(4120/g up:active seq 8493 v732) v7 from mds.1 172.21.15.17:6804/2576824570 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:06.924063 7f4f71beb700 10 mon.0@0(leader).mds e732 preprocess_beacon: GID exists in map: 4120 2017-03-02 15:08:06.924070 7f4f71beb700 10 mon.0@0(leader).mds e732 mds_beacon mdsbeacon(4120/g up:active seq 8493 v732) v7 ignoring requested state, because mds hasn't seen latest map 2017-03-02 15:08:06.924075 7f4f71beb700 15 mon.0@0(leader).mds e732 _note_beacon mdsbeacon(4120/g up:active seq 8493 v732) v7 noting time 2017-03-02 15:08:06.924083 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15d92e00 0x558a160c0d80 mdsbeacon(4120/g up:active seq 8493 v733) v7 2017-03-02 15:08:06.924090 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6804/2576824570 -- mdsbeacon(4120/g up:active seq 8493 v733) v7 -- 0x558a160c0d80 con 0 2017-03-02 15:08:06.934923 7f4f743f0700 10 mon.0@0(leader).paxosservice(mdsmap 1..732) propose_pending 2017-03-02 15:08:06.934989 7f4f743f0700 10 mon.0@0(leader).mds e732 encode_pending e733 2017-03-02 15:08:06.935113 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27319) queue_pending_finisher 0x558a1412ed50 2017-03-02 15:08:06.935130 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27319) trigger_propose active, proposing now 2017-03-02 15:08:06.935150 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27319) propose_pending 27320 2409 bytes 2017-03-02 15:08:06.935156 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27319) begin for 27320 2409 bytes 2017-03-02 15:08:06.945525 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27319) commit_start 27320 2017-03-02 15:08:06.968864 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.4 172.21.15.26:6805/2033388711 8550 ==== mdsbeacon(4117/b up:active seq 8493 v732) v7 ==== 365+0+0 (3574764046 0 0) 0x558a160c0d80 con 0x558a14904000 2017-03-02 15:08:06.968911 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b1600 for mds.? 172.21.15.26:6805/2033388711 2017-03-02 15:08:06.968923 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:06.968928 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..732) dispatch 0x558a160c0d80 mdsbeacon(4117/b up:active seq 8493 v732) v7 from mds.4 172.21.15.26:6805/2033388711 con 0x558a14904000 2017-03-02 15:08:06.968939 7f4f71beb700 5 mon.0@0(leader).paxos(paxos writing c 26607..27319) is_readable = 1 - now=2017-03-02 15:08:06.968940 lease_expire=0.000000 has v0 lc 27319 2017-03-02 15:08:06.968953 7f4f71beb700 10 mon.0@0(leader).mds e732 preprocess_query mdsbeacon(4117/b up:active seq 8493 v732) v7 from mds.4 172.21.15.26:6805/2033388711 2017-03-02 15:08:06.968965 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:06.968970 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:06.968972 7f4f71beb700 20 allow all 2017-03-02 15:08:06.968974 7f4f71beb700 12 mon.0@0(leader).mds e732 preprocess_beacon mdsbeacon(4117/b up:active seq 8493 v732) v7 from mds.4 172.21.15.26:6805/2033388711 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:06.968988 7f4f71beb700 10 mon.0@0(leader).mds e732 preprocess_beacon: GID exists in map: 4117 2017-03-02 15:08:06.968994 7f4f71beb700 10 mon.0@0(leader).mds e732 mds_beacon mdsbeacon(4117/b up:active seq 8493 v732) v7 ignoring requested state, because mds hasn't seen latest map 2017-03-02 15:08:06.969000 7f4f71beb700 15 mon.0@0(leader).mds e732 _note_beacon mdsbeacon(4117/b up:active seq 8493 v732) v7 noting time 2017-03-02 15:08:06.969007 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15d92e00 0x558a154b8d00 mdsbeacon(4117/b up:active seq 8493 v733) v7 2017-03-02 15:08:06.969013 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6805/2033388711 -- mdsbeacon(4117/b up:active seq 8493 v733) v7 -- 0x558a154b8d00 con 0 2017-03-02 15:08:06.971055 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27319) commit_finish 27320 2017-03-02 15:08:06.971134 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:06.971163 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12963) refresh 2017-03-02 15:08:06.971187 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) refresh 2017-03-02 15:08:06.971189 7f4f763f4700 10 mon.0@0(leader).mds e732 update_from_paxos version 733, my e 732 2017-03-02 15:08:06.971200 7f4f763f4700 10 mon.0@0(leader).mds e732 update_from_paxos got 733 2017-03-02 15:08:06.971252 7f4f763f4700 4 mon.0@0(leader).mds e733 new map 2017-03-02 15:08:06.971261 7f4f763f4700 0 mon.0@0(leader).mds e733 print_map 2017-03-02 15:08:06.971326 7f4f763f4700 20 mon.0@0(leader).mds e733 check_sub: mdsmap 2017-03-02 15:08:06.971333 7f4f763f4700 10 mon.0@0(leader).mds e733 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:06.971337 7f4f763f4700 10 mon.0@0(leader).mds e733 check_sub selected MDS map epoch 733 for namespace 1 for subscriber mds.? who wants epoch 733 2017-03-02 15:08:06.971359 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6804/2954782670 -- mdsmap(e 733) v1 -- 0x558a15c00240 con 0 2017-03-02 15:08:06.971389 7f4f763f4700 20 mon.0@0(leader).mds e733 check_sub: mdsmap 2017-03-02 15:08:06.971395 7f4f763f4700 10 mon.0@0(leader).mds e733 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:06.971398 7f4f763f4700 10 mon.0@0(leader).mds e733 check_sub selected MDS map epoch 733 for namespace 1 for subscriber mds.? who wants epoch 733 2017-03-02 15:08:06.971414 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6804/1637448123 -- mdsmap(e 733) v1 -- 0x558a15a10f40 con 0 2017-03-02 15:08:06.971436 7f4f763f4700 20 mon.0@0(leader).mds e733 check_sub: mdsmap 2017-03-02 15:08:06.971441 7f4f763f4700 10 mon.0@0(leader).mds e733 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:06.971443 7f4f763f4700 10 mon.0@0(leader).mds e733 check_sub selected MDS map epoch 733 for namespace 1 for subscriber mds.? who wants epoch 733 2017-03-02 15:08:06.971458 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6805/2033388711 -- mdsmap(e 733) v1 -- 0x558a16920f40 con 0 2017-03-02 15:08:06.971476 7f4f763f4700 20 mon.0@0(leader).mds e733 check_sub: mdsmap 2017-03-02 15:08:06.971493 7f4f763f4700 10 mon.0@0(leader).mds e733 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:06.971497 7f4f763f4700 10 mon.0@0(leader).mds e733 check_sub selected MDS map epoch 733 for namespace 1 for subscriber mds.? who wants epoch 733 2017-03-02 15:08:06.971523 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6805/1926402648 -- mdsmap(e 733) v1 -- 0x558a16920d00 con 0 2017-03-02 15:08:06.971554 7f4f763f4700 20 mon.0@0(leader).mds e733 check_sub: mdsmap 2017-03-02 15:08:06.971565 7f4f763f4700 10 mon.0@0(leader).mds e733 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:06.971569 7f4f763f4700 10 mon.0@0(leader).mds e733 check_sub selected MDS map epoch 733 for namespace 1 for subscriber mds.? who wants epoch 733 2017-03-02 15:08:06.971596 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6804/673088617 -- mdsmap(e 733) v1 -- 0x558a16921600 con 0 2017-03-02 15:08:06.971630 7f4f763f4700 20 mon.0@0(leader).mds e733 check_sub: mdsmap 2017-03-02 15:08:06.971638 7f4f763f4700 10 mon.0@0(leader).mds e733 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:06.971641 7f4f763f4700 10 mon.0@0(leader).mds e733 check_sub selected MDS map epoch 733 for namespace 1 for subscriber mds.? who wants epoch 733 2017-03-02 15:08:06.971656 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6804/2576824570 -- mdsmap(e 733) v1 -- 0x558a16921180 con 0 2017-03-02 15:08:06.971675 7f4f763f4700 20 mon.0@0(leader).mds e733 check_sub: mdsmap 2017-03-02 15:08:06.971683 7f4f763f4700 10 mon.0@0(leader).mds e733 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:06.971686 7f4f763f4700 10 mon.0@0(leader).mds e733 check_sub selected MDS map epoch 733 for namespace 1 for subscriber mds.? who wants epoch 733 2017-03-02 15:08:06.971712 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6805/2203629154 -- mdsmap(e 733) v1 -- 0x558a1691e000 con 0 2017-03-02 15:08:06.971741 7f4f763f4700 20 mon.0@0(leader).mds e733 check_sub: mdsmap 2017-03-02 15:08:06.971746 7f4f763f4700 10 mon.0@0(leader).mds e733 check_sub: is_mds=0, fscid= 1 2017-03-02 15:08:06.971750 7f4f763f4700 10 mon.0@0(leader).mds e733 check_sub selected MDS map epoch 733 for namespace 1 for subscriber unknown.0 who wants epoch 733 2017-03-02 15:08:06.971778 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.132:0/2597832251 -- mdsmap(e 733) v1 -- 0x558a169201c0 con 0 2017-03-02 15:08:06.971808 7f4f763f4700 20 mon.0@0(leader).mds e733 check_sub: mdsmap 2017-03-02 15:08:06.971813 7f4f763f4700 10 mon.0@0(leader).mds e733 check_sub: is_mds=0, fscid= 1 2017-03-02 15:08:06.971817 7f4f763f4700 10 mon.0@0(leader).mds e733 check_sub selected MDS map epoch 733 for namespace 1 for subscriber unknown.0 who wants epoch 733 2017-03-02 15:08:06.971872 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.133:0/3792964262 -- mdsmap(e 733) v1 -- 0x558a1691fb00 con 0 2017-03-02 15:08:06.971909 7f4f763f4700 10 mon.0@0(leader).mds e733 update_logger 2017-03-02 15:08:06.971963 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:06.971993 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13541) refresh 2017-03-02 15:08:06.971997 7f4f763f4700 10 mon.0@0(leader).log v13541 update_from_paxos 2017-03-02 15:08:06.972000 7f4f763f4700 10 mon.0@0(leader).log v13541 update_from_paxos version 13541 summary v 13541 2017-03-02 15:08:06.972024 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:06.972060 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:06.972063 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:06.972083 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:06.972086 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12963) post_refresh 2017-03-02 15:08:06.972089 7f4f763f4700 10 mon.0@0(leader).pg v12963 post_paxos_update 2017-03-02 15:08:06.972092 7f4f763f4700 10 mon.0@0(leader).pg v12963 check_subs 2017-03-02 15:08:06.972096 7f4f763f4700 20 mon.0@0(leader).pg v12963 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:06.972114 7f4f763f4700 20 mon.0@0(leader).pg v12963 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:06.972119 7f4f763f4700 20 mon.0@0(leader).pg v12963 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:06.972124 7f4f763f4700 20 mon.0@0(leader).pg v12963 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:06.972127 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) post_refresh 2017-03-02 15:08:06.972130 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:06.972132 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13541) post_refresh 2017-03-02 15:08:06.972134 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:06.972136 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:06.972138 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:06.972141 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27320) commit_proposal 2017-03-02 15:08:06.972146 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) _active - not active 2017-03-02 15:08:06.972149 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27320) finish_round 2017-03-02 15:08:06.972151 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27320) finish_round waiting_for_acting 2017-03-02 15:08:06.972154 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) _active 2017-03-02 15:08:06.972157 7f4f763f4700 7 mon.0@0(leader).paxosservice(mdsmap 1..733) _active creating new pending 2017-03-02 15:08:06.972180 7f4f763f4700 10 mon.0@0(leader).mds e733 create_pending e734 2017-03-02 15:08:06.972187 7f4f763f4700 10 mon.0@0(leader).mds e733 _updated mds.0 mdsbeacon(4121/d up:active seq 8489 v732) v7 2017-03-02 15:08:06.972201 7f4f763f4700 0 log_channel(cluster) log [INF] : mds.0 172.21.15.22:6805/2203629154 up:active 2017-03-02 15:08:06.972221 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14101 at 2017-03-02 15:08:06.972203) v1 -- 0x558a1456f3c0 con 0 2017-03-02 15:08:06.972250 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a15f71300 0x558a15204000 mdsbeacon(4121/d up:active seq 8489 v733) v7 2017-03-02 15:08:06.972255 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6805/2203629154 -- mdsbeacon(4121/d up:active seq 8489 v733) v7 -- 0x558a15204000 con 0 2017-03-02 15:08:06.972287 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14101 at 2017-03-02 15:08:06.972203) v1 ==== 0+0+0 (0 0 0) 0x558a1456f3c0 con 0x558a14482000 2017-03-02 15:08:06.972290 7f4f763f4700 10 mon.0@0(leader).mds e733 e733: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:06.972319 7f4f763f4700 10 mon.0@0(leader).mds e733 update_logger 2017-03-02 15:08:06.972336 7f4f763f4700 0 log_channel(cluster) log [INF] : fsmap e733: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:06.972360 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14102 at 2017-03-02 15:08:06.972338) v1 -- 0x558a1691eb40 con 0 2017-03-02 15:08:06.972381 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27320) finish_round waiting_for_readable 2017-03-02 15:08:06.972384 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27320) finish_round waiting_for_writeable 2017-03-02 15:08:06.972387 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27320) finish_round done w/ waiters, state active 2017-03-02 15:08:06.972413 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:06.972424 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:06.972427 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13541) dispatch 0x558a1456f3c0 log(1 entries from seq 14101 at 2017-03-02 15:08:06.972203) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:06.972438 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27320) is_readable = 1 - now=2017-03-02 15:08:06.972439 lease_expire=0.000000 has v0 lc 27320 2017-03-02 15:08:06.972447 7f4f71beb700 10 mon.0@0(leader).log v13541 preprocess_query log(1 entries from seq 14101 at 2017-03-02 15:08:06.972203) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:06.972455 7f4f71beb700 10 mon.0@0(leader).log v13541 preprocess_log log(1 entries from seq 14101 at 2017-03-02 15:08:06.972203) v1 from mon.0 2017-03-02 15:08:06.972460 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:06.972463 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:06.972464 7f4f71beb700 20 allow all 2017-03-02 15:08:06.972473 7f4f71beb700 10 mon.0@0(leader).log v13541 prepare_update log(1 entries from seq 14101 at 2017-03-02 15:08:06.972203) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:06.972480 7f4f71beb700 10 mon.0@0(leader).log v13541 prepare_log log(1 entries from seq 14101 at 2017-03-02 15:08:06.972203) v1 from mon.0 2017-03-02 15:08:06.972484 7f4f71beb700 10 mon.0@0(leader).log v13541 logging 2017-03-02 15:08:06.972203 mon.0 172.21.15.17:6789/0 14101 : cluster [INF] mds.0 172.21.15.22:6805/2203629154 up:active 2017-03-02 15:08:06.972493 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13541) setting proposal_timer 0x558a1412e3c0 with delay of 0.998607 2017-03-02 15:08:06.972519 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14102 at 2017-03-02 15:08:06.972338) v1 ==== 0+0+0 (0 0 0) 0x558a1691eb40 con 0x558a14482000 2017-03-02 15:08:06.972530 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:06.972534 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:06.972536 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13541) dispatch 0x558a1691eb40 log(1 entries from seq 14102 at 2017-03-02 15:08:06.972338) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:06.972542 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27320) is_readable = 1 - now=2017-03-02 15:08:06.972543 lease_expire=0.000000 has v0 lc 27320 2017-03-02 15:08:06.972548 7f4f71beb700 10 mon.0@0(leader).log v13541 preprocess_query log(1 entries from seq 14102 at 2017-03-02 15:08:06.972338) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:06.972554 7f4f71beb700 10 mon.0@0(leader).log v13541 preprocess_log log(1 entries from seq 14102 at 2017-03-02 15:08:06.972338) v1 from mon.0 2017-03-02 15:08:06.972560 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:06.972561 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:06.972562 7f4f71beb700 20 allow all 2017-03-02 15:08:06.972566 7f4f71beb700 10 mon.0@0(leader).log v13541 prepare_update log(1 entries from seq 14102 at 2017-03-02 15:08:06.972338) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:06.972572 7f4f71beb700 10 mon.0@0(leader).log v13541 prepare_log log(1 entries from seq 14102 at 2017-03-02 15:08:06.972338) v1 from mon.0 2017-03-02 15:08:06.972575 7f4f71beb700 10 mon.0@0(leader).log v13541 logging 2017-03-02 15:08:06.972338 mon.0 172.21.15.17:6789/0 14102 : cluster [INF] fsmap e733: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:06.972583 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13541) proposal_timer already set 2017-03-02 15:08:06.977962 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.6 172.21.15.26:6804/2954782670 8742 ==== mdsbeacon(4115/a up:active seq 8493 v733) v7 ==== 365+0+0 (3561447496 0 0) 0x558a15204000 con 0x558a1471f000 2017-03-02 15:08:06.978014 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14868d60 for mds.? 172.21.15.26:6804/2954782670 2017-03-02 15:08:06.978021 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:06.978024 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) dispatch 0x558a15204000 mdsbeacon(4115/a up:active seq 8493 v733) v7 from mds.6 172.21.15.26:6804/2954782670 con 0x558a1471f000 2017-03-02 15:08:06.978030 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27320) is_readable = 1 - now=2017-03-02 15:08:06.978031 lease_expire=0.000000 has v0 lc 27320 2017-03-02 15:08:06.978040 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_query mdsbeacon(4115/a up:active seq 8493 v733) v7 from mds.6 172.21.15.26:6804/2954782670 2017-03-02 15:08:06.978047 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:06.978050 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:06.978051 7f4f71beb700 20 allow all 2017-03-02 15:08:06.978053 7f4f71beb700 12 mon.0@0(leader).mds e733 preprocess_beacon mdsbeacon(4115/a up:active seq 8493 v733) v7 from mds.6 172.21.15.26:6804/2954782670 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:06.978061 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_beacon: GID exists in map: 4115 2017-03-02 15:08:06.978066 7f4f71beb700 15 mon.0@0(leader).mds e733 _note_beacon mdsbeacon(4115/a up:active seq 8493 v733) v7 noting time 2017-03-02 15:08:06.978070 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15a05300 0x558a160c0d80 mdsbeacon(4115/a up:active seq 8493 v733) v7 2017-03-02 15:08:06.978073 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6804/2954782670 -- mdsbeacon(4115/a up:active seq 8493 v733) v7 -- 0x558a160c0d80 con 0 2017-03-02 15:08:07.971234 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13541) propose_pending 2017-03-02 15:08:07.971260 7f4f743f0700 10 mon.0@0(leader).log v13541 encode_full log v 13541 2017-03-02 15:08:07.971336 7f4f743f0700 10 mon.0@0(leader).log v13541 encode_pending v13542 2017-03-02 15:08:07.971345 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27320) queue_pending_finisher 0x558a1412fcb0 2017-03-02 15:08:07.971350 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27320) trigger_propose active, proposing now 2017-03-02 15:08:07.971357 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27320) propose_pending 27321 10513 bytes 2017-03-02 15:08:07.971359 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27320) begin for 27321 10513 bytes 2017-03-02 15:08:07.996090 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27320) commit_start 27321 2017-03-02 15:08:08.021544 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27320) commit_finish 27321 2017-03-02 15:08:08.021598 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:08.021622 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12963) refresh 2017-03-02 15:08:08.021633 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) refresh 2017-03-02 15:08:08.021647 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:08.021661 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13542) refresh 2017-03-02 15:08:08.021662 7f4f763f4700 10 mon.0@0(leader).log v13542 update_from_paxos 2017-03-02 15:08:08.021665 7f4f763f4700 10 mon.0@0(leader).log v13542 update_from_paxos version 13542 summary v 13541 2017-03-02 15:08:08.021669 7f4f763f4700 10 mon.0@0(leader).log v13542 update_from_paxos latest full 13541 2017-03-02 15:08:08.021678 7f4f763f4700 7 mon.0@0(leader).log v13542 update_from_paxos applying incremental log 13542 2017-03-02 15:08:06.972203 mon.0 172.21.15.17:6789/0 14101 : cluster [INF] mds.0 172.21.15.22:6805/2203629154 up:active 2017-03-02 15:08:08.021693 7f4f763f4700 20 mon.0@0(leader).log v13542 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:08.021711 7f4f763f4700 7 mon.0@0(leader).log v13542 update_from_paxos applying incremental log 13542 2017-03-02 15:08:06.972338 mon.0 172.21.15.17:6789/0 14102 : cluster [INF] fsmap e733: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:08.021716 7f4f763f4700 20 mon.0@0(leader).log v13542 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:08.021723 7f4f763f4700 15 mon.0@0(leader).log v13542 update_from_paxos logging for 1 channels 2017-03-02 15:08:08.021725 7f4f763f4700 15 mon.0@0(leader).log v13542 update_from_paxos channel 'cluster' logging 316 bytes 2017-03-02 15:08:08.021798 7f4f763f4700 10 mon.0@0(leader).log v13542 check_subs 2017-03-02 15:08:08.021875 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:08.021892 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:08.021894 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:08.021911 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:08.021914 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12963) post_refresh 2017-03-02 15:08:08.021916 7f4f763f4700 10 mon.0@0(leader).pg v12963 post_paxos_update 2017-03-02 15:08:08.021919 7f4f763f4700 10 mon.0@0(leader).pg v12963 check_subs 2017-03-02 15:08:08.021922 7f4f763f4700 20 mon.0@0(leader).pg v12963 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:08.021930 7f4f763f4700 20 mon.0@0(leader).pg v12963 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:08.021935 7f4f763f4700 20 mon.0@0(leader).pg v12963 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:08.021938 7f4f763f4700 20 mon.0@0(leader).pg v12963 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:08.021942 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) post_refresh 2017-03-02 15:08:08.021945 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:08.021947 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13542) post_refresh 2017-03-02 15:08:08.021949 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:08.021951 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:08.021952 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:08.021956 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27321) commit_proposal 2017-03-02 15:08:08.021960 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13542) _active - not active 2017-03-02 15:08:08.021963 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27321) finish_round 2017-03-02 15:08:08.021966 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27321) finish_round waiting_for_acting 2017-03-02 15:08:08.021969 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13542) _active 2017-03-02 15:08:08.021971 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13542) _active creating new pending 2017-03-02 15:08:08.021988 7f4f763f4700 10 mon.0@0(leader).log v13542 create_pending v 13543 2017-03-02 15:08:08.021994 7f4f763f4700 7 mon.0@0(leader).log v13542 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:08.022001 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a15d92e00 0x558a14c1dde0 log(last 14101) v1 2017-03-02 15:08:08.022007 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14101) v1 -- 0x558a14c1dde0 con 0 2017-03-02 15:08:08.022037 7f4f763f4700 7 mon.0@0(leader).log v13542 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:08.022043 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a15dde900 0x558a1560f0e0 log(last 14102) v1 2017-03-02 15:08:08.022046 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14102) v1 -- 0x558a1560f0e0 con 0 2017-03-02 15:08:08.022102 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27321) finish_round waiting_for_readable 2017-03-02 15:08:08.022109 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27321) finish_round waiting_for_writeable 2017-03-02 15:08:08.022097 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14101) v1 ==== 0+0+0 (0 0 0) 0x558a14c1dde0 con 0x558a14482000 2017-03-02 15:08:08.022111 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27321) finish_round done w/ waiters, state active 2017-03-02 15:08:08.022137 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:08.022144 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:08.022147 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:08.022150 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:08.022151 7f4f71beb700 20 allow all 2017-03-02 15:08:08.022167 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14102) v1 ==== 0+0+0 (0 0 0) 0x558a1560f0e0 con 0x558a14482000 2017-03-02 15:08:08.022173 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:08.022175 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:08.022176 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:08.022178 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:08.022178 7f4f71beb700 20 allow all 2017-03-02 15:08:08.259791 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.3 172.21.15.17:6800/27100 6830 ==== pg_stats(1 pgs tid 6794 v 0) v1 ==== 706+0+0 (2078404550 0 0) 0x558a15aeb980 con 0x558a145d7000 2017-03-02 15:08:08.259841 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415f440 for osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:08.259850 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:08.259853 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12963) dispatch 0x558a15aeb980 pg_stats(1 pgs tid 6794 v 0) v1 from osd.3 172.21.15.17:6800/27100 con 0x558a145d7000 2017-03-02 15:08:08.259859 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27321) is_readable = 1 - now=2017-03-02 15:08:08.259859 lease_expire=0.000000 has v0 lc 27321 2017-03-02 15:08:08.259867 7f4f71beb700 10 mon.0@0(leader).pg v12963 preprocess_query pg_stats(1 pgs tid 6794 v 0) v1 from osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:08.259874 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:08.259877 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:08.259878 7f4f71beb700 20 allow all 2017-03-02 15:08:08.259880 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27321) is_readable = 1 - now=2017-03-02 15:08:08.259880 lease_expire=0.000000 has v0 lc 27321 2017-03-02 15:08:08.259884 7f4f71beb700 10 mon.0@0(leader).pg v12963 prepare_update pg_stats(1 pgs tid 6794 v 0) v1 from osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:08.259889 7f4f71beb700 10 mon.0@0(leader).pg v12963 prepare_pg_stats pg_stats(1 pgs tid 6794 v 0) v1 from osd.3 2017-03-02 15:08:08.259896 7f4f71beb700 10 mon.0@0(leader).pg v12963 got osd.3 osd_stat(42086 MB used, 53255 MB avail, 95341 MB total, peers [0,1,2] op hist [0,1]) (was osd_stat(42049 MB used, 53292 MB avail, 95341 MB total, peers [0,1,2] op hist [])) 2017-03-02 15:08:08.259905 7f4f71beb700 15 mon.0@0(leader).pg v12963 got 2.6 reported at 7:963712 state active+clean -> active+clean 2017-03-02 15:08:08.259915 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12963) setting proposal_timer 0x558a1412f760 with delay of 0.761655 2017-03-02 15:08:08.462412 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.1 172.21.15.22:6800/26531 6830 ==== pg_stats(5 pgs tid 6794 v 0) v1 ==== 2970+0+0 (2282682962 0 0) 0x558a15da0000 con 0x558a14643800 2017-03-02 15:08:08.462447 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1463b1e0 for osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:08.462454 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:08.462456 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12963) dispatch 0x558a15da0000 pg_stats(5 pgs tid 6794 v 0) v1 from osd.1 172.21.15.22:6800/26531 con 0x558a14643800 2017-03-02 15:08:08.462461 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27321) is_readable = 1 - now=2017-03-02 15:08:08.462461 lease_expire=0.000000 has v0 lc 27321 2017-03-02 15:08:08.462468 7f4f71beb700 10 mon.0@0(leader).pg v12963 preprocess_query pg_stats(5 pgs tid 6794 v 0) v1 from osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:08.462474 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:08.462476 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:08.462477 7f4f71beb700 20 allow all 2017-03-02 15:08:08.462478 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27321) is_readable = 1 - now=2017-03-02 15:08:08.462478 lease_expire=0.000000 has v0 lc 27321 2017-03-02 15:08:08.462482 7f4f71beb700 10 mon.0@0(leader).pg v12963 prepare_update pg_stats(5 pgs tid 6794 v 0) v1 from osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:08.462485 7f4f71beb700 10 mon.0@0(leader).pg v12963 prepare_pg_stats pg_stats(5 pgs tid 6794 v 0) v1 from osd.1 2017-03-02 15:08:08.462491 7f4f71beb700 10 mon.0@0(leader).pg v12963 got osd.1 osd_stat(52913 MB used, 42428 MB avail, 95341 MB total, peers [0,2,3] op hist [0,1]) (was osd_stat(52989 MB used, 42352 MB avail, 95341 MB total, peers [0,2,3] op hist [])) 2017-03-02 15:08:08.462498 7f4f71beb700 15 mon.0@0(leader).pg v12963 got 1.1 reported at 7:586145 state active+clean -> active+clean 2017-03-02 15:08:08.462505 7f4f71beb700 15 mon.0@0(leader).pg v12963 got 1.5 reported at 7:567405 state active+clean -> active+clean 2017-03-02 15:08:08.462509 7f4f71beb700 15 mon.0@0(leader).pg v12963 got 1.6 reported at 7:502691 state active+clean -> active+clean 2017-03-02 15:08:08.462513 7f4f71beb700 15 mon.0@0(leader).pg v12963 got 2.4 reported at 7:1025417 state active+clean -> active+clean 2017-03-02 15:08:08.462516 7f4f71beb700 15 mon.0@0(leader).pg v12963 got 2.5 reported at 7:1128695 state active+clean -> active+clean 2017-03-02 15:08:08.462520 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12963) proposal_timer already set 2017-03-02 15:08:08.555707 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.2 172.21.15.11:6800/26536 6830 ==== pg_stats(8 pgs tid 6794 v 0) v1 ==== 4660+0+0 (381852845 0 0) 0x558a159d7c00 con 0x558a145d5800 2017-03-02 15:08:08.555747 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a145c3700 for osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:08.555753 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:08.555756 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12963) dispatch 0x558a159d7c00 pg_stats(8 pgs tid 6794 v 0) v1 from osd.2 172.21.15.11:6800/26536 con 0x558a145d5800 2017-03-02 15:08:08.555763 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27321) is_readable = 1 - now=2017-03-02 15:08:08.555764 lease_expire=0.000000 has v0 lc 27321 2017-03-02 15:08:08.555773 7f4f71beb700 10 mon.0@0(leader).pg v12963 preprocess_query pg_stats(8 pgs tid 6794 v 0) v1 from osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:08.555780 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:08.555783 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:08.555785 7f4f71beb700 20 allow all 2017-03-02 15:08:08.555786 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27321) is_readable = 1 - now=2017-03-02 15:08:08.555787 lease_expire=0.000000 has v0 lc 27321 2017-03-02 15:08:08.555792 7f4f71beb700 10 mon.0@0(leader).pg v12963 prepare_update pg_stats(8 pgs tid 6794 v 0) v1 from osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:08.555797 7f4f71beb700 10 mon.0@0(leader).pg v12963 prepare_pg_stats pg_stats(8 pgs tid 6794 v 0) v1 from osd.2 2017-03-02 15:08:08.555804 7f4f71beb700 10 mon.0@0(leader).pg v12963 got osd.2 osd_stat(63661 MB used, 31679 MB avail, 95341 MB total, peers [0,1,3] op hist []) (was osd_stat(63682 MB used, 31659 MB avail, 95341 MB total, peers [0,1,3] op hist [])) 2017-03-02 15:08:08.555814 7f4f71beb700 15 mon.0@0(leader).pg v12963 got 1.0 reported at 7:496008 state active+clean -> active+clean 2017-03-02 15:08:08.555861 7f4f71beb700 15 mon.0@0(leader).pg v12963 got 1.2 reported at 7:576800 state active+clean -> active+clean 2017-03-02 15:08:08.555867 7f4f71beb700 15 mon.0@0(leader).pg v12963 got 1.7 reported at 7:529071 state active+clean -> active+clean 2017-03-02 15:08:08.555873 7f4f71beb700 15 mon.0@0(leader).pg v12963 got 2.0 reported at 7:1056015 state active+clean -> active+clean 2017-03-02 15:08:08.555879 7f4f71beb700 15 mon.0@0(leader).pg v12963 got 2.1 reported at 7:1088704 state active+clean -> active+clean 2017-03-02 15:08:08.555884 7f4f71beb700 15 mon.0@0(leader).pg v12963 got 2.2 reported at 7:1162442 state active+clean -> active+clean 2017-03-02 15:08:08.555889 7f4f71beb700 15 mon.0@0(leader).pg v12963 got 2.3 reported at 7:1094592 state active+clean -> active+clean 2017-03-02 15:08:08.555894 7f4f71beb700 15 mon.0@0(leader).pg v12963 got 2.7 reported at 7:1134624 state active+clean -> active+clean 2017-03-02 15:08:08.555901 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12963) proposal_timer already set 2017-03-02 15:08:08.709416 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.0 172.21.15.26:6800/26515 6830 ==== pg_stats(2 pgs tid 6794 v 0) v1 ==== 1300+0+0 (1971412914 0 0) 0x558a15c13200 con 0x558a145d8800 2017-03-02 15:08:08.709465 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1463af20 for osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:08.709472 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:08.709476 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12963) dispatch 0x558a15c13200 pg_stats(2 pgs tid 6794 v 0) v1 from osd.0 172.21.15.26:6800/26515 con 0x558a145d8800 2017-03-02 15:08:08.709482 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27321) is_readable = 1 - now=2017-03-02 15:08:08.709482 lease_expire=0.000000 has v0 lc 27321 2017-03-02 15:08:08.709494 7f4f71beb700 10 mon.0@0(leader).pg v12963 preprocess_query pg_stats(2 pgs tid 6794 v 0) v1 from osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:08.709501 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:08.709504 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:08.709509 7f4f71beb700 20 allow all 2017-03-02 15:08:08.709511 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27321) is_readable = 1 - now=2017-03-02 15:08:08.709512 lease_expire=0.000000 has v0 lc 27321 2017-03-02 15:08:08.709516 7f4f71beb700 10 mon.0@0(leader).pg v12963 prepare_update pg_stats(2 pgs tid 6794 v 0) v1 from osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:08.709521 7f4f71beb700 10 mon.0@0(leader).pg v12963 prepare_pg_stats pg_stats(2 pgs tid 6794 v 0) v1 from osd.0 2017-03-02 15:08:08.709530 7f4f71beb700 10 mon.0@0(leader).pg v12963 got osd.0 osd_stat(10928 MB used, 84413 MB avail, 95341 MB total, peers [1,2,3] op hist [0,0,2,0,0,0,10,25,28]) (was osd_stat(10935 MB used, 84405 MB avail, 95341 MB total, peers [1,2,3] op hist [])) 2017-03-02 15:08:08.709542 7f4f71beb700 15 mon.0@0(leader).pg v12963 got 1.3 reported at 7:499552 state active+clean -> active+clean 2017-03-02 15:08:08.709554 7f4f71beb700 15 mon.0@0(leader).pg v12963 got 1.4 reported at 7:485999 state active+clean -> active+clean 2017-03-02 15:08:08.709560 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12963) proposal_timer already set 2017-03-02 15:08:09.021714 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12963) propose_pending 2017-03-02 15:08:09.021742 7f4f743f0700 10 mon.0@0(leader).pg v12963 encode_pending v 12964 2017-03-02 15:08:09.021953 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27321) queue_pending_finisher 0x558a1412e3c0 2017-03-02 15:08:09.021970 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27321) trigger_propose active, proposing now 2017-03-02 15:08:09.021998 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27321) propose_pending 27322 10511 bytes 2017-03-02 15:08:09.022002 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27321) begin for 27322 10511 bytes 2017-03-02 15:08:09.046311 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27321) commit_start 27322 2017-03-02 15:08:09.071472 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27321) commit_finish 27322 2017-03-02 15:08:09.071537 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:09.071578 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12964) refresh 2017-03-02 15:08:09.071584 7f4f763f4700 10 mon.0@0(leader).pg v12963 update_from_paxos read_incremental 2017-03-02 15:08:09.071625 7f4f763f4700 20 mon.0@0(leader).pg v12963 refreshing pg 1.0 7:496008 active+clean 2017-03-02 15:08:09.071667 7f4f763f4700 20 mon.0@0(leader).pg v12963 refreshing pg 1.1 7:586145 active+clean 2017-03-02 15:08:09.071686 7f4f763f4700 20 mon.0@0(leader).pg v12963 refreshing pg 1.2 7:576800 active+clean 2017-03-02 15:08:09.071702 7f4f763f4700 20 mon.0@0(leader).pg v12963 refreshing pg 1.3 7:499552 active+clean 2017-03-02 15:08:09.071718 7f4f763f4700 20 mon.0@0(leader).pg v12963 refreshing pg 1.4 7:485999 active+clean 2017-03-02 15:08:09.071733 7f4f763f4700 20 mon.0@0(leader).pg v12963 refreshing pg 1.5 7:567405 active+clean 2017-03-02 15:08:09.071748 7f4f763f4700 20 mon.0@0(leader).pg v12963 refreshing pg 1.6 7:502691 active+clean 2017-03-02 15:08:09.071763 7f4f763f4700 20 mon.0@0(leader).pg v12963 refreshing pg 1.7 7:529071 active+clean 2017-03-02 15:08:09.071778 7f4f763f4700 20 mon.0@0(leader).pg v12963 refreshing pg 2.0 7:1056015 active+clean 2017-03-02 15:08:09.071793 7f4f763f4700 20 mon.0@0(leader).pg v12963 refreshing pg 2.1 7:1088704 active+clean 2017-03-02 15:08:09.071808 7f4f763f4700 20 mon.0@0(leader).pg v12963 refreshing pg 2.2 7:1162442 active+clean 2017-03-02 15:08:09.071866 7f4f763f4700 20 mon.0@0(leader).pg v12963 refreshing pg 2.3 7:1094592 active+clean 2017-03-02 15:08:09.071888 7f4f763f4700 20 mon.0@0(leader).pg v12963 refreshing pg 2.4 7:1025417 active+clean 2017-03-02 15:08:09.071903 7f4f763f4700 20 mon.0@0(leader).pg v12963 refreshing pg 2.5 7:1128695 active+clean 2017-03-02 15:08:09.071916 7f4f763f4700 20 mon.0@0(leader).pg v12963 refreshing pg 2.6 7:963712 active+clean 2017-03-02 15:08:09.071931 7f4f763f4700 20 mon.0@0(leader).pg v12963 refreshing pg 2.7 7:1134624 active+clean 2017-03-02 15:08:09.071937 7f4f763f4700 20 mon.0@0(leader).pg v12963 refreshing osd.0 2017-03-02 15:08:09.071952 7f4f763f4700 20 mon.0@0(leader).pg v12963 refreshing osd.1 2017-03-02 15:08:09.071962 7f4f763f4700 20 mon.0@0(leader).pg v12963 refreshing osd.2 2017-03-02 15:08:09.071971 7f4f763f4700 20 mon.0@0(leader).pg v12963 refreshing osd.3 2017-03-02 15:08:09.071990 7f4f763f4700 10 mon.0@0(leader).pg v12964 read_pgmap_meta 2017-03-02 15:08:09.072034 7f4f763f4700 10 mon.0@0(leader).pg v12964 update_logger 2017-03-02 15:08:09.072061 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) refresh 2017-03-02 15:08:09.072093 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:09.072122 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13542) refresh 2017-03-02 15:08:09.072126 7f4f763f4700 10 mon.0@0(leader).log v13542 update_from_paxos 2017-03-02 15:08:09.072130 7f4f763f4700 10 mon.0@0(leader).log v13542 update_from_paxos version 13542 summary v 13542 2017-03-02 15:08:09.072155 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:09.072185 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:09.072189 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:09.072217 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:09.072221 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12964) post_refresh 2017-03-02 15:08:09.072225 7f4f763f4700 10 mon.0@0(leader).pg v12964 post_paxos_update 2017-03-02 15:08:09.072228 7f4f763f4700 10 mon.0@0(leader).pg v12964 check_subs 2017-03-02 15:08:09.072233 7f4f763f4700 20 mon.0@0(leader).pg v12964 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:09.072244 7f4f763f4700 20 mon.0@0(leader).pg v12964 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:09.072250 7f4f763f4700 20 mon.0@0(leader).pg v12964 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:09.072256 7f4f763f4700 20 mon.0@0(leader).pg v12964 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:09.072262 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) post_refresh 2017-03-02 15:08:09.072265 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:09.072268 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13542) post_refresh 2017-03-02 15:08:09.072271 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:09.072273 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:09.072276 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:09.072280 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27322) commit_proposal 2017-03-02 15:08:09.072285 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12964) _active - not active 2017-03-02 15:08:09.072290 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27322) finish_round 2017-03-02 15:08:09.072293 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27322) finish_round waiting_for_acting 2017-03-02 15:08:09.072297 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12964) _active 2017-03-02 15:08:09.072301 7f4f763f4700 7 mon.0@0(leader).paxosservice(pgmap 12269..12964) _active creating new pending 2017-03-02 15:08:09.072311 7f4f763f4700 10 mon.0@0(leader).pg v12964 create_pending v 12965 2017-03-02 15:08:09.072321 7f4f763f4700 7 mon.0@0(leader).pg v12964 _updated_stats for osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:09.072329 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a15a05300 0x558a15e1fa00 pg_stats_ack(1 pgs tid 6794) v1 2017-03-02 15:08:09.072336 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6800/27100 -- pg_stats_ack(1 pgs tid 6794) v1 -- 0x558a15e1fa00 con 0 2017-03-02 15:08:09.072419 7f4f763f4700 7 mon.0@0(leader).pg v12964 _updated_stats for osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:09.072429 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a16385a00 0x558a1589cc00 pg_stats_ack(5 pgs tid 6794) v1 2017-03-02 15:08:09.072434 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6800/26531 -- pg_stats_ack(5 pgs tid 6794) v1 -- 0x558a1589cc00 con 0 2017-03-02 15:08:09.072469 7f4f763f4700 7 mon.0@0(leader).pg v12964 _updated_stats for osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:09.072477 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a16383a00 0x558a15434e00 pg_stats_ack(8 pgs tid 6794) v1 2017-03-02 15:08:09.072481 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6800/26536 -- pg_stats_ack(8 pgs tid 6794) v1 -- 0x558a15434e00 con 0 2017-03-02 15:08:09.072515 7f4f763f4700 7 mon.0@0(leader).pg v12964 _updated_stats for osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:09.072522 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a15c70100 0x558a15437e00 pg_stats_ack(2 pgs tid 6794) v1 2017-03-02 15:08:09.072526 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6800/26515 -- pg_stats_ack(2 pgs tid 6794) v1 -- 0x558a15437e00 con 0 2017-03-02 15:08:09.072569 7f4f763f4700 10 mon.0@0(leader).pg v12964 check_osd_map already seen 7 >= 7 2017-03-02 15:08:09.072575 7f4f763f4700 10 mon.0@0(leader).pg v12964 update_logger 2017-03-02 15:08:09.072604 7f4f763f4700 0 log_channel(cluster) log [INF] : pgmap v12964: 32 pgs: 32 active+clean; 88252 MB data, 165 GB used, 206 GB / 372 GB avail; 1080 kB/s rd, 21823 kB/s wr, 319 op/s 2017-03-02 15:08:09.072630 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14103 at 2017-03-02 15:08:09.072607) v1 -- 0x558a1456f3c0 con 0 2017-03-02 15:08:09.072659 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27322) finish_round waiting_for_readable 2017-03-02 15:08:09.072664 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27322) finish_round waiting_for_writeable 2017-03-02 15:08:09.072668 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27322) finish_round done w/ waiters, state active 2017-03-02 15:08:09.072724 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14103 at 2017-03-02 15:08:09.072607) v1 ==== 0+0+0 (0 0 0) 0x558a1456f3c0 con 0x558a14482000 2017-03-02 15:08:09.072770 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:09.072781 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:09.072787 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13542) dispatch 0x558a1456f3c0 log(1 entries from seq 14103 at 2017-03-02 15:08:09.072607) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:09.072803 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27322) is_readable = 1 - now=2017-03-02 15:08:09.072804 lease_expire=0.000000 has v0 lc 27322 2017-03-02 15:08:09.072816 7f4f71beb700 10 mon.0@0(leader).log v13542 preprocess_query log(1 entries from seq 14103 at 2017-03-02 15:08:09.072607) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:09.072856 7f4f71beb700 10 mon.0@0(leader).log v13542 preprocess_log log(1 entries from seq 14103 at 2017-03-02 15:08:09.072607) v1 from mon.0 2017-03-02 15:08:09.072867 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:09.072873 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:09.072875 7f4f71beb700 20 allow all 2017-03-02 15:08:09.072887 7f4f71beb700 10 mon.0@0(leader).log v13542 prepare_update log(1 entries from seq 14103 at 2017-03-02 15:08:09.072607) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:09.072898 7f4f71beb700 10 mon.0@0(leader).log v13542 prepare_log log(1 entries from seq 14103 at 2017-03-02 15:08:09.072607) v1 from mon.0 2017-03-02 15:08:09.072906 7f4f71beb700 10 mon.0@0(leader).log v13542 logging 2017-03-02 15:08:09.072607 mon.0 172.21.15.17:6789/0 14103 : cluster [INF] pgmap v12964: 32 pgs: 32 active+clean; 88252 MB data, 165 GB used, 206 GB / 372 GB avail; 1080 kB/s rd, 21823 kB/s wr, 319 op/s 2017-03-02 15:08:09.072923 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13542) setting proposal_timer 0x558a1412e9d0 with delay of 0.998572 2017-03-02 15:08:10.071594 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13542) propose_pending 2017-03-02 15:08:10.071623 7f4f743f0700 10 mon.0@0(leader).log v13542 encode_full log v 13542 2017-03-02 15:08:10.071735 7f4f743f0700 10 mon.0@0(leader).log v13542 encode_pending v13543 2017-03-02 15:08:10.071747 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27322) queue_pending_finisher 0x558a1412f760 2017-03-02 15:08:10.071752 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27322) trigger_propose active, proposing now 2017-03-02 15:08:10.071763 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27322) propose_pending 27323 10385 bytes 2017-03-02 15:08:10.071767 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27322) begin for 27323 10385 bytes 2017-03-02 15:08:10.096333 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27322) commit_start 27323 2017-03-02 15:08:10.121454 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27322) commit_finish 27323 2017-03-02 15:08:10.121510 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:10.121542 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12964) refresh 2017-03-02 15:08:10.121559 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) refresh 2017-03-02 15:08:10.121579 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:10.121607 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13543) refresh 2017-03-02 15:08:10.121611 7f4f763f4700 10 mon.0@0(leader).log v13543 update_from_paxos 2017-03-02 15:08:10.121614 7f4f763f4700 10 mon.0@0(leader).log v13543 update_from_paxos version 13543 summary v 13542 2017-03-02 15:08:10.121620 7f4f763f4700 10 mon.0@0(leader).log v13543 update_from_paxos latest full 13542 2017-03-02 15:08:10.121632 7f4f763f4700 7 mon.0@0(leader).log v13543 update_from_paxos applying incremental log 13543 2017-03-02 15:08:09.072607 mon.0 172.21.15.17:6789/0 14103 : cluster [INF] pgmap v12964: 32 pgs: 32 active+clean; 88252 MB data, 165 GB used, 206 GB / 372 GB avail; 1080 kB/s rd, 21823 kB/s wr, 319 op/s 2017-03-02 15:08:10.121650 7f4f763f4700 20 mon.0@0(leader).log v13543 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:10.121672 7f4f763f4700 15 mon.0@0(leader).log v13543 update_from_paxos logging for 1 channels 2017-03-02 15:08:10.121682 7f4f763f4700 15 mon.0@0(leader).log v13543 update_from_paxos channel 'cluster' logging 203 bytes 2017-03-02 15:08:10.121757 7f4f763f4700 10 mon.0@0(leader).log v13543 check_subs 2017-03-02 15:08:10.121808 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:10.121870 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:10.121875 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:10.121896 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:10.121899 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12964) post_refresh 2017-03-02 15:08:10.121902 7f4f763f4700 10 mon.0@0(leader).pg v12964 post_paxos_update 2017-03-02 15:08:10.121905 7f4f763f4700 10 mon.0@0(leader).pg v12964 check_subs 2017-03-02 15:08:10.121908 7f4f763f4700 20 mon.0@0(leader).pg v12964 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:10.121916 7f4f763f4700 20 mon.0@0(leader).pg v12964 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:10.121919 7f4f763f4700 20 mon.0@0(leader).pg v12964 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:10.121924 7f4f763f4700 20 mon.0@0(leader).pg v12964 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:10.121927 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) post_refresh 2017-03-02 15:08:10.121930 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:10.121932 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13543) post_refresh 2017-03-02 15:08:10.121933 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:10.121935 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:10.121937 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:10.121940 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27323) commit_proposal 2017-03-02 15:08:10.121944 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13543) _active - not active 2017-03-02 15:08:10.121948 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27323) finish_round 2017-03-02 15:08:10.121950 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27323) finish_round waiting_for_acting 2017-03-02 15:08:10.121953 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13543) _active 2017-03-02 15:08:10.121955 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13543) _active creating new pending 2017-03-02 15:08:10.121965 7f4f763f4700 10 mon.0@0(leader).log v13543 create_pending v 13544 2017-03-02 15:08:10.121969 7f4f763f4700 7 mon.0@0(leader).log v13543 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:10.121976 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a1498ea00 0x558a15153de0 log(last 14103) v1 2017-03-02 15:08:10.121981 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14103) v1 -- 0x558a15153de0 con 0 2017-03-02 15:08:10.122467 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27323) finish_round waiting_for_readable 2017-03-02 15:08:10.122482 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27323) finish_round waiting_for_writeable 2017-03-02 15:08:10.122486 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27323) finish_round done w/ waiters, state active 2017-03-02 15:08:10.122484 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14103) v1 ==== 0+0+0 (0 0 0) 0x558a15153de0 con 0x558a14482000 2017-03-02 15:08:10.122524 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:10.122531 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:10.122534 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:10.122536 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:10.122537 7f4f71beb700 20 allow all 2017-03-02 15:08:10.869885 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.5 172.21.15.11:6804/1637448123 8781 ==== mdsbeacon(4116/e up:active seq 8494 v733) v7 ==== 365+0+0 (1172283093 0 0) 0x558a148d7380 con 0x558a1471d800 2017-03-02 15:08:10.869912 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b0f20 for mds.? 172.21.15.11:6804/1637448123 2017-03-02 15:08:10.869916 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:10.869919 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) dispatch 0x558a148d7380 mdsbeacon(4116/e up:active seq 8494 v733) v7 from mds.5 172.21.15.11:6804/1637448123 con 0x558a1471d800 2017-03-02 15:08:10.869923 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27323) is_readable = 1 - now=2017-03-02 15:08:10.869924 lease_expire=0.000000 has v0 lc 27323 2017-03-02 15:08:10.869929 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_query mdsbeacon(4116/e up:active seq 8494 v733) v7 from mds.5 172.21.15.11:6804/1637448123 2017-03-02 15:08:10.869934 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:10.869936 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:10.869937 7f4f71beb700 20 allow all 2017-03-02 15:08:10.869938 7f4f71beb700 12 mon.0@0(leader).mds e733 preprocess_beacon mdsbeacon(4116/e up:active seq 8494 v733) v7 from mds.5 172.21.15.11:6804/1637448123 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:10.869943 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_beacon: GID exists in map: 4116 2017-03-02 15:08:10.869947 7f4f71beb700 15 mon.0@0(leader).mds e733 _note_beacon mdsbeacon(4116/e up:active seq 8494 v733) v7 noting time 2017-03-02 15:08:10.869950 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a1479f200 0x558a15204000 mdsbeacon(4116/e up:active seq 8494 v733) v7 2017-03-02 15:08:10.869952 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6804/1637448123 -- mdsbeacon(4116/e up:active seq 8494 v733) v7 -- 0x558a15204000 con 0 2017-03-02 15:08:10.883590 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.3 172.21.15.11:6805/1926402648 8557 ==== mdsbeacon(4118/f up:active seq 8494 v733) v7 ==== 406+0+0 (3018196071 0 0) 0x558a160c0d80 con 0x558a14905800 2017-03-02 15:08:10.883614 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b2260 for mds.? 172.21.15.11:6805/1926402648 2017-03-02 15:08:10.883620 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:10.883622 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) dispatch 0x558a160c0d80 mdsbeacon(4118/f up:active seq 8494 v733) v7 from mds.3 172.21.15.11:6805/1926402648 con 0x558a14905800 2017-03-02 15:08:10.883627 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27323) is_readable = 1 - now=2017-03-02 15:08:10.883627 lease_expire=0.000000 has v0 lc 27323 2017-03-02 15:08:10.883632 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_query mdsbeacon(4118/f up:active seq 8494 v733) v7 from mds.3 172.21.15.11:6805/1926402648 2017-03-02 15:08:10.883636 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:10.883637 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:10.883638 7f4f71beb700 20 allow all 2017-03-02 15:08:10.883639 7f4f71beb700 12 mon.0@0(leader).mds e733 preprocess_beacon mdsbeacon(4118/f up:active seq 8494 v733) v7 from mds.3 172.21.15.11:6805/1926402648 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:10.883643 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_beacon: GID exists in map: 4118 2017-03-02 15:08:10.883646 7f4f71beb700 15 mon.0@0(leader).mds e733 _note_beacon mdsbeacon(4118/f up:active seq 8494 v733) v7 noting time 2017-03-02 15:08:10.883649 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a1479f200 0x558a148d7380 mdsbeacon(4118/f up:active seq 8494 v733) v7 2017-03-02 15:08:10.883651 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6805/1926402648 -- mdsbeacon(4118/f up:active seq 8494 v733) v7 -- 0x558a148d7380 con 0 2017-03-02 15:08:10.884408 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.2 172.21.15.22:6804/673088617 8556 ==== mdsbeacon(4119/c up:active seq 8494 v733) v7 ==== 365+0+0 (1707514507 0 0) 0x558a15204000 con 0x558a14907000 2017-03-02 15:08:10.884422 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14916f20 for mds.? 172.21.15.22:6804/673088617 2017-03-02 15:08:10.884425 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:10.884427 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) dispatch 0x558a15204000 mdsbeacon(4119/c up:active seq 8494 v733) v7 from mds.2 172.21.15.22:6804/673088617 con 0x558a14907000 2017-03-02 15:08:10.884430 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27323) is_readable = 1 - now=2017-03-02 15:08:10.884430 lease_expire=0.000000 has v0 lc 27323 2017-03-02 15:08:10.884434 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_query mdsbeacon(4119/c up:active seq 8494 v733) v7 from mds.2 172.21.15.22:6804/673088617 2017-03-02 15:08:10.884437 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:10.884438 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:10.884439 7f4f71beb700 20 allow all 2017-03-02 15:08:10.884440 7f4f71beb700 12 mon.0@0(leader).mds e733 preprocess_beacon mdsbeacon(4119/c up:active seq 8494 v733) v7 from mds.2 172.21.15.22:6804/673088617 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:10.884443 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_beacon: GID exists in map: 4119 2017-03-02 15:08:10.884446 7f4f71beb700 15 mon.0@0(leader).mds e733 _note_beacon mdsbeacon(4119/c up:active seq 8494 v733) v7 noting time 2017-03-02 15:08:10.884448 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a1479f200 0x558a160c0d80 mdsbeacon(4119/c up:active seq 8494 v733) v7 2017-03-02 15:08:10.884449 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6804/673088617 -- mdsbeacon(4119/c up:active seq 8494 v733) v7 -- 0x558a160c0d80 con 0 2017-03-02 15:08:10.884673 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.0 172.21.15.22:6805/2203629154 8606 ==== mdsbeacon(4121/d up:active seq 8490 v733) v7 ==== 452+0+0 (1245286478 0 0) 0x558a148d7380 con 0x558a1492a000 2017-03-02 15:08:10.884690 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a149195a0 for mds.? 172.21.15.22:6805/2203629154 2017-03-02 15:08:10.884695 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:10.884697 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) dispatch 0x558a148d7380 mdsbeacon(4121/d up:active seq 8490 v733) v7 from mds.0 172.21.15.22:6805/2203629154 con 0x558a1492a000 2017-03-02 15:08:10.884701 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27323) is_readable = 1 - now=2017-03-02 15:08:10.884702 lease_expire=0.000000 has v0 lc 27323 2017-03-02 15:08:10.884705 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_query mdsbeacon(4121/d up:active seq 8490 v733) v7 from mds.0 172.21.15.22:6805/2203629154 2017-03-02 15:08:10.884708 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:10.884710 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:10.884710 7f4f71beb700 20 allow all 2017-03-02 15:08:10.884711 7f4f71beb700 12 mon.0@0(leader).mds e733 preprocess_beacon mdsbeacon(4121/d up:active seq 8490 v733) v7 from mds.0 172.21.15.22:6805/2203629154 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:10.884714 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_beacon: GID exists in map: 4121 2017-03-02 15:08:10.884717 7f4f71beb700 15 mon.0@0(leader).mds e733 _note_beacon mdsbeacon(4121/d up:active seq 8490 v733) v7 noting time 2017-03-02 15:08:10.884719 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a1479f200 0x558a15204000 mdsbeacon(4121/d up:active seq 8490 v733) v7 2017-03-02 15:08:10.884721 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6805/2203629154 -- mdsbeacon(4121/d up:active seq 8490 v733) v7 -- 0x558a15204000 con 0 2017-03-02 15:08:10.923800 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.1 172.21.15.17:6804/2576824570 8553 ==== mdsbeacon(4120/g up:active seq 8494 v733) v7 ==== 365+0+0 (2915686288 0 0) 0x558a154b8d00 con 0x558a14908800 2017-03-02 15:08:10.923840 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14917600 for mds.? 172.21.15.17:6804/2576824570 2017-03-02 15:08:10.923845 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:10.923847 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) dispatch 0x558a154b8d00 mdsbeacon(4120/g up:active seq 8494 v733) v7 from mds.1 172.21.15.17:6804/2576824570 con 0x558a14908800 2017-03-02 15:08:10.923852 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27323) is_readable = 1 - now=2017-03-02 15:08:10.923852 lease_expire=0.000000 has v0 lc 27323 2017-03-02 15:08:10.923858 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_query mdsbeacon(4120/g up:active seq 8494 v733) v7 from mds.1 172.21.15.17:6804/2576824570 2017-03-02 15:08:10.923862 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:10.923864 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:10.923865 7f4f71beb700 20 allow all 2017-03-02 15:08:10.923866 7f4f71beb700 12 mon.0@0(leader).mds e733 preprocess_beacon mdsbeacon(4120/g up:active seq 8494 v733) v7 from mds.1 172.21.15.17:6804/2576824570 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:10.923870 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_beacon: GID exists in map: 4120 2017-03-02 15:08:10.923874 7f4f71beb700 15 mon.0@0(leader).mds e733 _note_beacon mdsbeacon(4120/g up:active seq 8494 v733) v7 noting time 2017-03-02 15:08:10.923877 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a1479f200 0x558a148d7380 mdsbeacon(4120/g up:active seq 8494 v733) v7 2017-03-02 15:08:10.923879 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6804/2576824570 -- mdsbeacon(4120/g up:active seq 8494 v733) v7 -- 0x558a148d7380 con 0 2017-03-02 15:08:10.968943 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.4 172.21.15.26:6805/2033388711 8551 ==== mdsbeacon(4117/b up:active seq 8494 v733) v7 ==== 365+0+0 (1398851923 0 0) 0x558a148d7380 con 0x558a14904000 2017-03-02 15:08:10.968970 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b1600 for mds.? 172.21.15.26:6805/2033388711 2017-03-02 15:08:10.968977 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:10.968980 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) dispatch 0x558a148d7380 mdsbeacon(4117/b up:active seq 8494 v733) v7 from mds.4 172.21.15.26:6805/2033388711 con 0x558a14904000 2017-03-02 15:08:10.968986 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27323) is_readable = 1 - now=2017-03-02 15:08:10.968987 lease_expire=0.000000 has v0 lc 27323 2017-03-02 15:08:10.968993 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_query mdsbeacon(4117/b up:active seq 8494 v733) v7 from mds.4 172.21.15.26:6805/2033388711 2017-03-02 15:08:10.969000 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:10.969002 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:10.969003 7f4f71beb700 20 allow all 2017-03-02 15:08:10.969004 7f4f71beb700 12 mon.0@0(leader).mds e733 preprocess_beacon mdsbeacon(4117/b up:active seq 8494 v733) v7 from mds.4 172.21.15.26:6805/2033388711 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:10.969011 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_beacon: GID exists in map: 4117 2017-03-02 15:08:10.969016 7f4f71beb700 15 mon.0@0(leader).mds e733 _note_beacon mdsbeacon(4117/b up:active seq 8494 v733) v7 noting time 2017-03-02 15:08:10.969020 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a1479f200 0x558a154b8d00 mdsbeacon(4117/b up:active seq 8494 v733) v7 2017-03-02 15:08:10.969023 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6805/2033388711 -- mdsbeacon(4117/b up:active seq 8494 v733) v7 -- 0x558a154b8d00 con 0 2017-03-02 15:08:10.977885 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.6 172.21.15.26:6804/2954782670 8743 ==== mdsbeacon(4115/a up:active seq 8494 v733) v7 ==== 365+0+0 (143277371 0 0) 0x558a15204000 con 0x558a1471f000 2017-03-02 15:08:10.977905 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14868d60 for mds.? 172.21.15.26:6804/2954782670 2017-03-02 15:08:10.977910 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:10.977912 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) dispatch 0x558a15204000 mdsbeacon(4115/a up:active seq 8494 v733) v7 from mds.6 172.21.15.26:6804/2954782670 con 0x558a1471f000 2017-03-02 15:08:10.977918 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27323) is_readable = 1 - now=2017-03-02 15:08:10.977918 lease_expire=0.000000 has v0 lc 27323 2017-03-02 15:08:10.977924 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_query mdsbeacon(4115/a up:active seq 8494 v733) v7 from mds.6 172.21.15.26:6804/2954782670 2017-03-02 15:08:10.977930 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:10.977932 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:10.977934 7f4f71beb700 20 allow all 2017-03-02 15:08:10.977936 7f4f71beb700 12 mon.0@0(leader).mds e733 preprocess_beacon mdsbeacon(4115/a up:active seq 8494 v733) v7 from mds.6 172.21.15.26:6804/2954782670 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:10.977943 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_beacon: GID exists in map: 4115 2017-03-02 15:08:10.977946 7f4f71beb700 15 mon.0@0(leader).mds e733 _note_beacon mdsbeacon(4115/a up:active seq 8494 v733) v7 noting time 2017-03-02 15:08:10.977950 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a1479f200 0x558a148d7380 mdsbeacon(4115/a up:active seq 8494 v733) v7 2017-03-02 15:08:10.977952 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6804/2954782670 -- mdsbeacon(4115/a up:active seq 8494 v733) v7 -- 0x558a148d7380 con 0 2017-03-02 15:08:11.011102 7f4f743f0700 11 mon.0@0(leader) e2 tick 2017-03-02 15:08:11.011121 7f4f743f0700 10 mon.0@0(leader).pg v12964 v12964: 32 pgs: 32 active+clean; 88252 MB data, 165 GB used, 206 GB / 372 GB avail; 1080 kB/s rd, 21823 kB/s wr, 319 op/s 2017-03-02 15:08:11.011136 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12964) maybe_trim trim_to 12464 would only trim 195 < paxos_service_trim_min 250 2017-03-02 15:08:11.011139 7f4f743f0700 10 mon.0@0(leader).mds e733 e733: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:11.011168 7f4f743f0700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) maybe_trim trim_to 233 would only trim 232 < paxos_service_trim_min 250 2017-03-02 15:08:11.011171 7f4f743f0700 10 mon.0@0(leader).osd e7 e7: 4 osds: 4 up, 4 in 2017-03-02 15:08:11.011174 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27323) is_readable = 1 - now=2017-03-02 15:08:11.011175 lease_expire=0.000000 has v0 lc 27323 2017-03-02 15:08:11.011180 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27323) is_readable = 1 - now=2017-03-02 15:08:11.011181 lease_expire=0.000000 has v0 lc 27323 2017-03-02 15:08:11.011186 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27323) is_readable = 1 - now=2017-03-02 15:08:11.011186 lease_expire=0.000000 has v0 lc 27323 2017-03-02 15:08:11.011189 7f4f743f0700 10 mon.0@0(leader).osd e7 min_last_epoch_clean 6 2017-03-02 15:08:11.011191 7f4f743f0700 10 mon.0@0(leader).log v13543 log 2017-03-02 15:08:11.011192 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13543) maybe_trim trim_to 13043 would only trim 16 < paxos_service_trim_min 250 2017-03-02 15:08:11.011195 7f4f743f0700 10 mon.0@0(leader).auth v10 auth 2017-03-02 15:08:11.011198 7f4f743f0700 10 cephx keyserver: _check_rotating_secrets 2017-03-02 15:08:11.011206 7f4f743f0700 20 mon.0@0(leader) e2 sync_trim_providers 2017-03-02 15:08:13.260609 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.3 172.21.15.17:6800/27100 6831 ==== pg_stats(1 pgs tid 6795 v 0) v1 ==== 710+0+0 (211354829 0 0) 0x558a15da1900 con 0x558a145d7000 2017-03-02 15:08:13.260646 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415f440 for osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:13.260654 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:13.260659 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12964) dispatch 0x558a15da1900 pg_stats(1 pgs tid 6795 v 0) v1 from osd.3 172.21.15.17:6800/27100 con 0x558a145d7000 2017-03-02 15:08:13.260667 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27323) is_readable = 1 - now=2017-03-02 15:08:13.260667 lease_expire=0.000000 has v0 lc 27323 2017-03-02 15:08:13.260677 7f4f71beb700 10 mon.0@0(leader).pg v12964 preprocess_query pg_stats(1 pgs tid 6795 v 0) v1 from osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:13.260686 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:13.260690 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:13.260691 7f4f71beb700 20 allow all 2017-03-02 15:08:13.260693 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27323) is_readable = 1 - now=2017-03-02 15:08:13.260694 lease_expire=0.000000 has v0 lc 27323 2017-03-02 15:08:13.260700 7f4f71beb700 10 mon.0@0(leader).pg v12964 prepare_update pg_stats(1 pgs tid 6795 v 0) v1 from osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:13.260707 7f4f71beb700 10 mon.0@0(leader).pg v12964 prepare_pg_stats pg_stats(1 pgs tid 6795 v 0) v1 from osd.3 2017-03-02 15:08:13.260717 7f4f71beb700 10 mon.0@0(leader).pg v12964 got osd.3 osd_stat(42071 MB used, 53269 MB avail, 95341 MB total, peers [0,1,2] op hist [2,0,1]) (was osd_stat(42086 MB used, 53255 MB avail, 95341 MB total, peers [0,1,2] op hist [0,1])) 2017-03-02 15:08:13.260730 7f4f71beb700 15 mon.0@0(leader).pg v12964 got 2.6 reported at 7:963875 state active+clean -> active+clean 2017-03-02 15:08:13.260743 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12964) setting proposal_timer 0x558a1412cff0 with delay of 0.05 2017-03-02 15:08:13.310853 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12964) propose_pending 2017-03-02 15:08:13.310872 7f4f743f0700 10 mon.0@0(leader).pg v12964 encode_pending v 12965 2017-03-02 15:08:13.310903 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27323) queue_pending_finisher 0x558a1412e830 2017-03-02 15:08:13.310906 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27323) trigger_propose active, proposing now 2017-03-02 15:08:13.310915 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27323) propose_pending 27324 1102 bytes 2017-03-02 15:08:13.310917 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27323) begin for 27324 1102 bytes 2017-03-02 15:08:13.319944 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27323) commit_start 27324 2017-03-02 15:08:13.328308 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27323) commit_finish 27324 2017-03-02 15:08:13.328345 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:13.328363 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12965) refresh 2017-03-02 15:08:13.328366 7f4f763f4700 10 mon.0@0(leader).pg v12964 update_from_paxos read_incremental 2017-03-02 15:08:13.328392 7f4f763f4700 20 mon.0@0(leader).pg v12964 refreshing pg 2.6 7:963875 active+clean 2017-03-02 15:08:13.328395 7f4f763f4700 20 mon.0@0(leader).pg v12964 refreshing osd.3 2017-03-02 15:08:13.328407 7f4f763f4700 10 mon.0@0(leader).pg v12965 read_pgmap_meta 2017-03-02 15:08:13.328421 7f4f763f4700 10 mon.0@0(leader).pg v12965 update_logger 2017-03-02 15:08:13.328430 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) refresh 2017-03-02 15:08:13.328442 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:13.328461 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13543) refresh 2017-03-02 15:08:13.328467 7f4f763f4700 10 mon.0@0(leader).log v13543 update_from_paxos 2017-03-02 15:08:13.328469 7f4f763f4700 10 mon.0@0(leader).log v13543 update_from_paxos version 13543 summary v 13543 2017-03-02 15:08:13.328478 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:13.328490 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:13.328491 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:13.328501 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:13.328503 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12965) post_refresh 2017-03-02 15:08:13.328504 7f4f763f4700 10 mon.0@0(leader).pg v12965 post_paxos_update 2017-03-02 15:08:13.328506 7f4f763f4700 10 mon.0@0(leader).pg v12965 check_subs 2017-03-02 15:08:13.328508 7f4f763f4700 20 mon.0@0(leader).pg v12965 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:13.328515 7f4f763f4700 20 mon.0@0(leader).pg v12965 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:13.328519 7f4f763f4700 20 mon.0@0(leader).pg v12965 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:13.328522 7f4f763f4700 20 mon.0@0(leader).pg v12965 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:13.328525 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) post_refresh 2017-03-02 15:08:13.328527 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:13.328529 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13543) post_refresh 2017-03-02 15:08:13.328530 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:13.328532 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:13.328533 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:13.328536 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27324) commit_proposal 2017-03-02 15:08:13.328540 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12965) _active - not active 2017-03-02 15:08:13.328543 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27324) finish_round 2017-03-02 15:08:13.328545 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27324) finish_round waiting_for_acting 2017-03-02 15:08:13.328547 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12965) _active 2017-03-02 15:08:13.328549 7f4f763f4700 7 mon.0@0(leader).paxosservice(pgmap 12269..12965) _active creating new pending 2017-03-02 15:08:13.328553 7f4f763f4700 10 mon.0@0(leader).pg v12965 create_pending v 12966 2017-03-02 15:08:13.328559 7f4f763f4700 7 mon.0@0(leader).pg v12965 _updated_stats for osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:13.328564 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a1479f200 0x558a1589d200 pg_stats_ack(1 pgs tid 6795) v1 2017-03-02 15:08:13.328569 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6800/27100 -- pg_stats_ack(1 pgs tid 6795) v1 -- 0x558a1589d200 con 0 2017-03-02 15:08:13.328611 7f4f763f4700 10 mon.0@0(leader).pg v12965 check_osd_map already seen 7 >= 7 2017-03-02 15:08:13.328616 7f4f763f4700 10 mon.0@0(leader).pg v12965 update_logger 2017-03-02 15:08:13.328638 7f4f763f4700 0 log_channel(cluster) log [INF] : pgmap v12965: 32 pgs: 32 active+clean; 88238 MB data, 165 GB used, 206 GB / 372 GB avail; 653 kB/s rd, 13875 kB/s wr, 203 op/s 2017-03-02 15:08:13.328659 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14104 at 2017-03-02 15:08:13.328640) v1 -- 0x558a1691eb40 con 0 2017-03-02 15:08:13.328681 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27324) finish_round waiting_for_readable 2017-03-02 15:08:13.328685 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27324) finish_round waiting_for_writeable 2017-03-02 15:08:13.328688 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27324) finish_round done w/ waiters, state active 2017-03-02 15:08:13.328707 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14104 at 2017-03-02 15:08:13.328640) v1 ==== 0+0+0 (0 0 0) 0x558a1691eb40 con 0x558a14482000 2017-03-02 15:08:13.328732 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:13.328738 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:13.328741 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13543) dispatch 0x558a1691eb40 log(1 entries from seq 14104 at 2017-03-02 15:08:13.328640) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:13.328749 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27324) is_readable = 1 - now=2017-03-02 15:08:13.328750 lease_expire=0.000000 has v0 lc 27324 2017-03-02 15:08:13.328757 7f4f71beb700 10 mon.0@0(leader).log v13543 preprocess_query log(1 entries from seq 14104 at 2017-03-02 15:08:13.328640) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:13.328763 7f4f71beb700 10 mon.0@0(leader).log v13543 preprocess_log log(1 entries from seq 14104 at 2017-03-02 15:08:13.328640) v1 from mon.0 2017-03-02 15:08:13.328767 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:13.328769 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:13.328770 7f4f71beb700 20 allow all 2017-03-02 15:08:13.328777 7f4f71beb700 10 mon.0@0(leader).log v13543 prepare_update log(1 entries from seq 14104 at 2017-03-02 15:08:13.328640) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:13.328781 7f4f71beb700 10 mon.0@0(leader).log v13543 prepare_log log(1 entries from seq 14104 at 2017-03-02 15:08:13.328640) v1 from mon.0 2017-03-02 15:08:13.328783 7f4f71beb700 10 mon.0@0(leader).log v13543 logging 2017-03-02 15:08:13.328640 mon.0 172.21.15.17:6789/0 14104 : cluster [INF] pgmap v12965: 32 pgs: 32 active+clean; 88238 MB data, 165 GB used, 206 GB / 372 GB avail; 653 kB/s rd, 13875 kB/s wr, 203 op/s 2017-03-02 15:08:13.328789 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13543) setting proposal_timer 0x558a1412efd0 with delay of 0.999534 2017-03-02 15:08:13.462895 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.1 172.21.15.22:6800/26531 6831 ==== pg_stats(5 pgs tid 6795 v 0) v1 ==== 2962+0+0 (1245894249 0 0) 0x558a15da2580 con 0x558a14643800 2017-03-02 15:08:13.462928 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1463b1e0 for osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:13.462933 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:13.462936 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12965) dispatch 0x558a15da2580 pg_stats(5 pgs tid 6795 v 0) v1 from osd.1 172.21.15.22:6800/26531 con 0x558a14643800 2017-03-02 15:08:13.462940 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27324) is_readable = 1 - now=2017-03-02 15:08:13.462941 lease_expire=0.000000 has v0 lc 27324 2017-03-02 15:08:13.462948 7f4f71beb700 10 mon.0@0(leader).pg v12965 preprocess_query pg_stats(5 pgs tid 6795 v 0) v1 from osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:13.462953 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:13.462955 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:13.462956 7f4f71beb700 20 allow all 2017-03-02 15:08:13.462957 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27324) is_readable = 1 - now=2017-03-02 15:08:13.462957 lease_expire=0.000000 has v0 lc 27324 2017-03-02 15:08:13.462960 7f4f71beb700 10 mon.0@0(leader).pg v12965 prepare_update pg_stats(5 pgs tid 6795 v 0) v1 from osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:13.462962 7f4f71beb700 10 mon.0@0(leader).pg v12965 prepare_pg_stats pg_stats(5 pgs tid 6795 v 0) v1 from osd.1 2017-03-02 15:08:13.462969 7f4f71beb700 10 mon.0@0(leader).pg v12965 got osd.1 osd_stat(52994 MB used, 42347 MB avail, 95341 MB total, peers [0,2,3] op hist []) (was osd_stat(52913 MB used, 42428 MB avail, 95341 MB total, peers [0,2,3] op hist [0,1])) 2017-03-02 15:08:13.462978 7f4f71beb700 15 mon.0@0(leader).pg v12965 got 1.1 reported at 7:586155 state active+clean -> active+clean 2017-03-02 15:08:13.462989 7f4f71beb700 15 mon.0@0(leader).pg v12965 got 1.5 reported at 7:567418 state active+clean -> active+clean 2017-03-02 15:08:13.462993 7f4f71beb700 15 mon.0@0(leader).pg v12965 got 1.6 reported at 7:502712 state active+clean -> active+clean 2017-03-02 15:08:13.462998 7f4f71beb700 15 mon.0@0(leader).pg v12965 got 2.4 reported at 7:1025618 state active+clean -> active+clean 2017-03-02 15:08:13.463002 7f4f71beb700 15 mon.0@0(leader).pg v12965 got 2.5 reported at 7:1128973 state active+clean -> active+clean 2017-03-02 15:08:13.463008 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12965) setting proposal_timer 0x558a1412dc40 with delay of 0.865315 2017-03-02 15:08:13.556325 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.2 172.21.15.11:6800/26536 6831 ==== pg_stats(8 pgs tid 6795 v 0) v1 ==== 4708+0+0 (3008755911 0 0) 0x558a15d9e300 con 0x558a145d5800 2017-03-02 15:08:13.556351 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a145c3700 for osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:13.556356 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:13.556358 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12965) dispatch 0x558a15d9e300 pg_stats(8 pgs tid 6795 v 0) v1 from osd.2 172.21.15.11:6800/26536 con 0x558a145d5800 2017-03-02 15:08:13.556363 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27324) is_readable = 1 - now=2017-03-02 15:08:13.556363 lease_expire=0.000000 has v0 lc 27324 2017-03-02 15:08:13.556369 7f4f71beb700 10 mon.0@0(leader).pg v12965 preprocess_query pg_stats(8 pgs tid 6795 v 0) v1 from osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:13.556374 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:13.556376 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:13.556377 7f4f71beb700 20 allow all 2017-03-02 15:08:13.556378 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27324) is_readable = 1 - now=2017-03-02 15:08:13.556378 lease_expire=0.000000 has v0 lc 27324 2017-03-02 15:08:13.556381 7f4f71beb700 10 mon.0@0(leader).pg v12965 prepare_update pg_stats(8 pgs tid 6795 v 0) v1 from osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:13.556385 7f4f71beb700 10 mon.0@0(leader).pg v12965 prepare_pg_stats pg_stats(8 pgs tid 6795 v 0) v1 from osd.2 2017-03-02 15:08:13.556390 7f4f71beb700 10 mon.0@0(leader).pg v12965 got osd.2 osd_stat(63671 MB used, 31669 MB avail, 95341 MB total, peers [0,1,3] op hist [0,0,3,5,14,11,26,38,0,0,0,3]) (was osd_stat(63661 MB used, 31679 MB avail, 95341 MB total, peers [0,1,3] op hist [])) 2017-03-02 15:08:13.556397 7f4f71beb700 15 mon.0@0(leader).pg v12965 got 1.0 reported at 7:496022 state active+clean -> active+clean 2017-03-02 15:08:13.556404 7f4f71beb700 15 mon.0@0(leader).pg v12965 got 1.2 reported at 7:576839 state active+clean -> active+clean 2017-03-02 15:08:13.556408 7f4f71beb700 15 mon.0@0(leader).pg v12965 got 1.7 reported at 7:529081 state active+clean -> active+clean 2017-03-02 15:08:13.556411 7f4f71beb700 15 mon.0@0(leader).pg v12965 got 2.0 reported at 7:1056222 state active+clean -> active+clean 2017-03-02 15:08:13.556414 7f4f71beb700 15 mon.0@0(leader).pg v12965 got 2.1 reported at 7:1088940 state active+clean -> active+clean 2017-03-02 15:08:13.556416 7f4f71beb700 15 mon.0@0(leader).pg v12965 got 2.2 reported at 7:1162696 state active+clean -> active+clean 2017-03-02 15:08:13.556419 7f4f71beb700 15 mon.0@0(leader).pg v12965 got 2.3 reported at 7:1094812 state active+clean -> active+clean 2017-03-02 15:08:13.556422 7f4f71beb700 15 mon.0@0(leader).pg v12965 got 2.7 reported at 7:1134886 state active+clean -> active+clean 2017-03-02 15:08:13.556425 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12965) proposal_timer already set 2017-03-02 15:08:13.710411 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.0 172.21.15.26:6800/26515 6831 ==== pg_stats(2 pgs tid 6795 v 0) v1 ==== 1264+0+0 (1021383015 0 0) 0x558a15d27e00 con 0x558a145d8800 2017-03-02 15:08:13.710435 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1463af20 for osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:13.710438 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:13.710440 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12965) dispatch 0x558a15d27e00 pg_stats(2 pgs tid 6795 v 0) v1 from osd.0 172.21.15.26:6800/26515 con 0x558a145d8800 2017-03-02 15:08:13.710443 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27324) is_readable = 1 - now=2017-03-02 15:08:13.710443 lease_expire=0.000000 has v0 lc 27324 2017-03-02 15:08:13.710449 7f4f71beb700 10 mon.0@0(leader).pg v12965 preprocess_query pg_stats(2 pgs tid 6795 v 0) v1 from osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:13.710452 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:13.710454 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:13.710454 7f4f71beb700 20 allow all 2017-03-02 15:08:13.710455 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27324) is_readable = 1 - now=2017-03-02 15:08:13.710455 lease_expire=0.000000 has v0 lc 27324 2017-03-02 15:08:13.710458 7f4f71beb700 10 mon.0@0(leader).pg v12965 prepare_update pg_stats(2 pgs tid 6795 v 0) v1 from osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:13.710461 7f4f71beb700 10 mon.0@0(leader).pg v12965 prepare_pg_stats pg_stats(2 pgs tid 6795 v 0) v1 from osd.0 2017-03-02 15:08:13.710464 7f4f71beb700 10 mon.0@0(leader).pg v12965 got osd.0 osd_stat(10938 MB used, 84403 MB avail, 95341 MB total, peers [1,2,3] op hist []) (was osd_stat(10928 MB used, 84413 MB avail, 95341 MB total, peers [1,2,3] op hist [0,0,2,0,0,0,10,25,28])) 2017-03-02 15:08:13.710470 7f4f71beb700 15 mon.0@0(leader).pg v12965 got 1.3 reported at 7:499557 state active+clean -> active+clean 2017-03-02 15:08:13.710476 7f4f71beb700 15 mon.0@0(leader).pg v12965 got 1.4 reported at 7:486005 state active+clean -> active+clean 2017-03-02 15:08:13.710480 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12965) proposal_timer already set 2017-03-02 15:08:14.328463 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13543) propose_pending 2017-03-02 15:08:14.328486 7f4f743f0700 10 mon.0@0(leader).log v13543 encode_full log v 13543 2017-03-02 15:08:14.328581 7f4f743f0700 10 mon.0@0(leader).log v13543 encode_pending v13544 2017-03-02 15:08:14.328599 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27324) queue_pending_finisher 0x558a1412cff0 2017-03-02 15:08:14.328605 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27324) trigger_propose active, proposing now 2017-03-02 15:08:14.328615 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27324) propose_pending 27325 10391 bytes 2017-03-02 15:08:14.328619 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27324) begin for 27325 10391 bytes 2017-03-02 15:08:14.354537 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27324) commit_start 27325 2017-03-02 15:08:14.354598 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12965) propose_pending 2017-03-02 15:08:14.354603 7f4f743f0700 10 mon.0@0(leader).pg v12965 encode_pending v 12966 2017-03-02 15:08:14.354706 7f4f743f0700 5 mon.0@0(leader).paxos(paxos writing c 26607..27324) queue_pending_finisher 0x558a1412efd0 2017-03-02 15:08:14.354719 7f4f743f0700 10 mon.0@0(leader).paxos(paxos writing c 26607..27324) trigger_propose not active, will propose later 2017-03-02 15:08:14.379670 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27324) commit_finish 27325 2017-03-02 15:08:14.379735 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:14.379770 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12965) refresh 2017-03-02 15:08:14.379799 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) refresh 2017-03-02 15:08:14.379870 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:14.379901 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13544) refresh 2017-03-02 15:08:14.379906 7f4f763f4700 10 mon.0@0(leader).log v13544 update_from_paxos 2017-03-02 15:08:14.379909 7f4f763f4700 10 mon.0@0(leader).log v13544 update_from_paxos version 13544 summary v 13543 2017-03-02 15:08:14.379916 7f4f763f4700 10 mon.0@0(leader).log v13544 update_from_paxos latest full 13543 2017-03-02 15:08:14.379930 7f4f763f4700 7 mon.0@0(leader).log v13544 update_from_paxos applying incremental log 13544 2017-03-02 15:08:13.328640 mon.0 172.21.15.17:6789/0 14104 : cluster [INF] pgmap v12965: 32 pgs: 32 active+clean; 88238 MB data, 165 GB used, 206 GB / 372 GB avail; 653 kB/s rd, 13875 kB/s wr, 203 op/s 2017-03-02 15:08:14.379960 7f4f763f4700 20 mon.0@0(leader).log v13544 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:14.379982 7f4f763f4700 15 mon.0@0(leader).log v13544 update_from_paxos logging for 1 channels 2017-03-02 15:08:14.379985 7f4f763f4700 15 mon.0@0(leader).log v13544 update_from_paxos channel 'cluster' logging 202 bytes 2017-03-02 15:08:14.380085 7f4f763f4700 10 mon.0@0(leader).log v13544 check_subs 2017-03-02 15:08:14.380141 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:14.380177 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:14.380181 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:14.380207 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:14.380210 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12965) post_refresh 2017-03-02 15:08:14.380213 7f4f763f4700 10 mon.0@0(leader).pg v12965 post_paxos_update 2017-03-02 15:08:14.380216 7f4f763f4700 10 mon.0@0(leader).pg v12965 check_subs 2017-03-02 15:08:14.380219 7f4f763f4700 20 mon.0@0(leader).pg v12965 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:14.380227 7f4f763f4700 20 mon.0@0(leader).pg v12965 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:14.380231 7f4f763f4700 20 mon.0@0(leader).pg v12965 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:14.380235 7f4f763f4700 20 mon.0@0(leader).pg v12965 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:14.380239 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) post_refresh 2017-03-02 15:08:14.380241 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:14.380243 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13544) post_refresh 2017-03-02 15:08:14.380245 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:14.380247 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:14.380249 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:14.380252 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27325) commit_proposal 2017-03-02 15:08:14.380257 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13544) _active - not active 2017-03-02 15:08:14.380260 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27325) finish_round 2017-03-02 15:08:14.380263 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27325) finish_round waiting_for_acting 2017-03-02 15:08:14.380266 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13544) _active 2017-03-02 15:08:14.380268 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13544) _active creating new pending 2017-03-02 15:08:14.380278 7f4f763f4700 10 mon.0@0(leader).log v13544 create_pending v 13545 2017-03-02 15:08:14.380282 7f4f763f4700 7 mon.0@0(leader).log v13544 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:14.380289 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a15ca8900 0x558a1588f0e0 log(last 14104) v1 2017-03-02 15:08:14.380295 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14104) v1 -- 0x558a1588f0e0 con 0 2017-03-02 15:08:14.380323 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27325) finish_round waiting_for_readable 2017-03-02 15:08:14.380327 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27325) finish_round waiting_for_writeable 2017-03-02 15:08:14.380330 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27325) finish_round done w/ waiters, state active 2017-03-02 15:08:14.380362 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27325) propose_pending 27326 9784 bytes 2017-03-02 15:08:14.380371 7f4f763f4700 10 mon.0@0(leader).paxos(paxos updating c 26607..27325) begin for 27326 9784 bytes 2017-03-02 15:08:14.380443 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14104) v1 ==== 0+0+0 (0 0 0) 0x558a1588f0e0 con 0x558a14482000 2017-03-02 15:08:14.404714 7f4f763f4700 10 mon.0@0(leader).paxos(paxos updating c 26607..27325) commit_start 27326 2017-03-02 15:08:14.404859 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:14.404889 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:14.404894 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:14.404898 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:14.404900 7f4f71beb700 20 allow all 2017-03-02 15:08:14.429873 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27325) commit_finish 27326 2017-03-02 15:08:14.429921 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:14.429946 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12966) refresh 2017-03-02 15:08:14.429950 7f4f763f4700 10 mon.0@0(leader).pg v12965 update_from_paxos read_incremental 2017-03-02 15:08:14.429976 7f4f763f4700 20 mon.0@0(leader).pg v12965 refreshing pg 1.0 7:496022 active+clean 2017-03-02 15:08:14.429988 7f4f763f4700 20 mon.0@0(leader).pg v12965 refreshing pg 1.1 7:586155 active+clean 2017-03-02 15:08:14.429997 7f4f763f4700 20 mon.0@0(leader).pg v12965 refreshing pg 1.2 7:576839 active+clean 2017-03-02 15:08:14.430005 7f4f763f4700 20 mon.0@0(leader).pg v12965 refreshing pg 1.3 7:499557 active+clean 2017-03-02 15:08:14.430012 7f4f763f4700 20 mon.0@0(leader).pg v12965 refreshing pg 1.4 7:486005 active+clean 2017-03-02 15:08:14.430020 7f4f763f4700 20 mon.0@0(leader).pg v12965 refreshing pg 1.5 7:567418 active+clean 2017-03-02 15:08:14.430028 7f4f763f4700 20 mon.0@0(leader).pg v12965 refreshing pg 1.6 7:502712 active+clean 2017-03-02 15:08:14.430035 7f4f763f4700 20 mon.0@0(leader).pg v12965 refreshing pg 1.7 7:529081 active+clean 2017-03-02 15:08:14.430043 7f4f763f4700 20 mon.0@0(leader).pg v12965 refreshing pg 2.0 7:1056222 active+clean 2017-03-02 15:08:14.430051 7f4f763f4700 20 mon.0@0(leader).pg v12965 refreshing pg 2.1 7:1088940 active+clean 2017-03-02 15:08:14.430059 7f4f763f4700 20 mon.0@0(leader).pg v12965 refreshing pg 2.2 7:1162696 active+clean 2017-03-02 15:08:14.430066 7f4f763f4700 20 mon.0@0(leader).pg v12965 refreshing pg 2.3 7:1094812 active+clean 2017-03-02 15:08:14.430073 7f4f763f4700 20 mon.0@0(leader).pg v12965 refreshing pg 2.4 7:1025618 active+clean 2017-03-02 15:08:14.430081 7f4f763f4700 20 mon.0@0(leader).pg v12965 refreshing pg 2.5 7:1128973 active+clean 2017-03-02 15:08:14.430088 7f4f763f4700 20 mon.0@0(leader).pg v12965 refreshing pg 2.7 7:1134886 active+clean 2017-03-02 15:08:14.430091 7f4f763f4700 20 mon.0@0(leader).pg v12965 refreshing osd.0 2017-03-02 15:08:14.430100 7f4f763f4700 20 mon.0@0(leader).pg v12965 refreshing osd.1 2017-03-02 15:08:14.430105 7f4f763f4700 20 mon.0@0(leader).pg v12965 refreshing osd.2 2017-03-02 15:08:14.430118 7f4f763f4700 10 mon.0@0(leader).pg v12966 read_pgmap_meta 2017-03-02 15:08:14.430144 7f4f763f4700 10 mon.0@0(leader).pg v12966 update_logger 2017-03-02 15:08:14.430159 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) refresh 2017-03-02 15:08:14.430177 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:14.430192 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13544) refresh 2017-03-02 15:08:14.430194 7f4f763f4700 10 mon.0@0(leader).log v13544 update_from_paxos 2017-03-02 15:08:14.430196 7f4f763f4700 10 mon.0@0(leader).log v13544 update_from_paxos version 13544 summary v 13544 2017-03-02 15:08:14.430210 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:14.430226 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:14.430228 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:14.430244 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:14.430246 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12966) post_refresh 2017-03-02 15:08:14.430248 7f4f763f4700 10 mon.0@0(leader).pg v12966 post_paxos_update 2017-03-02 15:08:14.430250 7f4f763f4700 10 mon.0@0(leader).pg v12966 check_subs 2017-03-02 15:08:14.430252 7f4f763f4700 20 mon.0@0(leader).pg v12966 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:14.430261 7f4f763f4700 20 mon.0@0(leader).pg v12966 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:14.430264 7f4f763f4700 20 mon.0@0(leader).pg v12966 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:14.430267 7f4f763f4700 20 mon.0@0(leader).pg v12966 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:14.430270 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) post_refresh 2017-03-02 15:08:14.430272 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:14.430273 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13544) post_refresh 2017-03-02 15:08:14.430275 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:14.430276 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:14.430278 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:14.430280 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27326) commit_proposal 2017-03-02 15:08:14.430284 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12966) _active - not active 2017-03-02 15:08:14.430286 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27326) finish_round 2017-03-02 15:08:14.430288 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27326) finish_round waiting_for_acting 2017-03-02 15:08:14.430291 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12966) _active 2017-03-02 15:08:14.430292 7f4f763f4700 7 mon.0@0(leader).paxosservice(pgmap 12269..12966) _active creating new pending 2017-03-02 15:08:14.430300 7f4f763f4700 10 mon.0@0(leader).pg v12966 create_pending v 12967 2017-03-02 15:08:14.430306 7f4f763f4700 7 mon.0@0(leader).pg v12966 _updated_stats for osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:14.430311 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a166aee00 0x558a151aee00 pg_stats_ack(5 pgs tid 6795) v1 2017-03-02 15:08:14.430315 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6800/26531 -- pg_stats_ack(5 pgs tid 6795) v1 -- 0x558a151aee00 con 0 2017-03-02 15:08:14.430363 7f4f763f4700 7 mon.0@0(leader).pg v12966 _updated_stats for osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:14.430372 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a16385000 0x558a1589d800 pg_stats_ack(8 pgs tid 6795) v1 2017-03-02 15:08:14.430376 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6800/26536 -- pg_stats_ack(8 pgs tid 6795) v1 -- 0x558a1589d800 con 0 2017-03-02 15:08:14.430399 7f4f763f4700 7 mon.0@0(leader).pg v12966 _updated_stats for osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:14.430406 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a15ca8f00 0x558a15706e00 pg_stats_ack(2 pgs tid 6795) v1 2017-03-02 15:08:14.430409 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6800/26515 -- pg_stats_ack(2 pgs tid 6795) v1 -- 0x558a15706e00 con 0 2017-03-02 15:08:14.430451 7f4f763f4700 10 mon.0@0(leader).pg v12966 check_osd_map already seen 7 >= 7 2017-03-02 15:08:14.430458 7f4f763f4700 10 mon.0@0(leader).pg v12966 update_logger 2017-03-02 15:08:14.430485 7f4f763f4700 0 log_channel(cluster) log [INF] : pgmap v12966: 32 pgs: 32 active+clean; 88257 MB data, 165 GB used, 206 GB / 372 GB avail; 816 kB/s rd, 21448 kB/s wr, 516 op/s 2017-03-02 15:08:14.430511 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14105 at 2017-03-02 15:08:14.430488) v1 -- 0x558a1456f3c0 con 0 2017-03-02 15:08:14.430539 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27326) finish_round waiting_for_readable 2017-03-02 15:08:14.430545 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27326) finish_round waiting_for_writeable 2017-03-02 15:08:14.430562 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27326) finish_round done w/ waiters, state active 2017-03-02 15:08:14.430603 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14105 at 2017-03-02 15:08:14.430488) v1 ==== 0+0+0 (0 0 0) 0x558a1456f3c0 con 0x558a14482000 2017-03-02 15:08:14.430644 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:14.430654 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:14.430659 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13544) dispatch 0x558a1456f3c0 log(1 entries from seq 14105 at 2017-03-02 15:08:14.430488) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:14.430675 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27326) is_readable = 1 - now=2017-03-02 15:08:14.430677 lease_expire=0.000000 has v0 lc 27326 2017-03-02 15:08:14.430690 7f4f71beb700 10 mon.0@0(leader).log v13544 preprocess_query log(1 entries from seq 14105 at 2017-03-02 15:08:14.430488) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:14.430702 7f4f71beb700 10 mon.0@0(leader).log v13544 preprocess_log log(1 entries from seq 14105 at 2017-03-02 15:08:14.430488) v1 from mon.0 2017-03-02 15:08:14.430708 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:14.430711 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:14.430712 7f4f71beb700 20 allow all 2017-03-02 15:08:14.430720 7f4f71beb700 10 mon.0@0(leader).log v13544 prepare_update log(1 entries from seq 14105 at 2017-03-02 15:08:14.430488) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:14.430728 7f4f71beb700 10 mon.0@0(leader).log v13544 prepare_log log(1 entries from seq 14105 at 2017-03-02 15:08:14.430488) v1 from mon.0 2017-03-02 15:08:14.430732 7f4f71beb700 10 mon.0@0(leader).log v13544 logging 2017-03-02 15:08:14.430488 mon.0 172.21.15.17:6789/0 14105 : cluster [INF] pgmap v12966: 32 pgs: 32 active+clean; 88257 MB data, 165 GB used, 206 GB / 372 GB avail; 816 kB/s rd, 21448 kB/s wr, 516 op/s 2017-03-02 15:08:14.430743 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13544) setting proposal_timer 0x558a1412da40 with delay of 0.999149 2017-03-02 15:08:14.870127 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.5 172.21.15.11:6804/1637448123 8782 ==== mdsbeacon(4116/e up:active seq 8495 v733) v7 ==== 365+0+0 (241478148 0 0) 0x558a143e7a80 con 0x558a1471d800 2017-03-02 15:08:14.870188 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b0f20 for mds.? 172.21.15.11:6804/1637448123 2017-03-02 15:08:14.870203 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:14.870210 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) dispatch 0x558a143e7a80 mdsbeacon(4116/e up:active seq 8495 v733) v7 from mds.5 172.21.15.11:6804/1637448123 con 0x558a1471d800 2017-03-02 15:08:14.870222 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27326) is_readable = 1 - now=2017-03-02 15:08:14.870222 lease_expire=0.000000 has v0 lc 27326 2017-03-02 15:08:14.870235 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_query mdsbeacon(4116/e up:active seq 8495 v733) v7 from mds.5 172.21.15.11:6804/1637448123 2017-03-02 15:08:14.870246 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:14.870250 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:14.870251 7f4f71beb700 20 allow all 2017-03-02 15:08:14.870253 7f4f71beb700 12 mon.0@0(leader).mds e733 preprocess_beacon mdsbeacon(4116/e up:active seq 8495 v733) v7 from mds.5 172.21.15.11:6804/1637448123 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:14.870265 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_beacon: GID exists in map: 4116 2017-03-02 15:08:14.870272 7f4f71beb700 15 mon.0@0(leader).mds e733 _note_beacon mdsbeacon(4116/e up:active seq 8495 v733) v7 noting time 2017-03-02 15:08:14.870277 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a145da100 0x558a15204000 mdsbeacon(4116/e up:active seq 8495 v733) v7 2017-03-02 15:08:14.870282 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6804/1637448123 -- mdsbeacon(4116/e up:active seq 8495 v733) v7 -- 0x558a15204000 con 0 2017-03-02 15:08:14.883959 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.3 172.21.15.11:6805/1926402648 8558 ==== mdsbeacon(4118/f up:active seq 8495 v733) v7 ==== 406+0+0 (3527289393 0 0) 0x558a148d7380 con 0x558a14905800 2017-03-02 15:08:14.884003 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b2260 for mds.? 172.21.15.11:6805/1926402648 2017-03-02 15:08:14.884012 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:14.884016 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) dispatch 0x558a148d7380 mdsbeacon(4118/f up:active seq 8495 v733) v7 from mds.3 172.21.15.11:6805/1926402648 con 0x558a14905800 2017-03-02 15:08:14.884024 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27326) is_readable = 1 - now=2017-03-02 15:08:14.884025 lease_expire=0.000000 has v0 lc 27326 2017-03-02 15:08:14.884036 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_query mdsbeacon(4118/f up:active seq 8495 v733) v7 from mds.3 172.21.15.11:6805/1926402648 2017-03-02 15:08:14.884045 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:14.884048 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:14.884050 7f4f71beb700 20 allow all 2017-03-02 15:08:14.884052 7f4f71beb700 12 mon.0@0(leader).mds e733 preprocess_beacon mdsbeacon(4118/f up:active seq 8495 v733) v7 from mds.3 172.21.15.11:6805/1926402648 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:14.884062 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_beacon: GID exists in map: 4118 2017-03-02 15:08:14.884068 7f4f71beb700 15 mon.0@0(leader).mds e733 _note_beacon mdsbeacon(4118/f up:active seq 8495 v733) v7 noting time 2017-03-02 15:08:14.884074 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a145da100 0x558a143e7a80 mdsbeacon(4118/f up:active seq 8495 v733) v7 2017-03-02 15:08:14.884078 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6805/1926402648 -- mdsbeacon(4118/f up:active seq 8495 v733) v7 -- 0x558a143e7a80 con 0 2017-03-02 15:08:14.884815 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.2 172.21.15.22:6804/673088617 8557 ==== mdsbeacon(4119/c up:active seq 8495 v733) v7 ==== 365+0+0 (779988570 0 0) 0x558a15204000 con 0x558a14907000 2017-03-02 15:08:14.884902 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14916f20 for mds.? 172.21.15.22:6804/673088617 2017-03-02 15:08:14.884910 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:14.884914 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) dispatch 0x558a15204000 mdsbeacon(4119/c up:active seq 8495 v733) v7 from mds.2 172.21.15.22:6804/673088617 con 0x558a14907000 2017-03-02 15:08:14.884927 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27326) is_readable = 1 - now=2017-03-02 15:08:14.884927 lease_expire=0.000000 has v0 lc 27326 2017-03-02 15:08:14.884939 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_query mdsbeacon(4119/c up:active seq 8495 v733) v7 from mds.2 172.21.15.22:6804/673088617 2017-03-02 15:08:14.884948 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:14.884951 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:14.884963 7f4f71beb700 20 allow all 2017-03-02 15:08:14.884966 7f4f71beb700 12 mon.0@0(leader).mds e733 preprocess_beacon mdsbeacon(4119/c up:active seq 8495 v733) v7 from mds.2 172.21.15.22:6804/673088617 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:14.884977 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_beacon: GID exists in map: 4119 2017-03-02 15:08:14.884983 7f4f71beb700 15 mon.0@0(leader).mds e733 _note_beacon mdsbeacon(4119/c up:active seq 8495 v733) v7 noting time 2017-03-02 15:08:14.884988 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a145da100 0x558a148d7380 mdsbeacon(4119/c up:active seq 8495 v733) v7 2017-03-02 15:08:14.884993 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6804/673088617 -- mdsbeacon(4119/c up:active seq 8495 v733) v7 -- 0x558a148d7380 con 0 2017-03-02 15:08:14.885039 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.0 172.21.15.22:6805/2203629154 8607 ==== mdsbeacon(4121/d up:active seq 8491 v733) v7 ==== 452+0+0 (2147180556 0 0) 0x558a143e7a80 con 0x558a1492a000 2017-03-02 15:08:14.885053 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a149195a0 for mds.? 172.21.15.22:6805/2203629154 2017-03-02 15:08:14.885058 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:14.885061 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) dispatch 0x558a143e7a80 mdsbeacon(4121/d up:active seq 8491 v733) v7 from mds.0 172.21.15.22:6805/2203629154 con 0x558a1492a000 2017-03-02 15:08:14.885068 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27326) is_readable = 1 - now=2017-03-02 15:08:14.885069 lease_expire=0.000000 has v0 lc 27326 2017-03-02 15:08:14.885078 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_query mdsbeacon(4121/d up:active seq 8491 v733) v7 from mds.0 172.21.15.22:6805/2203629154 2017-03-02 15:08:14.885085 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:14.885088 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:14.885089 7f4f71beb700 20 allow all 2017-03-02 15:08:14.885091 7f4f71beb700 12 mon.0@0(leader).mds e733 preprocess_beacon mdsbeacon(4121/d up:active seq 8491 v733) v7 from mds.0 172.21.15.22:6805/2203629154 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:14.885100 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_beacon: GID exists in map: 4121 2017-03-02 15:08:14.885106 7f4f71beb700 15 mon.0@0(leader).mds e733 _note_beacon mdsbeacon(4121/d up:active seq 8491 v733) v7 noting time 2017-03-02 15:08:14.885110 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a145da100 0x558a15204000 mdsbeacon(4121/d up:active seq 8491 v733) v7 2017-03-02 15:08:14.885113 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6805/2203629154 -- mdsbeacon(4121/d up:active seq 8491 v733) v7 -- 0x558a15204000 con 0 2017-03-02 15:08:14.924170 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.1 172.21.15.17:6804/2576824570 8554 ==== mdsbeacon(4120/g up:active seq 8495 v733) v7 ==== 365+0+0 (3866288961 0 0) 0x558a154b8d00 con 0x558a14908800 2017-03-02 15:08:14.924210 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14917600 for mds.? 172.21.15.17:6804/2576824570 2017-03-02 15:08:14.924218 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:14.924222 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) dispatch 0x558a154b8d00 mdsbeacon(4120/g up:active seq 8495 v733) v7 from mds.1 172.21.15.17:6804/2576824570 con 0x558a14908800 2017-03-02 15:08:14.924230 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27326) is_readable = 1 - now=2017-03-02 15:08:14.924231 lease_expire=0.000000 has v0 lc 27326 2017-03-02 15:08:14.924242 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_query mdsbeacon(4120/g up:active seq 8495 v733) v7 from mds.1 172.21.15.17:6804/2576824570 2017-03-02 15:08:14.924251 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:14.924254 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:14.924256 7f4f71beb700 20 allow all 2017-03-02 15:08:14.924258 7f4f71beb700 12 mon.0@0(leader).mds e733 preprocess_beacon mdsbeacon(4120/g up:active seq 8495 v733) v7 from mds.1 172.21.15.17:6804/2576824570 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:14.924269 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_beacon: GID exists in map: 4120 2017-03-02 15:08:14.924274 7f4f71beb700 15 mon.0@0(leader).mds e733 _note_beacon mdsbeacon(4120/g up:active seq 8495 v733) v7 noting time 2017-03-02 15:08:14.924279 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a145da100 0x558a143e7a80 mdsbeacon(4120/g up:active seq 8495 v733) v7 2017-03-02 15:08:14.924283 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6804/2576824570 -- mdsbeacon(4120/g up:active seq 8495 v733) v7 -- 0x558a143e7a80 con 0 2017-03-02 15:08:14.969142 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.4 172.21.15.26:6805/2033388711 8552 ==== mdsbeacon(4117/b up:active seq 8495 v733) v7 ==== 365+0+0 (417070466 0 0) 0x558a143e7a80 con 0x558a14904000 2017-03-02 15:08:14.969195 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b1600 for mds.? 172.21.15.26:6805/2033388711 2017-03-02 15:08:14.969209 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:14.969214 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) dispatch 0x558a143e7a80 mdsbeacon(4117/b up:active seq 8495 v733) v7 from mds.4 172.21.15.26:6805/2033388711 con 0x558a14904000 2017-03-02 15:08:14.969227 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27326) is_readable = 1 - now=2017-03-02 15:08:14.969229 lease_expire=0.000000 has v0 lc 27326 2017-03-02 15:08:14.969242 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_query mdsbeacon(4117/b up:active seq 8495 v733) v7 from mds.4 172.21.15.26:6805/2033388711 2017-03-02 15:08:14.969251 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:14.969254 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:14.969256 7f4f71beb700 20 allow all 2017-03-02 15:08:14.969258 7f4f71beb700 12 mon.0@0(leader).mds e733 preprocess_beacon mdsbeacon(4117/b up:active seq 8495 v733) v7 from mds.4 172.21.15.26:6805/2033388711 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:14.969268 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_beacon: GID exists in map: 4117 2017-03-02 15:08:14.969273 7f4f71beb700 15 mon.0@0(leader).mds e733 _note_beacon mdsbeacon(4117/b up:active seq 8495 v733) v7 noting time 2017-03-02 15:08:14.969279 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a145da100 0x558a154b8d00 mdsbeacon(4117/b up:active seq 8495 v733) v7 2017-03-02 15:08:14.969285 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6805/2033388711 -- mdsbeacon(4117/b up:active seq 8495 v733) v7 -- 0x558a154b8d00 con 0 2017-03-02 15:08:14.978259 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.6 172.21.15.26:6804/2954782670 8744 ==== mdsbeacon(4115/a up:active seq 8495 v733) v7 ==== 365+0+0 (1127287274 0 0) 0x558a15204000 con 0x558a1471f000 2017-03-02 15:08:14.978303 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14868d60 for mds.? 172.21.15.26:6804/2954782670 2017-03-02 15:08:14.978311 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:14.978315 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) dispatch 0x558a15204000 mdsbeacon(4115/a up:active seq 8495 v733) v7 from mds.6 172.21.15.26:6804/2954782670 con 0x558a1471f000 2017-03-02 15:08:14.978324 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27326) is_readable = 1 - now=2017-03-02 15:08:14.978324 lease_expire=0.000000 has v0 lc 27326 2017-03-02 15:08:14.978336 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_query mdsbeacon(4115/a up:active seq 8495 v733) v7 from mds.6 172.21.15.26:6804/2954782670 2017-03-02 15:08:14.978345 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:14.978348 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:14.978349 7f4f71beb700 20 allow all 2017-03-02 15:08:14.978351 7f4f71beb700 12 mon.0@0(leader).mds e733 preprocess_beacon mdsbeacon(4115/a up:active seq 8495 v733) v7 from mds.6 172.21.15.26:6804/2954782670 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:14.978362 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_beacon: GID exists in map: 4115 2017-03-02 15:08:14.978367 7f4f71beb700 15 mon.0@0(leader).mds e733 _note_beacon mdsbeacon(4115/a up:active seq 8495 v733) v7 noting time 2017-03-02 15:08:14.978372 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a145da100 0x558a143e7a80 mdsbeacon(4115/a up:active seq 8495 v733) v7 2017-03-02 15:08:14.978376 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6804/2954782670 -- mdsbeacon(4115/a up:active seq 8495 v733) v7 -- 0x558a143e7a80 con 0 2017-03-02 15:08:15.430056 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13544) propose_pending 2017-03-02 15:08:15.430078 7f4f743f0700 10 mon.0@0(leader).log v13544 encode_full log v 13544 2017-03-02 15:08:15.430173 7f4f743f0700 10 mon.0@0(leader).log v13544 encode_pending v13545 2017-03-02 15:08:15.430183 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27326) queue_pending_finisher 0x558a1412dc40 2017-03-02 15:08:15.430188 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27326) trigger_propose active, proposing now 2017-03-02 15:08:15.430197 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27326) propose_pending 27327 10389 bytes 2017-03-02 15:08:15.430201 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27326) begin for 27327 10389 bytes 2017-03-02 15:08:15.454953 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27326) commit_start 27327 2017-03-02 15:08:15.488366 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27326) commit_finish 27327 2017-03-02 15:08:15.488427 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:15.488462 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12966) refresh 2017-03-02 15:08:15.488481 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) refresh 2017-03-02 15:08:15.488504 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:15.488538 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13545) refresh 2017-03-02 15:08:15.488541 7f4f763f4700 10 mon.0@0(leader).log v13545 update_from_paxos 2017-03-02 15:08:15.488544 7f4f763f4700 10 mon.0@0(leader).log v13545 update_from_paxos version 13545 summary v 13544 2017-03-02 15:08:15.488551 7f4f763f4700 10 mon.0@0(leader).log v13545 update_from_paxos latest full 13544 2017-03-02 15:08:15.488564 7f4f763f4700 7 mon.0@0(leader).log v13545 update_from_paxos applying incremental log 13545 2017-03-02 15:08:14.430488 mon.0 172.21.15.17:6789/0 14105 : cluster [INF] pgmap v12966: 32 pgs: 32 active+clean; 88257 MB data, 165 GB used, 206 GB / 372 GB avail; 816 kB/s rd, 21448 kB/s wr, 516 op/s 2017-03-02 15:08:15.488586 7f4f763f4700 20 mon.0@0(leader).log v13545 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:15.488606 7f4f763f4700 15 mon.0@0(leader).log v13545 update_from_paxos logging for 1 channels 2017-03-02 15:08:15.488609 7f4f763f4700 15 mon.0@0(leader).log v13545 update_from_paxos channel 'cluster' logging 202 bytes 2017-03-02 15:08:15.488715 7f4f763f4700 10 mon.0@0(leader).log v13545 check_subs 2017-03-02 15:08:15.488781 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:15.488880 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:15.488895 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:15.488919 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:15.488922 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12966) post_refresh 2017-03-02 15:08:15.488925 7f4f763f4700 10 mon.0@0(leader).pg v12966 post_paxos_update 2017-03-02 15:08:15.488928 7f4f763f4700 10 mon.0@0(leader).pg v12966 check_subs 2017-03-02 15:08:15.488932 7f4f763f4700 20 mon.0@0(leader).pg v12966 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:15.488941 7f4f763f4700 20 mon.0@0(leader).pg v12966 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:15.488946 7f4f763f4700 20 mon.0@0(leader).pg v12966 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:15.488949 7f4f763f4700 20 mon.0@0(leader).pg v12966 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:15.488953 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) post_refresh 2017-03-02 15:08:15.488956 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:15.488958 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13545) post_refresh 2017-03-02 15:08:15.488961 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:15.488962 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:15.488964 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:15.488968 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27327) commit_proposal 2017-03-02 15:08:15.488972 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13545) _active - not active 2017-03-02 15:08:15.488976 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27327) finish_round 2017-03-02 15:08:15.488978 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27327) finish_round waiting_for_acting 2017-03-02 15:08:15.488982 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13545) _active 2017-03-02 15:08:15.488984 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13545) _active creating new pending 2017-03-02 15:08:15.488994 7f4f763f4700 10 mon.0@0(leader).log v13545 create_pending v 13546 2017-03-02 15:08:15.488998 7f4f763f4700 7 mon.0@0(leader).log v13545 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:15.489005 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a14dbf200 0x558a150a3a20 log(last 14105) v1 2017-03-02 15:08:15.489010 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14105) v1 -- 0x558a150a3a20 con 0 2017-03-02 15:08:15.489037 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27327) finish_round waiting_for_readable 2017-03-02 15:08:15.489041 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27327) finish_round waiting_for_writeable 2017-03-02 15:08:15.489044 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27327) finish_round done w/ waiters, state active 2017-03-02 15:08:15.489116 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14105) v1 ==== 0+0+0 (0 0 0) 0x558a150a3a20 con 0x558a14482000 2017-03-02 15:08:15.489144 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:15.489151 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:15.489155 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:15.489158 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:15.489160 7f4f71beb700 20 allow all 2017-03-02 15:08:16.011333 7f4f743f0700 11 mon.0@0(leader) e2 tick 2017-03-02 15:08:16.011357 7f4f743f0700 10 mon.0@0(leader).pg v12966 v12966: 32 pgs: 32 active+clean; 88257 MB data, 165 GB used, 206 GB / 372 GB avail; 816 kB/s rd, 21448 kB/s wr, 516 op/s 2017-03-02 15:08:16.011377 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12966) maybe_trim trim_to 12466 would only trim 197 < paxos_service_trim_min 250 2017-03-02 15:08:16.011382 7f4f743f0700 10 mon.0@0(leader).mds e733 e733: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:16.011418 7f4f743f0700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) maybe_trim trim_to 233 would only trim 232 < paxos_service_trim_min 250 2017-03-02 15:08:16.011422 7f4f743f0700 10 mon.0@0(leader).osd e7 e7: 4 osds: 4 up, 4 in 2017-03-02 15:08:16.011427 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27327) is_readable = 1 - now=2017-03-02 15:08:16.011428 lease_expire=0.000000 has v0 lc 27327 2017-03-02 15:08:16.011437 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27327) is_readable = 1 - now=2017-03-02 15:08:16.011438 lease_expire=0.000000 has v0 lc 27327 2017-03-02 15:08:16.011445 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27327) is_readable = 1 - now=2017-03-02 15:08:16.011446 lease_expire=0.000000 has v0 lc 27327 2017-03-02 15:08:16.011450 7f4f743f0700 10 mon.0@0(leader).osd e7 min_last_epoch_clean 6 2017-03-02 15:08:16.011452 7f4f743f0700 10 mon.0@0(leader).log v13545 log 2017-03-02 15:08:16.011455 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13545) maybe_trim trim_to 13045 would only trim 18 < paxos_service_trim_min 250 2017-03-02 15:08:16.011459 7f4f743f0700 10 mon.0@0(leader).auth v10 auth 2017-03-02 15:08:16.011462 7f4f743f0700 10 cephx keyserver: _check_rotating_secrets 2017-03-02 15:08:16.011472 7f4f743f0700 20 mon.0@0(leader) e2 sync_trim_providers 2017-03-02 15:08:18.261588 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.3 172.21.15.17:6800/27100 6832 ==== pg_stats(1 pgs tid 6796 v 0) v1 ==== 698+0+0 (416384869 0 0) 0x558a15ae3b80 con 0x558a145d7000 2017-03-02 15:08:18.261658 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415f440 for osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:18.261673 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:18.261679 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12966) dispatch 0x558a15ae3b80 pg_stats(1 pgs tid 6796 v 0) v1 from osd.3 172.21.15.17:6800/27100 con 0x558a145d7000 2017-03-02 15:08:18.261691 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27327) is_readable = 1 - now=2017-03-02 15:08:18.261692 lease_expire=0.000000 has v0 lc 27327 2017-03-02 15:08:18.261704 7f4f71beb700 10 mon.0@0(leader).pg v12966 preprocess_query pg_stats(1 pgs tid 6796 v 0) v1 from osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:18.261713 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:18.261717 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:18.261719 7f4f71beb700 20 allow all 2017-03-02 15:08:18.261721 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27327) is_readable = 1 - now=2017-03-02 15:08:18.261721 lease_expire=0.000000 has v0 lc 27327 2017-03-02 15:08:18.261727 7f4f71beb700 10 mon.0@0(leader).pg v12966 prepare_update pg_stats(1 pgs tid 6796 v 0) v1 from osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:18.261733 7f4f71beb700 10 mon.0@0(leader).pg v12966 prepare_pg_stats pg_stats(1 pgs tid 6796 v 0) v1 from osd.3 2017-03-02 15:08:18.261742 7f4f71beb700 10 mon.0@0(leader).pg v12966 got osd.3 osd_stat(42082 MB used, 53259 MB avail, 95341 MB total, peers [0,1,2] op hist []) (was osd_stat(42071 MB used, 53269 MB avail, 95341 MB total, peers [0,1,2] op hist [2,0,1])) 2017-03-02 15:08:18.261754 7f4f71beb700 15 mon.0@0(leader).pg v12966 got 2.6 reported at 7:963876 state active+clean -> active+clean 2017-03-02 15:08:18.261768 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12966) setting proposal_timer 0x558a1412ed40 with delay of 0.05 2017-03-02 15:08:18.311878 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12966) propose_pending 2017-03-02 15:08:18.311909 7f4f743f0700 10 mon.0@0(leader).pg v12966 encode_pending v 12967 2017-03-02 15:08:18.311969 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27327) queue_pending_finisher 0x558a1412e9d0 2017-03-02 15:08:18.311976 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27327) trigger_propose active, proposing now 2017-03-02 15:08:18.311993 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27327) propose_pending 27328 1090 bytes 2017-03-02 15:08:18.311998 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27327) begin for 27328 1090 bytes 2017-03-02 15:08:18.338060 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27327) commit_start 27328 2017-03-02 15:08:18.353012 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27327) commit_finish 27328 2017-03-02 15:08:18.353086 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:18.353123 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12967) refresh 2017-03-02 15:08:18.353143 7f4f763f4700 10 mon.0@0(leader).pg v12966 update_from_paxos read_incremental 2017-03-02 15:08:18.353191 7f4f763f4700 20 mon.0@0(leader).pg v12966 refreshing pg 2.6 7:963876 active+clean 2017-03-02 15:08:18.353209 7f4f763f4700 20 mon.0@0(leader).pg v12966 refreshing osd.3 2017-03-02 15:08:18.353231 7f4f763f4700 10 mon.0@0(leader).pg v12967 read_pgmap_meta 2017-03-02 15:08:18.353266 7f4f763f4700 10 mon.0@0(leader).pg v12967 update_logger 2017-03-02 15:08:18.353298 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) refresh 2017-03-02 15:08:18.353323 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:18.353404 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13545) refresh 2017-03-02 15:08:18.353419 7f4f763f4700 10 mon.0@0(leader).log v13545 update_from_paxos 2017-03-02 15:08:18.353423 7f4f763f4700 10 mon.0@0(leader).log v13545 update_from_paxos version 13545 summary v 13545 2017-03-02 15:08:18.353448 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:18.353470 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:18.353474 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:18.353495 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:18.353498 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12967) post_refresh 2017-03-02 15:08:18.353501 7f4f763f4700 10 mon.0@0(leader).pg v12967 post_paxos_update 2017-03-02 15:08:18.353504 7f4f763f4700 10 mon.0@0(leader).pg v12967 check_subs 2017-03-02 15:08:18.353508 7f4f763f4700 20 mon.0@0(leader).pg v12967 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:18.353517 7f4f763f4700 20 mon.0@0(leader).pg v12967 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:18.353520 7f4f763f4700 20 mon.0@0(leader).pg v12967 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:18.353529 7f4f763f4700 20 mon.0@0(leader).pg v12967 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:18.353532 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) post_refresh 2017-03-02 15:08:18.353533 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:18.353535 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13545) post_refresh 2017-03-02 15:08:18.353536 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:18.353537 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:18.353538 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:18.353541 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27328) commit_proposal 2017-03-02 15:08:18.353544 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12967) _active - not active 2017-03-02 15:08:18.353546 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27328) finish_round 2017-03-02 15:08:18.353548 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27328) finish_round waiting_for_acting 2017-03-02 15:08:18.353550 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12967) _active 2017-03-02 15:08:18.353552 7f4f763f4700 7 mon.0@0(leader).paxosservice(pgmap 12269..12967) _active creating new pending 2017-03-02 15:08:18.353556 7f4f763f4700 10 mon.0@0(leader).pg v12967 create_pending v 12968 2017-03-02 15:08:18.353561 7f4f763f4700 7 mon.0@0(leader).pg v12967 _updated_stats for osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:18.353566 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a145da100 0x558a153f8600 pg_stats_ack(1 pgs tid 6796) v1 2017-03-02 15:08:18.353571 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6800/27100 -- pg_stats_ack(1 pgs tid 6796) v1 -- 0x558a153f8600 con 0 2017-03-02 15:08:18.353621 7f4f763f4700 10 mon.0@0(leader).pg v12967 check_osd_map already seen 7 >= 7 2017-03-02 15:08:18.353627 7f4f763f4700 10 mon.0@0(leader).pg v12967 update_logger 2017-03-02 15:08:18.353647 7f4f763f4700 0 log_channel(cluster) log [INF] : pgmap v12967: 32 pgs: 32 active+clean; 88257 MB data, 165 GB used, 206 GB / 372 GB avail; 258 kB/s rd, 17296 kB/s wr, 336 op/s 2017-03-02 15:08:18.353663 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14106 at 2017-03-02 15:08:18.353649) v1 -- 0x558a1691eb40 con 0 2017-03-02 15:08:18.353683 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27328) finish_round waiting_for_readable 2017-03-02 15:08:18.353688 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27328) finish_round waiting_for_writeable 2017-03-02 15:08:18.353690 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27328) finish_round done w/ waiters, state active 2017-03-02 15:08:18.353726 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14106 at 2017-03-02 15:08:18.353649) v1 ==== 0+0+0 (0 0 0) 0x558a1691eb40 con 0x558a14482000 2017-03-02 15:08:18.353759 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:18.353767 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:18.353771 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13545) dispatch 0x558a1691eb40 log(1 entries from seq 14106 at 2017-03-02 15:08:18.353649) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:18.353783 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27328) is_readable = 1 - now=2017-03-02 15:08:18.353784 lease_expire=0.000000 has v0 lc 27328 2017-03-02 15:08:18.353793 7f4f71beb700 10 mon.0@0(leader).log v13545 preprocess_query log(1 entries from seq 14106 at 2017-03-02 15:08:18.353649) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:18.353803 7f4f71beb700 10 mon.0@0(leader).log v13545 preprocess_log log(1 entries from seq 14106 at 2017-03-02 15:08:18.353649) v1 from mon.0 2017-03-02 15:08:18.353810 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:18.353814 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:18.353816 7f4f71beb700 20 allow all 2017-03-02 15:08:18.353856 7f4f71beb700 10 mon.0@0(leader).log v13545 prepare_update log(1 entries from seq 14106 at 2017-03-02 15:08:18.353649) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:18.353868 7f4f71beb700 10 mon.0@0(leader).log v13545 prepare_log log(1 entries from seq 14106 at 2017-03-02 15:08:18.353649) v1 from mon.0 2017-03-02 15:08:18.353873 7f4f71beb700 10 mon.0@0(leader).log v13545 logging 2017-03-02 15:08:18.353649 mon.0 172.21.15.17:6789/0 14106 : cluster [INF] pgmap v12967: 32 pgs: 32 active+clean; 88257 MB data, 165 GB used, 206 GB / 372 GB avail; 258 kB/s rd, 17296 kB/s wr, 336 op/s 2017-03-02 15:08:18.353886 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13545) setting proposal_timer 0x558a1412f8a0 with delay of 0.999155 2017-03-02 15:08:18.463548 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.1 172.21.15.22:6800/26531 6832 ==== pg_stats(5 pgs tid 6796 v 0) v1 ==== 2962+0+0 (669769610 0 0) 0x558a15da3c00 con 0x558a14643800 2017-03-02 15:08:18.463585 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1463b1e0 for osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:18.463592 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:18.463597 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12967) dispatch 0x558a15da3c00 pg_stats(5 pgs tid 6796 v 0) v1 from osd.1 172.21.15.22:6800/26531 con 0x558a14643800 2017-03-02 15:08:18.463603 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27328) is_readable = 1 - now=2017-03-02 15:08:18.463603 lease_expire=0.000000 has v0 lc 27328 2017-03-02 15:08:18.463612 7f4f71beb700 10 mon.0@0(leader).pg v12967 preprocess_query pg_stats(5 pgs tid 6796 v 0) v1 from osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:18.463619 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:18.463623 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:18.463624 7f4f71beb700 20 allow all 2017-03-02 15:08:18.463626 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27328) is_readable = 1 - now=2017-03-02 15:08:18.463626 lease_expire=0.000000 has v0 lc 27328 2017-03-02 15:08:18.463632 7f4f71beb700 10 mon.0@0(leader).pg v12967 prepare_update pg_stats(5 pgs tid 6796 v 0) v1 from osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:18.463638 7f4f71beb700 10 mon.0@0(leader).pg v12967 prepare_pg_stats pg_stats(5 pgs tid 6796 v 0) v1 from osd.1 2017-03-02 15:08:18.463646 7f4f71beb700 10 mon.0@0(leader).pg v12967 got osd.1 osd_stat(53047 MB used, 42294 MB avail, 95341 MB total, peers [0,2,3] op hist []) (was osd_stat(52994 MB used, 42347 MB avail, 95341 MB total, peers [0,2,3] op hist [])) 2017-03-02 15:08:18.463656 7f4f71beb700 15 mon.0@0(leader).pg v12967 got 1.1 reported at 7:586163 state active+clean -> active+clean 2017-03-02 15:08:18.463668 7f4f71beb700 15 mon.0@0(leader).pg v12967 got 1.5 reported at 7:567431 state active+clean -> active+clean 2017-03-02 15:08:18.463674 7f4f71beb700 15 mon.0@0(leader).pg v12967 got 1.6 reported at 7:502722 state active+clean -> active+clean 2017-03-02 15:08:18.463679 7f4f71beb700 15 mon.0@0(leader).pg v12967 got 2.4 reported at 7:1025619 state active+clean -> active+clean 2017-03-02 15:08:18.463684 7f4f71beb700 15 mon.0@0(leader).pg v12967 got 2.5 reported at 7:1128976 state active+clean -> active+clean 2017-03-02 15:08:18.463690 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12967) setting proposal_timer 0x558a1412d740 with delay of 0.889351 2017-03-02 15:08:18.557093 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.2 172.21.15.11:6800/26536 6832 ==== pg_stats(8 pgs tid 6796 v 0) v1 ==== 4660+0+0 (3162198393 0 0) 0x558a15ca2f00 con 0x558a145d5800 2017-03-02 15:08:18.557120 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a145c3700 for osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:18.557125 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:18.557127 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12967) dispatch 0x558a15ca2f00 pg_stats(8 pgs tid 6796 v 0) v1 from osd.2 172.21.15.11:6800/26536 con 0x558a145d5800 2017-03-02 15:08:18.557132 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27328) is_readable = 1 - now=2017-03-02 15:08:18.557132 lease_expire=0.000000 has v0 lc 27328 2017-03-02 15:08:18.557138 7f4f71beb700 10 mon.0@0(leader).pg v12967 preprocess_query pg_stats(8 pgs tid 6796 v 0) v1 from osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:18.557143 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:18.557145 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:18.557146 7f4f71beb700 20 allow all 2017-03-02 15:08:18.557147 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27328) is_readable = 1 - now=2017-03-02 15:08:18.557147 lease_expire=0.000000 has v0 lc 27328 2017-03-02 15:08:18.557150 7f4f71beb700 10 mon.0@0(leader).pg v12967 prepare_update pg_stats(8 pgs tid 6796 v 0) v1 from osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:18.557152 7f4f71beb700 10 mon.0@0(leader).pg v12967 prepare_pg_stats pg_stats(8 pgs tid 6796 v 0) v1 from osd.2 2017-03-02 15:08:18.557158 7f4f71beb700 10 mon.0@0(leader).pg v12967 got osd.2 osd_stat(63687 MB used, 31654 MB avail, 95341 MB total, peers [0,1,3] op hist []) (was osd_stat(63671 MB used, 31669 MB avail, 95341 MB total, peers [0,1,3] op hist [0,0,3,5,14,11,26,38,0,0,0,3])) 2017-03-02 15:08:18.557165 7f4f71beb700 15 mon.0@0(leader).pg v12967 got 1.0 reported at 7:496025 state active+clean -> active+clean 2017-03-02 15:08:18.557172 7f4f71beb700 15 mon.0@0(leader).pg v12967 got 1.2 reported at 7:576865 state active+clean -> active+clean 2017-03-02 15:08:18.557176 7f4f71beb700 15 mon.0@0(leader).pg v12967 got 1.7 reported at 7:529136 state active+clean -> active+clean 2017-03-02 15:08:18.557179 7f4f71beb700 15 mon.0@0(leader).pg v12967 got 2.0 reported at 7:1056224 state active+clean -> active+clean 2017-03-02 15:08:18.557181 7f4f71beb700 15 mon.0@0(leader).pg v12967 got 2.1 reported at 7:1088945 state active+clean -> active+clean 2017-03-02 15:08:18.557186 7f4f71beb700 15 mon.0@0(leader).pg v12967 got 2.2 reported at 7:1162701 state active+clean -> active+clean 2017-03-02 15:08:18.557190 7f4f71beb700 15 mon.0@0(leader).pg v12967 got 2.3 reported at 7:1094816 state active+clean -> active+clean 2017-03-02 15:08:18.557194 7f4f71beb700 15 mon.0@0(leader).pg v12967 got 2.7 reported at 7:1134889 state active+clean -> active+clean 2017-03-02 15:08:18.557199 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12967) proposal_timer already set 2017-03-02 15:08:18.711471 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.0 172.21.15.26:6800/26515 6832 ==== pg_stats(2 pgs tid 6796 v 0) v1 ==== 1264+0+0 (131332650 0 0) 0x558a15c09680 con 0x558a145d8800 2017-03-02 15:08:18.711503 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1463af20 for osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:18.711510 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:18.711513 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12967) dispatch 0x558a15c09680 pg_stats(2 pgs tid 6796 v 0) v1 from osd.0 172.21.15.26:6800/26515 con 0x558a145d8800 2017-03-02 15:08:18.711519 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27328) is_readable = 1 - now=2017-03-02 15:08:18.711520 lease_expire=0.000000 has v0 lc 27328 2017-03-02 15:08:18.711528 7f4f71beb700 10 mon.0@0(leader).pg v12967 preprocess_query pg_stats(2 pgs tid 6796 v 0) v1 from osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:18.711535 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:18.711538 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:18.711539 7f4f71beb700 20 allow all 2017-03-02 15:08:18.711541 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27328) is_readable = 1 - now=2017-03-02 15:08:18.711541 lease_expire=0.000000 has v0 lc 27328 2017-03-02 15:08:18.711545 7f4f71beb700 10 mon.0@0(leader).pg v12967 prepare_update pg_stats(2 pgs tid 6796 v 0) v1 from osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:18.711550 7f4f71beb700 10 mon.0@0(leader).pg v12967 prepare_pg_stats pg_stats(2 pgs tid 6796 v 0) v1 from osd.0 2017-03-02 15:08:18.711556 7f4f71beb700 10 mon.0@0(leader).pg v12967 got osd.0 osd_stat(10931 MB used, 84409 MB avail, 95341 MB total, peers [1,2,3] op hist []) (was osd_stat(10938 MB used, 84403 MB avail, 95341 MB total, peers [1,2,3] op hist [])) 2017-03-02 15:08:18.711566 7f4f71beb700 15 mon.0@0(leader).pg v12967 got 1.3 reported at 7:499563 state active+clean -> active+clean 2017-03-02 15:08:18.711576 7f4f71beb700 15 mon.0@0(leader).pg v12967 got 1.4 reported at 7:486009 state active+clean -> active+clean 2017-03-02 15:08:18.711582 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12967) proposal_timer already set 2017-03-02 15:08:18.870048 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.5 172.21.15.11:6804/1637448123 8783 ==== mdsbeacon(4116/e up:active seq 8496 v733) v7 ==== 365+0+0 (61509500 0 0) 0x558a148d8700 con 0x558a1471d800 2017-03-02 15:08:18.870080 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b0f20 for mds.? 172.21.15.11:6804/1637448123 2017-03-02 15:08:18.870087 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:18.870090 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) dispatch 0x558a148d8700 mdsbeacon(4116/e up:active seq 8496 v733) v7 from mds.5 172.21.15.11:6804/1637448123 con 0x558a1471d800 2017-03-02 15:08:18.870095 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27328) is_readable = 1 - now=2017-03-02 15:08:18.870096 lease_expire=0.000000 has v0 lc 27328 2017-03-02 15:08:18.870102 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_query mdsbeacon(4116/e up:active seq 8496 v733) v7 from mds.5 172.21.15.11:6804/1637448123 2017-03-02 15:08:18.870106 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:18.870108 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:18.870109 7f4f71beb700 20 allow all 2017-03-02 15:08:18.870110 7f4f71beb700 12 mon.0@0(leader).mds e733 preprocess_beacon mdsbeacon(4116/e up:active seq 8496 v733) v7 from mds.5 172.21.15.11:6804/1637448123 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:18.870116 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_beacon: GID exists in map: 4116 2017-03-02 15:08:18.870121 7f4f71beb700 15 mon.0@0(leader).mds e733 _note_beacon mdsbeacon(4116/e up:active seq 8496 v733) v7 noting time 2017-03-02 15:08:18.870125 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a167b5700 0x558a15204000 mdsbeacon(4116/e up:active seq 8496 v733) v7 2017-03-02 15:08:18.870128 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6804/1637448123 -- mdsbeacon(4116/e up:active seq 8496 v733) v7 -- 0x558a15204000 con 0 2017-03-02 15:08:18.883724 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.3 172.21.15.11:6805/1926402648 8559 ==== mdsbeacon(4118/f up:active seq 8496 v733) v7 ==== 406+0+0 (4089752967 0 0) 0x558a143e7a80 con 0x558a14905800 2017-03-02 15:08:18.883747 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b2260 for mds.? 172.21.15.11:6805/1926402648 2017-03-02 15:08:18.883753 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:18.883755 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) dispatch 0x558a143e7a80 mdsbeacon(4118/f up:active seq 8496 v733) v7 from mds.3 172.21.15.11:6805/1926402648 con 0x558a14905800 2017-03-02 15:08:18.883761 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27328) is_readable = 1 - now=2017-03-02 15:08:18.883761 lease_expire=0.000000 has v0 lc 27328 2017-03-02 15:08:18.883767 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_query mdsbeacon(4118/f up:active seq 8496 v733) v7 from mds.3 172.21.15.11:6805/1926402648 2017-03-02 15:08:18.883773 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:18.883775 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:18.883776 7f4f71beb700 20 allow all 2017-03-02 15:08:18.883777 7f4f71beb700 12 mon.0@0(leader).mds e733 preprocess_beacon mdsbeacon(4118/f up:active seq 8496 v733) v7 from mds.3 172.21.15.11:6805/1926402648 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:18.883785 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_beacon: GID exists in map: 4118 2017-03-02 15:08:18.883790 7f4f71beb700 15 mon.0@0(leader).mds e733 _note_beacon mdsbeacon(4118/f up:active seq 8496 v733) v7 noting time 2017-03-02 15:08:18.883794 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a167b5700 0x558a148d8700 mdsbeacon(4118/f up:active seq 8496 v733) v7 2017-03-02 15:08:18.883797 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6805/1926402648 -- mdsbeacon(4118/f up:active seq 8496 v733) v7 -- 0x558a148d8700 con 0 2017-03-02 15:08:18.884613 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.2 172.21.15.22:6804/673088617 8558 ==== mdsbeacon(4119/c up:active seq 8496 v733) v7 ==== 365+0+0 (598969122 0 0) 0x558a15204000 con 0x558a14907000 2017-03-02 15:08:18.884632 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14916f20 for mds.? 172.21.15.22:6804/673088617 2017-03-02 15:08:18.884637 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:18.884639 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) dispatch 0x558a15204000 mdsbeacon(4119/c up:active seq 8496 v733) v7 from mds.2 172.21.15.22:6804/673088617 con 0x558a14907000 2017-03-02 15:08:18.884644 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27328) is_readable = 1 - now=2017-03-02 15:08:18.884644 lease_expire=0.000000 has v0 lc 27328 2017-03-02 15:08:18.884650 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_query mdsbeacon(4119/c up:active seq 8496 v733) v7 from mds.2 172.21.15.22:6804/673088617 2017-03-02 15:08:18.884655 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:18.884657 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:18.884658 7f4f71beb700 20 allow all 2017-03-02 15:08:18.884659 7f4f71beb700 12 mon.0@0(leader).mds e733 preprocess_beacon mdsbeacon(4119/c up:active seq 8496 v733) v7 from mds.2 172.21.15.22:6804/673088617 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:18.884665 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_beacon: GID exists in map: 4119 2017-03-02 15:08:18.884668 7f4f71beb700 15 mon.0@0(leader).mds e733 _note_beacon mdsbeacon(4119/c up:active seq 8496 v733) v7 noting time 2017-03-02 15:08:18.884672 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a167b5700 0x558a143e7a80 mdsbeacon(4119/c up:active seq 8496 v733) v7 2017-03-02 15:08:18.884674 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6804/673088617 -- mdsbeacon(4119/c up:active seq 8496 v733) v7 -- 0x558a143e7a80 con 0 2017-03-02 15:08:18.884804 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.0 172.21.15.22:6805/2203629154 8608 ==== mdsbeacon(4121/d up:active seq 8492 v733) v7 ==== 454+0+0 (1656840140 0 0) 0x558a148d8700 con 0x558a1492a000 2017-03-02 15:08:18.884840 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a149195a0 for mds.? 172.21.15.22:6805/2203629154 2017-03-02 15:08:18.884845 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:18.884848 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) dispatch 0x558a148d8700 mdsbeacon(4121/d up:active seq 8492 v733) v7 from mds.0 172.21.15.22:6805/2203629154 con 0x558a1492a000 2017-03-02 15:08:18.884852 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27328) is_readable = 1 - now=2017-03-02 15:08:18.884853 lease_expire=0.000000 has v0 lc 27328 2017-03-02 15:08:18.884858 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_query mdsbeacon(4121/d up:active seq 8492 v733) v7 from mds.0 172.21.15.22:6805/2203629154 2017-03-02 15:08:18.884863 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:18.884865 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:18.884866 7f4f71beb700 20 allow all 2017-03-02 15:08:18.884867 7f4f71beb700 12 mon.0@0(leader).mds e733 preprocess_beacon mdsbeacon(4121/d up:active seq 8492 v733) v7 from mds.0 172.21.15.22:6805/2203629154 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:18.884872 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_beacon: GID exists in map: 4121 2017-03-02 15:08:18.884876 7f4f71beb700 20 mon.0@0(leader).mds e733 preprocess_beacon health metrics for gid 4121 were updated 2017-03-02 15:08:18.884878 7f4f71beb700 15 mon.0@0(leader).mds e733 _note_beacon mdsbeacon(4121/d up:active seq 8492 v733) v7 noting time 2017-03-02 15:08:18.884882 7f4f71beb700 7 mon.0@0(leader).mds e733 prepare_update mdsbeacon(4121/d up:active seq 8492 v733) v7 2017-03-02 15:08:18.884884 7f4f71beb700 12 mon.0@0(leader).mds e733 prepare_beacon mdsbeacon(4121/d up:active seq 8492 v733) v7 from mds.0 172.21.15.22:6805/2203629154 2017-03-02 15:08:18.884888 7f4f71beb700 20 mon.0@0(leader).mds e733 prepare_beacon got health from gid 4121 with 1 metrics. 2017-03-02 15:08:18.884892 7f4f71beb700 10 mon.0@0(leader).mds e733 prepare_beacon mds.0 up:active -> up:active standby_for_rank=-1 2017-03-02 15:08:18.884895 7f4f71beb700 7 mon.0@0(leader).mds e733 prepare_beacon pending map now: 2017-03-02 15:08:18.884897 7f4f71beb700 7 mon.0@0(leader).mds e733 print_map 2017-03-02 15:08:18.884939 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) setting proposal_timer 0x558a1412e330 with delay of 0.468102 2017-03-02 15:08:18.923978 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.1 172.21.15.17:6804/2576824570 8555 ==== mdsbeacon(4120/g up:active seq 8496 v733) v7 ==== 365+0+0 (3955017273 0 0) 0x558a154b8d00 con 0x558a14908800 2017-03-02 15:08:18.924009 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14917600 for mds.? 172.21.15.17:6804/2576824570 2017-03-02 15:08:18.924016 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:18.924019 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) dispatch 0x558a154b8d00 mdsbeacon(4120/g up:active seq 8496 v733) v7 from mds.1 172.21.15.17:6804/2576824570 con 0x558a14908800 2017-03-02 15:08:18.924023 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27328) is_readable = 1 - now=2017-03-02 15:08:18.924023 lease_expire=0.000000 has v0 lc 27328 2017-03-02 15:08:18.924029 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_query mdsbeacon(4120/g up:active seq 8496 v733) v7 from mds.1 172.21.15.17:6804/2576824570 2017-03-02 15:08:18.924034 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:18.924036 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:18.924037 7f4f71beb700 20 allow all 2017-03-02 15:08:18.924038 7f4f71beb700 12 mon.0@0(leader).mds e733 preprocess_beacon mdsbeacon(4120/g up:active seq 8496 v733) v7 from mds.1 172.21.15.17:6804/2576824570 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:18.924043 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_beacon: GID exists in map: 4120 2017-03-02 15:08:18.924046 7f4f71beb700 10 mon.0@0(leader).mds e733 mds_beacon mdsbeacon(4120/g up:active seq 8496 v733) v7 ignoring requested state, because mds hasn't seen latest map 2017-03-02 15:08:18.924048 7f4f71beb700 15 mon.0@0(leader).mds e733 _note_beacon mdsbeacon(4120/g up:active seq 8496 v733) v7 noting time 2017-03-02 15:08:18.924050 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a166a6700 0x558a15204000 mdsbeacon(4120/g up:active seq 8496 v734) v7 2017-03-02 15:08:18.924052 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6804/2576824570 -- mdsbeacon(4120/g up:active seq 8496 v734) v7 -- 0x558a15204000 con 0 2017-03-02 15:08:18.969120 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.4 172.21.15.26:6805/2033388711 8553 ==== mdsbeacon(4117/b up:active seq 8496 v733) v7 ==== 365+0+0 (353753338 0 0) 0x558a15204000 con 0x558a14904000 2017-03-02 15:08:18.969152 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b1600 for mds.? 172.21.15.26:6805/2033388711 2017-03-02 15:08:18.969158 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:18.969160 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) dispatch 0x558a15204000 mdsbeacon(4117/b up:active seq 8496 v733) v7 from mds.4 172.21.15.26:6805/2033388711 con 0x558a14904000 2017-03-02 15:08:18.969164 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27328) is_readable = 1 - now=2017-03-02 15:08:18.969164 lease_expire=0.000000 has v0 lc 27328 2017-03-02 15:08:18.969170 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_query mdsbeacon(4117/b up:active seq 8496 v733) v7 from mds.4 172.21.15.26:6805/2033388711 2017-03-02 15:08:18.969175 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:18.969177 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:18.969178 7f4f71beb700 20 allow all 2017-03-02 15:08:18.969179 7f4f71beb700 12 mon.0@0(leader).mds e733 preprocess_beacon mdsbeacon(4117/b up:active seq 8496 v733) v7 from mds.4 172.21.15.26:6805/2033388711 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:18.969184 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_beacon: GID exists in map: 4117 2017-03-02 15:08:18.969187 7f4f71beb700 10 mon.0@0(leader).mds e733 mds_beacon mdsbeacon(4117/b up:active seq 8496 v733) v7 ignoring requested state, because mds hasn't seen latest map 2017-03-02 15:08:18.969188 7f4f71beb700 15 mon.0@0(leader).mds e733 _note_beacon mdsbeacon(4117/b up:active seq 8496 v733) v7 noting time 2017-03-02 15:08:18.969191 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a166a6700 0x558a154b8d00 mdsbeacon(4117/b up:active seq 8496 v734) v7 2017-03-02 15:08:18.969193 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6805/2033388711 -- mdsbeacon(4117/b up:active seq 8496 v734) v7 -- 0x558a154b8d00 con 0 2017-03-02 15:08:18.978168 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.6 172.21.15.26:6804/2954782670 8745 ==== mdsbeacon(4115/a up:active seq 8496 v733) v7 ==== 365+0+0 (1325345938 0 0) 0x558a143e56c0 con 0x558a1471f000 2017-03-02 15:08:18.978200 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14868d60 for mds.? 172.21.15.26:6804/2954782670 2017-03-02 15:08:18.978207 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:18.978209 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) dispatch 0x558a143e56c0 mdsbeacon(4115/a up:active seq 8496 v733) v7 from mds.6 172.21.15.26:6804/2954782670 con 0x558a1471f000 2017-03-02 15:08:18.978213 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27328) is_readable = 1 - now=2017-03-02 15:08:18.978214 lease_expire=0.000000 has v0 lc 27328 2017-03-02 15:08:18.978219 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_query mdsbeacon(4115/a up:active seq 8496 v733) v7 from mds.6 172.21.15.26:6804/2954782670 2017-03-02 15:08:18.978224 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:18.978226 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:18.978227 7f4f71beb700 20 allow all 2017-03-02 15:08:18.978228 7f4f71beb700 12 mon.0@0(leader).mds e733 preprocess_beacon mdsbeacon(4115/a up:active seq 8496 v733) v7 from mds.6 172.21.15.26:6804/2954782670 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:18.978233 7f4f71beb700 10 mon.0@0(leader).mds e733 preprocess_beacon: GID exists in map: 4115 2017-03-02 15:08:18.978236 7f4f71beb700 10 mon.0@0(leader).mds e733 mds_beacon mdsbeacon(4115/a up:active seq 8496 v733) v7 ignoring requested state, because mds hasn't seen latest map 2017-03-02 15:08:18.978238 7f4f71beb700 15 mon.0@0(leader).mds e733 _note_beacon mdsbeacon(4115/a up:active seq 8496 v733) v7 noting time 2017-03-02 15:08:18.978240 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a166a6700 0x558a15204000 mdsbeacon(4115/a up:active seq 8496 v734) v7 2017-03-02 15:08:18.978242 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6804/2954782670 -- mdsbeacon(4115/a up:active seq 8496 v734) v7 -- 0x558a15204000 con 0 2017-03-02 15:08:19.353119 7f4f743f0700 10 mon.0@0(leader).paxosservice(mdsmap 1..733) propose_pending 2017-03-02 15:08:19.353156 7f4f743f0700 10 mon.0@0(leader).mds e733 encode_pending e734 2017-03-02 15:08:19.353220 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27328) queue_pending_finisher 0x558a1412ed40 2017-03-02 15:08:19.353229 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27328) trigger_propose active, proposing now 2017-03-02 15:08:19.353239 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27328) propose_pending 27329 2411 bytes 2017-03-02 15:08:19.353242 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27328) begin for 27329 2411 bytes 2017-03-02 15:08:19.361167 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27328) commit_start 27329 2017-03-02 15:08:19.361210 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12967) propose_pending 2017-03-02 15:08:19.361214 7f4f743f0700 10 mon.0@0(leader).pg v12967 encode_pending v 12968 2017-03-02 15:08:19.361292 7f4f743f0700 5 mon.0@0(leader).paxos(paxos writing c 26607..27328) queue_pending_finisher 0x558a1412e330 2017-03-02 15:08:19.361301 7f4f743f0700 10 mon.0@0(leader).paxos(paxos writing c 26607..27328) trigger_propose not active, will propose later 2017-03-02 15:08:19.361304 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13545) propose_pending 2017-03-02 15:08:19.361306 7f4f743f0700 10 mon.0@0(leader).log v13545 encode_full log v 13545 2017-03-02 15:08:19.361365 7f4f743f0700 10 mon.0@0(leader).log v13545 encode_pending v13546 2017-03-02 15:08:19.361374 7f4f743f0700 5 mon.0@0(leader).paxos(paxos writing c 26607..27328) queue_pending_finisher 0x558a1412d740 2017-03-02 15:08:19.361375 7f4f743f0700 10 mon.0@0(leader).paxos(paxos writing c 26607..27328) trigger_propose not active, will propose later 2017-03-02 15:08:19.387994 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27328) commit_finish 27329 2017-03-02 15:08:19.388032 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:19.388048 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12967) refresh 2017-03-02 15:08:19.388056 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) refresh 2017-03-02 15:08:19.388057 7f4f763f4700 10 mon.0@0(leader).mds e733 update_from_paxos version 734, my e 733 2017-03-02 15:08:19.388063 7f4f763f4700 10 mon.0@0(leader).mds e733 update_from_paxos got 734 2017-03-02 15:08:19.388095 7f4f763f4700 4 mon.0@0(leader).mds e734 new map 2017-03-02 15:08:19.388100 7f4f763f4700 0 mon.0@0(leader).mds e734 print_map 2017-03-02 15:08:19.388133 7f4f763f4700 20 mon.0@0(leader).mds e734 check_sub: mdsmap 2017-03-02 15:08:19.388137 7f4f763f4700 10 mon.0@0(leader).mds e734 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:19.388138 7f4f763f4700 10 mon.0@0(leader).mds e734 check_sub selected MDS map epoch 734 for namespace 1 for subscriber mds.? who wants epoch 734 2017-03-02 15:08:19.388151 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6804/2954782670 -- mdsmap(e 734) v1 -- 0x558a1456f3c0 con 0 2017-03-02 15:08:19.388166 7f4f763f4700 20 mon.0@0(leader).mds e734 check_sub: mdsmap 2017-03-02 15:08:19.388169 7f4f763f4700 10 mon.0@0(leader).mds e734 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:19.388171 7f4f763f4700 10 mon.0@0(leader).mds e734 check_sub selected MDS map epoch 734 for namespace 1 for subscriber mds.? who wants epoch 734 2017-03-02 15:08:19.388179 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6804/1637448123 -- mdsmap(e 734) v1 -- 0x558a15c02f40 con 0 2017-03-02 15:08:19.388190 7f4f763f4700 20 mon.0@0(leader).mds e734 check_sub: mdsmap 2017-03-02 15:08:19.388193 7f4f763f4700 10 mon.0@0(leader).mds e734 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:19.388195 7f4f763f4700 10 mon.0@0(leader).mds e734 check_sub selected MDS map epoch 734 for namespace 1 for subscriber mds.? who wants epoch 734 2017-03-02 15:08:19.388202 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6805/2033388711 -- mdsmap(e 734) v1 -- 0x558a1456f600 con 0 2017-03-02 15:08:19.388214 7f4f763f4700 20 mon.0@0(leader).mds e734 check_sub: mdsmap 2017-03-02 15:08:19.388217 7f4f763f4700 10 mon.0@0(leader).mds e734 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:19.388218 7f4f763f4700 10 mon.0@0(leader).mds e734 check_sub selected MDS map epoch 734 for namespace 1 for subscriber mds.? who wants epoch 734 2017-03-02 15:08:19.388225 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6805/1926402648 -- mdsmap(e 734) v1 -- 0x558a1456c6c0 con 0 2017-03-02 15:08:19.388234 7f4f763f4700 20 mon.0@0(leader).mds e734 check_sub: mdsmap 2017-03-02 15:08:19.388237 7f4f763f4700 10 mon.0@0(leader).mds e734 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:19.388238 7f4f763f4700 10 mon.0@0(leader).mds e734 check_sub selected MDS map epoch 734 for namespace 1 for subscriber mds.? who wants epoch 734 2017-03-02 15:08:19.388245 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6804/673088617 -- mdsmap(e 734) v1 -- 0x558a1456d8c0 con 0 2017-03-02 15:08:19.388250 7f4f763f4700 20 mon.0@0(leader).mds e734 check_sub: mdsmap 2017-03-02 15:08:19.388253 7f4f763f4700 10 mon.0@0(leader).mds e734 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:19.388254 7f4f763f4700 10 mon.0@0(leader).mds e734 check_sub selected MDS map epoch 734 for namespace 1 for subscriber mds.? who wants epoch 734 2017-03-02 15:08:19.388260 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6804/2576824570 -- mdsmap(e 734) v1 -- 0x558a1456e400 con 0 2017-03-02 15:08:19.388275 7f4f763f4700 20 mon.0@0(leader).mds e734 check_sub: mdsmap 2017-03-02 15:08:19.388278 7f4f763f4700 10 mon.0@0(leader).mds e734 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:19.388279 7f4f763f4700 10 mon.0@0(leader).mds e734 check_sub selected MDS map epoch 734 for namespace 1 for subscriber mds.? who wants epoch 734 2017-03-02 15:08:19.388286 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6805/2203629154 -- mdsmap(e 734) v1 -- 0x558a1456d680 con 0 2017-03-02 15:08:19.388292 7f4f763f4700 20 mon.0@0(leader).mds e734 check_sub: mdsmap 2017-03-02 15:08:19.388294 7f4f763f4700 10 mon.0@0(leader).mds e734 check_sub: is_mds=0, fscid= 1 2017-03-02 15:08:19.388295 7f4f763f4700 10 mon.0@0(leader).mds e734 check_sub selected MDS map epoch 734 for namespace 1 for subscriber unknown.0 who wants epoch 734 2017-03-02 15:08:19.388301 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.132:0/2597832251 -- mdsmap(e 734) v1 -- 0x558a1456e1c0 con 0 2017-03-02 15:08:19.388304 7f4f763f4700 20 mon.0@0(leader).mds e734 check_sub: mdsmap 2017-03-02 15:08:19.388305 7f4f763f4700 10 mon.0@0(leader).mds e734 check_sub: is_mds=0, fscid= 1 2017-03-02 15:08:19.388306 7f4f763f4700 10 mon.0@0(leader).mds e734 check_sub selected MDS map epoch 734 for namespace 1 for subscriber unknown.0 who wants epoch 734 2017-03-02 15:08:19.388312 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.133:0/3792964262 -- mdsmap(e 734) v1 -- 0x558a1426f180 con 0 2017-03-02 15:08:19.388316 7f4f763f4700 10 mon.0@0(leader).mds e734 update_logger 2017-03-02 15:08:19.388329 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:19.388341 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13545) refresh 2017-03-02 15:08:19.388342 7f4f763f4700 10 mon.0@0(leader).log v13545 update_from_paxos 2017-03-02 15:08:19.388343 7f4f763f4700 10 mon.0@0(leader).log v13545 update_from_paxos version 13545 summary v 13545 2017-03-02 15:08:19.388350 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:19.388362 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:19.388363 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:19.388371 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:19.388373 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12967) post_refresh 2017-03-02 15:08:19.388374 7f4f763f4700 10 mon.0@0(leader).pg v12967 post_paxos_update 2017-03-02 15:08:19.388375 7f4f763f4700 10 mon.0@0(leader).pg v12967 check_subs 2017-03-02 15:08:19.388377 7f4f763f4700 20 mon.0@0(leader).pg v12967 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:19.388379 7f4f763f4700 20 mon.0@0(leader).pg v12967 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:19.388381 7f4f763f4700 20 mon.0@0(leader).pg v12967 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:19.388382 7f4f763f4700 20 mon.0@0(leader).pg v12967 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:19.388384 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) post_refresh 2017-03-02 15:08:19.388385 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:19.388386 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13545) post_refresh 2017-03-02 15:08:19.388386 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:19.388389 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:19.388390 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:19.388391 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27329) commit_proposal 2017-03-02 15:08:19.388393 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) _active - not active 2017-03-02 15:08:19.388394 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27329) finish_round 2017-03-02 15:08:19.388395 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27329) finish_round waiting_for_acting 2017-03-02 15:08:19.388397 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) _active 2017-03-02 15:08:19.388398 7f4f763f4700 7 mon.0@0(leader).paxosservice(mdsmap 1..734) _active creating new pending 2017-03-02 15:08:19.388420 7f4f763f4700 10 mon.0@0(leader).mds e734 create_pending e735 2017-03-02 15:08:19.388424 7f4f763f4700 10 mon.0@0(leader).mds e734 _updated mds.0 mdsbeacon(4121/d up:active seq 8492 v733) v7 2017-03-02 15:08:19.388431 7f4f763f4700 0 log_channel(cluster) log [INF] : mds.0 172.21.15.22:6805/2203629154 up:active 2017-03-02 15:08:19.388440 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14107 at 2017-03-02 15:08:19.388432) v1 -- 0x558a1426df80 con 0 2017-03-02 15:08:19.388455 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a167b5700 0x558a148d7380 mdsbeacon(4121/d up:active seq 8492 v734) v7 2017-03-02 15:08:19.388459 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6805/2203629154 -- mdsbeacon(4121/d up:active seq 8492 v734) v7 -- 0x558a148d7380 con 0 2017-03-02 15:08:19.388482 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14107 at 2017-03-02 15:08:19.388432) v1 ==== 0+0+0 (0 0 0) 0x558a1426df80 con 0x558a14482000 2017-03-02 15:08:19.388509 7f4f763f4700 10 mon.0@0(leader).mds e734 e734: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:19.388529 7f4f763f4700 10 mon.0@0(leader).mds e734 update_logger 2017-03-02 15:08:19.388538 7f4f763f4700 0 log_channel(cluster) log [INF] : fsmap e734: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:19.388546 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14108 at 2017-03-02 15:08:19.388539) v1 -- 0x558a1426e1c0 con 0 2017-03-02 15:08:19.388556 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27329) finish_round waiting_for_readable 2017-03-02 15:08:19.388558 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27329) finish_round waiting_for_writeable 2017-03-02 15:08:19.388559 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27329) finish_round done w/ waiters, state active 2017-03-02 15:08:19.388584 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27329) propose_pending 27330 20097 bytes 2017-03-02 15:08:19.388590 7f4f763f4700 10 mon.0@0(leader).paxos(paxos updating c 26607..27329) begin for 27330 20097 bytes 2017-03-02 15:08:19.421478 7f4f763f4700 10 mon.0@0(leader).paxos(paxos updating c 26607..27329) commit_start 27330 2017-03-02 15:08:19.421578 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:19.421591 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:19.421593 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13545) dispatch 0x558a1426df80 log(1 entries from seq 14107 at 2017-03-02 15:08:19.388432) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:19.421601 7f4f71beb700 5 mon.0@0(leader).paxos(paxos writing c 26607..27329) is_readable = 1 - now=2017-03-02 15:08:19.421602 lease_expire=0.000000 has v0 lc 27329 2017-03-02 15:08:19.421606 7f4f71beb700 10 mon.0@0(leader).log v13545 preprocess_query log(1 entries from seq 14107 at 2017-03-02 15:08:19.388432) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:19.421610 7f4f71beb700 10 mon.0@0(leader).log v13545 preprocess_log log(1 entries from seq 14107 at 2017-03-02 15:08:19.388432) v1 from mon.0 2017-03-02 15:08:19.421613 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:19.421614 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:19.421615 7f4f71beb700 20 allow all 2017-03-02 15:08:19.421620 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13545) waiting for paxos -> writeable 2017-03-02 15:08:19.421626 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14108 at 2017-03-02 15:08:19.388539) v1 ==== 0+0+0 (0 0 0) 0x558a1426e1c0 con 0x558a14482000 2017-03-02 15:08:19.421631 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:19.421633 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:19.421634 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13545) dispatch 0x558a1426e1c0 log(1 entries from seq 14108 at 2017-03-02 15:08:19.388539) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:19.421637 7f4f71beb700 5 mon.0@0(leader).paxos(paxos writing c 26607..27329) is_readable = 1 - now=2017-03-02 15:08:19.421637 lease_expire=0.000000 has v0 lc 27329 2017-03-02 15:08:19.421640 7f4f71beb700 10 mon.0@0(leader).log v13545 preprocess_query log(1 entries from seq 14108 at 2017-03-02 15:08:19.388539) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:19.421642 7f4f71beb700 10 mon.0@0(leader).log v13545 preprocess_log log(1 entries from seq 14108 at 2017-03-02 15:08:19.388539) v1 from mon.0 2017-03-02 15:08:19.421644 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:19.421645 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:19.421645 7f4f71beb700 20 allow all 2017-03-02 15:08:19.421646 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13545) waiting for paxos -> writeable 2017-03-02 15:08:19.454965 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27329) commit_finish 27330 2017-03-02 15:08:19.455007 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:19.455021 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12968) refresh 2017-03-02 15:08:19.455024 7f4f763f4700 10 mon.0@0(leader).pg v12967 update_from_paxos read_incremental 2017-03-02 15:08:19.455046 7f4f763f4700 20 mon.0@0(leader).pg v12967 refreshing pg 1.0 7:496025 active+clean 2017-03-02 15:08:19.455056 7f4f763f4700 20 mon.0@0(leader).pg v12967 refreshing pg 1.1 7:586163 active+clean 2017-03-02 15:08:19.455063 7f4f763f4700 20 mon.0@0(leader).pg v12967 refreshing pg 1.2 7:576865 active+clean 2017-03-02 15:08:19.455071 7f4f763f4700 20 mon.0@0(leader).pg v12967 refreshing pg 1.3 7:499563 active+clean 2017-03-02 15:08:19.455077 7f4f763f4700 20 mon.0@0(leader).pg v12967 refreshing pg 1.4 7:486009 active+clean 2017-03-02 15:08:19.455084 7f4f763f4700 20 mon.0@0(leader).pg v12967 refreshing pg 1.5 7:567431 active+clean 2017-03-02 15:08:19.455091 7f4f763f4700 20 mon.0@0(leader).pg v12967 refreshing pg 1.6 7:502722 active+clean 2017-03-02 15:08:19.455096 7f4f763f4700 20 mon.0@0(leader).pg v12967 refreshing pg 1.7 7:529136 active+clean 2017-03-02 15:08:19.455100 7f4f763f4700 20 mon.0@0(leader).pg v12967 refreshing pg 2.0 7:1056224 active+clean 2017-03-02 15:08:19.455104 7f4f763f4700 20 mon.0@0(leader).pg v12967 refreshing pg 2.1 7:1088945 active+clean 2017-03-02 15:08:19.455110 7f4f763f4700 20 mon.0@0(leader).pg v12967 refreshing pg 2.2 7:1162701 active+clean 2017-03-02 15:08:19.455117 7f4f763f4700 20 mon.0@0(leader).pg v12967 refreshing pg 2.3 7:1094816 active+clean 2017-03-02 15:08:19.455123 7f4f763f4700 20 mon.0@0(leader).pg v12967 refreshing pg 2.4 7:1025619 active+clean 2017-03-02 15:08:19.455129 7f4f763f4700 20 mon.0@0(leader).pg v12967 refreshing pg 2.5 7:1128976 active+clean 2017-03-02 15:08:19.455135 7f4f763f4700 20 mon.0@0(leader).pg v12967 refreshing pg 2.7 7:1134889 active+clean 2017-03-02 15:08:19.455138 7f4f763f4700 20 mon.0@0(leader).pg v12967 refreshing osd.0 2017-03-02 15:08:19.455145 7f4f763f4700 20 mon.0@0(leader).pg v12967 refreshing osd.1 2017-03-02 15:08:19.455148 7f4f763f4700 20 mon.0@0(leader).pg v12967 refreshing osd.2 2017-03-02 15:08:19.455155 7f4f763f4700 10 mon.0@0(leader).pg v12968 read_pgmap_meta 2017-03-02 15:08:19.455169 7f4f763f4700 10 mon.0@0(leader).pg v12968 update_logger 2017-03-02 15:08:19.455177 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) refresh 2017-03-02 15:08:19.455185 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:19.455197 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13546) refresh 2017-03-02 15:08:19.455199 7f4f763f4700 10 mon.0@0(leader).log v13546 update_from_paxos 2017-03-02 15:08:19.455201 7f4f763f4700 10 mon.0@0(leader).log v13546 update_from_paxos version 13546 summary v 13545 2017-03-02 15:08:19.455205 7f4f763f4700 10 mon.0@0(leader).log v13546 update_from_paxos latest full 13545 2017-03-02 15:08:19.455211 7f4f763f4700 7 mon.0@0(leader).log v13546 update_from_paxos applying incremental log 13546 2017-03-02 15:08:18.353649 mon.0 172.21.15.17:6789/0 14106 : cluster [INF] pgmap v12967: 32 pgs: 32 active+clean; 88257 MB data, 165 GB used, 206 GB / 372 GB avail; 258 kB/s rd, 17296 kB/s wr, 336 op/s 2017-03-02 15:08:19.455223 7f4f763f4700 20 mon.0@0(leader).log v13546 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:19.455239 7f4f763f4700 15 mon.0@0(leader).log v13546 update_from_paxos logging for 1 channels 2017-03-02 15:08:19.455241 7f4f763f4700 15 mon.0@0(leader).log v13546 update_from_paxos channel 'cluster' logging 202 bytes 2017-03-02 15:08:19.455295 7f4f763f4700 10 mon.0@0(leader).log v13546 check_subs 2017-03-02 15:08:19.455329 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:19.455349 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:19.455351 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:19.455364 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:19.455370 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12968) post_refresh 2017-03-02 15:08:19.455371 7f4f763f4700 10 mon.0@0(leader).pg v12968 post_paxos_update 2017-03-02 15:08:19.455373 7f4f763f4700 10 mon.0@0(leader).pg v12968 check_subs 2017-03-02 15:08:19.455375 7f4f763f4700 20 mon.0@0(leader).pg v12968 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:19.455380 7f4f763f4700 20 mon.0@0(leader).pg v12968 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:19.455382 7f4f763f4700 20 mon.0@0(leader).pg v12968 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:19.455385 7f4f763f4700 20 mon.0@0(leader).pg v12968 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:19.455387 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) post_refresh 2017-03-02 15:08:19.455389 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:19.455390 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13546) post_refresh 2017-03-02 15:08:19.455391 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:19.455392 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:19.455393 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:19.455395 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27330) commit_proposal 2017-03-02 15:08:19.455398 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12968) _active - not active 2017-03-02 15:08:19.455400 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13546) _active - not active 2017-03-02 15:08:19.455402 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27330) finish_round 2017-03-02 15:08:19.455403 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27330) finish_round waiting_for_acting 2017-03-02 15:08:19.455405 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12968) _active 2017-03-02 15:08:19.455407 7f4f763f4700 7 mon.0@0(leader).paxosservice(pgmap 12269..12968) _active creating new pending 2017-03-02 15:08:19.455414 7f4f763f4700 10 mon.0@0(leader).pg v12968 create_pending v 12969 2017-03-02 15:08:19.455419 7f4f763f4700 7 mon.0@0(leader).pg v12968 _updated_stats for osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:19.455424 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a16461600 0x558a1547a600 pg_stats_ack(5 pgs tid 6796) v1 2017-03-02 15:08:19.455427 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6800/26531 -- pg_stats_ack(5 pgs tid 6796) v1 -- 0x558a1547a600 con 0 2017-03-02 15:08:19.455460 7f4f763f4700 7 mon.0@0(leader).pg v12968 _updated_stats for osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:19.455464 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a14987c00 0x558a152ab400 pg_stats_ack(8 pgs tid 6796) v1 2017-03-02 15:08:19.455467 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6800/26536 -- pg_stats_ack(8 pgs tid 6796) v1 -- 0x558a152ab400 con 0 2017-03-02 15:08:19.455484 7f4f763f4700 7 mon.0@0(leader).pg v12968 _updated_stats for osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:19.455488 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a16469b00 0x558a153f9c00 pg_stats_ack(2 pgs tid 6796) v1 2017-03-02 15:08:19.455490 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6800/26515 -- pg_stats_ack(2 pgs tid 6796) v1 -- 0x558a153f9c00 con 0 2017-03-02 15:08:19.455512 7f4f763f4700 10 mon.0@0(leader).pg v12968 check_osd_map already seen 7 >= 7 2017-03-02 15:08:19.455515 7f4f763f4700 10 mon.0@0(leader).pg v12968 update_logger 2017-03-02 15:08:19.455555 7f4f763f4700 0 log_channel(cluster) log [INF] : pgmap v12968: 32 pgs: 32 active+clean; 88141 MB data, 165 GB used, 206 GB / 372 GB avail; 360 kB/s rd, 21970 kB/s wr, 424 op/s 2017-03-02 15:08:19.455574 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14109 at 2017-03-02 15:08:19.455562) v1 -- 0x558a16742000 con 0 2017-03-02 15:08:19.455591 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13546) _active 2017-03-02 15:08:19.455597 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13546) _active creating new pending 2017-03-02 15:08:19.455605 7f4f763f4700 10 mon.0@0(leader).log v13546 create_pending v 13547 2017-03-02 15:08:19.455607 7f4f763f4700 7 mon.0@0(leader).log v13546 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:19.455612 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a163e1600 0x558a14eb4780 log(last 14106) v1 2017-03-02 15:08:19.455614 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14106) v1 -- 0x558a14eb4780 con 0 2017-03-02 15:08:19.455629 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13546) dispatch 0x558a1426df80 log(1 entries from seq 14107 at 2017-03-02 15:08:19.388432) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:19.455645 7f4f763f4700 5 mon.0@0(leader).paxos(paxos active c 26607..27330) is_readable = 1 - now=2017-03-02 15:08:19.455646 lease_expire=0.000000 has v0 lc 27330 2017-03-02 15:08:19.455655 7f4f763f4700 10 mon.0@0(leader).log v13546 preprocess_query log(1 entries from seq 14107 at 2017-03-02 15:08:19.388432) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:19.455660 7f4f763f4700 10 mon.0@0(leader).log v13546 preprocess_log log(1 entries from seq 14107 at 2017-03-02 15:08:19.388432) v1 from mon.0 2017-03-02 15:08:19.455615 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14109 at 2017-03-02 15:08:19.455562) v1 ==== 0+0+0 (0 0 0) 0x558a16742000 con 0x558a14482000 2017-03-02 15:08:19.455664 7f4f763f4700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:19.455667 7f4f763f4700 20 allow so far , doing grant allow * 2017-03-02 15:08:19.455668 7f4f763f4700 20 allow all 2017-03-02 15:08:19.455672 7f4f763f4700 10 mon.0@0(leader).log v13546 prepare_update log(1 entries from seq 14107 at 2017-03-02 15:08:19.388432) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:19.455676 7f4f763f4700 10 mon.0@0(leader).log v13546 prepare_log log(1 entries from seq 14107 at 2017-03-02 15:08:19.388432) v1 from mon.0 2017-03-02 15:08:19.455679 7f4f763f4700 10 mon.0@0(leader).log v13546 logging 2017-03-02 15:08:19.388432 mon.0 172.21.15.17:6789/0 14107 : cluster [INF] mds.0 172.21.15.22:6805/2203629154 up:active 2017-03-02 15:08:19.455687 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13546) setting proposal_timer 0x558a1412ff20 with delay of 0.999293 2017-03-02 15:08:19.455702 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13546) dispatch 0x558a1426e1c0 log(1 entries from seq 14108 at 2017-03-02 15:08:19.388539) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:19.455707 7f4f763f4700 5 mon.0@0(leader).paxos(paxos active c 26607..27330) is_readable = 1 - now=2017-03-02 15:08:19.455708 lease_expire=0.000000 has v0 lc 27330 2017-03-02 15:08:19.455712 7f4f763f4700 10 mon.0@0(leader).log v13546 preprocess_query log(1 entries from seq 14108 at 2017-03-02 15:08:19.388539) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:19.455716 7f4f763f4700 10 mon.0@0(leader).log v13546 preprocess_log log(1 entries from seq 14108 at 2017-03-02 15:08:19.388539) v1 from mon.0 2017-03-02 15:08:19.455718 7f4f763f4700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:19.455719 7f4f763f4700 20 allow so far , doing grant allow * 2017-03-02 15:08:19.455721 7f4f763f4700 20 allow all 2017-03-02 15:08:19.455723 7f4f763f4700 10 mon.0@0(leader).log v13546 prepare_update log(1 entries from seq 14108 at 2017-03-02 15:08:19.388539) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:19.455726 7f4f763f4700 10 mon.0@0(leader).log v13546 prepare_log log(1 entries from seq 14108 at 2017-03-02 15:08:19.388539) v1 from mon.0 2017-03-02 15:08:19.455727 7f4f763f4700 10 mon.0@0(leader).log v13546 logging 2017-03-02 15:08:19.388539 mon.0 172.21.15.17:6789/0 14108 : cluster [INF] fsmap e734: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:19.455731 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13546) proposal_timer already set 2017-03-02 15:08:19.455732 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27330) finish_round waiting_for_readable 2017-03-02 15:08:19.455734 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27330) finish_round waiting_for_writeable 2017-03-02 15:08:19.455735 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27330) finish_round done w/ waiters, state active 2017-03-02 15:08:19.455748 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:19.455755 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:19.455757 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13546) dispatch 0x558a16742000 log(1 entries from seq 14109 at 2017-03-02 15:08:19.455562) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:19.455764 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27330) is_readable = 1 - now=2017-03-02 15:08:19.455764 lease_expire=0.000000 has v0 lc 27330 2017-03-02 15:08:19.455769 7f4f71beb700 10 mon.0@0(leader).log v13546 preprocess_query log(1 entries from seq 14109 at 2017-03-02 15:08:19.455562) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:19.455775 7f4f71beb700 10 mon.0@0(leader).log v13546 preprocess_log log(1 entries from seq 14109 at 2017-03-02 15:08:19.455562) v1 from mon.0 2017-03-02 15:08:19.455778 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:19.455780 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:19.455781 7f4f71beb700 20 allow all 2017-03-02 15:08:19.455785 7f4f71beb700 10 mon.0@0(leader).log v13546 prepare_update log(1 entries from seq 14109 at 2017-03-02 15:08:19.455562) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:19.455789 7f4f71beb700 10 mon.0@0(leader).log v13546 prepare_log log(1 entries from seq 14109 at 2017-03-02 15:08:19.455562) v1 from mon.0 2017-03-02 15:08:19.455792 7f4f71beb700 10 mon.0@0(leader).log v13546 logging 2017-03-02 15:08:19.455562 mon.0 172.21.15.17:6789/0 14109 : cluster [INF] pgmap v12968: 32 pgs: 32 active+clean; 88141 MB data, 165 GB used, 206 GB / 372 GB avail; 360 kB/s rd, 21970 kB/s wr, 424 op/s 2017-03-02 15:08:19.455799 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13546) proposal_timer already set 2017-03-02 15:08:19.455808 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14106) v1 ==== 0+0+0 (0 0 0) 0x558a14eb4780 con 0x558a14482000 2017-03-02 15:08:19.455814 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:19.455817 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:19.455839 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:19.455842 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:19.455843 7f4f71beb700 20 allow all 2017-03-02 15:08:20.455058 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13546) propose_pending 2017-03-02 15:08:20.455072 7f4f743f0700 10 mon.0@0(leader).log v13546 encode_full log v 13546 2017-03-02 15:08:20.455124 7f4f743f0700 10 mon.0@0(leader).log v13546 encode_pending v13547 2017-03-02 15:08:20.455137 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27330) queue_pending_finisher 0x558a1412f8a0 2017-03-02 15:08:20.455140 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27330) trigger_propose active, proposing now 2017-03-02 15:08:20.455146 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27330) propose_pending 27331 10715 bytes 2017-03-02 15:08:20.455148 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27330) begin for 27331 10715 bytes 2017-03-02 15:08:20.488289 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27330) commit_start 27331 2017-03-02 15:08:20.521743 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27330) commit_finish 27331 2017-03-02 15:08:20.521791 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:20.521814 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12968) refresh 2017-03-02 15:08:20.521853 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) refresh 2017-03-02 15:08:20.521875 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:20.521889 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13547) refresh 2017-03-02 15:08:20.521891 7f4f763f4700 10 mon.0@0(leader).log v13547 update_from_paxos 2017-03-02 15:08:20.521893 7f4f763f4700 10 mon.0@0(leader).log v13547 update_from_paxos version 13547 summary v 13546 2017-03-02 15:08:20.521897 7f4f763f4700 10 mon.0@0(leader).log v13547 update_from_paxos latest full 13546 2017-03-02 15:08:20.521905 7f4f763f4700 7 mon.0@0(leader).log v13547 update_from_paxos applying incremental log 13547 2017-03-02 15:08:19.388432 mon.0 172.21.15.17:6789/0 14107 : cluster [INF] mds.0 172.21.15.22:6805/2203629154 up:active 2017-03-02 15:08:20.521917 7f4f763f4700 20 mon.0@0(leader).log v13547 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:20.521946 7f4f763f4700 7 mon.0@0(leader).log v13547 update_from_paxos applying incremental log 13547 2017-03-02 15:08:19.388539 mon.0 172.21.15.17:6789/0 14108 : cluster [INF] fsmap e734: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:20.521951 7f4f763f4700 20 mon.0@0(leader).log v13547 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:20.521959 7f4f763f4700 7 mon.0@0(leader).log v13547 update_from_paxos applying incremental log 13547 2017-03-02 15:08:19.455562 mon.0 172.21.15.17:6789/0 14109 : cluster [INF] pgmap v12968: 32 pgs: 32 active+clean; 88141 MB data, 165 GB used, 206 GB / 372 GB avail; 360 kB/s rd, 21970 kB/s wr, 424 op/s 2017-03-02 15:08:20.521963 7f4f763f4700 20 mon.0@0(leader).log v13547 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:20.521970 7f4f763f4700 15 mon.0@0(leader).log v13547 update_from_paxos logging for 1 channels 2017-03-02 15:08:20.521971 7f4f763f4700 15 mon.0@0(leader).log v13547 update_from_paxos channel 'cluster' logging 518 bytes 2017-03-02 15:08:20.522043 7f4f763f4700 10 mon.0@0(leader).log v13547 check_subs 2017-03-02 15:08:20.522071 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:20.522086 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:20.522088 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:20.522100 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:20.522102 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12968) post_refresh 2017-03-02 15:08:20.522104 7f4f763f4700 10 mon.0@0(leader).pg v12968 post_paxos_update 2017-03-02 15:08:20.522106 7f4f763f4700 10 mon.0@0(leader).pg v12968 check_subs 2017-03-02 15:08:20.522108 7f4f763f4700 20 mon.0@0(leader).pg v12968 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:20.522112 7f4f763f4700 20 mon.0@0(leader).pg v12968 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:20.522115 7f4f763f4700 20 mon.0@0(leader).pg v12968 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:20.522117 7f4f763f4700 20 mon.0@0(leader).pg v12968 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:20.522119 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) post_refresh 2017-03-02 15:08:20.522121 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:20.522122 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13547) post_refresh 2017-03-02 15:08:20.522123 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:20.522125 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:20.522126 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:20.522128 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27331) commit_proposal 2017-03-02 15:08:20.522131 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13547) _active - not active 2017-03-02 15:08:20.522133 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27331) finish_round 2017-03-02 15:08:20.522134 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27331) finish_round waiting_for_acting 2017-03-02 15:08:20.522136 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13547) _active 2017-03-02 15:08:20.522138 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13547) _active creating new pending 2017-03-02 15:08:20.522145 7f4f763f4700 10 mon.0@0(leader).log v13547 create_pending v 13548 2017-03-02 15:08:20.522148 7f4f763f4700 7 mon.0@0(leader).log v13547 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:20.522152 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a166a6700 0x558a15a4e3a0 log(last 14107) v1 2017-03-02 15:08:20.522155 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14107) v1 -- 0x558a15a4e3a0 con 0 2017-03-02 15:08:20.522176 7f4f763f4700 7 mon.0@0(leader).log v13547 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:20.522179 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a15c72a00 0x558a15affc20 log(last 14108) v1 2017-03-02 15:08:20.522181 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14108) v1 -- 0x558a15affc20 con 0 2017-03-02 15:08:20.522187 7f4f763f4700 7 mon.0@0(leader).log v13547 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:20.522190 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a165fe400 0x558a155a9c00 log(last 14109) v1 2017-03-02 15:08:20.522191 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14109) v1 -- 0x558a155a9c00 con 0 2017-03-02 15:08:20.522191 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14107) v1 ==== 0+0+0 (0 0 0) 0x558a15a4e3a0 con 0x558a14482000 2017-03-02 15:08:20.522202 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27331) finish_round waiting_for_readable 2017-03-02 15:08:20.522204 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27331) finish_round waiting_for_writeable 2017-03-02 15:08:20.522206 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27331) finish_round done w/ waiters, state active 2017-03-02 15:08:20.522220 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:20.522227 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:20.522229 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:20.522232 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:20.522233 7f4f71beb700 20 allow all 2017-03-02 15:08:20.522248 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14108) v1 ==== 0+0+0 (0 0 0) 0x558a15affc20 con 0x558a14482000 2017-03-02 15:08:20.522254 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:20.522257 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:20.522259 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:20.522260 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:20.522261 7f4f71beb700 20 allow all 2017-03-02 15:08:20.522270 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14109) v1 ==== 0+0+0 (0 0 0) 0x558a155a9c00 con 0x558a14482000 2017-03-02 15:08:20.522276 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:20.522279 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:20.522280 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:20.522281 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:20.522282 7f4f71beb700 20 allow all 2017-03-02 15:08:21.011543 7f4f743f0700 11 mon.0@0(leader) e2 tick 2017-03-02 15:08:21.011562 7f4f743f0700 10 mon.0@0(leader).pg v12968 v12968: 32 pgs: 32 active+clean; 88141 MB data, 165 GB used, 206 GB / 372 GB avail; 360 kB/s rd, 21970 kB/s wr, 424 op/s 2017-03-02 15:08:21.011572 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12968) maybe_trim trim_to 12468 would only trim 199 < paxos_service_trim_min 250 2017-03-02 15:08:21.011575 7f4f743f0700 10 mon.0@0(leader).mds e734 e734: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:21.011594 7f4f743f0700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) maybe_trim trim_to 234 would only trim 233 < paxos_service_trim_min 250 2017-03-02 15:08:21.011596 7f4f743f0700 10 mon.0@0(leader).osd e7 e7: 4 osds: 4 up, 4 in 2017-03-02 15:08:21.011598 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27331) is_readable = 1 - now=2017-03-02 15:08:21.011599 lease_expire=0.000000 has v0 lc 27331 2017-03-02 15:08:21.011603 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27331) is_readable = 1 - now=2017-03-02 15:08:21.011603 lease_expire=0.000000 has v0 lc 27331 2017-03-02 15:08:21.011607 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27331) is_readable = 1 - now=2017-03-02 15:08:21.011607 lease_expire=0.000000 has v0 lc 27331 2017-03-02 15:08:21.011609 7f4f743f0700 10 mon.0@0(leader).osd e7 min_last_epoch_clean 6 2017-03-02 15:08:21.011610 7f4f743f0700 10 mon.0@0(leader).log v13547 log 2017-03-02 15:08:21.011611 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13547) maybe_trim trim_to 13047 would only trim 20 < paxos_service_trim_min 250 2017-03-02 15:08:21.011613 7f4f743f0700 10 mon.0@0(leader).auth v10 auth 2017-03-02 15:08:21.011616 7f4f743f0700 10 cephx keyserver: _check_rotating_secrets 2017-03-02 15:08:21.011621 7f4f743f0700 20 mon.0@0(leader) e2 sync_trim_providers 2017-03-02 15:08:22.870054 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.5 172.21.15.11:6804/1637448123 8784 ==== mdsbeacon(4116/e up:active seq 8497 v734) v7 ==== 365+0+0 (2815472095 0 0) 0x558a143e7a80 con 0x558a1471d800 2017-03-02 15:08:22.870082 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b0f20 for mds.? 172.21.15.11:6804/1637448123 2017-03-02 15:08:22.870089 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:22.870092 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) dispatch 0x558a143e7a80 mdsbeacon(4116/e up:active seq 8497 v734) v7 from mds.5 172.21.15.11:6804/1637448123 con 0x558a1471d800 2017-03-02 15:08:22.870098 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27331) is_readable = 1 - now=2017-03-02 15:08:22.870098 lease_expire=0.000000 has v0 lc 27331 2017-03-02 15:08:22.870106 7f4f71beb700 10 mon.0@0(leader).mds e734 preprocess_query mdsbeacon(4116/e up:active seq 8497 v734) v7 from mds.5 172.21.15.11:6804/1637448123 2017-03-02 15:08:22.870112 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:22.870115 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:22.870116 7f4f71beb700 20 allow all 2017-03-02 15:08:22.870117 7f4f71beb700 12 mon.0@0(leader).mds e734 preprocess_beacon mdsbeacon(4116/e up:active seq 8497 v734) v7 from mds.5 172.21.15.11:6804/1637448123 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:22.870124 7f4f71beb700 10 mon.0@0(leader).mds e734 preprocess_beacon: GID exists in map: 4116 2017-03-02 15:08:22.870129 7f4f71beb700 15 mon.0@0(leader).mds e734 _note_beacon mdsbeacon(4116/e up:active seq 8497 v734) v7 noting time 2017-03-02 15:08:22.870132 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15c71100 0x558a143e56c0 mdsbeacon(4116/e up:active seq 8497 v734) v7 2017-03-02 15:08:22.870134 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6804/1637448123 -- mdsbeacon(4116/e up:active seq 8497 v734) v7 -- 0x558a143e56c0 con 0 2017-03-02 15:08:22.883777 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.3 172.21.15.11:6805/1926402648 8560 ==== mdsbeacon(4118/f up:active seq 8497 v734) v7 ==== 406+0+0 (3428553495 0 0) 0x558a148d7380 con 0x558a14905800 2017-03-02 15:08:22.883799 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b2260 for mds.? 172.21.15.11:6805/1926402648 2017-03-02 15:08:22.883803 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:22.883805 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) dispatch 0x558a148d7380 mdsbeacon(4118/f up:active seq 8497 v734) v7 from mds.3 172.21.15.11:6805/1926402648 con 0x558a14905800 2017-03-02 15:08:22.883809 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27331) is_readable = 1 - now=2017-03-02 15:08:22.883809 lease_expire=0.000000 has v0 lc 27331 2017-03-02 15:08:22.883814 7f4f71beb700 10 mon.0@0(leader).mds e734 preprocess_query mdsbeacon(4118/f up:active seq 8497 v734) v7 from mds.3 172.21.15.11:6805/1926402648 2017-03-02 15:08:22.883833 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:22.883835 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:22.883835 7f4f71beb700 20 allow all 2017-03-02 15:08:22.883836 7f4f71beb700 12 mon.0@0(leader).mds e734 preprocess_beacon mdsbeacon(4118/f up:active seq 8497 v734) v7 from mds.3 172.21.15.11:6805/1926402648 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:22.883841 7f4f71beb700 10 mon.0@0(leader).mds e734 preprocess_beacon: GID exists in map: 4118 2017-03-02 15:08:22.883844 7f4f71beb700 15 mon.0@0(leader).mds e734 _note_beacon mdsbeacon(4118/f up:active seq 8497 v734) v7 noting time 2017-03-02 15:08:22.883847 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15c71100 0x558a143e7a80 mdsbeacon(4118/f up:active seq 8497 v734) v7 2017-03-02 15:08:22.883848 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6805/1926402648 -- mdsbeacon(4118/f up:active seq 8497 v734) v7 -- 0x558a143e7a80 con 0 2017-03-02 15:08:22.884645 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.2 172.21.15.22:6804/673088617 8559 ==== mdsbeacon(4119/c up:active seq 8497 v734) v7 ==== 365+0+0 (2278143361 0 0) 0x558a143e56c0 con 0x558a14907000 2017-03-02 15:08:22.884665 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14916f20 for mds.? 172.21.15.22:6804/673088617 2017-03-02 15:08:22.884670 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:22.884672 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) dispatch 0x558a143e56c0 mdsbeacon(4119/c up:active seq 8497 v734) v7 from mds.2 172.21.15.22:6804/673088617 con 0x558a14907000 2017-03-02 15:08:22.884675 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27331) is_readable = 1 - now=2017-03-02 15:08:22.884675 lease_expire=0.000000 has v0 lc 27331 2017-03-02 15:08:22.884679 7f4f71beb700 10 mon.0@0(leader).mds e734 preprocess_query mdsbeacon(4119/c up:active seq 8497 v734) v7 from mds.2 172.21.15.22:6804/673088617 2017-03-02 15:08:22.884682 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:22.884684 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:22.884684 7f4f71beb700 20 allow all 2017-03-02 15:08:22.884685 7f4f71beb700 12 mon.0@0(leader).mds e734 preprocess_beacon mdsbeacon(4119/c up:active seq 8497 v734) v7 from mds.2 172.21.15.22:6804/673088617 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:22.884689 7f4f71beb700 10 mon.0@0(leader).mds e734 preprocess_beacon: GID exists in map: 4119 2017-03-02 15:08:22.884692 7f4f71beb700 15 mon.0@0(leader).mds e734 _note_beacon mdsbeacon(4119/c up:active seq 8497 v734) v7 noting time 2017-03-02 15:08:22.884695 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15c71100 0x558a148d7380 mdsbeacon(4119/c up:active seq 8497 v734) v7 2017-03-02 15:08:22.884697 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6804/673088617 -- mdsbeacon(4119/c up:active seq 8497 v734) v7 -- 0x558a148d7380 con 0 2017-03-02 15:08:22.884826 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.0 172.21.15.22:6805/2203629154 8609 ==== mdsbeacon(4121/d up:active seq 8493 v734) v7 ==== 454+0+0 (2079676206 0 0) 0x558a143e7a80 con 0x558a1492a000 2017-03-02 15:08:22.884847 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a149195a0 for mds.? 172.21.15.22:6805/2203629154 2017-03-02 15:08:22.884852 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:22.884854 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) dispatch 0x558a143e7a80 mdsbeacon(4121/d up:active seq 8493 v734) v7 from mds.0 172.21.15.22:6805/2203629154 con 0x558a1492a000 2017-03-02 15:08:22.884859 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27331) is_readable = 1 - now=2017-03-02 15:08:22.884859 lease_expire=0.000000 has v0 lc 27331 2017-03-02 15:08:22.884864 7f4f71beb700 10 mon.0@0(leader).mds e734 preprocess_query mdsbeacon(4121/d up:active seq 8493 v734) v7 from mds.0 172.21.15.22:6805/2203629154 2017-03-02 15:08:22.884869 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:22.884870 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:22.884871 7f4f71beb700 20 allow all 2017-03-02 15:08:22.884871 7f4f71beb700 12 mon.0@0(leader).mds e734 preprocess_beacon mdsbeacon(4121/d up:active seq 8493 v734) v7 from mds.0 172.21.15.22:6805/2203629154 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:22.884887 7f4f71beb700 10 mon.0@0(leader).mds e734 preprocess_beacon: GID exists in map: 4121 2017-03-02 15:08:22.884890 7f4f71beb700 15 mon.0@0(leader).mds e734 _note_beacon mdsbeacon(4121/d up:active seq 8493 v734) v7 noting time 2017-03-02 15:08:22.884893 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15c71100 0x558a143e56c0 mdsbeacon(4121/d up:active seq 8493 v734) v7 2017-03-02 15:08:22.884896 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6805/2203629154 -- mdsbeacon(4121/d up:active seq 8493 v734) v7 -- 0x558a143e56c0 con 0 2017-03-02 15:08:22.924024 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.1 172.21.15.17:6804/2576824570 8556 ==== mdsbeacon(4120/g up:active seq 8497 v734) v7 ==== 365+0+0 (1338439834 0 0) 0x558a154b8d00 con 0x558a14908800 2017-03-02 15:08:22.924051 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14917600 for mds.? 172.21.15.17:6804/2576824570 2017-03-02 15:08:22.924058 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:22.924061 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) dispatch 0x558a154b8d00 mdsbeacon(4120/g up:active seq 8497 v734) v7 from mds.1 172.21.15.17:6804/2576824570 con 0x558a14908800 2017-03-02 15:08:22.924067 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27331) is_readable = 1 - now=2017-03-02 15:08:22.924067 lease_expire=0.000000 has v0 lc 27331 2017-03-02 15:08:22.924074 7f4f71beb700 10 mon.0@0(leader).mds e734 preprocess_query mdsbeacon(4120/g up:active seq 8497 v734) v7 from mds.1 172.21.15.17:6804/2576824570 2017-03-02 15:08:22.924080 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:22.924083 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:22.924084 7f4f71beb700 20 allow all 2017-03-02 15:08:22.924085 7f4f71beb700 12 mon.0@0(leader).mds e734 preprocess_beacon mdsbeacon(4120/g up:active seq 8497 v734) v7 from mds.1 172.21.15.17:6804/2576824570 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:22.924092 7f4f71beb700 10 mon.0@0(leader).mds e734 preprocess_beacon: GID exists in map: 4120 2017-03-02 15:08:22.924096 7f4f71beb700 15 mon.0@0(leader).mds e734 _note_beacon mdsbeacon(4120/g up:active seq 8497 v734) v7 noting time 2017-03-02 15:08:22.924100 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15c71100 0x558a143e7a80 mdsbeacon(4120/g up:active seq 8497 v734) v7 2017-03-02 15:08:22.924103 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6804/2576824570 -- mdsbeacon(4120/g up:active seq 8497 v734) v7 -- 0x558a143e7a80 con 0 2017-03-02 15:08:22.969247 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.4 172.21.15.26:6805/2033388711 8554 ==== mdsbeacon(4117/b up:active seq 8497 v734) v7 ==== 365+0+0 (2976900697 0 0) 0x558a143e7a80 con 0x558a14904000 2017-03-02 15:08:22.969272 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b1600 for mds.? 172.21.15.26:6805/2033388711 2017-03-02 15:08:22.969276 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:22.969278 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) dispatch 0x558a143e7a80 mdsbeacon(4117/b up:active seq 8497 v734) v7 from mds.4 172.21.15.26:6805/2033388711 con 0x558a14904000 2017-03-02 15:08:22.969282 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27331) is_readable = 1 - now=2017-03-02 15:08:22.969283 lease_expire=0.000000 has v0 lc 27331 2017-03-02 15:08:22.969288 7f4f71beb700 10 mon.0@0(leader).mds e734 preprocess_query mdsbeacon(4117/b up:active seq 8497 v734) v7 from mds.4 172.21.15.26:6805/2033388711 2017-03-02 15:08:22.969292 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:22.969294 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:22.969295 7f4f71beb700 20 allow all 2017-03-02 15:08:22.969307 7f4f71beb700 12 mon.0@0(leader).mds e734 preprocess_beacon mdsbeacon(4117/b up:active seq 8497 v734) v7 from mds.4 172.21.15.26:6805/2033388711 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:22.969313 7f4f71beb700 10 mon.0@0(leader).mds e734 preprocess_beacon: GID exists in map: 4117 2017-03-02 15:08:22.969317 7f4f71beb700 15 mon.0@0(leader).mds e734 _note_beacon mdsbeacon(4117/b up:active seq 8497 v734) v7 noting time 2017-03-02 15:08:22.969319 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15c71100 0x558a154b8d00 mdsbeacon(4117/b up:active seq 8497 v734) v7 2017-03-02 15:08:22.969321 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6805/2033388711 -- mdsbeacon(4117/b up:active seq 8497 v734) v7 -- 0x558a154b8d00 con 0 2017-03-02 15:08:22.978450 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.6 172.21.15.26:6804/2954782670 8746 ==== mdsbeacon(4115/a up:active seq 8497 v734) v7 ==== 365+0+0 (3934589489 0 0) 0x558a143e56c0 con 0x558a1471f000 2017-03-02 15:08:22.978473 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14868d60 for mds.? 172.21.15.26:6804/2954782670 2017-03-02 15:08:22.978479 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:22.978480 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) dispatch 0x558a143e56c0 mdsbeacon(4115/a up:active seq 8497 v734) v7 from mds.6 172.21.15.26:6804/2954782670 con 0x558a1471f000 2017-03-02 15:08:22.978484 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27331) is_readable = 1 - now=2017-03-02 15:08:22.978484 lease_expire=0.000000 has v0 lc 27331 2017-03-02 15:08:22.978492 7f4f71beb700 10 mon.0@0(leader).mds e734 preprocess_query mdsbeacon(4115/a up:active seq 8497 v734) v7 from mds.6 172.21.15.26:6804/2954782670 2017-03-02 15:08:22.978496 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:22.978497 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:22.978498 7f4f71beb700 20 allow all 2017-03-02 15:08:22.978499 7f4f71beb700 12 mon.0@0(leader).mds e734 preprocess_beacon mdsbeacon(4115/a up:active seq 8497 v734) v7 from mds.6 172.21.15.26:6804/2954782670 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:22.978503 7f4f71beb700 10 mon.0@0(leader).mds e734 preprocess_beacon: GID exists in map: 4115 2017-03-02 15:08:22.978506 7f4f71beb700 15 mon.0@0(leader).mds e734 _note_beacon mdsbeacon(4115/a up:active seq 8497 v734) v7 noting time 2017-03-02 15:08:22.978508 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15c71100 0x558a143e7a80 mdsbeacon(4115/a up:active seq 8497 v734) v7 2017-03-02 15:08:22.978510 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6804/2954782670 -- mdsbeacon(4115/a up:active seq 8497 v734) v7 -- 0x558a143e7a80 con 0 2017-03-02 15:08:23.262074 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.3 172.21.15.17:6800/27100 6833 ==== pg_stats(1 pgs tid 6797 v 0) v1 ==== 698+0+0 (2708884744 0 0) 0x558a15ae9680 con 0x558a145d7000 2017-03-02 15:08:23.262110 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415f440 for osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:23.262118 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:23.262122 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12968) dispatch 0x558a15ae9680 pg_stats(1 pgs tid 6797 v 0) v1 from osd.3 172.21.15.17:6800/27100 con 0x558a145d7000 2017-03-02 15:08:23.262129 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27331) is_readable = 1 - now=2017-03-02 15:08:23.262129 lease_expire=0.000000 has v0 lc 27331 2017-03-02 15:08:23.262139 7f4f71beb700 10 mon.0@0(leader).pg v12968 preprocess_query pg_stats(1 pgs tid 6797 v 0) v1 from osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:23.262146 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:23.262149 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:23.262150 7f4f71beb700 20 allow all 2017-03-02 15:08:23.262152 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27331) is_readable = 1 - now=2017-03-02 15:08:23.262153 lease_expire=0.000000 has v0 lc 27331 2017-03-02 15:08:23.262158 7f4f71beb700 10 mon.0@0(leader).pg v12968 prepare_update pg_stats(1 pgs tid 6797 v 0) v1 from osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:23.262164 7f4f71beb700 10 mon.0@0(leader).pg v12968 prepare_pg_stats pg_stats(1 pgs tid 6797 v 0) v1 from osd.3 2017-03-02 15:08:23.262171 7f4f71beb700 10 mon.0@0(leader).pg v12968 got osd.3 osd_stat(42019 MB used, 53321 MB avail, 95341 MB total, peers [0,1,2] op hist []) (was osd_stat(42082 MB used, 53259 MB avail, 95341 MB total, peers [0,1,2] op hist [])) 2017-03-02 15:08:23.262180 7f4f71beb700 15 mon.0@0(leader).pg v12968 got 2.6 reported at 7:964072 state active+clean -> active+clean 2017-03-02 15:08:23.262191 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12968) setting proposal_timer 0x558a1412c7c0 with delay of 0.05 2017-03-02 15:08:23.312272 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12968) propose_pending 2017-03-02 15:08:23.312291 7f4f743f0700 10 mon.0@0(leader).pg v12968 encode_pending v 12969 2017-03-02 15:08:23.312328 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27331) queue_pending_finisher 0x558a1412da40 2017-03-02 15:08:23.312332 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27331) trigger_propose active, proposing now 2017-03-02 15:08:23.312342 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27331) propose_pending 27332 1090 bytes 2017-03-02 15:08:23.312345 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27331) begin for 27332 1090 bytes 2017-03-02 15:08:23.319368 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27331) commit_start 27332 2017-03-02 15:08:23.346547 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27331) commit_finish 27332 2017-03-02 15:08:23.346601 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:23.346628 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12969) refresh 2017-03-02 15:08:23.346632 7f4f763f4700 10 mon.0@0(leader).pg v12968 update_from_paxos read_incremental 2017-03-02 15:08:23.346659 7f4f763f4700 20 mon.0@0(leader).pg v12968 refreshing pg 2.6 7:964072 active+clean 2017-03-02 15:08:23.346666 7f4f763f4700 20 mon.0@0(leader).pg v12968 refreshing osd.3 2017-03-02 15:08:23.346680 7f4f763f4700 10 mon.0@0(leader).pg v12969 read_pgmap_meta 2017-03-02 15:08:23.346708 7f4f763f4700 10 mon.0@0(leader).pg v12969 update_logger 2017-03-02 15:08:23.346725 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) refresh 2017-03-02 15:08:23.346740 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:23.346754 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13547) refresh 2017-03-02 15:08:23.346755 7f4f763f4700 10 mon.0@0(leader).log v13547 update_from_paxos 2017-03-02 15:08:23.346757 7f4f763f4700 10 mon.0@0(leader).log v13547 update_from_paxos version 13547 summary v 13547 2017-03-02 15:08:23.346766 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:23.346778 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:23.346779 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:23.346789 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:23.346791 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12969) post_refresh 2017-03-02 15:08:23.346792 7f4f763f4700 10 mon.0@0(leader).pg v12969 post_paxos_update 2017-03-02 15:08:23.346794 7f4f763f4700 10 mon.0@0(leader).pg v12969 check_subs 2017-03-02 15:08:23.346796 7f4f763f4700 20 mon.0@0(leader).pg v12969 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:23.346802 7f4f763f4700 20 mon.0@0(leader).pg v12969 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:23.346804 7f4f763f4700 20 mon.0@0(leader).pg v12969 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:23.346807 7f4f763f4700 20 mon.0@0(leader).pg v12969 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:23.346808 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) post_refresh 2017-03-02 15:08:23.346810 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:23.346811 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13547) post_refresh 2017-03-02 15:08:23.346811 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:23.346813 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:23.346814 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:23.346815 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27332) commit_proposal 2017-03-02 15:08:23.346818 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12969) _active - not active 2017-03-02 15:08:23.346836 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27332) finish_round 2017-03-02 15:08:23.346838 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27332) finish_round waiting_for_acting 2017-03-02 15:08:23.346841 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12969) _active 2017-03-02 15:08:23.346843 7f4f763f4700 7 mon.0@0(leader).paxosservice(pgmap 12269..12969) _active creating new pending 2017-03-02 15:08:23.346846 7f4f763f4700 10 mon.0@0(leader).pg v12969 create_pending v 12970 2017-03-02 15:08:23.346850 7f4f763f4700 7 mon.0@0(leader).pg v12969 _updated_stats for osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:23.346854 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a15c71100 0x558a14dc6a00 pg_stats_ack(1 pgs tid 6797) v1 2017-03-02 15:08:23.346857 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6800/27100 -- pg_stats_ack(1 pgs tid 6797) v1 -- 0x558a14dc6a00 con 0 2017-03-02 15:08:23.346885 7f4f763f4700 10 mon.0@0(leader).pg v12969 check_osd_map already seen 7 >= 7 2017-03-02 15:08:23.346888 7f4f763f4700 10 mon.0@0(leader).pg v12969 update_logger 2017-03-02 15:08:23.346900 7f4f763f4700 0 log_channel(cluster) log [INF] : pgmap v12969: 32 pgs: 32 active+clean; 88143 MB data, 165 GB used, 206 GB / 372 GB avail; 304 kB/s rd, 18837 kB/s wr, 366 op/s 2017-03-02 15:08:23.346911 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14110 at 2017-03-02 15:08:23.346901) v1 -- 0x558a1426e1c0 con 0 2017-03-02 15:08:23.346923 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27332) finish_round waiting_for_readable 2017-03-02 15:08:23.346924 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27332) finish_round waiting_for_writeable 2017-03-02 15:08:23.346926 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27332) finish_round done w/ waiters, state active 2017-03-02 15:08:23.346949 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14110 at 2017-03-02 15:08:23.346901) v1 ==== 0+0+0 (0 0 0) 0x558a1426e1c0 con 0x558a14482000 2017-03-02 15:08:23.346968 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:23.346972 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:23.346975 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13547) dispatch 0x558a1426e1c0 log(1 entries from seq 14110 at 2017-03-02 15:08:23.346901) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:23.346981 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27332) is_readable = 1 - now=2017-03-02 15:08:23.346981 lease_expire=0.000000 has v0 lc 27332 2017-03-02 15:08:23.346986 7f4f71beb700 10 mon.0@0(leader).log v13547 preprocess_query log(1 entries from seq 14110 at 2017-03-02 15:08:23.346901) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:23.346992 7f4f71beb700 10 mon.0@0(leader).log v13547 preprocess_log log(1 entries from seq 14110 at 2017-03-02 15:08:23.346901) v1 from mon.0 2017-03-02 15:08:23.346995 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:23.346998 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:23.346999 7f4f71beb700 20 allow all 2017-03-02 15:08:23.347005 7f4f71beb700 10 mon.0@0(leader).log v13547 prepare_update log(1 entries from seq 14110 at 2017-03-02 15:08:23.346901) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:23.347010 7f4f71beb700 10 mon.0@0(leader).log v13547 prepare_log log(1 entries from seq 14110 at 2017-03-02 15:08:23.346901) v1 from mon.0 2017-03-02 15:08:23.347012 7f4f71beb700 10 mon.0@0(leader).log v13547 logging 2017-03-02 15:08:23.346901 mon.0 172.21.15.17:6789/0 14110 : cluster [INF] pgmap v12969: 32 pgs: 32 active+clean; 88143 MB data, 165 GB used, 206 GB / 372 GB avail; 304 kB/s rd, 18837 kB/s wr, 366 op/s 2017-03-02 15:08:23.347019 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13547) setting proposal_timer 0x558a1412fa00 with delay of 0.999552 2017-03-02 15:08:23.464200 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.1 172.21.15.22:6800/26531 6833 ==== pg_stats(5 pgs tid 6797 v 0) v1 ==== 2974+0+0 (3486099729 0 0) 0x558a15ca3900 con 0x558a14643800 2017-03-02 15:08:23.464230 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1463b1e0 for osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:23.464237 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:23.464241 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12969) dispatch 0x558a15ca3900 pg_stats(5 pgs tid 6797 v 0) v1 from osd.1 172.21.15.22:6800/26531 con 0x558a14643800 2017-03-02 15:08:23.464246 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27332) is_readable = 1 - now=2017-03-02 15:08:23.464247 lease_expire=0.000000 has v0 lc 27332 2017-03-02 15:08:23.464256 7f4f71beb700 10 mon.0@0(leader).pg v12969 preprocess_query pg_stats(5 pgs tid 6797 v 0) v1 from osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:23.464263 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:23.464266 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:23.464267 7f4f71beb700 20 allow all 2017-03-02 15:08:23.464268 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27332) is_readable = 1 - now=2017-03-02 15:08:23.464268 lease_expire=0.000000 has v0 lc 27332 2017-03-02 15:08:23.464273 7f4f71beb700 10 mon.0@0(leader).pg v12969 prepare_update pg_stats(5 pgs tid 6797 v 0) v1 from osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:23.464278 7f4f71beb700 10 mon.0@0(leader).pg v12969 prepare_pg_stats pg_stats(5 pgs tid 6797 v 0) v1 from osd.1 2017-03-02 15:08:23.464284 7f4f71beb700 10 mon.0@0(leader).pg v12969 got osd.1 osd_stat(53045 MB used, 42295 MB avail, 95341 MB total, peers [0,2,3] op hist [1,0,1]) (was osd_stat(53047 MB used, 42294 MB avail, 95341 MB total, peers [0,2,3] op hist [])) 2017-03-02 15:08:23.464293 7f4f71beb700 15 mon.0@0(leader).pg v12969 got 1.1 reported at 7:586650 state active+clean -> active+clean 2017-03-02 15:08:23.464302 7f4f71beb700 15 mon.0@0(leader).pg v12969 got 1.5 reported at 7:567540 state active+clean -> active+clean 2017-03-02 15:08:23.464306 7f4f71beb700 15 mon.0@0(leader).pg v12969 got 1.6 reported at 7:502888 state active+clean -> active+clean 2017-03-02 15:08:23.464310 7f4f71beb700 15 mon.0@0(leader).pg v12969 got 2.4 reported at 7:1025814 state active+clean -> active+clean 2017-03-02 15:08:23.464314 7f4f71beb700 15 mon.0@0(leader).pg v12969 got 2.5 reported at 7:1129210 state active+clean -> active+clean 2017-03-02 15:08:23.464319 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12969) setting proposal_timer 0x558a1412ea10 with delay of 0.882252 2017-03-02 15:08:23.557497 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.2 172.21.15.11:6800/26536 6833 ==== pg_stats(8 pgs tid 6797 v 0) v1 ==== 4676+0+0 (699232183 0 0) 0x558a1588be00 con 0x558a145d5800 2017-03-02 15:08:23.557521 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a145c3700 for osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:23.557525 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:23.557528 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12969) dispatch 0x558a1588be00 pg_stats(8 pgs tid 6797 v 0) v1 from osd.2 172.21.15.11:6800/26536 con 0x558a145d5800 2017-03-02 15:08:23.557531 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27332) is_readable = 1 - now=2017-03-02 15:08:23.557531 lease_expire=0.000000 has v0 lc 27332 2017-03-02 15:08:23.557537 7f4f71beb700 10 mon.0@0(leader).pg v12969 preprocess_query pg_stats(8 pgs tid 6797 v 0) v1 from osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:23.557541 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:23.557543 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:23.557544 7f4f71beb700 20 allow all 2017-03-02 15:08:23.557545 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27332) is_readable = 1 - now=2017-03-02 15:08:23.557545 lease_expire=0.000000 has v0 lc 27332 2017-03-02 15:08:23.557547 7f4f71beb700 10 mon.0@0(leader).pg v12969 prepare_update pg_stats(8 pgs tid 6797 v 0) v1 from osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:23.557550 7f4f71beb700 10 mon.0@0(leader).pg v12969 prepare_pg_stats pg_stats(8 pgs tid 6797 v 0) v1 from osd.2 2017-03-02 15:08:23.557555 7f4f71beb700 10 mon.0@0(leader).pg v12969 got osd.2 osd_stat(63652 MB used, 31689 MB avail, 95341 MB total, peers [0,1,3] op hist [2,2,3,2]) (was osd_stat(63687 MB used, 31654 MB avail, 95341 MB total, peers [0,1,3] op hist [])) 2017-03-02 15:08:23.557561 7f4f71beb700 15 mon.0@0(leader).pg v12969 got 1.0 reported at 7:496378 state active+clean -> active+clean 2017-03-02 15:08:23.557567 7f4f71beb700 15 mon.0@0(leader).pg v12969 got 1.2 reported at 7:577014 state active+clean -> active+clean 2017-03-02 15:08:23.557571 7f4f71beb700 15 mon.0@0(leader).pg v12969 got 1.7 reported at 7:529421 state active+clean -> active+clean 2017-03-02 15:08:23.557573 7f4f71beb700 15 mon.0@0(leader).pg v12969 got 2.0 reported at 7:1056421 state active+clean -> active+clean 2017-03-02 15:08:23.557576 7f4f71beb700 15 mon.0@0(leader).pg v12969 got 2.1 reported at 7:1089159 state active+clean -> active+clean 2017-03-02 15:08:23.557578 7f4f71beb700 15 mon.0@0(leader).pg v12969 got 2.2 reported at 7:1162918 state active+clean -> active+clean 2017-03-02 15:08:23.557580 7f4f71beb700 15 mon.0@0(leader).pg v12969 got 2.3 reported at 7:1095003 state active+clean -> active+clean 2017-03-02 15:08:23.557583 7f4f71beb700 15 mon.0@0(leader).pg v12969 got 2.7 reported at 7:1135083 state active+clean -> active+clean 2017-03-02 15:08:23.557586 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12969) proposal_timer already set 2017-03-02 15:08:23.712319 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.0 172.21.15.26:6800/26515 6833 ==== pg_stats(2 pgs tid 6797 v 0) v1 ==== 1264+0+0 (293921054 0 0) 0x558a15f24780 con 0x558a145d8800 2017-03-02 15:08:23.712349 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1463af20 for osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:23.712356 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:23.712359 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12969) dispatch 0x558a15f24780 pg_stats(2 pgs tid 6797 v 0) v1 from osd.0 172.21.15.26:6800/26515 con 0x558a145d8800 2017-03-02 15:08:23.712364 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27332) is_readable = 1 - now=2017-03-02 15:08:23.712365 lease_expire=0.000000 has v0 lc 27332 2017-03-02 15:08:23.712388 7f4f71beb700 10 mon.0@0(leader).pg v12969 preprocess_query pg_stats(2 pgs tid 6797 v 0) v1 from osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:23.712395 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:23.712397 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:23.712398 7f4f71beb700 20 allow all 2017-03-02 15:08:23.712400 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27332) is_readable = 1 - now=2017-03-02 15:08:23.712400 lease_expire=0.000000 has v0 lc 27332 2017-03-02 15:08:23.712404 7f4f71beb700 10 mon.0@0(leader).pg v12969 prepare_update pg_stats(2 pgs tid 6797 v 0) v1 from osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:23.712409 7f4f71beb700 10 mon.0@0(leader).pg v12969 prepare_pg_stats pg_stats(2 pgs tid 6797 v 0) v1 from osd.0 2017-03-02 15:08:23.712415 7f4f71beb700 10 mon.0@0(leader).pg v12969 got osd.0 osd_stat(10935 MB used, 84405 MB avail, 95341 MB total, peers [1,2,3] op hist []) (was osd_stat(10931 MB used, 84409 MB avail, 95341 MB total, peers [1,2,3] op hist [])) 2017-03-02 15:08:23.712423 7f4f71beb700 15 mon.0@0(leader).pg v12969 got 1.3 reported at 7:499753 state active+clean -> active+clean 2017-03-02 15:08:23.712433 7f4f71beb700 15 mon.0@0(leader).pg v12969 got 1.4 reported at 7:486067 state active+clean -> active+clean 2017-03-02 15:08:23.712438 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12969) proposal_timer already set 2017-03-02 15:08:24.346647 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13547) propose_pending 2017-03-02 15:08:24.346662 7f4f743f0700 10 mon.0@0(leader).log v13547 encode_full log v 13547 2017-03-02 15:08:24.346710 7f4f743f0700 10 mon.0@0(leader).log v13547 encode_pending v13548 2017-03-02 15:08:24.346715 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27332) queue_pending_finisher 0x558a1412c7c0 2017-03-02 15:08:24.346718 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27332) trigger_propose active, proposing now 2017-03-02 15:08:24.346723 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27332) propose_pending 27333 10291 bytes 2017-03-02 15:08:24.346725 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27332) begin for 27333 10291 bytes 2017-03-02 15:08:24.379856 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27332) commit_start 27333 2017-03-02 15:08:24.379891 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12969) propose_pending 2017-03-02 15:08:24.379893 7f4f743f0700 10 mon.0@0(leader).pg v12969 encode_pending v 12970 2017-03-02 15:08:24.379946 7f4f743f0700 5 mon.0@0(leader).paxos(paxos writing c 26607..27332) queue_pending_finisher 0x558a1412fa00 2017-03-02 15:08:24.379951 7f4f743f0700 10 mon.0@0(leader).paxos(paxos writing c 26607..27332) trigger_propose not active, will propose later 2017-03-02 15:08:24.413284 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27332) commit_finish 27333 2017-03-02 15:08:24.413319 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:24.413336 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12969) refresh 2017-03-02 15:08:24.413347 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) refresh 2017-03-02 15:08:24.413369 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:24.413388 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13548) refresh 2017-03-02 15:08:24.413393 7f4f763f4700 10 mon.0@0(leader).log v13548 update_from_paxos 2017-03-02 15:08:24.413395 7f4f763f4700 10 mon.0@0(leader).log v13548 update_from_paxos version 13548 summary v 13547 2017-03-02 15:08:24.413399 7f4f763f4700 10 mon.0@0(leader).log v13548 update_from_paxos latest full 13547 2017-03-02 15:08:24.413406 7f4f763f4700 7 mon.0@0(leader).log v13548 update_from_paxos applying incremental log 13548 2017-03-02 15:08:23.346901 mon.0 172.21.15.17:6789/0 14110 : cluster [INF] pgmap v12969: 32 pgs: 32 active+clean; 88143 MB data, 165 GB used, 206 GB / 372 GB avail; 304 kB/s rd, 18837 kB/s wr, 366 op/s 2017-03-02 15:08:24.413418 7f4f763f4700 20 mon.0@0(leader).log v13548 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:24.413434 7f4f763f4700 15 mon.0@0(leader).log v13548 update_from_paxos logging for 1 channels 2017-03-02 15:08:24.413442 7f4f763f4700 15 mon.0@0(leader).log v13548 update_from_paxos channel 'cluster' logging 202 bytes 2017-03-02 15:08:24.413503 7f4f763f4700 10 mon.0@0(leader).log v13548 check_subs 2017-03-02 15:08:24.413546 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:24.413573 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:24.413581 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:24.413599 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:24.413607 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12969) post_refresh 2017-03-02 15:08:24.413610 7f4f763f4700 10 mon.0@0(leader).pg v12969 post_paxos_update 2017-03-02 15:08:24.413612 7f4f763f4700 10 mon.0@0(leader).pg v12969 check_subs 2017-03-02 15:08:24.413615 7f4f763f4700 20 mon.0@0(leader).pg v12969 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:24.413622 7f4f763f4700 20 mon.0@0(leader).pg v12969 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:24.413625 7f4f763f4700 20 mon.0@0(leader).pg v12969 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:24.413629 7f4f763f4700 20 mon.0@0(leader).pg v12969 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:24.413633 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) post_refresh 2017-03-02 15:08:24.413635 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:24.413637 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13548) post_refresh 2017-03-02 15:08:24.413639 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:24.413641 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:24.413643 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:24.413647 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27333) commit_proposal 2017-03-02 15:08:24.413651 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13548) _active - not active 2017-03-02 15:08:24.413654 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27333) finish_round 2017-03-02 15:08:24.413657 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27333) finish_round waiting_for_acting 2017-03-02 15:08:24.413660 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13548) _active 2017-03-02 15:08:24.413662 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13548) _active creating new pending 2017-03-02 15:08:24.413671 7f4f763f4700 10 mon.0@0(leader).log v13548 create_pending v 13549 2017-03-02 15:08:24.413675 7f4f763f4700 7 mon.0@0(leader).log v13548 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:24.413681 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a145dd200 0x558a151d1480 log(last 14110) v1 2017-03-02 15:08:24.413686 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14110) v1 -- 0x558a151d1480 con 0 2017-03-02 15:08:24.413709 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27333) finish_round waiting_for_readable 2017-03-02 15:08:24.413713 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27333) finish_round waiting_for_writeable 2017-03-02 15:08:24.413715 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27333) finish_round done w/ waiters, state active 2017-03-02 15:08:24.413730 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14110) v1 ==== 0+0+0 (0 0 0) 0x558a151d1480 con 0x558a14482000 2017-03-02 15:08:24.413753 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27333) propose_pending 27334 9764 bytes 2017-03-02 15:08:24.413762 7f4f763f4700 10 mon.0@0(leader).paxos(paxos updating c 26607..27333) begin for 27334 9764 bytes 2017-03-02 15:08:24.446800 7f4f763f4700 10 mon.0@0(leader).paxos(paxos updating c 26607..27333) commit_start 27334 2017-03-02 15:08:24.446889 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:24.446901 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:24.446904 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:24.446907 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:24.446908 7f4f71beb700 20 allow all 2017-03-02 15:08:24.480263 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27333) commit_finish 27334 2017-03-02 15:08:24.480313 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:24.480334 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12970) refresh 2017-03-02 15:08:24.480337 7f4f763f4700 10 mon.0@0(leader).pg v12969 update_from_paxos read_incremental 2017-03-02 15:08:24.480361 7f4f763f4700 20 mon.0@0(leader).pg v12969 refreshing pg 1.0 7:496378 active+clean 2017-03-02 15:08:24.480377 7f4f763f4700 20 mon.0@0(leader).pg v12969 refreshing pg 1.1 7:586650 active+clean 2017-03-02 15:08:24.480385 7f4f763f4700 20 mon.0@0(leader).pg v12969 refreshing pg 1.2 7:577014 active+clean 2017-03-02 15:08:24.480393 7f4f763f4700 20 mon.0@0(leader).pg v12969 refreshing pg 1.3 7:499753 active+clean 2017-03-02 15:08:24.480400 7f4f763f4700 20 mon.0@0(leader).pg v12969 refreshing pg 1.4 7:486067 active+clean 2017-03-02 15:08:24.480407 7f4f763f4700 20 mon.0@0(leader).pg v12969 refreshing pg 1.5 7:567540 active+clean 2017-03-02 15:08:24.480415 7f4f763f4700 20 mon.0@0(leader).pg v12969 refreshing pg 1.6 7:502888 active+clean 2017-03-02 15:08:24.480422 7f4f763f4700 20 mon.0@0(leader).pg v12969 refreshing pg 1.7 7:529421 active+clean 2017-03-02 15:08:24.480429 7f4f763f4700 20 mon.0@0(leader).pg v12969 refreshing pg 2.0 7:1056421 active+clean 2017-03-02 15:08:24.480436 7f4f763f4700 20 mon.0@0(leader).pg v12969 refreshing pg 2.1 7:1089159 active+clean 2017-03-02 15:08:24.480443 7f4f763f4700 20 mon.0@0(leader).pg v12969 refreshing pg 2.2 7:1162918 active+clean 2017-03-02 15:08:24.480450 7f4f763f4700 20 mon.0@0(leader).pg v12969 refreshing pg 2.3 7:1095003 active+clean 2017-03-02 15:08:24.480457 7f4f763f4700 20 mon.0@0(leader).pg v12969 refreshing pg 2.4 7:1025814 active+clean 2017-03-02 15:08:24.480463 7f4f763f4700 20 mon.0@0(leader).pg v12969 refreshing pg 2.5 7:1129210 active+clean 2017-03-02 15:08:24.480470 7f4f763f4700 20 mon.0@0(leader).pg v12969 refreshing pg 2.7 7:1135083 active+clean 2017-03-02 15:08:24.480473 7f4f763f4700 20 mon.0@0(leader).pg v12969 refreshing osd.0 2017-03-02 15:08:24.480481 7f4f763f4700 20 mon.0@0(leader).pg v12969 refreshing osd.1 2017-03-02 15:08:24.480486 7f4f763f4700 20 mon.0@0(leader).pg v12969 refreshing osd.2 2017-03-02 15:08:24.480497 7f4f763f4700 10 mon.0@0(leader).pg v12970 read_pgmap_meta 2017-03-02 15:08:24.480518 7f4f763f4700 10 mon.0@0(leader).pg v12970 update_logger 2017-03-02 15:08:24.480531 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) refresh 2017-03-02 15:08:24.480547 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:24.480564 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13548) refresh 2017-03-02 15:08:24.480566 7f4f763f4700 10 mon.0@0(leader).log v13548 update_from_paxos 2017-03-02 15:08:24.480569 7f4f763f4700 10 mon.0@0(leader).log v13548 update_from_paxos version 13548 summary v 13548 2017-03-02 15:08:24.480581 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:24.480596 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:24.480598 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:24.480612 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:24.480614 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12970) post_refresh 2017-03-02 15:08:24.480616 7f4f763f4700 10 mon.0@0(leader).pg v12970 post_paxos_update 2017-03-02 15:08:24.480618 7f4f763f4700 10 mon.0@0(leader).pg v12970 check_subs 2017-03-02 15:08:24.480620 7f4f763f4700 20 mon.0@0(leader).pg v12970 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:24.480627 7f4f763f4700 20 mon.0@0(leader).pg v12970 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:24.480630 7f4f763f4700 20 mon.0@0(leader).pg v12970 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:24.480632 7f4f763f4700 20 mon.0@0(leader).pg v12970 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:24.480635 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) post_refresh 2017-03-02 15:08:24.480637 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:24.480638 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13548) post_refresh 2017-03-02 15:08:24.480639 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:24.480641 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:24.480642 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:24.480644 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27334) commit_proposal 2017-03-02 15:08:24.480648 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12970) _active - not active 2017-03-02 15:08:24.480650 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27334) finish_round 2017-03-02 15:08:24.480652 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27334) finish_round waiting_for_acting 2017-03-02 15:08:24.480654 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12970) _active 2017-03-02 15:08:24.480655 7f4f763f4700 7 mon.0@0(leader).paxosservice(pgmap 12269..12970) _active creating new pending 2017-03-02 15:08:24.480663 7f4f763f4700 10 mon.0@0(leader).pg v12970 create_pending v 12971 2017-03-02 15:08:24.480669 7f4f763f4700 7 mon.0@0(leader).pg v12970 _updated_stats for osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:24.480674 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a16475f00 0x558a1649e000 pg_stats_ack(5 pgs tid 6797) v1 2017-03-02 15:08:24.480677 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6800/26531 -- pg_stats_ack(5 pgs tid 6797) v1 -- 0x558a1649e000 con 0 2017-03-02 15:08:24.480713 7f4f763f4700 7 mon.0@0(leader).pg v12970 _updated_stats for osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:24.480718 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a1543af00 0x558a15e1c000 pg_stats_ack(8 pgs tid 6797) v1 2017-03-02 15:08:24.480721 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6800/26536 -- pg_stats_ack(8 pgs tid 6797) v1 -- 0x558a15e1c000 con 0 2017-03-02 15:08:24.480750 7f4f763f4700 7 mon.0@0(leader).pg v12970 _updated_stats for osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:24.480755 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a158d6d00 0x558a15560c00 pg_stats_ack(2 pgs tid 6797) v1 2017-03-02 15:08:24.480757 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6800/26515 -- pg_stats_ack(2 pgs tid 6797) v1 -- 0x558a15560c00 con 0 2017-03-02 15:08:24.480781 7f4f763f4700 10 mon.0@0(leader).pg v12970 check_osd_map already seen 7 >= 7 2017-03-02 15:08:24.480784 7f4f763f4700 10 mon.0@0(leader).pg v12970 update_logger 2017-03-02 15:08:24.480799 7f4f763f4700 0 log_channel(cluster) log [INF] : pgmap v12970: 32 pgs: 32 active+clean; 88297 MB data, 165 GB used, 206 GB / 372 GB avail; 157 kB/s rd, 18892 kB/s wr, 961 op/s 2017-03-02 15:08:24.480814 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14111 at 2017-03-02 15:08:24.480801) v1 -- 0x558a16742000 con 0 2017-03-02 15:08:24.480849 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27334) finish_round waiting_for_readable 2017-03-02 15:08:24.480853 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27334) finish_round waiting_for_writeable 2017-03-02 15:08:24.480855 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27334) finish_round done w/ waiters, state active 2017-03-02 15:08:24.480873 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14111 at 2017-03-02 15:08:24.480801) v1 ==== 0+0+0 (0 0 0) 0x558a16742000 con 0x558a14482000 2017-03-02 15:08:24.480892 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:24.480897 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:24.480899 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13548) dispatch 0x558a16742000 log(1 entries from seq 14111 at 2017-03-02 15:08:24.480801) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:24.480907 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27334) is_readable = 1 - now=2017-03-02 15:08:24.480907 lease_expire=0.000000 has v0 lc 27334 2017-03-02 15:08:24.480913 7f4f71beb700 10 mon.0@0(leader).log v13548 preprocess_query log(1 entries from seq 14111 at 2017-03-02 15:08:24.480801) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:24.480920 7f4f71beb700 10 mon.0@0(leader).log v13548 preprocess_log log(1 entries from seq 14111 at 2017-03-02 15:08:24.480801) v1 from mon.0 2017-03-02 15:08:24.480925 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:24.480927 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:24.480928 7f4f71beb700 20 allow all 2017-03-02 15:08:24.480935 7f4f71beb700 10 mon.0@0(leader).log v13548 prepare_update log(1 entries from seq 14111 at 2017-03-02 15:08:24.480801) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:24.480941 7f4f71beb700 10 mon.0@0(leader).log v13548 prepare_log log(1 entries from seq 14111 at 2017-03-02 15:08:24.480801) v1 from mon.0 2017-03-02 15:08:24.480944 7f4f71beb700 10 mon.0@0(leader).log v13548 logging 2017-03-02 15:08:24.480801 mon.0 172.21.15.17:6789/0 14111 : cluster [INF] pgmap v12970: 32 pgs: 32 active+clean; 88297 MB data, 165 GB used, 206 GB / 372 GB avail; 157 kB/s rd, 18892 kB/s wr, 961 op/s 2017-03-02 15:08:24.480951 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13548) setting proposal_timer 0x558a1412e7a0 with delay of 0.99933 2017-03-02 15:08:25.480383 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13548) propose_pending 2017-03-02 15:08:25.480404 7f4f743f0700 10 mon.0@0(leader).log v13548 encode_full log v 13548 2017-03-02 15:08:25.480463 7f4f743f0700 10 mon.0@0(leader).log v13548 encode_pending v13549 2017-03-02 15:08:25.480470 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27334) queue_pending_finisher 0x558a1412ea10 2017-03-02 15:08:25.480473 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27334) trigger_propose active, proposing now 2017-03-02 15:08:25.480479 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27334) propose_pending 27335 10290 bytes 2017-03-02 15:08:25.480481 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27334) begin for 27335 10290 bytes 2017-03-02 15:08:25.513610 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27334) commit_start 27335 2017-03-02 15:08:25.547156 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27334) commit_finish 27335 2017-03-02 15:08:25.547223 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:25.547258 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12970) refresh 2017-03-02 15:08:25.547268 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) refresh 2017-03-02 15:08:25.547279 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:25.547296 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13549) refresh 2017-03-02 15:08:25.547297 7f4f763f4700 10 mon.0@0(leader).log v13549 update_from_paxos 2017-03-02 15:08:25.547299 7f4f763f4700 10 mon.0@0(leader).log v13549 update_from_paxos version 13549 summary v 13548 2017-03-02 15:08:25.547303 7f4f763f4700 10 mon.0@0(leader).log v13549 update_from_paxos latest full 13548 2017-03-02 15:08:25.547311 7f4f763f4700 7 mon.0@0(leader).log v13549 update_from_paxos applying incremental log 13549 2017-03-02 15:08:24.480801 mon.0 172.21.15.17:6789/0 14111 : cluster [INF] pgmap v12970: 32 pgs: 32 active+clean; 88297 MB data, 165 GB used, 206 GB / 372 GB avail; 157 kB/s rd, 18892 kB/s wr, 961 op/s 2017-03-02 15:08:25.547324 7f4f763f4700 20 mon.0@0(leader).log v13549 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:25.547339 7f4f763f4700 15 mon.0@0(leader).log v13549 update_from_paxos logging for 1 channels 2017-03-02 15:08:25.547345 7f4f763f4700 15 mon.0@0(leader).log v13549 update_from_paxos channel 'cluster' logging 202 bytes 2017-03-02 15:08:25.547393 7f4f763f4700 10 mon.0@0(leader).log v13549 check_subs 2017-03-02 15:08:25.547426 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:25.547442 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:25.547448 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:25.547458 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:25.547460 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12970) post_refresh 2017-03-02 15:08:25.547462 7f4f763f4700 10 mon.0@0(leader).pg v12970 post_paxos_update 2017-03-02 15:08:25.547464 7f4f763f4700 10 mon.0@0(leader).pg v12970 check_subs 2017-03-02 15:08:25.547466 7f4f763f4700 20 mon.0@0(leader).pg v12970 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:25.547470 7f4f763f4700 20 mon.0@0(leader).pg v12970 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:25.547472 7f4f763f4700 20 mon.0@0(leader).pg v12970 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:25.547474 7f4f763f4700 20 mon.0@0(leader).pg v12970 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:25.547476 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) post_refresh 2017-03-02 15:08:25.547477 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:25.547478 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13549) post_refresh 2017-03-02 15:08:25.547479 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:25.547480 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:25.547481 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:25.547483 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27335) commit_proposal 2017-03-02 15:08:25.547486 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13549) _active - not active 2017-03-02 15:08:25.547487 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27335) finish_round 2017-03-02 15:08:25.547489 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27335) finish_round waiting_for_acting 2017-03-02 15:08:25.547490 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13549) _active 2017-03-02 15:08:25.547491 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13549) _active creating new pending 2017-03-02 15:08:25.547498 7f4f763f4700 10 mon.0@0(leader).log v13549 create_pending v 13550 2017-03-02 15:08:25.547501 7f4f763f4700 7 mon.0@0(leader).log v13549 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:25.547505 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a14985900 0x558a15a2cd20 log(last 14111) v1 2017-03-02 15:08:25.547508 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14111) v1 -- 0x558a15a2cd20 con 0 2017-03-02 15:08:25.547524 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27335) finish_round waiting_for_readable 2017-03-02 15:08:25.547526 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27335) finish_round waiting_for_writeable 2017-03-02 15:08:25.547527 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27335) finish_round done w/ waiters, state active 2017-03-02 15:08:25.547568 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14111) v1 ==== 0+0+0 (0 0 0) 0x558a15a2cd20 con 0x558a14482000 2017-03-02 15:08:25.547586 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:25.547590 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:25.547592 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:25.547594 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:25.547595 7f4f71beb700 20 allow all 2017-03-02 15:08:26.011710 7f4f743f0700 11 mon.0@0(leader) e2 tick 2017-03-02 15:08:26.011738 7f4f743f0700 10 mon.0@0(leader).pg v12970 v12970: 32 pgs: 32 active+clean; 88297 MB data, 165 GB used, 206 GB / 372 GB avail; 157 kB/s rd, 18892 kB/s wr, 961 op/s 2017-03-02 15:08:26.011758 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12970) maybe_trim trim_to 12470 would only trim 201 < paxos_service_trim_min 250 2017-03-02 15:08:26.011763 7f4f743f0700 10 mon.0@0(leader).mds e734 e734: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:26.011799 7f4f743f0700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) maybe_trim trim_to 234 would only trim 233 < paxos_service_trim_min 250 2017-03-02 15:08:26.011802 7f4f743f0700 10 mon.0@0(leader).osd e7 e7: 4 osds: 4 up, 4 in 2017-03-02 15:08:26.011807 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27335) is_readable = 1 - now=2017-03-02 15:08:26.011808 lease_expire=0.000000 has v0 lc 27335 2017-03-02 15:08:26.011816 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27335) is_readable = 1 - now=2017-03-02 15:08:26.011817 lease_expire=0.000000 has v0 lc 27335 2017-03-02 15:08:26.011841 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27335) is_readable = 1 - now=2017-03-02 15:08:26.011842 lease_expire=0.000000 has v0 lc 27335 2017-03-02 15:08:26.011847 7f4f743f0700 10 mon.0@0(leader).osd e7 min_last_epoch_clean 6 2017-03-02 15:08:26.011849 7f4f743f0700 10 mon.0@0(leader).log v13549 log 2017-03-02 15:08:26.011852 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13549) maybe_trim trim_to 13049 would only trim 22 < paxos_service_trim_min 250 2017-03-02 15:08:26.011856 7f4f743f0700 10 mon.0@0(leader).auth v10 auth 2017-03-02 15:08:26.011860 7f4f743f0700 10 cephx keyserver: _check_rotating_secrets 2017-03-02 15:08:26.011870 7f4f743f0700 20 mon.0@0(leader) e2 sync_trim_providers 2017-03-02 15:08:26.870116 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.5 172.21.15.11:6804/1637448123 8785 ==== mdsbeacon(4116/e up:active seq 8498 v734) v7 ==== 365+0+0 (2065558700 0 0) 0x558a148d8d80 con 0x558a1471d800 2017-03-02 15:08:26.870145 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b0f20 for mds.? 172.21.15.11:6804/1637448123 2017-03-02 15:08:26.870152 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:26.870155 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) dispatch 0x558a148d8d80 mdsbeacon(4116/e up:active seq 8498 v734) v7 from mds.5 172.21.15.11:6804/1637448123 con 0x558a1471d800 2017-03-02 15:08:26.870160 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27335) is_readable = 1 - now=2017-03-02 15:08:26.870161 lease_expire=0.000000 has v0 lc 27335 2017-03-02 15:08:26.870167 7f4f71beb700 10 mon.0@0(leader).mds e734 preprocess_query mdsbeacon(4116/e up:active seq 8498 v734) v7 from mds.5 172.21.15.11:6804/1637448123 2017-03-02 15:08:26.870173 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:26.870176 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:26.870177 7f4f71beb700 20 allow all 2017-03-02 15:08:26.870178 7f4f71beb700 12 mon.0@0(leader).mds e734 preprocess_beacon mdsbeacon(4116/e up:active seq 8498 v734) v7 from mds.5 172.21.15.11:6804/1637448123 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:26.870185 7f4f71beb700 10 mon.0@0(leader).mds e734 preprocess_beacon: GID exists in map: 4116 2017-03-02 15:08:26.870190 7f4f71beb700 15 mon.0@0(leader).mds e734 _note_beacon mdsbeacon(4116/e up:active seq 8498 v734) v7 noting time 2017-03-02 15:08:26.870194 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a14955e00 0x558a143e56c0 mdsbeacon(4116/e up:active seq 8498 v734) v7 2017-03-02 15:08:26.870196 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6804/1637448123 -- mdsbeacon(4116/e up:active seq 8498 v734) v7 -- 0x558a143e56c0 con 0 2017-03-02 15:08:26.883919 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.3 172.21.15.11:6805/1926402648 8561 ==== mdsbeacon(4118/f up:active seq 8498 v734) v7 ==== 406+0+0 (1848893933 0 0) 0x558a143e7a80 con 0x558a14905800 2017-03-02 15:08:26.883947 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b2260 for mds.? 172.21.15.11:6805/1926402648 2017-03-02 15:08:26.883953 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:26.883956 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) dispatch 0x558a143e7a80 mdsbeacon(4118/f up:active seq 8498 v734) v7 from mds.3 172.21.15.11:6805/1926402648 con 0x558a14905800 2017-03-02 15:08:26.883961 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27335) is_readable = 1 - now=2017-03-02 15:08:26.883961 lease_expire=0.000000 has v0 lc 27335 2017-03-02 15:08:26.883968 7f4f71beb700 10 mon.0@0(leader).mds e734 preprocess_query mdsbeacon(4118/f up:active seq 8498 v734) v7 from mds.3 172.21.15.11:6805/1926402648 2017-03-02 15:08:26.883974 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:26.883976 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:26.883977 7f4f71beb700 20 allow all 2017-03-02 15:08:26.883978 7f4f71beb700 12 mon.0@0(leader).mds e734 preprocess_beacon mdsbeacon(4118/f up:active seq 8498 v734) v7 from mds.3 172.21.15.11:6805/1926402648 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:26.883986 7f4f71beb700 10 mon.0@0(leader).mds e734 preprocess_beacon: GID exists in map: 4118 2017-03-02 15:08:26.883990 7f4f71beb700 15 mon.0@0(leader).mds e734 _note_beacon mdsbeacon(4118/f up:active seq 8498 v734) v7 noting time 2017-03-02 15:08:26.883994 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a14955e00 0x558a148d8d80 mdsbeacon(4118/f up:active seq 8498 v734) v7 2017-03-02 15:08:26.883997 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6805/1926402648 -- mdsbeacon(4118/f up:active seq 8498 v734) v7 -- 0x558a148d8d80 con 0 2017-03-02 15:08:26.884690 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.2 172.21.15.22:6804/673088617 8560 ==== mdsbeacon(4119/c up:active seq 8498 v734) v7 ==== 365+0+0 (1527048434 0 0) 0x558a143e56c0 con 0x558a14907000 2017-03-02 15:08:26.884709 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14916f20 for mds.? 172.21.15.22:6804/673088617 2017-03-02 15:08:26.884713 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:26.884715 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) dispatch 0x558a143e56c0 mdsbeacon(4119/c up:active seq 8498 v734) v7 from mds.2 172.21.15.22:6804/673088617 con 0x558a14907000 2017-03-02 15:08:26.884720 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27335) is_readable = 1 - now=2017-03-02 15:08:26.884720 lease_expire=0.000000 has v0 lc 27335 2017-03-02 15:08:26.884725 7f4f71beb700 10 mon.0@0(leader).mds e734 preprocess_query mdsbeacon(4119/c up:active seq 8498 v734) v7 from mds.2 172.21.15.22:6804/673088617 2017-03-02 15:08:26.884730 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:26.884733 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:26.884734 7f4f71beb700 20 allow all 2017-03-02 15:08:26.884735 7f4f71beb700 12 mon.0@0(leader).mds e734 preprocess_beacon mdsbeacon(4119/c up:active seq 8498 v734) v7 from mds.2 172.21.15.22:6804/673088617 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:26.884741 7f4f71beb700 10 mon.0@0(leader).mds e734 preprocess_beacon: GID exists in map: 4119 2017-03-02 15:08:26.884744 7f4f71beb700 15 mon.0@0(leader).mds e734 _note_beacon mdsbeacon(4119/c up:active seq 8498 v734) v7 noting time 2017-03-02 15:08:26.884747 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a14955e00 0x558a143e7a80 mdsbeacon(4119/c up:active seq 8498 v734) v7 2017-03-02 15:08:26.884750 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6804/673088617 -- mdsbeacon(4119/c up:active seq 8498 v734) v7 -- 0x558a143e7a80 con 0 2017-03-02 15:08:26.884912 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.0 172.21.15.22:6805/2203629154 8610 ==== mdsbeacon(4121/d up:active seq 8494 v734) v7 ==== 454+0+0 (3956203460 0 0) 0x558a148d8d80 con 0x558a1492a000 2017-03-02 15:08:26.884928 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a149195a0 for mds.? 172.21.15.22:6805/2203629154 2017-03-02 15:08:26.884932 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:26.884935 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) dispatch 0x558a148d8d80 mdsbeacon(4121/d up:active seq 8494 v734) v7 from mds.0 172.21.15.22:6805/2203629154 con 0x558a1492a000 2017-03-02 15:08:26.884939 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27335) is_readable = 1 - now=2017-03-02 15:08:26.884939 lease_expire=0.000000 has v0 lc 27335 2017-03-02 15:08:26.884947 7f4f71beb700 10 mon.0@0(leader).mds e734 preprocess_query mdsbeacon(4121/d up:active seq 8494 v734) v7 from mds.0 172.21.15.22:6805/2203629154 2017-03-02 15:08:26.884952 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:26.884954 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:26.884955 7f4f71beb700 20 allow all 2017-03-02 15:08:26.884956 7f4f71beb700 12 mon.0@0(leader).mds e734 preprocess_beacon mdsbeacon(4121/d up:active seq 8494 v734) v7 from mds.0 172.21.15.22:6805/2203629154 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:26.884961 7f4f71beb700 10 mon.0@0(leader).mds e734 preprocess_beacon: GID exists in map: 4121 2017-03-02 15:08:26.884965 7f4f71beb700 20 mon.0@0(leader).mds e734 preprocess_beacon health metrics for gid 4121 were updated 2017-03-02 15:08:26.884966 7f4f71beb700 15 mon.0@0(leader).mds e734 _note_beacon mdsbeacon(4121/d up:active seq 8494 v734) v7 noting time 2017-03-02 15:08:26.884970 7f4f71beb700 7 mon.0@0(leader).mds e734 prepare_update mdsbeacon(4121/d up:active seq 8494 v734) v7 2017-03-02 15:08:26.884974 7f4f71beb700 12 mon.0@0(leader).mds e734 prepare_beacon mdsbeacon(4121/d up:active seq 8494 v734) v7 from mds.0 172.21.15.22:6805/2203629154 2017-03-02 15:08:26.884978 7f4f71beb700 20 mon.0@0(leader).mds e734 prepare_beacon got health from gid 4121 with 1 metrics. 2017-03-02 15:08:26.884981 7f4f71beb700 10 mon.0@0(leader).mds e734 prepare_beacon mds.0 up:active -> up:active standby_for_rank=-1 2017-03-02 15:08:26.884984 7f4f71beb700 7 mon.0@0(leader).mds e734 prepare_beacon pending map now: 2017-03-02 15:08:26.884986 7f4f71beb700 7 mon.0@0(leader).mds e734 print_map 2017-03-02 15:08:26.885024 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) setting proposal_timer 0x558a1412e400 with delay of 0.05 2017-03-02 15:08:26.924130 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.1 172.21.15.17:6804/2576824570 8557 ==== mdsbeacon(4120/g up:active seq 8498 v734) v7 ==== 365+0+0 (2467014121 0 0) 0x558a154b8d00 con 0x558a14908800 2017-03-02 15:08:26.924159 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14917600 for mds.? 172.21.15.17:6804/2576824570 2017-03-02 15:08:26.924166 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:26.924169 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) dispatch 0x558a154b8d00 mdsbeacon(4120/g up:active seq 8498 v734) v7 from mds.1 172.21.15.17:6804/2576824570 con 0x558a14908800 2017-03-02 15:08:26.924175 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27335) is_readable = 1 - now=2017-03-02 15:08:26.924175 lease_expire=0.000000 has v0 lc 27335 2017-03-02 15:08:26.924183 7f4f71beb700 10 mon.0@0(leader).mds e734 preprocess_query mdsbeacon(4120/g up:active seq 8498 v734) v7 from mds.1 172.21.15.17:6804/2576824570 2017-03-02 15:08:26.924190 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:26.924192 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:26.924193 7f4f71beb700 20 allow all 2017-03-02 15:08:26.924195 7f4f71beb700 12 mon.0@0(leader).mds e734 preprocess_beacon mdsbeacon(4120/g up:active seq 8498 v734) v7 from mds.1 172.21.15.17:6804/2576824570 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:26.924202 7f4f71beb700 10 mon.0@0(leader).mds e734 preprocess_beacon: GID exists in map: 4120 2017-03-02 15:08:26.924206 7f4f71beb700 10 mon.0@0(leader).mds e734 mds_beacon mdsbeacon(4120/g up:active seq 8498 v734) v7 ignoring requested state, because mds hasn't seen latest map 2017-03-02 15:08:26.924208 7f4f71beb700 15 mon.0@0(leader).mds e734 _note_beacon mdsbeacon(4120/g up:active seq 8498 v734) v7 noting time 2017-03-02 15:08:26.924212 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a163ba000 0x558a143e56c0 mdsbeacon(4120/g up:active seq 8498 v735) v7 2017-03-02 15:08:26.924215 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6804/2576824570 -- mdsbeacon(4120/g up:active seq 8498 v735) v7 -- 0x558a143e56c0 con 0 2017-03-02 15:08:26.935099 7f4f743f0700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) propose_pending 2017-03-02 15:08:26.935134 7f4f743f0700 10 mon.0@0(leader).mds e734 encode_pending e735 2017-03-02 15:08:26.935192 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27335) queue_pending_finisher 0x558a1412ff20 2017-03-02 15:08:26.935199 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27335) trigger_propose active, proposing now 2017-03-02 15:08:26.935208 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27335) propose_pending 27336 2411 bytes 2017-03-02 15:08:26.935210 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27335) begin for 27336 2411 bytes 2017-03-02 15:08:26.963629 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27335) commit_start 27336 2017-03-02 15:08:26.969196 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.4 172.21.15.26:6805/2033388711 8555 ==== mdsbeacon(4117/b up:active seq 8498 v734) v7 ==== 365+0+0 (1839381290 0 0) 0x558a143e56c0 con 0x558a14904000 2017-03-02 15:08:26.969223 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b1600 for mds.? 172.21.15.26:6805/2033388711 2017-03-02 15:08:26.969230 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:26.969233 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..734) dispatch 0x558a143e56c0 mdsbeacon(4117/b up:active seq 8498 v734) v7 from mds.4 172.21.15.26:6805/2033388711 con 0x558a14904000 2017-03-02 15:08:26.969237 7f4f71beb700 5 mon.0@0(leader).paxos(paxos writing c 26607..27335) is_readable = 1 - now=2017-03-02 15:08:26.969237 lease_expire=0.000000 has v0 lc 27335 2017-03-02 15:08:26.969242 7f4f71beb700 10 mon.0@0(leader).mds e734 preprocess_query mdsbeacon(4117/b up:active seq 8498 v734) v7 from mds.4 172.21.15.26:6805/2033388711 2017-03-02 15:08:26.969247 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:26.969249 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:26.969250 7f4f71beb700 20 allow all 2017-03-02 15:08:26.969250 7f4f71beb700 12 mon.0@0(leader).mds e734 preprocess_beacon mdsbeacon(4117/b up:active seq 8498 v734) v7 from mds.4 172.21.15.26:6805/2033388711 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:26.969255 7f4f71beb700 10 mon.0@0(leader).mds e734 preprocess_beacon: GID exists in map: 4117 2017-03-02 15:08:26.969258 7f4f71beb700 10 mon.0@0(leader).mds e734 mds_beacon mdsbeacon(4117/b up:active seq 8498 v734) v7 ignoring requested state, because mds hasn't seen latest map 2017-03-02 15:08:26.969259 7f4f71beb700 15 mon.0@0(leader).mds e734 _note_beacon mdsbeacon(4117/b up:active seq 8498 v734) v7 noting time 2017-03-02 15:08:26.969262 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a163ba000 0x558a154b8d00 mdsbeacon(4117/b up:active seq 8498 v735) v7 2017-03-02 15:08:26.969264 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6805/2033388711 -- mdsbeacon(4117/b up:active seq 8498 v735) v7 -- 0x558a154b8d00 con 0 2017-03-02 15:08:26.977532 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27335) commit_finish 27336 2017-03-02 15:08:26.977575 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:26.977599 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12970) refresh 2017-03-02 15:08:26.977618 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..735) refresh 2017-03-02 15:08:26.977620 7f4f763f4700 10 mon.0@0(leader).mds e734 update_from_paxos version 735, my e 734 2017-03-02 15:08:26.977629 7f4f763f4700 10 mon.0@0(leader).mds e734 update_from_paxos got 735 2017-03-02 15:08:26.977673 7f4f763f4700 4 mon.0@0(leader).mds e735 new map 2017-03-02 15:08:26.977682 7f4f763f4700 0 mon.0@0(leader).mds e735 print_map 2017-03-02 15:08:26.977737 7f4f763f4700 20 mon.0@0(leader).mds e735 check_sub: mdsmap 2017-03-02 15:08:26.977747 7f4f763f4700 10 mon.0@0(leader).mds e735 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:26.977751 7f4f763f4700 10 mon.0@0(leader).mds e735 check_sub selected MDS map epoch 735 for namespace 1 for subscriber mds.? who wants epoch 735 2017-03-02 15:08:26.977770 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6804/2954782670 -- mdsmap(e 735) v1 -- 0x558a16742000 con 0 2017-03-02 15:08:26.977894 7f4f763f4700 20 mon.0@0(leader).mds e735 check_sub: mdsmap 2017-03-02 15:08:26.977908 7f4f763f4700 10 mon.0@0(leader).mds e735 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:26.977911 7f4f763f4700 10 mon.0@0(leader).mds e735 check_sub selected MDS map epoch 735 for namespace 1 for subscriber mds.? who wants epoch 735 2017-03-02 15:08:26.977927 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6804/1637448123 -- mdsmap(e 735) v1 -- 0x558a1426e1c0 con 0 2017-03-02 15:08:26.977946 7f4f763f4700 20 mon.0@0(leader).mds e735 check_sub: mdsmap 2017-03-02 15:08:26.977952 7f4f763f4700 10 mon.0@0(leader).mds e735 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:26.977955 7f4f763f4700 10 mon.0@0(leader).mds e735 check_sub selected MDS map epoch 735 for namespace 1 for subscriber mds.? who wants epoch 735 2017-03-02 15:08:26.977968 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6805/2033388711 -- mdsmap(e 735) v1 -- 0x558a1426df80 con 0 2017-03-02 15:08:26.977986 7f4f763f4700 20 mon.0@0(leader).mds e735 check_sub: mdsmap 2017-03-02 15:08:26.977991 7f4f763f4700 10 mon.0@0(leader).mds e735 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:26.977993 7f4f763f4700 10 mon.0@0(leader).mds e735 check_sub selected MDS map epoch 735 for namespace 1 for subscriber mds.? who wants epoch 735 2017-03-02 15:08:26.978007 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6805/1926402648 -- mdsmap(e 735) v1 -- 0x558a1691eb40 con 0 2017-03-02 15:08:26.978230 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.6 172.21.15.26:6804/2954782670 8747 ==== mdsbeacon(4115/a up:active seq 8498 v734) v7 ==== 365+0+0 (910708546 0 0) 0x558a15204000 con 0x558a1471f000 2017-03-02 15:08:26.978259 7f4f763f4700 20 mon.0@0(leader).mds e735 check_sub: mdsmap 2017-03-02 15:08:26.978268 7f4f763f4700 10 mon.0@0(leader).mds e735 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:26.978271 7f4f763f4700 10 mon.0@0(leader).mds e735 check_sub selected MDS map epoch 735 for namespace 1 for subscriber mds.? who wants epoch 735 2017-03-02 15:08:26.978286 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6804/673088617 -- mdsmap(e 735) v1 -- 0x558a16920d00 con 0 2017-03-02 15:08:26.978309 7f4f763f4700 20 mon.0@0(leader).mds e735 check_sub: mdsmap 2017-03-02 15:08:26.978314 7f4f763f4700 10 mon.0@0(leader).mds e735 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:26.978316 7f4f763f4700 10 mon.0@0(leader).mds e735 check_sub selected MDS map epoch 735 for namespace 1 for subscriber mds.? who wants epoch 735 2017-03-02 15:08:26.978331 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6804/2576824570 -- mdsmap(e 735) v1 -- 0x558a16742240 con 0 2017-03-02 15:08:26.978512 7f4f763f4700 20 mon.0@0(leader).mds e735 check_sub: mdsmap 2017-03-02 15:08:26.978525 7f4f763f4700 10 mon.0@0(leader).mds e735 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:26.978528 7f4f763f4700 10 mon.0@0(leader).mds e735 check_sub selected MDS map epoch 735 for namespace 1 for subscriber mds.? who wants epoch 735 2017-03-02 15:08:26.978544 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6805/2203629154 -- mdsmap(e 735) v1 -- 0x558a16742480 con 0 2017-03-02 15:08:26.978655 7f4f763f4700 20 mon.0@0(leader).mds e735 check_sub: mdsmap 2017-03-02 15:08:26.978664 7f4f763f4700 10 mon.0@0(leader).mds e735 check_sub: is_mds=0, fscid= 1 2017-03-02 15:08:26.978667 7f4f763f4700 10 mon.0@0(leader).mds e735 check_sub selected MDS map epoch 735 for namespace 1 for subscriber unknown.0 who wants epoch 735 2017-03-02 15:08:26.978683 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.132:0/2597832251 -- mdsmap(e 735) v1 -- 0x558a167426c0 con 0 2017-03-02 15:08:26.978700 7f4f763f4700 20 mon.0@0(leader).mds e735 check_sub: mdsmap 2017-03-02 15:08:26.978703 7f4f763f4700 10 mon.0@0(leader).mds e735 check_sub: is_mds=0, fscid= 1 2017-03-02 15:08:26.978705 7f4f763f4700 10 mon.0@0(leader).mds e735 check_sub selected MDS map epoch 735 for namespace 1 for subscriber unknown.0 who wants epoch 735 2017-03-02 15:08:26.978718 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.133:0/3792964262 -- mdsmap(e 735) v1 -- 0x558a16742900 con 0 2017-03-02 15:08:26.978830 7f4f763f4700 10 mon.0@0(leader).mds e735 update_logger 2017-03-02 15:08:26.978861 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:26.978884 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13549) refresh 2017-03-02 15:08:26.978891 7f4f763f4700 10 mon.0@0(leader).log v13549 update_from_paxos 2017-03-02 15:08:26.978893 7f4f763f4700 10 mon.0@0(leader).log v13549 update_from_paxos version 13549 summary v 13549 2017-03-02 15:08:26.978906 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:26.978920 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:26.978922 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:26.978935 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:26.978942 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12970) post_refresh 2017-03-02 15:08:26.978944 7f4f763f4700 10 mon.0@0(leader).pg v12970 post_paxos_update 2017-03-02 15:08:26.978946 7f4f763f4700 10 mon.0@0(leader).pg v12970 check_subs 2017-03-02 15:08:26.978948 7f4f763f4700 20 mon.0@0(leader).pg v12970 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:26.978953 7f4f763f4700 20 mon.0@0(leader).pg v12970 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:26.978956 7f4f763f4700 20 mon.0@0(leader).pg v12970 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:26.978958 7f4f763f4700 20 mon.0@0(leader).pg v12970 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:26.978960 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..735) post_refresh 2017-03-02 15:08:26.978962 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:26.978963 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13549) post_refresh 2017-03-02 15:08:26.978964 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:26.978965 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:26.978966 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:26.978968 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27336) commit_proposal 2017-03-02 15:08:26.978970 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..735) _active - not active 2017-03-02 15:08:26.978972 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27336) finish_round 2017-03-02 15:08:26.978974 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27336) finish_round waiting_for_acting 2017-03-02 15:08:26.978976 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..735) _active 2017-03-02 15:08:26.978977 7f4f763f4700 7 mon.0@0(leader).paxosservice(mdsmap 1..735) _active creating new pending 2017-03-02 15:08:26.978992 7f4f763f4700 10 mon.0@0(leader).mds e735 create_pending e736 2017-03-02 15:08:26.978996 7f4f763f4700 10 mon.0@0(leader).mds e735 _updated mds.0 mdsbeacon(4121/d up:active seq 8494 v734) v7 2017-03-02 15:08:26.979004 7f4f763f4700 0 log_channel(cluster) log [INF] : mds.0 172.21.15.22:6805/2203629154 up:active 2017-03-02 15:08:26.979016 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14112 at 2017-03-02 15:08:26.979005) v1 -- 0x558a16742d80 con 0 2017-03-02 15:08:26.979029 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a14955e00 0x558a148d7380 mdsbeacon(4121/d up:active seq 8494 v735) v7 2017-03-02 15:08:26.979032 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6805/2203629154 -- mdsbeacon(4121/d up:active seq 8494 v735) v7 -- 0x558a148d7380 con 0 2017-03-02 15:08:26.979053 7f4f763f4700 10 mon.0@0(leader).mds e735 e735: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:26.979070 7f4f763f4700 10 mon.0@0(leader).mds e735 update_logger 2017-03-02 15:08:26.979080 7f4f763f4700 0 log_channel(cluster) log [INF] : fsmap e735: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:26.979090 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14113 at 2017-03-02 15:08:26.979081) v1 -- 0x558a16742b40 con 0 2017-03-02 15:08:26.979123 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27336) finish_round waiting_for_readable 2017-03-02 15:08:26.979126 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27336) finish_round waiting_for_writeable 2017-03-02 15:08:26.979127 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27336) finish_round done w/ waiters, state active 2017-03-02 15:08:26.979141 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14868d60 for mds.? 172.21.15.26:6804/2954782670 2017-03-02 15:08:26.979149 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:26.979152 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..735) dispatch 0x558a15204000 mdsbeacon(4115/a up:active seq 8498 v734) v7 from mds.6 172.21.15.26:6804/2954782670 con 0x558a1471f000 2017-03-02 15:08:26.979157 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27336) is_readable = 1 - now=2017-03-02 15:08:26.979158 lease_expire=0.000000 has v0 lc 27336 2017-03-02 15:08:26.979164 7f4f71beb700 10 mon.0@0(leader).mds e735 preprocess_query mdsbeacon(4115/a up:active seq 8498 v734) v7 from mds.6 172.21.15.26:6804/2954782670 2017-03-02 15:08:26.979170 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:26.979172 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:26.979173 7f4f71beb700 20 allow all 2017-03-02 15:08:26.979174 7f4f71beb700 12 mon.0@0(leader).mds e735 preprocess_beacon mdsbeacon(4115/a up:active seq 8498 v734) v7 from mds.6 172.21.15.26:6804/2954782670 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:26.979181 7f4f71beb700 10 mon.0@0(leader).mds e735 preprocess_beacon: GID exists in map: 4115 2017-03-02 15:08:26.979183 7f4f71beb700 10 mon.0@0(leader).mds e735 mds_beacon mdsbeacon(4115/a up:active seq 8498 v734) v7 ignoring requested state, because mds hasn't seen latest map 2017-03-02 15:08:26.979186 7f4f71beb700 15 mon.0@0(leader).mds e735 _note_beacon mdsbeacon(4115/a up:active seq 8498 v734) v7 noting time 2017-03-02 15:08:26.979189 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a163ba000 0x558a143e56c0 mdsbeacon(4115/a up:active seq 8498 v735) v7 2017-03-02 15:08:26.979192 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6804/2954782670 -- mdsbeacon(4115/a up:active seq 8498 v735) v7 -- 0x558a143e56c0 con 0 2017-03-02 15:08:26.979273 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14112 at 2017-03-02 15:08:26.979005) v1 ==== 0+0+0 (0 0 0) 0x558a16742d80 con 0x558a14482000 2017-03-02 15:08:26.979284 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:26.979288 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:26.979290 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13549) dispatch 0x558a16742d80 log(1 entries from seq 14112 at 2017-03-02 15:08:26.979005) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:26.979295 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27336) is_readable = 1 - now=2017-03-02 15:08:26.979295 lease_expire=0.000000 has v0 lc 27336 2017-03-02 15:08:26.979300 7f4f71beb700 10 mon.0@0(leader).log v13549 preprocess_query log(1 entries from seq 14112 at 2017-03-02 15:08:26.979005) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:26.979305 7f4f71beb700 10 mon.0@0(leader).log v13549 preprocess_log log(1 entries from seq 14112 at 2017-03-02 15:08:26.979005) v1 from mon.0 2017-03-02 15:08:26.979309 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:26.979311 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:26.979312 7f4f71beb700 20 allow all 2017-03-02 15:08:26.979318 7f4f71beb700 10 mon.0@0(leader).log v13549 prepare_update log(1 entries from seq 14112 at 2017-03-02 15:08:26.979005) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:26.979323 7f4f71beb700 10 mon.0@0(leader).log v13549 prepare_log log(1 entries from seq 14112 at 2017-03-02 15:08:26.979005) v1 from mon.0 2017-03-02 15:08:26.979326 7f4f71beb700 10 mon.0@0(leader).log v13549 logging 2017-03-02 15:08:26.979005 mon.0 172.21.15.17:6789/0 14112 : cluster [INF] mds.0 172.21.15.22:6805/2203629154 up:active 2017-03-02 15:08:26.979334 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13549) setting proposal_timer 0x558a1412cfa0 with delay of 0.998215 2017-03-02 15:08:26.979353 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14113 at 2017-03-02 15:08:26.979081) v1 ==== 0+0+0 (0 0 0) 0x558a16742b40 con 0x558a14482000 2017-03-02 15:08:26.979361 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:26.979364 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:26.979366 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13549) dispatch 0x558a16742b40 log(1 entries from seq 14113 at 2017-03-02 15:08:26.979081) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:26.979371 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27336) is_readable = 1 - now=2017-03-02 15:08:26.979371 lease_expire=0.000000 has v0 lc 27336 2017-03-02 15:08:26.979375 7f4f71beb700 10 mon.0@0(leader).log v13549 preprocess_query log(1 entries from seq 14113 at 2017-03-02 15:08:26.979081) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:26.979405 7f4f71beb700 10 mon.0@0(leader).log v13549 preprocess_log log(1 entries from seq 14113 at 2017-03-02 15:08:26.979081) v1 from mon.0 2017-03-02 15:08:26.979409 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:26.979410 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:26.979411 7f4f71beb700 20 allow all 2017-03-02 15:08:26.979414 7f4f71beb700 10 mon.0@0(leader).log v13549 prepare_update log(1 entries from seq 14113 at 2017-03-02 15:08:26.979081) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:26.979419 7f4f71beb700 10 mon.0@0(leader).log v13549 prepare_log log(1 entries from seq 14113 at 2017-03-02 15:08:26.979081) v1 from mon.0 2017-03-02 15:08:26.979421 7f4f71beb700 10 mon.0@0(leader).log v13549 logging 2017-03-02 15:08:26.979081 mon.0 172.21.15.17:6789/0 14113 : cluster [INF] fsmap e735: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:26.979428 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13549) proposal_timer already set 2017-03-02 15:08:27.977627 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13549) propose_pending 2017-03-02 15:08:27.977645 7f4f743f0700 10 mon.0@0(leader).log v13549 encode_full log v 13549 2017-03-02 15:08:27.977694 7f4f743f0700 10 mon.0@0(leader).log v13549 encode_pending v13550 2017-03-02 15:08:27.977705 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27336) queue_pending_finisher 0x558a1412e400 2017-03-02 15:08:27.977708 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27336) trigger_propose active, proposing now 2017-03-02 15:08:27.977712 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27336) propose_pending 27337 10493 bytes 2017-03-02 15:08:27.977713 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27336) begin for 27337 10493 bytes 2017-03-02 15:08:28.005317 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27336) commit_start 27337 2017-03-02 15:08:28.038767 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27336) commit_finish 27337 2017-03-02 15:08:28.038804 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:28.038837 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12970) refresh 2017-03-02 15:08:28.038863 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..735) refresh 2017-03-02 15:08:28.038883 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:28.038902 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13550) refresh 2017-03-02 15:08:28.038909 7f4f763f4700 10 mon.0@0(leader).log v13550 update_from_paxos 2017-03-02 15:08:28.038912 7f4f763f4700 10 mon.0@0(leader).log v13550 update_from_paxos version 13550 summary v 13549 2017-03-02 15:08:28.038917 7f4f763f4700 10 mon.0@0(leader).log v13550 update_from_paxos latest full 13549 2017-03-02 15:08:28.038927 7f4f763f4700 7 mon.0@0(leader).log v13550 update_from_paxos applying incremental log 13550 2017-03-02 15:08:26.979005 mon.0 172.21.15.17:6789/0 14112 : cluster [INF] mds.0 172.21.15.22:6805/2203629154 up:active 2017-03-02 15:08:28.038937 7f4f763f4700 20 mon.0@0(leader).log v13550 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:28.038949 7f4f763f4700 7 mon.0@0(leader).log v13550 update_from_paxos applying incremental log 13550 2017-03-02 15:08:26.979081 mon.0 172.21.15.17:6789/0 14113 : cluster [INF] fsmap e735: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:28.038952 7f4f763f4700 20 mon.0@0(leader).log v13550 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:28.038956 7f4f763f4700 15 mon.0@0(leader).log v13550 update_from_paxos logging for 1 channels 2017-03-02 15:08:28.038958 7f4f763f4700 15 mon.0@0(leader).log v13550 update_from_paxos channel 'cluster' logging 316 bytes 2017-03-02 15:08:28.038994 7f4f763f4700 10 mon.0@0(leader).log v13550 check_subs 2017-03-02 15:08:28.039018 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:28.039031 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:28.039035 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:28.039043 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:28.039045 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12970) post_refresh 2017-03-02 15:08:28.039046 7f4f763f4700 10 mon.0@0(leader).pg v12970 post_paxos_update 2017-03-02 15:08:28.039047 7f4f763f4700 10 mon.0@0(leader).pg v12970 check_subs 2017-03-02 15:08:28.039049 7f4f763f4700 20 mon.0@0(leader).pg v12970 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:28.039051 7f4f763f4700 20 mon.0@0(leader).pg v12970 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:28.039053 7f4f763f4700 20 mon.0@0(leader).pg v12970 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:28.039055 7f4f763f4700 20 mon.0@0(leader).pg v12970 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:28.039056 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..735) post_refresh 2017-03-02 15:08:28.039057 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:28.039058 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13550) post_refresh 2017-03-02 15:08:28.039059 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:28.039059 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:28.039060 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:28.039061 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27337) commit_proposal 2017-03-02 15:08:28.039063 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13550) _active - not active 2017-03-02 15:08:28.039065 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27337) finish_round 2017-03-02 15:08:28.039066 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27337) finish_round waiting_for_acting 2017-03-02 15:08:28.039067 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13550) _active 2017-03-02 15:08:28.039068 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13550) _active creating new pending 2017-03-02 15:08:28.039074 7f4f763f4700 10 mon.0@0(leader).log v13550 create_pending v 13551 2017-03-02 15:08:28.039076 7f4f763f4700 7 mon.0@0(leader).log v13550 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:28.039079 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a163ba000 0x558a144a4f00 log(last 14112) v1 2017-03-02 15:08:28.039081 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14112) v1 -- 0x558a144a4f00 con 0 2017-03-02 15:08:28.039094 7f4f763f4700 7 mon.0@0(leader).log v13550 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:28.039096 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a151a7600 0x558a15a4ed00 log(last 14113) v1 2017-03-02 15:08:28.039097 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14113) v1 -- 0x558a15a4ed00 con 0 2017-03-02 15:08:28.039100 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27337) finish_round waiting_for_readable 2017-03-02 15:08:28.039102 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27337) finish_round waiting_for_writeable 2017-03-02 15:08:28.039103 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27337) finish_round done w/ waiters, state active 2017-03-02 15:08:28.039116 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14112) v1 ==== 0+0+0 (0 0 0) 0x558a144a4f00 con 0x558a14482000 2017-03-02 15:08:28.039132 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:28.039137 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:28.039140 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:28.039142 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:28.039144 7f4f71beb700 20 allow all 2017-03-02 15:08:28.039157 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14113) v1 ==== 0+0+0 (0 0 0) 0x558a15a4ed00 con 0x558a14482000 2017-03-02 15:08:28.039163 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:28.039165 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:28.039167 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:28.039168 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:28.039169 7f4f71beb700 20 allow all 2017-03-02 15:08:28.262802 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.3 172.21.15.17:6800/27100 6834 ==== pg_stats(1 pgs tid 6798 v 0) v1 ==== 706+0+0 (873372303 0 0) 0x558a14d98000 con 0x558a145d7000 2017-03-02 15:08:28.262847 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415f440 for osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:28.262854 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:28.262856 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12970) dispatch 0x558a14d98000 pg_stats(1 pgs tid 6798 v 0) v1 from osd.3 172.21.15.17:6800/27100 con 0x558a145d7000 2017-03-02 15:08:28.262862 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27337) is_readable = 1 - now=2017-03-02 15:08:28.262862 lease_expire=0.000000 has v0 lc 27337 2017-03-02 15:08:28.262869 7f4f71beb700 10 mon.0@0(leader).pg v12970 preprocess_query pg_stats(1 pgs tid 6798 v 0) v1 from osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:28.262875 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:28.262878 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:28.262879 7f4f71beb700 20 allow all 2017-03-02 15:08:28.262881 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27337) is_readable = 1 - now=2017-03-02 15:08:28.262881 lease_expire=0.000000 has v0 lc 27337 2017-03-02 15:08:28.262886 7f4f71beb700 10 mon.0@0(leader).pg v12970 prepare_update pg_stats(1 pgs tid 6798 v 0) v1 from osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:28.262890 7f4f71beb700 10 mon.0@0(leader).pg v12970 prepare_pg_stats pg_stats(1 pgs tid 6798 v 0) v1 from osd.3 2017-03-02 15:08:28.262896 7f4f71beb700 10 mon.0@0(leader).pg v12970 got osd.3 osd_stat(41976 MB used, 53365 MB avail, 95341 MB total, peers [0,1,2] op hist [0,1]) (was osd_stat(42019 MB used, 53321 MB avail, 95341 MB total, peers [0,1,2] op hist [])) 2017-03-02 15:08:28.262904 7f4f71beb700 15 mon.0@0(leader).pg v12970 got 2.6 reported at 7:964314 state active+clean -> active+clean 2017-03-02 15:08:28.262913 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12970) setting proposal_timer 0x558a1412e880 with delay of 0.775868 2017-03-02 15:08:28.464693 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.1 172.21.15.22:6800/26531 6834 ==== pg_stats(5 pgs tid 6798 v 0) v1 ==== 2970+0+0 (1981840968 0 0) 0x558a15ca2c80 con 0x558a14643800 2017-03-02 15:08:28.464723 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1463b1e0 for osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:28.464730 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:28.464733 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12970) dispatch 0x558a15ca2c80 pg_stats(5 pgs tid 6798 v 0) v1 from osd.1 172.21.15.22:6800/26531 con 0x558a14643800 2017-03-02 15:08:28.464739 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27337) is_readable = 1 - now=2017-03-02 15:08:28.464739 lease_expire=0.000000 has v0 lc 27337 2017-03-02 15:08:28.464747 7f4f71beb700 10 mon.0@0(leader).pg v12970 preprocess_query pg_stats(5 pgs tid 6798 v 0) v1 from osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:28.464753 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:28.464756 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:28.464757 7f4f71beb700 20 allow all 2017-03-02 15:08:28.464759 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27337) is_readable = 1 - now=2017-03-02 15:08:28.464759 lease_expire=0.000000 has v0 lc 27337 2017-03-02 15:08:28.464763 7f4f71beb700 10 mon.0@0(leader).pg v12970 prepare_update pg_stats(5 pgs tid 6798 v 0) v1 from osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:28.464767 7f4f71beb700 10 mon.0@0(leader).pg v12970 prepare_pg_stats pg_stats(5 pgs tid 6798 v 0) v1 from osd.1 2017-03-02 15:08:28.464773 7f4f71beb700 10 mon.0@0(leader).pg v12970 got osd.1 osd_stat(52172 MB used, 43168 MB avail, 95341 MB total, peers [0,2,3] op hist [0,1]) (was osd_stat(53045 MB used, 42295 MB avail, 95341 MB total, peers [0,2,3] op hist [1,0,1])) 2017-03-02 15:08:28.464782 7f4f71beb700 15 mon.0@0(leader).pg v12970 got 1.1 reported at 7:586782 state active+clean -> active+clean 2017-03-02 15:08:28.464791 7f4f71beb700 15 mon.0@0(leader).pg v12970 got 1.5 reported at 7:567661 state active+clean -> active+clean 2017-03-02 15:08:28.464795 7f4f71beb700 15 mon.0@0(leader).pg v12970 got 1.6 reported at 7:503140 state active+clean -> active+clean 2017-03-02 15:08:28.464800 7f4f71beb700 15 mon.0@0(leader).pg v12970 got 2.4 reported at 7:1026100 state active+clean -> active+clean 2017-03-02 15:08:28.464804 7f4f71beb700 15 mon.0@0(leader).pg v12970 got 2.5 reported at 7:1129458 state active+clean -> active+clean 2017-03-02 15:08:28.464809 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12970) proposal_timer already set 2017-03-02 15:08:28.558044 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.2 172.21.15.11:6800/26536 6834 ==== pg_stats(8 pgs tid 6798 v 0) v1 ==== 4676+0+0 (4165320639 0 0) 0x558a15f27980 con 0x558a145d5800 2017-03-02 15:08:28.558073 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a145c3700 for osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:28.558079 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:28.558081 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12970) dispatch 0x558a15f27980 pg_stats(8 pgs tid 6798 v 0) v1 from osd.2 172.21.15.11:6800/26536 con 0x558a145d5800 2017-03-02 15:08:28.558087 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27337) is_readable = 1 - now=2017-03-02 15:08:28.558087 lease_expire=0.000000 has v0 lc 27337 2017-03-02 15:08:28.558096 7f4f71beb700 10 mon.0@0(leader).pg v12970 preprocess_query pg_stats(8 pgs tid 6798 v 0) v1 from osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:28.558102 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:28.558105 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:28.558106 7f4f71beb700 20 allow all 2017-03-02 15:08:28.558107 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27337) is_readable = 1 - now=2017-03-02 15:08:28.558108 lease_expire=0.000000 has v0 lc 27337 2017-03-02 15:08:28.558112 7f4f71beb700 10 mon.0@0(leader).pg v12970 prepare_update pg_stats(8 pgs tid 6798 v 0) v1 from osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:28.558116 7f4f71beb700 10 mon.0@0(leader).pg v12970 prepare_pg_stats pg_stats(8 pgs tid 6798 v 0) v1 from osd.2 2017-03-02 15:08:28.558122 7f4f71beb700 10 mon.0@0(leader).pg v12970 got osd.2 osd_stat(63400 MB used, 31941 MB avail, 95341 MB total, peers [0,1,3] op hist [0,2,0,1]) (was osd_stat(63652 MB used, 31689 MB avail, 95341 MB total, peers [0,1,3] op hist [2,2,3,2])) 2017-03-02 15:08:28.558131 7f4f71beb700 15 mon.0@0(leader).pg v12970 got 1.0 reported at 7:496485 state active+clean -> active+clean 2017-03-02 15:08:28.558141 7f4f71beb700 15 mon.0@0(leader).pg v12970 got 1.2 reported at 7:577222 state active+clean -> active+clean 2017-03-02 15:08:28.558146 7f4f71beb700 15 mon.0@0(leader).pg v12970 got 1.7 reported at 7:529726 state active+clean -> active+clean 2017-03-02 15:08:28.558150 7f4f71beb700 15 mon.0@0(leader).pg v12970 got 2.0 reported at 7:1056667 state active+clean -> active+clean 2017-03-02 15:08:28.558154 7f4f71beb700 15 mon.0@0(leader).pg v12970 got 2.1 reported at 7:1089437 state active+clean -> active+clean 2017-03-02 15:08:28.558158 7f4f71beb700 15 mon.0@0(leader).pg v12970 got 2.2 reported at 7:1163199 state active+clean -> active+clean 2017-03-02 15:08:28.558162 7f4f71beb700 15 mon.0@0(leader).pg v12970 got 2.3 reported at 7:1095267 state active+clean -> active+clean 2017-03-02 15:08:28.558167 7f4f71beb700 15 mon.0@0(leader).pg v12970 got 2.7 reported at 7:1135346 state active+clean -> active+clean 2017-03-02 15:08:28.558172 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12970) proposal_timer already set 2017-03-02 15:08:28.712940 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.0 172.21.15.26:6800/26515 6834 ==== pg_stats(2 pgs tid 6798 v 0) v1 ==== 1264+0+0 (1791548720 0 0) 0x558a15f27700 con 0x558a145d8800 2017-03-02 15:08:28.712964 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1463af20 for osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:28.712968 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:28.712971 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12970) dispatch 0x558a15f27700 pg_stats(2 pgs tid 6798 v 0) v1 from osd.0 172.21.15.26:6800/26515 con 0x558a145d8800 2017-03-02 15:08:28.712975 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27337) is_readable = 1 - now=2017-03-02 15:08:28.712975 lease_expire=0.000000 has v0 lc 27337 2017-03-02 15:08:28.712980 7f4f71beb700 10 mon.0@0(leader).pg v12970 preprocess_query pg_stats(2 pgs tid 6798 v 0) v1 from osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:28.712985 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:28.712987 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:28.712987 7f4f71beb700 20 allow all 2017-03-02 15:08:28.712988 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27337) is_readable = 1 - now=2017-03-02 15:08:28.712989 lease_expire=0.000000 has v0 lc 27337 2017-03-02 15:08:28.712991 7f4f71beb700 10 mon.0@0(leader).pg v12970 prepare_update pg_stats(2 pgs tid 6798 v 0) v1 from osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:28.712994 7f4f71beb700 10 mon.0@0(leader).pg v12970 prepare_pg_stats pg_stats(2 pgs tid 6798 v 0) v1 from osd.0 2017-03-02 15:08:28.712999 7f4f71beb700 10 mon.0@0(leader).pg v12970 got osd.0 osd_stat(10912 MB used, 84428 MB avail, 95341 MB total, peers [1,2,3] op hist []) (was osd_stat(10935 MB used, 84405 MB avail, 95341 MB total, peers [1,2,3] op hist [])) 2017-03-02 15:08:28.713005 7f4f71beb700 15 mon.0@0(leader).pg v12970 got 1.3 reported at 7:500015 state active+clean -> active+clean 2017-03-02 15:08:28.713012 7f4f71beb700 15 mon.0@0(leader).pg v12970 got 1.4 reported at 7:486191 state active+clean -> active+clean 2017-03-02 15:08:28.713015 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12970) proposal_timer already set 2017-03-02 15:08:29.038852 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12970) propose_pending 2017-03-02 15:08:29.038867 7f4f743f0700 10 mon.0@0(leader).pg v12970 encode_pending v 12971 2017-03-02 15:08:29.038942 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27337) queue_pending_finisher 0x558a1412cfa0 2017-03-02 15:08:29.038948 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27337) trigger_propose active, proposing now 2017-03-02 15:08:29.038964 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27337) propose_pending 27338 10491 bytes 2017-03-02 15:08:29.038966 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27337) begin for 27338 10491 bytes 2017-03-02 15:08:29.072081 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27337) commit_start 27338 2017-03-02 15:08:29.105560 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27337) commit_finish 27338 2017-03-02 15:08:29.105597 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:29.105611 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12971) refresh 2017-03-02 15:08:29.105618 7f4f763f4700 10 mon.0@0(leader).pg v12970 update_from_paxos read_incremental 2017-03-02 15:08:29.105636 7f4f763f4700 20 mon.0@0(leader).pg v12970 refreshing pg 1.0 7:496485 active+clean 2017-03-02 15:08:29.105644 7f4f763f4700 20 mon.0@0(leader).pg v12970 refreshing pg 1.1 7:586782 active+clean 2017-03-02 15:08:29.105649 7f4f763f4700 20 mon.0@0(leader).pg v12970 refreshing pg 1.2 7:577222 active+clean 2017-03-02 15:08:29.105653 7f4f763f4700 20 mon.0@0(leader).pg v12970 refreshing pg 1.3 7:500015 active+clean 2017-03-02 15:08:29.105657 7f4f763f4700 20 mon.0@0(leader).pg v12970 refreshing pg 1.4 7:486191 active+clean 2017-03-02 15:08:29.105660 7f4f763f4700 20 mon.0@0(leader).pg v12970 refreshing pg 1.5 7:567661 active+clean 2017-03-02 15:08:29.105664 7f4f763f4700 20 mon.0@0(leader).pg v12970 refreshing pg 1.6 7:503140 active+clean 2017-03-02 15:08:29.105668 7f4f763f4700 20 mon.0@0(leader).pg v12970 refreshing pg 1.7 7:529726 active+clean 2017-03-02 15:08:29.105672 7f4f763f4700 20 mon.0@0(leader).pg v12970 refreshing pg 2.0 7:1056667 active+clean 2017-03-02 15:08:29.105675 7f4f763f4700 20 mon.0@0(leader).pg v12970 refreshing pg 2.1 7:1089437 active+clean 2017-03-02 15:08:29.105678 7f4f763f4700 20 mon.0@0(leader).pg v12970 refreshing pg 2.2 7:1163199 active+clean 2017-03-02 15:08:29.105682 7f4f763f4700 20 mon.0@0(leader).pg v12970 refreshing pg 2.3 7:1095267 active+clean 2017-03-02 15:08:29.105685 7f4f763f4700 20 mon.0@0(leader).pg v12970 refreshing pg 2.4 7:1026100 active+clean 2017-03-02 15:08:29.105689 7f4f763f4700 20 mon.0@0(leader).pg v12970 refreshing pg 2.5 7:1129458 active+clean 2017-03-02 15:08:29.105692 7f4f763f4700 20 mon.0@0(leader).pg v12970 refreshing pg 2.6 7:964314 active+clean 2017-03-02 15:08:29.105695 7f4f763f4700 20 mon.0@0(leader).pg v12970 refreshing pg 2.7 7:1135346 active+clean 2017-03-02 15:08:29.105697 7f4f763f4700 20 mon.0@0(leader).pg v12970 refreshing osd.0 2017-03-02 15:08:29.105701 7f4f763f4700 20 mon.0@0(leader).pg v12970 refreshing osd.1 2017-03-02 15:08:29.105704 7f4f763f4700 20 mon.0@0(leader).pg v12970 refreshing osd.2 2017-03-02 15:08:29.105707 7f4f763f4700 20 mon.0@0(leader).pg v12970 refreshing osd.3 2017-03-02 15:08:29.105714 7f4f763f4700 10 mon.0@0(leader).pg v12971 read_pgmap_meta 2017-03-02 15:08:29.105726 7f4f763f4700 10 mon.0@0(leader).pg v12971 update_logger 2017-03-02 15:08:29.105734 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..735) refresh 2017-03-02 15:08:29.105745 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:29.105761 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13550) refresh 2017-03-02 15:08:29.105764 7f4f763f4700 10 mon.0@0(leader).log v13550 update_from_paxos 2017-03-02 15:08:29.105765 7f4f763f4700 10 mon.0@0(leader).log v13550 update_from_paxos version 13550 summary v 13550 2017-03-02 15:08:29.105776 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:29.105790 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:29.105792 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:29.105807 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:29.105809 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12971) post_refresh 2017-03-02 15:08:29.105811 7f4f763f4700 10 mon.0@0(leader).pg v12971 post_paxos_update 2017-03-02 15:08:29.105812 7f4f763f4700 10 mon.0@0(leader).pg v12971 check_subs 2017-03-02 15:08:29.105815 7f4f763f4700 20 mon.0@0(leader).pg v12971 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:29.105837 7f4f763f4700 20 mon.0@0(leader).pg v12971 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:29.105841 7f4f763f4700 20 mon.0@0(leader).pg v12971 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:29.105844 7f4f763f4700 20 mon.0@0(leader).pg v12971 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:29.105846 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..735) post_refresh 2017-03-02 15:08:29.105848 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:29.105849 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13550) post_refresh 2017-03-02 15:08:29.105850 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:29.105851 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:29.105853 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:29.105855 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27338) commit_proposal 2017-03-02 15:08:29.105858 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12971) _active - not active 2017-03-02 15:08:29.105860 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27338) finish_round 2017-03-02 15:08:29.105861 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27338) finish_round waiting_for_acting 2017-03-02 15:08:29.105863 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12971) _active 2017-03-02 15:08:29.105864 7f4f763f4700 7 mon.0@0(leader).paxosservice(pgmap 12269..12971) _active creating new pending 2017-03-02 15:08:29.105872 7f4f763f4700 10 mon.0@0(leader).pg v12971 create_pending v 12972 2017-03-02 15:08:29.105878 7f4f763f4700 7 mon.0@0(leader).pg v12971 _updated_stats for osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:29.105882 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a142aba00 0x558a14b9fa00 pg_stats_ack(1 pgs tid 6798) v1 2017-03-02 15:08:29.105886 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6800/27100 -- pg_stats_ack(1 pgs tid 6798) v1 -- 0x558a14b9fa00 con 0 2017-03-02 15:08:29.105916 7f4f763f4700 7 mon.0@0(leader).pg v12971 _updated_stats for osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:29.105920 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a1623ec00 0x558a15646400 pg_stats_ack(5 pgs tid 6798) v1 2017-03-02 15:08:29.105921 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6800/26531 -- pg_stats_ack(5 pgs tid 6798) v1 -- 0x558a15646400 con 0 2017-03-02 15:08:29.105934 7f4f763f4700 7 mon.0@0(leader).pg v12971 _updated_stats for osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:29.105936 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a16209800 0x558a153f6800 pg_stats_ack(8 pgs tid 6798) v1 2017-03-02 15:08:29.105937 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6800/26536 -- pg_stats_ack(8 pgs tid 6798) v1 -- 0x558a153f6800 con 0 2017-03-02 15:08:29.105944 7f4f763f4700 7 mon.0@0(leader).pg v12971 _updated_stats for osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:29.105945 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a167b6f00 0x558a151ae400 pg_stats_ack(2 pgs tid 6798) v1 2017-03-02 15:08:29.105947 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6800/26515 -- pg_stats_ack(2 pgs tid 6798) v1 -- 0x558a151ae400 con 0 2017-03-02 15:08:29.105961 7f4f763f4700 10 mon.0@0(leader).pg v12971 check_osd_map already seen 7 >= 7 2017-03-02 15:08:29.105963 7f4f763f4700 10 mon.0@0(leader).pg v12971 update_logger 2017-03-02 15:08:29.105973 7f4f763f4700 0 log_channel(cluster) log [INF] : pgmap v12971: 32 pgs: 32 active+clean; 86314 MB data, 164 GB used, 207 GB / 372 GB avail; 450 kB/s rd, 30145 kB/s wr, 1541 op/s 2017-03-02 15:08:29.105981 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14114 at 2017-03-02 15:08:29.105974) v1 -- 0x558a16742d80 con 0 2017-03-02 15:08:29.105990 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27338) finish_round waiting_for_readable 2017-03-02 15:08:29.105992 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27338) finish_round waiting_for_writeable 2017-03-02 15:08:29.105993 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27338) finish_round done w/ waiters, state active 2017-03-02 15:08:29.106016 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14114 at 2017-03-02 15:08:29.105974) v1 ==== 0+0+0 (0 0 0) 0x558a16742d80 con 0x558a14482000 2017-03-02 15:08:29.106030 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:29.106034 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:29.106035 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13550) dispatch 0x558a16742d80 log(1 entries from seq 14114 at 2017-03-02 15:08:29.105974) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:29.106041 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27338) is_readable = 1 - now=2017-03-02 15:08:29.106041 lease_expire=0.000000 has v0 lc 27338 2017-03-02 15:08:29.106045 7f4f71beb700 10 mon.0@0(leader).log v13550 preprocess_query log(1 entries from seq 14114 at 2017-03-02 15:08:29.105974) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:29.106049 7f4f71beb700 10 mon.0@0(leader).log v13550 preprocess_log log(1 entries from seq 14114 at 2017-03-02 15:08:29.105974) v1 from mon.0 2017-03-02 15:08:29.106051 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:29.106053 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:29.106053 7f4f71beb700 20 allow all 2017-03-02 15:08:29.106059 7f4f71beb700 10 mon.0@0(leader).log v13550 prepare_update log(1 entries from seq 14114 at 2017-03-02 15:08:29.105974) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:29.106062 7f4f71beb700 10 mon.0@0(leader).log v13550 prepare_log log(1 entries from seq 14114 at 2017-03-02 15:08:29.105974) v1 from mon.0 2017-03-02 15:08:29.106064 7f4f71beb700 10 mon.0@0(leader).log v13550 logging 2017-03-02 15:08:29.105974 mon.0 172.21.15.17:6789/0 14114 : cluster [INF] pgmap v12971: 32 pgs: 32 active+clean; 86314 MB data, 164 GB used, 207 GB / 372 GB avail; 450 kB/s rd, 30145 kB/s wr, 1541 op/s 2017-03-02 15:08:29.106069 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13550) setting proposal_timer 0x558a1412d530 with delay of 0.999505 2017-03-02 15:08:30.105648 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13550) propose_pending 2017-03-02 15:08:30.105662 7f4f743f0700 10 mon.0@0(leader).log v13550 encode_full log v 13550 2017-03-02 15:08:30.105708 7f4f743f0700 10 mon.0@0(leader).log v13550 encode_pending v13551 2017-03-02 15:08:30.105713 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27338) queue_pending_finisher 0x558a1412e880 2017-03-02 15:08:30.105716 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27338) trigger_propose active, proposing now 2017-03-02 15:08:30.105721 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27338) propose_pending 27339 10289 bytes 2017-03-02 15:08:30.105723 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27338) begin for 27339 10289 bytes 2017-03-02 15:08:30.138906 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27338) commit_start 27339 2017-03-02 15:08:30.172484 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27338) commit_finish 27339 2017-03-02 15:08:30.172544 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:30.172577 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12971) refresh 2017-03-02 15:08:30.172597 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..735) refresh 2017-03-02 15:08:30.172620 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:30.172640 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13551) refresh 2017-03-02 15:08:30.172643 7f4f763f4700 10 mon.0@0(leader).log v13551 update_from_paxos 2017-03-02 15:08:30.172646 7f4f763f4700 10 mon.0@0(leader).log v13551 update_from_paxos version 13551 summary v 13550 2017-03-02 15:08:30.172652 7f4f763f4700 10 mon.0@0(leader).log v13551 update_from_paxos latest full 13550 2017-03-02 15:08:30.172668 7f4f763f4700 7 mon.0@0(leader).log v13551 update_from_paxos applying incremental log 13551 2017-03-02 15:08:29.105974 mon.0 172.21.15.17:6789/0 14114 : cluster [INF] pgmap v12971: 32 pgs: 32 active+clean; 86314 MB data, 164 GB used, 207 GB / 372 GB avail; 450 kB/s rd, 30145 kB/s wr, 1541 op/s 2017-03-02 15:08:30.172688 7f4f763f4700 20 mon.0@0(leader).log v13551 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:30.172711 7f4f763f4700 15 mon.0@0(leader).log v13551 update_from_paxos logging for 1 channels 2017-03-02 15:08:30.172714 7f4f763f4700 15 mon.0@0(leader).log v13551 update_from_paxos channel 'cluster' logging 203 bytes 2017-03-02 15:08:30.172800 7f4f763f4700 10 mon.0@0(leader).log v13551 check_subs 2017-03-02 15:08:30.172867 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:30.172878 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:30.172879 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:30.172887 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:30.172889 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12971) post_refresh 2017-03-02 15:08:30.172890 7f4f763f4700 10 mon.0@0(leader).pg v12971 post_paxos_update 2017-03-02 15:08:30.172891 7f4f763f4700 10 mon.0@0(leader).pg v12971 check_subs 2017-03-02 15:08:30.172893 7f4f763f4700 20 mon.0@0(leader).pg v12971 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:30.172897 7f4f763f4700 20 mon.0@0(leader).pg v12971 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:30.172899 7f4f763f4700 20 mon.0@0(leader).pg v12971 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:30.172901 7f4f763f4700 20 mon.0@0(leader).pg v12971 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:30.172902 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..735) post_refresh 2017-03-02 15:08:30.172903 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:30.172904 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13551) post_refresh 2017-03-02 15:08:30.172905 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:30.172906 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:30.172906 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:30.172908 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27339) commit_proposal 2017-03-02 15:08:30.172910 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13551) _active - not active 2017-03-02 15:08:30.172911 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27339) finish_round 2017-03-02 15:08:30.172913 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27339) finish_round waiting_for_acting 2017-03-02 15:08:30.172914 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13551) _active 2017-03-02 15:08:30.172915 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13551) _active creating new pending 2017-03-02 15:08:30.172920 7f4f763f4700 10 mon.0@0(leader).log v13551 create_pending v 13552 2017-03-02 15:08:30.172923 7f4f763f4700 7 mon.0@0(leader).log v13551 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:30.172926 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a164aa600 0x558a14a13480 log(last 14114) v1 2017-03-02 15:08:30.172929 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14114) v1 -- 0x558a14a13480 con 0 2017-03-02 15:08:30.172942 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27339) finish_round waiting_for_readable 2017-03-02 15:08:30.172944 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27339) finish_round waiting_for_writeable 2017-03-02 15:08:30.172945 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27339) finish_round done w/ waiters, state active 2017-03-02 15:08:30.172991 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14114) v1 ==== 0+0+0 (0 0 0) 0x558a14a13480 con 0x558a14482000 2017-03-02 15:08:30.173005 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:30.173009 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:30.173012 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:30.173013 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:30.173014 7f4f71beb700 20 allow all 2017-03-02 15:08:30.686386 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.6 172.21.15.26:6804/2954782670 8748 ==== mds_load_targets(4115 1,2,3,4) v1 ==== 46+0+0 (3586277686 0 0) 0x558a16742900 con 0x558a1471f000 2017-03-02 15:08:30.686421 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14868d60 for mds.? 172.21.15.26:6804/2954782670 2017-03-02 15:08:30.686429 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:30.686433 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..735) dispatch 0x558a16742900 mds_load_targets(4115 1,2,3,4) v1 from mds.6 172.21.15.26:6804/2954782670 con 0x558a1471f000 2017-03-02 15:08:30.686440 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27339) is_readable = 1 - now=2017-03-02 15:08:30.686441 lease_expire=0.000000 has v0 lc 27339 2017-03-02 15:08:30.686449 7f4f71beb700 10 mon.0@0(leader).mds e735 preprocess_query mds_load_targets(4115 1,2,3,4) v1 from mds.6 172.21.15.26:6804/2954782670 2017-03-02 15:08:30.686456 7f4f71beb700 10 mon.0@0(leader).mds e735 preprocess_offload_targets mds_load_targets(4115 1,2,3,4) v1 from mds.6 2017-03-02 15:08:30.686460 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:30.686464 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:30.686465 7f4f71beb700 20 allow all 2017-03-02 15:08:30.686470 7f4f71beb700 7 mon.0@0(leader).mds e735 prepare_update mds_load_targets(4115 1,2,3,4) v1 2017-03-02 15:08:30.686474 7f4f71beb700 10 mon.0@0(leader).mds e735 prepare_offload_targets 4115 1,2,3,4 2017-03-02 15:08:30.686480 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..735) setting proposal_timer 0x558a1412d980 with delay of 0.486031 2017-03-02 15:08:30.870250 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.5 172.21.15.11:6804/1637448123 8786 ==== mdsbeacon(4116/e up:active seq 8499 v735) v7 ==== 365+0+0 (1780080211 0 0) 0x558a160be9c0 con 0x558a1471d800 2017-03-02 15:08:30.870283 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b0f20 for mds.? 172.21.15.11:6804/1637448123 2017-03-02 15:08:30.870291 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:30.870294 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..735) dispatch 0x558a160be9c0 mdsbeacon(4116/e up:active seq 8499 v735) v7 from mds.5 172.21.15.11:6804/1637448123 con 0x558a1471d800 2017-03-02 15:08:30.870301 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27339) is_readable = 1 - now=2017-03-02 15:08:30.870301 lease_expire=0.000000 has v0 lc 27339 2017-03-02 15:08:30.870308 7f4f71beb700 10 mon.0@0(leader).mds e735 preprocess_query mdsbeacon(4116/e up:active seq 8499 v735) v7 from mds.5 172.21.15.11:6804/1637448123 2017-03-02 15:08:30.870316 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:30.870319 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:30.870320 7f4f71beb700 20 allow all 2017-03-02 15:08:30.870321 7f4f71beb700 12 mon.0@0(leader).mds e735 preprocess_beacon mdsbeacon(4116/e up:active seq 8499 v735) v7 from mds.5 172.21.15.11:6804/1637448123 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:30.870329 7f4f71beb700 10 mon.0@0(leader).mds e735 preprocess_beacon: GID exists in map: 4116 2017-03-02 15:08:30.870333 7f4f71beb700 10 mon.0@0(leader).mds e735 mds_beacon mdsbeacon(4116/e up:active seq 8499 v735) v7 ignoring requested state, because mds hasn't seen latest map 2017-03-02 15:08:30.870336 7f4f71beb700 15 mon.0@0(leader).mds e735 _note_beacon mdsbeacon(4116/e up:active seq 8499 v735) v7 noting time 2017-03-02 15:08:30.870339 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a14f44800 0x558a15204000 mdsbeacon(4116/e up:active seq 8499 v736) v7 2017-03-02 15:08:30.870342 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6804/1637448123 -- mdsbeacon(4116/e up:active seq 8499 v736) v7 -- 0x558a15204000 con 0 2017-03-02 15:08:30.883927 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.3 172.21.15.11:6805/1926402648 8562 ==== mdsbeacon(4118/f up:active seq 8499 v735) v7 ==== 406+0+0 (975781881 0 0) 0x558a143e56c0 con 0x558a14905800 2017-03-02 15:08:30.883952 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b2260 for mds.? 172.21.15.11:6805/1926402648 2017-03-02 15:08:30.883958 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:30.883961 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..735) dispatch 0x558a143e56c0 mdsbeacon(4118/f up:active seq 8499 v735) v7 from mds.3 172.21.15.11:6805/1926402648 con 0x558a14905800 2017-03-02 15:08:30.883966 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27339) is_readable = 1 - now=2017-03-02 15:08:30.883967 lease_expire=0.000000 has v0 lc 27339 2017-03-02 15:08:30.883974 7f4f71beb700 10 mon.0@0(leader).mds e735 preprocess_query mdsbeacon(4118/f up:active seq 8499 v735) v7 from mds.3 172.21.15.11:6805/1926402648 2017-03-02 15:08:30.883980 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:30.883983 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:30.883984 7f4f71beb700 20 allow all 2017-03-02 15:08:30.883985 7f4f71beb700 12 mon.0@0(leader).mds e735 preprocess_beacon mdsbeacon(4118/f up:active seq 8499 v735) v7 from mds.3 172.21.15.11:6805/1926402648 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:30.883993 7f4f71beb700 10 mon.0@0(leader).mds e735 preprocess_beacon: GID exists in map: 4118 2017-03-02 15:08:30.883997 7f4f71beb700 10 mon.0@0(leader).mds e735 mds_beacon mdsbeacon(4118/f up:active seq 8499 v735) v7 ignoring requested state, because mds hasn't seen latest map 2017-03-02 15:08:30.883999 7f4f71beb700 15 mon.0@0(leader).mds e735 _note_beacon mdsbeacon(4118/f up:active seq 8499 v735) v7 noting time 2017-03-02 15:08:30.884003 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a14f44800 0x558a160be9c0 mdsbeacon(4118/f up:active seq 8499 v736) v7 2017-03-02 15:08:30.884006 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6805/1926402648 -- mdsbeacon(4118/f up:active seq 8499 v736) v7 -- 0x558a160be9c0 con 0 2017-03-02 15:08:30.884774 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.2 172.21.15.22:6804/673088617 8561 ==== mdsbeacon(4119/c up:active seq 8499 v735) v7 ==== 365+0+0 (1241569805 0 0) 0x558a15204000 con 0x558a14907000 2017-03-02 15:08:30.884789 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14916f20 for mds.? 172.21.15.22:6804/673088617 2017-03-02 15:08:30.884793 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:30.884796 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..735) dispatch 0x558a15204000 mdsbeacon(4119/c up:active seq 8499 v735) v7 from mds.2 172.21.15.22:6804/673088617 con 0x558a14907000 2017-03-02 15:08:30.884800 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27339) is_readable = 1 - now=2017-03-02 15:08:30.884801 lease_expire=0.000000 has v0 lc 27339 2017-03-02 15:08:30.884806 7f4f71beb700 10 mon.0@0(leader).mds e735 preprocess_query mdsbeacon(4119/c up:active seq 8499 v735) v7 from mds.2 172.21.15.22:6804/673088617 2017-03-02 15:08:30.884811 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:30.884813 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:30.884814 7f4f71beb700 20 allow all 2017-03-02 15:08:30.884816 7f4f71beb700 12 mon.0@0(leader).mds e735 preprocess_beacon mdsbeacon(4119/c up:active seq 8499 v735) v7 from mds.2 172.21.15.22:6804/673088617 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:30.884843 7f4f71beb700 10 mon.0@0(leader).mds e735 preprocess_beacon: GID exists in map: 4119 2017-03-02 15:08:30.884846 7f4f71beb700 10 mon.0@0(leader).mds e735 mds_beacon mdsbeacon(4119/c up:active seq 8499 v735) v7 ignoring requested state, because mds hasn't seen latest map 2017-03-02 15:08:30.884849 7f4f71beb700 15 mon.0@0(leader).mds e735 _note_beacon mdsbeacon(4119/c up:active seq 8499 v735) v7 noting time 2017-03-02 15:08:30.884852 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a14f44800 0x558a143e56c0 mdsbeacon(4119/c up:active seq 8499 v736) v7 2017-03-02 15:08:30.884855 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6804/673088617 -- mdsbeacon(4119/c up:active seq 8499 v736) v7 -- 0x558a143e56c0 con 0 2017-03-02 15:08:30.884997 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.0 172.21.15.22:6805/2203629154 8611 ==== mdsbeacon(4121/d up:active seq 8495 v735) v7 ==== 452+0+0 (1938064227 0 0) 0x558a160be9c0 con 0x558a1492a000 2017-03-02 15:08:30.885010 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a149195a0 for mds.? 172.21.15.22:6805/2203629154 2017-03-02 15:08:30.885014 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:30.885016 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..735) dispatch 0x558a160be9c0 mdsbeacon(4121/d up:active seq 8495 v735) v7 from mds.0 172.21.15.22:6805/2203629154 con 0x558a1492a000 2017-03-02 15:08:30.885020 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27339) is_readable = 1 - now=2017-03-02 15:08:30.885021 lease_expire=0.000000 has v0 lc 27339 2017-03-02 15:08:30.885026 7f4f71beb700 10 mon.0@0(leader).mds e735 preprocess_query mdsbeacon(4121/d up:active seq 8495 v735) v7 from mds.0 172.21.15.22:6805/2203629154 2017-03-02 15:08:30.885031 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:30.885033 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:30.885034 7f4f71beb700 20 allow all 2017-03-02 15:08:30.885035 7f4f71beb700 12 mon.0@0(leader).mds e735 preprocess_beacon mdsbeacon(4121/d up:active seq 8495 v735) v7 from mds.0 172.21.15.22:6805/2203629154 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:30.885041 7f4f71beb700 10 mon.0@0(leader).mds e735 preprocess_beacon: GID exists in map: 4121 2017-03-02 15:08:30.885044 7f4f71beb700 10 mon.0@0(leader).mds e735 mds_beacon mdsbeacon(4121/d up:active seq 8495 v735) v7 ignoring requested state, because mds hasn't seen latest map 2017-03-02 15:08:30.885046 7f4f71beb700 15 mon.0@0(leader).mds e735 _note_beacon mdsbeacon(4121/d up:active seq 8495 v735) v7 noting time 2017-03-02 15:08:30.885049 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a14f44800 0x558a15204000 mdsbeacon(4121/d up:active seq 8495 v736) v7 2017-03-02 15:08:30.885051 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6805/2203629154 -- mdsbeacon(4121/d up:active seq 8495 v736) v7 -- 0x558a15204000 con 0 2017-03-02 15:08:30.924167 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.1 172.21.15.17:6804/2576824570 8558 ==== mdsbeacon(4120/g up:active seq 8499 v735) v7 ==== 365+0+0 (2182059798 0 0) 0x558a154b8d00 con 0x558a14908800 2017-03-02 15:08:30.924196 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14917600 for mds.? 172.21.15.17:6804/2576824570 2017-03-02 15:08:30.924204 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:30.924207 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..735) dispatch 0x558a154b8d00 mdsbeacon(4120/g up:active seq 8499 v735) v7 from mds.1 172.21.15.17:6804/2576824570 con 0x558a14908800 2017-03-02 15:08:30.924213 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27339) is_readable = 1 - now=2017-03-02 15:08:30.924214 lease_expire=0.000000 has v0 lc 27339 2017-03-02 15:08:30.924220 7f4f71beb700 10 mon.0@0(leader).mds e735 preprocess_query mdsbeacon(4120/g up:active seq 8499 v735) v7 from mds.1 172.21.15.17:6804/2576824570 2017-03-02 15:08:30.924225 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:30.924227 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:30.924228 7f4f71beb700 20 allow all 2017-03-02 15:08:30.924229 7f4f71beb700 12 mon.0@0(leader).mds e735 preprocess_beacon mdsbeacon(4120/g up:active seq 8499 v735) v7 from mds.1 172.21.15.17:6804/2576824570 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:30.924234 7f4f71beb700 10 mon.0@0(leader).mds e735 preprocess_beacon: GID exists in map: 4120 2017-03-02 15:08:30.924237 7f4f71beb700 10 mon.0@0(leader).mds e735 mds_beacon mdsbeacon(4120/g up:active seq 8499 v735) v7 ignoring requested state, because mds hasn't seen latest map 2017-03-02 15:08:30.924239 7f4f71beb700 15 mon.0@0(leader).mds e735 _note_beacon mdsbeacon(4120/g up:active seq 8499 v735) v7 noting time 2017-03-02 15:08:30.924241 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a14f44800 0x558a160be9c0 mdsbeacon(4120/g up:active seq 8499 v736) v7 2017-03-02 15:08:30.924243 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6804/2576824570 -- mdsbeacon(4120/g up:active seq 8499 v736) v7 -- 0x558a160be9c0 con 0 2017-03-02 15:08:30.969465 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.4 172.21.15.26:6805/2033388711 8556 ==== mdsbeacon(4117/b up:active seq 8499 v735) v7 ==== 365+0+0 (2091289045 0 0) 0x558a160be9c0 con 0x558a14904000 2017-03-02 15:08:30.969496 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b1600 for mds.? 172.21.15.26:6805/2033388711 2017-03-02 15:08:30.969503 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:30.969505 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..735) dispatch 0x558a160be9c0 mdsbeacon(4117/b up:active seq 8499 v735) v7 from mds.4 172.21.15.26:6805/2033388711 con 0x558a14904000 2017-03-02 15:08:30.969509 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27339) is_readable = 1 - now=2017-03-02 15:08:30.969510 lease_expire=0.000000 has v0 lc 27339 2017-03-02 15:08:30.969516 7f4f71beb700 10 mon.0@0(leader).mds e735 preprocess_query mdsbeacon(4117/b up:active seq 8499 v735) v7 from mds.4 172.21.15.26:6805/2033388711 2017-03-02 15:08:30.969521 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:30.969523 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:30.969523 7f4f71beb700 20 allow all 2017-03-02 15:08:30.969524 7f4f71beb700 12 mon.0@0(leader).mds e735 preprocess_beacon mdsbeacon(4117/b up:active seq 8499 v735) v7 from mds.4 172.21.15.26:6805/2033388711 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:30.969529 7f4f71beb700 10 mon.0@0(leader).mds e735 preprocess_beacon: GID exists in map: 4117 2017-03-02 15:08:30.969532 7f4f71beb700 10 mon.0@0(leader).mds e735 mds_beacon mdsbeacon(4117/b up:active seq 8499 v735) v7 ignoring requested state, because mds hasn't seen latest map 2017-03-02 15:08:30.969533 7f4f71beb700 15 mon.0@0(leader).mds e735 _note_beacon mdsbeacon(4117/b up:active seq 8499 v735) v7 noting time 2017-03-02 15:08:30.969536 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a14f44800 0x558a154b8d00 mdsbeacon(4117/b up:active seq 8499 v736) v7 2017-03-02 15:08:30.969538 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6805/2033388711 -- mdsbeacon(4117/b up:active seq 8499 v736) v7 -- 0x558a154b8d00 con 0 2017-03-02 15:08:30.978478 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.6 172.21.15.26:6804/2954782670 8749 ==== mdsbeacon(4115/a up:active seq 8499 v735) v7 ==== 365+0+0 (659324349 0 0) 0x558a15204000 con 0x558a1471f000 2017-03-02 15:08:30.978505 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14868d60 for mds.? 172.21.15.26:6804/2954782670 2017-03-02 15:08:30.978531 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:30.978534 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..735) dispatch 0x558a15204000 mdsbeacon(4115/a up:active seq 8499 v735) v7 from mds.6 172.21.15.26:6804/2954782670 con 0x558a1471f000 2017-03-02 15:08:30.978541 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27339) is_readable = 1 - now=2017-03-02 15:08:30.978541 lease_expire=0.000000 has v0 lc 27339 2017-03-02 15:08:30.978548 7f4f71beb700 10 mon.0@0(leader).mds e735 preprocess_query mdsbeacon(4115/a up:active seq 8499 v735) v7 from mds.6 172.21.15.26:6804/2954782670 2017-03-02 15:08:30.978555 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:30.978557 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:30.978561 7f4f71beb700 20 allow all 2017-03-02 15:08:30.978562 7f4f71beb700 12 mon.0@0(leader).mds e735 preprocess_beacon mdsbeacon(4115/a up:active seq 8499 v735) v7 from mds.6 172.21.15.26:6804/2954782670 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:30.978570 7f4f71beb700 10 mon.0@0(leader).mds e735 preprocess_beacon: GID exists in map: 4115 2017-03-02 15:08:30.978573 7f4f71beb700 10 mon.0@0(leader).mds e735 mds_beacon mdsbeacon(4115/a up:active seq 8499 v735) v7 ignoring requested state, because mds hasn't seen latest map 2017-03-02 15:08:30.978575 7f4f71beb700 15 mon.0@0(leader).mds e735 _note_beacon mdsbeacon(4115/a up:active seq 8499 v735) v7 noting time 2017-03-02 15:08:30.978579 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a14f44800 0x558a160be9c0 mdsbeacon(4115/a up:active seq 8499 v736) v7 2017-03-02 15:08:30.978582 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6804/2954782670 -- mdsbeacon(4115/a up:active seq 8499 v736) v7 -- 0x558a160be9c0 con 0 2017-03-02 15:08:31.011945 7f4f743f0700 11 mon.0@0(leader) e2 tick 2017-03-02 15:08:31.011969 7f4f743f0700 10 mon.0@0(leader).pg v12971 v12971: 32 pgs: 32 active+clean; 86314 MB data, 164 GB used, 207 GB / 372 GB avail; 450 kB/s rd, 30145 kB/s wr, 1541 op/s 2017-03-02 15:08:31.011984 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12971) maybe_trim trim_to 12471 would only trim 202 < paxos_service_trim_min 250 2017-03-02 15:08:31.011988 7f4f743f0700 10 mon.0@0(leader).mds e735 e735: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:31.012018 7f4f743f0700 10 mon.0@0(leader).paxosservice(mdsmap 1..735) maybe_trim trim_to 235 would only trim 234 < paxos_service_trim_min 250 2017-03-02 15:08:31.012022 7f4f743f0700 10 mon.0@0(leader).osd e7 e7: 4 osds: 4 up, 4 in 2017-03-02 15:08:31.012025 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27339) is_readable = 1 - now=2017-03-02 15:08:31.012026 lease_expire=0.000000 has v0 lc 27339 2017-03-02 15:08:31.012033 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27339) is_readable = 1 - now=2017-03-02 15:08:31.012033 lease_expire=0.000000 has v0 lc 27339 2017-03-02 15:08:31.012038 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27339) is_readable = 1 - now=2017-03-02 15:08:31.012038 lease_expire=0.000000 has v0 lc 27339 2017-03-02 15:08:31.012041 7f4f743f0700 10 mon.0@0(leader).osd e7 min_last_epoch_clean 6 2017-03-02 15:08:31.012043 7f4f743f0700 10 mon.0@0(leader).log v13551 log 2017-03-02 15:08:31.012044 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13551) maybe_trim trim_to 13051 would only trim 24 < paxos_service_trim_min 250 2017-03-02 15:08:31.012047 7f4f743f0700 10 mon.0@0(leader).auth v10 auth 2017-03-02 15:08:31.012051 7f4f743f0700 10 cephx keyserver: _check_rotating_secrets 2017-03-02 15:08:31.012058 7f4f743f0700 20 mon.0@0(leader) e2 sync_trim_providers 2017-03-02 15:08:31.172593 7f4f743f0700 10 mon.0@0(leader).paxosservice(mdsmap 1..735) propose_pending 2017-03-02 15:08:31.172635 7f4f743f0700 10 mon.0@0(leader).mds e735 encode_pending e736 2017-03-02 15:08:31.172696 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27339) queue_pending_finisher 0x558a1412e7a0 2017-03-02 15:08:31.172704 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27339) trigger_propose active, proposing now 2017-03-02 15:08:31.172711 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27339) propose_pending 27340 2415 bytes 2017-03-02 15:08:31.172713 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27339) begin for 27340 2415 bytes 2017-03-02 15:08:31.185685 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27339) commit_start 27340 2017-03-02 15:08:31.214045 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27339) commit_finish 27340 2017-03-02 15:08:31.214096 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:31.214120 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12971) refresh 2017-03-02 15:08:31.214138 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..736) refresh 2017-03-02 15:08:31.214144 7f4f763f4700 10 mon.0@0(leader).mds e735 update_from_paxos version 736, my e 735 2017-03-02 15:08:31.214154 7f4f763f4700 10 mon.0@0(leader).mds e735 update_from_paxos got 736 2017-03-02 15:08:31.214202 7f4f763f4700 4 mon.0@0(leader).mds e736 new map 2017-03-02 15:08:31.214209 7f4f763f4700 0 mon.0@0(leader).mds e736 print_map 2017-03-02 15:08:31.214260 7f4f763f4700 20 mon.0@0(leader).mds e736 check_sub: mdsmap 2017-03-02 15:08:31.214266 7f4f763f4700 10 mon.0@0(leader).mds e736 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:31.214269 7f4f763f4700 10 mon.0@0(leader).mds e736 check_sub selected MDS map epoch 736 for namespace 1 for subscriber mds.? who wants epoch 736 2017-03-02 15:08:31.214289 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6804/2954782670 -- mdsmap(e 736) v1 -- 0x558a16742d80 con 0 2017-03-02 15:08:31.214317 7f4f763f4700 20 mon.0@0(leader).mds e736 check_sub: mdsmap 2017-03-02 15:08:31.214328 7f4f763f4700 10 mon.0@0(leader).mds e736 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:31.214331 7f4f763f4700 10 mon.0@0(leader).mds e736 check_sub selected MDS map epoch 736 for namespace 1 for subscriber mds.? who wants epoch 736 2017-03-02 15:08:31.214346 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6804/1637448123 -- mdsmap(e 736) v1 -- 0x558a16742b40 con 0 2017-03-02 15:08:31.214364 7f4f763f4700 20 mon.0@0(leader).mds e736 check_sub: mdsmap 2017-03-02 15:08:31.214369 7f4f763f4700 10 mon.0@0(leader).mds e736 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:31.214371 7f4f763f4700 10 mon.0@0(leader).mds e736 check_sub selected MDS map epoch 736 for namespace 1 for subscriber mds.? who wants epoch 736 2017-03-02 15:08:31.214385 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6805/2033388711 -- mdsmap(e 736) v1 -- 0x558a1456d680 con 0 2017-03-02 15:08:31.214426 7f4f763f4700 20 mon.0@0(leader).mds e736 check_sub: mdsmap 2017-03-02 15:08:31.214437 7f4f763f4700 10 mon.0@0(leader).mds e736 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:31.214439 7f4f763f4700 10 mon.0@0(leader).mds e736 check_sub selected MDS map epoch 736 for namespace 1 for subscriber mds.? who wants epoch 736 2017-03-02 15:08:31.214454 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6805/1926402648 -- mdsmap(e 736) v1 -- 0x558a16742fc0 con 0 2017-03-02 15:08:31.214540 7f4f763f4700 20 mon.0@0(leader).mds e736 check_sub: mdsmap 2017-03-02 15:08:31.214555 7f4f763f4700 10 mon.0@0(leader).mds e736 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:31.214558 7f4f763f4700 10 mon.0@0(leader).mds e736 check_sub selected MDS map epoch 736 for namespace 1 for subscriber mds.? who wants epoch 736 2017-03-02 15:08:31.214574 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6804/673088617 -- mdsmap(e 736) v1 -- 0x558a16743200 con 0 2017-03-02 15:08:31.214592 7f4f763f4700 20 mon.0@0(leader).mds e736 check_sub: mdsmap 2017-03-02 15:08:31.214596 7f4f763f4700 10 mon.0@0(leader).mds e736 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:31.214598 7f4f763f4700 10 mon.0@0(leader).mds e736 check_sub selected MDS map epoch 736 for namespace 1 for subscriber mds.? who wants epoch 736 2017-03-02 15:08:31.214629 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6804/2576824570 -- mdsmap(e 736) v1 -- 0x558a16743440 con 0 2017-03-02 15:08:31.214646 7f4f763f4700 20 mon.0@0(leader).mds e736 check_sub: mdsmap 2017-03-02 15:08:31.214651 7f4f763f4700 10 mon.0@0(leader).mds e736 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:31.214653 7f4f763f4700 10 mon.0@0(leader).mds e736 check_sub selected MDS map epoch 736 for namespace 1 for subscriber mds.? who wants epoch 736 2017-03-02 15:08:31.214667 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6805/2203629154 -- mdsmap(e 736) v1 -- 0x558a16743680 con 0 2017-03-02 15:08:31.214950 7f4f763f4700 20 mon.0@0(leader).mds e736 check_sub: mdsmap 2017-03-02 15:08:31.214961 7f4f763f4700 10 mon.0@0(leader).mds e736 check_sub: is_mds=0, fscid= 1 2017-03-02 15:08:31.214963 7f4f763f4700 10 mon.0@0(leader).mds e736 check_sub selected MDS map epoch 736 for namespace 1 for subscriber unknown.0 who wants epoch 736 2017-03-02 15:08:31.214996 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.132:0/2597832251 -- mdsmap(e 736) v1 -- 0x558a167438c0 con 0 2017-03-02 15:08:31.215021 7f4f763f4700 20 mon.0@0(leader).mds e736 check_sub: mdsmap 2017-03-02 15:08:31.215027 7f4f763f4700 10 mon.0@0(leader).mds e736 check_sub: is_mds=0, fscid= 1 2017-03-02 15:08:31.215064 7f4f763f4700 10 mon.0@0(leader).mds e736 check_sub selected MDS map epoch 736 for namespace 1 for subscriber unknown.0 who wants epoch 736 2017-03-02 15:08:31.215081 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.133:0/3792964262 -- mdsmap(e 736) v1 -- 0x558a16743b00 con 0 2017-03-02 15:08:31.215100 7f4f763f4700 10 mon.0@0(leader).mds e736 update_logger 2017-03-02 15:08:31.215123 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:31.215146 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13551) refresh 2017-03-02 15:08:31.215153 7f4f763f4700 10 mon.0@0(leader).log v13551 update_from_paxos 2017-03-02 15:08:31.215156 7f4f763f4700 10 mon.0@0(leader).log v13551 update_from_paxos version 13551 summary v 13551 2017-03-02 15:08:31.215167 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:31.215181 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:31.215183 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:31.215196 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:31.215199 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12971) post_refresh 2017-03-02 15:08:31.215200 7f4f763f4700 10 mon.0@0(leader).pg v12971 post_paxos_update 2017-03-02 15:08:31.215203 7f4f763f4700 10 mon.0@0(leader).pg v12971 check_subs 2017-03-02 15:08:31.215205 7f4f763f4700 20 mon.0@0(leader).pg v12971 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:31.215211 7f4f763f4700 20 mon.0@0(leader).pg v12971 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:31.215213 7f4f763f4700 20 mon.0@0(leader).pg v12971 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:31.215216 7f4f763f4700 20 mon.0@0(leader).pg v12971 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:31.215218 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..736) post_refresh 2017-03-02 15:08:31.215220 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:31.215221 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13551) post_refresh 2017-03-02 15:08:31.215222 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:31.215224 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:31.215225 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:31.215227 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27340) commit_proposal 2017-03-02 15:08:31.215231 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..736) _active - not active 2017-03-02 15:08:31.215233 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27340) finish_round 2017-03-02 15:08:31.215234 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27340) finish_round waiting_for_acting 2017-03-02 15:08:31.215236 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..736) _active 2017-03-02 15:08:31.215237 7f4f763f4700 7 mon.0@0(leader).paxosservice(mdsmap 1..736) _active creating new pending 2017-03-02 15:08:31.215249 7f4f763f4700 10 mon.0@0(leader).mds e736 create_pending e737 2017-03-02 15:08:31.215251 7f4f763f4700 10 mon.0@0(leader).mds e736 e736: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:31.215263 7f4f763f4700 10 mon.0@0(leader).mds e736 update_logger 2017-03-02 15:08:31.215274 7f4f763f4700 0 log_channel(cluster) log [INF] : fsmap e736: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:31.215284 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14115 at 2017-03-02 15:08:31.215275) v1 -- 0x558a16743f80 con 0 2017-03-02 15:08:31.215298 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27340) finish_round waiting_for_readable 2017-03-02 15:08:31.215300 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27340) finish_round waiting_for_writeable 2017-03-02 15:08:31.215301 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27340) finish_round done w/ waiters, state active 2017-03-02 15:08:31.215323 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14115 at 2017-03-02 15:08:31.215275) v1 ==== 0+0+0 (0 0 0) 0x558a16743f80 con 0x558a14482000 2017-03-02 15:08:31.215337 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:31.215340 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:31.215343 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13551) dispatch 0x558a16743f80 log(1 entries from seq 14115 at 2017-03-02 15:08:31.215275) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:31.215347 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27340) is_readable = 1 - now=2017-03-02 15:08:31.215347 lease_expire=0.000000 has v0 lc 27340 2017-03-02 15:08:31.215353 7f4f71beb700 10 mon.0@0(leader).log v13551 preprocess_query log(1 entries from seq 14115 at 2017-03-02 15:08:31.215275) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:31.215359 7f4f71beb700 10 mon.0@0(leader).log v13551 preprocess_log log(1 entries from seq 14115 at 2017-03-02 15:08:31.215275) v1 from mon.0 2017-03-02 15:08:31.215363 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:31.215365 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:31.215366 7f4f71beb700 20 allow all 2017-03-02 15:08:31.215373 7f4f71beb700 10 mon.0@0(leader).log v13551 prepare_update log(1 entries from seq 14115 at 2017-03-02 15:08:31.215275) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:31.215377 7f4f71beb700 10 mon.0@0(leader).log v13551 prepare_log log(1 entries from seq 14115 at 2017-03-02 15:08:31.215275) v1 from mon.0 2017-03-02 15:08:31.215379 7f4f71beb700 10 mon.0@0(leader).log v13551 logging 2017-03-02 15:08:31.215275 mon.0 172.21.15.17:6789/0 14115 : cluster [INF] fsmap e736: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:31.215384 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13551) setting proposal_timer 0x558a1412df60 with delay of 0.998683 2017-03-02 15:08:32.214158 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13551) propose_pending 2017-03-02 15:08:32.214172 7f4f743f0700 10 mon.0@0(leader).log v13551 encode_full log v 13551 2017-03-02 15:08:32.214221 7f4f743f0700 10 mon.0@0(leader).log v13551 encode_pending v13552 2017-03-02 15:08:32.214226 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27340) queue_pending_finisher 0x558a1412d980 2017-03-02 15:08:32.214228 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27340) trigger_propose active, proposing now 2017-03-02 15:08:32.214232 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27340) propose_pending 27341 10365 bytes 2017-03-02 15:08:32.214234 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27340) begin for 27341 10365 bytes 2017-03-02 15:08:32.247362 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27340) commit_start 27341 2017-03-02 15:08:32.280810 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27340) commit_finish 27341 2017-03-02 15:08:32.280866 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:32.280882 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12971) refresh 2017-03-02 15:08:32.280890 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..736) refresh 2017-03-02 15:08:32.280899 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:32.280912 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13552) refresh 2017-03-02 15:08:32.280914 7f4f763f4700 10 mon.0@0(leader).log v13552 update_from_paxos 2017-03-02 15:08:32.280916 7f4f763f4700 10 mon.0@0(leader).log v13552 update_from_paxos version 13552 summary v 13551 2017-03-02 15:08:32.280919 7f4f763f4700 10 mon.0@0(leader).log v13552 update_from_paxos latest full 13551 2017-03-02 15:08:32.280926 7f4f763f4700 7 mon.0@0(leader).log v13552 update_from_paxos applying incremental log 13552 2017-03-02 15:08:31.215275 mon.0 172.21.15.17:6789/0 14115 : cluster [INF] fsmap e736: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:32.280938 7f4f763f4700 20 mon.0@0(leader).log v13552 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:32.280953 7f4f763f4700 15 mon.0@0(leader).log v13552 update_from_paxos logging for 1 channels 2017-03-02 15:08:32.280954 7f4f763f4700 15 mon.0@0(leader).log v13552 update_from_paxos channel 'cluster' logging 196 bytes 2017-03-02 15:08:32.280997 7f4f763f4700 10 mon.0@0(leader).log v13552 check_subs 2017-03-02 15:08:32.281021 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:32.281039 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:32.281045 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:32.281054 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:32.281056 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12971) post_refresh 2017-03-02 15:08:32.281057 7f4f763f4700 10 mon.0@0(leader).pg v12971 post_paxos_update 2017-03-02 15:08:32.281058 7f4f763f4700 10 mon.0@0(leader).pg v12971 check_subs 2017-03-02 15:08:32.281060 7f4f763f4700 20 mon.0@0(leader).pg v12971 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:32.281063 7f4f763f4700 20 mon.0@0(leader).pg v12971 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:32.281064 7f4f763f4700 20 mon.0@0(leader).pg v12971 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:32.281066 7f4f763f4700 20 mon.0@0(leader).pg v12971 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:32.281067 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..736) post_refresh 2017-03-02 15:08:32.281068 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:32.281069 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13552) post_refresh 2017-03-02 15:08:32.281069 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:32.281070 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:32.281071 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:32.281072 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27341) commit_proposal 2017-03-02 15:08:32.281074 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13552) _active - not active 2017-03-02 15:08:32.281076 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27341) finish_round 2017-03-02 15:08:32.281077 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27341) finish_round waiting_for_acting 2017-03-02 15:08:32.281078 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13552) _active 2017-03-02 15:08:32.281079 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13552) _active creating new pending 2017-03-02 15:08:32.281085 7f4f763f4700 10 mon.0@0(leader).log v13552 create_pending v 13553 2017-03-02 15:08:32.281087 7f4f763f4700 7 mon.0@0(leader).log v13552 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:32.281090 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a14f44800 0x558a154d4780 log(last 14115) v1 2017-03-02 15:08:32.281093 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14115) v1 -- 0x558a154d4780 con 0 2017-03-02 15:08:32.281125 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14115) v1 ==== 0+0+0 (0 0 0) 0x558a154d4780 con 0x558a14482000 2017-03-02 15:08:32.281216 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27341) finish_round waiting_for_readable 2017-03-02 15:08:32.281223 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27341) finish_round waiting_for_writeable 2017-03-02 15:08:32.281224 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27341) finish_round done w/ waiters, state active 2017-03-02 15:08:32.281239 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:32.281246 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:32.281249 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:32.281252 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:32.281252 7f4f71beb700 20 allow all 2017-03-02 15:08:33.263287 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.3 172.21.15.17:6800/27100 6835 ==== pg_stats(1 pgs tid 6799 v 0) v1 ==== 706+0+0 (1057950902 0 0) 0x558a15da3700 con 0x558a145d7000 2017-03-02 15:08:33.263312 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415f440 for osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:33.263316 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:33.263318 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12971) dispatch 0x558a15da3700 pg_stats(1 pgs tid 6799 v 0) v1 from osd.3 172.21.15.17:6800/27100 con 0x558a145d7000 2017-03-02 15:08:33.263322 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27341) is_readable = 1 - now=2017-03-02 15:08:33.263322 lease_expire=0.000000 has v0 lc 27341 2017-03-02 15:08:33.263328 7f4f71beb700 10 mon.0@0(leader).pg v12971 preprocess_query pg_stats(1 pgs tid 6799 v 0) v1 from osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:33.263333 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:33.263336 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:33.263339 7f4f71beb700 20 allow all 2017-03-02 15:08:33.263340 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27341) is_readable = 1 - now=2017-03-02 15:08:33.263341 lease_expire=0.000000 has v0 lc 27341 2017-03-02 15:08:33.263345 7f4f71beb700 10 mon.0@0(leader).pg v12971 prepare_update pg_stats(1 pgs tid 6799 v 0) v1 from osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:33.263350 7f4f71beb700 10 mon.0@0(leader).pg v12971 prepare_pg_stats pg_stats(1 pgs tid 6799 v 0) v1 from osd.3 2017-03-02 15:08:33.263354 7f4f71beb700 10 mon.0@0(leader).pg v12971 got osd.3 osd_stat(40871 MB used, 54469 MB avail, 95341 MB total, peers [0,1,2] op hist [0,2]) (was osd_stat(41976 MB used, 53365 MB avail, 95341 MB total, peers [0,1,2] op hist [0,1])) 2017-03-02 15:08:33.263360 7f4f71beb700 15 mon.0@0(leader).pg v12971 got 2.6 reported at 7:964513 state active+clean -> active+clean 2017-03-02 15:08:33.263367 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12971) setting proposal_timer 0x558a1412e600 with delay of 0.0174804 2017-03-02 15:08:33.280935 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12971) propose_pending 2017-03-02 15:08:33.280946 7f4f743f0700 10 mon.0@0(leader).pg v12971 encode_pending v 12972 2017-03-02 15:08:33.280967 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27341) queue_pending_finisher 0x558a1412df60 2017-03-02 15:08:33.280969 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27341) trigger_propose active, proposing now 2017-03-02 15:08:33.280976 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27341) propose_pending 27342 1098 bytes 2017-03-02 15:08:33.280978 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27341) begin for 27342 1098 bytes 2017-03-02 15:08:33.293851 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27341) commit_start 27342 2017-03-02 15:08:33.302179 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27341) commit_finish 27342 2017-03-02 15:08:33.302203 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:33.302216 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12972) refresh 2017-03-02 15:08:33.302218 7f4f763f4700 10 mon.0@0(leader).pg v12971 update_from_paxos read_incremental 2017-03-02 15:08:33.302232 7f4f763f4700 20 mon.0@0(leader).pg v12971 refreshing pg 2.6 7:964513 active+clean 2017-03-02 15:08:33.302234 7f4f763f4700 20 mon.0@0(leader).pg v12971 refreshing osd.3 2017-03-02 15:08:33.302243 7f4f763f4700 10 mon.0@0(leader).pg v12972 read_pgmap_meta 2017-03-02 15:08:33.302254 7f4f763f4700 10 mon.0@0(leader).pg v12972 update_logger 2017-03-02 15:08:33.302262 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..736) refresh 2017-03-02 15:08:33.302271 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:33.302282 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13552) refresh 2017-03-02 15:08:33.302283 7f4f763f4700 10 mon.0@0(leader).log v13552 update_from_paxos 2017-03-02 15:08:33.302285 7f4f763f4700 10 mon.0@0(leader).log v13552 update_from_paxos version 13552 summary v 13552 2017-03-02 15:08:33.302291 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:33.302300 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:33.302301 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:33.302309 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:33.302310 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12972) post_refresh 2017-03-02 15:08:33.302311 7f4f763f4700 10 mon.0@0(leader).pg v12972 post_paxos_update 2017-03-02 15:08:33.302312 7f4f763f4700 10 mon.0@0(leader).pg v12972 check_subs 2017-03-02 15:08:33.302314 7f4f763f4700 20 mon.0@0(leader).pg v12972 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:33.302318 7f4f763f4700 20 mon.0@0(leader).pg v12972 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:33.302320 7f4f763f4700 20 mon.0@0(leader).pg v12972 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:33.302322 7f4f763f4700 20 mon.0@0(leader).pg v12972 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:33.302323 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..736) post_refresh 2017-03-02 15:08:33.302324 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:33.302325 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13552) post_refresh 2017-03-02 15:08:33.302326 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:33.302326 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:33.302327 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:33.302328 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27342) commit_proposal 2017-03-02 15:08:33.302330 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12972) _active - not active 2017-03-02 15:08:33.302332 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27342) finish_round 2017-03-02 15:08:33.302333 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27342) finish_round waiting_for_acting 2017-03-02 15:08:33.302334 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12972) _active 2017-03-02 15:08:33.302335 7f4f763f4700 7 mon.0@0(leader).paxosservice(pgmap 12269..12972) _active creating new pending 2017-03-02 15:08:33.302337 7f4f763f4700 10 mon.0@0(leader).pg v12972 create_pending v 12973 2017-03-02 15:08:33.302340 7f4f763f4700 7 mon.0@0(leader).pg v12972 _updated_stats for osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:33.302343 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a142adb00 0x558a14f25e00 pg_stats_ack(1 pgs tid 6799) v1 2017-03-02 15:08:33.302345 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6800/27100 -- pg_stats_ack(1 pgs tid 6799) v1 -- 0x558a14f25e00 con 0 2017-03-02 15:08:33.302367 7f4f763f4700 10 mon.0@0(leader).pg v12972 check_osd_map already seen 7 >= 7 2017-03-02 15:08:33.302370 7f4f763f4700 10 mon.0@0(leader).pg v12972 update_logger 2017-03-02 15:08:33.302383 7f4f763f4700 0 log_channel(cluster) log [INF] : pgmap v12972: 32 pgs: 32 active+clean; 86080 MB data, 163 GB used, 208 GB / 372 GB avail; 394 kB/s rd, 20232 kB/s wr, 1052 op/s 2017-03-02 15:08:33.302398 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14116 at 2017-03-02 15:08:33.302384) v1 -- 0x558a16743d40 con 0 2017-03-02 15:08:33.302411 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27342) finish_round waiting_for_readable 2017-03-02 15:08:33.302413 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27342) finish_round waiting_for_writeable 2017-03-02 15:08:33.302415 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27342) finish_round done w/ waiters, state active 2017-03-02 15:08:33.302437 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14116 at 2017-03-02 15:08:33.302384) v1 ==== 0+0+0 (0 0 0) 0x558a16743d40 con 0x558a14482000 2017-03-02 15:08:33.302457 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:33.302462 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:33.302465 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13552) dispatch 0x558a16743d40 log(1 entries from seq 14116 at 2017-03-02 15:08:33.302384) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:33.302471 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27342) is_readable = 1 - now=2017-03-02 15:08:33.302472 lease_expire=0.000000 has v0 lc 27342 2017-03-02 15:08:33.302478 7f4f71beb700 10 mon.0@0(leader).log v13552 preprocess_query log(1 entries from seq 14116 at 2017-03-02 15:08:33.302384) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:33.302485 7f4f71beb700 10 mon.0@0(leader).log v13552 preprocess_log log(1 entries from seq 14116 at 2017-03-02 15:08:33.302384) v1 from mon.0 2017-03-02 15:08:33.302489 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:33.302492 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:33.302493 7f4f71beb700 20 allow all 2017-03-02 15:08:33.302501 7f4f71beb700 10 mon.0@0(leader).log v13552 prepare_update log(1 entries from seq 14116 at 2017-03-02 15:08:33.302384) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:33.302507 7f4f71beb700 10 mon.0@0(leader).log v13552 prepare_log log(1 entries from seq 14116 at 2017-03-02 15:08:33.302384) v1 from mon.0 2017-03-02 15:08:33.302511 7f4f71beb700 10 mon.0@0(leader).log v13552 logging 2017-03-02 15:08:33.302384 mon.0 172.21.15.17:6789/0 14116 : cluster [INF] pgmap v12972: 32 pgs: 32 active+clean; 86080 MB data, 163 GB used, 208 GB / 372 GB avail; 394 kB/s rd, 20232 kB/s wr, 1052 op/s 2017-03-02 15:08:33.302520 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13552) setting proposal_timer 0x558a1412f780 with delay of 0.999669 2017-03-02 15:08:33.465252 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.1 172.21.15.22:6800/26531 6835 ==== pg_stats(5 pgs tid 6799 v 0) v1 ==== 2974+0+0 (352113976 0 0) 0x558a15d13680 con 0x558a14643800 2017-03-02 15:08:33.465284 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1463b1e0 for osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:33.465291 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:33.465294 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12972) dispatch 0x558a15d13680 pg_stats(5 pgs tid 6799 v 0) v1 from osd.1 172.21.15.22:6800/26531 con 0x558a14643800 2017-03-02 15:08:33.465301 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27342) is_readable = 1 - now=2017-03-02 15:08:33.465301 lease_expire=0.000000 has v0 lc 27342 2017-03-02 15:08:33.465310 7f4f71beb700 10 mon.0@0(leader).pg v12972 preprocess_query pg_stats(5 pgs tid 6799 v 0) v1 from osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:33.465316 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:33.465319 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:33.465320 7f4f71beb700 20 allow all 2017-03-02 15:08:33.465322 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27342) is_readable = 1 - now=2017-03-02 15:08:33.465322 lease_expire=0.000000 has v0 lc 27342 2017-03-02 15:08:33.465327 7f4f71beb700 10 mon.0@0(leader).pg v12972 prepare_update pg_stats(5 pgs tid 6799 v 0) v1 from osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:33.465332 7f4f71beb700 10 mon.0@0(leader).pg v12972 prepare_pg_stats pg_stats(5 pgs tid 6799 v 0) v1 from osd.1 2017-03-02 15:08:33.465340 7f4f71beb700 10 mon.0@0(leader).pg v12972 got osd.1 osd_stat(51614 MB used, 43726 MB avail, 95341 MB total, peers [0,2,3] op hist [0,0,2]) (was osd_stat(52172 MB used, 43168 MB avail, 95341 MB total, peers [0,2,3] op hist [0,1])) 2017-03-02 15:08:33.465350 7f4f71beb700 15 mon.0@0(leader).pg v12972 got 1.1 reported at 7:586994 state active+clean -> active+clean 2017-03-02 15:08:33.465361 7f4f71beb700 15 mon.0@0(leader).pg v12972 got 1.5 reported at 7:567866 state active+clean -> active+clean 2017-03-02 15:08:33.465366 7f4f71beb700 15 mon.0@0(leader).pg v12972 got 1.6 reported at 7:503243 state active+clean -> active+clean 2017-03-02 15:08:33.465372 7f4f71beb700 15 mon.0@0(leader).pg v12972 got 2.4 reported at 7:1026332 state active+clean -> active+clean 2017-03-02 15:08:33.465377 7f4f71beb700 15 mon.0@0(leader).pg v12972 got 2.5 reported at 7:1129681 state active+clean -> active+clean 2017-03-02 15:08:33.465384 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12972) setting proposal_timer 0x558a1412f230 with delay of 0.836806 2017-03-02 15:08:33.558586 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.2 172.21.15.11:6800/26536 6835 ==== pg_stats(8 pgs tid 6799 v 0) v1 ==== 4676+0+0 (1546459290 0 0) 0x558a15d9c500 con 0x558a145d5800 2017-03-02 15:08:33.558616 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a145c3700 for osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:33.558622 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:33.558624 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12972) dispatch 0x558a15d9c500 pg_stats(8 pgs tid 6799 v 0) v1 from osd.2 172.21.15.11:6800/26536 con 0x558a145d5800 2017-03-02 15:08:33.558630 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27342) is_readable = 1 - now=2017-03-02 15:08:33.558630 lease_expire=0.000000 has v0 lc 27342 2017-03-02 15:08:33.558638 7f4f71beb700 10 mon.0@0(leader).pg v12972 preprocess_query pg_stats(8 pgs tid 6799 v 0) v1 from osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:33.558645 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:33.558648 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:33.558649 7f4f71beb700 20 allow all 2017-03-02 15:08:33.558650 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27342) is_readable = 1 - now=2017-03-02 15:08:33.558651 lease_expire=0.000000 has v0 lc 27342 2017-03-02 15:08:33.558656 7f4f71beb700 10 mon.0@0(leader).pg v12972 prepare_update pg_stats(8 pgs tid 6799 v 0) v1 from osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:33.558661 7f4f71beb700 10 mon.0@0(leader).pg v12972 prepare_pg_stats pg_stats(8 pgs tid 6799 v 0) v1 from osd.2 2017-03-02 15:08:33.558667 7f4f71beb700 10 mon.0@0(leader).pg v12972 got osd.2 osd_stat(61738 MB used, 33602 MB avail, 95341 MB total, peers [0,1,3] op hist [0,2,3,2]) (was osd_stat(63400 MB used, 31941 MB avail, 95341 MB total, peers [0,1,3] op hist [0,2,0,1])) 2017-03-02 15:08:33.558677 7f4f71beb700 15 mon.0@0(leader).pg v12972 got 1.0 reported at 7:496674 state active+clean -> active+clean 2017-03-02 15:08:33.558687 7f4f71beb700 15 mon.0@0(leader).pg v12972 got 1.2 reported at 7:577479 state active+clean -> active+clean 2017-03-02 15:08:33.558710 7f4f71beb700 15 mon.0@0(leader).pg v12972 got 1.7 reported at 7:529841 state active+clean -> active+clean 2017-03-02 15:08:33.558715 7f4f71beb700 15 mon.0@0(leader).pg v12972 got 2.0 reported at 7:1056893 state active+clean -> active+clean 2017-03-02 15:08:33.558720 7f4f71beb700 15 mon.0@0(leader).pg v12972 got 2.1 reported at 7:1089651 state active+clean -> active+clean 2017-03-02 15:08:33.558725 7f4f71beb700 15 mon.0@0(leader).pg v12972 got 2.2 reported at 7:1163420 state active+clean -> active+clean 2017-03-02 15:08:33.558731 7f4f71beb700 15 mon.0@0(leader).pg v12972 got 2.3 reported at 7:1095487 state active+clean -> active+clean 2017-03-02 15:08:33.558736 7f4f71beb700 15 mon.0@0(leader).pg v12972 got 2.7 reported at 7:1135577 state active+clean -> active+clean 2017-03-02 15:08:33.558742 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12972) proposal_timer already set 2017-03-02 15:08:33.713827 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.0 172.21.15.26:6800/26515 6835 ==== pg_stats(2 pgs tid 6799 v 0) v1 ==== 1264+0+0 (3766816235 0 0) 0x558a15da2800 con 0x558a145d8800 2017-03-02 15:08:33.713858 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1463af20 for osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:33.713865 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:33.713868 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12972) dispatch 0x558a15da2800 pg_stats(2 pgs tid 6799 v 0) v1 from osd.0 172.21.15.26:6800/26515 con 0x558a145d8800 2017-03-02 15:08:33.713874 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27342) is_readable = 1 - now=2017-03-02 15:08:33.713874 lease_expire=0.000000 has v0 lc 27342 2017-03-02 15:08:33.713883 7f4f71beb700 10 mon.0@0(leader).pg v12972 preprocess_query pg_stats(2 pgs tid 6799 v 0) v1 from osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:33.713890 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:33.713892 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:33.713893 7f4f71beb700 20 allow all 2017-03-02 15:08:33.713895 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27342) is_readable = 1 - now=2017-03-02 15:08:33.713895 lease_expire=0.000000 has v0 lc 27342 2017-03-02 15:08:33.713902 7f4f71beb700 10 mon.0@0(leader).pg v12972 prepare_update pg_stats(2 pgs tid 6799 v 0) v1 from osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:33.713907 7f4f71beb700 10 mon.0@0(leader).pg v12972 prepare_pg_stats pg_stats(2 pgs tid 6799 v 0) v1 from osd.0 2017-03-02 15:08:33.713914 7f4f71beb700 10 mon.0@0(leader).pg v12972 got osd.0 osd_stat(10661 MB used, 84679 MB avail, 95341 MB total, peers [1,2,3] op hist []) (was osd_stat(10912 MB used, 84428 MB avail, 95341 MB total, peers [1,2,3] op hist [])) 2017-03-02 15:08:33.713924 7f4f71beb700 15 mon.0@0(leader).pg v12972 got 1.3 reported at 7:500294 state active+clean -> active+clean 2017-03-02 15:08:33.713934 7f4f71beb700 15 mon.0@0(leader).pg v12972 got 1.4 reported at 7:486238 state active+clean -> active+clean 2017-03-02 15:08:33.713940 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12972) proposal_timer already set 2017-03-02 15:08:34.302266 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13552) propose_pending 2017-03-02 15:08:34.302281 7f4f743f0700 10 mon.0@0(leader).log v13552 encode_full log v 13552 2017-03-02 15:08:34.302328 7f4f743f0700 10 mon.0@0(leader).log v13552 encode_pending v13553 2017-03-02 15:08:34.302332 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27342) queue_pending_finisher 0x558a1412e600 2017-03-02 15:08:34.302335 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27342) trigger_propose active, proposing now 2017-03-02 15:08:34.302340 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27342) propose_pending 27343 10372 bytes 2017-03-02 15:08:34.302343 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27342) begin for 27343 10372 bytes 2017-03-02 15:08:34.330772 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27342) commit_start 27343 2017-03-02 15:08:34.330808 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12972) propose_pending 2017-03-02 15:08:34.330812 7f4f743f0700 10 mon.0@0(leader).pg v12972 encode_pending v 12973 2017-03-02 15:08:34.330912 7f4f743f0700 5 mon.0@0(leader).paxos(paxos writing c 26607..27342) queue_pending_finisher 0x558a1412f780 2017-03-02 15:08:34.330921 7f4f743f0700 10 mon.0@0(leader).paxos(paxos writing c 26607..27342) trigger_propose not active, will propose later 2017-03-02 15:08:34.364184 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27342) commit_finish 27343 2017-03-02 15:08:34.364212 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:34.364228 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12972) refresh 2017-03-02 15:08:34.364237 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..736) refresh 2017-03-02 15:08:34.364246 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:34.364256 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13553) refresh 2017-03-02 15:08:34.364257 7f4f763f4700 10 mon.0@0(leader).log v13553 update_from_paxos 2017-03-02 15:08:34.364258 7f4f763f4700 10 mon.0@0(leader).log v13553 update_from_paxos version 13553 summary v 13552 2017-03-02 15:08:34.364261 7f4f763f4700 10 mon.0@0(leader).log v13553 update_from_paxos latest full 13552 2017-03-02 15:08:34.364267 7f4f763f4700 7 mon.0@0(leader).log v13553 update_from_paxos applying incremental log 13553 2017-03-02 15:08:33.302384 mon.0 172.21.15.17:6789/0 14116 : cluster [INF] pgmap v12972: 32 pgs: 32 active+clean; 86080 MB data, 163 GB used, 208 GB / 372 GB avail; 394 kB/s rd, 20232 kB/s wr, 1052 op/s 2017-03-02 15:08:34.364278 7f4f763f4700 20 mon.0@0(leader).log v13553 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:34.364289 7f4f763f4700 15 mon.0@0(leader).log v13553 update_from_paxos logging for 1 channels 2017-03-02 15:08:34.364290 7f4f763f4700 15 mon.0@0(leader).log v13553 update_from_paxos channel 'cluster' logging 203 bytes 2017-03-02 15:08:34.364334 7f4f763f4700 10 mon.0@0(leader).log v13553 check_subs 2017-03-02 15:08:34.364359 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:34.364372 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:34.364377 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:34.364386 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:34.364388 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12972) post_refresh 2017-03-02 15:08:34.364389 7f4f763f4700 10 mon.0@0(leader).pg v12972 post_paxos_update 2017-03-02 15:08:34.364390 7f4f763f4700 10 mon.0@0(leader).pg v12972 check_subs 2017-03-02 15:08:34.364392 7f4f763f4700 20 mon.0@0(leader).pg v12972 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:34.364395 7f4f763f4700 20 mon.0@0(leader).pg v12972 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:34.364397 7f4f763f4700 20 mon.0@0(leader).pg v12972 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:34.364398 7f4f763f4700 20 mon.0@0(leader).pg v12972 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:34.364400 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..736) post_refresh 2017-03-02 15:08:34.364401 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:34.364402 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13553) post_refresh 2017-03-02 15:08:34.364403 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:34.364403 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:34.364404 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:34.364405 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27343) commit_proposal 2017-03-02 15:08:34.364408 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13553) _active - not active 2017-03-02 15:08:34.364409 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27343) finish_round 2017-03-02 15:08:34.364411 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27343) finish_round waiting_for_acting 2017-03-02 15:08:34.364412 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13553) _active 2017-03-02 15:08:34.364413 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13553) _active creating new pending 2017-03-02 15:08:34.364419 7f4f763f4700 10 mon.0@0(leader).log v13553 create_pending v 13554 2017-03-02 15:08:34.364421 7f4f763f4700 7 mon.0@0(leader).log v13553 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:34.364424 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a14955c00 0x558a1592d2a0 log(last 14116) v1 2017-03-02 15:08:34.364427 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14116) v1 -- 0x558a1592d2a0 con 0 2017-03-02 15:08:34.364439 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27343) finish_round waiting_for_readable 2017-03-02 15:08:34.364441 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27343) finish_round waiting_for_writeable 2017-03-02 15:08:34.364442 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27343) finish_round done w/ waiters, state active 2017-03-02 15:08:34.364460 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27343) propose_pending 27344 9764 bytes 2017-03-02 15:08:34.364464 7f4f763f4700 10 mon.0@0(leader).paxos(paxos updating c 26607..27343) begin for 27344 9764 bytes 2017-03-02 15:08:34.364478 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14116) v1 ==== 0+0+0 (0 0 0) 0x558a1592d2a0 con 0x558a14482000 2017-03-02 15:08:34.397659 7f4f763f4700 10 mon.0@0(leader).paxos(paxos updating c 26607..27343) commit_start 27344 2017-03-02 15:08:34.397743 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:34.397775 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:34.397779 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:34.397783 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:34.397784 7f4f71beb700 20 allow all 2017-03-02 15:08:34.431155 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27343) commit_finish 27344 2017-03-02 15:08:34.431188 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:34.431204 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12973) refresh 2017-03-02 15:08:34.431206 7f4f763f4700 10 mon.0@0(leader).pg v12972 update_from_paxos read_incremental 2017-03-02 15:08:34.431225 7f4f763f4700 20 mon.0@0(leader).pg v12972 refreshing pg 1.0 7:496674 active+clean 2017-03-02 15:08:34.431237 7f4f763f4700 20 mon.0@0(leader).pg v12972 refreshing pg 1.1 7:586994 active+clean 2017-03-02 15:08:34.431242 7f4f763f4700 20 mon.0@0(leader).pg v12972 refreshing pg 1.2 7:577479 active+clean 2017-03-02 15:08:34.431247 7f4f763f4700 20 mon.0@0(leader).pg v12972 refreshing pg 1.3 7:500294 active+clean 2017-03-02 15:08:34.431252 7f4f763f4700 20 mon.0@0(leader).pg v12972 refreshing pg 1.4 7:486238 active+clean 2017-03-02 15:08:34.431256 7f4f763f4700 20 mon.0@0(leader).pg v12972 refreshing pg 1.5 7:567866 active+clean 2017-03-02 15:08:34.431260 7f4f763f4700 20 mon.0@0(leader).pg v12972 refreshing pg 1.6 7:503243 active+clean 2017-03-02 15:08:34.431265 7f4f763f4700 20 mon.0@0(leader).pg v12972 refreshing pg 1.7 7:529841 active+clean 2017-03-02 15:08:34.431269 7f4f763f4700 20 mon.0@0(leader).pg v12972 refreshing pg 2.0 7:1056893 active+clean 2017-03-02 15:08:34.431273 7f4f763f4700 20 mon.0@0(leader).pg v12972 refreshing pg 2.1 7:1089651 active+clean 2017-03-02 15:08:34.431278 7f4f763f4700 20 mon.0@0(leader).pg v12972 refreshing pg 2.2 7:1163420 active+clean 2017-03-02 15:08:34.431282 7f4f763f4700 20 mon.0@0(leader).pg v12972 refreshing pg 2.3 7:1095487 active+clean 2017-03-02 15:08:34.431286 7f4f763f4700 20 mon.0@0(leader).pg v12972 refreshing pg 2.4 7:1026332 active+clean 2017-03-02 15:08:34.431290 7f4f763f4700 20 mon.0@0(leader).pg v12972 refreshing pg 2.5 7:1129681 active+clean 2017-03-02 15:08:34.431294 7f4f763f4700 20 mon.0@0(leader).pg v12972 refreshing pg 2.7 7:1135577 active+clean 2017-03-02 15:08:34.431295 7f4f763f4700 20 mon.0@0(leader).pg v12972 refreshing osd.0 2017-03-02 15:08:34.431301 7f4f763f4700 20 mon.0@0(leader).pg v12972 refreshing osd.1 2017-03-02 15:08:34.431304 7f4f763f4700 20 mon.0@0(leader).pg v12972 refreshing osd.2 2017-03-02 15:08:34.431313 7f4f763f4700 10 mon.0@0(leader).pg v12973 read_pgmap_meta 2017-03-02 15:08:34.431328 7f4f763f4700 10 mon.0@0(leader).pg v12973 update_logger 2017-03-02 15:08:34.431337 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..736) refresh 2017-03-02 15:08:34.431347 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:34.431359 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13553) refresh 2017-03-02 15:08:34.431360 7f4f763f4700 10 mon.0@0(leader).log v13553 update_from_paxos 2017-03-02 15:08:34.431362 7f4f763f4700 10 mon.0@0(leader).log v13553 update_from_paxos version 13553 summary v 13553 2017-03-02 15:08:34.431370 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:34.431379 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:34.431382 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:34.431391 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:34.431392 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12973) post_refresh 2017-03-02 15:08:34.431393 7f4f763f4700 10 mon.0@0(leader).pg v12973 post_paxos_update 2017-03-02 15:08:34.431394 7f4f763f4700 10 mon.0@0(leader).pg v12973 check_subs 2017-03-02 15:08:34.431396 7f4f763f4700 20 mon.0@0(leader).pg v12973 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:34.431401 7f4f763f4700 20 mon.0@0(leader).pg v12973 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:34.431403 7f4f763f4700 20 mon.0@0(leader).pg v12973 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:34.431405 7f4f763f4700 20 mon.0@0(leader).pg v12973 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:34.431407 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..736) post_refresh 2017-03-02 15:08:34.431408 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:34.431409 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13553) post_refresh 2017-03-02 15:08:34.431410 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:34.431410 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:34.431411 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:34.431413 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27344) commit_proposal 2017-03-02 15:08:34.431415 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12973) _active - not active 2017-03-02 15:08:34.431416 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27344) finish_round 2017-03-02 15:08:34.431418 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27344) finish_round waiting_for_acting 2017-03-02 15:08:34.431419 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12973) _active 2017-03-02 15:08:34.431421 7f4f763f4700 7 mon.0@0(leader).paxosservice(pgmap 12269..12973) _active creating new pending 2017-03-02 15:08:34.431427 7f4f763f4700 10 mon.0@0(leader).pg v12973 create_pending v 12974 2017-03-02 15:08:34.431431 7f4f763f4700 7 mon.0@0(leader).pg v12973 _updated_stats for osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:34.431434 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a15a18300 0x558a150dfe00 pg_stats_ack(5 pgs tid 6799) v1 2017-03-02 15:08:34.431437 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6800/26531 -- pg_stats_ack(5 pgs tid 6799) v1 -- 0x558a150dfe00 con 0 2017-03-02 15:08:34.431462 7f4f763f4700 7 mon.0@0(leader).pg v12973 _updated_stats for osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:34.431465 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a1645ea00 0x558a15562a00 pg_stats_ack(8 pgs tid 6799) v1 2017-03-02 15:08:34.431466 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6800/26536 -- pg_stats_ack(8 pgs tid 6799) v1 -- 0x558a15562a00 con 0 2017-03-02 15:08:34.431477 7f4f763f4700 7 mon.0@0(leader).pg v12973 _updated_stats for osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:34.431480 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a15f38d00 0x558a1547c600 pg_stats_ack(2 pgs tid 6799) v1 2017-03-02 15:08:34.431481 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6800/26515 -- pg_stats_ack(2 pgs tid 6799) v1 -- 0x558a1547c600 con 0 2017-03-02 15:08:34.431496 7f4f763f4700 10 mon.0@0(leader).pg v12973 check_osd_map already seen 7 >= 7 2017-03-02 15:08:34.431498 7f4f763f4700 10 mon.0@0(leader).pg v12973 update_logger 2017-03-02 15:08:34.431509 7f4f763f4700 0 log_channel(cluster) log [INF] : pgmap v12973: 32 pgs: 32 active+clean; 84371 MB data, 161 GB used, 211 GB / 372 GB avail; 522 kB/s rd, 32084 kB/s wr, 1614 op/s 2017-03-02 15:08:34.431519 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14117 at 2017-03-02 15:08:34.431511) v1 -- 0x558a16743f80 con 0 2017-03-02 15:08:34.431530 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27344) finish_round waiting_for_readable 2017-03-02 15:08:34.431533 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27344) finish_round waiting_for_writeable 2017-03-02 15:08:34.431534 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27344) finish_round done w/ waiters, state active 2017-03-02 15:08:34.431563 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14117 at 2017-03-02 15:08:34.431511) v1 ==== 0+0+0 (0 0 0) 0x558a16743f80 con 0x558a14482000 2017-03-02 15:08:34.431580 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:34.431584 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:34.431585 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13553) dispatch 0x558a16743f80 log(1 entries from seq 14117 at 2017-03-02 15:08:34.431511) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:34.431591 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27344) is_readable = 1 - now=2017-03-02 15:08:34.431592 lease_expire=0.000000 has v0 lc 27344 2017-03-02 15:08:34.431596 7f4f71beb700 10 mon.0@0(leader).log v13553 preprocess_query log(1 entries from seq 14117 at 2017-03-02 15:08:34.431511) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:34.431601 7f4f71beb700 10 mon.0@0(leader).log v13553 preprocess_log log(1 entries from seq 14117 at 2017-03-02 15:08:34.431511) v1 from mon.0 2017-03-02 15:08:34.431604 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:34.431606 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:34.431606 7f4f71beb700 20 allow all 2017-03-02 15:08:34.431612 7f4f71beb700 10 mon.0@0(leader).log v13553 prepare_update log(1 entries from seq 14117 at 2017-03-02 15:08:34.431511) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:34.431616 7f4f71beb700 10 mon.0@0(leader).log v13553 prepare_log log(1 entries from seq 14117 at 2017-03-02 15:08:34.431511) v1 from mon.0 2017-03-02 15:08:34.431618 7f4f71beb700 10 mon.0@0(leader).log v13553 logging 2017-03-02 15:08:34.431511 mon.0 172.21.15.17:6789/0 14117 : cluster [INF] pgmap v12973: 32 pgs: 32 active+clean; 84371 MB data, 161 GB used, 211 GB / 372 GB avail; 522 kB/s rd, 32084 kB/s wr, 1614 op/s 2017-03-02 15:08:34.431624 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13553) setting proposal_timer 0x558a1412def0 with delay of 0.999543 2017-03-02 15:08:34.870290 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.5 172.21.15.11:6804/1637448123 8787 ==== mdsbeacon(4116/e up:active seq 8500 v736) v7 ==== 365+0+0 (205502642 0 0) 0x558a15205380 con 0x558a1471d800 2017-03-02 15:08:34.870317 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b0f20 for mds.? 172.21.15.11:6804/1637448123 2017-03-02 15:08:34.870322 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:34.870324 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..736) dispatch 0x558a15205380 mdsbeacon(4116/e up:active seq 8500 v736) v7 from mds.5 172.21.15.11:6804/1637448123 con 0x558a1471d800 2017-03-02 15:08:34.870328 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27344) is_readable = 1 - now=2017-03-02 15:08:34.870329 lease_expire=0.000000 has v0 lc 27344 2017-03-02 15:08:34.870334 7f4f71beb700 10 mon.0@0(leader).mds e736 preprocess_query mdsbeacon(4116/e up:active seq 8500 v736) v7 from mds.5 172.21.15.11:6804/1637448123 2017-03-02 15:08:34.870339 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:34.870341 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:34.870342 7f4f71beb700 20 allow all 2017-03-02 15:08:34.870343 7f4f71beb700 12 mon.0@0(leader).mds e736 preprocess_beacon mdsbeacon(4116/e up:active seq 8500 v736) v7 from mds.5 172.21.15.11:6804/1637448123 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:34.870347 7f4f71beb700 10 mon.0@0(leader).mds e736 preprocess_beacon: GID exists in map: 4116 2017-03-02 15:08:34.870351 7f4f71beb700 15 mon.0@0(leader).mds e736 _note_beacon mdsbeacon(4116/e up:active seq 8500 v736) v7 noting time 2017-03-02 15:08:34.870353 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a16384100 0x558a15204000 mdsbeacon(4116/e up:active seq 8500 v736) v7 2017-03-02 15:08:34.870355 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6804/1637448123 -- mdsbeacon(4116/e up:active seq 8500 v736) v7 -- 0x558a15204000 con 0 2017-03-02 15:08:34.884208 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.3 172.21.15.11:6805/1926402648 8563 ==== mdsbeacon(4118/f up:active seq 8500 v736) v7 ==== 406+0+0 (3461744336 0 0) 0x558a160be9c0 con 0x558a14905800 2017-03-02 15:08:34.884226 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b2260 for mds.? 172.21.15.11:6805/1926402648 2017-03-02 15:08:34.884229 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:34.884231 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..736) dispatch 0x558a160be9c0 mdsbeacon(4118/f up:active seq 8500 v736) v7 from mds.3 172.21.15.11:6805/1926402648 con 0x558a14905800 2017-03-02 15:08:34.884235 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27344) is_readable = 1 - now=2017-03-02 15:08:34.884235 lease_expire=0.000000 has v0 lc 27344 2017-03-02 15:08:34.884239 7f4f71beb700 10 mon.0@0(leader).mds e736 preprocess_query mdsbeacon(4118/f up:active seq 8500 v736) v7 from mds.3 172.21.15.11:6805/1926402648 2017-03-02 15:08:34.884244 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:34.884245 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:34.884246 7f4f71beb700 20 allow all 2017-03-02 15:08:34.884246 7f4f71beb700 12 mon.0@0(leader).mds e736 preprocess_beacon mdsbeacon(4118/f up:active seq 8500 v736) v7 from mds.3 172.21.15.11:6805/1926402648 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:34.884250 7f4f71beb700 10 mon.0@0(leader).mds e736 preprocess_beacon: GID exists in map: 4118 2017-03-02 15:08:34.884254 7f4f71beb700 15 mon.0@0(leader).mds e736 _note_beacon mdsbeacon(4118/f up:active seq 8500 v736) v7 noting time 2017-03-02 15:08:34.884256 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a16384100 0x558a15205380 mdsbeacon(4118/f up:active seq 8500 v736) v7 2017-03-02 15:08:34.884258 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6805/1926402648 -- mdsbeacon(4118/f up:active seq 8500 v736) v7 -- 0x558a15205380 con 0 2017-03-02 15:08:34.884954 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.2 172.21.15.22:6804/673088617 8562 ==== mdsbeacon(4119/c up:active seq 8500 v736) v7 ==== 365+0+0 (740736236 0 0) 0x558a15204000 con 0x558a14907000 2017-03-02 15:08:34.884982 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14916f20 for mds.? 172.21.15.22:6804/673088617 2017-03-02 15:08:34.884989 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:34.884991 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..736) dispatch 0x558a15204000 mdsbeacon(4119/c up:active seq 8500 v736) v7 from mds.2 172.21.15.22:6804/673088617 con 0x558a14907000 2017-03-02 15:08:34.884998 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27344) is_readable = 1 - now=2017-03-02 15:08:34.884999 lease_expire=0.000000 has v0 lc 27344 2017-03-02 15:08:34.885008 7f4f71beb700 10 mon.0@0(leader).mds e736 preprocess_query mdsbeacon(4119/c up:active seq 8500 v736) v7 from mds.2 172.21.15.22:6804/673088617 2017-03-02 15:08:34.885015 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:34.885018 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:34.885020 7f4f71beb700 20 allow all 2017-03-02 15:08:34.885021 7f4f71beb700 12 mon.0@0(leader).mds e736 preprocess_beacon mdsbeacon(4119/c up:active seq 8500 v736) v7 from mds.2 172.21.15.22:6804/673088617 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:34.885030 7f4f71beb700 10 mon.0@0(leader).mds e736 preprocess_beacon: GID exists in map: 4119 2017-03-02 15:08:34.885035 7f4f71beb700 15 mon.0@0(leader).mds e736 _note_beacon mdsbeacon(4119/c up:active seq 8500 v736) v7 noting time 2017-03-02 15:08:34.885039 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a16384100 0x558a160be9c0 mdsbeacon(4119/c up:active seq 8500 v736) v7 2017-03-02 15:08:34.885043 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6804/673088617 -- mdsbeacon(4119/c up:active seq 8500 v736) v7 -- 0x558a160be9c0 con 0 2017-03-02 15:08:34.885315 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.0 172.21.15.22:6805/2203629154 8612 ==== mdsbeacon(4121/d up:active seq 8496 v736) v7 ==== 452+0+0 (2455706224 0 0) 0x558a15205380 con 0x558a1492a000 2017-03-02 15:08:34.885346 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a149195a0 for mds.? 172.21.15.22:6805/2203629154 2017-03-02 15:08:34.885349 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:34.885351 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..736) dispatch 0x558a15205380 mdsbeacon(4121/d up:active seq 8496 v736) v7 from mds.0 172.21.15.22:6805/2203629154 con 0x558a1492a000 2017-03-02 15:08:34.885354 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27344) is_readable = 1 - now=2017-03-02 15:08:34.885354 lease_expire=0.000000 has v0 lc 27344 2017-03-02 15:08:34.885358 7f4f71beb700 10 mon.0@0(leader).mds e736 preprocess_query mdsbeacon(4121/d up:active seq 8496 v736) v7 from mds.0 172.21.15.22:6805/2203629154 2017-03-02 15:08:34.885362 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:34.885363 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:34.885364 7f4f71beb700 20 allow all 2017-03-02 15:08:34.885365 7f4f71beb700 12 mon.0@0(leader).mds e736 preprocess_beacon mdsbeacon(4121/d up:active seq 8496 v736) v7 from mds.0 172.21.15.22:6805/2203629154 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:34.885369 7f4f71beb700 10 mon.0@0(leader).mds e736 preprocess_beacon: GID exists in map: 4121 2017-03-02 15:08:34.885372 7f4f71beb700 20 mon.0@0(leader).mds e736 preprocess_beacon health metrics for gid 4121 were updated 2017-03-02 15:08:34.885373 7f4f71beb700 15 mon.0@0(leader).mds e736 _note_beacon mdsbeacon(4121/d up:active seq 8496 v736) v7 noting time 2017-03-02 15:08:34.885375 7f4f71beb700 7 mon.0@0(leader).mds e736 prepare_update mdsbeacon(4121/d up:active seq 8496 v736) v7 2017-03-02 15:08:34.885377 7f4f71beb700 12 mon.0@0(leader).mds e736 prepare_beacon mdsbeacon(4121/d up:active seq 8496 v736) v7 from mds.0 172.21.15.22:6805/2203629154 2017-03-02 15:08:34.885379 7f4f71beb700 20 mon.0@0(leader).mds e736 prepare_beacon got health from gid 4121 with 1 metrics. 2017-03-02 15:08:34.885382 7f4f71beb700 10 mon.0@0(leader).mds e736 prepare_beacon mds.0 up:active -> up:active standby_for_rank=-1 2017-03-02 15:08:34.885384 7f4f71beb700 7 mon.0@0(leader).mds e736 prepare_beacon pending map now: 2017-03-02 15:08:34.885385 7f4f71beb700 7 mon.0@0(leader).mds e736 print_map 2017-03-02 15:08:34.885411 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..736) setting proposal_timer 0x558a1412ca70 with delay of 0.545757 2017-03-02 15:08:34.924258 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.1 172.21.15.17:6804/2576824570 8559 ==== mdsbeacon(4120/g up:active seq 8500 v736) v7 ==== 365+0+0 (3827954167 0 0) 0x558a154b8d00 con 0x558a14908800 2017-03-02 15:08:34.924284 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14917600 for mds.? 172.21.15.17:6804/2576824570 2017-03-02 15:08:34.924289 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:34.924291 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..736) dispatch 0x558a154b8d00 mdsbeacon(4120/g up:active seq 8500 v736) v7 from mds.1 172.21.15.17:6804/2576824570 con 0x558a14908800 2017-03-02 15:08:34.924295 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27344) is_readable = 1 - now=2017-03-02 15:08:34.924295 lease_expire=0.000000 has v0 lc 27344 2017-03-02 15:08:34.924303 7f4f71beb700 10 mon.0@0(leader).mds e736 preprocess_query mdsbeacon(4120/g up:active seq 8500 v736) v7 from mds.1 172.21.15.17:6804/2576824570 2017-03-02 15:08:34.924307 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:34.924309 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:34.924310 7f4f71beb700 20 allow all 2017-03-02 15:08:34.924310 7f4f71beb700 12 mon.0@0(leader).mds e736 preprocess_beacon mdsbeacon(4120/g up:active seq 8500 v736) v7 from mds.1 172.21.15.17:6804/2576824570 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:34.924315 7f4f71beb700 10 mon.0@0(leader).mds e736 preprocess_beacon: GID exists in map: 4120 2017-03-02 15:08:34.924318 7f4f71beb700 10 mon.0@0(leader).mds e736 mds_beacon mdsbeacon(4120/g up:active seq 8500 v736) v7 ignoring requested state, because mds hasn't seen latest map 2017-03-02 15:08:34.924319 7f4f71beb700 15 mon.0@0(leader).mds e736 _note_beacon mdsbeacon(4120/g up:active seq 8500 v736) v7 noting time 2017-03-02 15:08:34.924322 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a166b1a00 0x558a15204000 mdsbeacon(4120/g up:active seq 8500 v737) v7 2017-03-02 15:08:34.924324 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6804/2576824570 -- mdsbeacon(4120/g up:active seq 8500 v737) v7 -- 0x558a15204000 con 0 2017-03-02 15:08:34.969713 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.4 172.21.15.26:6805/2033388711 8557 ==== mdsbeacon(4117/b up:active seq 8500 v736) v7 ==== 365+0+0 (444657460 0 0) 0x558a15204000 con 0x558a14904000 2017-03-02 15:08:34.969736 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b1600 for mds.? 172.21.15.26:6805/2033388711 2017-03-02 15:08:34.969740 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:34.969742 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..736) dispatch 0x558a15204000 mdsbeacon(4117/b up:active seq 8500 v736) v7 from mds.4 172.21.15.26:6805/2033388711 con 0x558a14904000 2017-03-02 15:08:34.969746 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27344) is_readable = 1 - now=2017-03-02 15:08:34.969746 lease_expire=0.000000 has v0 lc 27344 2017-03-02 15:08:34.969752 7f4f71beb700 10 mon.0@0(leader).mds e736 preprocess_query mdsbeacon(4117/b up:active seq 8500 v736) v7 from mds.4 172.21.15.26:6805/2033388711 2017-03-02 15:08:34.969756 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:34.969758 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:34.969758 7f4f71beb700 20 allow all 2017-03-02 15:08:34.969759 7f4f71beb700 12 mon.0@0(leader).mds e736 preprocess_beacon mdsbeacon(4117/b up:active seq 8500 v736) v7 from mds.4 172.21.15.26:6805/2033388711 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:34.969764 7f4f71beb700 10 mon.0@0(leader).mds e736 preprocess_beacon: GID exists in map: 4117 2017-03-02 15:08:34.969767 7f4f71beb700 10 mon.0@0(leader).mds e736 mds_beacon mdsbeacon(4117/b up:active seq 8500 v736) v7 ignoring requested state, because mds hasn't seen latest map 2017-03-02 15:08:34.969768 7f4f71beb700 15 mon.0@0(leader).mds e736 _note_beacon mdsbeacon(4117/b up:active seq 8500 v736) v7 noting time 2017-03-02 15:08:34.969771 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a166b1a00 0x558a154b8d00 mdsbeacon(4117/b up:active seq 8500 v737) v7 2017-03-02 15:08:34.969773 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6805/2033388711 -- mdsbeacon(4117/b up:active seq 8500 v737) v7 -- 0x558a154b8d00 con 0 2017-03-02 15:08:34.978742 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.6 172.21.15.26:6804/2954782670 8750 ==== mdsbeacon(4115/a up:active seq 8500 v736) v7 ==== 365+0+0 (1097472860 0 0) 0x558a148d7380 con 0x558a1471f000 2017-03-02 15:08:34.978761 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14868d60 for mds.? 172.21.15.26:6804/2954782670 2017-03-02 15:08:34.978765 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:34.978767 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..736) dispatch 0x558a148d7380 mdsbeacon(4115/a up:active seq 8500 v736) v7 from mds.6 172.21.15.26:6804/2954782670 con 0x558a1471f000 2017-03-02 15:08:34.978771 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27344) is_readable = 1 - now=2017-03-02 15:08:34.978771 lease_expire=0.000000 has v0 lc 27344 2017-03-02 15:08:34.978776 7f4f71beb700 10 mon.0@0(leader).mds e736 preprocess_query mdsbeacon(4115/a up:active seq 8500 v736) v7 from mds.6 172.21.15.26:6804/2954782670 2017-03-02 15:08:34.978780 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:34.978782 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:34.978783 7f4f71beb700 20 allow all 2017-03-02 15:08:34.978783 7f4f71beb700 12 mon.0@0(leader).mds e736 preprocess_beacon mdsbeacon(4115/a up:active seq 8500 v736) v7 from mds.6 172.21.15.26:6804/2954782670 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:34.978788 7f4f71beb700 10 mon.0@0(leader).mds e736 preprocess_beacon: GID exists in map: 4115 2017-03-02 15:08:34.978791 7f4f71beb700 10 mon.0@0(leader).mds e736 mds_beacon mdsbeacon(4115/a up:active seq 8500 v736) v7 ignoring requested state, because mds hasn't seen latest map 2017-03-02 15:08:34.978793 7f4f71beb700 15 mon.0@0(leader).mds e736 _note_beacon mdsbeacon(4115/a up:active seq 8500 v736) v7 noting time 2017-03-02 15:08:34.978795 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a166b1a00 0x558a15204000 mdsbeacon(4115/a up:active seq 8500 v737) v7 2017-03-02 15:08:34.978797 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6804/2954782670 -- mdsbeacon(4115/a up:active seq 8500 v737) v7 -- 0x558a15204000 con 0 2017-03-02 15:08:35.431245 7f4f743f0700 10 mon.0@0(leader).paxosservice(mdsmap 1..736) propose_pending 2017-03-02 15:08:35.431292 7f4f743f0700 10 mon.0@0(leader).mds e736 encode_pending e737 2017-03-02 15:08:35.431367 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27344) queue_pending_finisher 0x558a1412f230 2017-03-02 15:08:35.431377 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27344) trigger_propose active, proposing now 2017-03-02 15:08:35.431388 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27344) propose_pending 27345 2413 bytes 2017-03-02 15:08:35.431392 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27344) begin for 27345 2413 bytes 2017-03-02 15:08:35.464472 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27344) commit_start 27345 2017-03-02 15:08:35.464553 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13553) propose_pending 2017-03-02 15:08:35.464559 7f4f743f0700 10 mon.0@0(leader).log v13553 encode_full log v 13553 2017-03-02 15:08:35.464655 7f4f743f0700 10 mon.0@0(leader).log v13553 encode_pending v13554 2017-03-02 15:08:35.464672 7f4f743f0700 5 mon.0@0(leader).paxos(paxos writing c 26607..27344) queue_pending_finisher 0x558a1412ca70 2017-03-02 15:08:35.464676 7f4f743f0700 10 mon.0@0(leader).paxos(paxos writing c 26607..27344) trigger_propose not active, will propose later 2017-03-02 15:08:35.477139 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27344) commit_finish 27345 2017-03-02 15:08:35.477177 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:35.477197 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12973) refresh 2017-03-02 15:08:35.477216 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) refresh 2017-03-02 15:08:35.477218 7f4f763f4700 10 mon.0@0(leader).mds e736 update_from_paxos version 737, my e 736 2017-03-02 15:08:35.477225 7f4f763f4700 10 mon.0@0(leader).mds e736 update_from_paxos got 737 2017-03-02 15:08:35.477264 7f4f763f4700 4 mon.0@0(leader).mds e737 new map 2017-03-02 15:08:35.477271 7f4f763f4700 0 mon.0@0(leader).mds e737 print_map 2017-03-02 15:08:35.477313 7f4f763f4700 20 mon.0@0(leader).mds e737 check_sub: mdsmap 2017-03-02 15:08:35.477319 7f4f763f4700 10 mon.0@0(leader).mds e737 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:35.477321 7f4f763f4700 10 mon.0@0(leader).mds e737 check_sub selected MDS map epoch 737 for namespace 1 for subscriber mds.? who wants epoch 737 2017-03-02 15:08:35.477336 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6804/2954782670 -- mdsmap(e 737) v1 -- 0x558a16743d40 con 0 2017-03-02 15:08:35.477362 7f4f763f4700 20 mon.0@0(leader).mds e737 check_sub: mdsmap 2017-03-02 15:08:35.477366 7f4f763f4700 10 mon.0@0(leader).mds e737 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:35.477368 7f4f763f4700 10 mon.0@0(leader).mds e737 check_sub selected MDS map epoch 737 for namespace 1 for subscriber mds.? who wants epoch 737 2017-03-02 15:08:35.477378 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6804/1637448123 -- mdsmap(e 737) v1 -- 0x558a167441c0 con 0 2017-03-02 15:08:35.477393 7f4f763f4700 20 mon.0@0(leader).mds e737 check_sub: mdsmap 2017-03-02 15:08:35.477396 7f4f763f4700 10 mon.0@0(leader).mds e737 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:35.477397 7f4f763f4700 10 mon.0@0(leader).mds e737 check_sub selected MDS map epoch 737 for namespace 1 for subscriber mds.? who wants epoch 737 2017-03-02 15:08:35.477407 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6805/2033388711 -- mdsmap(e 737) v1 -- 0x558a16744400 con 0 2017-03-02 15:08:35.477422 7f4f763f4700 20 mon.0@0(leader).mds e737 check_sub: mdsmap 2017-03-02 15:08:35.477425 7f4f763f4700 10 mon.0@0(leader).mds e737 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:35.477427 7f4f763f4700 10 mon.0@0(leader).mds e737 check_sub selected MDS map epoch 737 for namespace 1 for subscriber mds.? who wants epoch 737 2017-03-02 15:08:35.477436 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6805/1926402648 -- mdsmap(e 737) v1 -- 0x558a16744640 con 0 2017-03-02 15:08:35.477447 7f4f763f4700 20 mon.0@0(leader).mds e737 check_sub: mdsmap 2017-03-02 15:08:35.477451 7f4f763f4700 10 mon.0@0(leader).mds e737 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:35.477452 7f4f763f4700 10 mon.0@0(leader).mds e737 check_sub selected MDS map epoch 737 for namespace 1 for subscriber mds.? who wants epoch 737 2017-03-02 15:08:35.477461 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6804/673088617 -- mdsmap(e 737) v1 -- 0x558a16744880 con 0 2017-03-02 15:08:35.477469 7f4f763f4700 20 mon.0@0(leader).mds e737 check_sub: mdsmap 2017-03-02 15:08:35.477472 7f4f763f4700 10 mon.0@0(leader).mds e737 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:35.477473 7f4f763f4700 10 mon.0@0(leader).mds e737 check_sub selected MDS map epoch 737 for namespace 1 for subscriber mds.? who wants epoch 737 2017-03-02 15:08:35.477482 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6804/2576824570 -- mdsmap(e 737) v1 -- 0x558a16744ac0 con 0 2017-03-02 15:08:35.477489 7f4f763f4700 20 mon.0@0(leader).mds e737 check_sub: mdsmap 2017-03-02 15:08:35.477492 7f4f763f4700 10 mon.0@0(leader).mds e737 check_sub: is_mds=1, fscid= 1 2017-03-02 15:08:35.477493 7f4f763f4700 10 mon.0@0(leader).mds e737 check_sub selected MDS map epoch 737 for namespace 1 for subscriber mds.? who wants epoch 737 2017-03-02 15:08:35.477501 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6805/2203629154 -- mdsmap(e 737) v1 -- 0x558a16744d00 con 0 2017-03-02 15:08:35.477517 7f4f763f4700 20 mon.0@0(leader).mds e737 check_sub: mdsmap 2017-03-02 15:08:35.477519 7f4f763f4700 10 mon.0@0(leader).mds e737 check_sub: is_mds=0, fscid= 1 2017-03-02 15:08:35.477520 7f4f763f4700 10 mon.0@0(leader).mds e737 check_sub selected MDS map epoch 737 for namespace 1 for subscriber unknown.0 who wants epoch 737 2017-03-02 15:08:35.477531 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.132:0/2597832251 -- mdsmap(e 737) v1 -- 0x558a16744f40 con 0 2017-03-02 15:08:35.477544 7f4f763f4700 20 mon.0@0(leader).mds e737 check_sub: mdsmap 2017-03-02 15:08:35.477545 7f4f763f4700 10 mon.0@0(leader).mds e737 check_sub: is_mds=0, fscid= 1 2017-03-02 15:08:35.477546 7f4f763f4700 10 mon.0@0(leader).mds e737 check_sub selected MDS map epoch 737 for namespace 1 for subscriber unknown.0 who wants epoch 737 2017-03-02 15:08:35.477555 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.133:0/3792964262 -- mdsmap(e 737) v1 -- 0x558a16745180 con 0 2017-03-02 15:08:35.477571 7f4f763f4700 10 mon.0@0(leader).mds e737 update_logger 2017-03-02 15:08:35.477593 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:35.477608 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13553) refresh 2017-03-02 15:08:35.477610 7f4f763f4700 10 mon.0@0(leader).log v13553 update_from_paxos 2017-03-02 15:08:35.477611 7f4f763f4700 10 mon.0@0(leader).log v13553 update_from_paxos version 13553 summary v 13553 2017-03-02 15:08:35.477620 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:35.477631 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:35.477633 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:35.477643 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:35.477645 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12973) post_refresh 2017-03-02 15:08:35.477646 7f4f763f4700 10 mon.0@0(leader).pg v12973 post_paxos_update 2017-03-02 15:08:35.477648 7f4f763f4700 10 mon.0@0(leader).pg v12973 check_subs 2017-03-02 15:08:35.477650 7f4f763f4700 20 mon.0@0(leader).pg v12973 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:35.477655 7f4f763f4700 20 mon.0@0(leader).pg v12973 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:35.477657 7f4f763f4700 20 mon.0@0(leader).pg v12973 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:35.477659 7f4f763f4700 20 mon.0@0(leader).pg v12973 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:35.477661 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) post_refresh 2017-03-02 15:08:35.477662 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:35.477663 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13553) post_refresh 2017-03-02 15:08:35.477664 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:35.477665 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:35.477666 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:35.477668 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27345) commit_proposal 2017-03-02 15:08:35.477671 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) _active - not active 2017-03-02 15:08:35.477673 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27345) finish_round 2017-03-02 15:08:35.477674 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27345) finish_round waiting_for_acting 2017-03-02 15:08:35.477676 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) _active 2017-03-02 15:08:35.477677 7f4f763f4700 7 mon.0@0(leader).paxosservice(mdsmap 1..737) _active creating new pending 2017-03-02 15:08:35.477693 7f4f763f4700 10 mon.0@0(leader).mds e737 create_pending e738 2017-03-02 15:08:35.477698 7f4f763f4700 10 mon.0@0(leader).mds e737 _updated mds.0 mdsbeacon(4121/d up:active seq 8496 v736) v7 2017-03-02 15:08:35.477707 7f4f763f4700 0 log_channel(cluster) log [INF] : mds.0 172.21.15.22:6805/2203629154 up:active 2017-03-02 15:08:35.477718 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14118 at 2017-03-02 15:08:35.477708) v1 -- 0x558a16745600 con 0 2017-03-02 15:08:35.477735 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a16384100 0x558a143e56c0 mdsbeacon(4121/d up:active seq 8496 v737) v7 2017-03-02 15:08:35.477738 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6805/2203629154 -- mdsbeacon(4121/d up:active seq 8496 v737) v7 -- 0x558a143e56c0 con 0 2017-03-02 15:08:35.477762 7f4f763f4700 10 mon.0@0(leader).mds e737 e737: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:35.477780 7f4f763f4700 10 mon.0@0(leader).mds e737 update_logger 2017-03-02 15:08:35.477788 7f4f763f4700 0 log_channel(cluster) log [INF] : fsmap e737: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:35.477796 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14119 at 2017-03-02 15:08:35.477790) v1 -- 0x558a167453c0 con 0 2017-03-02 15:08:35.477809 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27345) finish_round waiting_for_readable 2017-03-02 15:08:35.477811 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27345) finish_round waiting_for_writeable 2017-03-02 15:08:35.477813 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27345) finish_round done w/ waiters, state active 2017-03-02 15:08:35.477835 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27345) propose_pending 27346 10371 bytes 2017-03-02 15:08:35.477841 7f4f763f4700 10 mon.0@0(leader).paxos(paxos updating c 26607..27345) begin for 27346 10371 bytes 2017-03-02 15:08:35.477901 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14118 at 2017-03-02 15:08:35.477708) v1 ==== 0+0+0 (0 0 0) 0x558a16745600 con 0x558a14482000 2017-03-02 15:08:35.506294 7f4f763f4700 10 mon.0@0(leader).paxos(paxos updating c 26607..27345) commit_start 27346 2017-03-02 15:08:35.506356 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:35.506375 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:35.506378 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13553) dispatch 0x558a16745600 log(1 entries from seq 14118 at 2017-03-02 15:08:35.477708) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:35.506387 7f4f71beb700 5 mon.0@0(leader).paxos(paxos writing c 26607..27345) is_readable = 1 - now=2017-03-02 15:08:35.506387 lease_expire=0.000000 has v0 lc 27345 2017-03-02 15:08:35.506394 7f4f71beb700 10 mon.0@0(leader).log v13553 preprocess_query log(1 entries from seq 14118 at 2017-03-02 15:08:35.477708) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:35.506399 7f4f71beb700 10 mon.0@0(leader).log v13553 preprocess_log log(1 entries from seq 14118 at 2017-03-02 15:08:35.477708) v1 from mon.0 2017-03-02 15:08:35.506403 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:35.506406 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:35.506407 7f4f71beb700 20 allow all 2017-03-02 15:08:35.506414 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13553) waiting for paxos -> writeable 2017-03-02 15:08:35.506423 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14119 at 2017-03-02 15:08:35.477790) v1 ==== 0+0+0 (0 0 0) 0x558a167453c0 con 0x558a14482000 2017-03-02 15:08:35.506431 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:35.506434 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:35.506438 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13553) dispatch 0x558a167453c0 log(1 entries from seq 14119 at 2017-03-02 15:08:35.477790) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:35.506442 7f4f71beb700 5 mon.0@0(leader).paxos(paxos writing c 26607..27345) is_readable = 1 - now=2017-03-02 15:08:35.506442 lease_expire=0.000000 has v0 lc 27345 2017-03-02 15:08:35.506445 7f4f71beb700 10 mon.0@0(leader).log v13553 preprocess_query log(1 entries from seq 14119 at 2017-03-02 15:08:35.477790) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:35.506449 7f4f71beb700 10 mon.0@0(leader).log v13553 preprocess_log log(1 entries from seq 14119 at 2017-03-02 15:08:35.477790) v1 from mon.0 2017-03-02 15:08:35.506452 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:35.506454 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:35.506455 7f4f71beb700 20 allow all 2017-03-02 15:08:35.506457 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13553) waiting for paxos -> writeable 2017-03-02 15:08:35.539761 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27345) commit_finish 27346 2017-03-02 15:08:35.539805 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:35.539856 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12973) refresh 2017-03-02 15:08:35.539881 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) refresh 2017-03-02 15:08:35.539903 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:35.539924 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13554) refresh 2017-03-02 15:08:35.539931 7f4f763f4700 10 mon.0@0(leader).log v13554 update_from_paxos 2017-03-02 15:08:35.539934 7f4f763f4700 10 mon.0@0(leader).log v13554 update_from_paxos version 13554 summary v 13553 2017-03-02 15:08:35.539939 7f4f763f4700 10 mon.0@0(leader).log v13554 update_from_paxos latest full 13553 2017-03-02 15:08:35.539949 7f4f763f4700 7 mon.0@0(leader).log v13554 update_from_paxos applying incremental log 13554 2017-03-02 15:08:34.431511 mon.0 172.21.15.17:6789/0 14117 : cluster [INF] pgmap v12973: 32 pgs: 32 active+clean; 84371 MB data, 161 GB used, 211 GB / 372 GB avail; 522 kB/s rd, 32084 kB/s wr, 1614 op/s 2017-03-02 15:08:35.539964 7f4f763f4700 20 mon.0@0(leader).log v13554 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:35.539983 7f4f763f4700 15 mon.0@0(leader).log v13554 update_from_paxos logging for 1 channels 2017-03-02 15:08:35.539991 7f4f763f4700 15 mon.0@0(leader).log v13554 update_from_paxos channel 'cluster' logging 203 bytes 2017-03-02 15:08:35.540057 7f4f763f4700 10 mon.0@0(leader).log v13554 check_subs 2017-03-02 15:08:35.540098 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:35.540120 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:35.540127 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:35.540142 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:35.540144 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12973) post_refresh 2017-03-02 15:08:35.540147 7f4f763f4700 10 mon.0@0(leader).pg v12973 post_paxos_update 2017-03-02 15:08:35.540149 7f4f763f4700 10 mon.0@0(leader).pg v12973 check_subs 2017-03-02 15:08:35.540151 7f4f763f4700 20 mon.0@0(leader).pg v12973 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:35.540157 7f4f763f4700 20 mon.0@0(leader).pg v12973 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:35.540160 7f4f763f4700 20 mon.0@0(leader).pg v12973 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:35.540163 7f4f763f4700 20 mon.0@0(leader).pg v12973 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:35.540166 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) post_refresh 2017-03-02 15:08:35.540167 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:35.540169 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13554) post_refresh 2017-03-02 15:08:35.540170 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:35.540172 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:35.540173 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:35.540176 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27346) commit_proposal 2017-03-02 15:08:35.540179 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13554) _active - not active 2017-03-02 15:08:35.540182 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27346) finish_round 2017-03-02 15:08:35.540183 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27346) finish_round waiting_for_acting 2017-03-02 15:08:35.540186 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13554) _active 2017-03-02 15:08:35.540187 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13554) _active creating new pending 2017-03-02 15:08:35.540195 7f4f763f4700 10 mon.0@0(leader).log v13554 create_pending v 13555 2017-03-02 15:08:35.540199 7f4f763f4700 7 mon.0@0(leader).log v13554 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:35.540204 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a15fc7400 0x558a15938000 log(last 14117) v1 2017-03-02 15:08:35.540208 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14117) v1 -- 0x558a15938000 con 0 2017-03-02 15:08:35.540230 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13554) dispatch 0x558a16745600 log(1 entries from seq 14118 at 2017-03-02 15:08:35.477708) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:35.540239 7f4f763f4700 5 mon.0@0(leader).paxos(paxos active c 26607..27346) is_readable = 1 - now=2017-03-02 15:08:35.540239 lease_expire=0.000000 has v0 lc 27346 2017-03-02 15:08:35.540245 7f4f763f4700 10 mon.0@0(leader).log v13554 preprocess_query log(1 entries from seq 14118 at 2017-03-02 15:08:35.477708) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:35.540251 7f4f763f4700 10 mon.0@0(leader).log v13554 preprocess_log log(1 entries from seq 14118 at 2017-03-02 15:08:35.477708) v1 from mon.0 2017-03-02 15:08:35.540254 7f4f763f4700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:35.540257 7f4f763f4700 20 allow so far , doing grant allow * 2017-03-02 15:08:35.540258 7f4f763f4700 20 allow all 2017-03-02 15:08:35.540262 7f4f763f4700 10 mon.0@0(leader).log v13554 prepare_update log(1 entries from seq 14118 at 2017-03-02 15:08:35.477708) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:35.540267 7f4f763f4700 10 mon.0@0(leader).log v13554 prepare_log log(1 entries from seq 14118 at 2017-03-02 15:08:35.477708) v1 from mon.0 2017-03-02 15:08:35.540271 7f4f763f4700 10 mon.0@0(leader).log v13554 logging 2017-03-02 15:08:35.477708 mon.0 172.21.15.17:6789/0 14118 : cluster [INF] mds.0 172.21.15.22:6805/2203629154 up:active 2017-03-02 15:08:35.540278 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13554) setting proposal_timer 0x558a1412ca70 with delay of 0.999502 2017-03-02 15:08:35.540280 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14117) v1 ==== 0+0+0 (0 0 0) 0x558a15938000 con 0x558a14482000 2017-03-02 15:08:35.540291 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13554) dispatch 0x558a167453c0 log(1 entries from seq 14119 at 2017-03-02 15:08:35.477790) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:35.540297 7f4f763f4700 5 mon.0@0(leader).paxos(paxos active c 26607..27346) is_readable = 1 - now=2017-03-02 15:08:35.540297 lease_expire=0.000000 has v0 lc 27346 2017-03-02 15:08:35.540301 7f4f763f4700 10 mon.0@0(leader).log v13554 preprocess_query log(1 entries from seq 14119 at 2017-03-02 15:08:35.477790) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:35.540306 7f4f763f4700 10 mon.0@0(leader).log v13554 preprocess_log log(1 entries from seq 14119 at 2017-03-02 15:08:35.477790) v1 from mon.0 2017-03-02 15:08:35.540309 7f4f763f4700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:35.540310 7f4f763f4700 20 allow so far , doing grant allow * 2017-03-02 15:08:35.540311 7f4f763f4700 20 allow all 2017-03-02 15:08:35.540314 7f4f763f4700 10 mon.0@0(leader).log v13554 prepare_update log(1 entries from seq 14119 at 2017-03-02 15:08:35.477790) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:35.540319 7f4f763f4700 10 mon.0@0(leader).log v13554 prepare_log log(1 entries from seq 14119 at 2017-03-02 15:08:35.477790) v1 from mon.0 2017-03-02 15:08:35.540322 7f4f763f4700 10 mon.0@0(leader).log v13554 logging 2017-03-02 15:08:35.477790 mon.0 172.21.15.17:6789/0 14119 : cluster [INF] fsmap e737: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:35.540329 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13554) proposal_timer already set 2017-03-02 15:08:35.540331 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27346) finish_round waiting_for_readable 2017-03-02 15:08:35.540333 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27346) finish_round waiting_for_writeable 2017-03-02 15:08:35.540336 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27346) finish_round done w/ waiters, state active 2017-03-02 15:08:35.540355 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:35.540360 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:35.540363 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:35.540365 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:35.540366 7f4f71beb700 20 allow all 2017-03-02 15:08:36.012134 7f4f743f0700 11 mon.0@0(leader) e2 tick 2017-03-02 15:08:36.012159 7f4f743f0700 10 mon.0@0(leader).pg v12973 v12973: 32 pgs: 32 active+clean; 84371 MB data, 161 GB used, 211 GB / 372 GB avail; 522 kB/s rd, 32084 kB/s wr, 1614 op/s 2017-03-02 15:08:36.012174 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12973) maybe_trim trim_to 12473 would only trim 204 < paxos_service_trim_min 250 2017-03-02 15:08:36.012178 7f4f743f0700 10 mon.0@0(leader).mds e737 e737: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:36.012209 7f4f743f0700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) maybe_trim trim_to 237 would only trim 236 < paxos_service_trim_min 250 2017-03-02 15:08:36.012212 7f4f743f0700 10 mon.0@0(leader).osd e7 e7: 4 osds: 4 up, 4 in 2017-03-02 15:08:36.012216 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27346) is_readable = 1 - now=2017-03-02 15:08:36.012216 lease_expire=0.000000 has v0 lc 27346 2017-03-02 15:08:36.012223 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27346) is_readable = 1 - now=2017-03-02 15:08:36.012223 lease_expire=0.000000 has v0 lc 27346 2017-03-02 15:08:36.012228 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27346) is_readable = 1 - now=2017-03-02 15:08:36.012228 lease_expire=0.000000 has v0 lc 27346 2017-03-02 15:08:36.012231 7f4f743f0700 10 mon.0@0(leader).osd e7 min_last_epoch_clean 6 2017-03-02 15:08:36.012233 7f4f743f0700 10 mon.0@0(leader).log v13554 log 2017-03-02 15:08:36.012234 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13554) maybe_trim trim_to 13054 would only trim 27 < paxos_service_trim_min 250 2017-03-02 15:08:36.012237 7f4f743f0700 10 mon.0@0(leader).auth v10 auth 2017-03-02 15:08:36.012241 7f4f743f0700 10 cephx keyserver: _check_rotating_secrets 2017-03-02 15:08:36.012249 7f4f743f0700 20 mon.0@0(leader) e2 sync_trim_providers 2017-03-02 15:08:36.539880 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13554) propose_pending 2017-03-02 15:08:36.539899 7f4f743f0700 10 mon.0@0(leader).log v13554 encode_full log v 13554 2017-03-02 15:08:36.539955 7f4f743f0700 10 mon.0@0(leader).log v13554 encode_pending v13555 2017-03-02 15:08:36.539966 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27346) queue_pending_finisher 0x558a1412d530 2017-03-02 15:08:36.539969 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27346) trigger_propose active, proposing now 2017-03-02 15:08:36.539975 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27346) propose_pending 27347 10490 bytes 2017-03-02 15:08:36.539977 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27346) begin for 27347 10490 bytes 2017-03-02 15:08:36.573076 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27346) commit_start 27347 2017-03-02 15:08:36.606533 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27346) commit_finish 27347 2017-03-02 15:08:36.606572 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:36.606592 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12973) refresh 2017-03-02 15:08:36.606602 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) refresh 2017-03-02 15:08:36.606613 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:36.606632 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13555) refresh 2017-03-02 15:08:36.606634 7f4f763f4700 10 mon.0@0(leader).log v13555 update_from_paxos 2017-03-02 15:08:36.606636 7f4f763f4700 10 mon.0@0(leader).log v13555 update_from_paxos version 13555 summary v 13554 2017-03-02 15:08:36.606640 7f4f763f4700 10 mon.0@0(leader).log v13555 update_from_paxos latest full 13554 2017-03-02 15:08:36.606647 7f4f763f4700 7 mon.0@0(leader).log v13555 update_from_paxos applying incremental log 13555 2017-03-02 15:08:35.477708 mon.0 172.21.15.17:6789/0 14118 : cluster [INF] mds.0 172.21.15.22:6805/2203629154 up:active 2017-03-02 15:08:36.606660 7f4f763f4700 20 mon.0@0(leader).log v13555 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:36.606675 7f4f763f4700 7 mon.0@0(leader).log v13555 update_from_paxos applying incremental log 13555 2017-03-02 15:08:35.477790 mon.0 172.21.15.17:6789/0 14119 : cluster [INF] fsmap e737: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:36.606683 7f4f763f4700 20 mon.0@0(leader).log v13555 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:36.606688 7f4f763f4700 15 mon.0@0(leader).log v13555 update_from_paxos logging for 1 channels 2017-03-02 15:08:36.606690 7f4f763f4700 15 mon.0@0(leader).log v13555 update_from_paxos channel 'cluster' logging 316 bytes 2017-03-02 15:08:36.606739 7f4f763f4700 10 mon.0@0(leader).log v13555 check_subs 2017-03-02 15:08:36.606772 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:36.606783 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:36.606785 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:36.606794 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:36.606796 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12973) post_refresh 2017-03-02 15:08:36.606797 7f4f763f4700 10 mon.0@0(leader).pg v12973 post_paxos_update 2017-03-02 15:08:36.606799 7f4f763f4700 10 mon.0@0(leader).pg v12973 check_subs 2017-03-02 15:08:36.606801 7f4f763f4700 20 mon.0@0(leader).pg v12973 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:36.606805 7f4f763f4700 20 mon.0@0(leader).pg v12973 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:36.606807 7f4f763f4700 20 mon.0@0(leader).pg v12973 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:36.606809 7f4f763f4700 20 mon.0@0(leader).pg v12973 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:36.606811 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) post_refresh 2017-03-02 15:08:36.606812 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:36.606813 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13555) post_refresh 2017-03-02 15:08:36.606814 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:36.606815 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:36.606816 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:36.606817 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27347) commit_proposal 2017-03-02 15:08:36.606843 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13555) _active - not active 2017-03-02 15:08:36.606845 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27347) finish_round 2017-03-02 15:08:36.606847 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27347) finish_round waiting_for_acting 2017-03-02 15:08:36.606848 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13555) _active 2017-03-02 15:08:36.606849 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13555) _active creating new pending 2017-03-02 15:08:36.606857 7f4f763f4700 10 mon.0@0(leader).log v13555 create_pending v 13556 2017-03-02 15:08:36.606860 7f4f763f4700 7 mon.0@0(leader).log v13555 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:36.606865 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a166b1a00 0x558a1592d2a0 log(last 14118) v1 2017-03-02 15:08:36.606868 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14118) v1 -- 0x558a1592d2a0 con 0 2017-03-02 15:08:36.606886 7f4f763f4700 7 mon.0@0(leader).log v13555 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:36.606889 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a15fc8f00 0x558a1599c5a0 log(last 14119) v1 2017-03-02 15:08:36.606890 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14119) v1 -- 0x558a1599c5a0 con 0 2017-03-02 15:08:36.606895 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27347) finish_round waiting_for_readable 2017-03-02 15:08:36.606897 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27347) finish_round waiting_for_writeable 2017-03-02 15:08:36.606898 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27347) finish_round done w/ waiters, state active 2017-03-02 15:08:36.606914 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14118) v1 ==== 0+0+0 (0 0 0) 0x558a1592d2a0 con 0x558a14482000 2017-03-02 15:08:36.606936 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:36.606943 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:36.606947 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:36.606951 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:36.606952 7f4f71beb700 20 allow all 2017-03-02 15:08:36.606971 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14119) v1 ==== 0+0+0 (0 0 0) 0x558a1599c5a0 con 0x558a14482000 2017-03-02 15:08:36.606981 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:36.606984 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:36.606986 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:36.606988 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:36.606989 7f4f71beb700 20 allow all 2017-03-02 15:08:38.263801 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.3 172.21.15.17:6800/27100 6836 ==== pg_stats(1 pgs tid 6800 v 0) v1 ==== 706+0+0 (1847319482 0 0) 0x558a15aea800 con 0x558a145d7000 2017-03-02 15:08:38.263861 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415f440 for osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:38.263869 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:38.263871 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12973) dispatch 0x558a15aea800 pg_stats(1 pgs tid 6800 v 0) v1 from osd.3 172.21.15.17:6800/27100 con 0x558a145d7000 2017-03-02 15:08:38.263876 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27347) is_readable = 1 - now=2017-03-02 15:08:38.263876 lease_expire=0.000000 has v0 lc 27347 2017-03-02 15:08:38.263883 7f4f71beb700 10 mon.0@0(leader).pg v12973 preprocess_query pg_stats(1 pgs tid 6800 v 0) v1 from osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:38.263887 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:38.263890 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:38.263891 7f4f71beb700 20 allow all 2017-03-02 15:08:38.263892 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27347) is_readable = 1 - now=2017-03-02 15:08:38.263892 lease_expire=0.000000 has v0 lc 27347 2017-03-02 15:08:38.263895 7f4f71beb700 10 mon.0@0(leader).pg v12973 prepare_update pg_stats(1 pgs tid 6800 v 0) v1 from osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:38.263898 7f4f71beb700 10 mon.0@0(leader).pg v12973 prepare_pg_stats pg_stats(1 pgs tid 6800 v 0) v1 from osd.3 2017-03-02 15:08:38.263903 7f4f71beb700 10 mon.0@0(leader).pg v12973 got osd.3 osd_stat(39526 MB used, 55815 MB avail, 95341 MB total, peers [0,1,2] op hist [0,1]) (was osd_stat(40871 MB used, 54469 MB avail, 95341 MB total, peers [0,1,2] op hist [0,2])) 2017-03-02 15:08:38.263910 7f4f71beb700 15 mon.0@0(leader).pg v12973 got 2.6 reported at 7:964776 state active+clean -> active+clean 2017-03-02 15:08:38.263917 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12973) setting proposal_timer 0x558a1412d960 with delay of 0.05 2017-03-02 15:08:38.314009 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12973) propose_pending 2017-03-02 15:08:38.314025 7f4f743f0700 10 mon.0@0(leader).pg v12973 encode_pending v 12974 2017-03-02 15:08:38.314053 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27347) queue_pending_finisher 0x558a1412ca70 2017-03-02 15:08:38.314056 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27347) trigger_propose active, proposing now 2017-03-02 15:08:38.314063 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27347) propose_pending 27348 1098 bytes 2017-03-02 15:08:38.314065 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27347) begin for 27348 1098 bytes 2017-03-02 15:08:38.318684 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27347) commit_start 27348 2017-03-02 15:08:38.327030 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27347) commit_finish 27348 2017-03-02 15:08:38.327057 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:38.327072 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12974) refresh 2017-03-02 15:08:38.327074 7f4f763f4700 10 mon.0@0(leader).pg v12973 update_from_paxos read_incremental 2017-03-02 15:08:38.327091 7f4f763f4700 20 mon.0@0(leader).pg v12973 refreshing pg 2.6 7:964776 active+clean 2017-03-02 15:08:38.327094 7f4f763f4700 20 mon.0@0(leader).pg v12973 refreshing osd.3 2017-03-02 15:08:38.327104 7f4f763f4700 10 mon.0@0(leader).pg v12974 read_pgmap_meta 2017-03-02 15:08:38.327116 7f4f763f4700 10 mon.0@0(leader).pg v12974 update_logger 2017-03-02 15:08:38.327124 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) refresh 2017-03-02 15:08:38.327134 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:38.327152 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13555) refresh 2017-03-02 15:08:38.327156 7f4f763f4700 10 mon.0@0(leader).log v13555 update_from_paxos 2017-03-02 15:08:38.327158 7f4f763f4700 10 mon.0@0(leader).log v13555 update_from_paxos version 13555 summary v 13555 2017-03-02 15:08:38.327166 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:38.327175 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:38.327177 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:38.327185 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:38.327186 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12974) post_refresh 2017-03-02 15:08:38.327187 7f4f763f4700 10 mon.0@0(leader).pg v12974 post_paxos_update 2017-03-02 15:08:38.327189 7f4f763f4700 10 mon.0@0(leader).pg v12974 check_subs 2017-03-02 15:08:38.327191 7f4f763f4700 20 mon.0@0(leader).pg v12974 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:38.327196 7f4f763f4700 20 mon.0@0(leader).pg v12974 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:38.327198 7f4f763f4700 20 mon.0@0(leader).pg v12974 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:38.327200 7f4f763f4700 20 mon.0@0(leader).pg v12974 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:38.327202 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) post_refresh 2017-03-02 15:08:38.327203 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:38.327204 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13555) post_refresh 2017-03-02 15:08:38.327204 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:38.327205 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:38.327206 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:38.327208 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27348) commit_proposal 2017-03-02 15:08:38.327210 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12974) _active - not active 2017-03-02 15:08:38.327211 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27348) finish_round 2017-03-02 15:08:38.327212 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27348) finish_round waiting_for_acting 2017-03-02 15:08:38.327214 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12974) _active 2017-03-02 15:08:38.327215 7f4f763f4700 7 mon.0@0(leader).paxosservice(pgmap 12269..12974) _active creating new pending 2017-03-02 15:08:38.327217 7f4f763f4700 10 mon.0@0(leader).pg v12974 create_pending v 12975 2017-03-02 15:08:38.327221 7f4f763f4700 7 mon.0@0(leader).pg v12974 _updated_stats for osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:38.327224 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a163bd400 0x558a14dc5400 pg_stats_ack(1 pgs tid 6800) v1 2017-03-02 15:08:38.327226 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6800/27100 -- pg_stats_ack(1 pgs tid 6800) v1 -- 0x558a14dc5400 con 0 2017-03-02 15:08:38.327250 7f4f763f4700 10 mon.0@0(leader).pg v12974 check_osd_map already seen 7 >= 7 2017-03-02 15:08:38.327252 7f4f763f4700 10 mon.0@0(leader).pg v12974 update_logger 2017-03-02 15:08:38.327263 7f4f763f4700 0 log_channel(cluster) log [INF] : pgmap v12974: 32 pgs: 32 active+clean; 84046 MB data, 159 GB used, 212 GB / 372 GB avail; 394 kB/s rd, 25799 kB/s wr, 1354 op/s 2017-03-02 15:08:38.327273 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14120 at 2017-03-02 15:08:38.327264) v1 -- 0x558a16745600 con 0 2017-03-02 15:08:38.327283 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27348) finish_round waiting_for_readable 2017-03-02 15:08:38.327284 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27348) finish_round waiting_for_writeable 2017-03-02 15:08:38.327286 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27348) finish_round done w/ waiters, state active 2017-03-02 15:08:38.327331 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14120 at 2017-03-02 15:08:38.327264) v1 ==== 0+0+0 (0 0 0) 0x558a16745600 con 0x558a14482000 2017-03-02 15:08:38.327351 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:38.327356 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:38.327359 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13555) dispatch 0x558a16745600 log(1 entries from seq 14120 at 2017-03-02 15:08:38.327264) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:38.327365 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27348) is_readable = 1 - now=2017-03-02 15:08:38.327365 lease_expire=0.000000 has v0 lc 27348 2017-03-02 15:08:38.327371 7f4f71beb700 10 mon.0@0(leader).log v13555 preprocess_query log(1 entries from seq 14120 at 2017-03-02 15:08:38.327264) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:38.327378 7f4f71beb700 10 mon.0@0(leader).log v13555 preprocess_log log(1 entries from seq 14120 at 2017-03-02 15:08:38.327264) v1 from mon.0 2017-03-02 15:08:38.327382 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:38.327384 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:38.327385 7f4f71beb700 20 allow all 2017-03-02 15:08:38.327393 7f4f71beb700 10 mon.0@0(leader).log v13555 prepare_update log(1 entries from seq 14120 at 2017-03-02 15:08:38.327264) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:38.327399 7f4f71beb700 10 mon.0@0(leader).log v13555 prepare_log log(1 entries from seq 14120 at 2017-03-02 15:08:38.327264) v1 from mon.0 2017-03-02 15:08:38.327402 7f4f71beb700 10 mon.0@0(leader).log v13555 logging 2017-03-02 15:08:38.327264 mon.0 172.21.15.17:6789/0 14120 : cluster [INF] pgmap v12974: 32 pgs: 32 active+clean; 84046 MB data, 159 GB used, 212 GB / 372 GB avail; 394 kB/s rd, 25799 kB/s wr, 1354 op/s 2017-03-02 15:08:38.327410 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13555) setting proposal_timer 0x558a1412f870 with delay of 0.999631 2017-03-02 15:08:38.465974 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.1 172.21.15.22:6800/26531 6836 ==== pg_stats(5 pgs tid 6800 v 0) v1 ==== 2966+0+0 (1693014274 0 0) 0x558a14d98f00 con 0x558a14643800 2017-03-02 15:08:38.466037 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1463b1e0 for osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:38.466045 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:38.466048 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12974) dispatch 0x558a14d98f00 pg_stats(5 pgs tid 6800 v 0) v1 from osd.1 172.21.15.22:6800/26531 con 0x558a14643800 2017-03-02 15:08:38.466054 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27348) is_readable = 1 - now=2017-03-02 15:08:38.466055 lease_expire=0.000000 has v0 lc 27348 2017-03-02 15:08:38.466064 7f4f71beb700 10 mon.0@0(leader).pg v12974 preprocess_query pg_stats(5 pgs tid 6800 v 0) v1 from osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:38.466072 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:38.466074 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:38.466076 7f4f71beb700 20 allow all 2017-03-02 15:08:38.466077 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27348) is_readable = 1 - now=2017-03-02 15:08:38.466078 lease_expire=0.000000 has v0 lc 27348 2017-03-02 15:08:38.466082 7f4f71beb700 10 mon.0@0(leader).pg v12974 prepare_update pg_stats(5 pgs tid 6800 v 0) v1 from osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:38.466087 7f4f71beb700 10 mon.0@0(leader).pg v12974 prepare_pg_stats pg_stats(5 pgs tid 6800 v 0) v1 from osd.1 2017-03-02 15:08:38.466095 7f4f71beb700 10 mon.0@0(leader).pg v12974 got osd.1 osd_stat(49225 MB used, 46115 MB avail, 95341 MB total, peers [0,2,3] op hist [2]) (was osd_stat(51614 MB used, 43726 MB avail, 95341 MB total, peers [0,2,3] op hist [0,0,2])) 2017-03-02 15:08:38.466106 7f4f71beb700 15 mon.0@0(leader).pg v12974 got 1.1 reported at 7:587088 state active+clean -> active+clean 2017-03-02 15:08:38.466116 7f4f71beb700 15 mon.0@0(leader).pg v12974 got 1.5 reported at 7:567937 state active+clean -> active+clean 2017-03-02 15:08:38.466121 7f4f71beb700 15 mon.0@0(leader).pg v12974 got 1.6 reported at 7:503328 state active+clean -> active+clean 2017-03-02 15:08:38.466126 7f4f71beb700 15 mon.0@0(leader).pg v12974 got 2.4 reported at 7:1026594 state active+clean -> active+clean 2017-03-02 15:08:38.466130 7f4f71beb700 15 mon.0@0(leader).pg v12974 got 2.5 reported at 7:1129924 state active+clean -> active+clean 2017-03-02 15:08:38.466136 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12974) setting proposal_timer 0x558a1412dd10 with delay of 0.860905 2017-03-02 15:08:38.559265 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.2 172.21.15.11:6800/26536 6836 ==== pg_stats(8 pgs tid 6800 v 0) v1 ==== 4672+0+0 (748521510 0 0) 0x558a15da0280 con 0x558a145d5800 2017-03-02 15:08:38.559315 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a145c3700 for osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:38.559324 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:38.559329 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12974) dispatch 0x558a15da0280 pg_stats(8 pgs tid 6800 v 0) v1 from osd.2 172.21.15.11:6800/26536 con 0x558a145d5800 2017-03-02 15:08:38.559338 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27348) is_readable = 1 - now=2017-03-02 15:08:38.559339 lease_expire=0.000000 has v0 lc 27348 2017-03-02 15:08:38.559357 7f4f71beb700 10 mon.0@0(leader).pg v12974 preprocess_query pg_stats(8 pgs tid 6800 v 0) v1 from osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:38.559367 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:38.559371 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:38.559373 7f4f71beb700 20 allow all 2017-03-02 15:08:38.559375 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27348) is_readable = 1 - now=2017-03-02 15:08:38.559376 lease_expire=0.000000 has v0 lc 27348 2017-03-02 15:08:38.559383 7f4f71beb700 10 mon.0@0(leader).pg v12974 prepare_update pg_stats(8 pgs tid 6800 v 0) v1 from osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:38.559389 7f4f71beb700 10 mon.0@0(leader).pg v12974 prepare_pg_stats pg_stats(8 pgs tid 6800 v 0) v1 from osd.2 2017-03-02 15:08:38.559399 7f4f71beb700 10 mon.0@0(leader).pg v12974 got osd.2 osd_stat(59585 MB used, 35756 MB avail, 95341 MB total, peers [0,1,3] op hist [0,0,1]) (was osd_stat(61738 MB used, 33602 MB avail, 95341 MB total, peers [0,1,3] op hist [0,2,3,2])) 2017-03-02 15:08:38.559413 7f4f71beb700 15 mon.0@0(leader).pg v12974 got 1.0 reported at 7:496749 state active+clean -> active+clean 2017-03-02 15:08:38.559428 7f4f71beb700 15 mon.0@0(leader).pg v12974 got 1.2 reported at 7:577621 state active+clean -> active+clean 2017-03-02 15:08:38.559436 7f4f71beb700 15 mon.0@0(leader).pg v12974 got 1.7 reported at 7:529924 state active+clean -> active+clean 2017-03-02 15:08:38.559443 7f4f71beb700 15 mon.0@0(leader).pg v12974 got 2.0 reported at 7:1057161 state active+clean -> active+clean 2017-03-02 15:08:38.559450 7f4f71beb700 15 mon.0@0(leader).pg v12974 got 2.1 reported at 7:1089895 state active+clean -> active+clean 2017-03-02 15:08:38.559457 7f4f71beb700 15 mon.0@0(leader).pg v12974 got 2.2 reported at 7:1163640 state active+clean -> active+clean 2017-03-02 15:08:38.559463 7f4f71beb700 15 mon.0@0(leader).pg v12974 got 2.3 reported at 7:1095730 state active+clean -> active+clean 2017-03-02 15:08:38.559470 7f4f71beb700 15 mon.0@0(leader).pg v12974 got 2.7 reported at 7:1135807 state active+clean -> active+clean 2017-03-02 15:08:38.559478 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12974) proposal_timer already set 2017-03-02 15:08:38.714655 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.0 172.21.15.26:6800/26515 6836 ==== pg_stats(2 pgs tid 6800 v 0) v1 ==== 1264+0+0 (2009003800 0 0) 0x558a15c08f00 con 0x558a145d8800 2017-03-02 15:08:38.714712 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1463af20 for osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:38.714726 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:38.714732 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12974) dispatch 0x558a15c08f00 pg_stats(2 pgs tid 6800 v 0) v1 from osd.0 172.21.15.26:6800/26515 con 0x558a145d8800 2017-03-02 15:08:38.714743 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27348) is_readable = 1 - now=2017-03-02 15:08:38.714744 lease_expire=0.000000 has v0 lc 27348 2017-03-02 15:08:38.714759 7f4f71beb700 10 mon.0@0(leader).pg v12974 preprocess_query pg_stats(2 pgs tid 6800 v 0) v1 from osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:38.714768 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:38.714771 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:38.714773 7f4f71beb700 20 allow all 2017-03-02 15:08:38.714775 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27348) is_readable = 1 - now=2017-03-02 15:08:38.714775 lease_expire=0.000000 has v0 lc 27348 2017-03-02 15:08:38.714780 7f4f71beb700 10 mon.0@0(leader).pg v12974 prepare_update pg_stats(2 pgs tid 6800 v 0) v1 from osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:38.714786 7f4f71beb700 10 mon.0@0(leader).pg v12974 prepare_pg_stats pg_stats(2 pgs tid 6800 v 0) v1 from osd.0 2017-03-02 15:08:38.714794 7f4f71beb700 10 mon.0@0(leader).pg v12974 got osd.0 osd_stat(10457 MB used, 84884 MB avail, 95341 MB total, peers [1,2,3] op hist []) (was osd_stat(10661 MB used, 84679 MB avail, 95341 MB total, peers [1,2,3] op hist [])) 2017-03-02 15:08:38.714806 7f4f71beb700 15 mon.0@0(leader).pg v12974 got 1.3 reported at 7:500364 state active+clean -> active+clean 2017-03-02 15:08:38.714853 7f4f71beb700 15 mon.0@0(leader).pg v12974 got 1.4 reported at 7:486316 state active+clean -> active+clean 2017-03-02 15:08:38.714862 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12974) proposal_timer already set 2017-03-02 15:08:38.870466 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.5 172.21.15.11:6804/1637448123 8788 ==== mdsbeacon(4116/e up:active seq 8501 v737) v7 ==== 365+0+0 (490441293 0 0) 0x558a15207740 con 0x558a1471d800 2017-03-02 15:08:38.870510 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b0f20 for mds.? 172.21.15.11:6804/1637448123 2017-03-02 15:08:38.870521 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:38.870524 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a15207740 mdsbeacon(4116/e up:active seq 8501 v737) v7 from mds.5 172.21.15.11:6804/1637448123 con 0x558a1471d800 2017-03-02 15:08:38.870530 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27348) is_readable = 1 - now=2017-03-02 15:08:38.870531 lease_expire=0.000000 has v0 lc 27348 2017-03-02 15:08:38.870539 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4116/e up:active seq 8501 v737) v7 from mds.5 172.21.15.11:6804/1637448123 2017-03-02 15:08:38.870545 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:38.870548 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:38.870549 7f4f71beb700 20 allow all 2017-03-02 15:08:38.870550 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4116/e up:active seq 8501 v737) v7 from mds.5 172.21.15.11:6804/1637448123 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:38.870557 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4116 2017-03-02 15:08:38.870562 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4116/e up:active seq 8501 v737) v7 noting time 2017-03-02 15:08:38.870566 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15d6a100 0x558a148d7380 mdsbeacon(4116/e up:active seq 8501 v737) v7 2017-03-02 15:08:38.870568 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6804/1637448123 -- mdsbeacon(4116/e up:active seq 8501 v737) v7 -- 0x558a148d7380 con 0 2017-03-02 15:08:38.884127 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.3 172.21.15.11:6805/1926402648 8564 ==== mdsbeacon(4118/f up:active seq 8501 v737) v7 ==== 406+0+0 (2588727492 0 0) 0x558a143e56c0 con 0x558a14905800 2017-03-02 15:08:38.884163 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b2260 for mds.? 172.21.15.11:6805/1926402648 2017-03-02 15:08:38.884172 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:38.884175 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a143e56c0 mdsbeacon(4118/f up:active seq 8501 v737) v7 from mds.3 172.21.15.11:6805/1926402648 con 0x558a14905800 2017-03-02 15:08:38.884182 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27348) is_readable = 1 - now=2017-03-02 15:08:38.884182 lease_expire=0.000000 has v0 lc 27348 2017-03-02 15:08:38.884189 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4118/f up:active seq 8501 v737) v7 from mds.3 172.21.15.11:6805/1926402648 2017-03-02 15:08:38.884194 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:38.884196 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:38.884197 7f4f71beb700 20 allow all 2017-03-02 15:08:38.884198 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4118/f up:active seq 8501 v737) v7 from mds.3 172.21.15.11:6805/1926402648 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:38.884205 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4118 2017-03-02 15:08:38.884209 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4118/f up:active seq 8501 v737) v7 noting time 2017-03-02 15:08:38.884212 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15d6a100 0x558a15207740 mdsbeacon(4118/f up:active seq 8501 v737) v7 2017-03-02 15:08:38.884214 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6805/1926402648 -- mdsbeacon(4118/f up:active seq 8501 v737) v7 -- 0x558a15207740 con 0 2017-03-02 15:08:38.884982 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.2 172.21.15.22:6804/673088617 8563 ==== mdsbeacon(4119/c up:active seq 8501 v737) v7 ==== 365+0+0 (1025674771 0 0) 0x558a148d7380 con 0x558a14907000 2017-03-02 15:08:38.885005 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14916f20 for mds.? 172.21.15.22:6804/673088617 2017-03-02 15:08:38.885009 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:38.885011 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a148d7380 mdsbeacon(4119/c up:active seq 8501 v737) v7 from mds.2 172.21.15.22:6804/673088617 con 0x558a14907000 2017-03-02 15:08:38.885016 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27348) is_readable = 1 - now=2017-03-02 15:08:38.885016 lease_expire=0.000000 has v0 lc 27348 2017-03-02 15:08:38.885022 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4119/c up:active seq 8501 v737) v7 from mds.2 172.21.15.22:6804/673088617 2017-03-02 15:08:38.885027 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:38.885029 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:38.885030 7f4f71beb700 20 allow all 2017-03-02 15:08:38.885031 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4119/c up:active seq 8501 v737) v7 from mds.2 172.21.15.22:6804/673088617 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:38.885037 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4119 2017-03-02 15:08:38.885040 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4119/c up:active seq 8501 v737) v7 noting time 2017-03-02 15:08:38.885044 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15d6a100 0x558a143e56c0 mdsbeacon(4119/c up:active seq 8501 v737) v7 2017-03-02 15:08:38.885046 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6804/673088617 -- mdsbeacon(4119/c up:active seq 8501 v737) v7 -- 0x558a143e56c0 con 0 2017-03-02 15:08:38.885194 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.0 172.21.15.22:6805/2203629154 8613 ==== mdsbeacon(4121/d up:active seq 8497 v736) v7 ==== 452+0+0 (2812144178 0 0) 0x558a15207740 con 0x558a1492a000 2017-03-02 15:08:38.885220 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a149195a0 for mds.? 172.21.15.22:6805/2203629154 2017-03-02 15:08:38.885226 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:38.885229 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a15207740 mdsbeacon(4121/d up:active seq 8497 v736) v7 from mds.0 172.21.15.22:6805/2203629154 con 0x558a1492a000 2017-03-02 15:08:38.885235 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27348) is_readable = 1 - now=2017-03-02 15:08:38.885235 lease_expire=0.000000 has v0 lc 27348 2017-03-02 15:08:38.885253 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4121/d up:active seq 8497 v736) v7 from mds.0 172.21.15.22:6805/2203629154 2017-03-02 15:08:38.885259 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:38.885261 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:38.885262 7f4f71beb700 20 allow all 2017-03-02 15:08:38.885263 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4121/d up:active seq 8497 v736) v7 from mds.0 172.21.15.22:6805/2203629154 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:38.885272 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4121 2017-03-02 15:08:38.885276 7f4f71beb700 10 mon.0@0(leader).mds e737 mds_beacon mdsbeacon(4121/d up:active seq 8497 v736) v7 ignoring requested state, because mds hasn't seen latest map 2017-03-02 15:08:38.885279 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4121/d up:active seq 8497 v736) v7 noting time 2017-03-02 15:08:38.885282 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15d6a100 0x558a148d7380 mdsbeacon(4121/d up:active seq 8497 v737) v7 2017-03-02 15:08:38.885285 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6805/2203629154 -- mdsbeacon(4121/d up:active seq 8497 v737) v7 -- 0x558a148d7380 con 0 2017-03-02 15:08:38.924496 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.1 172.21.15.17:6804/2576824570 8560 ==== mdsbeacon(4120/g up:active seq 8501 v737) v7 ==== 365+0+0 (4113416968 0 0) 0x558a154b8d00 con 0x558a14908800 2017-03-02 15:08:38.924523 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14917600 for mds.? 172.21.15.17:6804/2576824570 2017-03-02 15:08:38.924529 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:38.924532 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a154b8d00 mdsbeacon(4120/g up:active seq 8501 v737) v7 from mds.1 172.21.15.17:6804/2576824570 con 0x558a14908800 2017-03-02 15:08:38.924538 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27348) is_readable = 1 - now=2017-03-02 15:08:38.924539 lease_expire=0.000000 has v0 lc 27348 2017-03-02 15:08:38.924545 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4120/g up:active seq 8501 v737) v7 from mds.1 172.21.15.17:6804/2576824570 2017-03-02 15:08:38.924551 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:38.924554 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:38.924555 7f4f71beb700 20 allow all 2017-03-02 15:08:38.924556 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4120/g up:active seq 8501 v737) v7 from mds.1 172.21.15.17:6804/2576824570 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:38.924564 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4120 2017-03-02 15:08:38.924569 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4120/g up:active seq 8501 v737) v7 noting time 2017-03-02 15:08:38.924573 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15d6a100 0x558a15207740 mdsbeacon(4120/g up:active seq 8501 v737) v7 2017-03-02 15:08:38.924576 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6804/2576824570 -- mdsbeacon(4120/g up:active seq 8501 v737) v7 -- 0x558a15207740 con 0 2017-03-02 15:08:38.969935 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.4 172.21.15.26:6805/2033388711 8558 ==== mdsbeacon(4117/b up:active seq 8501 v737) v7 ==== 365+0+0 (193256907 0 0) 0x558a15207740 con 0x558a14904000 2017-03-02 15:08:38.969982 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b1600 for mds.? 172.21.15.26:6805/2033388711 2017-03-02 15:08:38.969993 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:38.969997 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a15207740 mdsbeacon(4117/b up:active seq 8501 v737) v7 from mds.4 172.21.15.26:6805/2033388711 con 0x558a14904000 2017-03-02 15:08:38.970007 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27348) is_readable = 1 - now=2017-03-02 15:08:38.970008 lease_expire=0.000000 has v0 lc 27348 2017-03-02 15:08:38.970022 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4117/b up:active seq 8501 v737) v7 from mds.4 172.21.15.26:6805/2033388711 2017-03-02 15:08:38.970033 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:38.970037 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:38.970039 7f4f71beb700 20 allow all 2017-03-02 15:08:38.970041 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4117/b up:active seq 8501 v737) v7 from mds.4 172.21.15.26:6805/2033388711 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:38.970054 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4117 2017-03-02 15:08:38.970061 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4117/b up:active seq 8501 v737) v7 noting time 2017-03-02 15:08:38.970067 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15d6a100 0x558a154b8d00 mdsbeacon(4117/b up:active seq 8501 v737) v7 2017-03-02 15:08:38.970072 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6805/2033388711 -- mdsbeacon(4117/b up:active seq 8501 v737) v7 -- 0x558a154b8d00 con 0 2017-03-02 15:08:38.978870 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.6 172.21.15.26:6804/2954782670 8751 ==== mdsbeacon(4115/a up:active seq 8501 v737) v7 ==== 365+0+0 (1349396899 0 0) 0x558a148d7380 con 0x558a1471f000 2017-03-02 15:08:38.978903 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14868d60 for mds.? 172.21.15.26:6804/2954782670 2017-03-02 15:08:38.978909 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:38.978914 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a148d7380 mdsbeacon(4115/a up:active seq 8501 v737) v7 from mds.6 172.21.15.26:6804/2954782670 con 0x558a1471f000 2017-03-02 15:08:38.978920 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27348) is_readable = 1 - now=2017-03-02 15:08:38.978921 lease_expire=0.000000 has v0 lc 27348 2017-03-02 15:08:38.978929 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4115/a up:active seq 8501 v737) v7 from mds.6 172.21.15.26:6804/2954782670 2017-03-02 15:08:38.978936 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:38.978938 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:38.978939 7f4f71beb700 20 allow all 2017-03-02 15:08:38.978941 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4115/a up:active seq 8501 v737) v7 from mds.6 172.21.15.26:6804/2954782670 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:38.978948 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4115 2017-03-02 15:08:38.978953 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4115/a up:active seq 8501 v737) v7 noting time 2017-03-02 15:08:38.978958 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15d6a100 0x558a15207740 mdsbeacon(4115/a up:active seq 8501 v737) v7 2017-03-02 15:08:38.978961 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6804/2954782670 -- mdsbeacon(4115/a up:active seq 8501 v737) v7 -- 0x558a15207740 con 0 2017-03-02 15:08:39.327126 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13555) propose_pending 2017-03-02 15:08:39.327148 7f4f743f0700 10 mon.0@0(leader).log v13555 encode_full log v 13555 2017-03-02 15:08:39.327209 7f4f743f0700 10 mon.0@0(leader).log v13555 encode_pending v13556 2017-03-02 15:08:39.327215 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27348) queue_pending_finisher 0x558a1412d960 2017-03-02 15:08:39.327218 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27348) trigger_propose active, proposing now 2017-03-02 15:08:39.327224 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27348) propose_pending 27349 10278 bytes 2017-03-02 15:08:39.327226 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27348) begin for 27349 10278 bytes 2017-03-02 15:08:39.356316 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27348) commit_start 27349 2017-03-02 15:08:39.356361 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12974) propose_pending 2017-03-02 15:08:39.356364 7f4f743f0700 10 mon.0@0(leader).pg v12974 encode_pending v 12975 2017-03-02 15:08:39.356435 7f4f743f0700 5 mon.0@0(leader).paxos(paxos writing c 26607..27348) queue_pending_finisher 0x558a1412f870 2017-03-02 15:08:39.356442 7f4f743f0700 10 mon.0@0(leader).paxos(paxos writing c 26607..27348) trigger_propose not active, will propose later 2017-03-02 15:08:39.389816 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27348) commit_finish 27349 2017-03-02 15:08:39.389871 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:39.389891 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12974) refresh 2017-03-02 15:08:39.389899 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) refresh 2017-03-02 15:08:39.389909 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:39.389919 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13556) refresh 2017-03-02 15:08:39.389920 7f4f763f4700 10 mon.0@0(leader).log v13556 update_from_paxos 2017-03-02 15:08:39.389922 7f4f763f4700 10 mon.0@0(leader).log v13556 update_from_paxos version 13556 summary v 13555 2017-03-02 15:08:39.389925 7f4f763f4700 10 mon.0@0(leader).log v13556 update_from_paxos latest full 13555 2017-03-02 15:08:39.389933 7f4f763f4700 7 mon.0@0(leader).log v13556 update_from_paxos applying incremental log 13556 2017-03-02 15:08:38.327264 mon.0 172.21.15.17:6789/0 14120 : cluster [INF] pgmap v12974: 32 pgs: 32 active+clean; 84046 MB data, 159 GB used, 212 GB / 372 GB avail; 394 kB/s rd, 25799 kB/s wr, 1354 op/s 2017-03-02 15:08:39.389946 7f4f763f4700 20 mon.0@0(leader).log v13556 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:39.389961 7f4f763f4700 15 mon.0@0(leader).log v13556 update_from_paxos logging for 1 channels 2017-03-02 15:08:39.389962 7f4f763f4700 15 mon.0@0(leader).log v13556 update_from_paxos channel 'cluster' logging 203 bytes 2017-03-02 15:08:39.390020 7f4f763f4700 10 mon.0@0(leader).log v13556 check_subs 2017-03-02 15:08:39.390052 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:39.390066 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:39.390068 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:39.390078 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:39.390079 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12974) post_refresh 2017-03-02 15:08:39.390081 7f4f763f4700 10 mon.0@0(leader).pg v12974 post_paxos_update 2017-03-02 15:08:39.390083 7f4f763f4700 10 mon.0@0(leader).pg v12974 check_subs 2017-03-02 15:08:39.390084 7f4f763f4700 20 mon.0@0(leader).pg v12974 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:39.390088 7f4f763f4700 20 mon.0@0(leader).pg v12974 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:39.390090 7f4f763f4700 20 mon.0@0(leader).pg v12974 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:39.390091 7f4f763f4700 20 mon.0@0(leader).pg v12974 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:39.390093 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) post_refresh 2017-03-02 15:08:39.390094 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:39.390095 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13556) post_refresh 2017-03-02 15:08:39.390096 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:39.390097 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:39.390098 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:39.390099 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27349) commit_proposal 2017-03-02 15:08:39.390102 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13556) _active - not active 2017-03-02 15:08:39.390103 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27349) finish_round 2017-03-02 15:08:39.390105 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27349) finish_round waiting_for_acting 2017-03-02 15:08:39.390106 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13556) _active 2017-03-02 15:08:39.390107 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13556) _active creating new pending 2017-03-02 15:08:39.390114 7f4f763f4700 10 mon.0@0(leader).log v13556 create_pending v 13557 2017-03-02 15:08:39.390116 7f4f763f4700 7 mon.0@0(leader).log v13556 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:39.390120 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a166aef00 0x558a1592d2a0 log(last 14120) v1 2017-03-02 15:08:39.390123 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14120) v1 -- 0x558a1592d2a0 con 0 2017-03-02 15:08:39.390138 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27349) finish_round waiting_for_readable 2017-03-02 15:08:39.390140 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27349) finish_round waiting_for_writeable 2017-03-02 15:08:39.390141 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27349) finish_round done w/ waiters, state active 2017-03-02 15:08:39.390171 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27349) propose_pending 27350 9752 bytes 2017-03-02 15:08:39.390176 7f4f763f4700 10 mon.0@0(leader).paxos(paxos updating c 26607..27349) begin for 27350 9752 bytes 2017-03-02 15:08:39.390181 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14120) v1 ==== 0+0+0 (0 0 0) 0x558a1592d2a0 con 0x558a14482000 2017-03-02 15:08:39.423225 7f4f763f4700 10 mon.0@0(leader).paxos(paxos updating c 26607..27349) commit_start 27350 2017-03-02 15:08:39.423318 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:39.423335 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:39.423339 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:39.423342 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:39.423343 7f4f71beb700 20 allow all 2017-03-02 15:08:39.456727 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27349) commit_finish 27350 2017-03-02 15:08:39.456782 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:39.456802 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12975) refresh 2017-03-02 15:08:39.456810 7f4f763f4700 10 mon.0@0(leader).pg v12974 update_from_paxos read_incremental 2017-03-02 15:08:39.456846 7f4f763f4700 20 mon.0@0(leader).pg v12974 refreshing pg 1.0 7:496749 active+clean 2017-03-02 15:08:39.456861 7f4f763f4700 20 mon.0@0(leader).pg v12974 refreshing pg 1.1 7:587088 active+clean 2017-03-02 15:08:39.456868 7f4f763f4700 20 mon.0@0(leader).pg v12974 refreshing pg 1.2 7:577621 active+clean 2017-03-02 15:08:39.456874 7f4f763f4700 20 mon.0@0(leader).pg v12974 refreshing pg 1.3 7:500364 active+clean 2017-03-02 15:08:39.456880 7f4f763f4700 20 mon.0@0(leader).pg v12974 refreshing pg 1.4 7:486316 active+clean 2017-03-02 15:08:39.456886 7f4f763f4700 20 mon.0@0(leader).pg v12974 refreshing pg 1.5 7:567937 active+clean 2017-03-02 15:08:39.456892 7f4f763f4700 20 mon.0@0(leader).pg v12974 refreshing pg 1.6 7:503328 active+clean 2017-03-02 15:08:39.456897 7f4f763f4700 20 mon.0@0(leader).pg v12974 refreshing pg 1.7 7:529924 active+clean 2017-03-02 15:08:39.456903 7f4f763f4700 20 mon.0@0(leader).pg v12974 refreshing pg 2.0 7:1057161 active+clean 2017-03-02 15:08:39.456908 7f4f763f4700 20 mon.0@0(leader).pg v12974 refreshing pg 2.1 7:1089895 active+clean 2017-03-02 15:08:39.456914 7f4f763f4700 20 mon.0@0(leader).pg v12974 refreshing pg 2.2 7:1163640 active+clean 2017-03-02 15:08:39.456919 7f4f763f4700 20 mon.0@0(leader).pg v12974 refreshing pg 2.3 7:1095730 active+clean 2017-03-02 15:08:39.456924 7f4f763f4700 20 mon.0@0(leader).pg v12974 refreshing pg 2.4 7:1026594 active+clean 2017-03-02 15:08:39.456929 7f4f763f4700 20 mon.0@0(leader).pg v12974 refreshing pg 2.5 7:1129924 active+clean 2017-03-02 15:08:39.456934 7f4f763f4700 20 mon.0@0(leader).pg v12974 refreshing pg 2.7 7:1135807 active+clean 2017-03-02 15:08:39.456937 7f4f763f4700 20 mon.0@0(leader).pg v12974 refreshing osd.0 2017-03-02 15:08:39.456944 7f4f763f4700 20 mon.0@0(leader).pg v12974 refreshing osd.1 2017-03-02 15:08:39.456948 7f4f763f4700 20 mon.0@0(leader).pg v12974 refreshing osd.2 2017-03-02 15:08:39.456959 7f4f763f4700 10 mon.0@0(leader).pg v12975 read_pgmap_meta 2017-03-02 15:08:39.456978 7f4f763f4700 10 mon.0@0(leader).pg v12975 update_logger 2017-03-02 15:08:39.456995 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) refresh 2017-03-02 15:08:39.457013 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:39.457033 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13556) refresh 2017-03-02 15:08:39.457039 7f4f763f4700 10 mon.0@0(leader).log v13556 update_from_paxos 2017-03-02 15:08:39.457043 7f4f763f4700 10 mon.0@0(leader).log v13556 update_from_paxos version 13556 summary v 13556 2017-03-02 15:08:39.457054 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:39.457067 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:39.457072 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:39.457086 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:39.457091 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12975) post_refresh 2017-03-02 15:08:39.457093 7f4f763f4700 10 mon.0@0(leader).pg v12975 post_paxos_update 2017-03-02 15:08:39.457095 7f4f763f4700 10 mon.0@0(leader).pg v12975 check_subs 2017-03-02 15:08:39.457098 7f4f763f4700 20 mon.0@0(leader).pg v12975 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:39.457105 7f4f763f4700 20 mon.0@0(leader).pg v12975 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:39.457107 7f4f763f4700 20 mon.0@0(leader).pg v12975 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:39.457109 7f4f763f4700 20 mon.0@0(leader).pg v12975 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:39.457112 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) post_refresh 2017-03-02 15:08:39.457113 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:39.457114 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13556) post_refresh 2017-03-02 15:08:39.457115 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:39.457116 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:39.457118 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:39.457119 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27350) commit_proposal 2017-03-02 15:08:39.457122 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12975) _active - not active 2017-03-02 15:08:39.457124 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27350) finish_round 2017-03-02 15:08:39.457126 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27350) finish_round waiting_for_acting 2017-03-02 15:08:39.457128 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12975) _active 2017-03-02 15:08:39.457129 7f4f763f4700 7 mon.0@0(leader).paxosservice(pgmap 12269..12975) _active creating new pending 2017-03-02 15:08:39.457137 7f4f763f4700 10 mon.0@0(leader).pg v12975 create_pending v 12976 2017-03-02 15:08:39.457142 7f4f763f4700 7 mon.0@0(leader).pg v12975 _updated_stats for osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:39.457146 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a16382800 0x558a150de200 pg_stats_ack(5 pgs tid 6800) v1 2017-03-02 15:08:39.457149 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6800/26531 -- pg_stats_ack(5 pgs tid 6800) v1 -- 0x558a150de200 con 0 2017-03-02 15:08:39.457192 7f4f763f4700 7 mon.0@0(leader).pg v12975 _updated_stats for osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:39.457196 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a15a07200 0x558a153f8400 pg_stats_ack(8 pgs tid 6800) v1 2017-03-02 15:08:39.457197 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6800/26536 -- pg_stats_ack(8 pgs tid 6800) v1 -- 0x558a153f8400 con 0 2017-03-02 15:08:39.457213 7f4f763f4700 7 mon.0@0(leader).pg v12975 _updated_stats for osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:39.457216 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a15db0600 0x558a15707a00 pg_stats_ack(2 pgs tid 6800) v1 2017-03-02 15:08:39.457218 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6800/26515 -- pg_stats_ack(2 pgs tid 6800) v1 -- 0x558a15707a00 con 0 2017-03-02 15:08:39.457241 7f4f763f4700 10 mon.0@0(leader).pg v12975 check_osd_map already seen 7 >= 7 2017-03-02 15:08:39.457244 7f4f763f4700 10 mon.0@0(leader).pg v12975 update_logger 2017-03-02 15:08:39.457264 7f4f763f4700 0 log_channel(cluster) log [INF] : pgmap v12975: 32 pgs: 32 active+clean; 81438 MB data, 155 GB used, 217 GB / 372 GB avail; 465 kB/s rd, 31233 kB/s wr, 1860 op/s 2017-03-02 15:08:39.457284 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14121 at 2017-03-02 15:08:39.457267) v1 -- 0x558a167453c0 con 0 2017-03-02 15:08:39.457350 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27350) finish_round waiting_for_readable 2017-03-02 15:08:39.457356 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27350) finish_round waiting_for_writeable 2017-03-02 15:08:39.457359 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27350) finish_round done w/ waiters, state active 2017-03-02 15:08:39.457400 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14121 at 2017-03-02 15:08:39.457267) v1 ==== 0+0+0 (0 0 0) 0x558a167453c0 con 0x558a14482000 2017-03-02 15:08:39.457428 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:39.457438 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:39.457442 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13556) dispatch 0x558a167453c0 log(1 entries from seq 14121 at 2017-03-02 15:08:39.457267) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:39.457449 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27350) is_readable = 1 - now=2017-03-02 15:08:39.457450 lease_expire=0.000000 has v0 lc 27350 2017-03-02 15:08:39.457456 7f4f71beb700 10 mon.0@0(leader).log v13556 preprocess_query log(1 entries from seq 14121 at 2017-03-02 15:08:39.457267) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:39.457462 7f4f71beb700 10 mon.0@0(leader).log v13556 preprocess_log log(1 entries from seq 14121 at 2017-03-02 15:08:39.457267) v1 from mon.0 2017-03-02 15:08:39.457466 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:39.457469 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:39.457470 7f4f71beb700 20 allow all 2017-03-02 15:08:39.457478 7f4f71beb700 10 mon.0@0(leader).log v13556 prepare_update log(1 entries from seq 14121 at 2017-03-02 15:08:39.457267) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:39.457483 7f4f71beb700 10 mon.0@0(leader).log v13556 prepare_log log(1 entries from seq 14121 at 2017-03-02 15:08:39.457267) v1 from mon.0 2017-03-02 15:08:39.457486 7f4f71beb700 10 mon.0@0(leader).log v13556 logging 2017-03-02 15:08:39.457267 mon.0 172.21.15.17:6789/0 14121 : cluster [INF] pgmap v12975: 32 pgs: 32 active+clean; 81438 MB data, 155 GB used, 217 GB / 372 GB avail; 465 kB/s rd, 31233 kB/s wr, 1860 op/s 2017-03-02 15:08:39.457494 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13556) setting proposal_timer 0x558a1412cc90 with delay of 0.999254 2017-03-02 15:08:40.456844 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13556) propose_pending 2017-03-02 15:08:40.456876 7f4f743f0700 10 mon.0@0(leader).log v13556 encode_full log v 13556 2017-03-02 15:08:40.456931 7f4f743f0700 10 mon.0@0(leader).log v13556 encode_pending v13557 2017-03-02 15:08:40.456952 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27350) queue_pending_finisher 0x558a1412dd10 2017-03-02 15:08:40.456962 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27350) trigger_propose active, proposing now 2017-03-02 15:08:40.456970 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27350) propose_pending 27351 10277 bytes 2017-03-02 15:08:40.456974 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27350) begin for 27351 10277 bytes 2017-03-02 15:08:40.490064 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27350) commit_start 27351 2017-03-02 15:08:40.523554 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27350) commit_finish 27351 2017-03-02 15:08:40.523610 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:40.523632 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12975) refresh 2017-03-02 15:08:40.523641 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) refresh 2017-03-02 15:08:40.523651 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:40.523661 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13557) refresh 2017-03-02 15:08:40.523667 7f4f763f4700 10 mon.0@0(leader).log v13557 update_from_paxos 2017-03-02 15:08:40.523669 7f4f763f4700 10 mon.0@0(leader).log v13557 update_from_paxos version 13557 summary v 13556 2017-03-02 15:08:40.523672 7f4f763f4700 10 mon.0@0(leader).log v13557 update_from_paxos latest full 13556 2017-03-02 15:08:40.523680 7f4f763f4700 7 mon.0@0(leader).log v13557 update_from_paxos applying incremental log 13557 2017-03-02 15:08:39.457267 mon.0 172.21.15.17:6789/0 14121 : cluster [INF] pgmap v12975: 32 pgs: 32 active+clean; 81438 MB data, 155 GB used, 217 GB / 372 GB avail; 465 kB/s rd, 31233 kB/s wr, 1860 op/s 2017-03-02 15:08:40.523693 7f4f763f4700 20 mon.0@0(leader).log v13557 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:40.523706 7f4f763f4700 15 mon.0@0(leader).log v13557 update_from_paxos logging for 1 channels 2017-03-02 15:08:40.523707 7f4f763f4700 15 mon.0@0(leader).log v13557 update_from_paxos channel 'cluster' logging 203 bytes 2017-03-02 15:08:40.523766 7f4f763f4700 10 mon.0@0(leader).log v13557 check_subs 2017-03-02 15:08:40.523795 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:40.523806 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:40.523807 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:40.523816 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:40.523817 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12975) post_refresh 2017-03-02 15:08:40.523847 7f4f763f4700 10 mon.0@0(leader).pg v12975 post_paxos_update 2017-03-02 15:08:40.523850 7f4f763f4700 10 mon.0@0(leader).pg v12975 check_subs 2017-03-02 15:08:40.523851 7f4f763f4700 20 mon.0@0(leader).pg v12975 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:40.523856 7f4f763f4700 20 mon.0@0(leader).pg v12975 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:40.523858 7f4f763f4700 20 mon.0@0(leader).pg v12975 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:40.523859 7f4f763f4700 20 mon.0@0(leader).pg v12975 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:40.523861 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) post_refresh 2017-03-02 15:08:40.523862 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:40.523863 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13557) post_refresh 2017-03-02 15:08:40.523864 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:40.523865 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:40.523865 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:40.523867 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27351) commit_proposal 2017-03-02 15:08:40.523870 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13557) _active - not active 2017-03-02 15:08:40.523872 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27351) finish_round 2017-03-02 15:08:40.523873 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27351) finish_round waiting_for_acting 2017-03-02 15:08:40.523874 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13557) _active 2017-03-02 15:08:40.523875 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13557) _active creating new pending 2017-03-02 15:08:40.523884 7f4f763f4700 10 mon.0@0(leader).log v13557 create_pending v 13558 2017-03-02 15:08:40.523888 7f4f763f4700 7 mon.0@0(leader).log v13557 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:40.523892 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a15d6a100 0x558a15c62d20 log(last 14121) v1 2017-03-02 15:08:40.523896 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14121) v1 -- 0x558a15c62d20 con 0 2017-03-02 15:08:40.523917 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27351) finish_round waiting_for_readable 2017-03-02 15:08:40.523920 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27351) finish_round waiting_for_writeable 2017-03-02 15:08:40.523921 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27351) finish_round done w/ waiters, state active 2017-03-02 15:08:40.523959 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14121) v1 ==== 0+0+0 (0 0 0) 0x558a15c62d20 con 0x558a14482000 2017-03-02 15:08:40.523978 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:40.523984 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:40.523987 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:40.523990 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:40.523991 7f4f71beb700 20 allow all 2017-03-02 15:08:41.012325 7f4f743f0700 11 mon.0@0(leader) e2 tick 2017-03-02 15:08:41.012350 7f4f743f0700 10 mon.0@0(leader).pg v12975 v12975: 32 pgs: 32 active+clean; 81438 MB data, 155 GB used, 217 GB / 372 GB avail; 465 kB/s rd, 31233 kB/s wr, 1860 op/s 2017-03-02 15:08:41.012364 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12975) maybe_trim trim_to 12475 would only trim 206 < paxos_service_trim_min 250 2017-03-02 15:08:41.012368 7f4f743f0700 10 mon.0@0(leader).mds e737 e737: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:41.012398 7f4f743f0700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) maybe_trim trim_to 237 would only trim 236 < paxos_service_trim_min 250 2017-03-02 15:08:41.012400 7f4f743f0700 10 mon.0@0(leader).osd e7 e7: 4 osds: 4 up, 4 in 2017-03-02 15:08:41.012404 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27351) is_readable = 1 - now=2017-03-02 15:08:41.012405 lease_expire=0.000000 has v0 lc 27351 2017-03-02 15:08:41.012412 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27351) is_readable = 1 - now=2017-03-02 15:08:41.012412 lease_expire=0.000000 has v0 lc 27351 2017-03-02 15:08:41.012416 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27351) is_readable = 1 - now=2017-03-02 15:08:41.012416 lease_expire=0.000000 has v0 lc 27351 2017-03-02 15:08:41.012421 7f4f743f0700 10 mon.0@0(leader).osd e7 min_last_epoch_clean 6 2017-03-02 15:08:41.012423 7f4f743f0700 10 mon.0@0(leader).log v13557 log 2017-03-02 15:08:41.012424 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13557) maybe_trim trim_to 13057 would only trim 30 < paxos_service_trim_min 250 2017-03-02 15:08:41.012427 7f4f743f0700 10 mon.0@0(leader).auth v10 auth 2017-03-02 15:08:41.012431 7f4f743f0700 10 cephx keyserver: _check_rotating_secrets 2017-03-02 15:08:41.012439 7f4f743f0700 20 mon.0@0(leader) e2 sync_trim_providers 2017-03-02 15:08:42.870494 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.5 172.21.15.11:6804/1637448123 8789 ==== mdsbeacon(4116/e up:active seq 8502 v737) v7 ==== 365+0+0 (3254179646 0 0) 0x558a143e56c0 con 0x558a1471d800 2017-03-02 15:08:42.870523 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b0f20 for mds.? 172.21.15.11:6804/1637448123 2017-03-02 15:08:42.870527 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:42.870530 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a143e56c0 mdsbeacon(4116/e up:active seq 8502 v737) v7 from mds.5 172.21.15.11:6804/1637448123 con 0x558a1471d800 2017-03-02 15:08:42.870534 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27351) is_readable = 1 - now=2017-03-02 15:08:42.870534 lease_expire=0.000000 has v0 lc 27351 2017-03-02 15:08:42.870541 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4116/e up:active seq 8502 v737) v7 from mds.5 172.21.15.11:6804/1637448123 2017-03-02 15:08:42.870546 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:42.870548 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:42.870549 7f4f71beb700 20 allow all 2017-03-02 15:08:42.870550 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4116/e up:active seq 8502 v737) v7 from mds.5 172.21.15.11:6804/1637448123 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:42.870555 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4116 2017-03-02 15:08:42.870560 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4116/e up:active seq 8502 v737) v7 noting time 2017-03-02 15:08:42.870563 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a1621b900 0x558a148d7380 mdsbeacon(4116/e up:active seq 8502 v737) v7 2017-03-02 15:08:42.870564 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6804/1637448123 -- mdsbeacon(4116/e up:active seq 8502 v737) v7 -- 0x558a148d7380 con 0 2017-03-02 15:08:42.884205 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.3 172.21.15.11:6805/1926402648 8565 ==== mdsbeacon(4118/f up:active seq 8502 v737) v7 ==== 406+0+0 (941926974 0 0) 0x558a15207740 con 0x558a14905800 2017-03-02 15:08:42.884232 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b2260 for mds.? 172.21.15.11:6805/1926402648 2017-03-02 15:08:42.884239 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:42.884243 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a15207740 mdsbeacon(4118/f up:active seq 8502 v737) v7 from mds.3 172.21.15.11:6805/1926402648 con 0x558a14905800 2017-03-02 15:08:42.884248 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27351) is_readable = 1 - now=2017-03-02 15:08:42.884249 lease_expire=0.000000 has v0 lc 27351 2017-03-02 15:08:42.884254 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4118/f up:active seq 8502 v737) v7 from mds.3 172.21.15.11:6805/1926402648 2017-03-02 15:08:42.884259 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:42.884260 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:42.884261 7f4f71beb700 20 allow all 2017-03-02 15:08:42.884262 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4118/f up:active seq 8502 v737) v7 from mds.3 172.21.15.11:6805/1926402648 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:42.884266 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4118 2017-03-02 15:08:42.884270 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4118/f up:active seq 8502 v737) v7 noting time 2017-03-02 15:08:42.884273 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a1621b900 0x558a143e56c0 mdsbeacon(4118/f up:active seq 8502 v737) v7 2017-03-02 15:08:42.884274 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6805/1926402648 -- mdsbeacon(4118/f up:active seq 8502 v737) v7 -- 0x558a143e56c0 con 0 2017-03-02 15:08:42.885058 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.2 172.21.15.22:6804/673088617 8564 ==== mdsbeacon(4119/c up:active seq 8502 v737) v7 ==== 365+0+0 (3790590816 0 0) 0x558a148d7380 con 0x558a14907000 2017-03-02 15:08:42.885080 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14916f20 for mds.? 172.21.15.22:6804/673088617 2017-03-02 15:08:42.885087 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:42.885090 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a148d7380 mdsbeacon(4119/c up:active seq 8502 v737) v7 from mds.2 172.21.15.22:6804/673088617 con 0x558a14907000 2017-03-02 15:08:42.885095 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27351) is_readable = 1 - now=2017-03-02 15:08:42.885095 lease_expire=0.000000 has v0 lc 27351 2017-03-02 15:08:42.885100 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4119/c up:active seq 8502 v737) v7 from mds.2 172.21.15.22:6804/673088617 2017-03-02 15:08:42.885106 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:42.885108 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:42.885108 7f4f71beb700 20 allow all 2017-03-02 15:08:42.885110 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4119/c up:active seq 8502 v737) v7 from mds.2 172.21.15.22:6804/673088617 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:42.885116 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4119 2017-03-02 15:08:42.885119 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4119/c up:active seq 8502 v737) v7 noting time 2017-03-02 15:08:42.885121 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a1621b900 0x558a15207740 mdsbeacon(4119/c up:active seq 8502 v737) v7 2017-03-02 15:08:42.885123 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6804/673088617 -- mdsbeacon(4119/c up:active seq 8502 v737) v7 -- 0x558a15207740 con 0 2017-03-02 15:08:42.885223 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.0 172.21.15.22:6805/2203629154 8614 ==== mdsbeacon(4121/d up:active seq 8498 v736) v7 ==== 452+0+0 (4191877876 0 0) 0x558a143e56c0 con 0x558a1492a000 2017-03-02 15:08:42.885241 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a149195a0 for mds.? 172.21.15.22:6805/2203629154 2017-03-02 15:08:42.885244 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:42.885246 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a143e56c0 mdsbeacon(4121/d up:active seq 8498 v736) v7 from mds.0 172.21.15.22:6805/2203629154 con 0x558a1492a000 2017-03-02 15:08:42.885248 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27351) is_readable = 1 - now=2017-03-02 15:08:42.885249 lease_expire=0.000000 has v0 lc 27351 2017-03-02 15:08:42.885253 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4121/d up:active seq 8498 v736) v7 from mds.0 172.21.15.22:6805/2203629154 2017-03-02 15:08:42.885256 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:42.885257 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:42.885257 7f4f71beb700 20 allow all 2017-03-02 15:08:42.885258 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4121/d up:active seq 8498 v736) v7 from mds.0 172.21.15.22:6805/2203629154 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:42.885263 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4121 2017-03-02 15:08:42.885266 7f4f71beb700 10 mon.0@0(leader).mds e737 mds_beacon mdsbeacon(4121/d up:active seq 8498 v736) v7 ignoring requested state, because mds hasn't seen latest map 2017-03-02 15:08:42.885268 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4121/d up:active seq 8498 v736) v7 noting time 2017-03-02 15:08:42.885271 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a1621b900 0x558a148d7380 mdsbeacon(4121/d up:active seq 8498 v737) v7 2017-03-02 15:08:42.885273 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6805/2203629154 -- mdsbeacon(4121/d up:active seq 8498 v737) v7 -- 0x558a148d7380 con 0 2017-03-02 15:08:42.924466 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.1 172.21.15.17:6804/2576824570 8561 ==== mdsbeacon(4120/g up:active seq 8502 v737) v7 ==== 365+0+0 (702586491 0 0) 0x558a154b8d00 con 0x558a14908800 2017-03-02 15:08:42.924494 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14917600 for mds.? 172.21.15.17:6804/2576824570 2017-03-02 15:08:42.924501 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:42.924504 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a154b8d00 mdsbeacon(4120/g up:active seq 8502 v737) v7 from mds.1 172.21.15.17:6804/2576824570 con 0x558a14908800 2017-03-02 15:08:42.924511 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27351) is_readable = 1 - now=2017-03-02 15:08:42.924512 lease_expire=0.000000 has v0 lc 27351 2017-03-02 15:08:42.924520 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4120/g up:active seq 8502 v737) v7 from mds.1 172.21.15.17:6804/2576824570 2017-03-02 15:08:42.924528 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:42.924530 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:42.924531 7f4f71beb700 20 allow all 2017-03-02 15:08:42.924532 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4120/g up:active seq 8502 v737) v7 from mds.1 172.21.15.17:6804/2576824570 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:42.924538 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4120 2017-03-02 15:08:42.924542 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4120/g up:active seq 8502 v737) v7 noting time 2017-03-02 15:08:42.924545 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a1621b900 0x558a143e56c0 mdsbeacon(4120/g up:active seq 8502 v737) v7 2017-03-02 15:08:42.924547 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6804/2576824570 -- mdsbeacon(4120/g up:active seq 8502 v737) v7 -- 0x558a143e56c0 con 0 2017-03-02 15:08:42.970018 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.4 172.21.15.26:6805/2033388711 8559 ==== mdsbeacon(4117/b up:active seq 8502 v737) v7 ==== 365+0+0 (3611918520 0 0) 0x558a143e56c0 con 0x558a14904000 2017-03-02 15:08:42.970045 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b1600 for mds.? 172.21.15.26:6805/2033388711 2017-03-02 15:08:42.970049 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:42.970052 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a143e56c0 mdsbeacon(4117/b up:active seq 8502 v737) v7 from mds.4 172.21.15.26:6805/2033388711 con 0x558a14904000 2017-03-02 15:08:42.970056 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27351) is_readable = 1 - now=2017-03-02 15:08:42.970057 lease_expire=0.000000 has v0 lc 27351 2017-03-02 15:08:42.970063 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4117/b up:active seq 8502 v737) v7 from mds.4 172.21.15.26:6805/2033388711 2017-03-02 15:08:42.970067 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:42.970069 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:42.970070 7f4f71beb700 20 allow all 2017-03-02 15:08:42.970071 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4117/b up:active seq 8502 v737) v7 from mds.4 172.21.15.26:6805/2033388711 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:42.970076 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4117 2017-03-02 15:08:42.970080 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4117/b up:active seq 8502 v737) v7 noting time 2017-03-02 15:08:42.970082 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a1621b900 0x558a154b8d00 mdsbeacon(4117/b up:active seq 8502 v737) v7 2017-03-02 15:08:42.970084 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6805/2033388711 -- mdsbeacon(4117/b up:active seq 8502 v737) v7 -- 0x558a154b8d00 con 0 2017-03-02 15:08:42.978901 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.6 172.21.15.26:6804/2954782670 8752 ==== mdsbeacon(4115/a up:active seq 8502 v737) v7 ==== 365+0+0 (2359522512 0 0) 0x558a148d7380 con 0x558a1471f000 2017-03-02 15:08:42.978922 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14868d60 for mds.? 172.21.15.26:6804/2954782670 2017-03-02 15:08:42.978927 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:42.978929 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a148d7380 mdsbeacon(4115/a up:active seq 8502 v737) v7 from mds.6 172.21.15.26:6804/2954782670 con 0x558a1471f000 2017-03-02 15:08:42.978933 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27351) is_readable = 1 - now=2017-03-02 15:08:42.978934 lease_expire=0.000000 has v0 lc 27351 2017-03-02 15:08:42.978939 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4115/a up:active seq 8502 v737) v7 from mds.6 172.21.15.26:6804/2954782670 2017-03-02 15:08:42.978943 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:42.978944 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:42.978945 7f4f71beb700 20 allow all 2017-03-02 15:08:42.978945 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4115/a up:active seq 8502 v737) v7 from mds.6 172.21.15.26:6804/2954782670 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:42.978950 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4115 2017-03-02 15:08:42.978954 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4115/a up:active seq 8502 v737) v7 noting time 2017-03-02 15:08:42.978956 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a1621b900 0x558a143e56c0 mdsbeacon(4115/a up:active seq 8502 v737) v7 2017-03-02 15:08:42.978958 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6804/2954782670 -- mdsbeacon(4115/a up:active seq 8502 v737) v7 -- 0x558a143e56c0 con 0 2017-03-02 15:08:43.264397 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.3 172.21.15.17:6800/27100 6837 ==== pg_stats(1 pgs tid 6801 v 0) v1 ==== 706+0+0 (649959040 0 0) 0x558a15d28300 con 0x558a145d7000 2017-03-02 15:08:43.264443 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415f440 for osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:43.264453 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:43.264456 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12975) dispatch 0x558a15d28300 pg_stats(1 pgs tid 6801 v 0) v1 from osd.3 172.21.15.17:6800/27100 con 0x558a145d7000 2017-03-02 15:08:43.264464 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27351) is_readable = 1 - now=2017-03-02 15:08:43.264464 lease_expire=0.000000 has v0 lc 27351 2017-03-02 15:08:43.264475 7f4f71beb700 10 mon.0@0(leader).pg v12975 preprocess_query pg_stats(1 pgs tid 6801 v 0) v1 from osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:43.264481 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:43.264483 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:43.264484 7f4f71beb700 20 allow all 2017-03-02 15:08:43.264486 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27351) is_readable = 1 - now=2017-03-02 15:08:43.264486 lease_expire=0.000000 has v0 lc 27351 2017-03-02 15:08:43.264489 7f4f71beb700 10 mon.0@0(leader).pg v12975 prepare_update pg_stats(1 pgs tid 6801 v 0) v1 from osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:43.264493 7f4f71beb700 10 mon.0@0(leader).pg v12975 prepare_pg_stats pg_stats(1 pgs tid 6801 v 0) v1 from osd.3 2017-03-02 15:08:43.264499 7f4f71beb700 10 mon.0@0(leader).pg v12975 got osd.3 osd_stat(38198 MB used, 57142 MB avail, 95341 MB total, peers [0,1,2] op hist [0,1]) (was osd_stat(39526 MB used, 55815 MB avail, 95341 MB total, peers [0,1,2] op hist [0,1])) 2017-03-02 15:08:43.264506 7f4f71beb700 15 mon.0@0(leader).pg v12975 got 2.6 reported at 7:965046 state active+clean -> active+clean 2017-03-02 15:08:43.264515 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12975) setting proposal_timer 0x558a1412f990 with delay of 0.05 2017-03-02 15:08:43.314613 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12975) propose_pending 2017-03-02 15:08:43.314635 7f4f743f0700 10 mon.0@0(leader).pg v12975 encode_pending v 12976 2017-03-02 15:08:43.314683 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27351) queue_pending_finisher 0x558a1412def0 2017-03-02 15:08:43.314688 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27351) trigger_propose active, proposing now 2017-03-02 15:08:43.314701 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27351) propose_pending 27352 1098 bytes 2017-03-02 15:08:43.314704 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27351) begin for 27352 1098 bytes 2017-03-02 15:08:43.326719 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27351) commit_start 27352 2017-03-02 15:08:43.335024 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27351) commit_finish 27352 2017-03-02 15:08:43.335056 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:43.335074 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12976) refresh 2017-03-02 15:08:43.335076 7f4f763f4700 10 mon.0@0(leader).pg v12975 update_from_paxos read_incremental 2017-03-02 15:08:43.335097 7f4f763f4700 20 mon.0@0(leader).pg v12975 refreshing pg 2.6 7:965046 active+clean 2017-03-02 15:08:43.335119 7f4f763f4700 20 mon.0@0(leader).pg v12975 refreshing osd.3 2017-03-02 15:08:43.335130 7f4f763f4700 10 mon.0@0(leader).pg v12976 read_pgmap_meta 2017-03-02 15:08:43.335145 7f4f763f4700 10 mon.0@0(leader).pg v12976 update_logger 2017-03-02 15:08:43.335155 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) refresh 2017-03-02 15:08:43.335166 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:43.335186 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13557) refresh 2017-03-02 15:08:43.335192 7f4f763f4700 10 mon.0@0(leader).log v13557 update_from_paxos 2017-03-02 15:08:43.335194 7f4f763f4700 10 mon.0@0(leader).log v13557 update_from_paxos version 13557 summary v 13557 2017-03-02 15:08:43.335202 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:43.335213 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:43.335215 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:43.335224 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:43.335226 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12976) post_refresh 2017-03-02 15:08:43.335227 7f4f763f4700 10 mon.0@0(leader).pg v12976 post_paxos_update 2017-03-02 15:08:43.335229 7f4f763f4700 10 mon.0@0(leader).pg v12976 check_subs 2017-03-02 15:08:43.335231 7f4f763f4700 20 mon.0@0(leader).pg v12976 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:43.335237 7f4f763f4700 20 mon.0@0(leader).pg v12976 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:43.335240 7f4f763f4700 20 mon.0@0(leader).pg v12976 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:43.335242 7f4f763f4700 20 mon.0@0(leader).pg v12976 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:43.335243 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) post_refresh 2017-03-02 15:08:43.335244 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:43.335245 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13557) post_refresh 2017-03-02 15:08:43.335246 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:43.335247 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:43.335248 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:43.335250 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27352) commit_proposal 2017-03-02 15:08:43.335252 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12976) _active - not active 2017-03-02 15:08:43.335254 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27352) finish_round 2017-03-02 15:08:43.335255 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27352) finish_round waiting_for_acting 2017-03-02 15:08:43.335257 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12976) _active 2017-03-02 15:08:43.335258 7f4f763f4700 7 mon.0@0(leader).paxosservice(pgmap 12269..12976) _active creating new pending 2017-03-02 15:08:43.335261 7f4f763f4700 10 mon.0@0(leader).pg v12976 create_pending v 12977 2017-03-02 15:08:43.335265 7f4f763f4700 7 mon.0@0(leader).pg v12976 _updated_stats for osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:43.335269 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a1621b900 0x558a1547d000 pg_stats_ack(1 pgs tid 6801) v1 2017-03-02 15:08:43.335272 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6800/27100 -- pg_stats_ack(1 pgs tid 6801) v1 -- 0x558a1547d000 con 0 2017-03-02 15:08:43.335303 7f4f763f4700 10 mon.0@0(leader).pg v12976 check_osd_map already seen 7 >= 7 2017-03-02 15:08:43.335308 7f4f763f4700 10 mon.0@0(leader).pg v12976 update_logger 2017-03-02 15:08:43.335325 7f4f763f4700 0 log_channel(cluster) log [INF] : pgmap v12976: 32 pgs: 32 active+clean; 80992 MB data, 153 GB used, 218 GB / 372 GB avail; 353 kB/s rd, 17932 kB/s wr, 1128 op/s 2017-03-02 15:08:43.335343 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14122 at 2017-03-02 15:08:43.335327) v1 -- 0x558a16745600 con 0 2017-03-02 15:08:43.335362 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27352) finish_round waiting_for_readable 2017-03-02 15:08:43.335366 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27352) finish_round waiting_for_writeable 2017-03-02 15:08:43.335368 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27352) finish_round done w/ waiters, state active 2017-03-02 15:08:43.335421 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14122 at 2017-03-02 15:08:43.335327) v1 ==== 0+0+0 (0 0 0) 0x558a16745600 con 0x558a14482000 2017-03-02 15:08:43.335444 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:43.335449 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:43.335451 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13557) dispatch 0x558a16745600 log(1 entries from seq 14122 at 2017-03-02 15:08:43.335327) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:43.335457 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27352) is_readable = 1 - now=2017-03-02 15:08:43.335457 lease_expire=0.000000 has v0 lc 27352 2017-03-02 15:08:43.335463 7f4f71beb700 10 mon.0@0(leader).log v13557 preprocess_query log(1 entries from seq 14122 at 2017-03-02 15:08:43.335327) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:43.335468 7f4f71beb700 10 mon.0@0(leader).log v13557 preprocess_log log(1 entries from seq 14122 at 2017-03-02 15:08:43.335327) v1 from mon.0 2017-03-02 15:08:43.335472 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:43.335475 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:43.335476 7f4f71beb700 20 allow all 2017-03-02 15:08:43.335484 7f4f71beb700 10 mon.0@0(leader).log v13557 prepare_update log(1 entries from seq 14122 at 2017-03-02 15:08:43.335327) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:43.335488 7f4f71beb700 10 mon.0@0(leader).log v13557 prepare_log log(1 entries from seq 14122 at 2017-03-02 15:08:43.335327) v1 from mon.0 2017-03-02 15:08:43.335491 7f4f71beb700 10 mon.0@0(leader).log v13557 logging 2017-03-02 15:08:43.335327 mon.0 172.21.15.17:6789/0 14122 : cluster [INF] pgmap v12976: 32 pgs: 32 active+clean; 80992 MB data, 153 GB used, 218 GB / 372 GB avail; 353 kB/s rd, 17932 kB/s wr, 1128 op/s 2017-03-02 15:08:43.335498 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13557) setting proposal_timer 0x558a1412c3d0 with delay of 0.999539 2017-03-02 15:08:43.466294 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.1 172.21.15.22:6800/26531 6837 ==== pg_stats(5 pgs tid 6801 v 0) v1 ==== 2966+0+0 (2274094456 0 0) 0x558a15da2f80 con 0x558a14643800 2017-03-02 15:08:43.466331 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1463b1e0 for osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:43.466336 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:43.466339 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12976) dispatch 0x558a15da2f80 pg_stats(5 pgs tid 6801 v 0) v1 from osd.1 172.21.15.22:6800/26531 con 0x558a14643800 2017-03-02 15:08:43.466343 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27352) is_readable = 1 - now=2017-03-02 15:08:43.466343 lease_expire=0.000000 has v0 lc 27352 2017-03-02 15:08:43.466350 7f4f71beb700 10 mon.0@0(leader).pg v12976 preprocess_query pg_stats(5 pgs tid 6801 v 0) v1 from osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:43.466355 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:43.466357 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:43.466357 7f4f71beb700 20 allow all 2017-03-02 15:08:43.466358 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27352) is_readable = 1 - now=2017-03-02 15:08:43.466359 lease_expire=0.000000 has v0 lc 27352 2017-03-02 15:08:43.466361 7f4f71beb700 10 mon.0@0(leader).pg v12976 prepare_update pg_stats(5 pgs tid 6801 v 0) v1 from osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:43.466364 7f4f71beb700 10 mon.0@0(leader).pg v12976 prepare_pg_stats pg_stats(5 pgs tid 6801 v 0) v1 from osd.1 2017-03-02 15:08:43.466371 7f4f71beb700 10 mon.0@0(leader).pg v12976 got osd.1 osd_stat(49225 MB used, 46115 MB avail, 95341 MB total, peers [0,2,3] op hist [2]) (was osd_stat(49225 MB used, 46115 MB avail, 95341 MB total, peers [0,2,3] op hist [2])) 2017-03-02 15:08:43.466377 7f4f71beb700 15 mon.0@0(leader).pg v12976 got 1.1 reported at 7:587151 state active+clean -> active+clean 2017-03-02 15:08:43.466383 7f4f71beb700 15 mon.0@0(leader).pg v12976 got 1.5 reported at 7:567986 state active+clean -> active+clean 2017-03-02 15:08:43.466386 7f4f71beb700 15 mon.0@0(leader).pg v12976 got 1.6 reported at 7:503387 state active+clean -> active+clean 2017-03-02 15:08:43.466389 7f4f71beb700 15 mon.0@0(leader).pg v12976 got 2.4 reported at 7:1026889 state active+clean -> active+clean 2017-03-02 15:08:43.466391 7f4f71beb700 15 mon.0@0(leader).pg v12976 got 2.5 reported at 7:1130212 state active+clean -> active+clean 2017-03-02 15:08:43.466395 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12976) setting proposal_timer 0x558a1412eb90 with delay of 0.868643 2017-03-02 15:08:43.559701 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.2 172.21.15.11:6800/26536 6837 ==== pg_stats(8 pgs tid 6801 v 0) v1 ==== 4664+0+0 (1784509473 0 0) 0x558a155d4580 con 0x558a145d5800 2017-03-02 15:08:43.559730 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a145c3700 for osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:43.559734 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:43.559737 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12976) dispatch 0x558a155d4580 pg_stats(8 pgs tid 6801 v 0) v1 from osd.2 172.21.15.11:6800/26536 con 0x558a145d5800 2017-03-02 15:08:43.559741 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27352) is_readable = 1 - now=2017-03-02 15:08:43.559741 lease_expire=0.000000 has v0 lc 27352 2017-03-02 15:08:43.559750 7f4f71beb700 10 mon.0@0(leader).pg v12976 preprocess_query pg_stats(8 pgs tid 6801 v 0) v1 from osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:43.559754 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:43.559757 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:43.559757 7f4f71beb700 20 allow all 2017-03-02 15:08:43.559759 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27352) is_readable = 1 - now=2017-03-02 15:08:43.559759 lease_expire=0.000000 has v0 lc 27352 2017-03-02 15:08:43.559762 7f4f71beb700 10 mon.0@0(leader).pg v12976 prepare_update pg_stats(8 pgs tid 6801 v 0) v1 from osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:43.559764 7f4f71beb700 10 mon.0@0(leader).pg v12976 prepare_pg_stats pg_stats(8 pgs tid 6801 v 0) v1 from osd.2 2017-03-02 15:08:43.559770 7f4f71beb700 10 mon.0@0(leader).pg v12976 got osd.2 osd_stat(57791 MB used, 37549 MB avail, 95341 MB total, peers [0,1,3] op hist [1]) (was osd_stat(59585 MB used, 35756 MB avail, 95341 MB total, peers [0,1,3] op hist [0,0,1])) 2017-03-02 15:08:43.559777 7f4f71beb700 15 mon.0@0(leader).pg v12976 got 1.0 reported at 7:496815 state active+clean -> active+clean 2017-03-02 15:08:43.559784 7f4f71beb700 15 mon.0@0(leader).pg v12976 got 1.2 reported at 7:577691 state active+clean -> active+clean 2017-03-02 15:08:43.559787 7f4f71beb700 15 mon.0@0(leader).pg v12976 got 1.7 reported at 7:529991 state active+clean -> active+clean 2017-03-02 15:08:43.559790 7f4f71beb700 15 mon.0@0(leader).pg v12976 got 2.0 reported at 7:1057458 state active+clean -> active+clean 2017-03-02 15:08:43.559792 7f4f71beb700 15 mon.0@0(leader).pg v12976 got 2.1 reported at 7:1090182 state active+clean -> active+clean 2017-03-02 15:08:43.559795 7f4f71beb700 15 mon.0@0(leader).pg v12976 got 2.2 reported at 7:1163918 state active+clean -> active+clean 2017-03-02 15:08:43.559798 7f4f71beb700 15 mon.0@0(leader).pg v12976 got 2.3 reported at 7:1096003 state active+clean -> active+clean 2017-03-02 15:08:43.559800 7f4f71beb700 15 mon.0@0(leader).pg v12976 got 2.7 reported at 7:1136110 state active+clean -> active+clean 2017-03-02 15:08:43.559804 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12976) proposal_timer already set 2017-03-02 15:08:43.715523 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.0 172.21.15.26:6800/26515 6837 ==== pg_stats(2 pgs tid 6801 v 0) v1 ==== 1264+0+0 (3991217721 0 0) 0x558a15ca2500 con 0x558a145d8800 2017-03-02 15:08:43.715552 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1463af20 for osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:43.715557 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:43.715559 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12976) dispatch 0x558a15ca2500 pg_stats(2 pgs tid 6801 v 0) v1 from osd.0 172.21.15.26:6800/26515 con 0x558a145d8800 2017-03-02 15:08:43.715564 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27352) is_readable = 1 - now=2017-03-02 15:08:43.715564 lease_expire=0.000000 has v0 lc 27352 2017-03-02 15:08:43.715572 7f4f71beb700 10 mon.0@0(leader).pg v12976 preprocess_query pg_stats(2 pgs tid 6801 v 0) v1 from osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:43.715577 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:43.715579 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:43.715580 7f4f71beb700 20 allow all 2017-03-02 15:08:43.715581 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27352) is_readable = 1 - now=2017-03-02 15:08:43.715581 lease_expire=0.000000 has v0 lc 27352 2017-03-02 15:08:43.715584 7f4f71beb700 10 mon.0@0(leader).pg v12976 prepare_update pg_stats(2 pgs tid 6801 v 0) v1 from osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:43.715587 7f4f71beb700 10 mon.0@0(leader).pg v12976 prepare_pg_stats pg_stats(2 pgs tid 6801 v 0) v1 from osd.0 2017-03-02 15:08:43.715592 7f4f71beb700 10 mon.0@0(leader).pg v12976 got osd.0 osd_stat(10127 MB used, 85214 MB avail, 95341 MB total, peers [1,2,3] op hist []) (was osd_stat(10457 MB used, 84884 MB avail, 95341 MB total, peers [1,2,3] op hist [])) 2017-03-02 15:08:43.715600 7f4f71beb700 15 mon.0@0(leader).pg v12976 got 1.3 reported at 7:500432 state active+clean -> active+clean 2017-03-02 15:08:43.715607 7f4f71beb700 15 mon.0@0(leader).pg v12976 got 1.4 reported at 7:486372 state active+clean -> active+clean 2017-03-02 15:08:43.715611 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12976) proposal_timer already set 2017-03-02 15:08:44.335117 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12976) propose_pending 2017-03-02 15:08:44.335138 7f4f743f0700 10 mon.0@0(leader).pg v12976 encode_pending v 12977 2017-03-02 15:08:44.335222 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27352) queue_pending_finisher 0x558a1412f990 2017-03-02 15:08:44.335231 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27352) trigger_propose active, proposing now 2017-03-02 15:08:44.335248 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27352) propose_pending 27353 9744 bytes 2017-03-02 15:08:44.335250 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27352) begin for 27353 9744 bytes 2017-03-02 15:08:44.364813 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27352) commit_start 27353 2017-03-02 15:08:44.364918 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13557) propose_pending 2017-03-02 15:08:44.364929 7f4f743f0700 10 mon.0@0(leader).log v13557 encode_full log v 13557 2017-03-02 15:08:44.364980 7f4f743f0700 10 mon.0@0(leader).log v13557 encode_pending v13558 2017-03-02 15:08:44.364990 7f4f743f0700 5 mon.0@0(leader).paxos(paxos writing c 26607..27352) queue_pending_finisher 0x558a1412eb90 2017-03-02 15:08:44.364992 7f4f743f0700 10 mon.0@0(leader).paxos(paxos writing c 26607..27352) trigger_propose not active, will propose later 2017-03-02 15:08:44.398399 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27352) commit_finish 27353 2017-03-02 15:08:44.398476 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:44.398508 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12977) refresh 2017-03-02 15:08:44.398512 7f4f763f4700 10 mon.0@0(leader).pg v12976 update_from_paxos read_incremental 2017-03-02 15:08:44.398548 7f4f763f4700 20 mon.0@0(leader).pg v12976 refreshing pg 1.0 7:496815 active+clean 2017-03-02 15:08:44.398565 7f4f763f4700 20 mon.0@0(leader).pg v12976 refreshing pg 1.1 7:587151 active+clean 2017-03-02 15:08:44.398577 7f4f763f4700 20 mon.0@0(leader).pg v12976 refreshing pg 1.2 7:577691 active+clean 2017-03-02 15:08:44.398588 7f4f763f4700 20 mon.0@0(leader).pg v12976 refreshing pg 1.3 7:500432 active+clean 2017-03-02 15:08:44.398598 7f4f763f4700 20 mon.0@0(leader).pg v12976 refreshing pg 1.4 7:486372 active+clean 2017-03-02 15:08:44.398608 7f4f763f4700 20 mon.0@0(leader).pg v12976 refreshing pg 1.5 7:567986 active+clean 2017-03-02 15:08:44.398619 7f4f763f4700 20 mon.0@0(leader).pg v12976 refreshing pg 1.6 7:503387 active+clean 2017-03-02 15:08:44.398630 7f4f763f4700 20 mon.0@0(leader).pg v12976 refreshing pg 1.7 7:529991 active+clean 2017-03-02 15:08:44.398640 7f4f763f4700 20 mon.0@0(leader).pg v12976 refreshing pg 2.0 7:1057458 active+clean 2017-03-02 15:08:44.398651 7f4f763f4700 20 mon.0@0(leader).pg v12976 refreshing pg 2.1 7:1090182 active+clean 2017-03-02 15:08:44.398662 7f4f763f4700 20 mon.0@0(leader).pg v12976 refreshing pg 2.2 7:1163918 active+clean 2017-03-02 15:08:44.398672 7f4f763f4700 20 mon.0@0(leader).pg v12976 refreshing pg 2.3 7:1096003 active+clean 2017-03-02 15:08:44.398682 7f4f763f4700 20 mon.0@0(leader).pg v12976 refreshing pg 2.4 7:1026889 active+clean 2017-03-02 15:08:44.398692 7f4f763f4700 20 mon.0@0(leader).pg v12976 refreshing pg 2.5 7:1130212 active+clean 2017-03-02 15:08:44.398702 7f4f763f4700 20 mon.0@0(leader).pg v12976 refreshing pg 2.7 7:1136110 active+clean 2017-03-02 15:08:44.398706 7f4f763f4700 20 mon.0@0(leader).pg v12976 refreshing osd.0 2017-03-02 15:08:44.398718 7f4f763f4700 20 mon.0@0(leader).pg v12976 refreshing osd.1 2017-03-02 15:08:44.398724 7f4f763f4700 20 mon.0@0(leader).pg v12976 refreshing osd.2 2017-03-02 15:08:44.398740 7f4f763f4700 10 mon.0@0(leader).pg v12977 read_pgmap_meta 2017-03-02 15:08:44.398771 7f4f763f4700 10 mon.0@0(leader).pg v12977 update_logger 2017-03-02 15:08:44.398792 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) refresh 2017-03-02 15:08:44.398814 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:44.398877 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13557) refresh 2017-03-02 15:08:44.398881 7f4f763f4700 10 mon.0@0(leader).log v13557 update_from_paxos 2017-03-02 15:08:44.398885 7f4f763f4700 10 mon.0@0(leader).log v13557 update_from_paxos version 13557 summary v 13557 2017-03-02 15:08:44.398903 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:44.398924 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:44.398927 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:44.398947 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:44.398950 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12977) post_refresh 2017-03-02 15:08:44.398952 7f4f763f4700 10 mon.0@0(leader).pg v12977 post_paxos_update 2017-03-02 15:08:44.398955 7f4f763f4700 10 mon.0@0(leader).pg v12977 check_subs 2017-03-02 15:08:44.398959 7f4f763f4700 20 mon.0@0(leader).pg v12977 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:44.398969 7f4f763f4700 20 mon.0@0(leader).pg v12977 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:44.398975 7f4f763f4700 20 mon.0@0(leader).pg v12977 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:44.398979 7f4f763f4700 20 mon.0@0(leader).pg v12977 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:44.398983 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) post_refresh 2017-03-02 15:08:44.398986 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:44.398988 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13557) post_refresh 2017-03-02 15:08:44.398990 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:44.398992 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:44.398994 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:44.398997 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27353) commit_proposal 2017-03-02 15:08:44.399002 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12977) _active - not active 2017-03-02 15:08:44.399005 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27353) finish_round 2017-03-02 15:08:44.399008 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27353) finish_round waiting_for_acting 2017-03-02 15:08:44.399011 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12977) _active 2017-03-02 15:08:44.399014 7f4f763f4700 7 mon.0@0(leader).paxosservice(pgmap 12269..12977) _active creating new pending 2017-03-02 15:08:44.399025 7f4f763f4700 10 mon.0@0(leader).pg v12977 create_pending v 12978 2017-03-02 15:08:44.399033 7f4f763f4700 7 mon.0@0(leader).pg v12977 _updated_stats for osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:44.399039 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a14f45f00 0x558a14f24a00 pg_stats_ack(5 pgs tid 6801) v1 2017-03-02 15:08:44.399045 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6800/26531 -- pg_stats_ack(5 pgs tid 6801) v1 -- 0x558a14f24a00 con 0 2017-03-02 15:08:44.399100 7f4f763f4700 7 mon.0@0(leader).pg v12977 _updated_stats for osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:44.399107 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a14521600 0x558a15d2ae00 pg_stats_ack(8 pgs tid 6801) v1 2017-03-02 15:08:44.399110 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6800/26536 -- pg_stats_ack(8 pgs tid 6801) v1 -- 0x558a15d2ae00 con 0 2017-03-02 15:08:44.399138 7f4f763f4700 7 mon.0@0(leader).pg v12977 _updated_stats for osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:44.399143 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a15a19800 0x558a15657600 pg_stats_ack(2 pgs tid 6801) v1 2017-03-02 15:08:44.399146 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6800/26515 -- pg_stats_ack(2 pgs tid 6801) v1 -- 0x558a15657600 con 0 2017-03-02 15:08:44.399181 7f4f763f4700 10 mon.0@0(leader).pg v12977 check_osd_map already seen 7 >= 7 2017-03-02 15:08:44.399185 7f4f763f4700 10 mon.0@0(leader).pg v12977 update_logger 2017-03-02 15:08:44.399208 7f4f763f4700 0 log_channel(cluster) log [INF] : pgmap v12977: 32 pgs: 32 active+clean; 78308 MB data, 151 GB used, 220 GB / 372 GB avail; 133 kB/s rd, 10747 kB/s wr, 941 op/s 2017-03-02 15:08:44.399230 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14123 at 2017-03-02 15:08:44.399211) v1 -- 0x558a16743f80 con 0 2017-03-02 15:08:44.399253 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27353) finish_round waiting_for_readable 2017-03-02 15:08:44.399258 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27353) finish_round waiting_for_writeable 2017-03-02 15:08:44.399261 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27353) finish_round done w/ waiters, state active 2017-03-02 15:08:44.399274 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27353) propose_pending 27354 10276 bytes 2017-03-02 15:08:44.399280 7f4f763f4700 10 mon.0@0(leader).paxos(paxos updating c 26607..27353) begin for 27354 10276 bytes 2017-03-02 15:08:44.399292 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14123 at 2017-03-02 15:08:44.399211) v1 ==== 0+0+0 (0 0 0) 0x558a16743f80 con 0x558a14482000 2017-03-02 15:08:44.431773 7f4f763f4700 10 mon.0@0(leader).paxos(paxos updating c 26607..27353) commit_start 27354 2017-03-02 15:08:44.431875 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:44.431899 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:44.431905 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13557) dispatch 0x558a16743f80 log(1 entries from seq 14123 at 2017-03-02 15:08:44.399211) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:44.431918 7f4f71beb700 5 mon.0@0(leader).paxos(paxos writing c 26607..27353) is_readable = 1 - now=2017-03-02 15:08:44.431919 lease_expire=0.000000 has v0 lc 27353 2017-03-02 15:08:44.431927 7f4f71beb700 10 mon.0@0(leader).log v13557 preprocess_query log(1 entries from seq 14123 at 2017-03-02 15:08:44.399211) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:44.431936 7f4f71beb700 10 mon.0@0(leader).log v13557 preprocess_log log(1 entries from seq 14123 at 2017-03-02 15:08:44.399211) v1 from mon.0 2017-03-02 15:08:44.431942 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:44.431946 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:44.431948 7f4f71beb700 20 allow all 2017-03-02 15:08:44.431956 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13557) waiting for paxos -> writeable 2017-03-02 15:08:44.465570 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27353) commit_finish 27354 2017-03-02 15:08:44.465626 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:44.465656 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12977) refresh 2017-03-02 15:08:44.465683 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) refresh 2017-03-02 15:08:44.465709 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:44.465765 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13558) refresh 2017-03-02 15:08:44.465776 7f4f763f4700 10 mon.0@0(leader).log v13558 update_from_paxos 2017-03-02 15:08:44.465779 7f4f763f4700 10 mon.0@0(leader).log v13558 update_from_paxos version 13558 summary v 13557 2017-03-02 15:08:44.465785 7f4f763f4700 10 mon.0@0(leader).log v13558 update_from_paxos latest full 13557 2017-03-02 15:08:44.465798 7f4f763f4700 7 mon.0@0(leader).log v13558 update_from_paxos applying incremental log 13558 2017-03-02 15:08:43.335327 mon.0 172.21.15.17:6789/0 14122 : cluster [INF] pgmap v12976: 32 pgs: 32 active+clean; 80992 MB data, 153 GB used, 218 GB / 372 GB avail; 353 kB/s rd, 17932 kB/s wr, 1128 op/s 2017-03-02 15:08:44.465817 7f4f763f4700 20 mon.0@0(leader).log v13558 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:44.465874 7f4f763f4700 15 mon.0@0(leader).log v13558 update_from_paxos logging for 1 channels 2017-03-02 15:08:44.465884 7f4f763f4700 15 mon.0@0(leader).log v13558 update_from_paxos channel 'cluster' logging 203 bytes 2017-03-02 15:08:44.465966 7f4f763f4700 10 mon.0@0(leader).log v13558 check_subs 2017-03-02 15:08:44.466016 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:44.466045 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:44.466048 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:44.466067 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:44.466070 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12977) post_refresh 2017-03-02 15:08:44.466072 7f4f763f4700 10 mon.0@0(leader).pg v12977 post_paxos_update 2017-03-02 15:08:44.466075 7f4f763f4700 10 mon.0@0(leader).pg v12977 check_subs 2017-03-02 15:08:44.466078 7f4f763f4700 20 mon.0@0(leader).pg v12977 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:44.466085 7f4f763f4700 20 mon.0@0(leader).pg v12977 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:44.466089 7f4f763f4700 20 mon.0@0(leader).pg v12977 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:44.466092 7f4f763f4700 20 mon.0@0(leader).pg v12977 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:44.466095 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) post_refresh 2017-03-02 15:08:44.466097 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:44.466099 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13558) post_refresh 2017-03-02 15:08:44.466101 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:44.466103 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:44.466105 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:44.466107 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27354) commit_proposal 2017-03-02 15:08:44.466112 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13558) _active - not active 2017-03-02 15:08:44.466115 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27354) finish_round 2017-03-02 15:08:44.466117 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27354) finish_round waiting_for_acting 2017-03-02 15:08:44.466120 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13558) _active 2017-03-02 15:08:44.466122 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13558) _active creating new pending 2017-03-02 15:08:44.466132 7f4f763f4700 10 mon.0@0(leader).log v13558 create_pending v 13559 2017-03-02 15:08:44.466135 7f4f763f4700 7 mon.0@0(leader).log v13558 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:44.466142 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a15d92000 0x558a15c44760 log(last 14122) v1 2017-03-02 15:08:44.466146 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14122) v1 -- 0x558a15c44760 con 0 2017-03-02 15:08:44.466172 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13558) dispatch 0x558a16743f80 log(1 entries from seq 14123 at 2017-03-02 15:08:44.399211) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:44.466183 7f4f763f4700 5 mon.0@0(leader).paxos(paxos active c 26607..27354) is_readable = 1 - now=2017-03-02 15:08:44.466184 lease_expire=0.000000 has v0 lc 27354 2017-03-02 15:08:44.466190 7f4f763f4700 10 mon.0@0(leader).log v13558 preprocess_query log(1 entries from seq 14123 at 2017-03-02 15:08:44.399211) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:44.466197 7f4f763f4700 10 mon.0@0(leader).log v13558 preprocess_log log(1 entries from seq 14123 at 2017-03-02 15:08:44.399211) v1 from mon.0 2017-03-02 15:08:44.466203 7f4f763f4700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:44.466192 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14122) v1 ==== 0+0+0 (0 0 0) 0x558a15c44760 con 0x558a14482000 2017-03-02 15:08:44.466206 7f4f763f4700 20 allow so far , doing grant allow * 2017-03-02 15:08:44.466208 7f4f763f4700 20 allow all 2017-03-02 15:08:44.466212 7f4f763f4700 10 mon.0@0(leader).log v13558 prepare_update log(1 entries from seq 14123 at 2017-03-02 15:08:44.399211) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:44.466221 7f4f763f4700 10 mon.0@0(leader).log v13558 prepare_log log(1 entries from seq 14123 at 2017-03-02 15:08:44.399211) v1 from mon.0 2017-03-02 15:08:44.466225 7f4f763f4700 10 mon.0@0(leader).log v13558 logging 2017-03-02 15:08:44.399211 mon.0 172.21.15.17:6789/0 14123 : cluster [INF] pgmap v12977: 32 pgs: 32 active+clean; 78308 MB data, 151 GB used, 220 GB / 372 GB avail; 133 kB/s rd, 10747 kB/s wr, 941 op/s 2017-03-02 15:08:44.466236 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13558) setting proposal_timer 0x558a1412eb90 with delay of 0.99936 2017-03-02 15:08:44.466258 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27354) finish_round waiting_for_readable 2017-03-02 15:08:44.466261 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27354) finish_round waiting_for_writeable 2017-03-02 15:08:44.466264 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27354) finish_round done w/ waiters, state active 2017-03-02 15:08:44.466282 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:44.466288 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:44.466291 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:44.466293 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:44.466294 7f4f71beb700 20 allow all 2017-03-02 15:08:45.465694 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13558) propose_pending 2017-03-02 15:08:45.465719 7f4f743f0700 10 mon.0@0(leader).log v13558 encode_full log v 13558 2017-03-02 15:08:45.465817 7f4f743f0700 10 mon.0@0(leader).log v13558 encode_pending v13559 2017-03-02 15:08:45.465864 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27354) queue_pending_finisher 0x558a1412c3d0 2017-03-02 15:08:45.465869 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27354) trigger_propose active, proposing now 2017-03-02 15:08:45.465878 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27354) propose_pending 27355 10275 bytes 2017-03-02 15:08:45.465881 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27354) begin for 27355 10275 bytes 2017-03-02 15:08:45.498556 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27354) commit_start 27355 2017-03-02 15:08:45.532004 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27354) commit_finish 27355 2017-03-02 15:08:45.532258 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:45.532289 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12977) refresh 2017-03-02 15:08:45.532306 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) refresh 2017-03-02 15:08:45.532320 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:45.532337 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13559) refresh 2017-03-02 15:08:45.532344 7f4f763f4700 10 mon.0@0(leader).log v13559 update_from_paxos 2017-03-02 15:08:45.532346 7f4f763f4700 10 mon.0@0(leader).log v13559 update_from_paxos version 13559 summary v 13558 2017-03-02 15:08:45.532350 7f4f763f4700 10 mon.0@0(leader).log v13559 update_from_paxos latest full 13558 2017-03-02 15:08:45.532359 7f4f763f4700 7 mon.0@0(leader).log v13559 update_from_paxos applying incremental log 13559 2017-03-02 15:08:44.399211 mon.0 172.21.15.17:6789/0 14123 : cluster [INF] pgmap v12977: 32 pgs: 32 active+clean; 78308 MB data, 151 GB used, 220 GB / 372 GB avail; 133 kB/s rd, 10747 kB/s wr, 941 op/s 2017-03-02 15:08:45.532373 7f4f763f4700 20 mon.0@0(leader).log v13559 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:45.532389 7f4f763f4700 15 mon.0@0(leader).log v13559 update_from_paxos logging for 1 channels 2017-03-02 15:08:45.532396 7f4f763f4700 15 mon.0@0(leader).log v13559 update_from_paxos channel 'cluster' logging 202 bytes 2017-03-02 15:08:45.532452 7f4f763f4700 10 mon.0@0(leader).log v13559 check_subs 2017-03-02 15:08:45.532487 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:45.532505 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:45.532512 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:45.532523 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:45.532525 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12977) post_refresh 2017-03-02 15:08:45.532527 7f4f763f4700 10 mon.0@0(leader).pg v12977 post_paxos_update 2017-03-02 15:08:45.532529 7f4f763f4700 10 mon.0@0(leader).pg v12977 check_subs 2017-03-02 15:08:45.532531 7f4f763f4700 20 mon.0@0(leader).pg v12977 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:45.532536 7f4f763f4700 20 mon.0@0(leader).pg v12977 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:45.532537 7f4f763f4700 20 mon.0@0(leader).pg v12977 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:45.532539 7f4f763f4700 20 mon.0@0(leader).pg v12977 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:45.532541 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) post_refresh 2017-03-02 15:08:45.532543 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:45.532544 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13559) post_refresh 2017-03-02 15:08:45.532545 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:45.532546 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:45.532547 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:45.532549 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27355) commit_proposal 2017-03-02 15:08:45.532552 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13559) _active - not active 2017-03-02 15:08:45.532554 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27355) finish_round 2017-03-02 15:08:45.532555 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27355) finish_round waiting_for_acting 2017-03-02 15:08:45.532557 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13559) _active 2017-03-02 15:08:45.532558 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13559) _active creating new pending 2017-03-02 15:08:45.532566 7f4f763f4700 10 mon.0@0(leader).log v13559 create_pending v 13560 2017-03-02 15:08:45.532569 7f4f763f4700 7 mon.0@0(leader).log v13559 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:45.532573 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a1623c800 0x558a15b69660 log(last 14123) v1 2017-03-02 15:08:45.532576 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14123) v1 -- 0x558a15b69660 con 0 2017-03-02 15:08:45.532595 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27355) finish_round waiting_for_readable 2017-03-02 15:08:45.532597 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27355) finish_round waiting_for_writeable 2017-03-02 15:08:45.532599 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27355) finish_round done w/ waiters, state active 2017-03-02 15:08:45.532651 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14123) v1 ==== 0+0+0 (0 0 0) 0x558a15b69660 con 0x558a14482000 2017-03-02 15:08:45.532665 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:45.532668 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:45.532671 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:45.532674 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:45.532675 7f4f71beb700 20 allow all 2017-03-02 15:08:46.012523 7f4f743f0700 11 mon.0@0(leader) e2 tick 2017-03-02 15:08:46.012547 7f4f743f0700 10 mon.0@0(leader).pg v12977 v12977: 32 pgs: 32 active+clean; 78308 MB data, 151 GB used, 220 GB / 372 GB avail; 133 kB/s rd, 10747 kB/s wr, 941 op/s 2017-03-02 15:08:46.012573 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12977) maybe_trim trim_to 12477 would only trim 208 < paxos_service_trim_min 250 2017-03-02 15:08:46.012578 7f4f743f0700 10 mon.0@0(leader).mds e737 e737: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:46.012605 7f4f743f0700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) maybe_trim trim_to 237 would only trim 236 < paxos_service_trim_min 250 2017-03-02 15:08:46.012607 7f4f743f0700 10 mon.0@0(leader).osd e7 e7: 4 osds: 4 up, 4 in 2017-03-02 15:08:46.012610 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27355) is_readable = 1 - now=2017-03-02 15:08:46.012611 lease_expire=0.000000 has v0 lc 27355 2017-03-02 15:08:46.012617 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27355) is_readable = 1 - now=2017-03-02 15:08:46.012617 lease_expire=0.000000 has v0 lc 27355 2017-03-02 15:08:46.012621 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27355) is_readable = 1 - now=2017-03-02 15:08:46.012621 lease_expire=0.000000 has v0 lc 27355 2017-03-02 15:08:46.012623 7f4f743f0700 10 mon.0@0(leader).osd e7 min_last_epoch_clean 6 2017-03-02 15:08:46.012624 7f4f743f0700 10 mon.0@0(leader).log v13559 log 2017-03-02 15:08:46.012626 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13559) maybe_trim trim_to 13059 would only trim 32 < paxos_service_trim_min 250 2017-03-02 15:08:46.012629 7f4f743f0700 10 mon.0@0(leader).auth v10 auth 2017-03-02 15:08:46.012632 7f4f743f0700 10 cephx keyserver: _check_rotating_secrets 2017-03-02 15:08:46.012639 7f4f743f0700 20 mon.0@0(leader) e2 sync_trim_providers 2017-03-02 15:08:46.870525 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.5 172.21.15.11:6804/1637448123 8790 ==== mdsbeacon(4116/e up:active seq 8503 v737) v7 ==== 365+0+0 (2320360431 0 0) 0x558a160c0a40 con 0x558a1471d800 2017-03-02 15:08:46.870563 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b0f20 for mds.? 172.21.15.11:6804/1637448123 2017-03-02 15:08:46.870569 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:46.870573 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a160c0a40 mdsbeacon(4116/e up:active seq 8503 v737) v7 from mds.5 172.21.15.11:6804/1637448123 con 0x558a1471d800 2017-03-02 15:08:46.870579 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27355) is_readable = 1 - now=2017-03-02 15:08:46.870579 lease_expire=0.000000 has v0 lc 27355 2017-03-02 15:08:46.870588 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4116/e up:active seq 8503 v737) v7 from mds.5 172.21.15.11:6804/1637448123 2017-03-02 15:08:46.870595 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:46.870598 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:46.870599 7f4f71beb700 20 allow all 2017-03-02 15:08:46.870600 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4116/e up:active seq 8503 v737) v7 from mds.5 172.21.15.11:6804/1637448123 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:46.870608 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4116 2017-03-02 15:08:46.870613 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4116/e up:active seq 8503 v737) v7 noting time 2017-03-02 15:08:46.870617 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a16630600 0x558a148d7380 mdsbeacon(4116/e up:active seq 8503 v737) v7 2017-03-02 15:08:46.870620 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6804/1637448123 -- mdsbeacon(4116/e up:active seq 8503 v737) v7 -- 0x558a148d7380 con 0 2017-03-02 15:08:46.884326 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.3 172.21.15.11:6805/1926402648 8566 ==== mdsbeacon(4118/f up:active seq 8503 v737) v7 ==== 406+0+0 (1509721192 0 0) 0x558a143e56c0 con 0x558a14905800 2017-03-02 15:08:46.884361 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b2260 for mds.? 172.21.15.11:6805/1926402648 2017-03-02 15:08:46.884367 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:46.884369 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a143e56c0 mdsbeacon(4118/f up:active seq 8503 v737) v7 from mds.3 172.21.15.11:6805/1926402648 con 0x558a14905800 2017-03-02 15:08:46.884375 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27355) is_readable = 1 - now=2017-03-02 15:08:46.884376 lease_expire=0.000000 has v0 lc 27355 2017-03-02 15:08:46.884383 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4118/f up:active seq 8503 v737) v7 from mds.3 172.21.15.11:6805/1926402648 2017-03-02 15:08:46.884389 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:46.884391 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:46.884392 7f4f71beb700 20 allow all 2017-03-02 15:08:46.884393 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4118/f up:active seq 8503 v737) v7 from mds.3 172.21.15.11:6805/1926402648 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:46.884400 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4118 2017-03-02 15:08:46.884404 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4118/f up:active seq 8503 v737) v7 noting time 2017-03-02 15:08:46.884408 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a16630600 0x558a160c0a40 mdsbeacon(4118/f up:active seq 8503 v737) v7 2017-03-02 15:08:46.884411 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6805/1926402648 -- mdsbeacon(4118/f up:active seq 8503 v737) v7 -- 0x558a160c0a40 con 0 2017-03-02 15:08:46.885322 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.2 172.21.15.22:6804/673088617 8565 ==== mdsbeacon(4119/c up:active seq 8503 v737) v7 ==== 365+0+0 (2857691057 0 0) 0x558a148d7380 con 0x558a14907000 2017-03-02 15:08:46.885363 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14916f20 for mds.? 172.21.15.22:6804/673088617 2017-03-02 15:08:46.885371 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:46.885375 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a148d7380 mdsbeacon(4119/c up:active seq 8503 v737) v7 from mds.2 172.21.15.22:6804/673088617 con 0x558a14907000 2017-03-02 15:08:46.885383 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27355) is_readable = 1 - now=2017-03-02 15:08:46.885384 lease_expire=0.000000 has v0 lc 27355 2017-03-02 15:08:46.885395 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4119/c up:active seq 8503 v737) v7 from mds.2 172.21.15.22:6804/673088617 2017-03-02 15:08:46.885405 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:46.885408 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:46.885410 7f4f71beb700 20 allow all 2017-03-02 15:08:46.885411 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4119/c up:active seq 8503 v737) v7 from mds.2 172.21.15.22:6804/673088617 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:46.885422 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4119 2017-03-02 15:08:46.885428 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4119/c up:active seq 8503 v737) v7 noting time 2017-03-02 15:08:46.885433 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a16630600 0x558a143e56c0 mdsbeacon(4119/c up:active seq 8503 v737) v7 2017-03-02 15:08:46.885437 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6804/673088617 -- mdsbeacon(4119/c up:active seq 8503 v737) v7 -- 0x558a143e56c0 con 0 2017-03-02 15:08:46.885494 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.0 172.21.15.22:6805/2203629154 8615 ==== mdsbeacon(4121/d up:active seq 8499 v736) v7 ==== 452+0+0 (3424127670 0 0) 0x558a160c0a40 con 0x558a1492a000 2017-03-02 15:08:46.885504 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a149195a0 for mds.? 172.21.15.22:6805/2203629154 2017-03-02 15:08:46.885508 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:46.885510 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a160c0a40 mdsbeacon(4121/d up:active seq 8499 v736) v7 from mds.0 172.21.15.22:6805/2203629154 con 0x558a1492a000 2017-03-02 15:08:46.885514 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27355) is_readable = 1 - now=2017-03-02 15:08:46.885514 lease_expire=0.000000 has v0 lc 27355 2017-03-02 15:08:46.885519 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4121/d up:active seq 8499 v736) v7 from mds.0 172.21.15.22:6805/2203629154 2017-03-02 15:08:46.885524 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:46.885526 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:46.885527 7f4f71beb700 20 allow all 2017-03-02 15:08:46.885528 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4121/d up:active seq 8499 v736) v7 from mds.0 172.21.15.22:6805/2203629154 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:46.885533 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4121 2017-03-02 15:08:46.885536 7f4f71beb700 10 mon.0@0(leader).mds e737 mds_beacon mdsbeacon(4121/d up:active seq 8499 v736) v7 ignoring requested state, because mds hasn't seen latest map 2017-03-02 15:08:46.885538 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4121/d up:active seq 8499 v736) v7 noting time 2017-03-02 15:08:46.885541 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a16630600 0x558a148d7380 mdsbeacon(4121/d up:active seq 8499 v737) v7 2017-03-02 15:08:46.885543 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6805/2203629154 -- mdsbeacon(4121/d up:active seq 8499 v737) v7 -- 0x558a148d7380 con 0 2017-03-02 15:08:46.924605 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.1 172.21.15.17:6804/2576824570 8562 ==== mdsbeacon(4120/g up:active seq 8503 v737) v7 ==== 365+0+0 (1650174634 0 0) 0x558a154b8d00 con 0x558a14908800 2017-03-02 15:08:46.924638 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14917600 for mds.? 172.21.15.17:6804/2576824570 2017-03-02 15:08:46.924644 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:46.924647 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a154b8d00 mdsbeacon(4120/g up:active seq 8503 v737) v7 from mds.1 172.21.15.17:6804/2576824570 con 0x558a14908800 2017-03-02 15:08:46.924652 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27355) is_readable = 1 - now=2017-03-02 15:08:46.924653 lease_expire=0.000000 has v0 lc 27355 2017-03-02 15:08:46.924661 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4120/g up:active seq 8503 v737) v7 from mds.1 172.21.15.17:6804/2576824570 2017-03-02 15:08:46.924667 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:46.924670 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:46.924671 7f4f71beb700 20 allow all 2017-03-02 15:08:46.924672 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4120/g up:active seq 8503 v737) v7 from mds.1 172.21.15.17:6804/2576824570 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:46.924679 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4120 2017-03-02 15:08:46.924684 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4120/g up:active seq 8503 v737) v7 noting time 2017-03-02 15:08:46.924687 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a16630600 0x558a160c0a40 mdsbeacon(4120/g up:active seq 8503 v737) v7 2017-03-02 15:08:46.924690 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6804/2576824570 -- mdsbeacon(4120/g up:active seq 8503 v737) v7 -- 0x558a160c0a40 con 0 2017-03-02 15:08:46.970052 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.4 172.21.15.26:6805/2033388711 8560 ==== mdsbeacon(4117/b up:active seq 8503 v737) v7 ==== 365+0+0 (2633151593 0 0) 0x558a160c0a40 con 0x558a14904000 2017-03-02 15:08:46.970090 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b1600 for mds.? 172.21.15.26:6805/2033388711 2017-03-02 15:08:46.970100 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:46.970104 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a160c0a40 mdsbeacon(4117/b up:active seq 8503 v737) v7 from mds.4 172.21.15.26:6805/2033388711 con 0x558a14904000 2017-03-02 15:08:46.970111 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27355) is_readable = 1 - now=2017-03-02 15:08:46.970112 lease_expire=0.000000 has v0 lc 27355 2017-03-02 15:08:46.970119 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4117/b up:active seq 8503 v737) v7 from mds.4 172.21.15.26:6805/2033388711 2017-03-02 15:08:46.970125 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:46.970127 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:46.970129 7f4f71beb700 20 allow all 2017-03-02 15:08:46.970130 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4117/b up:active seq 8503 v737) v7 from mds.4 172.21.15.26:6805/2033388711 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:46.970137 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4117 2017-03-02 15:08:46.970141 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4117/b up:active seq 8503 v737) v7 noting time 2017-03-02 15:08:46.970144 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a16630600 0x558a154b8d00 mdsbeacon(4117/b up:active seq 8503 v737) v7 2017-03-02 15:08:46.970147 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6805/2033388711 -- mdsbeacon(4117/b up:active seq 8503 v737) v7 -- 0x558a154b8d00 con 0 2017-03-02 15:08:46.979109 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.6 172.21.15.26:6804/2954782670 8753 ==== mdsbeacon(4115/a up:active seq 8503 v737) v7 ==== 365+0+0 (3340255233 0 0) 0x558a148d7380 con 0x558a1471f000 2017-03-02 15:08:46.979146 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14868d60 for mds.? 172.21.15.26:6804/2954782670 2017-03-02 15:08:46.979154 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:46.979157 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a148d7380 mdsbeacon(4115/a up:active seq 8503 v737) v7 from mds.6 172.21.15.26:6804/2954782670 con 0x558a1471f000 2017-03-02 15:08:46.979162 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27355) is_readable = 1 - now=2017-03-02 15:08:46.979162 lease_expire=0.000000 has v0 lc 27355 2017-03-02 15:08:46.979168 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4115/a up:active seq 8503 v737) v7 from mds.6 172.21.15.26:6804/2954782670 2017-03-02 15:08:46.979174 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:46.979176 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:46.979177 7f4f71beb700 20 allow all 2017-03-02 15:08:46.979178 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4115/a up:active seq 8503 v737) v7 from mds.6 172.21.15.26:6804/2954782670 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:46.979187 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4115 2017-03-02 15:08:46.979191 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4115/a up:active seq 8503 v737) v7 noting time 2017-03-02 15:08:46.979194 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a16630600 0x558a160c0a40 mdsbeacon(4115/a up:active seq 8503 v737) v7 2017-03-02 15:08:46.979197 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6804/2954782670 -- mdsbeacon(4115/a up:active seq 8503 v737) v7 -- 0x558a160c0a40 con 0 2017-03-02 15:08:48.265152 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.3 172.21.15.17:6800/27100 6838 ==== pg_stats(1 pgs tid 6802 v 0) v1 ==== 706+0+0 (6429013 0 0) 0x558a15d2e000 con 0x558a145d7000 2017-03-02 15:08:48.265181 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415f440 for osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:48.265187 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:48.265191 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12977) dispatch 0x558a15d2e000 pg_stats(1 pgs tid 6802 v 0) v1 from osd.3 172.21.15.17:6800/27100 con 0x558a145d7000 2017-03-02 15:08:48.265197 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27355) is_readable = 1 - now=2017-03-02 15:08:48.265198 lease_expire=0.000000 has v0 lc 27355 2017-03-02 15:08:48.265205 7f4f71beb700 10 mon.0@0(leader).pg v12977 preprocess_query pg_stats(1 pgs tid 6802 v 0) v1 from osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:48.265211 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:48.265214 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:48.265216 7f4f71beb700 20 allow all 2017-03-02 15:08:48.265217 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27355) is_readable = 1 - now=2017-03-02 15:08:48.265218 lease_expire=0.000000 has v0 lc 27355 2017-03-02 15:08:48.265224 7f4f71beb700 10 mon.0@0(leader).pg v12977 prepare_update pg_stats(1 pgs tid 6802 v 0) v1 from osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:48.265229 7f4f71beb700 10 mon.0@0(leader).pg v12977 prepare_pg_stats pg_stats(1 pgs tid 6802 v 0) v1 from osd.3 2017-03-02 15:08:48.265236 7f4f71beb700 10 mon.0@0(leader).pg v12977 got osd.3 osd_stat(36360 MB used, 58981 MB avail, 95341 MB total, peers [0,1,2] op hist [1,1]) (was osd_stat(38198 MB used, 57142 MB avail, 95341 MB total, peers [0,1,2] op hist [0,1])) 2017-03-02 15:08:48.265246 7f4f71beb700 15 mon.0@0(leader).pg v12977 got 2.6 reported at 7:965339 state active+clean -> active+clean 2017-03-02 15:08:48.265256 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12977) setting proposal_timer 0x558a1412de90 with delay of 0.05 2017-03-02 15:08:48.315334 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12977) propose_pending 2017-03-02 15:08:48.315349 7f4f743f0700 10 mon.0@0(leader).pg v12977 encode_pending v 12978 2017-03-02 15:08:48.315376 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27355) queue_pending_finisher 0x558a1412cc90 2017-03-02 15:08:48.315378 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27355) trigger_propose active, proposing now 2017-03-02 15:08:48.315385 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27355) propose_pending 27356 1098 bytes 2017-03-02 15:08:48.315387 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27355) begin for 27356 1098 bytes 2017-03-02 15:08:48.326336 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27355) commit_start 27356 2017-03-02 15:08:48.365335 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27355) commit_finish 27356 2017-03-02 15:08:48.365371 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:48.365387 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12978) refresh 2017-03-02 15:08:48.365389 7f4f763f4700 10 mon.0@0(leader).pg v12977 update_from_paxos read_incremental 2017-03-02 15:08:48.365408 7f4f763f4700 20 mon.0@0(leader).pg v12977 refreshing pg 2.6 7:965339 active+clean 2017-03-02 15:08:48.365411 7f4f763f4700 20 mon.0@0(leader).pg v12977 refreshing osd.3 2017-03-02 15:08:48.365422 7f4f763f4700 10 mon.0@0(leader).pg v12978 read_pgmap_meta 2017-03-02 15:08:48.365434 7f4f763f4700 10 mon.0@0(leader).pg v12978 update_logger 2017-03-02 15:08:48.365448 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) refresh 2017-03-02 15:08:48.365457 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:48.365470 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13559) refresh 2017-03-02 15:08:48.365471 7f4f763f4700 10 mon.0@0(leader).log v13559 update_from_paxos 2017-03-02 15:08:48.365473 7f4f763f4700 10 mon.0@0(leader).log v13559 update_from_paxos version 13559 summary v 13559 2017-03-02 15:08:48.365480 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:48.365490 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:48.365491 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:48.365500 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:48.365501 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12978) post_refresh 2017-03-02 15:08:48.365502 7f4f763f4700 10 mon.0@0(leader).pg v12978 post_paxos_update 2017-03-02 15:08:48.365504 7f4f763f4700 10 mon.0@0(leader).pg v12978 check_subs 2017-03-02 15:08:48.365506 7f4f763f4700 20 mon.0@0(leader).pg v12978 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:48.365511 7f4f763f4700 20 mon.0@0(leader).pg v12978 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:48.365513 7f4f763f4700 20 mon.0@0(leader).pg v12978 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:48.365515 7f4f763f4700 20 mon.0@0(leader).pg v12978 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:48.365517 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) post_refresh 2017-03-02 15:08:48.365518 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:48.365518 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13559) post_refresh 2017-03-02 15:08:48.365519 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:48.365520 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:48.365521 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:48.365524 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27356) commit_proposal 2017-03-02 15:08:48.365527 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12978) _active - not active 2017-03-02 15:08:48.365529 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27356) finish_round 2017-03-02 15:08:48.365530 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27356) finish_round waiting_for_acting 2017-03-02 15:08:48.365531 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12978) _active 2017-03-02 15:08:48.365532 7f4f763f4700 7 mon.0@0(leader).paxosservice(pgmap 12269..12978) _active creating new pending 2017-03-02 15:08:48.365535 7f4f763f4700 10 mon.0@0(leader).pg v12978 create_pending v 12979 2017-03-02 15:08:48.365539 7f4f763f4700 7 mon.0@0(leader).pg v12978 _updated_stats for osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:48.365542 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a16630600 0x558a1547aa00 pg_stats_ack(1 pgs tid 6802) v1 2017-03-02 15:08:48.365545 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6800/27100 -- pg_stats_ack(1 pgs tid 6802) v1 -- 0x558a1547aa00 con 0 2017-03-02 15:08:48.365570 7f4f763f4700 10 mon.0@0(leader).pg v12978 check_osd_map already seen 7 >= 7 2017-03-02 15:08:48.365571 7f4f763f4700 10 mon.0@0(leader).pg v12978 update_logger 2017-03-02 15:08:48.365582 7f4f763f4700 0 log_channel(cluster) log [INF] : pgmap v12978: 32 pgs: 32 active+clean; 77884 MB data, 149 GB used, 222 GB / 372 GB avail; 61299 B/s rd, 10934 kB/s wr, 960 op/s 2017-03-02 15:08:48.365592 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14124 at 2017-03-02 15:08:48.365584) v1 -- 0x558a16745600 con 0 2017-03-02 15:08:48.365603 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27356) finish_round waiting_for_readable 2017-03-02 15:08:48.365605 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27356) finish_round waiting_for_writeable 2017-03-02 15:08:48.365606 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27356) finish_round done w/ waiters, state active 2017-03-02 15:08:48.365641 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14124 at 2017-03-02 15:08:48.365584) v1 ==== 0+0+0 (0 0 0) 0x558a16745600 con 0x558a14482000 2017-03-02 15:08:48.365653 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:48.365657 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:48.365658 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13559) dispatch 0x558a16745600 log(1 entries from seq 14124 at 2017-03-02 15:08:48.365584) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:48.365663 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27356) is_readable = 1 - now=2017-03-02 15:08:48.365664 lease_expire=0.000000 has v0 lc 27356 2017-03-02 15:08:48.365668 7f4f71beb700 10 mon.0@0(leader).log v13559 preprocess_query log(1 entries from seq 14124 at 2017-03-02 15:08:48.365584) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:48.365674 7f4f71beb700 10 mon.0@0(leader).log v13559 preprocess_log log(1 entries from seq 14124 at 2017-03-02 15:08:48.365584) v1 from mon.0 2017-03-02 15:08:48.365677 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:48.365680 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:48.365681 7f4f71beb700 20 allow all 2017-03-02 15:08:48.365689 7f4f71beb700 10 mon.0@0(leader).log v13559 prepare_update log(1 entries from seq 14124 at 2017-03-02 15:08:48.365584) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:48.365694 7f4f71beb700 10 mon.0@0(leader).log v13559 prepare_log log(1 entries from seq 14124 at 2017-03-02 15:08:48.365584) v1 from mon.0 2017-03-02 15:08:48.365697 7f4f71beb700 10 mon.0@0(leader).log v13559 logging 2017-03-02 15:08:48.365584 mon.0 172.21.15.17:6789/0 14124 : cluster [INF] pgmap v12978: 32 pgs: 32 active+clean; 77884 MB data, 149 GB used, 222 GB / 372 GB avail; 61299 B/s rd, 10934 kB/s wr, 960 op/s 2017-03-02 15:08:48.365705 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13559) setting proposal_timer 0x558a1412f270 with delay of 0.999646 2017-03-02 15:08:48.466723 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.1 172.21.15.22:6800/26531 6838 ==== pg_stats(5 pgs tid 6802 v 0) v1 ==== 2970+0+0 (2692448869 0 0) 0x558a15c0b980 con 0x558a14643800 2017-03-02 15:08:48.466750 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1463b1e0 for osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:48.466755 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:48.466757 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12978) dispatch 0x558a15c0b980 pg_stats(5 pgs tid 6802 v 0) v1 from osd.1 172.21.15.22:6800/26531 con 0x558a14643800 2017-03-02 15:08:48.466762 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27356) is_readable = 1 - now=2017-03-02 15:08:48.466762 lease_expire=0.000000 has v0 lc 27356 2017-03-02 15:08:48.466769 7f4f71beb700 10 mon.0@0(leader).pg v12978 preprocess_query pg_stats(5 pgs tid 6802 v 0) v1 from osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:48.466774 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:48.466776 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:48.466777 7f4f71beb700 20 allow all 2017-03-02 15:08:48.466778 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27356) is_readable = 1 - now=2017-03-02 15:08:48.466778 lease_expire=0.000000 has v0 lc 27356 2017-03-02 15:08:48.466781 7f4f71beb700 10 mon.0@0(leader).pg v12978 prepare_update pg_stats(5 pgs tid 6802 v 0) v1 from osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:48.466785 7f4f71beb700 10 mon.0@0(leader).pg v12978 prepare_pg_stats pg_stats(5 pgs tid 6802 v 0) v1 from osd.1 2017-03-02 15:08:48.466790 7f4f71beb700 10 mon.0@0(leader).pg v12978 got osd.1 osd_stat(46745 MB used, 48596 MB avail, 95341 MB total, peers [0,2,3] op hist [0,2]) (was osd_stat(49225 MB used, 46115 MB avail, 95341 MB total, peers [0,2,3] op hist [2])) 2017-03-02 15:08:48.466797 7f4f71beb700 15 mon.0@0(leader).pg v12978 got 1.1 reported at 7:587205 state active+clean -> active+clean 2017-03-02 15:08:48.466805 7f4f71beb700 15 mon.0@0(leader).pg v12978 got 1.5 reported at 7:568053 state active+clean -> active+clean 2017-03-02 15:08:48.466809 7f4f71beb700 15 mon.0@0(leader).pg v12978 got 1.6 reported at 7:503459 state active+clean -> active+clean 2017-03-02 15:08:48.466813 7f4f71beb700 15 mon.0@0(leader).pg v12978 got 2.4 reported at 7:1027160 state active+clean -> active+clean 2017-03-02 15:08:48.466816 7f4f71beb700 15 mon.0@0(leader).pg v12978 got 2.5 reported at 7:1130500 state active+clean -> active+clean 2017-03-02 15:08:48.466844 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12978) setting proposal_timer 0x558a1412ce70 with delay of 0.898506 2017-03-02 15:08:48.560442 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.2 172.21.15.11:6800/26536 6838 ==== pg_stats(8 pgs tid 6802 v 0) v1 ==== 4668+0+0 (2518752472 0 0) 0x558a14d9a580 con 0x558a145d5800 2017-03-02 15:08:48.560488 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a145c3700 for osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:48.560497 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:48.560502 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12978) dispatch 0x558a14d9a580 pg_stats(8 pgs tid 6802 v 0) v1 from osd.2 172.21.15.11:6800/26536 con 0x558a145d5800 2017-03-02 15:08:48.560510 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27356) is_readable = 1 - now=2017-03-02 15:08:48.560511 lease_expire=0.000000 has v0 lc 27356 2017-03-02 15:08:48.560523 7f4f71beb700 10 mon.0@0(leader).pg v12978 preprocess_query pg_stats(8 pgs tid 6802 v 0) v1 from osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:48.560534 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:48.560538 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:48.560539 7f4f71beb700 20 allow all 2017-03-02 15:08:48.560542 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27356) is_readable = 1 - now=2017-03-02 15:08:48.560542 lease_expire=0.000000 has v0 lc 27356 2017-03-02 15:08:48.560550 7f4f71beb700 10 mon.0@0(leader).pg v12978 prepare_update pg_stats(8 pgs tid 6802 v 0) v1 from osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:48.560557 7f4f71beb700 10 mon.0@0(leader).pg v12978 prepare_pg_stats pg_stats(8 pgs tid 6802 v 0) v1 from osd.2 2017-03-02 15:08:48.560566 7f4f71beb700 10 mon.0@0(leader).pg v12978 got osd.2 osd_stat(55482 MB used, 39858 MB avail, 95341 MB total, peers [0,1,3] op hist [0,2]) (was osd_stat(57791 MB used, 37549 MB avail, 95341 MB total, peers [0,1,3] op hist [1])) 2017-03-02 15:08:48.560581 7f4f71beb700 15 mon.0@0(leader).pg v12978 got 1.0 reported at 7:496901 state active+clean -> active+clean 2017-03-02 15:08:48.560594 7f4f71beb700 15 mon.0@0(leader).pg v12978 got 1.2 reported at 7:577757 state active+clean -> active+clean 2017-03-02 15:08:48.560602 7f4f71beb700 15 mon.0@0(leader).pg v12978 got 1.7 reported at 7:530054 state active+clean -> active+clean 2017-03-02 15:08:48.560609 7f4f71beb700 15 mon.0@0(leader).pg v12978 got 2.0 reported at 7:1057769 state active+clean -> active+clean 2017-03-02 15:08:48.560616 7f4f71beb700 15 mon.0@0(leader).pg v12978 got 2.1 reported at 7:1090470 state active+clean -> active+clean 2017-03-02 15:08:48.560623 7f4f71beb700 15 mon.0@0(leader).pg v12978 got 2.2 reported at 7:1164223 state active+clean -> active+clean 2017-03-02 15:08:48.560630 7f4f71beb700 15 mon.0@0(leader).pg v12978 got 2.3 reported at 7:1096301 state active+clean -> active+clean 2017-03-02 15:08:48.560638 7f4f71beb700 15 mon.0@0(leader).pg v12978 got 2.7 reported at 7:1136405 state active+clean -> active+clean 2017-03-02 15:08:48.560646 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12978) proposal_timer already set 2017-03-02 15:08:48.716541 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.0 172.21.15.26:6800/26515 6838 ==== pg_stats(2 pgs tid 6802 v 0) v1 ==== 1264+0+0 (734681761 0 0) 0x558a15da1680 con 0x558a145d8800 2017-03-02 15:08:48.716583 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1463af20 for osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:48.716591 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:48.716595 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12978) dispatch 0x558a15da1680 pg_stats(2 pgs tid 6802 v 0) v1 from osd.0 172.21.15.26:6800/26515 con 0x558a145d8800 2017-03-02 15:08:48.716600 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27356) is_readable = 1 - now=2017-03-02 15:08:48.716601 lease_expire=0.000000 has v0 lc 27356 2017-03-02 15:08:48.716609 7f4f71beb700 10 mon.0@0(leader).pg v12978 preprocess_query pg_stats(2 pgs tid 6802 v 0) v1 from osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:48.716615 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:48.716618 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:48.716619 7f4f71beb700 20 allow all 2017-03-02 15:08:48.716620 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26607..27356) is_readable = 1 - now=2017-03-02 15:08:48.716620 lease_expire=0.000000 has v0 lc 27356 2017-03-02 15:08:48.716624 7f4f71beb700 10 mon.0@0(leader).pg v12978 prepare_update pg_stats(2 pgs tid 6802 v 0) v1 from osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:48.716627 7f4f71beb700 10 mon.0@0(leader).pg v12978 prepare_pg_stats pg_stats(2 pgs tid 6802 v 0) v1 from osd.0 2017-03-02 15:08:48.716634 7f4f71beb700 10 mon.0@0(leader).pg v12978 got osd.0 osd_stat(9578 MB used, 85763 MB avail, 95341 MB total, peers [1,2,3] op hist []) (was osd_stat(10127 MB used, 85214 MB avail, 95341 MB total, peers [1,2,3] op hist [])) 2017-03-02 15:08:48.716643 7f4f71beb700 15 mon.0@0(leader).pg v12978 got 1.3 reported at 7:500495 state active+clean -> active+clean 2017-03-02 15:08:48.716652 7f4f71beb700 15 mon.0@0(leader).pg v12978 got 1.4 reported at 7:486439 state active+clean -> active+clean 2017-03-02 15:08:48.716657 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12978) proposal_timer already set 2017-03-02 15:08:49.365436 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13559) propose_pending 2017-03-02 15:08:49.365456 7f4f743f0700 10 mon.0@0(leader).log v13559 encode_full log v 13559 2017-03-02 15:08:49.365514 7f4f743f0700 10 mon.0@0(leader).log v13559 encode_pending v13560 2017-03-02 15:08:49.365520 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26607..27356) queue_pending_finisher 0x558a1412de90 2017-03-02 15:08:49.365523 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27356) trigger_propose active, proposing now 2017-03-02 15:08:49.365529 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26607..27356) propose_pending 27357 10275 bytes 2017-03-02 15:08:49.365531 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27356) begin for 27357 10275 bytes 2017-03-02 15:08:49.398731 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26607..27356) commit_start 27357 2017-03-02 15:08:49.398796 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12978) propose_pending 2017-03-02 15:08:49.398802 7f4f743f0700 10 mon.0@0(leader).pg v12978 encode_pending v 12979 2017-03-02 15:08:49.398896 7f4f743f0700 5 mon.0@0(leader).paxos(paxos writing c 26607..27356) queue_pending_finisher 0x558a1412f270 2017-03-02 15:08:49.398901 7f4f743f0700 10 mon.0@0(leader).paxos(paxos writing c 26607..27356) trigger_propose not active, will propose later 2017-03-02 15:08:49.432163 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27356) commit_finish 27357 2017-03-02 15:08:49.432206 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:49.432229 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12978) refresh 2017-03-02 15:08:49.432259 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) refresh 2017-03-02 15:08:49.432274 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:49.432292 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13560) refresh 2017-03-02 15:08:49.432298 7f4f763f4700 10 mon.0@0(leader).log v13560 update_from_paxos 2017-03-02 15:08:49.432300 7f4f763f4700 10 mon.0@0(leader).log v13560 update_from_paxos version 13560 summary v 13559 2017-03-02 15:08:49.432304 7f4f763f4700 10 mon.0@0(leader).log v13560 update_from_paxos latest full 13559 2017-03-02 15:08:49.432325 7f4f763f4700 7 mon.0@0(leader).log v13560 update_from_paxos applying incremental log 13560 2017-03-02 15:08:48.365584 mon.0 172.21.15.17:6789/0 14124 : cluster [INF] pgmap v12978: 32 pgs: 32 active+clean; 77884 MB data, 149 GB used, 222 GB / 372 GB avail; 61299 B/s rd, 10934 kB/s wr, 960 op/s 2017-03-02 15:08:49.432339 7f4f763f4700 20 mon.0@0(leader).log v13560 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:49.432355 7f4f763f4700 15 mon.0@0(leader).log v13560 update_from_paxos logging for 1 channels 2017-03-02 15:08:49.432362 7f4f763f4700 15 mon.0@0(leader).log v13560 update_from_paxos channel 'cluster' logging 203 bytes 2017-03-02 15:08:49.432424 7f4f763f4700 10 mon.0@0(leader).log v13560 check_subs 2017-03-02 15:08:49.432458 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:49.432476 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:49.432482 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:49.432494 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:49.432496 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12978) post_refresh 2017-03-02 15:08:49.432497 7f4f763f4700 10 mon.0@0(leader).pg v12978 post_paxos_update 2017-03-02 15:08:49.432499 7f4f763f4700 10 mon.0@0(leader).pg v12978 check_subs 2017-03-02 15:08:49.432501 7f4f763f4700 20 mon.0@0(leader).pg v12978 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:49.432505 7f4f763f4700 20 mon.0@0(leader).pg v12978 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:49.432508 7f4f763f4700 20 mon.0@0(leader).pg v12978 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:49.432510 7f4f763f4700 20 mon.0@0(leader).pg v12978 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:49.432512 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) post_refresh 2017-03-02 15:08:49.432514 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:49.432515 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13560) post_refresh 2017-03-02 15:08:49.432516 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:49.432517 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:49.432518 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:49.432520 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27357) commit_proposal 2017-03-02 15:08:49.432523 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13560) _active - not active 2017-03-02 15:08:49.432525 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27357) finish_round 2017-03-02 15:08:49.432526 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27357) finish_round waiting_for_acting 2017-03-02 15:08:49.432528 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13560) _active 2017-03-02 15:08:49.432530 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13560) _active creating new pending 2017-03-02 15:08:49.432537 7f4f763f4700 10 mon.0@0(leader).log v13560 create_pending v 13561 2017-03-02 15:08:49.432540 7f4f763f4700 7 mon.0@0(leader).log v13560 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:49.432544 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a14957700 0x558a150d65a0 log(last 14124) v1 2017-03-02 15:08:49.432548 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14124) v1 -- 0x558a150d65a0 con 0 2017-03-02 15:08:49.432568 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27357) finish_round waiting_for_readable 2017-03-02 15:08:49.432570 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27357) finish_round waiting_for_writeable 2017-03-02 15:08:49.432571 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27357) finish_round done w/ waiters, state active 2017-03-02 15:08:49.432597 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27357) propose_pending 27358 9752 bytes 2017-03-02 15:08:49.432603 7f4f763f4700 10 mon.0@0(leader).paxos(paxos updating c 26607..27357) begin for 27358 9752 bytes 2017-03-02 15:08:49.432591 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14124) v1 ==== 0+0+0 (0 0 0) 0x558a150d65a0 con 0x558a14482000 2017-03-02 15:08:49.465608 7f4f763f4700 10 mon.0@0(leader).paxos(paxos updating c 26607..27357) commit_start 27358 2017-03-02 15:08:49.465688 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:49.465705 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:49.465709 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:49.465712 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:49.465713 7f4f71beb700 20 allow all 2017-03-02 15:08:49.499092 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27357) commit_finish 27358 2017-03-02 15:08:49.499132 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:49.499148 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12979) refresh 2017-03-02 15:08:49.499150 7f4f763f4700 10 mon.0@0(leader).pg v12978 update_from_paxos read_incremental 2017-03-02 15:08:49.499170 7f4f763f4700 20 mon.0@0(leader).pg v12978 refreshing pg 1.0 7:496901 active+clean 2017-03-02 15:08:49.499177 7f4f763f4700 20 mon.0@0(leader).pg v12978 refreshing pg 1.1 7:587205 active+clean 2017-03-02 15:08:49.499182 7f4f763f4700 20 mon.0@0(leader).pg v12978 refreshing pg 1.2 7:577757 active+clean 2017-03-02 15:08:49.499186 7f4f763f4700 20 mon.0@0(leader).pg v12978 refreshing pg 1.3 7:500495 active+clean 2017-03-02 15:08:49.499190 7f4f763f4700 20 mon.0@0(leader).pg v12978 refreshing pg 1.4 7:486439 active+clean 2017-03-02 15:08:49.499194 7f4f763f4700 20 mon.0@0(leader).pg v12978 refreshing pg 1.5 7:568053 active+clean 2017-03-02 15:08:49.499199 7f4f763f4700 20 mon.0@0(leader).pg v12978 refreshing pg 1.6 7:503459 active+clean 2017-03-02 15:08:49.499203 7f4f763f4700 20 mon.0@0(leader).pg v12978 refreshing pg 1.7 7:530054 active+clean 2017-03-02 15:08:49.499207 7f4f763f4700 20 mon.0@0(leader).pg v12978 refreshing pg 2.0 7:1057769 active+clean 2017-03-02 15:08:49.499211 7f4f763f4700 20 mon.0@0(leader).pg v12978 refreshing pg 2.1 7:1090470 active+clean 2017-03-02 15:08:49.499215 7f4f763f4700 20 mon.0@0(leader).pg v12978 refreshing pg 2.2 7:1164223 active+clean 2017-03-02 15:08:49.499219 7f4f763f4700 20 mon.0@0(leader).pg v12978 refreshing pg 2.3 7:1096301 active+clean 2017-03-02 15:08:49.499222 7f4f763f4700 20 mon.0@0(leader).pg v12978 refreshing pg 2.4 7:1027160 active+clean 2017-03-02 15:08:49.499226 7f4f763f4700 20 mon.0@0(leader).pg v12978 refreshing pg 2.5 7:1130500 active+clean 2017-03-02 15:08:49.499230 7f4f763f4700 20 mon.0@0(leader).pg v12978 refreshing pg 2.7 7:1136405 active+clean 2017-03-02 15:08:49.499231 7f4f763f4700 20 mon.0@0(leader).pg v12978 refreshing osd.0 2017-03-02 15:08:49.499237 7f4f763f4700 20 mon.0@0(leader).pg v12978 refreshing osd.1 2017-03-02 15:08:49.499240 7f4f763f4700 20 mon.0@0(leader).pg v12978 refreshing osd.2 2017-03-02 15:08:49.499249 7f4f763f4700 10 mon.0@0(leader).pg v12979 read_pgmap_meta 2017-03-02 15:08:49.499262 7f4f763f4700 10 mon.0@0(leader).pg v12979 update_logger 2017-03-02 15:08:49.499271 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) refresh 2017-03-02 15:08:49.499281 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:49.499294 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13560) refresh 2017-03-02 15:08:49.499295 7f4f763f4700 10 mon.0@0(leader).log v13560 update_from_paxos 2017-03-02 15:08:49.499297 7f4f763f4700 10 mon.0@0(leader).log v13560 update_from_paxos version 13560 summary v 13560 2017-03-02 15:08:49.499304 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:49.499313 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:49.499314 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:49.499323 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:49.499324 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12979) post_refresh 2017-03-02 15:08:49.499325 7f4f763f4700 10 mon.0@0(leader).pg v12979 post_paxos_update 2017-03-02 15:08:49.499327 7f4f763f4700 10 mon.0@0(leader).pg v12979 check_subs 2017-03-02 15:08:49.499329 7f4f763f4700 20 mon.0@0(leader).pg v12979 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:49.499334 7f4f763f4700 20 mon.0@0(leader).pg v12979 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:49.499336 7f4f763f4700 20 mon.0@0(leader).pg v12979 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:49.499338 7f4f763f4700 20 mon.0@0(leader).pg v12979 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:49.499340 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) post_refresh 2017-03-02 15:08:49.499341 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:49.499342 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13560) post_refresh 2017-03-02 15:08:49.499343 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:49.499343 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:49.499344 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:49.499346 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27358) commit_proposal 2017-03-02 15:08:49.499348 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12979) _active - not active 2017-03-02 15:08:49.499350 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26607..27358) finish_round 2017-03-02 15:08:49.499351 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27358) finish_round waiting_for_acting 2017-03-02 15:08:49.499352 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12979) _active 2017-03-02 15:08:49.499356 7f4f763f4700 7 mon.0@0(leader).paxosservice(pgmap 12269..12979) _active creating new pending 2017-03-02 15:08:49.499363 7f4f763f4700 10 mon.0@0(leader).pg v12979 create_pending v 12980 2017-03-02 15:08:49.499367 7f4f763f4700 7 mon.0@0(leader).pg v12979 _updated_stats for osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:49.499370 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a14956e00 0x558a1589f000 pg_stats_ack(5 pgs tid 6802) v1 2017-03-02 15:08:49.499373 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6800/26531 -- pg_stats_ack(5 pgs tid 6802) v1 -- 0x558a1589f000 con 0 2017-03-02 15:08:49.499401 7f4f763f4700 7 mon.0@0(leader).pg v12979 _updated_stats for osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:49.499404 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a1483e600 0x558a15562200 pg_stats_ack(8 pgs tid 6802) v1 2017-03-02 15:08:49.499406 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6800/26536 -- pg_stats_ack(8 pgs tid 6802) v1 -- 0x558a15562200 con 0 2017-03-02 15:08:49.499415 7f4f763f4700 7 mon.0@0(leader).pg v12979 _updated_stats for osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:49.499417 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a16600200 0x558a15659600 pg_stats_ack(2 pgs tid 6802) v1 2017-03-02 15:08:49.499418 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6800/26515 -- pg_stats_ack(2 pgs tid 6802) v1 -- 0x558a15659600 con 0 2017-03-02 15:08:49.499434 7f4f763f4700 10 mon.0@0(leader).pg v12979 check_osd_map already seen 7 >= 7 2017-03-02 15:08:49.499436 7f4f763f4700 10 mon.0@0(leader).pg v12979 update_logger 2017-03-02 15:08:49.499447 7f4f763f4700 0 log_channel(cluster) log [INF] : pgmap v12979: 32 pgs: 32 active+clean; 74914 MB data, 144 GB used, 227 GB / 372 GB avail; 0 B/s rd, 3293 kB/s wr, 769 op/s 2017-03-02 15:08:49.499456 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14125 at 2017-03-02 15:08:49.499448) v1 -- 0x558a16743f80 con 0 2017-03-02 15:08:49.499466 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27358) finish_round waiting_for_readable 2017-03-02 15:08:49.499469 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26607..27358) finish_round waiting_for_writeable 2017-03-02 15:08:49.499471 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) finish_round done w/ waiters, state active 2017-03-02 15:08:49.499473 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim to 26858 (was 26607) 2017-03-02 15:08:49.499475 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26607 2017-03-02 15:08:49.499478 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26608 2017-03-02 15:08:49.499481 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26609 2017-03-02 15:08:49.499483 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26610 2017-03-02 15:08:49.499486 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26611 2017-03-02 15:08:49.499488 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26612 2017-03-02 15:08:49.499491 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26613 2017-03-02 15:08:49.499494 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26614 2017-03-02 15:08:49.499496 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26615 2017-03-02 15:08:49.499498 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26616 2017-03-02 15:08:49.499501 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26617 2017-03-02 15:08:49.499503 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26618 2017-03-02 15:08:49.499495 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14125 at 2017-03-02 15:08:49.499448) v1 ==== 0+0+0 (0 0 0) 0x558a16743f80 con 0x558a14482000 2017-03-02 15:08:49.499506 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26619 2017-03-02 15:08:49.499508 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26620 2017-03-02 15:08:49.499511 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26621 2017-03-02 15:08:49.499513 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26622 2017-03-02 15:08:49.499516 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26623 2017-03-02 15:08:49.499518 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26624 2017-03-02 15:08:49.499520 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26625 2017-03-02 15:08:49.499521 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26626 2017-03-02 15:08:49.499523 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26627 2017-03-02 15:08:49.499524 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26628 2017-03-02 15:08:49.499526 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26629 2017-03-02 15:08:49.499527 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26630 2017-03-02 15:08:49.499529 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26631 2017-03-02 15:08:49.499530 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26632 2017-03-02 15:08:49.499532 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26633 2017-03-02 15:08:49.499533 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26634 2017-03-02 15:08:49.499534 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26635 2017-03-02 15:08:49.499536 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26636 2017-03-02 15:08:49.499537 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26637 2017-03-02 15:08:49.499539 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26638 2017-03-02 15:08:49.499540 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26639 2017-03-02 15:08:49.499542 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26640 2017-03-02 15:08:49.499543 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26641 2017-03-02 15:08:49.499545 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26642 2017-03-02 15:08:49.499546 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26643 2017-03-02 15:08:49.499548 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26644 2017-03-02 15:08:49.499566 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26645 2017-03-02 15:08:49.499568 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26646 2017-03-02 15:08:49.499570 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26647 2017-03-02 15:08:49.499572 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26648 2017-03-02 15:08:49.499573 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26649 2017-03-02 15:08:49.499575 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26650 2017-03-02 15:08:49.499576 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26651 2017-03-02 15:08:49.499578 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26652 2017-03-02 15:08:49.499579 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26653 2017-03-02 15:08:49.499581 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26654 2017-03-02 15:08:49.499582 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26655 2017-03-02 15:08:49.499584 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26656 2017-03-02 15:08:49.499585 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26657 2017-03-02 15:08:49.499586 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26658 2017-03-02 15:08:49.499588 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26659 2017-03-02 15:08:49.499589 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26660 2017-03-02 15:08:49.499591 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26661 2017-03-02 15:08:49.499592 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26662 2017-03-02 15:08:49.499594 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26663 2017-03-02 15:08:49.499595 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26664 2017-03-02 15:08:49.499596 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26665 2017-03-02 15:08:49.499598 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26666 2017-03-02 15:08:49.499599 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26667 2017-03-02 15:08:49.499601 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26668 2017-03-02 15:08:49.499602 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26669 2017-03-02 15:08:49.499604 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26670 2017-03-02 15:08:49.499605 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26671 2017-03-02 15:08:49.499606 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26672 2017-03-02 15:08:49.499608 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26673 2017-03-02 15:08:49.499610 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26674 2017-03-02 15:08:49.499611 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26675 2017-03-02 15:08:49.499613 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26676 2017-03-02 15:08:49.499614 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26677 2017-03-02 15:08:49.499616 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26678 2017-03-02 15:08:49.499617 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26679 2017-03-02 15:08:49.499619 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26680 2017-03-02 15:08:49.499620 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26681 2017-03-02 15:08:49.499621 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26682 2017-03-02 15:08:49.499623 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26683 2017-03-02 15:08:49.499637 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26684 2017-03-02 15:08:49.499639 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26685 2017-03-02 15:08:49.499641 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26686 2017-03-02 15:08:49.499642 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26687 2017-03-02 15:08:49.499643 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26688 2017-03-02 15:08:49.499645 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26689 2017-03-02 15:08:49.499647 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26690 2017-03-02 15:08:49.499648 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26691 2017-03-02 15:08:49.499649 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26692 2017-03-02 15:08:49.499651 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26693 2017-03-02 15:08:49.499653 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26694 2017-03-02 15:08:49.499655 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26695 2017-03-02 15:08:49.499657 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26696 2017-03-02 15:08:49.499658 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26697 2017-03-02 15:08:49.499660 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26698 2017-03-02 15:08:49.499661 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26699 2017-03-02 15:08:49.499663 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26700 2017-03-02 15:08:49.499664 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26701 2017-03-02 15:08:49.499666 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26702 2017-03-02 15:08:49.499667 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26703 2017-03-02 15:08:49.499669 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26704 2017-03-02 15:08:49.499670 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26705 2017-03-02 15:08:49.499672 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26706 2017-03-02 15:08:49.499673 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26707 2017-03-02 15:08:49.499675 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26708 2017-03-02 15:08:49.499676 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26709 2017-03-02 15:08:49.499678 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26710 2017-03-02 15:08:49.499679 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26711 2017-03-02 15:08:49.499680 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26712 2017-03-02 15:08:49.499682 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26713 2017-03-02 15:08:49.499683 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26714 2017-03-02 15:08:49.499685 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26715 2017-03-02 15:08:49.499686 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26716 2017-03-02 15:08:49.499688 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26717 2017-03-02 15:08:49.499689 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26718 2017-03-02 15:08:49.499691 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26719 2017-03-02 15:08:49.499692 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26720 2017-03-02 15:08:49.499694 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26721 2017-03-02 15:08:49.499695 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26722 2017-03-02 15:08:49.499696 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26723 2017-03-02 15:08:49.499698 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26724 2017-03-02 15:08:49.499699 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26725 2017-03-02 15:08:49.499701 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26726 2017-03-02 15:08:49.499703 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26727 2017-03-02 15:08:49.499704 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26728 2017-03-02 15:08:49.499706 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26729 2017-03-02 15:08:49.499707 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26730 2017-03-02 15:08:49.499709 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26731 2017-03-02 15:08:49.499710 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26732 2017-03-02 15:08:49.499712 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26733 2017-03-02 15:08:49.499713 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26734 2017-03-02 15:08:49.499715 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26735 2017-03-02 15:08:49.499716 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26736 2017-03-02 15:08:49.499718 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26737 2017-03-02 15:08:49.499719 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26738 2017-03-02 15:08:49.499721 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26739 2017-03-02 15:08:49.499722 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26740 2017-03-02 15:08:49.499723 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26741 2017-03-02 15:08:49.499725 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26742 2017-03-02 15:08:49.499726 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26743 2017-03-02 15:08:49.499728 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26744 2017-03-02 15:08:49.499729 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26745 2017-03-02 15:08:49.499731 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26746 2017-03-02 15:08:49.499732 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26747 2017-03-02 15:08:49.499734 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26748 2017-03-02 15:08:49.499735 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26749 2017-03-02 15:08:49.499737 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26750 2017-03-02 15:08:49.499738 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26751 2017-03-02 15:08:49.499740 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26752 2017-03-02 15:08:49.499741 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26753 2017-03-02 15:08:49.499743 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26754 2017-03-02 15:08:49.499744 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26755 2017-03-02 15:08:49.499746 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26756 2017-03-02 15:08:49.499747 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26757 2017-03-02 15:08:49.499749 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26758 2017-03-02 15:08:49.499750 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26759 2017-03-02 15:08:49.499752 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26760 2017-03-02 15:08:49.499753 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26761 2017-03-02 15:08:49.499755 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26762 2017-03-02 15:08:49.499756 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26763 2017-03-02 15:08:49.499758 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26764 2017-03-02 15:08:49.499759 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26765 2017-03-02 15:08:49.499760 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26766 2017-03-02 15:08:49.499762 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26767 2017-03-02 15:08:49.499763 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26768 2017-03-02 15:08:49.499765 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26769 2017-03-02 15:08:49.499767 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26770 2017-03-02 15:08:49.499768 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26771 2017-03-02 15:08:49.499769 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26772 2017-03-02 15:08:49.499771 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26773 2017-03-02 15:08:49.499772 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26774 2017-03-02 15:08:49.499774 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26775 2017-03-02 15:08:49.499775 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26776 2017-03-02 15:08:49.499776 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26777 2017-03-02 15:08:49.499778 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26778 2017-03-02 15:08:49.499779 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26779 2017-03-02 15:08:49.499781 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26780 2017-03-02 15:08:49.499782 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26781 2017-03-02 15:08:49.499784 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26782 2017-03-02 15:08:49.499791 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26783 2017-03-02 15:08:49.499793 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26784 2017-03-02 15:08:49.499794 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26785 2017-03-02 15:08:49.499796 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26786 2017-03-02 15:08:49.499797 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26787 2017-03-02 15:08:49.499799 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26788 2017-03-02 15:08:49.499800 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26789 2017-03-02 15:08:49.499802 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26790 2017-03-02 15:08:49.499803 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26791 2017-03-02 15:08:49.499805 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26792 2017-03-02 15:08:49.499806 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26793 2017-03-02 15:08:49.499808 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26794 2017-03-02 15:08:49.499809 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26795 2017-03-02 15:08:49.499811 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26796 2017-03-02 15:08:49.499812 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26797 2017-03-02 15:08:49.499814 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26798 2017-03-02 15:08:49.499815 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26799 2017-03-02 15:08:49.499817 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26800 2017-03-02 15:08:49.499837 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26801 2017-03-02 15:08:49.499839 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26802 2017-03-02 15:08:49.499841 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26803 2017-03-02 15:08:49.499842 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26804 2017-03-02 15:08:49.499844 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26805 2017-03-02 15:08:49.499845 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26806 2017-03-02 15:08:49.499846 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26807 2017-03-02 15:08:49.499848 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26808 2017-03-02 15:08:49.499850 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26809 2017-03-02 15:08:49.499851 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26810 2017-03-02 15:08:49.499853 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26811 2017-03-02 15:08:49.499854 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26812 2017-03-02 15:08:49.499856 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26813 2017-03-02 15:08:49.499857 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26814 2017-03-02 15:08:49.499859 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26815 2017-03-02 15:08:49.499860 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26816 2017-03-02 15:08:49.499862 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26817 2017-03-02 15:08:49.499863 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26818 2017-03-02 15:08:49.499865 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26819 2017-03-02 15:08:49.499866 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26820 2017-03-02 15:08:49.499868 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26821 2017-03-02 15:08:49.499869 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26822 2017-03-02 15:08:49.499870 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26823 2017-03-02 15:08:49.499872 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26824 2017-03-02 15:08:49.499873 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26825 2017-03-02 15:08:49.499875 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26826 2017-03-02 15:08:49.499876 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26827 2017-03-02 15:08:49.499878 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26828 2017-03-02 15:08:49.499879 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26829 2017-03-02 15:08:49.499881 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26830 2017-03-02 15:08:49.499882 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26831 2017-03-02 15:08:49.499884 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26832 2017-03-02 15:08:49.499885 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26833 2017-03-02 15:08:49.499887 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26834 2017-03-02 15:08:49.499888 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26835 2017-03-02 15:08:49.499890 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26836 2017-03-02 15:08:49.499891 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26837 2017-03-02 15:08:49.499893 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26838 2017-03-02 15:08:49.499894 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26839 2017-03-02 15:08:49.499896 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26840 2017-03-02 15:08:49.499897 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26841 2017-03-02 15:08:49.499898 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26842 2017-03-02 15:08:49.499900 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26843 2017-03-02 15:08:49.499902 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26844 2017-03-02 15:08:49.499903 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26845 2017-03-02 15:08:49.499904 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26846 2017-03-02 15:08:49.499906 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26847 2017-03-02 15:08:49.499907 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26848 2017-03-02 15:08:49.499909 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26849 2017-03-02 15:08:49.499910 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26850 2017-03-02 15:08:49.499912 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26851 2017-03-02 15:08:49.499913 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26852 2017-03-02 15:08:49.499915 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26853 2017-03-02 15:08:49.499916 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26854 2017-03-02 15:08:49.499918 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26855 2017-03-02 15:08:49.499919 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26856 2017-03-02 15:08:49.499921 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) trim 26857 2017-03-02 15:08:49.499924 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) compacting trimmed range 2017-03-02 15:08:49.499927 7f4f763f4700 5 mon.0@0(leader).paxos(paxos active c 26607..27358) queue_pending_finisher 0x558a1412ed00 2017-03-02 15:08:49.499985 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26607..27358) propose_pending 27359 8398 bytes 2017-03-02 15:08:49.499997 7f4f763f4700 10 mon.0@0(leader).paxos(paxos updating c 26607..27358) begin for 27359 8398 bytes 2017-03-02 15:08:49.532518 7f4f763f4700 10 mon.0@0(leader).paxos(paxos updating c 26607..27358) commit_start 27359 2017-03-02 15:08:49.532675 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:49.532693 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:49.532696 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13560) dispatch 0x558a16743f80 log(1 entries from seq 14125 at 2017-03-02 15:08:49.499448) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:49.532706 7f4f71beb700 5 mon.0@0(leader).paxos(paxos writing c 26607..27358) is_readable = 1 - now=2017-03-02 15:08:49.532707 lease_expire=0.000000 has v0 lc 27358 2017-03-02 15:08:49.532713 7f4f71beb700 10 mon.0@0(leader).log v13560 preprocess_query log(1 entries from seq 14125 at 2017-03-02 15:08:49.499448) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:49.532719 7f4f71beb700 10 mon.0@0(leader).log v13560 preprocess_log log(1 entries from seq 14125 at 2017-03-02 15:08:49.499448) v1 from mon.0 2017-03-02 15:08:49.532724 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:49.532727 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:49.532728 7f4f71beb700 20 allow all 2017-03-02 15:08:49.532736 7f4f71beb700 10 mon.0@0(leader).log v13560 prepare_update log(1 entries from seq 14125 at 2017-03-02 15:08:49.499448) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:49.532742 7f4f71beb700 10 mon.0@0(leader).log v13560 prepare_log log(1 entries from seq 14125 at 2017-03-02 15:08:49.499448) v1 from mon.0 2017-03-02 15:08:49.532745 7f4f71beb700 10 mon.0@0(leader).log v13560 logging 2017-03-02 15:08:49.499448 mon.0 172.21.15.17:6789/0 14125 : cluster [INF] pgmap v12979: 32 pgs: 32 active+clean; 74914 MB data, 144 GB used, 227 GB / 372 GB avail; 0 B/s rd, 3293 kB/s wr, 769 op/s 2017-03-02 15:08:49.532754 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13560) setting proposal_timer 0x558a1412d6c0 with delay of 0.966354 2017-03-02 15:08:49.566128 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26607..27358) commit_finish 27359 2017-03-02 15:08:49.566456 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:49.566470 7f4f6fbe7700 1 leveldb: Level-0 table #1875: started 2017-03-02 15:08:49.566510 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12979) refresh 2017-03-02 15:08:49.566526 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) refresh 2017-03-02 15:08:49.566538 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:49.566551 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13560) refresh 2017-03-02 15:08:49.566559 7f4f763f4700 10 mon.0@0(leader).log v13560 update_from_paxos 2017-03-02 15:08:49.566561 7f4f763f4700 10 mon.0@0(leader).log v13560 update_from_paxos version 13560 summary v 13560 2017-03-02 15:08:49.566570 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:49.566581 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:49.566582 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:49.566592 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:49.566593 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12979) post_refresh 2017-03-02 15:08:49.566595 7f4f763f4700 10 mon.0@0(leader).pg v12979 post_paxos_update 2017-03-02 15:08:49.566597 7f4f763f4700 10 mon.0@0(leader).pg v12979 check_subs 2017-03-02 15:08:49.566599 7f4f763f4700 20 mon.0@0(leader).pg v12979 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:49.566605 7f4f763f4700 20 mon.0@0(leader).pg v12979 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:49.566607 7f4f763f4700 20 mon.0@0(leader).pg v12979 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:49.566609 7f4f763f4700 20 mon.0@0(leader).pg v12979 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:49.566611 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) post_refresh 2017-03-02 15:08:49.566612 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:49.566613 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13560) post_refresh 2017-03-02 15:08:49.566614 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:49.566615 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:49.566616 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:49.566618 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27359) commit_proposal 2017-03-02 15:08:49.566621 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27359) finish_round 2017-03-02 15:08:49.566622 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27359) finish_round waiting_for_acting 2017-03-02 15:08:49.566624 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27359) finish_round waiting_for_readable 2017-03-02 15:08:49.566625 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27359) finish_round waiting_for_writeable 2017-03-02 15:08:49.566626 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26858..27359) finish_round done w/ waiters, state active 2017-03-02 15:08:49.616593 7f4f6fbe7700 1 leveldb: Level-0 table #1875: 1055499 bytes OK 2017-03-02 15:08:49.627034 7f4f6fbe7700 1 leveldb: Delete type=0 #1872 2017-03-02 15:08:49.627745 7f4f6fbe7700 1 leveldb: Manual compaction at level-0 from 'paxos\x0026606' @ 72057594037927935 : 1 .. 'paxos\x0026858' @ 0 : 0; will stop at (end) 2017-03-02 15:08:49.627782 7f4f6fbe7700 1 leveldb: Manual compaction at level-1 from 'paxos\x0026606' @ 72057594037927935 : 1 .. 'paxos\x0026858' @ 0 : 0; will stop at 'pgmap_pg\x002.7' @ 458575 : 1 2017-03-02 15:08:49.627789 7f4f6fbe7700 1 leveldb: Compacting 1@1 + 6@2 files 2017-03-02 15:08:49.684068 7f4f6fbe7700 1 leveldb: Generated table #1876: 739 keys, 2099826 bytes 2017-03-02 15:08:49.750635 7f4f6fbe7700 1 leveldb: Generated table #1877: 210 keys, 2155159 bytes 2017-03-02 15:08:49.809104 7f4f6fbe7700 1 leveldb: Generated table #1878: 566 keys, 2146075 bytes 2017-03-02 15:08:49.867616 7f4f6fbe7700 1 leveldb: Generated table #1879: 539 keys, 2098188 bytes 2017-03-02 15:08:49.934690 7f4f6fbe7700 1 leveldb: Generated table #1880: 267 keys, 2144903 bytes 2017-03-02 15:08:49.976646 7f4f6fbe7700 1 leveldb: Generated table #1881: 809 keys, 533176 bytes 2017-03-02 15:08:49.976671 7f4f6fbe7700 1 leveldb: Compacted 1@1 + 6@2 files => 11177327 bytes 2017-03-02 15:08:49.985191 7f4f6fbe7700 1 leveldb: compacted to: files[ 0 0 6 0 0 0 0 ] 2017-03-02 15:08:49.985554 7f4f6fbe7700 1 leveldb: Delete type=2 #1867 2017-03-02 15:08:49.986759 7f4f6fbe7700 1 leveldb: Delete type=2 #1866 2017-03-02 15:08:49.987939 7f4f6fbe7700 1 leveldb: Delete type=2 #1869 2017-03-02 15:08:49.988985 7f4f6fbe7700 1 leveldb: Delete type=2 #1875 2017-03-02 15:08:49.989741 7f4f6fbe7700 1 leveldb: Delete type=2 #1871 2017-03-02 15:08:49.990756 7f4f6fbe7700 1 leveldb: Delete type=2 #1868 2017-03-02 15:08:49.992032 7f4f6fbe7700 1 leveldb: Delete type=2 #1870 2017-03-02 15:08:49.992917 7f4f6fbe7700 1 leveldb: Manual compaction at level-1 from 'pgmap_pg\x002.7' @ 458575 : 1 .. 'paxos\x0026858' @ 0 : 0; will stop at (end) 2017-03-02 15:08:50.499206 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13560) propose_pending 2017-03-02 15:08:50.499230 7f4f743f0700 10 mon.0@0(leader).log v13560 encode_full log v 13560 2017-03-02 15:08:50.499309 7f4f743f0700 10 mon.0@0(leader).log v13560 encode_pending v13561 2017-03-02 15:08:50.499318 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26858..27359) queue_pending_finisher 0x558a1412ce70 2017-03-02 15:08:50.499322 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26858..27359) trigger_propose active, proposing now 2017-03-02 15:08:50.499328 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26858..27359) propose_pending 27360 10270 bytes 2017-03-02 15:08:50.499331 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26858..27359) begin for 27360 10270 bytes 2017-03-02 15:08:50.542966 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26858..27359) commit_start 27360 2017-03-02 15:08:50.584758 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26858..27359) commit_finish 27360 2017-03-02 15:08:50.584813 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:50.584865 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12979) refresh 2017-03-02 15:08:50.584881 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) refresh 2017-03-02 15:08:50.584897 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:50.584919 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13561) refresh 2017-03-02 15:08:50.584924 7f4f763f4700 10 mon.0@0(leader).log v13561 update_from_paxos 2017-03-02 15:08:50.584926 7f4f763f4700 10 mon.0@0(leader).log v13561 update_from_paxos version 13561 summary v 13560 2017-03-02 15:08:50.584929 7f4f763f4700 10 mon.0@0(leader).log v13561 update_from_paxos latest full 13560 2017-03-02 15:08:50.584936 7f4f763f4700 7 mon.0@0(leader).log v13561 update_from_paxos applying incremental log 13561 2017-03-02 15:08:49.499448 mon.0 172.21.15.17:6789/0 14125 : cluster [INF] pgmap v12979: 32 pgs: 32 active+clean; 74914 MB data, 144 GB used, 227 GB / 372 GB avail; 0 B/s rd, 3293 kB/s wr, 769 op/s 2017-03-02 15:08:50.584947 7f4f763f4700 20 mon.0@0(leader).log v13561 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:50.584960 7f4f763f4700 15 mon.0@0(leader).log v13561 update_from_paxos logging for 1 channels 2017-03-02 15:08:50.584961 7f4f763f4700 15 mon.0@0(leader).log v13561 update_from_paxos channel 'cluster' logging 198 bytes 2017-03-02 15:08:50.585005 7f4f763f4700 10 mon.0@0(leader).log v13561 check_subs 2017-03-02 15:08:50.585030 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:50.585052 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:50.585059 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:50.585068 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:50.585070 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12979) post_refresh 2017-03-02 15:08:50.585071 7f4f763f4700 10 mon.0@0(leader).pg v12979 post_paxos_update 2017-03-02 15:08:50.585073 7f4f763f4700 10 mon.0@0(leader).pg v12979 check_subs 2017-03-02 15:08:50.585074 7f4f763f4700 20 mon.0@0(leader).pg v12979 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:50.585078 7f4f763f4700 20 mon.0@0(leader).pg v12979 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:50.585080 7f4f763f4700 20 mon.0@0(leader).pg v12979 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:50.585081 7f4f763f4700 20 mon.0@0(leader).pg v12979 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:50.585083 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) post_refresh 2017-03-02 15:08:50.585084 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:50.585085 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13561) post_refresh 2017-03-02 15:08:50.585085 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:50.585086 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:50.585087 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:50.585088 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27360) commit_proposal 2017-03-02 15:08:50.585091 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13561) _active - not active 2017-03-02 15:08:50.585092 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27360) finish_round 2017-03-02 15:08:50.585093 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27360) finish_round waiting_for_acting 2017-03-02 15:08:50.585094 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13561) _active 2017-03-02 15:08:50.585095 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13561) _active creating new pending 2017-03-02 15:08:50.585102 7f4f763f4700 10 mon.0@0(leader).log v13561 create_pending v 13562 2017-03-02 15:08:50.585104 7f4f763f4700 7 mon.0@0(leader).log v13561 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:50.585109 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a16405400 0x558a157fa1e0 log(last 14125) v1 2017-03-02 15:08:50.585112 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14125) v1 -- 0x558a157fa1e0 con 0 2017-03-02 15:08:50.585129 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27360) finish_round waiting_for_readable 2017-03-02 15:08:50.585131 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27360) finish_round waiting_for_writeable 2017-03-02 15:08:50.585133 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26858..27360) finish_round done w/ waiters, state active 2017-03-02 15:08:50.585152 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14125) v1 ==== 0+0+0 (0 0 0) 0x558a157fa1e0 con 0x558a14482000 2017-03-02 15:08:50.585166 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:50.585170 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:50.585172 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:50.585174 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:50.585175 7f4f71beb700 20 allow all 2017-03-02 15:08:50.870665 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.5 172.21.15.11:6804/1637448123 8791 ==== mdsbeacon(4116/e up:active seq 8504 v737) v7 ==== 365+0+0 (729894979 0 0) 0x558a15204d00 con 0x558a1471d800 2017-03-02 15:08:50.870701 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b0f20 for mds.? 172.21.15.11:6804/1637448123 2017-03-02 15:08:50.870709 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:50.870713 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a15204d00 mdsbeacon(4116/e up:active seq 8504 v737) v7 from mds.5 172.21.15.11:6804/1637448123 con 0x558a1471d800 2017-03-02 15:08:50.870721 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27360) is_readable = 1 - now=2017-03-02 15:08:50.870721 lease_expire=0.000000 has v0 lc 27360 2017-03-02 15:08:50.870730 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4116/e up:active seq 8504 v737) v7 from mds.5 172.21.15.11:6804/1637448123 2017-03-02 15:08:50.870745 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:50.870748 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:50.870750 7f4f71beb700 20 allow all 2017-03-02 15:08:50.870752 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4116/e up:active seq 8504 v737) v7 from mds.5 172.21.15.11:6804/1637448123 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:50.870762 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4116 2017-03-02 15:08:50.870769 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4116/e up:active seq 8504 v737) v7 noting time 2017-03-02 15:08:50.870774 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a1491e900 0x558a148d7380 mdsbeacon(4116/e up:active seq 8504 v737) v7 2017-03-02 15:08:50.870779 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6804/1637448123 -- mdsbeacon(4116/e up:active seq 8504 v737) v7 -- 0x558a148d7380 con 0 2017-03-02 15:08:50.884386 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.3 172.21.15.11:6805/1926402648 8567 ==== mdsbeacon(4118/f up:active seq 8504 v737) v7 ==== 406+0+0 (2045628568 0 0) 0x558a160c0a40 con 0x558a14905800 2017-03-02 15:08:50.884412 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b2260 for mds.? 172.21.15.11:6805/1926402648 2017-03-02 15:08:50.884417 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:50.884419 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a160c0a40 mdsbeacon(4118/f up:active seq 8504 v737) v7 from mds.3 172.21.15.11:6805/1926402648 con 0x558a14905800 2017-03-02 15:08:50.884423 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27360) is_readable = 1 - now=2017-03-02 15:08:50.884423 lease_expire=0.000000 has v0 lc 27360 2017-03-02 15:08:50.884429 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4118/f up:active seq 8504 v737) v7 from mds.3 172.21.15.11:6805/1926402648 2017-03-02 15:08:50.884433 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:50.884435 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:50.884436 7f4f71beb700 20 allow all 2017-03-02 15:08:50.884437 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4118/f up:active seq 8504 v737) v7 from mds.3 172.21.15.11:6805/1926402648 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:50.884442 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4118 2017-03-02 15:08:50.884446 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4118/f up:active seq 8504 v737) v7 noting time 2017-03-02 15:08:50.884449 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a1491e900 0x558a15204d00 mdsbeacon(4118/f up:active seq 8504 v737) v7 2017-03-02 15:08:50.884451 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6805/1926402648 -- mdsbeacon(4118/f up:active seq 8504 v737) v7 -- 0x558a15204d00 con 0 2017-03-02 15:08:50.885209 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.2 172.21.15.22:6804/673088617 8566 ==== mdsbeacon(4119/c up:active seq 8504 v737) v7 ==== 365+0+0 (194532381 0 0) 0x558a148d7380 con 0x558a14907000 2017-03-02 15:08:50.885231 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14916f20 for mds.? 172.21.15.22:6804/673088617 2017-03-02 15:08:50.885236 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:50.885238 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a148d7380 mdsbeacon(4119/c up:active seq 8504 v737) v7 from mds.2 172.21.15.22:6804/673088617 con 0x558a14907000 2017-03-02 15:08:50.885241 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27360) is_readable = 1 - now=2017-03-02 15:08:50.885241 lease_expire=0.000000 has v0 lc 27360 2017-03-02 15:08:50.885246 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4119/c up:active seq 8504 v737) v7 from mds.2 172.21.15.22:6804/673088617 2017-03-02 15:08:50.885250 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:50.885252 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:50.885252 7f4f71beb700 20 allow all 2017-03-02 15:08:50.885253 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4119/c up:active seq 8504 v737) v7 from mds.2 172.21.15.22:6804/673088617 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:50.885257 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4119 2017-03-02 15:08:50.885260 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4119/c up:active seq 8504 v737) v7 noting time 2017-03-02 15:08:50.885263 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a1491e900 0x558a160c0a40 mdsbeacon(4119/c up:active seq 8504 v737) v7 2017-03-02 15:08:50.885264 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6804/673088617 -- mdsbeacon(4119/c up:active seq 8504 v737) v7 -- 0x558a160c0a40 con 0 2017-03-02 15:08:50.924648 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.1 172.21.15.17:6804/2576824570 8563 ==== mdsbeacon(4120/g up:active seq 8504 v737) v7 ==== 365+0+0 (3281459462 0 0) 0x558a154b8d00 con 0x558a14908800 2017-03-02 15:08:50.924679 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14917600 for mds.? 172.21.15.17:6804/2576824570 2017-03-02 15:08:50.924684 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:50.924686 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a154b8d00 mdsbeacon(4120/g up:active seq 8504 v737) v7 from mds.1 172.21.15.17:6804/2576824570 con 0x558a14908800 2017-03-02 15:08:50.924690 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27360) is_readable = 1 - now=2017-03-02 15:08:50.924690 lease_expire=0.000000 has v0 lc 27360 2017-03-02 15:08:50.924696 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4120/g up:active seq 8504 v737) v7 from mds.1 172.21.15.17:6804/2576824570 2017-03-02 15:08:50.924701 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:50.924703 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:50.924703 7f4f71beb700 20 allow all 2017-03-02 15:08:50.924704 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4120/g up:active seq 8504 v737) v7 from mds.1 172.21.15.17:6804/2576824570 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:50.924710 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4120 2017-03-02 15:08:50.924714 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4120/g up:active seq 8504 v737) v7 noting time 2017-03-02 15:08:50.924718 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a1491e900 0x558a148d7380 mdsbeacon(4120/g up:active seq 8504 v737) v7 2017-03-02 15:08:50.924721 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6804/2576824570 -- mdsbeacon(4120/g up:active seq 8504 v737) v7 -- 0x558a148d7380 con 0 2017-03-02 15:08:50.970383 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.4 172.21.15.26:6805/2033388711 8561 ==== mdsbeacon(4117/b up:active seq 8504 v737) v7 ==== 365+0+0 (1027475397 0 0) 0x558a148d7380 con 0x558a14904000 2017-03-02 15:08:50.970415 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b1600 for mds.? 172.21.15.26:6805/2033388711 2017-03-02 15:08:50.970422 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:50.970424 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a148d7380 mdsbeacon(4117/b up:active seq 8504 v737) v7 from mds.4 172.21.15.26:6805/2033388711 con 0x558a14904000 2017-03-02 15:08:50.970429 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27360) is_readable = 1 - now=2017-03-02 15:08:50.970429 lease_expire=0.000000 has v0 lc 27360 2017-03-02 15:08:50.970436 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4117/b up:active seq 8504 v737) v7 from mds.4 172.21.15.26:6805/2033388711 2017-03-02 15:08:50.970443 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:50.970445 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:50.970446 7f4f71beb700 20 allow all 2017-03-02 15:08:50.970447 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4117/b up:active seq 8504 v737) v7 from mds.4 172.21.15.26:6805/2033388711 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:50.970452 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4117 2017-03-02 15:08:50.970456 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4117/b up:active seq 8504 v737) v7 noting time 2017-03-02 15:08:50.970459 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a1491e900 0x558a154b8d00 mdsbeacon(4117/b up:active seq 8504 v737) v7 2017-03-02 15:08:50.970461 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6805/2033388711 -- mdsbeacon(4117/b up:active seq 8504 v737) v7 -- 0x558a154b8d00 con 0 2017-03-02 15:08:50.979223 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.6 172.21.15.26:6804/2954782670 8754 ==== mdsbeacon(4115/a up:active seq 8504 v737) v7 ==== 365+0+0 (1725232045 0 0) 0x558a15204d00 con 0x558a1471f000 2017-03-02 15:08:50.979250 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14868d60 for mds.? 172.21.15.26:6804/2954782670 2017-03-02 15:08:50.979257 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:50.979260 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a15204d00 mdsbeacon(4115/a up:active seq 8504 v737) v7 from mds.6 172.21.15.26:6804/2954782670 con 0x558a1471f000 2017-03-02 15:08:50.979265 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27360) is_readable = 1 - now=2017-03-02 15:08:50.979266 lease_expire=0.000000 has v0 lc 27360 2017-03-02 15:08:50.979271 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4115/a up:active seq 8504 v737) v7 from mds.6 172.21.15.26:6804/2954782670 2017-03-02 15:08:50.979275 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:50.979279 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:50.979280 7f4f71beb700 20 allow all 2017-03-02 15:08:50.979281 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4115/a up:active seq 8504 v737) v7 from mds.6 172.21.15.26:6804/2954782670 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:50.979285 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4115 2017-03-02 15:08:50.979289 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4115/a up:active seq 8504 v737) v7 noting time 2017-03-02 15:08:50.979291 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a1491e900 0x558a148d7380 mdsbeacon(4115/a up:active seq 8504 v737) v7 2017-03-02 15:08:50.979293 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6804/2954782670 -- mdsbeacon(4115/a up:active seq 8504 v737) v7 -- 0x558a148d7380 con 0 2017-03-02 15:08:51.012735 7f4f743f0700 11 mon.0@0(leader) e2 tick 2017-03-02 15:08:51.012753 7f4f743f0700 10 mon.0@0(leader).pg v12979 v12979: 32 pgs: 32 active+clean; 74914 MB data, 144 GB used, 227 GB / 372 GB avail; 0 B/s rd, 3293 kB/s wr, 769 op/s 2017-03-02 15:08:51.012771 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12979) maybe_trim trim_to 12479 would only trim 210 < paxos_service_trim_min 250 2017-03-02 15:08:51.012775 7f4f743f0700 10 mon.0@0(leader).mds e737 e737: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:51.012807 7f4f743f0700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) maybe_trim trim_to 237 would only trim 236 < paxos_service_trim_min 250 2017-03-02 15:08:51.012811 7f4f743f0700 10 mon.0@0(leader).osd e7 e7: 4 osds: 4 up, 4 in 2017-03-02 15:08:51.012815 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26858..27360) is_readable = 1 - now=2017-03-02 15:08:51.012816 lease_expire=0.000000 has v0 lc 27360 2017-03-02 15:08:51.012837 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26858..27360) is_readable = 1 - now=2017-03-02 15:08:51.012838 lease_expire=0.000000 has v0 lc 27360 2017-03-02 15:08:51.012844 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26858..27360) is_readable = 1 - now=2017-03-02 15:08:51.012845 lease_expire=0.000000 has v0 lc 27360 2017-03-02 15:08:51.012848 7f4f743f0700 10 mon.0@0(leader).osd e7 min_last_epoch_clean 6 2017-03-02 15:08:51.012851 7f4f743f0700 10 mon.0@0(leader).log v13561 log 2017-03-02 15:08:51.012853 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13561) maybe_trim trim_to 13061 would only trim 34 < paxos_service_trim_min 250 2017-03-02 15:08:51.012856 7f4f743f0700 10 mon.0@0(leader).auth v10 auth 2017-03-02 15:08:51.012860 7f4f743f0700 10 cephx keyserver: _check_rotating_secrets 2017-03-02 15:08:51.012868 7f4f743f0700 20 mon.0@0(leader) e2 sync_trim_providers 2017-03-02 15:08:52.507648 7f4f743f0700 10 mon.0@0(leader).data_health(4) service_tick 2017-03-02 15:08:52.507684 7f4f743f0700 0 mon.0@0(leader).data_health(4) update_stats avail 93% total 916 GB, used 9634 MB, avail 860 GB 2017-03-02 15:08:52.507814 7f4f743f0700 10 mon.0@0(leader).data_health(4) share_stats 2017-03-02 15:08:53.265917 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.3 172.21.15.17:6800/27100 6839 ==== pg_stats(1 pgs tid 6803 v 0) v1 ==== 702+0+0 (2276688686 0 0) 0x558a15f26580 con 0x558a145d7000 2017-03-02 15:08:53.265959 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415f440 for osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:53.265969 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:53.265974 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12979) dispatch 0x558a15f26580 pg_stats(1 pgs tid 6803 v 0) v1 from osd.3 172.21.15.17:6800/27100 con 0x558a145d7000 2017-03-02 15:08:53.265984 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27360) is_readable = 1 - now=2017-03-02 15:08:53.265985 lease_expire=0.000000 has v0 lc 27360 2017-03-02 15:08:53.265996 7f4f71beb700 10 mon.0@0(leader).pg v12979 preprocess_query pg_stats(1 pgs tid 6803 v 0) v1 from osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:53.266004 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:53.266007 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:53.266008 7f4f71beb700 20 allow all 2017-03-02 15:08:53.266010 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27360) is_readable = 1 - now=2017-03-02 15:08:53.266010 lease_expire=0.000000 has v0 lc 27360 2017-03-02 15:08:53.266015 7f4f71beb700 10 mon.0@0(leader).pg v12979 prepare_update pg_stats(1 pgs tid 6803 v 0) v1 from osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:53.266020 7f4f71beb700 10 mon.0@0(leader).pg v12979 prepare_pg_stats pg_stats(1 pgs tid 6803 v 0) v1 from osd.3 2017-03-02 15:08:53.266027 7f4f71beb700 10 mon.0@0(leader).pg v12979 got osd.3 osd_stat(35388 MB used, 59953 MB avail, 95341 MB total, peers [0,1,2] op hist [1]) (was osd_stat(36360 MB used, 58981 MB avail, 95341 MB total, peers [0,1,2] op hist [1,1])) 2017-03-02 15:08:53.266037 7f4f71beb700 15 mon.0@0(leader).pg v12979 got 2.6 reported at 7:965600 state active+clean -> active+clean 2017-03-02 15:08:53.266047 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12979) setting proposal_timer 0x558a1412e230 with delay of 0.05 2017-03-02 15:08:53.316143 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12979) propose_pending 2017-03-02 15:08:53.316160 7f4f743f0700 10 mon.0@0(leader).pg v12979 encode_pending v 12980 2017-03-02 15:08:53.316194 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26858..27360) queue_pending_finisher 0x558a1412e740 2017-03-02 15:08:53.316203 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26858..27360) trigger_propose active, proposing now 2017-03-02 15:08:53.316212 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26858..27360) propose_pending 27361 1094 bytes 2017-03-02 15:08:53.316215 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26858..27360) begin for 27361 1094 bytes 2017-03-02 15:08:53.325631 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26858..27360) commit_start 27361 2017-03-02 15:08:53.333943 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26858..27360) commit_finish 27361 2017-03-02 15:08:53.333982 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:53.334004 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12980) refresh 2017-03-02 15:08:53.334007 7f4f763f4700 10 mon.0@0(leader).pg v12979 update_from_paxos read_incremental 2017-03-02 15:08:53.334032 7f4f763f4700 20 mon.0@0(leader).pg v12979 refreshing pg 2.6 7:965600 active+clean 2017-03-02 15:08:53.334046 7f4f763f4700 20 mon.0@0(leader).pg v12979 refreshing osd.3 2017-03-02 15:08:53.334062 7f4f763f4700 10 mon.0@0(leader).pg v12980 read_pgmap_meta 2017-03-02 15:08:53.334082 7f4f763f4700 10 mon.0@0(leader).pg v12980 update_logger 2017-03-02 15:08:53.334107 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) refresh 2017-03-02 15:08:53.334130 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:53.334148 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13561) refresh 2017-03-02 15:08:53.334156 7f4f763f4700 10 mon.0@0(leader).log v13561 update_from_paxos 2017-03-02 15:08:53.334158 7f4f763f4700 10 mon.0@0(leader).log v13561 update_from_paxos version 13561 summary v 13561 2017-03-02 15:08:53.334174 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:53.334188 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:53.334190 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:53.334204 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:53.334207 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12980) post_refresh 2017-03-02 15:08:53.334208 7f4f763f4700 10 mon.0@0(leader).pg v12980 post_paxos_update 2017-03-02 15:08:53.334210 7f4f763f4700 10 mon.0@0(leader).pg v12980 check_subs 2017-03-02 15:08:53.334213 7f4f763f4700 20 mon.0@0(leader).pg v12980 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:53.334220 7f4f763f4700 20 mon.0@0(leader).pg v12980 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:53.334223 7f4f763f4700 20 mon.0@0(leader).pg v12980 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:53.334226 7f4f763f4700 20 mon.0@0(leader).pg v12980 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:53.334229 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) post_refresh 2017-03-02 15:08:53.334231 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:53.334232 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13561) post_refresh 2017-03-02 15:08:53.334234 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:53.334235 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:53.334237 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:53.334239 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27361) commit_proposal 2017-03-02 15:08:53.334242 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12980) _active - not active 2017-03-02 15:08:53.334244 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27361) finish_round 2017-03-02 15:08:53.334246 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27361) finish_round waiting_for_acting 2017-03-02 15:08:53.334248 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12980) _active 2017-03-02 15:08:53.334249 7f4f763f4700 7 mon.0@0(leader).paxosservice(pgmap 12269..12980) _active creating new pending 2017-03-02 15:08:53.334253 7f4f763f4700 10 mon.0@0(leader).pg v12980 create_pending v 12981 2017-03-02 15:08:53.334259 7f4f763f4700 7 mon.0@0(leader).pg v12980 _updated_stats for osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:53.334263 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a1491e900 0x558a150e0800 pg_stats_ack(1 pgs tid 6803) v1 2017-03-02 15:08:53.334267 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6800/27100 -- pg_stats_ack(1 pgs tid 6803) v1 -- 0x558a150e0800 con 0 2017-03-02 15:08:53.334300 7f4f763f4700 10 mon.0@0(leader).pg v12980 check_osd_map already seen 7 >= 7 2017-03-02 15:08:53.334304 7f4f763f4700 10 mon.0@0(leader).pg v12980 update_logger 2017-03-02 15:08:53.334320 7f4f763f4700 0 log_channel(cluster) log [INF] : pgmap v12980: 32 pgs: 32 active+clean; 74521 MB data, 143 GB used, 228 GB / 372 GB avail; 0 B/s rd, 3358 kB/s wr, 774 op/s 2017-03-02 15:08:53.334335 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14126 at 2017-03-02 15:08:53.334322) v1 -- 0x558a16745600 con 0 2017-03-02 15:08:53.334349 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27361) finish_round waiting_for_readable 2017-03-02 15:08:53.334351 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27361) finish_round waiting_for_writeable 2017-03-02 15:08:53.334353 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26858..27361) finish_round done w/ waiters, state active 2017-03-02 15:08:53.334380 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14126 at 2017-03-02 15:08:53.334322) v1 ==== 0+0+0 (0 0 0) 0x558a16745600 con 0x558a14482000 2017-03-02 15:08:53.334402 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:53.334406 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:53.334409 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13561) dispatch 0x558a16745600 log(1 entries from seq 14126 at 2017-03-02 15:08:53.334322) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:53.334415 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27361) is_readable = 1 - now=2017-03-02 15:08:53.334416 lease_expire=0.000000 has v0 lc 27361 2017-03-02 15:08:53.334422 7f4f71beb700 10 mon.0@0(leader).log v13561 preprocess_query log(1 entries from seq 14126 at 2017-03-02 15:08:53.334322) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:53.334427 7f4f71beb700 10 mon.0@0(leader).log v13561 preprocess_log log(1 entries from seq 14126 at 2017-03-02 15:08:53.334322) v1 from mon.0 2017-03-02 15:08:53.334432 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:53.334434 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:53.334435 7f4f71beb700 20 allow all 2017-03-02 15:08:53.334442 7f4f71beb700 10 mon.0@0(leader).log v13561 prepare_update log(1 entries from seq 14126 at 2017-03-02 15:08:53.334322) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:53.334447 7f4f71beb700 10 mon.0@0(leader).log v13561 prepare_log log(1 entries from seq 14126 at 2017-03-02 15:08:53.334322) v1 from mon.0 2017-03-02 15:08:53.334450 7f4f71beb700 10 mon.0@0(leader).log v13561 logging 2017-03-02 15:08:53.334322 mon.0 172.21.15.17:6789/0 14126 : cluster [INF] pgmap v12980: 32 pgs: 32 active+clean; 74521 MB data, 143 GB used, 228 GB / 372 GB avail; 0 B/s rd, 3358 kB/s wr, 774 op/s 2017-03-02 15:08:53.334457 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13561) setting proposal_timer 0x558a1412f860 with delay of 0.999501 2017-03-02 15:08:53.467437 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.1 172.21.15.22:6800/26531 6839 ==== pg_stats(5 pgs tid 6803 v 0) v1 ==== 2974+0+0 (1620746903 0 0) 0x558a15ae4800 con 0x558a14643800 2017-03-02 15:08:53.467468 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1463b1e0 for osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:53.467473 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:53.467476 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12980) dispatch 0x558a15ae4800 pg_stats(5 pgs tid 6803 v 0) v1 from osd.1 172.21.15.22:6800/26531 con 0x558a14643800 2017-03-02 15:08:53.467481 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27361) is_readable = 1 - now=2017-03-02 15:08:53.467481 lease_expire=0.000000 has v0 lc 27361 2017-03-02 15:08:53.467488 7f4f71beb700 10 mon.0@0(leader).pg v12980 preprocess_query pg_stats(5 pgs tid 6803 v 0) v1 from osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:53.467494 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:53.467496 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:53.467497 7f4f71beb700 20 allow all 2017-03-02 15:08:53.467499 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27361) is_readable = 1 - now=2017-03-02 15:08:53.467499 lease_expire=0.000000 has v0 lc 27361 2017-03-02 15:08:53.467502 7f4f71beb700 10 mon.0@0(leader).pg v12980 prepare_update pg_stats(5 pgs tid 6803 v 0) v1 from osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:53.467505 7f4f71beb700 10 mon.0@0(leader).pg v12980 prepare_pg_stats pg_stats(5 pgs tid 6803 v 0) v1 from osd.1 2017-03-02 15:08:53.467512 7f4f71beb700 10 mon.0@0(leader).pg v12980 got osd.1 osd_stat(44321 MB used, 51020 MB avail, 95341 MB total, peers [0,2,3] op hist [0,1,2]) (was osd_stat(46745 MB used, 48596 MB avail, 95341 MB total, peers [0,2,3] op hist [0,2])) 2017-03-02 15:08:53.467520 7f4f71beb700 15 mon.0@0(leader).pg v12980 got 1.1 reported at 7:587276 state active+clean -> active+clean 2017-03-02 15:08:53.467528 7f4f71beb700 15 mon.0@0(leader).pg v12980 got 1.5 reported at 7:568141 state active+clean -> active+clean 2017-03-02 15:08:53.467532 7f4f71beb700 15 mon.0@0(leader).pg v12980 got 1.6 reported at 7:503542 state active+clean -> active+clean 2017-03-02 15:08:53.467535 7f4f71beb700 15 mon.0@0(leader).pg v12980 got 2.4 reported at 7:1027409 state active+clean -> active+clean 2017-03-02 15:08:53.467538 7f4f71beb700 15 mon.0@0(leader).pg v12980 got 2.5 reported at 7:1130773 state active+clean -> active+clean 2017-03-02 15:08:53.467542 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12980) setting proposal_timer 0x558a1412ee20 with delay of 0.866416 2017-03-02 15:08:53.561131 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.2 172.21.15.11:6800/26536 6839 ==== pg_stats(8 pgs tid 6803 v 0) v1 ==== 4668+0+0 (1274434369 0 0) 0x558a15da2d00 con 0x558a145d5800 2017-03-02 15:08:53.561164 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a145c3700 for osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:53.561169 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:53.561171 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12980) dispatch 0x558a15da2d00 pg_stats(8 pgs tid 6803 v 0) v1 from osd.2 172.21.15.11:6800/26536 con 0x558a145d5800 2017-03-02 15:08:53.561177 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27361) is_readable = 1 - now=2017-03-02 15:08:53.561177 lease_expire=0.000000 has v0 lc 27361 2017-03-02 15:08:53.561184 7f4f71beb700 10 mon.0@0(leader).pg v12980 preprocess_query pg_stats(8 pgs tid 6803 v 0) v1 from osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:53.561189 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:53.561192 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:53.561192 7f4f71beb700 20 allow all 2017-03-02 15:08:53.561194 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27361) is_readable = 1 - now=2017-03-02 15:08:53.561194 lease_expire=0.000000 has v0 lc 27361 2017-03-02 15:08:53.561197 7f4f71beb700 10 mon.0@0(leader).pg v12980 prepare_update pg_stats(8 pgs tid 6803 v 0) v1 from osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:53.561200 7f4f71beb700 10 mon.0@0(leader).pg v12980 prepare_pg_stats pg_stats(8 pgs tid 6803 v 0) v1 from osd.2 2017-03-02 15:08:53.561206 7f4f71beb700 10 mon.0@0(leader).pg v12980 got osd.2 osd_stat(53438 MB used, 41902 MB avail, 95341 MB total, peers [0,1,3] op hist [1,2]) (was osd_stat(55482 MB used, 39858 MB avail, 95341 MB total, peers [0,1,3] op hist [0,2])) 2017-03-02 15:08:53.561214 7f4f71beb700 15 mon.0@0(leader).pg v12980 got 1.0 reported at 7:496988 state active+clean -> active+clean 2017-03-02 15:08:53.561222 7f4f71beb700 15 mon.0@0(leader).pg v12980 got 1.2 reported at 7:577858 state active+clean -> active+clean 2017-03-02 15:08:53.561226 7f4f71beb700 15 mon.0@0(leader).pg v12980 got 1.7 reported at 7:530128 state active+clean -> active+clean 2017-03-02 15:08:53.561229 7f4f71beb700 15 mon.0@0(leader).pg v12980 got 2.0 reported at 7:1058019 state active+clean -> active+clean 2017-03-02 15:08:53.561232 7f4f71beb700 15 mon.0@0(leader).pg v12980 got 2.1 reported at 7:1090768 state active+clean -> active+clean 2017-03-02 15:08:53.561235 7f4f71beb700 15 mon.0@0(leader).pg v12980 got 2.2 reported at 7:1164499 state active+clean -> active+clean 2017-03-02 15:08:53.561239 7f4f71beb700 15 mon.0@0(leader).pg v12980 got 2.3 reported at 7:1096541 state active+clean -> active+clean 2017-03-02 15:08:53.561242 7f4f71beb700 15 mon.0@0(leader).pg v12980 got 2.7 reported at 7:1136684 state active+clean -> active+clean 2017-03-02 15:08:53.561245 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12980) proposal_timer already set 2017-03-02 15:08:53.717673 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.0 172.21.15.26:6800/26515 6839 ==== pg_stats(2 pgs tid 6803 v 0) v1 ==== 1268+0+0 (1127013678 0 0) 0x558a15aea080 con 0x558a145d8800 2017-03-02 15:08:53.717732 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1463af20 for osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:53.717745 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:53.717752 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12980) dispatch 0x558a15aea080 pg_stats(2 pgs tid 6803 v 0) v1 from osd.0 172.21.15.26:6800/26515 con 0x558a145d8800 2017-03-02 15:08:53.717765 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27361) is_readable = 1 - now=2017-03-02 15:08:53.717766 lease_expire=0.000000 has v0 lc 27361 2017-03-02 15:08:53.717783 7f4f71beb700 10 mon.0@0(leader).pg v12980 preprocess_query pg_stats(2 pgs tid 6803 v 0) v1 from osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:53.717797 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:53.717803 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:53.717806 7f4f71beb700 20 allow all 2017-03-02 15:08:53.717809 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27361) is_readable = 1 - now=2017-03-02 15:08:53.717810 lease_expire=0.000000 has v0 lc 27361 2017-03-02 15:08:53.717872 7f4f71beb700 10 mon.0@0(leader).pg v12980 prepare_update pg_stats(2 pgs tid 6803 v 0) v1 from osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:53.717888 7f4f71beb700 10 mon.0@0(leader).pg v12980 prepare_pg_stats pg_stats(2 pgs tid 6803 v 0) v1 from osd.0 2017-03-02 15:08:53.717902 7f4f71beb700 10 mon.0@0(leader).pg v12980 got osd.0 osd_stat(9114 MB used, 86226 MB avail, 95341 MB total, peers [1,2,3] op hist [1]) (was osd_stat(9578 MB used, 85763 MB avail, 95341 MB total, peers [1,2,3] op hist [])) 2017-03-02 15:08:53.717923 7f4f71beb700 15 mon.0@0(leader).pg v12980 got 1.3 reported at 7:500571 state active+clean -> active+clean 2017-03-02 15:08:53.717944 7f4f71beb700 15 mon.0@0(leader).pg v12980 got 1.4 reported at 7:486509 state active+clean -> active+clean 2017-03-02 15:08:53.717959 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12980) proposal_timer already set 2017-03-02 15:08:54.334123 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12980) propose_pending 2017-03-02 15:08:54.334155 7f4f743f0700 10 mon.0@0(leader).pg v12980 encode_pending v 12981 2017-03-02 15:08:54.334295 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26858..27361) queue_pending_finisher 0x558a1412e230 2017-03-02 15:08:54.334310 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26858..27361) trigger_propose active, proposing now 2017-03-02 15:08:54.334339 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26858..27361) propose_pending 27362 9760 bytes 2017-03-02 15:08:54.334343 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26858..27361) begin for 27362 9760 bytes 2017-03-02 15:08:54.367905 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26858..27361) commit_start 27362 2017-03-02 15:08:54.367991 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13561) propose_pending 2017-03-02 15:08:54.367998 7f4f743f0700 10 mon.0@0(leader).log v13561 encode_full log v 13561 2017-03-02 15:08:54.368099 7f4f743f0700 10 mon.0@0(leader).log v13561 encode_pending v13562 2017-03-02 15:08:54.368116 7f4f743f0700 5 mon.0@0(leader).paxos(paxos writing c 26858..27361) queue_pending_finisher 0x558a1412ee20 2017-03-02 15:08:54.368120 7f4f743f0700 10 mon.0@0(leader).paxos(paxos writing c 26858..27361) trigger_propose not active, will propose later 2017-03-02 15:08:54.418042 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26858..27361) commit_finish 27362 2017-03-02 15:08:54.418115 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:54.418145 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12981) refresh 2017-03-02 15:08:54.418159 7f4f763f4700 10 mon.0@0(leader).pg v12980 update_from_paxos read_incremental 2017-03-02 15:08:54.418196 7f4f763f4700 20 mon.0@0(leader).pg v12980 refreshing pg 1.0 7:496988 active+clean 2017-03-02 15:08:54.418223 7f4f763f4700 20 mon.0@0(leader).pg v12980 refreshing pg 1.1 7:587276 active+clean 2017-03-02 15:08:54.418246 7f4f763f4700 20 mon.0@0(leader).pg v12980 refreshing pg 1.2 7:577858 active+clean 2017-03-02 15:08:54.418260 7f4f763f4700 20 mon.0@0(leader).pg v12980 refreshing pg 1.3 7:500571 active+clean 2017-03-02 15:08:54.418271 7f4f763f4700 20 mon.0@0(leader).pg v12980 refreshing pg 1.4 7:486509 active+clean 2017-03-02 15:08:54.418295 7f4f763f4700 20 mon.0@0(leader).pg v12980 refreshing pg 1.5 7:568141 active+clean 2017-03-02 15:08:54.418309 7f4f763f4700 20 mon.0@0(leader).pg v12980 refreshing pg 1.6 7:503542 active+clean 2017-03-02 15:08:54.418336 7f4f763f4700 20 mon.0@0(leader).pg v12980 refreshing pg 1.7 7:530128 active+clean 2017-03-02 15:08:54.418359 7f4f763f4700 20 mon.0@0(leader).pg v12980 refreshing pg 2.0 7:1058019 active+clean 2017-03-02 15:08:54.418382 7f4f763f4700 20 mon.0@0(leader).pg v12980 refreshing pg 2.1 7:1090768 active+clean 2017-03-02 15:08:54.418396 7f4f763f4700 20 mon.0@0(leader).pg v12980 refreshing pg 2.2 7:1164499 active+clean 2017-03-02 15:08:54.418409 7f4f763f4700 20 mon.0@0(leader).pg v12980 refreshing pg 2.3 7:1096541 active+clean 2017-03-02 15:08:54.418421 7f4f763f4700 20 mon.0@0(leader).pg v12980 refreshing pg 2.4 7:1027409 active+clean 2017-03-02 15:08:54.418434 7f4f763f4700 20 mon.0@0(leader).pg v12980 refreshing pg 2.5 7:1130773 active+clean 2017-03-02 15:08:54.418446 7f4f763f4700 20 mon.0@0(leader).pg v12980 refreshing pg 2.7 7:1136684 active+clean 2017-03-02 15:08:54.418452 7f4f763f4700 20 mon.0@0(leader).pg v12980 refreshing osd.0 2017-03-02 15:08:54.418465 7f4f763f4700 20 mon.0@0(leader).pg v12980 refreshing osd.1 2017-03-02 15:08:54.418474 7f4f763f4700 20 mon.0@0(leader).pg v12980 refreshing osd.2 2017-03-02 15:08:54.418492 7f4f763f4700 10 mon.0@0(leader).pg v12981 read_pgmap_meta 2017-03-02 15:08:54.418528 7f4f763f4700 10 mon.0@0(leader).pg v12981 update_logger 2017-03-02 15:08:54.418560 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) refresh 2017-03-02 15:08:54.418585 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:54.418620 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13561) refresh 2017-03-02 15:08:54.418624 7f4f763f4700 10 mon.0@0(leader).log v13561 update_from_paxos 2017-03-02 15:08:54.418628 7f4f763f4700 10 mon.0@0(leader).log v13561 update_from_paxos version 13561 summary v 13561 2017-03-02 15:08:54.418650 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:54.418672 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:54.418683 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:54.418708 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:54.418719 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12981) post_refresh 2017-03-02 15:08:54.418722 7f4f763f4700 10 mon.0@0(leader).pg v12981 post_paxos_update 2017-03-02 15:08:54.418725 7f4f763f4700 10 mon.0@0(leader).pg v12981 check_subs 2017-03-02 15:08:54.418729 7f4f763f4700 20 mon.0@0(leader).pg v12981 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:54.418740 7f4f763f4700 20 mon.0@0(leader).pg v12981 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:54.418745 7f4f763f4700 20 mon.0@0(leader).pg v12981 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:54.418749 7f4f763f4700 20 mon.0@0(leader).pg v12981 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:54.418754 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) post_refresh 2017-03-02 15:08:54.418756 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:54.418759 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13561) post_refresh 2017-03-02 15:08:54.418761 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:54.418763 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:54.418765 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:54.418768 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27362) commit_proposal 2017-03-02 15:08:54.418774 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12981) _active - not active 2017-03-02 15:08:54.418777 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27362) finish_round 2017-03-02 15:08:54.418780 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27362) finish_round waiting_for_acting 2017-03-02 15:08:54.418784 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12981) _active 2017-03-02 15:08:54.418786 7f4f763f4700 7 mon.0@0(leader).paxosservice(pgmap 12269..12981) _active creating new pending 2017-03-02 15:08:54.418798 7f4f763f4700 10 mon.0@0(leader).pg v12981 create_pending v 12982 2017-03-02 15:08:54.418806 7f4f763f4700 7 mon.0@0(leader).pg v12981 _updated_stats for osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:54.418813 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a1451f000 0x558a14f22c00 pg_stats_ack(5 pgs tid 6803) v1 2017-03-02 15:08:54.418855 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6800/26531 -- pg_stats_ack(5 pgs tid 6803) v1 -- 0x558a14f22c00 con 0 2017-03-02 15:08:54.418924 7f4f763f4700 7 mon.0@0(leader).pg v12981 _updated_stats for osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:54.418931 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a1621d700 0x558a15058600 pg_stats_ack(8 pgs tid 6803) v1 2017-03-02 15:08:54.418936 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6800/26536 -- pg_stats_ack(8 pgs tid 6803) v1 -- 0x558a15058600 con 0 2017-03-02 15:08:54.418965 7f4f763f4700 7 mon.0@0(leader).pg v12981 _updated_stats for osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:54.418971 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a16477600 0x558a14f24400 pg_stats_ack(2 pgs tid 6803) v1 2017-03-02 15:08:54.418974 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6800/26515 -- pg_stats_ack(2 pgs tid 6803) v1 -- 0x558a14f24400 con 0 2017-03-02 15:08:54.419008 7f4f763f4700 10 mon.0@0(leader).pg v12981 check_osd_map already seen 7 >= 7 2017-03-02 15:08:54.419013 7f4f763f4700 10 mon.0@0(leader).pg v12981 update_logger 2017-03-02 15:08:54.419037 7f4f763f4700 0 log_channel(cluster) log [INF] : pgmap v12981: 32 pgs: 32 active+clean; 71853 MB data, 138 GB used, 233 GB / 372 GB avail; 0 B/s rd, 3417 kB/s wr, 678 op/s 2017-03-02 15:08:54.419060 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14127 at 2017-03-02 15:08:54.419040) v1 -- 0x558a167453c0 con 0 2017-03-02 15:08:54.419093 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27362) finish_round waiting_for_readable 2017-03-02 15:08:54.419103 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27362) finish_round waiting_for_writeable 2017-03-02 15:08:54.419107 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26858..27362) finish_round done w/ waiters, state active 2017-03-02 15:08:54.419121 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26858..27362) propose_pending 27363 10265 bytes 2017-03-02 15:08:54.419127 7f4f763f4700 10 mon.0@0(leader).paxos(paxos updating c 26858..27362) begin for 27363 10265 bytes 2017-03-02 15:08:54.419167 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14127 at 2017-03-02 15:08:54.419040) v1 ==== 0+0+0 (0 0 0) 0x558a167453c0 con 0x558a14482000 2017-03-02 15:08:54.451511 7f4f763f4700 10 mon.0@0(leader).paxos(paxos updating c 26858..27362) commit_start 27363 2017-03-02 15:08:54.451565 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:54.451579 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:54.451583 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13561) dispatch 0x558a167453c0 log(1 entries from seq 14127 at 2017-03-02 15:08:54.419040) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:54.451595 7f4f71beb700 5 mon.0@0(leader).paxos(paxos writing c 26858..27362) is_readable = 1 - now=2017-03-02 15:08:54.451596 lease_expire=0.000000 has v0 lc 27362 2017-03-02 15:08:54.451605 7f4f71beb700 10 mon.0@0(leader).log v13561 preprocess_query log(1 entries from seq 14127 at 2017-03-02 15:08:54.419040) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:54.451614 7f4f71beb700 10 mon.0@0(leader).log v13561 preprocess_log log(1 entries from seq 14127 at 2017-03-02 15:08:54.419040) v1 from mon.0 2017-03-02 15:08:54.451620 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:54.451624 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:54.451626 7f4f71beb700 20 allow all 2017-03-02 15:08:54.451635 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13561) waiting for paxos -> writeable 2017-03-02 15:08:54.518391 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26858..27362) commit_finish 27363 2017-03-02 15:08:54.518442 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:54.518463 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12981) refresh 2017-03-02 15:08:54.518481 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) refresh 2017-03-02 15:08:54.518495 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:54.518510 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13562) refresh 2017-03-02 15:08:54.518511 7f4f763f4700 10 mon.0@0(leader).log v13562 update_from_paxos 2017-03-02 15:08:54.518514 7f4f763f4700 10 mon.0@0(leader).log v13562 update_from_paxos version 13562 summary v 13561 2017-03-02 15:08:54.518518 7f4f763f4700 10 mon.0@0(leader).log v13562 update_from_paxos latest full 13561 2017-03-02 15:08:54.518527 7f4f763f4700 7 mon.0@0(leader).log v13562 update_from_paxos applying incremental log 13562 2017-03-02 15:08:53.334322 mon.0 172.21.15.17:6789/0 14126 : cluster [INF] pgmap v12980: 32 pgs: 32 active+clean; 74521 MB data, 143 GB used, 228 GB / 372 GB avail; 0 B/s rd, 3358 kB/s wr, 774 op/s 2017-03-02 15:08:54.518543 7f4f763f4700 20 mon.0@0(leader).log v13562 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:54.518562 7f4f763f4700 15 mon.0@0(leader).log v13562 update_from_paxos logging for 1 channels 2017-03-02 15:08:54.518564 7f4f763f4700 15 mon.0@0(leader).log v13562 update_from_paxos channel 'cluster' logging 198 bytes 2017-03-02 15:08:54.518662 7f4f763f4700 10 mon.0@0(leader).log v13562 check_subs 2017-03-02 15:08:54.518704 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:54.518721 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:54.518724 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:54.518740 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:54.518743 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12981) post_refresh 2017-03-02 15:08:54.518745 7f4f763f4700 10 mon.0@0(leader).pg v12981 post_paxos_update 2017-03-02 15:08:54.518747 7f4f763f4700 10 mon.0@0(leader).pg v12981 check_subs 2017-03-02 15:08:54.518750 7f4f763f4700 20 mon.0@0(leader).pg v12981 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:54.518756 7f4f763f4700 20 mon.0@0(leader).pg v12981 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:54.518758 7f4f763f4700 20 mon.0@0(leader).pg v12981 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:54.518766 7f4f763f4700 20 mon.0@0(leader).pg v12981 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:54.518769 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) post_refresh 2017-03-02 15:08:54.518770 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:54.518772 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13562) post_refresh 2017-03-02 15:08:54.518773 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:54.518775 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:54.518776 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:54.518778 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27363) commit_proposal 2017-03-02 15:08:54.518783 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13562) _active - not active 2017-03-02 15:08:54.518785 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27363) finish_round 2017-03-02 15:08:54.518787 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27363) finish_round waiting_for_acting 2017-03-02 15:08:54.518790 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13562) _active 2017-03-02 15:08:54.518791 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13562) _active creating new pending 2017-03-02 15:08:54.518799 7f4f763f4700 10 mon.0@0(leader).log v13562 create_pending v 13563 2017-03-02 15:08:54.518802 7f4f763f4700 7 mon.0@0(leader).log v13562 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:54.518807 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a166a8f00 0x558a14eb0f00 log(last 14126) v1 2017-03-02 15:08:54.518812 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14126) v1 -- 0x558a14eb0f00 con 0 2017-03-02 15:08:54.518854 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13562) dispatch 0x558a167453c0 log(1 entries from seq 14127 at 2017-03-02 15:08:54.419040) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:54.518865 7f4f763f4700 5 mon.0@0(leader).paxos(paxos active c 26858..27363) is_readable = 1 - now=2017-03-02 15:08:54.518865 lease_expire=0.000000 has v0 lc 27363 2017-03-02 15:08:54.518874 7f4f763f4700 10 mon.0@0(leader).log v13562 preprocess_query log(1 entries from seq 14127 at 2017-03-02 15:08:54.419040) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:54.518884 7f4f763f4700 10 mon.0@0(leader).log v13562 preprocess_log log(1 entries from seq 14127 at 2017-03-02 15:08:54.419040) v1 from mon.0 2017-03-02 15:08:54.518890 7f4f763f4700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:54.518894 7f4f763f4700 20 allow so far , doing grant allow * 2017-03-02 15:08:54.518896 7f4f763f4700 20 allow all 2017-03-02 15:08:54.518902 7f4f763f4700 10 mon.0@0(leader).log v13562 prepare_update log(1 entries from seq 14127 at 2017-03-02 15:08:54.419040) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:54.518896 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14126) v1 ==== 0+0+0 (0 0 0) 0x558a14eb0f00 con 0x558a14482000 2017-03-02 15:08:54.518914 7f4f763f4700 10 mon.0@0(leader).log v13562 prepare_log log(1 entries from seq 14127 at 2017-03-02 15:08:54.419040) v1 from mon.0 2017-03-02 15:08:54.518920 7f4f763f4700 10 mon.0@0(leader).log v13562 logging 2017-03-02 15:08:54.419040 mon.0 172.21.15.17:6789/0 14127 : cluster [INF] pgmap v12981: 32 pgs: 32 active+clean; 71853 MB data, 138 GB used, 233 GB / 372 GB avail; 0 B/s rd, 3417 kB/s wr, 678 op/s 2017-03-02 15:08:54.518934 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13562) setting proposal_timer 0x558a1412ee20 with delay of 0.999482 2017-03-02 15:08:54.518957 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27363) finish_round waiting_for_readable 2017-03-02 15:08:54.518960 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27363) finish_round waiting_for_writeable 2017-03-02 15:08:54.518962 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26858..27363) finish_round done w/ waiters, state active 2017-03-02 15:08:54.518986 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:54.518998 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:54.519002 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:54.519005 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:54.519006 7f4f71beb700 20 allow all 2017-03-02 15:08:54.870843 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.5 172.21.15.11:6804/1637448123 8792 ==== mdsbeacon(4116/e up:active seq 8505 v737) v7 ==== 365+0+0 (1614439570 0 0) 0x558a143e7740 con 0x558a1471d800 2017-03-02 15:08:54.870888 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b0f20 for mds.? 172.21.15.11:6804/1637448123 2017-03-02 15:08:54.870895 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:54.870899 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a143e7740 mdsbeacon(4116/e up:active seq 8505 v737) v7 from mds.5 172.21.15.11:6804/1637448123 con 0x558a1471d800 2017-03-02 15:08:54.870907 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27363) is_readable = 1 - now=2017-03-02 15:08:54.870907 lease_expire=0.000000 has v0 lc 27363 2017-03-02 15:08:54.870917 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4116/e up:active seq 8505 v737) v7 from mds.5 172.21.15.11:6804/1637448123 2017-03-02 15:08:54.870925 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:54.870929 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:54.870930 7f4f71beb700 20 allow all 2017-03-02 15:08:54.870932 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4116/e up:active seq 8505 v737) v7 from mds.5 172.21.15.11:6804/1637448123 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:54.870941 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4116 2017-03-02 15:08:54.870947 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4116/e up:active seq 8505 v737) v7 noting time 2017-03-02 15:08:54.870951 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a164ad700 0x558a15204d00 mdsbeacon(4116/e up:active seq 8505 v737) v7 2017-03-02 15:08:54.870955 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6804/1637448123 -- mdsbeacon(4116/e up:active seq 8505 v737) v7 -- 0x558a15204d00 con 0 2017-03-02 15:08:54.884622 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.3 172.21.15.11:6805/1926402648 8568 ==== mdsbeacon(4118/f up:active seq 8505 v737) v7 ==== 406+0+0 (406191822 0 0) 0x558a148d7380 con 0x558a14905800 2017-03-02 15:08:54.884685 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b2260 for mds.? 172.21.15.11:6805/1926402648 2017-03-02 15:08:54.884714 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:54.884718 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a148d7380 mdsbeacon(4118/f up:active seq 8505 v737) v7 from mds.3 172.21.15.11:6805/1926402648 con 0x558a14905800 2017-03-02 15:08:54.884729 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27363) is_readable = 1 - now=2017-03-02 15:08:54.884730 lease_expire=0.000000 has v0 lc 27363 2017-03-02 15:08:54.884743 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4118/f up:active seq 8505 v737) v7 from mds.3 172.21.15.11:6805/1926402648 2017-03-02 15:08:54.884755 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:54.884759 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:54.884761 7f4f71beb700 20 allow all 2017-03-02 15:08:54.884763 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4118/f up:active seq 8505 v737) v7 from mds.3 172.21.15.11:6805/1926402648 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:54.884776 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4118 2017-03-02 15:08:54.884783 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4118/f up:active seq 8505 v737) v7 noting time 2017-03-02 15:08:54.884790 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a164ad700 0x558a143e7740 mdsbeacon(4118/f up:active seq 8505 v737) v7 2017-03-02 15:08:54.884795 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6805/1926402648 -- mdsbeacon(4118/f up:active seq 8505 v737) v7 -- 0x558a143e7740 con 0 2017-03-02 15:08:54.885416 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.2 172.21.15.22:6804/673088617 8567 ==== mdsbeacon(4119/c up:active seq 8505 v737) v7 ==== 365+0+0 (1076060364 0 0) 0x558a15204d00 con 0x558a14907000 2017-03-02 15:08:54.885464 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14916f20 for mds.? 172.21.15.22:6804/673088617 2017-03-02 15:08:54.885476 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:54.885481 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a15204d00 mdsbeacon(4119/c up:active seq 8505 v737) v7 from mds.2 172.21.15.22:6804/673088617 con 0x558a14907000 2017-03-02 15:08:54.885492 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27363) is_readable = 1 - now=2017-03-02 15:08:54.885493 lease_expire=0.000000 has v0 lc 27363 2017-03-02 15:08:54.885507 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4119/c up:active seq 8505 v737) v7 from mds.2 172.21.15.22:6804/673088617 2017-03-02 15:08:54.885520 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:54.885524 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:54.885526 7f4f71beb700 20 allow all 2017-03-02 15:08:54.885528 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4119/c up:active seq 8505 v737) v7 from mds.2 172.21.15.22:6804/673088617 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:54.885537 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4119 2017-03-02 15:08:54.885543 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4119/c up:active seq 8505 v737) v7 noting time 2017-03-02 15:08:54.885548 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a164ad700 0x558a148d7380 mdsbeacon(4119/c up:active seq 8505 v737) v7 2017-03-02 15:08:54.885552 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6804/673088617 -- mdsbeacon(4119/c up:active seq 8505 v737) v7 -- 0x558a148d7380 con 0 2017-03-02 15:08:54.925023 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.1 172.21.15.17:6804/2576824570 8564 ==== mdsbeacon(4120/g up:active seq 8505 v737) v7 ==== 365+0+0 (2284596695 0 0) 0x558a154b8d00 con 0x558a14908800 2017-03-02 15:08:54.925060 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14917600 for mds.? 172.21.15.17:6804/2576824570 2017-03-02 15:08:54.925066 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:54.925069 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a154b8d00 mdsbeacon(4120/g up:active seq 8505 v737) v7 from mds.1 172.21.15.17:6804/2576824570 con 0x558a14908800 2017-03-02 15:08:54.925076 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27363) is_readable = 1 - now=2017-03-02 15:08:54.925077 lease_expire=0.000000 has v0 lc 27363 2017-03-02 15:08:54.925086 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4120/g up:active seq 8505 v737) v7 from mds.1 172.21.15.17:6804/2576824570 2017-03-02 15:08:54.925096 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:54.925099 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:54.925101 7f4f71beb700 20 allow all 2017-03-02 15:08:54.925103 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4120/g up:active seq 8505 v737) v7 from mds.1 172.21.15.17:6804/2576824570 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:54.925116 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4120 2017-03-02 15:08:54.925122 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4120/g up:active seq 8505 v737) v7 noting time 2017-03-02 15:08:54.925129 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a164ad700 0x558a15204d00 mdsbeacon(4120/g up:active seq 8505 v737) v7 2017-03-02 15:08:54.925135 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6804/2576824570 -- mdsbeacon(4120/g up:active seq 8505 v737) v7 -- 0x558a15204d00 con 0 2017-03-02 15:08:54.970454 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.4 172.21.15.26:6805/2033388711 8562 ==== mdsbeacon(4117/b up:active seq 8505 v737) v7 ==== 365+0+0 (1988440852 0 0) 0x558a15204d00 con 0x558a14904000 2017-03-02 15:08:54.970497 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b1600 for mds.? 172.21.15.26:6805/2033388711 2017-03-02 15:08:54.970504 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:54.970508 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a15204d00 mdsbeacon(4117/b up:active seq 8505 v737) v7 from mds.4 172.21.15.26:6805/2033388711 con 0x558a14904000 2017-03-02 15:08:54.970516 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27363) is_readable = 1 - now=2017-03-02 15:08:54.970516 lease_expire=0.000000 has v0 lc 27363 2017-03-02 15:08:54.970527 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4117/b up:active seq 8505 v737) v7 from mds.4 172.21.15.26:6805/2033388711 2017-03-02 15:08:54.970536 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:54.970539 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:54.970540 7f4f71beb700 20 allow all 2017-03-02 15:08:54.970542 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4117/b up:active seq 8505 v737) v7 from mds.4 172.21.15.26:6805/2033388711 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:54.970551 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4117 2017-03-02 15:08:54.970556 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4117/b up:active seq 8505 v737) v7 noting time 2017-03-02 15:08:54.970561 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a164ad700 0x558a154b8d00 mdsbeacon(4117/b up:active seq 8505 v737) v7 2017-03-02 15:08:54.970565 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6805/2033388711 -- mdsbeacon(4117/b up:active seq 8505 v737) v7 -- 0x558a154b8d00 con 0 2017-03-02 15:08:54.979452 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.6 172.21.15.26:6804/2954782670 8755 ==== mdsbeacon(4115/a up:active seq 8505 v737) v7 ==== 365+0+0 (762300284 0 0) 0x558a143e7740 con 0x558a1471f000 2017-03-02 15:08:54.979492 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14868d60 for mds.? 172.21.15.26:6804/2954782670 2017-03-02 15:08:54.979499 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:54.979502 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a143e7740 mdsbeacon(4115/a up:active seq 8505 v737) v7 from mds.6 172.21.15.26:6804/2954782670 con 0x558a1471f000 2017-03-02 15:08:54.979509 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27363) is_readable = 1 - now=2017-03-02 15:08:54.979509 lease_expire=0.000000 has v0 lc 27363 2017-03-02 15:08:54.979519 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4115/a up:active seq 8505 v737) v7 from mds.6 172.21.15.26:6804/2954782670 2017-03-02 15:08:54.979527 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:54.979529 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:54.979530 7f4f71beb700 20 allow all 2017-03-02 15:08:54.979532 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4115/a up:active seq 8505 v737) v7 from mds.6 172.21.15.26:6804/2954782670 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:54.979541 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4115 2017-03-02 15:08:54.979546 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4115/a up:active seq 8505 v737) v7 noting time 2017-03-02 15:08:54.979550 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a164ad700 0x558a15204d00 mdsbeacon(4115/a up:active seq 8505 v737) v7 2017-03-02 15:08:54.979553 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6804/2954782670 -- mdsbeacon(4115/a up:active seq 8505 v737) v7 -- 0x558a15204d00 con 0 2017-03-02 15:08:55.518521 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13562) propose_pending 2017-03-02 15:08:55.518552 7f4f743f0700 10 mon.0@0(leader).log v13562 encode_full log v 13562 2017-03-02 15:08:55.518640 7f4f743f0700 10 mon.0@0(leader).log v13562 encode_pending v13563 2017-03-02 15:08:55.518689 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26858..27363) queue_pending_finisher 0x558a1412f860 2017-03-02 15:08:55.518694 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26858..27363) trigger_propose active, proposing now 2017-03-02 15:08:55.518703 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26858..27363) propose_pending 27364 10260 bytes 2017-03-02 15:08:55.518706 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26858..27363) begin for 27364 10260 bytes 2017-03-02 15:08:55.560464 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26858..27363) commit_start 27364 2017-03-02 15:08:55.593561 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26858..27363) commit_finish 27364 2017-03-02 15:08:55.593610 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:55.593633 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12981) refresh 2017-03-02 15:08:55.593652 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) refresh 2017-03-02 15:08:55.593667 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:55.593682 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13563) refresh 2017-03-02 15:08:55.593684 7f4f763f4700 10 mon.0@0(leader).log v13563 update_from_paxos 2017-03-02 15:08:55.593689 7f4f763f4700 10 mon.0@0(leader).log v13563 update_from_paxos version 13563 summary v 13562 2017-03-02 15:08:55.593694 7f4f763f4700 10 mon.0@0(leader).log v13563 update_from_paxos latest full 13562 2017-03-02 15:08:55.593704 7f4f763f4700 7 mon.0@0(leader).log v13563 update_from_paxos applying incremental log 13563 2017-03-02 15:08:54.419040 mon.0 172.21.15.17:6789/0 14127 : cluster [INF] pgmap v12981: 32 pgs: 32 active+clean; 71853 MB data, 138 GB used, 233 GB / 372 GB avail; 0 B/s rd, 3417 kB/s wr, 678 op/s 2017-03-02 15:08:55.593720 7f4f763f4700 20 mon.0@0(leader).log v13563 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:55.593740 7f4f763f4700 15 mon.0@0(leader).log v13563 update_from_paxos logging for 1 channels 2017-03-02 15:08:55.593742 7f4f763f4700 15 mon.0@0(leader).log v13563 update_from_paxos channel 'cluster' logging 198 bytes 2017-03-02 15:08:55.593812 7f4f763f4700 10 mon.0@0(leader).log v13563 check_subs 2017-03-02 15:08:55.593874 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:55.593892 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:55.593894 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:55.593910 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:55.593912 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12981) post_refresh 2017-03-02 15:08:55.593914 7f4f763f4700 10 mon.0@0(leader).pg v12981 post_paxos_update 2017-03-02 15:08:55.593917 7f4f763f4700 10 mon.0@0(leader).pg v12981 check_subs 2017-03-02 15:08:55.593919 7f4f763f4700 20 mon.0@0(leader).pg v12981 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:55.593925 7f4f763f4700 20 mon.0@0(leader).pg v12981 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:55.593928 7f4f763f4700 20 mon.0@0(leader).pg v12981 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:55.593931 7f4f763f4700 20 mon.0@0(leader).pg v12981 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:55.593934 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) post_refresh 2017-03-02 15:08:55.593935 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:55.593937 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13563) post_refresh 2017-03-02 15:08:55.593938 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:55.593939 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:55.593941 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:55.593943 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27364) commit_proposal 2017-03-02 15:08:55.593946 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13563) _active - not active 2017-03-02 15:08:55.593949 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27364) finish_round 2017-03-02 15:08:55.593951 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27364) finish_round waiting_for_acting 2017-03-02 15:08:55.593953 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13563) _active 2017-03-02 15:08:55.593954 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13563) _active creating new pending 2017-03-02 15:08:55.593963 7f4f763f4700 10 mon.0@0(leader).log v13563 create_pending v 13564 2017-03-02 15:08:55.593967 7f4f763f4700 7 mon.0@0(leader).log v13563 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:55.593972 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a14957800 0x558a15baaee0 log(last 14127) v1 2017-03-02 15:08:55.593976 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14127) v1 -- 0x558a15baaee0 con 0 2017-03-02 15:08:55.594001 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27364) finish_round waiting_for_readable 2017-03-02 15:08:55.594004 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27364) finish_round waiting_for_writeable 2017-03-02 15:08:55.594007 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26858..27364) finish_round done w/ waiters, state active 2017-03-02 15:08:55.594069 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14127) v1 ==== 0+0+0 (0 0 0) 0x558a15baaee0 con 0x558a14482000 2017-03-02 15:08:55.594087 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:55.594092 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:55.594095 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:55.594098 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:55.594099 7f4f71beb700 20 allow all 2017-03-02 15:08:56.012990 7f4f743f0700 11 mon.0@0(leader) e2 tick 2017-03-02 15:08:56.013019 7f4f743f0700 10 mon.0@0(leader).pg v12981 v12981: 32 pgs: 32 active+clean; 71853 MB data, 138 GB used, 233 GB / 372 GB avail; 0 B/s rd, 3417 kB/s wr, 678 op/s 2017-03-02 15:08:56.013039 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12981) maybe_trim trim_to 12481 would only trim 212 < paxos_service_trim_min 250 2017-03-02 15:08:56.013043 7f4f743f0700 10 mon.0@0(leader).mds e737 e737: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:08:56.013078 7f4f743f0700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) maybe_trim trim_to 237 would only trim 236 < paxos_service_trim_min 250 2017-03-02 15:08:56.013081 7f4f743f0700 10 mon.0@0(leader).osd e7 e7: 4 osds: 4 up, 4 in 2017-03-02 15:08:56.013086 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26858..27364) is_readable = 1 - now=2017-03-02 15:08:56.013086 lease_expire=0.000000 has v0 lc 27364 2017-03-02 15:08:56.013095 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26858..27364) is_readable = 1 - now=2017-03-02 15:08:56.013095 lease_expire=0.000000 has v0 lc 27364 2017-03-02 15:08:56.013101 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26858..27364) is_readable = 1 - now=2017-03-02 15:08:56.013101 lease_expire=0.000000 has v0 lc 27364 2017-03-02 15:08:56.013104 7f4f743f0700 10 mon.0@0(leader).osd e7 min_last_epoch_clean 6 2017-03-02 15:08:56.013107 7f4f743f0700 10 mon.0@0(leader).log v13563 log 2017-03-02 15:08:56.013109 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13563) maybe_trim trim_to 13063 would only trim 36 < paxos_service_trim_min 250 2017-03-02 15:08:56.013113 7f4f743f0700 10 mon.0@0(leader).auth v10 auth 2017-03-02 15:08:56.013117 7f4f743f0700 10 cephx keyserver: _check_rotating_secrets 2017-03-02 15:08:56.013125 7f4f743f0700 20 mon.0@0(leader) e2 sync_trim_providers 2017-03-02 15:08:58.266644 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.3 172.21.15.17:6800/27100 6840 ==== pg_stats(1 pgs tid 6804 v 0) v1 ==== 698+0+0 (4167388284 0 0) 0x558a15ae9e00 con 0x558a145d7000 2017-03-02 15:08:58.266678 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415f440 for osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:58.266686 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:58.266688 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12981) dispatch 0x558a15ae9e00 pg_stats(1 pgs tid 6804 v 0) v1 from osd.3 172.21.15.17:6800/27100 con 0x558a145d7000 2017-03-02 15:08:58.266693 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27364) is_readable = 1 - now=2017-03-02 15:08:58.266693 lease_expire=0.000000 has v0 lc 27364 2017-03-02 15:08:58.266700 7f4f71beb700 10 mon.0@0(leader).pg v12981 preprocess_query pg_stats(1 pgs tid 6804 v 0) v1 from osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:58.266705 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:58.266707 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:58.266708 7f4f71beb700 20 allow all 2017-03-02 15:08:58.266709 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27364) is_readable = 1 - now=2017-03-02 15:08:58.266709 lease_expire=0.000000 has v0 lc 27364 2017-03-02 15:08:58.266712 7f4f71beb700 10 mon.0@0(leader).pg v12981 prepare_update pg_stats(1 pgs tid 6804 v 0) v1 from osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:58.266715 7f4f71beb700 10 mon.0@0(leader).pg v12981 prepare_pg_stats pg_stats(1 pgs tid 6804 v 0) v1 from osd.3 2017-03-02 15:08:58.266720 7f4f71beb700 10 mon.0@0(leader).pg v12981 got osd.3 osd_stat(34184 MB used, 61157 MB avail, 95341 MB total, peers [0,1,2] op hist []) (was osd_stat(35388 MB used, 59953 MB avail, 95341 MB total, peers [0,1,2] op hist [1])) 2017-03-02 15:08:58.266727 7f4f71beb700 15 mon.0@0(leader).pg v12981 got 2.6 reported at 7:965713 state active+clean -> active+clean 2017-03-02 15:08:58.266734 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12981) setting proposal_timer 0x558a1412ee60 with delay of 0.05 2017-03-02 15:08:58.316838 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12981) propose_pending 2017-03-02 15:08:58.316852 7f4f743f0700 10 mon.0@0(leader).pg v12981 encode_pending v 12982 2017-03-02 15:08:58.316880 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26858..27364) queue_pending_finisher 0x558a1412d6c0 2017-03-02 15:08:58.316883 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26858..27364) trigger_propose active, proposing now 2017-03-02 15:08:58.316891 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26858..27364) propose_pending 27365 1090 bytes 2017-03-02 15:08:58.316893 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26858..27364) begin for 27365 1090 bytes 2017-03-02 15:08:58.329789 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26858..27364) commit_start 27365 2017-03-02 15:08:58.338095 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26858..27364) commit_finish 27365 2017-03-02 15:08:58.338130 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:58.338146 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12982) refresh 2017-03-02 15:08:58.338149 7f4f763f4700 10 mon.0@0(leader).pg v12981 update_from_paxos read_incremental 2017-03-02 15:08:58.338167 7f4f763f4700 20 mon.0@0(leader).pg v12981 refreshing pg 2.6 7:965713 active+clean 2017-03-02 15:08:58.338171 7f4f763f4700 20 mon.0@0(leader).pg v12981 refreshing osd.3 2017-03-02 15:08:58.338182 7f4f763f4700 10 mon.0@0(leader).pg v12982 read_pgmap_meta 2017-03-02 15:08:58.338196 7f4f763f4700 10 mon.0@0(leader).pg v12982 update_logger 2017-03-02 15:08:58.338209 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) refresh 2017-03-02 15:08:58.338227 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:58.338239 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13563) refresh 2017-03-02 15:08:58.338245 7f4f763f4700 10 mon.0@0(leader).log v13563 update_from_paxos 2017-03-02 15:08:58.338247 7f4f763f4700 10 mon.0@0(leader).log v13563 update_from_paxos version 13563 summary v 13563 2017-03-02 15:08:58.338259 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:58.338271 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:58.338273 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:58.338284 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:58.338286 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12982) post_refresh 2017-03-02 15:08:58.338288 7f4f763f4700 10 mon.0@0(leader).pg v12982 post_paxos_update 2017-03-02 15:08:58.338289 7f4f763f4700 10 mon.0@0(leader).pg v12982 check_subs 2017-03-02 15:08:58.338292 7f4f763f4700 20 mon.0@0(leader).pg v12982 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:58.338299 7f4f763f4700 20 mon.0@0(leader).pg v12982 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:58.338301 7f4f763f4700 20 mon.0@0(leader).pg v12982 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:58.338303 7f4f763f4700 20 mon.0@0(leader).pg v12982 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:58.338305 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) post_refresh 2017-03-02 15:08:58.338307 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:58.338308 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13563) post_refresh 2017-03-02 15:08:58.338309 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:58.338310 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:58.338311 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:58.338312 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27365) commit_proposal 2017-03-02 15:08:58.338315 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12982) _active - not active 2017-03-02 15:08:58.338317 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27365) finish_round 2017-03-02 15:08:58.338318 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27365) finish_round waiting_for_acting 2017-03-02 15:08:58.338320 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12982) _active 2017-03-02 15:08:58.338321 7f4f763f4700 7 mon.0@0(leader).paxosservice(pgmap 12269..12982) _active creating new pending 2017-03-02 15:08:58.338325 7f4f763f4700 10 mon.0@0(leader).pg v12982 create_pending v 12983 2017-03-02 15:08:58.338330 7f4f763f4700 7 mon.0@0(leader).pg v12982 _updated_stats for osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:58.338335 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a164ad700 0x558a153f8a00 pg_stats_ack(1 pgs tid 6804) v1 2017-03-02 15:08:58.338340 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6800/27100 -- pg_stats_ack(1 pgs tid 6804) v1 -- 0x558a153f8a00 con 0 2017-03-02 15:08:58.338376 7f4f763f4700 10 mon.0@0(leader).pg v12982 check_osd_map already seen 7 >= 7 2017-03-02 15:08:58.338379 7f4f763f4700 10 mon.0@0(leader).pg v12982 update_logger 2017-03-02 15:08:58.338401 7f4f763f4700 0 log_channel(cluster) log [INF] : pgmap v12982: 32 pgs: 32 active+clean; 71727 MB data, 137 GB used, 234 GB / 372 GB avail; 0 B/s rd, 3411 kB/s wr, 664 op/s 2017-03-02 15:08:58.338414 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14128 at 2017-03-02 15:08:58.338402) v1 -- 0x558a16745600 con 0 2017-03-02 15:08:58.338427 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27365) finish_round waiting_for_readable 2017-03-02 15:08:58.338429 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27365) finish_round waiting_for_writeable 2017-03-02 15:08:58.338431 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26858..27365) finish_round done w/ waiters, state active 2017-03-02 15:08:58.338461 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14128 at 2017-03-02 15:08:58.338402) v1 ==== 0+0+0 (0 0 0) 0x558a16745600 con 0x558a14482000 2017-03-02 15:08:58.338479 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:58.338483 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:58.338485 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13563) dispatch 0x558a16745600 log(1 entries from seq 14128 at 2017-03-02 15:08:58.338402) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:58.338490 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27365) is_readable = 1 - now=2017-03-02 15:08:58.338490 lease_expire=0.000000 has v0 lc 27365 2017-03-02 15:08:58.338494 7f4f71beb700 10 mon.0@0(leader).log v13563 preprocess_query log(1 entries from seq 14128 at 2017-03-02 15:08:58.338402) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:58.338498 7f4f71beb700 10 mon.0@0(leader).log v13563 preprocess_log log(1 entries from seq 14128 at 2017-03-02 15:08:58.338402) v1 from mon.0 2017-03-02 15:08:58.338501 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:58.338503 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:58.338504 7f4f71beb700 20 allow all 2017-03-02 15:08:58.338511 7f4f71beb700 10 mon.0@0(leader).log v13563 prepare_update log(1 entries from seq 14128 at 2017-03-02 15:08:58.338402) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:58.338514 7f4f71beb700 10 mon.0@0(leader).log v13563 prepare_log log(1 entries from seq 14128 at 2017-03-02 15:08:58.338402) v1 from mon.0 2017-03-02 15:08:58.338516 7f4f71beb700 10 mon.0@0(leader).log v13563 logging 2017-03-02 15:08:58.338402 mon.0 172.21.15.17:6789/0 14128 : cluster [INF] pgmap v12982: 32 pgs: 32 active+clean; 71727 MB data, 137 GB used, 234 GB / 372 GB avail; 0 B/s rd, 3411 kB/s wr, 664 op/s 2017-03-02 15:08:58.338521 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13563) setting proposal_timer 0x558a1412d340 with delay of 0.999588 2017-03-02 15:08:58.467847 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.1 172.21.15.22:6800/26531 6840 ==== pg_stats(5 pgs tid 6804 v 0) v1 ==== 2974+0+0 (3515542479 0 0) 0x558a15ae9b80 con 0x558a14643800 2017-03-02 15:08:58.467891 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1463b1e0 for osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:58.467900 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:58.467903 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12982) dispatch 0x558a15ae9b80 pg_stats(5 pgs tid 6804 v 0) v1 from osd.1 172.21.15.22:6800/26531 con 0x558a14643800 2017-03-02 15:08:58.467909 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27365) is_readable = 1 - now=2017-03-02 15:08:58.467909 lease_expire=0.000000 has v0 lc 27365 2017-03-02 15:08:58.467919 7f4f71beb700 10 mon.0@0(leader).pg v12982 preprocess_query pg_stats(5 pgs tid 6804 v 0) v1 from osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:58.467924 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:58.467927 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:58.467928 7f4f71beb700 20 allow all 2017-03-02 15:08:58.467930 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27365) is_readable = 1 - now=2017-03-02 15:08:58.467930 lease_expire=0.000000 has v0 lc 27365 2017-03-02 15:08:58.467934 7f4f71beb700 10 mon.0@0(leader).pg v12982 prepare_update pg_stats(5 pgs tid 6804 v 0) v1 from osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:58.467938 7f4f71beb700 10 mon.0@0(leader).pg v12982 prepare_pg_stats pg_stats(5 pgs tid 6804 v 0) v1 from osd.1 2017-03-02 15:08:58.467945 7f4f71beb700 10 mon.0@0(leader).pg v12982 got osd.1 osd_stat(42763 MB used, 52577 MB avail, 95341 MB total, peers [0,2,3] op hist [1,0,1]) (was osd_stat(44321 MB used, 51020 MB avail, 95341 MB total, peers [0,2,3] op hist [0,1,2])) 2017-03-02 15:08:58.467956 7f4f71beb700 15 mon.0@0(leader).pg v12982 got 1.1 reported at 7:587302 state active+clean -> active+clean 2017-03-02 15:08:58.467969 7f4f71beb700 15 mon.0@0(leader).pg v12982 got 1.5 reported at 7:568178 state active+clean -> active+clean 2017-03-02 15:08:58.467977 7f4f71beb700 15 mon.0@0(leader).pg v12982 got 1.6 reported at 7:503572 state active+clean -> active+clean 2017-03-02 15:08:58.467983 7f4f71beb700 15 mon.0@0(leader).pg v12982 got 2.4 reported at 7:1027536 state active+clean -> active+clean 2017-03-02 15:08:58.467987 7f4f71beb700 15 mon.0@0(leader).pg v12982 got 2.5 reported at 7:1130917 state active+clean -> active+clean 2017-03-02 15:08:58.467992 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12982) setting proposal_timer 0x558a1412cf10 with delay of 0.870118 2017-03-02 15:08:58.561937 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.2 172.21.15.11:6800/26536 6840 ==== pg_stats(8 pgs tid 6804 v 0) v1 ==== 4668+0+0 (2866011718 0 0) 0x558a15ae4f80 con 0x558a145d5800 2017-03-02 15:08:58.561978 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a145c3700 for osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:58.561987 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:58.561991 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12982) dispatch 0x558a15ae4f80 pg_stats(8 pgs tid 6804 v 0) v1 from osd.2 172.21.15.11:6800/26536 con 0x558a145d5800 2017-03-02 15:08:58.561999 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27365) is_readable = 1 - now=2017-03-02 15:08:58.562000 lease_expire=0.000000 has v0 lc 27365 2017-03-02 15:08:58.562010 7f4f71beb700 10 mon.0@0(leader).pg v12982 preprocess_query pg_stats(8 pgs tid 6804 v 0) v1 from osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:58.562017 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:58.562019 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:58.562020 7f4f71beb700 20 allow all 2017-03-02 15:08:58.562022 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27365) is_readable = 1 - now=2017-03-02 15:08:58.562022 lease_expire=0.000000 has v0 lc 27365 2017-03-02 15:08:58.562026 7f4f71beb700 10 mon.0@0(leader).pg v12982 prepare_update pg_stats(8 pgs tid 6804 v 0) v1 from osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:58.562029 7f4f71beb700 10 mon.0@0(leader).pg v12982 prepare_pg_stats pg_stats(8 pgs tid 6804 v 0) v1 from osd.2 2017-03-02 15:08:58.562035 7f4f71beb700 10 mon.0@0(leader).pg v12982 got osd.2 osd_stat(51692 MB used, 43648 MB avail, 95341 MB total, peers [0,1,3] op hist [0,1]) (was osd_stat(53438 MB used, 41902 MB avail, 95341 MB total, peers [0,1,3] op hist [1,2])) 2017-03-02 15:08:58.562044 7f4f71beb700 15 mon.0@0(leader).pg v12982 got 1.0 reported at 7:497018 state active+clean -> active+clean 2017-03-02 15:08:58.562053 7f4f71beb700 15 mon.0@0(leader).pg v12982 got 1.2 reported at 7:577891 state active+clean -> active+clean 2017-03-02 15:08:58.562057 7f4f71beb700 15 mon.0@0(leader).pg v12982 got 1.7 reported at 7:530160 state active+clean -> active+clean 2017-03-02 15:08:58.562061 7f4f71beb700 15 mon.0@0(leader).pg v12982 got 2.0 reported at 7:1058154 state active+clean -> active+clean 2017-03-02 15:08:58.562065 7f4f71beb700 15 mon.0@0(leader).pg v12982 got 2.1 reported at 7:1090890 state active+clean -> active+clean 2017-03-02 15:08:58.562068 7f4f71beb700 15 mon.0@0(leader).pg v12982 got 2.2 reported at 7:1164629 state active+clean -> active+clean 2017-03-02 15:08:58.562071 7f4f71beb700 15 mon.0@0(leader).pg v12982 got 2.3 reported at 7:1096675 state active+clean -> active+clean 2017-03-02 15:08:58.562075 7f4f71beb700 15 mon.0@0(leader).pg v12982 got 2.7 reported at 7:1136824 state active+clean -> active+clean 2017-03-02 15:08:58.562080 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12982) proposal_timer already set 2017-03-02 15:08:58.718480 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.0 172.21.15.26:6800/26515 6840 ==== pg_stats(2 pgs tid 6804 v 0) v1 ==== 1264+0+0 (3476969350 0 0) 0x558a15c0b700 con 0x558a145d8800 2017-03-02 15:08:58.718524 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1463af20 for osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:58.718530 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:58.718533 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12982) dispatch 0x558a15c0b700 pg_stats(2 pgs tid 6804 v 0) v1 from osd.0 172.21.15.26:6800/26515 con 0x558a145d8800 2017-03-02 15:08:58.718538 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27365) is_readable = 1 - now=2017-03-02 15:08:58.718539 lease_expire=0.000000 has v0 lc 27365 2017-03-02 15:08:58.718548 7f4f71beb700 10 mon.0@0(leader).pg v12982 preprocess_query pg_stats(2 pgs tid 6804 v 0) v1 from osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:58.718554 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:08:58.718556 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:58.718557 7f4f71beb700 20 allow all 2017-03-02 15:08:58.718559 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27365) is_readable = 1 - now=2017-03-02 15:08:58.718559 lease_expire=0.000000 has v0 lc 27365 2017-03-02 15:08:58.718562 7f4f71beb700 10 mon.0@0(leader).pg v12982 prepare_update pg_stats(2 pgs tid 6804 v 0) v1 from osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:58.718566 7f4f71beb700 10 mon.0@0(leader).pg v12982 prepare_pg_stats pg_stats(2 pgs tid 6804 v 0) v1 from osd.0 2017-03-02 15:08:58.718572 7f4f71beb700 10 mon.0@0(leader).pg v12982 got osd.0 osd_stat(8955 MB used, 86385 MB avail, 95341 MB total, peers [1,2,3] op hist []) (was osd_stat(9114 MB used, 86226 MB avail, 95341 MB total, peers [1,2,3] op hist [1])) 2017-03-02 15:08:58.718581 7f4f71beb700 15 mon.0@0(leader).pg v12982 got 1.3 reported at 7:500601 state active+clean -> active+clean 2017-03-02 15:08:58.718594 7f4f71beb700 15 mon.0@0(leader).pg v12982 got 1.4 reported at 7:486533 state active+clean -> active+clean 2017-03-02 15:08:58.718601 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12982) proposal_timer already set 2017-03-02 15:08:58.871106 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.5 172.21.15.11:6804/1637448123 8793 ==== mdsbeacon(4116/e up:active seq 8506 v737) v7 ==== 365+0+0 (3170318817 0 0) 0x558a143e49c0 con 0x558a1471d800 2017-03-02 15:08:58.871163 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b0f20 for mds.? 172.21.15.11:6804/1637448123 2017-03-02 15:08:58.871175 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:58.871181 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a143e49c0 mdsbeacon(4116/e up:active seq 8506 v737) v7 from mds.5 172.21.15.11:6804/1637448123 con 0x558a1471d800 2017-03-02 15:08:58.871192 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27365) is_readable = 1 - now=2017-03-02 15:08:58.871193 lease_expire=0.000000 has v0 lc 27365 2017-03-02 15:08:58.871212 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4116/e up:active seq 8506 v737) v7 from mds.5 172.21.15.11:6804/1637448123 2017-03-02 15:08:58.871224 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:58.871229 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:58.871231 7f4f71beb700 20 allow all 2017-03-02 15:08:58.871233 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4116/e up:active seq 8506 v737) v7 from mds.5 172.21.15.11:6804/1637448123 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:58.871247 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4116 2017-03-02 15:08:58.871255 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4116/e up:active seq 8506 v737) v7 noting time 2017-03-02 15:08:58.871262 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15c73600 0x558a143e7740 mdsbeacon(4116/e up:active seq 8506 v737) v7 2017-03-02 15:08:58.871267 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6804/1637448123 -- mdsbeacon(4116/e up:active seq 8506 v737) v7 -- 0x558a143e7740 con 0 2017-03-02 15:08:58.884633 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.3 172.21.15.11:6805/1926402648 8569 ==== mdsbeacon(4118/f up:active seq 8506 v737) v7 ==== 406+0+0 (3126692916 0 0) 0x558a15204d00 con 0x558a14905800 2017-03-02 15:08:58.884678 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b2260 for mds.? 172.21.15.11:6805/1926402648 2017-03-02 15:08:58.884686 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:58.884689 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a15204d00 mdsbeacon(4118/f up:active seq 8506 v737) v7 from mds.3 172.21.15.11:6805/1926402648 con 0x558a14905800 2017-03-02 15:08:58.884695 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27365) is_readable = 1 - now=2017-03-02 15:08:58.884696 lease_expire=0.000000 has v0 lc 27365 2017-03-02 15:08:58.884706 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4118/f up:active seq 8506 v737) v7 from mds.3 172.21.15.11:6805/1926402648 2017-03-02 15:08:58.884713 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:58.884716 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:58.884717 7f4f71beb700 20 allow all 2017-03-02 15:08:58.884719 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4118/f up:active seq 8506 v737) v7 from mds.3 172.21.15.11:6805/1926402648 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:58.884727 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4118 2017-03-02 15:08:58.884732 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4118/f up:active seq 8506 v737) v7 noting time 2017-03-02 15:08:58.884736 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15c73600 0x558a143e49c0 mdsbeacon(4118/f up:active seq 8506 v737) v7 2017-03-02 15:08:58.884739 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6805/1926402648 -- mdsbeacon(4118/f up:active seq 8506 v737) v7 -- 0x558a143e49c0 con 0 2017-03-02 15:08:58.885443 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.2 172.21.15.22:6804/673088617 8568 ==== mdsbeacon(4119/c up:active seq 8506 v737) v7 ==== 365+0+0 (2632859071 0 0) 0x558a143e7740 con 0x558a14907000 2017-03-02 15:08:58.885479 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14916f20 for mds.? 172.21.15.22:6804/673088617 2017-03-02 15:08:58.885488 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:58.885492 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a143e7740 mdsbeacon(4119/c up:active seq 8506 v737) v7 from mds.2 172.21.15.22:6804/673088617 con 0x558a14907000 2017-03-02 15:08:58.885501 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27365) is_readable = 1 - now=2017-03-02 15:08:58.885502 lease_expire=0.000000 has v0 lc 27365 2017-03-02 15:08:58.885511 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4119/c up:active seq 8506 v737) v7 from mds.2 172.21.15.22:6804/673088617 2017-03-02 15:08:58.885518 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:58.885520 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:58.885521 7f4f71beb700 20 allow all 2017-03-02 15:08:58.885523 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4119/c up:active seq 8506 v737) v7 from mds.2 172.21.15.22:6804/673088617 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:58.885530 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4119 2017-03-02 15:08:58.885534 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4119/c up:active seq 8506 v737) v7 noting time 2017-03-02 15:08:58.885537 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15c73600 0x558a15204d00 mdsbeacon(4119/c up:active seq 8506 v737) v7 2017-03-02 15:08:58.885540 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6804/673088617 -- mdsbeacon(4119/c up:active seq 8506 v737) v7 -- 0x558a15204d00 con 0 2017-03-02 15:08:58.924993 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.1 172.21.15.17:6804/2576824570 8565 ==== mdsbeacon(4120/g up:active seq 8506 v737) v7 ==== 365+0+0 (1424060580 0 0) 0x558a154b8d00 con 0x558a14908800 2017-03-02 15:08:58.925045 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14917600 for mds.? 172.21.15.17:6804/2576824570 2017-03-02 15:08:58.925057 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:58.925062 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a154b8d00 mdsbeacon(4120/g up:active seq 8506 v737) v7 from mds.1 172.21.15.17:6804/2576824570 con 0x558a14908800 2017-03-02 15:08:58.925073 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27365) is_readable = 1 - now=2017-03-02 15:08:58.925074 lease_expire=0.000000 has v0 lc 27365 2017-03-02 15:08:58.925087 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4120/g up:active seq 8506 v737) v7 from mds.1 172.21.15.17:6804/2576824570 2017-03-02 15:08:58.925100 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:58.925105 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:58.925107 7f4f71beb700 20 allow all 2017-03-02 15:08:58.925109 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4120/g up:active seq 8506 v737) v7 from mds.1 172.21.15.17:6804/2576824570 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:58.925118 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4120 2017-03-02 15:08:58.925124 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4120/g up:active seq 8506 v737) v7 noting time 2017-03-02 15:08:58.925130 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15c73600 0x558a143e7740 mdsbeacon(4120/g up:active seq 8506 v737) v7 2017-03-02 15:08:58.925133 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6804/2576824570 -- mdsbeacon(4120/g up:active seq 8506 v737) v7 -- 0x558a143e7740 con 0 2017-03-02 15:08:58.970622 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.4 172.21.15.26:6805/2033388711 8563 ==== mdsbeacon(4117/b up:active seq 8506 v737) v7 ==== 365+0+0 (2856873575 0 0) 0x558a143e7740 con 0x558a14904000 2017-03-02 15:08:58.970688 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b1600 for mds.? 172.21.15.26:6805/2033388711 2017-03-02 15:08:58.970717 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:58.970721 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a143e7740 mdsbeacon(4117/b up:active seq 8506 v737) v7 from mds.4 172.21.15.26:6805/2033388711 con 0x558a14904000 2017-03-02 15:08:58.970730 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27365) is_readable = 1 - now=2017-03-02 15:08:58.970730 lease_expire=0.000000 has v0 lc 27365 2017-03-02 15:08:58.970742 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4117/b up:active seq 8506 v737) v7 from mds.4 172.21.15.26:6805/2033388711 2017-03-02 15:08:58.970751 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:58.970755 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:58.970756 7f4f71beb700 20 allow all 2017-03-02 15:08:58.970758 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4117/b up:active seq 8506 v737) v7 from mds.4 172.21.15.26:6805/2033388711 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:58.970768 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4117 2017-03-02 15:08:58.970774 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4117/b up:active seq 8506 v737) v7 noting time 2017-03-02 15:08:58.970779 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15c73600 0x558a154b8d00 mdsbeacon(4117/b up:active seq 8506 v737) v7 2017-03-02 15:08:58.970783 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6805/2033388711 -- mdsbeacon(4117/b up:active seq 8506 v737) v7 -- 0x558a154b8d00 con 0 2017-03-02 15:08:58.979683 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.6 172.21.15.26:6804/2954782670 8756 ==== mdsbeacon(4115/a up:active seq 8506 v737) v7 ==== 365+0+0 (4053965327 0 0) 0x558a143e49c0 con 0x558a1471f000 2017-03-02 15:08:58.979735 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14868d60 for mds.? 172.21.15.26:6804/2954782670 2017-03-02 15:08:58.979742 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:58.979745 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a143e49c0 mdsbeacon(4115/a up:active seq 8506 v737) v7 from mds.6 172.21.15.26:6804/2954782670 con 0x558a1471f000 2017-03-02 15:08:58.979753 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27365) is_readable = 1 - now=2017-03-02 15:08:58.979753 lease_expire=0.000000 has v0 lc 27365 2017-03-02 15:08:58.979763 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mdsbeacon(4115/a up:active seq 8506 v737) v7 from mds.6 172.21.15.26:6804/2954782670 2017-03-02 15:08:58.979771 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:08:58.979774 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:58.979775 7f4f71beb700 20 allow all 2017-03-02 15:08:58.979777 7f4f71beb700 12 mon.0@0(leader).mds e737 preprocess_beacon mdsbeacon(4115/a up:active seq 8506 v737) v7 from mds.6 172.21.15.26:6804/2954782670 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:08:58.979786 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_beacon: GID exists in map: 4115 2017-03-02 15:08:58.979791 7f4f71beb700 15 mon.0@0(leader).mds e737 _note_beacon mdsbeacon(4115/a up:active seq 8506 v737) v7 noting time 2017-03-02 15:08:58.979796 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15c73600 0x558a143e7740 mdsbeacon(4115/a up:active seq 8506 v737) v7 2017-03-02 15:08:58.979799 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6804/2954782670 -- mdsbeacon(4115/a up:active seq 8506 v737) v7 -- 0x558a143e7740 con 0 2017-03-02 15:08:59.338270 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13563) propose_pending 2017-03-02 15:08:59.338298 7f4f743f0700 10 mon.0@0(leader).log v13563 encode_full log v 13563 2017-03-02 15:08:59.338415 7f4f743f0700 10 mon.0@0(leader).log v13563 encode_pending v13564 2017-03-02 15:08:59.338425 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26858..27365) queue_pending_finisher 0x558a1412ee60 2017-03-02 15:08:59.338431 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26858..27365) trigger_propose active, proposing now 2017-03-02 15:08:59.338441 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26858..27365) propose_pending 27366 10255 bytes 2017-03-02 15:08:59.338444 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26858..27365) begin for 27366 10255 bytes 2017-03-02 15:08:59.368278 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26858..27365) commit_start 27366 2017-03-02 15:08:59.368366 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12982) propose_pending 2017-03-02 15:08:59.368374 7f4f743f0700 10 mon.0@0(leader).pg v12982 encode_pending v 12983 2017-03-02 15:08:59.368522 7f4f743f0700 5 mon.0@0(leader).paxos(paxos writing c 26858..27365) queue_pending_finisher 0x558a1412d340 2017-03-02 15:08:59.368538 7f4f743f0700 10 mon.0@0(leader).paxos(paxos writing c 26858..27365) trigger_propose not active, will propose later 2017-03-02 15:08:59.401771 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26858..27365) commit_finish 27366 2017-03-02 15:08:59.401853 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:59.401880 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12982) refresh 2017-03-02 15:08:59.401901 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) refresh 2017-03-02 15:08:59.401918 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:59.401937 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13564) refresh 2017-03-02 15:08:59.401939 7f4f763f4700 10 mon.0@0(leader).log v13564 update_from_paxos 2017-03-02 15:08:59.401942 7f4f763f4700 10 mon.0@0(leader).log v13564 update_from_paxos version 13564 summary v 13563 2017-03-02 15:08:59.401947 7f4f763f4700 10 mon.0@0(leader).log v13564 update_from_paxos latest full 13563 2017-03-02 15:08:59.401959 7f4f763f4700 7 mon.0@0(leader).log v13564 update_from_paxos applying incremental log 13564 2017-03-02 15:08:58.338402 mon.0 172.21.15.17:6789/0 14128 : cluster [INF] pgmap v12982: 32 pgs: 32 active+clean; 71727 MB data, 137 GB used, 234 GB / 372 GB avail; 0 B/s rd, 3411 kB/s wr, 664 op/s 2017-03-02 15:08:59.401979 7f4f763f4700 20 mon.0@0(leader).log v13564 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:08:59.402000 7f4f763f4700 15 mon.0@0(leader).log v13564 update_from_paxos logging for 1 channels 2017-03-02 15:08:59.402003 7f4f763f4700 15 mon.0@0(leader).log v13564 update_from_paxos channel 'cluster' logging 198 bytes 2017-03-02 15:08:59.402091 7f4f763f4700 10 mon.0@0(leader).log v13564 check_subs 2017-03-02 15:08:59.402136 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:59.402156 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:59.402159 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:59.402178 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:59.402180 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12982) post_refresh 2017-03-02 15:08:59.402183 7f4f763f4700 10 mon.0@0(leader).pg v12982 post_paxos_update 2017-03-02 15:08:59.402185 7f4f763f4700 10 mon.0@0(leader).pg v12982 check_subs 2017-03-02 15:08:59.402188 7f4f763f4700 20 mon.0@0(leader).pg v12982 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:59.402195 7f4f763f4700 20 mon.0@0(leader).pg v12982 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:59.402198 7f4f763f4700 20 mon.0@0(leader).pg v12982 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:59.402201 7f4f763f4700 20 mon.0@0(leader).pg v12982 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:59.402204 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) post_refresh 2017-03-02 15:08:59.402206 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:59.402208 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13564) post_refresh 2017-03-02 15:08:59.402210 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:59.402211 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:59.402213 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:59.402216 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27366) commit_proposal 2017-03-02 15:08:59.402220 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13564) _active - not active 2017-03-02 15:08:59.402223 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27366) finish_round 2017-03-02 15:08:59.402225 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27366) finish_round waiting_for_acting 2017-03-02 15:08:59.402228 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13564) _active 2017-03-02 15:08:59.402230 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13564) _active creating new pending 2017-03-02 15:08:59.402240 7f4f763f4700 10 mon.0@0(leader).log v13564 create_pending v 13565 2017-03-02 15:08:59.402246 7f4f763f4700 7 mon.0@0(leader).log v13564 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:59.402255 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a1621a700 0x558a15229840 log(last 14128) v1 2017-03-02 15:08:59.402262 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14128) v1 -- 0x558a15229840 con 0 2017-03-02 15:08:59.402333 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27366) finish_round waiting_for_readable 2017-03-02 15:08:59.402340 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27366) finish_round waiting_for_writeable 2017-03-02 15:08:59.402344 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26858..27366) finish_round done w/ waiters, state active 2017-03-02 15:08:59.402343 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14128) v1 ==== 0+0+0 (0 0 0) 0x558a15229840 con 0x558a14482000 2017-03-02 15:08:59.402394 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26858..27366) propose_pending 27367 9756 bytes 2017-03-02 15:08:59.402408 7f4f763f4700 10 mon.0@0(leader).paxos(paxos updating c 26858..27366) begin for 27367 9756 bytes 2017-03-02 15:08:59.435179 7f4f763f4700 10 mon.0@0(leader).paxos(paxos updating c 26858..27366) commit_start 27367 2017-03-02 15:08:59.435297 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:59.435325 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:59.435330 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:08:59.435334 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:59.435335 7f4f71beb700 20 allow all 2017-03-02 15:08:59.468728 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26858..27366) commit_finish 27367 2017-03-02 15:08:59.468784 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:08:59.468806 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12983) refresh 2017-03-02 15:08:59.468809 7f4f763f4700 10 mon.0@0(leader).pg v12982 update_from_paxos read_incremental 2017-03-02 15:08:59.468878 7f4f763f4700 20 mon.0@0(leader).pg v12982 refreshing pg 1.0 7:497018 active+clean 2017-03-02 15:08:59.468891 7f4f763f4700 20 mon.0@0(leader).pg v12982 refreshing pg 1.1 7:587302 active+clean 2017-03-02 15:08:59.468899 7f4f763f4700 20 mon.0@0(leader).pg v12982 refreshing pg 1.2 7:577891 active+clean 2017-03-02 15:08:59.468906 7f4f763f4700 20 mon.0@0(leader).pg v12982 refreshing pg 1.3 7:500601 active+clean 2017-03-02 15:08:59.468914 7f4f763f4700 20 mon.0@0(leader).pg v12982 refreshing pg 1.4 7:486533 active+clean 2017-03-02 15:08:59.468921 7f4f763f4700 20 mon.0@0(leader).pg v12982 refreshing pg 1.5 7:568178 active+clean 2017-03-02 15:08:59.468928 7f4f763f4700 20 mon.0@0(leader).pg v12982 refreshing pg 1.6 7:503572 active+clean 2017-03-02 15:08:59.468935 7f4f763f4700 20 mon.0@0(leader).pg v12982 refreshing pg 1.7 7:530160 active+clean 2017-03-02 15:08:59.468942 7f4f763f4700 20 mon.0@0(leader).pg v12982 refreshing pg 2.0 7:1058154 active+clean 2017-03-02 15:08:59.468952 7f4f763f4700 20 mon.0@0(leader).pg v12982 refreshing pg 2.1 7:1090890 active+clean 2017-03-02 15:08:59.468960 7f4f763f4700 20 mon.0@0(leader).pg v12982 refreshing pg 2.2 7:1164629 active+clean 2017-03-02 15:08:59.468967 7f4f763f4700 20 mon.0@0(leader).pg v12982 refreshing pg 2.3 7:1096675 active+clean 2017-03-02 15:08:59.468973 7f4f763f4700 20 mon.0@0(leader).pg v12982 refreshing pg 2.4 7:1027536 active+clean 2017-03-02 15:08:59.468979 7f4f763f4700 20 mon.0@0(leader).pg v12982 refreshing pg 2.5 7:1130917 active+clean 2017-03-02 15:08:59.468985 7f4f763f4700 20 mon.0@0(leader).pg v12982 refreshing pg 2.7 7:1136824 active+clean 2017-03-02 15:08:59.468988 7f4f763f4700 20 mon.0@0(leader).pg v12982 refreshing osd.0 2017-03-02 15:08:59.468996 7f4f763f4700 20 mon.0@0(leader).pg v12982 refreshing osd.1 2017-03-02 15:08:59.469002 7f4f763f4700 20 mon.0@0(leader).pg v12982 refreshing osd.2 2017-03-02 15:08:59.469014 7f4f763f4700 10 mon.0@0(leader).pg v12983 read_pgmap_meta 2017-03-02 15:08:59.469038 7f4f763f4700 10 mon.0@0(leader).pg v12983 update_logger 2017-03-02 15:08:59.469056 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) refresh 2017-03-02 15:08:59.469072 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:08:59.469088 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13564) refresh 2017-03-02 15:08:59.469090 7f4f763f4700 10 mon.0@0(leader).log v13564 update_from_paxos 2017-03-02 15:08:59.469092 7f4f763f4700 10 mon.0@0(leader).log v13564 update_from_paxos version 13564 summary v 13564 2017-03-02 15:08:59.469107 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:08:59.469123 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:08:59.469125 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:08:59.469142 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:08:59.469144 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12983) post_refresh 2017-03-02 15:08:59.469146 7f4f763f4700 10 mon.0@0(leader).pg v12983 post_paxos_update 2017-03-02 15:08:59.469147 7f4f763f4700 10 mon.0@0(leader).pg v12983 check_subs 2017-03-02 15:08:59.469151 7f4f763f4700 20 mon.0@0(leader).pg v12983 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:08:59.469159 7f4f763f4700 20 mon.0@0(leader).pg v12983 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:59.469162 7f4f763f4700 20 mon.0@0(leader).pg v12983 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:59.469165 7f4f763f4700 20 mon.0@0(leader).pg v12983 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:59.469168 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) post_refresh 2017-03-02 15:08:59.469170 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:08:59.469172 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13564) post_refresh 2017-03-02 15:08:59.469173 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:08:59.469174 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:08:59.469175 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:08:59.469178 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27367) commit_proposal 2017-03-02 15:08:59.469182 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12983) _active - not active 2017-03-02 15:08:59.469184 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27367) finish_round 2017-03-02 15:08:59.469186 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27367) finish_round waiting_for_acting 2017-03-02 15:08:59.469188 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12983) _active 2017-03-02 15:08:59.469190 7f4f763f4700 7 mon.0@0(leader).paxosservice(pgmap 12269..12983) _active creating new pending 2017-03-02 15:08:59.469218 7f4f763f4700 10 mon.0@0(leader).pg v12983 create_pending v 12984 2017-03-02 15:08:59.469224 7f4f763f4700 7 mon.0@0(leader).pg v12983 _updated_stats for osd.1 172.21.15.22:6800/26531 2017-03-02 15:08:59.469229 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a16405500 0x558a15e1c600 pg_stats_ack(5 pgs tid 6804) v1 2017-03-02 15:08:59.469233 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6800/26531 -- pg_stats_ack(5 pgs tid 6804) v1 -- 0x558a15e1c600 con 0 2017-03-02 15:08:59.469278 7f4f763f4700 7 mon.0@0(leader).pg v12983 _updated_stats for osd.2 172.21.15.11:6800/26536 2017-03-02 15:08:59.469283 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a162cb200 0x558a153fc400 pg_stats_ack(8 pgs tid 6804) v1 2017-03-02 15:08:59.469286 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6800/26536 -- pg_stats_ack(8 pgs tid 6804) v1 -- 0x558a153fc400 con 0 2017-03-02 15:08:59.469300 7f4f763f4700 7 mon.0@0(leader).pg v12983 _updated_stats for osd.0 172.21.15.26:6800/26515 2017-03-02 15:08:59.469304 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a145dcd00 0x558a15561a00 pg_stats_ack(2 pgs tid 6804) v1 2017-03-02 15:08:59.469305 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6800/26515 -- pg_stats_ack(2 pgs tid 6804) v1 -- 0x558a15561a00 con 0 2017-03-02 15:08:59.469329 7f4f763f4700 10 mon.0@0(leader).pg v12983 check_osd_map already seen 7 >= 7 2017-03-02 15:08:59.469332 7f4f763f4700 10 mon.0@0(leader).pg v12983 update_logger 2017-03-02 15:08:59.469350 7f4f763f4700 0 log_channel(cluster) log [INF] : pgmap v12983: 32 pgs: 32 active+clean; 70456 MB data, 134 GB used, 238 GB / 372 GB avail; 0 B/s rd, 2723 kB/s wr, 562 op/s 2017-03-02 15:08:59.469365 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14129 at 2017-03-02 15:08:59.469351) v1 -- 0x558a167453c0 con 0 2017-03-02 15:08:59.469382 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27367) finish_round waiting_for_readable 2017-03-02 15:08:59.469385 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27367) finish_round waiting_for_writeable 2017-03-02 15:08:59.469387 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26858..27367) finish_round done w/ waiters, state active 2017-03-02 15:08:59.469430 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14129 at 2017-03-02 15:08:59.469351) v1 ==== 0+0+0 (0 0 0) 0x558a167453c0 con 0x558a14482000 2017-03-02 15:08:59.469470 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:08:59.469479 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:08:59.469484 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13564) dispatch 0x558a167453c0 log(1 entries from seq 14129 at 2017-03-02 15:08:59.469351) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:08:59.469498 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27367) is_readable = 1 - now=2017-03-02 15:08:59.469498 lease_expire=0.000000 has v0 lc 27367 2017-03-02 15:08:59.469507 7f4f71beb700 10 mon.0@0(leader).log v13564 preprocess_query log(1 entries from seq 14129 at 2017-03-02 15:08:59.469351) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:59.469515 7f4f71beb700 10 mon.0@0(leader).log v13564 preprocess_log log(1 entries from seq 14129 at 2017-03-02 15:08:59.469351) v1 from mon.0 2017-03-02 15:08:59.469521 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:08:59.469524 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:08:59.469526 7f4f71beb700 20 allow all 2017-03-02 15:08:59.469535 7f4f71beb700 10 mon.0@0(leader).log v13564 prepare_update log(1 entries from seq 14129 at 2017-03-02 15:08:59.469351) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:08:59.469542 7f4f71beb700 10 mon.0@0(leader).log v13564 prepare_log log(1 entries from seq 14129 at 2017-03-02 15:08:59.469351) v1 from mon.0 2017-03-02 15:08:59.469546 7f4f71beb700 10 mon.0@0(leader).log v13564 logging 2017-03-02 15:08:59.469351 mon.0 172.21.15.17:6789/0 14129 : cluster [INF] pgmap v12983: 32 pgs: 32 active+clean; 70456 MB data, 134 GB used, 238 GB / 372 GB avail; 0 B/s rd, 2723 kB/s wr, 562 op/s 2017-03-02 15:08:59.469556 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13564) setting proposal_timer 0x558a1412cf00 with delay of 0.999193 2017-03-02 15:09:00.468865 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13564) propose_pending 2017-03-02 15:09:00.468895 7f4f743f0700 10 mon.0@0(leader).log v13564 encode_full log v 13564 2017-03-02 15:09:00.468968 7f4f743f0700 10 mon.0@0(leader).log v13564 encode_pending v13565 2017-03-02 15:09:00.468975 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26858..27367) queue_pending_finisher 0x558a1412cf10 2017-03-02 15:09:00.468979 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26858..27367) trigger_propose active, proposing now 2017-03-02 15:09:00.468985 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26858..27367) propose_pending 27368 10250 bytes 2017-03-02 15:09:00.468987 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26858..27367) begin for 27368 10250 bytes 2017-03-02 15:09:00.502065 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26858..27367) commit_start 27368 2017-03-02 15:09:00.535520 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26858..27367) commit_finish 27368 2017-03-02 15:09:00.535574 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:09:00.535617 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12983) refresh 2017-03-02 15:09:00.535651 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) refresh 2017-03-02 15:09:00.535694 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:09:00.535734 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13565) refresh 2017-03-02 15:09:00.535739 7f4f763f4700 10 mon.0@0(leader).log v13565 update_from_paxos 2017-03-02 15:09:00.535743 7f4f763f4700 10 mon.0@0(leader).log v13565 update_from_paxos version 13565 summary v 13564 2017-03-02 15:09:00.535753 7f4f763f4700 10 mon.0@0(leader).log v13565 update_from_paxos latest full 13564 2017-03-02 15:09:00.535770 7f4f763f4700 7 mon.0@0(leader).log v13565 update_from_paxos applying incremental log 13565 2017-03-02 15:08:59.469351 mon.0 172.21.15.17:6789/0 14129 : cluster [INF] pgmap v12983: 32 pgs: 32 active+clean; 70456 MB data, 134 GB used, 238 GB / 372 GB avail; 0 B/s rd, 2723 kB/s wr, 562 op/s 2017-03-02 15:09:00.535797 7f4f763f4700 20 mon.0@0(leader).log v13565 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:09:00.535852 7f4f763f4700 15 mon.0@0(leader).log v13565 update_from_paxos logging for 1 channels 2017-03-02 15:09:00.535857 7f4f763f4700 15 mon.0@0(leader).log v13565 update_from_paxos channel 'cluster' logging 198 bytes 2017-03-02 15:09:00.535981 7f4f763f4700 10 mon.0@0(leader).log v13565 check_subs 2017-03-02 15:09:00.536060 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:09:00.536107 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:09:00.536122 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:09:00.536159 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:09:00.536175 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12983) post_refresh 2017-03-02 15:09:00.536179 7f4f763f4700 10 mon.0@0(leader).pg v12983 post_paxos_update 2017-03-02 15:09:00.536184 7f4f763f4700 10 mon.0@0(leader).pg v12983 check_subs 2017-03-02 15:09:00.536188 7f4f763f4700 20 mon.0@0(leader).pg v12983 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:09:00.536200 7f4f763f4700 20 mon.0@0(leader).pg v12983 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:09:00.536206 7f4f763f4700 20 mon.0@0(leader).pg v12983 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:09:00.536212 7f4f763f4700 20 mon.0@0(leader).pg v12983 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:09:00.536218 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) post_refresh 2017-03-02 15:09:00.536221 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:09:00.536224 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13565) post_refresh 2017-03-02 15:09:00.536227 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:09:00.536229 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:09:00.536232 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:09:00.536236 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27368) commit_proposal 2017-03-02 15:09:00.536242 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13565) _active - not active 2017-03-02 15:09:00.536246 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27368) finish_round 2017-03-02 15:09:00.536249 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27368) finish_round waiting_for_acting 2017-03-02 15:09:00.536253 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13565) _active 2017-03-02 15:09:00.536256 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13565) _active creating new pending 2017-03-02 15:09:00.536268 7f4f763f4700 10 mon.0@0(leader).log v13565 create_pending v 13566 2017-03-02 15:09:00.536274 7f4f763f4700 7 mon.0@0(leader).log v13565 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:09:00.536282 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a15c73600 0x558a16108ee0 log(last 14129) v1 2017-03-02 15:09:00.536289 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14129) v1 -- 0x558a16108ee0 con 0 2017-03-02 15:09:00.536325 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27368) finish_round waiting_for_readable 2017-03-02 15:09:00.536330 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27368) finish_round waiting_for_writeable 2017-03-02 15:09:00.536334 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26858..27368) finish_round done w/ waiters, state active 2017-03-02 15:09:00.536393 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14129) v1 ==== 0+0+0 (0 0 0) 0x558a16108ee0 con 0x558a14482000 2017-03-02 15:09:00.536428 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:09:00.536440 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:00.536447 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:09:00.536452 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:00.536455 7f4f71beb700 20 allow all 2017-03-02 15:09:01.013212 7f4f743f0700 11 mon.0@0(leader) e2 tick 2017-03-02 15:09:01.013241 7f4f743f0700 10 mon.0@0(leader).pg v12983 v12983: 32 pgs: 32 active+clean; 70456 MB data, 134 GB used, 238 GB / 372 GB avail; 0 B/s rd, 2723 kB/s wr, 562 op/s 2017-03-02 15:09:01.013267 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12983) maybe_trim trim_to 12483 would only trim 214 < paxos_service_trim_min 250 2017-03-02 15:09:01.013273 7f4f743f0700 10 mon.0@0(leader).mds e737 e737: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:09:01.013320 7f4f743f0700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) maybe_trim trim_to 237 would only trim 236 < paxos_service_trim_min 250 2017-03-02 15:09:01.013326 7f4f743f0700 10 mon.0@0(leader).osd e7 e7: 4 osds: 4 up, 4 in 2017-03-02 15:09:01.013333 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26858..27368) is_readable = 1 - now=2017-03-02 15:09:01.013334 lease_expire=0.000000 has v0 lc 27368 2017-03-02 15:09:01.013345 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26858..27368) is_readable = 1 - now=2017-03-02 15:09:01.013346 lease_expire=0.000000 has v0 lc 27368 2017-03-02 15:09:01.013355 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26858..27368) is_readable = 1 - now=2017-03-02 15:09:01.013356 lease_expire=0.000000 has v0 lc 27368 2017-03-02 15:09:01.013362 7f4f743f0700 10 mon.0@0(leader).osd e7 min_last_epoch_clean 6 2017-03-02 15:09:01.013366 7f4f743f0700 10 mon.0@0(leader).log v13565 log 2017-03-02 15:09:01.013369 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13565) maybe_trim trim_to 13065 would only trim 38 < paxos_service_trim_min 250 2017-03-02 15:09:01.013375 7f4f743f0700 10 mon.0@0(leader).auth v10 auth 2017-03-02 15:09:01.013381 7f4f743f0700 10 cephx keyserver: _check_rotating_secrets 2017-03-02 15:09:01.013391 7f4f743f0700 20 mon.0@0(leader) e2 sync_trim_providers 2017-03-02 15:09:02.637859 7f4f74bf1700 1 -- 172.21.15.17:6789/0 >> 172.21.15.22:6805/2203629154 conn(0x558a1492a000 :6789 s=STATE_OPEN pgs=1 cs=1 l=1).read_bulk peer close file descriptor 39 2017-03-02 15:09:02.637884 7f4f74bf1700 1 -- 172.21.15.17:6789/0 >> 172.21.15.22:6805/2203629154 conn(0x558a1492a000 :6789 s=STATE_OPEN pgs=1 cs=1 l=1).read_until read failed 2017-03-02 15:09:02.637887 7f4f74bf1700 1 -- 172.21.15.17:6789/0 >> 172.21.15.22:6805/2203629154 conn(0x558a1492a000 :6789 s=STATE_OPEN pgs=1 cs=1 l=1).process read tag failed 2017-03-02 15:09:02.637890 7f4f74bf1700 1 -- 172.21.15.17:6789/0 >> 172.21.15.22:6805/2203629154 conn(0x558a1492a000 :6789 s=STATE_OPEN pgs=1 cs=1 l=1).fault on lossy channel, failing 2017-03-02 15:09:02.637980 7f4f71beb700 10 mon.0@0(leader) e2 ms_handle_reset 0x558a1492a000 172.21.15.22:6805/2203629154 2017-03-02 15:09:02.637992 7f4f71beb700 10 mon.0@0(leader) e2 reset/close on session mds.? 172.21.15.22:6805/2203629154 2017-03-02 15:09:02.637995 7f4f71beb700 10 mon.0@0(leader) e2 remove_session 0x558a149195a0 mds.? 172.21.15.22:6805/2203629154 2017-03-02 15:09:02.638285 7f4f74bf1700 1 -- 172.21.15.17:6789/0 >> - conn(0x558a15290000 :6789 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=9 - 2017-03-02 15:09:02.638491 7f4f74bf1700 10 mon.0@0(leader) e2 ms_verify_authorizer 172.21.15.22:6805/2203629154 mds protocol 0 2017-03-02 15:09:02.638518 7f4f74bf1700 10 In get_auth_session_handler for protocol 0 2017-03-02 15:09:02.638698 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.0 172.21.15.22:6805/2203629154 1 ==== auth(proto 0 26 bytes epoch 2) v1 ==== 56+0+0 (1275721920 0 0) 0x558a16743d40 con 0x558a15290000 2017-03-02 15:09:02.638722 7f4f71beb700 10 mon.0@0(leader) e2 _ms_dispatch new session 0x558a1636a580 MonSession(mds.0 172.21.15.22:6805/2203629154 is open) 2017-03-02 15:09:02.638729 7f4f71beb700 20 mon.0@0(leader) e2 caps 2017-03-02 15:09:02.638733 7f4f71beb700 10 mon.0@0(leader).paxosservice(auth 1..10) dispatch 0x558a16743d40 auth(proto 0 26 bytes epoch 2) v1 from mds.0 172.21.15.22:6805/2203629154 con 0x558a15290000 2017-03-02 15:09:02.638739 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27368) is_readable = 1 - now=2017-03-02 15:09:02.638740 lease_expire=0.000000 has v0 lc 27368 2017-03-02 15:09:02.638746 7f4f71beb700 10 mon.0@0(leader).auth v10 preprocess_query auth(proto 0 26 bytes epoch 2) v1 from mds.0 172.21.15.22:6805/2203629154 2017-03-02 15:09:02.638752 7f4f71beb700 10 mon.0@0(leader).auth v10 prep_auth() blob_size=26 2017-03-02 15:09:02.638980 7f4f71beb700 10 cephx server mds.d: start_session server_challenge a75aa997631b1650 2017-03-02 15:09:02.638995 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a16404e00 0x558a16744640 auth_reply(proto 2 0 (0) Success) v1 2017-03-02 15:09:02.639006 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6805/2203629154 -- auth_reply(proto 2 0 (0) Success) v1 -- 0x558a16744640 con 0 2017-03-02 15:09:02.645014 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.5 172.21.15.11:6804/1637448123 8794 ==== mds_load_targets(4116 1,2,3,4) v1 ==== 46+0+0 (3047058345 0 0) 0x558a16744f40 con 0x558a1471d800 2017-03-02 15:09:02.645044 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b0f20 for mds.? 172.21.15.11:6804/1637448123 2017-03-02 15:09:02.645049 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:02.645052 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) dispatch 0x558a16744f40 mds_load_targets(4116 1,2,3,4) v1 from mds.5 172.21.15.11:6804/1637448123 con 0x558a1471d800 2017-03-02 15:09:02.645058 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27368) is_readable = 1 - now=2017-03-02 15:09:02.645058 lease_expire=0.000000 has v0 lc 27368 2017-03-02 15:09:02.645065 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_query mds_load_targets(4116 1,2,3,4) v1 from mds.5 172.21.15.11:6804/1637448123 2017-03-02 15:09:02.645071 7f4f71beb700 10 mon.0@0(leader).mds e737 preprocess_offload_targets mds_load_targets(4116 1,2,3,4) v1 from mds.5 2017-03-02 15:09:02.645074 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:09:02.645076 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:02.645077 7f4f71beb700 20 allow all 2017-03-02 15:09:02.645082 7f4f71beb700 7 mon.0@0(leader).mds e737 prepare_update mds_load_targets(4116 1,2,3,4) v1 2017-03-02 15:09:02.645085 7f4f71beb700 10 mon.0@0(leader).mds e737 prepare_offload_targets 4116 1,2,3,4 2017-03-02 15:09:02.645091 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) setting proposal_timer 0x558a1412fa30 with delay of 0.05 2017-03-02 15:09:02.648702 7f4f74bf1700 1 -- 172.21.15.17:6789/0 >> 172.21.15.22:6805/2203629154 conn(0x558a15290000 :6789 s=STATE_OPEN pgs=16 cs=1 l=1).read_bulk peer close file descriptor 9 2017-03-02 15:09:02.648721 7f4f74bf1700 1 -- 172.21.15.17:6789/0 >> 172.21.15.22:6805/2203629154 conn(0x558a15290000 :6789 s=STATE_OPEN pgs=16 cs=1 l=1).read_until read failed 2017-03-02 15:09:02.648725 7f4f74bf1700 1 -- 172.21.15.17:6789/0 >> 172.21.15.22:6805/2203629154 conn(0x558a15290000 :6789 s=STATE_OPEN pgs=16 cs=1 l=1).process read tag failed 2017-03-02 15:09:02.648729 7f4f74bf1700 1 -- 172.21.15.17:6789/0 >> 172.21.15.22:6805/2203629154 conn(0x558a15290000 :6789 s=STATE_OPEN pgs=16 cs=1 l=1).fault on lossy channel, failing 2017-03-02 15:09:02.648751 7f4f71beb700 10 mon.0@0(leader) e2 ms_handle_reset 0x558a15290000 172.21.15.22:6805/2203629154 2017-03-02 15:09:02.648761 7f4f71beb700 10 mon.0@0(leader) e2 reset/close on session mds.0 172.21.15.22:6805/2203629154 2017-03-02 15:09:02.648764 7f4f71beb700 10 mon.0@0(leader) e2 remove_session 0x558a1636a580 mds.0 172.21.15.22:6805/2203629154 2017-03-02 15:09:02.695168 7f4f743f0700 10 mon.0@0(leader).paxosservice(mdsmap 1..737) propose_pending 2017-03-02 15:09:02.695208 7f4f743f0700 10 mon.0@0(leader).mds e737 encode_pending e738 2017-03-02 15:09:02.695276 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26858..27368) queue_pending_finisher 0x558a1412ee20 2017-03-02 15:09:02.695287 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26858..27368) trigger_propose active, proposing now 2017-03-02 15:09:02.695296 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26858..27368) propose_pending 27369 2417 bytes 2017-03-02 15:09:02.695299 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26858..27368) begin for 27369 2417 bytes 2017-03-02 15:09:02.727072 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26858..27368) commit_start 27369 2017-03-02 15:09:02.737930 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26858..27368) commit_finish 27369 2017-03-02 15:09:02.737972 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:09:02.737991 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12983) refresh 2017-03-02 15:09:02.738004 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..738) refresh 2017-03-02 15:09:02.738006 7f4f763f4700 10 mon.0@0(leader).mds e737 update_from_paxos version 738, my e 737 2017-03-02 15:09:02.738014 7f4f763f4700 10 mon.0@0(leader).mds e737 update_from_paxos got 738 2017-03-02 15:09:02.738059 7f4f763f4700 4 mon.0@0(leader).mds e738 new map 2017-03-02 15:09:02.738068 7f4f763f4700 0 mon.0@0(leader).mds e738 print_map 2017-03-02 15:09:02.738119 7f4f763f4700 20 mon.0@0(leader).mds e738 check_sub: mdsmap 2017-03-02 15:09:02.738125 7f4f763f4700 10 mon.0@0(leader).mds e738 check_sub: is_mds=1, fscid= 1 2017-03-02 15:09:02.738128 7f4f763f4700 10 mon.0@0(leader).mds e738 check_sub selected MDS map epoch 738 for namespace 1 for subscriber mds.? who wants epoch 738 2017-03-02 15:09:02.738146 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6804/2954782670 -- mdsmap(e 738) v1 -- 0x558a167453c0 con 0 2017-03-02 15:09:02.738173 7f4f763f4700 20 mon.0@0(leader).mds e738 check_sub: mdsmap 2017-03-02 15:09:02.738189 7f4f763f4700 10 mon.0@0(leader).mds e738 check_sub: is_mds=1, fscid= 1 2017-03-02 15:09:02.738193 7f4f763f4700 10 mon.0@0(leader).mds e738 check_sub selected MDS map epoch 738 for namespace 1 for subscriber mds.? who wants epoch 738 2017-03-02 15:09:02.738213 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6804/1637448123 -- mdsmap(e 738) v1 -- 0x558a16745600 con 0 2017-03-02 15:09:02.738242 7f4f763f4700 20 mon.0@0(leader).mds e738 check_sub: mdsmap 2017-03-02 15:09:02.738252 7f4f763f4700 10 mon.0@0(leader).mds e738 check_sub: is_mds=1, fscid= 1 2017-03-02 15:09:02.738256 7f4f763f4700 10 mon.0@0(leader).mds e738 check_sub selected MDS map epoch 738 for namespace 1 for subscriber mds.? who wants epoch 738 2017-03-02 15:09:02.738277 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6805/2033388711 -- mdsmap(e 738) v1 -- 0x558a16743f80 con 0 2017-03-02 15:09:02.738302 7f4f763f4700 20 mon.0@0(leader).mds e738 check_sub: mdsmap 2017-03-02 15:09:02.738315 7f4f763f4700 10 mon.0@0(leader).mds e738 check_sub: is_mds=1, fscid= 1 2017-03-02 15:09:02.738318 7f4f763f4700 10 mon.0@0(leader).mds e738 check_sub selected MDS map epoch 738 for namespace 1 for subscriber mds.? who wants epoch 738 2017-03-02 15:09:02.738330 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6805/1926402648 -- mdsmap(e 738) v1 -- 0x558a16743b00 con 0 2017-03-02 15:09:02.738345 7f4f763f4700 20 mon.0@0(leader).mds e738 check_sub: mdsmap 2017-03-02 15:09:02.738351 7f4f763f4700 10 mon.0@0(leader).mds e738 check_sub: is_mds=1, fscid= 1 2017-03-02 15:09:02.738354 7f4f763f4700 10 mon.0@0(leader).mds e738 check_sub selected MDS map epoch 738 for namespace 1 for subscriber mds.? who wants epoch 738 2017-03-02 15:09:02.738374 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6804/673088617 -- mdsmap(e 738) v1 -- 0x558a16745840 con 0 2017-03-02 15:09:02.738391 7f4f763f4700 20 mon.0@0(leader).mds e738 check_sub: mdsmap 2017-03-02 15:09:02.738398 7f4f763f4700 10 mon.0@0(leader).mds e738 check_sub: is_mds=1, fscid= 1 2017-03-02 15:09:02.738401 7f4f763f4700 10 mon.0@0(leader).mds e738 check_sub selected MDS map epoch 738 for namespace 1 for subscriber mds.? who wants epoch 738 2017-03-02 15:09:02.738421 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6804/2576824570 -- mdsmap(e 738) v1 -- 0x558a16745a80 con 0 2017-03-02 15:09:02.738440 7f4f763f4700 20 mon.0@0(leader).mds e738 check_sub: mdsmap 2017-03-02 15:09:02.738442 7f4f763f4700 10 mon.0@0(leader).mds e738 check_sub: is_mds=0, fscid= 1 2017-03-02 15:09:02.738444 7f4f763f4700 10 mon.0@0(leader).mds e738 check_sub selected MDS map epoch 738 for namespace 1 for subscriber unknown.0 who wants epoch 738 2017-03-02 15:09:02.738457 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.132:0/2597832251 -- mdsmap(e 738) v1 -- 0x558a16745cc0 con 0 2017-03-02 15:09:02.738467 7f4f763f4700 20 mon.0@0(leader).mds e738 check_sub: mdsmap 2017-03-02 15:09:02.738468 7f4f763f4700 10 mon.0@0(leader).mds e738 check_sub: is_mds=0, fscid= 1 2017-03-02 15:09:02.738470 7f4f763f4700 10 mon.0@0(leader).mds e738 check_sub selected MDS map epoch 738 for namespace 1 for subscriber unknown.0 who wants epoch 738 2017-03-02 15:09:02.738505 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.133:0/3792964262 -- mdsmap(e 738) v1 -- 0x558a14d60000 con 0 2017-03-02 15:09:02.738523 7f4f763f4700 10 mon.0@0(leader).mds e738 update_logger 2017-03-02 15:09:02.738594 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:09:02.738622 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13565) refresh 2017-03-02 15:09:02.738625 7f4f763f4700 10 mon.0@0(leader).log v13565 update_from_paxos 2017-03-02 15:09:02.738628 7f4f763f4700 10 mon.0@0(leader).log v13565 update_from_paxos version 13565 summary v 13565 2017-03-02 15:09:02.738650 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:09:02.738674 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:09:02.738677 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:09:02.738700 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:09:02.738702 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12983) post_refresh 2017-03-02 15:09:02.738704 7f4f763f4700 10 mon.0@0(leader).pg v12983 post_paxos_update 2017-03-02 15:09:02.738706 7f4f763f4700 10 mon.0@0(leader).pg v12983 check_subs 2017-03-02 15:09:02.738709 7f4f763f4700 20 mon.0@0(leader).pg v12983 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:09:02.738743 7f4f763f4700 20 mon.0@0(leader).pg v12983 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:09:02.738746 7f4f763f4700 20 mon.0@0(leader).pg v12983 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:09:02.738749 7f4f763f4700 20 mon.0@0(leader).pg v12983 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:09:02.738752 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..738) post_refresh 2017-03-02 15:09:02.738753 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:09:02.738755 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13565) post_refresh 2017-03-02 15:09:02.738756 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:09:02.738757 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:09:02.738758 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:09:02.738760 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27369) commit_proposal 2017-03-02 15:09:02.738764 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..738) _active - not active 2017-03-02 15:09:02.738766 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27369) finish_round 2017-03-02 15:09:02.738768 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27369) finish_round waiting_for_acting 2017-03-02 15:09:02.738770 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..738) _active 2017-03-02 15:09:02.738771 7f4f763f4700 7 mon.0@0(leader).paxosservice(mdsmap 1..738) _active creating new pending 2017-03-02 15:09:02.738788 7f4f763f4700 10 mon.0@0(leader).mds e738 create_pending e739 2017-03-02 15:09:02.738791 7f4f763f4700 10 mon.0@0(leader).mds e738 e738: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:09:02.738810 7f4f763f4700 10 mon.0@0(leader).mds e738 no beacon from 4121 172.21.15.22:6805/2203629154 mds.0.6 up:active since 2017-03-02 15:08:46.885540 2017-03-02 15:09:02.738817 7f4f763f4700 10 mon.0@0(leader).mds e738 marking 4121 172.21.15.22:6805/2203629154 mds.0.6 up:active laggy 2017-03-02 15:09:02.738855 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..738) propose_pending 2017-03-02 15:09:02.738869 7f4f763f4700 10 mon.0@0(leader).mds e738 encode_pending e739 2017-03-02 15:09:02.738914 7f4f763f4700 5 mon.0@0(leader).paxos(paxos active c 26858..27369) queue_pending_finisher 0x558a1412ee20 2017-03-02 15:09:02.738917 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26858..27369) trigger_propose active, proposing now 2017-03-02 15:09:02.738926 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26858..27369) propose_pending 27370 2417 bytes 2017-03-02 15:09:02.738928 7f4f763f4700 10 mon.0@0(leader).paxos(paxos updating c 26858..27369) begin for 27370 2417 bytes 2017-03-02 15:09:02.768913 7f4f763f4700 10 mon.0@0(leader).paxos(paxos updating c 26858..27369) commit_start 27370 2017-03-02 15:09:02.768961 7f4f763f4700 10 mon.0@0(leader).mds e738 update_logger 2017-03-02 15:09:02.768985 7f4f763f4700 0 log_channel(cluster) log [INF] : fsmap e738: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:09:02.769001 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14130 at 2017-03-02 15:09:02.768986) v1 -- 0x558a14d60480 con 0 2017-03-02 15:09:02.769018 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26858..27369) finish_round waiting_for_readable 2017-03-02 15:09:02.769026 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26858..27369) finish_round waiting_for_writeable 2017-03-02 15:09:02.769028 7f4f763f4700 10 mon.0@0(leader).paxos(paxos writing c 26858..27369) finish_round done w/ waiters, state writing 2017-03-02 15:09:02.769061 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14130 at 2017-03-02 15:09:02.768986) v1 ==== 0+0+0 (0 0 0) 0x558a14d60480 con 0x558a14482000 2017-03-02 15:09:02.769093 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:09:02.769099 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:02.769102 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13565) dispatch 0x558a14d60480 log(1 entries from seq 14130 at 2017-03-02 15:09:02.768986) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:09:02.769111 7f4f71beb700 5 mon.0@0(leader).paxos(paxos writing c 26858..27369) is_readable = 1 - now=2017-03-02 15:09:02.769111 lease_expire=0.000000 has v0 lc 27369 2017-03-02 15:09:02.769118 7f4f71beb700 10 mon.0@0(leader).log v13565 preprocess_query log(1 entries from seq 14130 at 2017-03-02 15:09:02.768986) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:09:02.769126 7f4f71beb700 10 mon.0@0(leader).log v13565 preprocess_log log(1 entries from seq 14130 at 2017-03-02 15:09:02.768986) v1 from mon.0 2017-03-02 15:09:02.769130 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:09:02.769133 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:02.769134 7f4f71beb700 20 allow all 2017-03-02 15:09:02.769143 7f4f71beb700 10 mon.0@0(leader).log v13565 prepare_update log(1 entries from seq 14130 at 2017-03-02 15:09:02.768986) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:09:02.769149 7f4f71beb700 10 mon.0@0(leader).log v13565 prepare_log log(1 entries from seq 14130 at 2017-03-02 15:09:02.768986) v1 from mon.0 2017-03-02 15:09:02.769152 7f4f71beb700 10 mon.0@0(leader).log v13565 logging 2017-03-02 15:09:02.768986 mon.0 172.21.15.17:6789/0 14130 : cluster [INF] fsmap e738: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:09:02.769161 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13565) setting proposal_timer 0x558a1412fe50 with delay of 0.968788 2017-03-02 15:09:02.779529 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26858..27369) commit_finish 27370 2017-03-02 15:09:02.779584 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:09:02.779614 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12983) refresh 2017-03-02 15:09:02.779637 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..739) refresh 2017-03-02 15:09:02.779640 7f4f763f4700 10 mon.0@0(leader).mds e738 update_from_paxos version 739, my e 738 2017-03-02 15:09:02.779652 7f4f763f4700 10 mon.0@0(leader).mds e738 update_from_paxos got 739 2017-03-02 15:09:02.779707 7f4f763f4700 4 mon.0@0(leader).mds e739 new map 2017-03-02 15:09:02.779721 7f4f763f4700 0 mon.0@0(leader).mds e739 print_map 2017-03-02 15:09:02.779805 7f4f763f4700 20 mon.0@0(leader).mds e739 check_sub: mdsmap 2017-03-02 15:09:02.779857 7f4f763f4700 10 mon.0@0(leader).mds e739 check_sub: is_mds=1, fscid= 1 2017-03-02 15:09:02.779861 7f4f763f4700 10 mon.0@0(leader).mds e739 check_sub selected MDS map epoch 739 for namespace 1 for subscriber mds.? who wants epoch 739 2017-03-02 15:09:02.779876 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6804/2954782670 -- mdsmap(e 739) v1 -- 0x558a14d60240 con 0 2017-03-02 15:09:02.779897 7f4f763f4700 20 mon.0@0(leader).mds e739 check_sub: mdsmap 2017-03-02 15:09:02.779904 7f4f763f4700 10 mon.0@0(leader).mds e739 check_sub: is_mds=1, fscid= 1 2017-03-02 15:09:02.779907 7f4f763f4700 10 mon.0@0(leader).mds e739 check_sub selected MDS map epoch 739 for namespace 1 for subscriber mds.? who wants epoch 739 2017-03-02 15:09:02.779923 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6804/1637448123 -- mdsmap(e 739) v1 -- 0x558a14d606c0 con 0 2017-03-02 15:09:02.779946 7f4f763f4700 20 mon.0@0(leader).mds e739 check_sub: mdsmap 2017-03-02 15:09:02.779960 7f4f763f4700 10 mon.0@0(leader).mds e739 check_sub: is_mds=1, fscid= 1 2017-03-02 15:09:02.779964 7f4f763f4700 10 mon.0@0(leader).mds e739 check_sub selected MDS map epoch 739 for namespace 1 for subscriber mds.? who wants epoch 739 2017-03-02 15:09:02.779981 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6805/2033388711 -- mdsmap(e 739) v1 -- 0x558a14d60900 con 0 2017-03-02 15:09:02.779999 7f4f763f4700 20 mon.0@0(leader).mds e739 check_sub: mdsmap 2017-03-02 15:09:02.780003 7f4f763f4700 10 mon.0@0(leader).mds e739 check_sub: is_mds=1, fscid= 1 2017-03-02 15:09:02.780005 7f4f763f4700 10 mon.0@0(leader).mds e739 check_sub selected MDS map epoch 739 for namespace 1 for subscriber mds.? who wants epoch 739 2017-03-02 15:09:02.780015 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6805/1926402648 -- mdsmap(e 739) v1 -- 0x558a14d60b40 con 0 2017-03-02 15:09:02.780026 7f4f763f4700 20 mon.0@0(leader).mds e739 check_sub: mdsmap 2017-03-02 15:09:02.780031 7f4f763f4700 10 mon.0@0(leader).mds e739 check_sub: is_mds=1, fscid= 1 2017-03-02 15:09:02.780033 7f4f763f4700 10 mon.0@0(leader).mds e739 check_sub selected MDS map epoch 739 for namespace 1 for subscriber mds.? who wants epoch 739 2017-03-02 15:09:02.780048 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6804/673088617 -- mdsmap(e 739) v1 -- 0x558a14d60d80 con 0 2017-03-02 15:09:02.780067 7f4f763f4700 20 mon.0@0(leader).mds e739 check_sub: mdsmap 2017-03-02 15:09:02.780074 7f4f763f4700 10 mon.0@0(leader).mds e739 check_sub: is_mds=1, fscid= 1 2017-03-02 15:09:02.780077 7f4f763f4700 10 mon.0@0(leader).mds e739 check_sub selected MDS map epoch 739 for namespace 1 for subscriber mds.? who wants epoch 739 2017-03-02 15:09:02.780093 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6804/2576824570 -- mdsmap(e 739) v1 -- 0x558a14d60fc0 con 0 2017-03-02 15:09:02.780111 7f4f763f4700 20 mon.0@0(leader).mds e739 check_sub: mdsmap 2017-03-02 15:09:02.780113 7f4f763f4700 10 mon.0@0(leader).mds e739 check_sub: is_mds=0, fscid= 1 2017-03-02 15:09:02.780115 7f4f763f4700 10 mon.0@0(leader).mds e739 check_sub selected MDS map epoch 739 for namespace 1 for subscriber unknown.0 who wants epoch 739 2017-03-02 15:09:02.780125 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.132:0/2597832251 -- mdsmap(e 739) v1 -- 0x558a14d61200 con 0 2017-03-02 15:09:02.780133 7f4f763f4700 20 mon.0@0(leader).mds e739 check_sub: mdsmap 2017-03-02 15:09:02.780134 7f4f763f4700 10 mon.0@0(leader).mds e739 check_sub: is_mds=0, fscid= 1 2017-03-02 15:09:02.780136 7f4f763f4700 10 mon.0@0(leader).mds e739 check_sub selected MDS map epoch 739 for namespace 1 for subscriber unknown.0 who wants epoch 739 2017-03-02 15:09:02.780144 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.133:0/3792964262 -- mdsmap(e 739) v1 -- 0x558a14d61440 con 0 2017-03-02 15:09:02.780156 7f4f763f4700 10 mon.0@0(leader).mds e739 update_logger 2017-03-02 15:09:02.780183 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:09:02.780234 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13565) refresh 2017-03-02 15:09:02.780240 7f4f763f4700 10 mon.0@0(leader).log v13565 update_from_paxos 2017-03-02 15:09:02.780242 7f4f763f4700 10 mon.0@0(leader).log v13565 update_from_paxos version 13565 summary v 13565 2017-03-02 15:09:02.780291 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:09:02.780310 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:09:02.780313 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:09:02.780335 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:09:02.780338 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12983) post_refresh 2017-03-02 15:09:02.780340 7f4f763f4700 10 mon.0@0(leader).pg v12983 post_paxos_update 2017-03-02 15:09:02.780342 7f4f763f4700 10 mon.0@0(leader).pg v12983 check_subs 2017-03-02 15:09:02.780345 7f4f763f4700 20 mon.0@0(leader).pg v12983 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:09:02.780352 7f4f763f4700 20 mon.0@0(leader).pg v12983 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:09:02.780356 7f4f763f4700 20 mon.0@0(leader).pg v12983 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:09:02.780359 7f4f763f4700 20 mon.0@0(leader).pg v12983 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:09:02.780362 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..739) post_refresh 2017-03-02 15:09:02.780364 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:09:02.780365 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13565) post_refresh 2017-03-02 15:09:02.780367 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:09:02.780368 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:09:02.780369 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:09:02.780372 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27370) commit_proposal 2017-03-02 15:09:02.780376 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..739) _active - not active 2017-03-02 15:09:02.780377 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27370) finish_round 2017-03-02 15:09:02.780379 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27370) finish_round waiting_for_acting 2017-03-02 15:09:02.780380 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..739) _active 2017-03-02 15:09:02.780381 7f4f763f4700 7 mon.0@0(leader).paxosservice(mdsmap 1..739) _active creating new pending 2017-03-02 15:09:02.780395 7f4f763f4700 10 mon.0@0(leader).mds e739 create_pending e740 2017-03-02 15:09:02.780397 7f4f763f4700 10 mon.0@0(leader).mds e739 e739: 7/7/7 up {0=d=up:active(laggy or crashed),1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:09:02.780411 7f4f763f4700 10 mon.0@0(leader).mds e739 no beacon from 4121 172.21.15.22:6805/2203629154 mds.0.6 up:active since 2017-03-02 15:08:46.885540 2017-03-02 15:09:02.780417 7f4f763f4700 10 mon.0@0(leader).mds e739 update_logger 2017-03-02 15:09:02.780427 7f4f763f4700 0 log_channel(cluster) log [INF] : fsmap e739: 7/7/7 up {0=d=up:active(laggy or crashed),1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:09:02.780436 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14131 at 2017-03-02 15:09:02.780429) v1 -- 0x558a14d618c0 con 0 2017-03-02 15:09:02.780451 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27370) finish_round waiting_for_readable 2017-03-02 15:09:02.780453 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27370) finish_round waiting_for_writeable 2017-03-02 15:09:02.780454 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26858..27370) finish_round done w/ waiters, state active 2017-03-02 15:09:02.780498 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14131 at 2017-03-02 15:09:02.780429) v1 ==== 0+0+0 (0 0 0) 0x558a14d618c0 con 0x558a14482000 2017-03-02 15:09:02.780524 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:09:02.780529 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:02.780532 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13565) dispatch 0x558a14d618c0 log(1 entries from seq 14131 at 2017-03-02 15:09:02.780429) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:09:02.780539 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27370) is_readable = 1 - now=2017-03-02 15:09:02.780540 lease_expire=0.000000 has v0 lc 27370 2017-03-02 15:09:02.780547 7f4f71beb700 10 mon.0@0(leader).log v13565 preprocess_query log(1 entries from seq 14131 at 2017-03-02 15:09:02.780429) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:09:02.780554 7f4f71beb700 10 mon.0@0(leader).log v13565 preprocess_log log(1 entries from seq 14131 at 2017-03-02 15:09:02.780429) v1 from mon.0 2017-03-02 15:09:02.780562 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:09:02.780565 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:02.780566 7f4f71beb700 20 allow all 2017-03-02 15:09:02.780573 7f4f71beb700 10 mon.0@0(leader).log v13565 prepare_update log(1 entries from seq 14131 at 2017-03-02 15:09:02.780429) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:09:02.780579 7f4f71beb700 10 mon.0@0(leader).log v13565 prepare_log log(1 entries from seq 14131 at 2017-03-02 15:09:02.780429) v1 from mon.0 2017-03-02 15:09:02.780584 7f4f71beb700 10 mon.0@0(leader).log v13565 logging 2017-03-02 15:09:02.780429 mon.0 172.21.15.17:6789/0 14131 : cluster [INF] fsmap e739: 7/7/7 up {0=d=up:active(laggy or crashed),1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:09:02.780594 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13565) proposal_timer already set 2017-03-02 15:09:02.871073 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.5 172.21.15.11:6804/1637448123 8795 ==== mdsbeacon(4116/e up:active seq 8507 v739) v7 ==== 365+0+0 (1110573420 0 0) 0x558a15204d00 con 0x558a1471d800 2017-03-02 15:09:02.871110 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b0f20 for mds.? 172.21.15.11:6804/1637448123 2017-03-02 15:09:02.871115 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:02.871118 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..739) dispatch 0x558a15204d00 mdsbeacon(4116/e up:active seq 8507 v739) v7 from mds.5 172.21.15.11:6804/1637448123 con 0x558a1471d800 2017-03-02 15:09:02.871126 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27370) is_readable = 1 - now=2017-03-02 15:09:02.871126 lease_expire=0.000000 has v0 lc 27370 2017-03-02 15:09:02.871135 7f4f71beb700 10 mon.0@0(leader).mds e739 preprocess_query mdsbeacon(4116/e up:active seq 8507 v739) v7 from mds.5 172.21.15.11:6804/1637448123 2017-03-02 15:09:02.871141 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:09:02.871143 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:02.871144 7f4f71beb700 20 allow all 2017-03-02 15:09:02.871146 7f4f71beb700 12 mon.0@0(leader).mds e739 preprocess_beacon mdsbeacon(4116/e up:active seq 8507 v739) v7 from mds.5 172.21.15.11:6804/1637448123 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:09:02.871153 7f4f71beb700 10 mon.0@0(leader).mds e739 preprocess_beacon: GID exists in map: 4116 2017-03-02 15:09:02.871158 7f4f71beb700 15 mon.0@0(leader).mds e739 _note_beacon mdsbeacon(4116/e up:active seq 8507 v739) v7 noting time 2017-03-02 15:09:02.871161 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a14921b00 0x558a143e49c0 mdsbeacon(4116/e up:active seq 8507 v739) v7 2017-03-02 15:09:02.871164 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6804/1637448123 -- mdsbeacon(4116/e up:active seq 8507 v739) v7 -- 0x558a143e49c0 con 0 2017-03-02 15:09:02.884688 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.3 172.21.15.11:6805/1926402648 8570 ==== mdsbeacon(4118/f up:active seq 8507 v739) v7 ==== 406+0+0 (2952807142 0 0) 0x558a143e7740 con 0x558a14905800 2017-03-02 15:09:02.884721 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b2260 for mds.? 172.21.15.11:6805/1926402648 2017-03-02 15:09:02.884727 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:02.884731 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..739) dispatch 0x558a143e7740 mdsbeacon(4118/f up:active seq 8507 v739) v7 from mds.3 172.21.15.11:6805/1926402648 con 0x558a14905800 2017-03-02 15:09:02.884738 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27370) is_readable = 1 - now=2017-03-02 15:09:02.884738 lease_expire=0.000000 has v0 lc 27370 2017-03-02 15:09:02.884747 7f4f71beb700 10 mon.0@0(leader).mds e739 preprocess_query mdsbeacon(4118/f up:active seq 8507 v739) v7 from mds.3 172.21.15.11:6805/1926402648 2017-03-02 15:09:02.884754 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:09:02.884756 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:02.884758 7f4f71beb700 20 allow all 2017-03-02 15:09:02.884759 7f4f71beb700 12 mon.0@0(leader).mds e739 preprocess_beacon mdsbeacon(4118/f up:active seq 8507 v739) v7 from mds.3 172.21.15.11:6805/1926402648 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:09:02.884768 7f4f71beb700 10 mon.0@0(leader).mds e739 preprocess_beacon: GID exists in map: 4118 2017-03-02 15:09:02.884773 7f4f71beb700 15 mon.0@0(leader).mds e739 _note_beacon mdsbeacon(4118/f up:active seq 8507 v739) v7 noting time 2017-03-02 15:09:02.884778 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a14921b00 0x558a15204d00 mdsbeacon(4118/f up:active seq 8507 v739) v7 2017-03-02 15:09:02.884781 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6805/1926402648 -- mdsbeacon(4118/f up:active seq 8507 v739) v7 -- 0x558a15204d00 con 0 2017-03-02 15:09:02.885595 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.2 172.21.15.22:6804/673088617 8569 ==== mdsbeacon(4119/c up:active seq 8507 v739) v7 ==== 365+0+0 (1646986546 0 0) 0x558a143e49c0 con 0x558a14907000 2017-03-02 15:09:02.885627 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14916f20 for mds.? 172.21.15.22:6804/673088617 2017-03-02 15:09:02.885635 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:02.885638 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..739) dispatch 0x558a143e49c0 mdsbeacon(4119/c up:active seq 8507 v739) v7 from mds.2 172.21.15.22:6804/673088617 con 0x558a14907000 2017-03-02 15:09:02.885644 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27370) is_readable = 1 - now=2017-03-02 15:09:02.885645 lease_expire=0.000000 has v0 lc 27370 2017-03-02 15:09:02.885654 7f4f71beb700 10 mon.0@0(leader).mds e739 preprocess_query mdsbeacon(4119/c up:active seq 8507 v739) v7 from mds.2 172.21.15.22:6804/673088617 2017-03-02 15:09:02.885659 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:09:02.885661 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:02.885662 7f4f71beb700 20 allow all 2017-03-02 15:09:02.885663 7f4f71beb700 12 mon.0@0(leader).mds e739 preprocess_beacon mdsbeacon(4119/c up:active seq 8507 v739) v7 from mds.2 172.21.15.22:6804/673088617 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:09:02.885669 7f4f71beb700 10 mon.0@0(leader).mds e739 preprocess_beacon: GID exists in map: 4119 2017-03-02 15:09:02.885672 7f4f71beb700 15 mon.0@0(leader).mds e739 _note_beacon mdsbeacon(4119/c up:active seq 8507 v739) v7 noting time 2017-03-02 15:09:02.885675 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a14921b00 0x558a143e7740 mdsbeacon(4119/c up:active seq 8507 v739) v7 2017-03-02 15:09:02.885677 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6804/673088617 -- mdsbeacon(4119/c up:active seq 8507 v739) v7 -- 0x558a143e7740 con 0 2017-03-02 15:09:02.924981 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.1 172.21.15.17:6804/2576824570 8566 ==== mdsbeacon(4120/g up:active seq 8507 v739) v7 ==== 365+0+0 (2854504489 0 0) 0x558a154b8d00 con 0x558a14908800 2017-03-02 15:09:02.925011 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14917600 for mds.? 172.21.15.17:6804/2576824570 2017-03-02 15:09:02.925016 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:02.925019 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..739) dispatch 0x558a154b8d00 mdsbeacon(4120/g up:active seq 8507 v739) v7 from mds.1 172.21.15.17:6804/2576824570 con 0x558a14908800 2017-03-02 15:09:02.925024 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27370) is_readable = 1 - now=2017-03-02 15:09:02.925025 lease_expire=0.000000 has v0 lc 27370 2017-03-02 15:09:02.925032 7f4f71beb700 10 mon.0@0(leader).mds e739 preprocess_query mdsbeacon(4120/g up:active seq 8507 v739) v7 from mds.1 172.21.15.17:6804/2576824570 2017-03-02 15:09:02.925038 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:09:02.925040 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:02.925041 7f4f71beb700 20 allow all 2017-03-02 15:09:02.925042 7f4f71beb700 12 mon.0@0(leader).mds e739 preprocess_beacon mdsbeacon(4120/g up:active seq 8507 v739) v7 from mds.1 172.21.15.17:6804/2576824570 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:09:02.925048 7f4f71beb700 10 mon.0@0(leader).mds e739 preprocess_beacon: GID exists in map: 4120 2017-03-02 15:09:02.925052 7f4f71beb700 15 mon.0@0(leader).mds e739 _note_beacon mdsbeacon(4120/g up:active seq 8507 v739) v7 noting time 2017-03-02 15:09:02.925055 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a14921b00 0x558a143e49c0 mdsbeacon(4120/g up:active seq 8507 v739) v7 2017-03-02 15:09:02.925058 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6804/2576824570 -- mdsbeacon(4120/g up:active seq 8507 v739) v7 -- 0x558a143e49c0 con 0 2017-03-02 15:09:02.970627 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.4 172.21.15.26:6805/2033388711 8564 ==== mdsbeacon(4117/b up:active seq 8507 v739) v7 ==== 365+0+0 (1418548970 0 0) 0x558a143e49c0 con 0x558a14904000 2017-03-02 15:09:02.970660 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b1600 for mds.? 172.21.15.26:6805/2033388711 2017-03-02 15:09:02.970666 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:02.970669 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..739) dispatch 0x558a143e49c0 mdsbeacon(4117/b up:active seq 8507 v739) v7 from mds.4 172.21.15.26:6805/2033388711 con 0x558a14904000 2017-03-02 15:09:02.970674 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27370) is_readable = 1 - now=2017-03-02 15:09:02.970675 lease_expire=0.000000 has v0 lc 27370 2017-03-02 15:09:02.970682 7f4f71beb700 10 mon.0@0(leader).mds e739 preprocess_query mdsbeacon(4117/b up:active seq 8507 v739) v7 from mds.4 172.21.15.26:6805/2033388711 2017-03-02 15:09:02.970688 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:09:02.970690 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:02.970691 7f4f71beb700 20 allow all 2017-03-02 15:09:02.970692 7f4f71beb700 12 mon.0@0(leader).mds e739 preprocess_beacon mdsbeacon(4117/b up:active seq 8507 v739) v7 from mds.4 172.21.15.26:6805/2033388711 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:09:02.970699 7f4f71beb700 10 mon.0@0(leader).mds e739 preprocess_beacon: GID exists in map: 4117 2017-03-02 15:09:02.970703 7f4f71beb700 15 mon.0@0(leader).mds e739 _note_beacon mdsbeacon(4117/b up:active seq 8507 v739) v7 noting time 2017-03-02 15:09:02.970706 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a14921b00 0x558a154b8d00 mdsbeacon(4117/b up:active seq 8507 v739) v7 2017-03-02 15:09:02.970709 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6805/2033388711 -- mdsbeacon(4117/b up:active seq 8507 v739) v7 -- 0x558a154b8d00 con 0 2017-03-02 15:09:02.979881 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.6 172.21.15.26:6804/2954782670 8757 ==== mdsbeacon(4115/a up:active seq 8507 v739) v7 ==== 365+0+0 (258450050 0 0) 0x558a15204d00 con 0x558a1471f000 2017-03-02 15:09:02.979941 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14868d60 for mds.? 172.21.15.26:6804/2954782670 2017-03-02 15:09:02.979946 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:02.979948 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..739) dispatch 0x558a15204d00 mdsbeacon(4115/a up:active seq 8507 v739) v7 from mds.6 172.21.15.26:6804/2954782670 con 0x558a1471f000 2017-03-02 15:09:02.979952 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27370) is_readable = 1 - now=2017-03-02 15:09:02.979953 lease_expire=0.000000 has v0 lc 27370 2017-03-02 15:09:02.979959 7f4f71beb700 10 mon.0@0(leader).mds e739 preprocess_query mdsbeacon(4115/a up:active seq 8507 v739) v7 from mds.6 172.21.15.26:6804/2954782670 2017-03-02 15:09:02.979963 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:09:02.979965 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:02.979966 7f4f71beb700 20 allow all 2017-03-02 15:09:02.979967 7f4f71beb700 12 mon.0@0(leader).mds e739 preprocess_beacon mdsbeacon(4115/a up:active seq 8507 v739) v7 from mds.6 172.21.15.26:6804/2954782670 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:09:02.979972 7f4f71beb700 10 mon.0@0(leader).mds e739 preprocess_beacon: GID exists in map: 4115 2017-03-02 15:09:02.979976 7f4f71beb700 15 mon.0@0(leader).mds e739 _note_beacon mdsbeacon(4115/a up:active seq 8507 v739) v7 noting time 2017-03-02 15:09:02.979979 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a14921b00 0x558a143e49c0 mdsbeacon(4115/a up:active seq 8507 v739) v7 2017-03-02 15:09:02.979982 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6804/2954782670 -- mdsbeacon(4115/a up:active seq 8507 v739) v7 -- 0x558a143e49c0 con 0 2017-03-02 15:09:03.267635 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.3 172.21.15.17:6800/27100 6841 ==== pg_stats(1 pgs tid 6805 v 0) v1 ==== 710+0+0 (3700825459 0 0) 0x558a15ae3e00 con 0x558a145d7000 2017-03-02 15:09:03.267679 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415f440 for osd.3 172.21.15.17:6800/27100 2017-03-02 15:09:03.267687 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:03.267691 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12983) dispatch 0x558a15ae3e00 pg_stats(1 pgs tid 6805 v 0) v1 from osd.3 172.21.15.17:6800/27100 con 0x558a145d7000 2017-03-02 15:09:03.267698 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27370) is_readable = 1 - now=2017-03-02 15:09:03.267699 lease_expire=0.000000 has v0 lc 27370 2017-03-02 15:09:03.267711 7f4f71beb700 10 mon.0@0(leader).pg v12983 preprocess_query pg_stats(1 pgs tid 6805 v 0) v1 from osd.3 172.21.15.17:6800/27100 2017-03-02 15:09:03.267719 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:09:03.267722 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:03.267724 7f4f71beb700 20 allow all 2017-03-02 15:09:03.267726 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27370) is_readable = 1 - now=2017-03-02 15:09:03.267726 lease_expire=0.000000 has v0 lc 27370 2017-03-02 15:09:03.267732 7f4f71beb700 10 mon.0@0(leader).pg v12983 prepare_update pg_stats(1 pgs tid 6805 v 0) v1 from osd.3 172.21.15.17:6800/27100 2017-03-02 15:09:03.267737 7f4f71beb700 10 mon.0@0(leader).pg v12983 prepare_pg_stats pg_stats(1 pgs tid 6805 v 0) v1 from osd.3 2017-03-02 15:09:03.267746 7f4f71beb700 10 mon.0@0(leader).pg v12983 got osd.3 osd_stat(33528 MB used, 61812 MB avail, 95341 MB total, peers [0,1,2] op hist [0,0,1]) (was osd_stat(34184 MB used, 61157 MB avail, 95341 MB total, peers [0,1,2] op hist [])) 2017-03-02 15:09:03.267757 7f4f71beb700 15 mon.0@0(leader).pg v12983 got 2.6 reported at 7:965847 state active+clean -> active+clean 2017-03-02 15:09:03.267770 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12983) setting proposal_timer 0x558a1412e690 with delay of 0.511782 2017-03-02 15:09:03.468673 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.1 172.21.15.22:6800/26531 6841 ==== pg_stats(5 pgs tid 6805 v 0) v1 ==== 2962+0+0 (1650875541 0 0) 0x558a15f27200 con 0x558a14643800 2017-03-02 15:09:03.468721 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1463b1e0 for osd.1 172.21.15.22:6800/26531 2017-03-02 15:09:03.468728 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:03.468732 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12983) dispatch 0x558a15f27200 pg_stats(5 pgs tid 6805 v 0) v1 from osd.1 172.21.15.22:6800/26531 con 0x558a14643800 2017-03-02 15:09:03.468738 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27370) is_readable = 1 - now=2017-03-02 15:09:03.468739 lease_expire=0.000000 has v0 lc 27370 2017-03-02 15:09:03.468775 7f4f71beb700 10 mon.0@0(leader).pg v12983 preprocess_query pg_stats(5 pgs tid 6805 v 0) v1 from osd.1 172.21.15.22:6800/26531 2017-03-02 15:09:03.468783 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:09:03.468786 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:03.468788 7f4f71beb700 20 allow all 2017-03-02 15:09:03.468789 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27370) is_readable = 1 - now=2017-03-02 15:09:03.468790 lease_expire=0.000000 has v0 lc 27370 2017-03-02 15:09:03.468794 7f4f71beb700 10 mon.0@0(leader).pg v12983 prepare_update pg_stats(5 pgs tid 6805 v 0) v1 from osd.1 172.21.15.22:6800/26531 2017-03-02 15:09:03.468799 7f4f71beb700 10 mon.0@0(leader).pg v12983 prepare_pg_stats pg_stats(5 pgs tid 6805 v 0) v1 from osd.1 2017-03-02 15:09:03.468806 7f4f71beb700 10 mon.0@0(leader).pg v12983 got osd.1 osd_stat(42170 MB used, 53170 MB avail, 95341 MB total, peers [0,2,3] op hist []) (was osd_stat(42763 MB used, 52577 MB avail, 95341 MB total, peers [0,2,3] op hist [1,0,1])) 2017-03-02 15:09:03.468817 7f4f71beb700 15 mon.0@0(leader).pg v12983 got 1.1 reported at 7:587324 state active+clean -> active+clean 2017-03-02 15:09:03.468861 7f4f71beb700 15 mon.0@0(leader).pg v12983 got 1.5 reported at 7:568203 state active+clean -> active+clean 2017-03-02 15:09:03.468866 7f4f71beb700 15 mon.0@0(leader).pg v12983 got 1.6 reported at 7:503599 state active+clean -> active+clean 2017-03-02 15:09:03.468871 7f4f71beb700 15 mon.0@0(leader).pg v12983 got 2.4 reported at 7:1027661 state active+clean -> active+clean 2017-03-02 15:09:03.468876 7f4f71beb700 15 mon.0@0(leader).pg v12983 got 2.5 reported at 7:1131028 state active+clean -> active+clean 2017-03-02 15:09:03.468882 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12983) proposal_timer already set 2017-03-02 15:09:03.562760 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.2 172.21.15.11:6800/26536 6841 ==== pg_stats(8 pgs tid 6805 v 0) v1 ==== 4672+0+0 (4119456140 0 0) 0x558a155d5480 con 0x558a145d5800 2017-03-02 15:09:03.562800 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a145c3700 for osd.2 172.21.15.11:6800/26536 2017-03-02 15:09:03.562811 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:03.562817 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12983) dispatch 0x558a155d5480 pg_stats(8 pgs tid 6805 v 0) v1 from osd.2 172.21.15.11:6800/26536 con 0x558a145d5800 2017-03-02 15:09:03.562861 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27370) is_readable = 1 - now=2017-03-02 15:09:03.562863 lease_expire=0.000000 has v0 lc 27370 2017-03-02 15:09:03.562874 7f4f71beb700 10 mon.0@0(leader).pg v12983 preprocess_query pg_stats(8 pgs tid 6805 v 0) v1 from osd.2 172.21.15.11:6800/26536 2017-03-02 15:09:03.562881 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:09:03.562885 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:03.562886 7f4f71beb700 20 allow all 2017-03-02 15:09:03.562888 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27370) is_readable = 1 - now=2017-03-02 15:09:03.562888 lease_expire=0.000000 has v0 lc 27370 2017-03-02 15:09:03.562895 7f4f71beb700 10 mon.0@0(leader).pg v12983 prepare_update pg_stats(8 pgs tid 6805 v 0) v1 from osd.2 172.21.15.11:6800/26536 2017-03-02 15:09:03.562901 7f4f71beb700 10 mon.0@0(leader).pg v12983 prepare_pg_stats pg_stats(8 pgs tid 6805 v 0) v1 from osd.2 2017-03-02 15:09:03.562908 7f4f71beb700 10 mon.0@0(leader).pg v12983 got osd.2 osd_stat(50727 MB used, 44613 MB avail, 95341 MB total, peers [0,1,3] op hist [0,0,1]) (was osd_stat(51692 MB used, 43648 MB avail, 95341 MB total, peers [0,1,3] op hist [0,1])) 2017-03-02 15:09:03.562922 7f4f71beb700 15 mon.0@0(leader).pg v12983 got 1.0 reported at 7:497043 state active+clean -> active+clean 2017-03-02 15:09:03.562938 7f4f71beb700 15 mon.0@0(leader).pg v12983 got 1.2 reported at 7:577922 state active+clean -> active+clean 2017-03-02 15:09:03.562948 7f4f71beb700 15 mon.0@0(leader).pg v12983 got 1.7 reported at 7:530180 state active+clean -> active+clean 2017-03-02 15:09:03.562957 7f4f71beb700 15 mon.0@0(leader).pg v12983 got 2.0 reported at 7:1058264 state active+clean -> active+clean 2017-03-02 15:09:03.562966 7f4f71beb700 15 mon.0@0(leader).pg v12983 got 2.1 reported at 7:1091007 state active+clean -> active+clean 2017-03-02 15:09:03.562976 7f4f71beb700 15 mon.0@0(leader).pg v12983 got 2.2 reported at 7:1164763 state active+clean -> active+clean 2017-03-02 15:09:03.562984 7f4f71beb700 15 mon.0@0(leader).pg v12983 got 2.3 reported at 7:1096800 state active+clean -> active+clean 2017-03-02 15:09:03.562993 7f4f71beb700 15 mon.0@0(leader).pg v12983 got 2.7 reported at 7:1136949 state active+clean -> active+clean 2017-03-02 15:09:03.563003 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12983) proposal_timer already set 2017-03-02 15:09:03.719388 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.0 172.21.15.26:6800/26515 6841 ==== pg_stats(2 pgs tid 6805 v 0) v1 ==== 1264+0+0 (219628236 0 0) 0x558a15da1b80 con 0x558a145d8800 2017-03-02 15:09:03.719429 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1463af20 for osd.0 172.21.15.26:6800/26515 2017-03-02 15:09:03.719436 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:03.719440 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12983) dispatch 0x558a15da1b80 pg_stats(2 pgs tid 6805 v 0) v1 from osd.0 172.21.15.26:6800/26515 con 0x558a145d8800 2017-03-02 15:09:03.719466 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27370) is_readable = 1 - now=2017-03-02 15:09:03.719467 lease_expire=0.000000 has v0 lc 27370 2017-03-02 15:09:03.719476 7f4f71beb700 10 mon.0@0(leader).pg v12983 preprocess_query pg_stats(2 pgs tid 6805 v 0) v1 from osd.0 172.21.15.26:6800/26515 2017-03-02 15:09:03.719483 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:09:03.719486 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:03.719487 7f4f71beb700 20 allow all 2017-03-02 15:09:03.719489 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27370) is_readable = 1 - now=2017-03-02 15:09:03.719489 lease_expire=0.000000 has v0 lc 27370 2017-03-02 15:09:03.719493 7f4f71beb700 10 mon.0@0(leader).pg v12983 prepare_update pg_stats(2 pgs tid 6805 v 0) v1 from osd.0 172.21.15.26:6800/26515 2017-03-02 15:09:03.719498 7f4f71beb700 10 mon.0@0(leader).pg v12983 prepare_pg_stats pg_stats(2 pgs tid 6805 v 0) v1 from osd.0 2017-03-02 15:09:03.719505 7f4f71beb700 10 mon.0@0(leader).pg v12983 got osd.0 osd_stat(8809 MB used, 86531 MB avail, 95341 MB total, peers [1,2,3] op hist []) (was osd_stat(8955 MB used, 86385 MB avail, 95341 MB total, peers [1,2,3] op hist [])) 2017-03-02 15:09:03.719516 7f4f71beb700 15 mon.0@0(leader).pg v12983 got 1.3 reported at 7:500632 state active+clean -> active+clean 2017-03-02 15:09:03.719526 7f4f71beb700 15 mon.0@0(leader).pg v12983 got 1.4 reported at 7:486555 state active+clean -> active+clean 2017-03-02 15:09:03.719532 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12983) proposal_timer already set 2017-03-02 15:09:03.738062 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13565) propose_pending 2017-03-02 15:09:03.738086 7f4f743f0700 10 mon.0@0(leader).log v13565 encode_full log v 13565 2017-03-02 15:09:03.738183 7f4f743f0700 10 mon.0@0(leader).log v13565 encode_pending v13566 2017-03-02 15:09:03.738204 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26858..27370) queue_pending_finisher 0x558a1412fa30 2017-03-02 15:09:03.738209 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26858..27370) trigger_propose active, proposing now 2017-03-02 15:09:03.738219 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26858..27370) propose_pending 27371 10464 bytes 2017-03-02 15:09:03.738224 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26858..27370) begin for 27371 10464 bytes 2017-03-02 15:09:03.768949 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26858..27370) commit_start 27371 2017-03-02 15:09:03.779660 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12983) propose_pending 2017-03-02 15:09:03.779674 7f4f743f0700 10 mon.0@0(leader).pg v12983 encode_pending v 12984 2017-03-02 15:09:03.779762 7f4f743f0700 5 mon.0@0(leader).paxos(paxos writing c 26858..27370) queue_pending_finisher 0x558a1412fe50 2017-03-02 15:09:03.779775 7f4f743f0700 10 mon.0@0(leader).paxos(paxos writing c 26858..27370) trigger_propose not active, will propose later 2017-03-02 15:09:03.802511 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26858..27370) commit_finish 27371 2017-03-02 15:09:03.802571 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:09:03.802601 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12983) refresh 2017-03-02 15:09:03.802626 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..739) refresh 2017-03-02 15:09:03.802649 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:09:03.802693 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13566) refresh 2017-03-02 15:09:03.802695 7f4f763f4700 10 mon.0@0(leader).log v13566 update_from_paxos 2017-03-02 15:09:03.802698 7f4f763f4700 10 mon.0@0(leader).log v13566 update_from_paxos version 13566 summary v 13565 2017-03-02 15:09:03.802703 7f4f763f4700 10 mon.0@0(leader).log v13566 update_from_paxos latest full 13565 2017-03-02 15:09:03.802717 7f4f763f4700 7 mon.0@0(leader).log v13566 update_from_paxos applying incremental log 13566 2017-03-02 15:09:02.768986 mon.0 172.21.15.17:6789/0 14130 : cluster [INF] fsmap e738: 7/7/7 up {0=d=up:active,1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:09:03.802735 7f4f763f4700 20 mon.0@0(leader).log v13566 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:09:03.802756 7f4f763f4700 7 mon.0@0(leader).log v13566 update_from_paxos applying incremental log 13566 2017-03-02 15:09:02.780429 mon.0 172.21.15.17:6789/0 14131 : cluster [INF] fsmap e739: 7/7/7 up {0=d=up:active(laggy or crashed),1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:09:03.802764 7f4f763f4700 20 mon.0@0(leader).log v13566 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:09:03.802773 7f4f763f4700 15 mon.0@0(leader).log v13566 update_from_paxos logging for 1 channels 2017-03-02 15:09:03.802776 7f4f763f4700 15 mon.0@0(leader).log v13566 update_from_paxos channel 'cluster' logging 410 bytes 2017-03-02 15:09:03.802933 7f4f763f4700 10 mon.0@0(leader).log v13566 check_subs 2017-03-02 15:09:03.802989 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:09:03.803018 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:09:03.803026 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:09:03.803047 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:09:03.803050 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12983) post_refresh 2017-03-02 15:09:03.803052 7f4f763f4700 10 mon.0@0(leader).pg v12983 post_paxos_update 2017-03-02 15:09:03.803055 7f4f763f4700 10 mon.0@0(leader).pg v12983 check_subs 2017-03-02 15:09:03.803058 7f4f763f4700 20 mon.0@0(leader).pg v12983 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:09:03.803066 7f4f763f4700 20 mon.0@0(leader).pg v12983 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:09:03.803069 7f4f763f4700 20 mon.0@0(leader).pg v12983 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:09:03.803072 7f4f763f4700 20 mon.0@0(leader).pg v12983 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:09:03.803075 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..739) post_refresh 2017-03-02 15:09:03.803077 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:09:03.803079 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13566) post_refresh 2017-03-02 15:09:03.803081 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:09:03.803082 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:09:03.803084 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:09:03.803087 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27371) commit_proposal 2017-03-02 15:09:03.803091 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13566) _active - not active 2017-03-02 15:09:03.803094 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27371) finish_round 2017-03-02 15:09:03.803096 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27371) finish_round waiting_for_acting 2017-03-02 15:09:03.803099 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13566) _active 2017-03-02 15:09:03.803101 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13566) _active creating new pending 2017-03-02 15:09:03.803111 7f4f763f4700 10 mon.0@0(leader).log v13566 create_pending v 13567 2017-03-02 15:09:03.803114 7f4f763f4700 7 mon.0@0(leader).log v13566 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:09:03.803120 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a16404e00 0x558a14a870c0 log(last 14130) v1 2017-03-02 15:09:03.803125 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14130) v1 -- 0x558a14a870c0 con 0 2017-03-02 15:09:03.803148 7f4f763f4700 7 mon.0@0(leader).log v13566 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:09:03.803153 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a15fc8b00 0x558a16108ee0 log(last 14131) v1 2017-03-02 15:09:03.803155 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14131) v1 -- 0x558a16108ee0 con 0 2017-03-02 15:09:03.803163 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27371) finish_round waiting_for_readable 2017-03-02 15:09:03.803166 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27371) finish_round waiting_for_writeable 2017-03-02 15:09:03.803169 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26858..27371) finish_round done w/ waiters, state active 2017-03-02 15:09:03.803198 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26858..27371) propose_pending 27372 10483 bytes 2017-03-02 15:09:03.803206 7f4f763f4700 10 mon.0@0(leader).paxos(paxos updating c 26858..27371) begin for 27372 10483 bytes 2017-03-02 15:09:03.803215 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14130) v1 ==== 0+0+0 (0 0 0) 0x558a14a870c0 con 0x558a14482000 2017-03-02 15:09:03.835967 7f4f763f4700 10 mon.0@0(leader).paxos(paxos updating c 26858..27371) commit_start 27372 2017-03-02 15:09:03.836116 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:09:03.836143 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:03.836148 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:09:03.836155 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:03.836158 7f4f71beb700 20 allow all 2017-03-02 15:09:03.836185 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14131) v1 ==== 0+0+0 (0 0 0) 0x558a16108ee0 con 0x558a14482000 2017-03-02 15:09:03.836198 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:09:03.836203 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:03.836207 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:09:03.836209 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:03.836211 7f4f71beb700 20 allow all 2017-03-02 15:09:03.869465 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26858..27371) commit_finish 27372 2017-03-02 15:09:03.869534 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:09:03.869573 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12984) refresh 2017-03-02 15:09:03.869579 7f4f763f4700 10 mon.0@0(leader).pg v12983 update_from_paxos read_incremental 2017-03-02 15:09:03.869621 7f4f763f4700 20 mon.0@0(leader).pg v12983 refreshing pg 1.0 7:497043 active+clean 2017-03-02 15:09:03.869644 7f4f763f4700 20 mon.0@0(leader).pg v12983 refreshing pg 1.1 7:587324 active+clean 2017-03-02 15:09:03.869662 7f4f763f4700 20 mon.0@0(leader).pg v12983 refreshing pg 1.2 7:577922 active+clean 2017-03-02 15:09:03.869678 7f4f763f4700 20 mon.0@0(leader).pg v12983 refreshing pg 1.3 7:500632 active+clean 2017-03-02 15:09:03.869694 7f4f763f4700 20 mon.0@0(leader).pg v12983 refreshing pg 1.4 7:486555 active+clean 2017-03-02 15:09:03.869710 7f4f763f4700 20 mon.0@0(leader).pg v12983 refreshing pg 1.5 7:568203 active+clean 2017-03-02 15:09:03.869726 7f4f763f4700 20 mon.0@0(leader).pg v12983 refreshing pg 1.6 7:503599 active+clean 2017-03-02 15:09:03.869742 7f4f763f4700 20 mon.0@0(leader).pg v12983 refreshing pg 1.7 7:530180 active+clean 2017-03-02 15:09:03.869758 7f4f763f4700 20 mon.0@0(leader).pg v12983 refreshing pg 2.0 7:1058264 active+clean 2017-03-02 15:09:03.869773 7f4f763f4700 20 mon.0@0(leader).pg v12983 refreshing pg 2.1 7:1091007 active+clean 2017-03-02 15:09:03.869788 7f4f763f4700 20 mon.0@0(leader).pg v12983 refreshing pg 2.2 7:1164763 active+clean 2017-03-02 15:09:03.869803 7f4f763f4700 20 mon.0@0(leader).pg v12983 refreshing pg 2.3 7:1096800 active+clean 2017-03-02 15:09:03.869818 7f4f763f4700 20 mon.0@0(leader).pg v12983 refreshing pg 2.4 7:1027661 active+clean 2017-03-02 15:09:03.869882 7f4f763f4700 20 mon.0@0(leader).pg v12983 refreshing pg 2.5 7:1131028 active+clean 2017-03-02 15:09:03.869896 7f4f763f4700 20 mon.0@0(leader).pg v12983 refreshing pg 2.6 7:965847 active+clean 2017-03-02 15:09:03.869911 7f4f763f4700 20 mon.0@0(leader).pg v12983 refreshing pg 2.7 7:1136949 active+clean 2017-03-02 15:09:03.869917 7f4f763f4700 20 mon.0@0(leader).pg v12983 refreshing osd.0 2017-03-02 15:09:03.869932 7f4f763f4700 20 mon.0@0(leader).pg v12983 refreshing osd.1 2017-03-02 15:09:03.869943 7f4f763f4700 20 mon.0@0(leader).pg v12983 refreshing osd.2 2017-03-02 15:09:03.869952 7f4f763f4700 20 mon.0@0(leader).pg v12983 refreshing osd.3 2017-03-02 15:09:03.869972 7f4f763f4700 10 mon.0@0(leader).pg v12984 read_pgmap_meta 2017-03-02 15:09:03.870020 7f4f763f4700 10 mon.0@0(leader).pg v12984 update_logger 2017-03-02 15:09:03.870054 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..739) refresh 2017-03-02 15:09:03.870085 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:09:03.870114 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13566) refresh 2017-03-02 15:09:03.870118 7f4f763f4700 10 mon.0@0(leader).log v13566 update_from_paxos 2017-03-02 15:09:03.870122 7f4f763f4700 10 mon.0@0(leader).log v13566 update_from_paxos version 13566 summary v 13566 2017-03-02 15:09:03.870153 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:09:03.870184 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:09:03.870188 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:09:03.870222 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:09:03.870226 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12984) post_refresh 2017-03-02 15:09:03.870229 7f4f763f4700 10 mon.0@0(leader).pg v12984 post_paxos_update 2017-03-02 15:09:03.870233 7f4f763f4700 10 mon.0@0(leader).pg v12984 check_subs 2017-03-02 15:09:03.870237 7f4f763f4700 20 mon.0@0(leader).pg v12984 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:09:03.870248 7f4f763f4700 20 mon.0@0(leader).pg v12984 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:09:03.870255 7f4f763f4700 20 mon.0@0(leader).pg v12984 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:09:03.870260 7f4f763f4700 20 mon.0@0(leader).pg v12984 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:09:03.870266 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..739) post_refresh 2017-03-02 15:09:03.870270 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:09:03.870272 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13566) post_refresh 2017-03-02 15:09:03.870275 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:09:03.870278 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:09:03.870281 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:09:03.870285 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27372) commit_proposal 2017-03-02 15:09:03.870291 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12984) _active - not active 2017-03-02 15:09:03.870295 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27372) finish_round 2017-03-02 15:09:03.870299 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27372) finish_round waiting_for_acting 2017-03-02 15:09:03.870303 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12984) _active 2017-03-02 15:09:03.870306 7f4f763f4700 7 mon.0@0(leader).paxosservice(pgmap 12269..12984) _active creating new pending 2017-03-02 15:09:03.870320 7f4f763f4700 10 mon.0@0(leader).pg v12984 create_pending v 12985 2017-03-02 15:09:03.870329 7f4f763f4700 7 mon.0@0(leader).pg v12984 _updated_stats for osd.3 172.21.15.17:6800/27100 2017-03-02 15:09:03.870338 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a14921b00 0x558a15437600 pg_stats_ack(1 pgs tid 6805) v1 2017-03-02 15:09:03.870345 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6800/27100 -- pg_stats_ack(1 pgs tid 6805) v1 -- 0x558a15437600 con 0 2017-03-02 15:09:03.870409 7f4f763f4700 7 mon.0@0(leader).pg v12984 _updated_stats for osd.1 172.21.15.22:6800/26531 2017-03-02 15:09:03.870419 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a1469ad00 0x558a15707800 pg_stats_ack(5 pgs tid 6805) v1 2017-03-02 15:09:03.870424 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6800/26531 -- pg_stats_ack(5 pgs tid 6805) v1 -- 0x558a15707800 con 0 2017-03-02 15:09:03.870460 7f4f763f4700 7 mon.0@0(leader).pg v12984 _updated_stats for osd.2 172.21.15.11:6800/26536 2017-03-02 15:09:03.870468 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a15d0c200 0x558a1547b800 pg_stats_ack(8 pgs tid 6805) v1 2017-03-02 15:09:03.870472 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6800/26536 -- pg_stats_ack(8 pgs tid 6805) v1 -- 0x558a1547b800 con 0 2017-03-02 15:09:03.870497 7f4f763f4700 7 mon.0@0(leader).pg v12984 _updated_stats for osd.0 172.21.15.26:6800/26515 2017-03-02 15:09:03.870504 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a145dcb00 0x558a153f9e00 pg_stats_ack(2 pgs tid 6805) v1 2017-03-02 15:09:03.870508 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6800/26515 -- pg_stats_ack(2 pgs tid 6805) v1 -- 0x558a153f9e00 con 0 2017-03-02 15:09:03.870549 7f4f763f4700 10 mon.0@0(leader).pg v12984 check_osd_map already seen 7 >= 7 2017-03-02 15:09:03.870559 7f4f763f4700 10 mon.0@0(leader).pg v12984 update_logger 2017-03-02 15:09:03.870588 7f4f763f4700 0 log_channel(cluster) log [INF] : pgmap v12984: 32 pgs: 32 active+clean; 69253 MB data, 132 GB used, 240 GB / 372 GB avail; 0 B/s rd, 3156 kB/s wr, 635 op/s 2017-03-02 15:09:03.870615 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14132 at 2017-03-02 15:09:03.870591) v1 -- 0x558a14d60480 con 0 2017-03-02 15:09:03.870656 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27372) finish_round waiting_for_readable 2017-03-02 15:09:03.870662 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27372) finish_round waiting_for_writeable 2017-03-02 15:09:03.870666 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26858..27372) finish_round done w/ waiters, state active 2017-03-02 15:09:03.870725 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14132 at 2017-03-02 15:09:03.870591) v1 ==== 0+0+0 (0 0 0) 0x558a14d60480 con 0x558a14482000 2017-03-02 15:09:03.870750 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:09:03.870754 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:03.870757 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13566) dispatch 0x558a14d60480 log(1 entries from seq 14132 at 2017-03-02 15:09:03.870591) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:09:03.870763 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27372) is_readable = 1 - now=2017-03-02 15:09:03.870763 lease_expire=0.000000 has v0 lc 27372 2017-03-02 15:09:03.870769 7f4f71beb700 10 mon.0@0(leader).log v13566 preprocess_query log(1 entries from seq 14132 at 2017-03-02 15:09:03.870591) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:09:03.870778 7f4f71beb700 10 mon.0@0(leader).log v13566 preprocess_log log(1 entries from seq 14132 at 2017-03-02 15:09:03.870591) v1 from mon.0 2017-03-02 15:09:03.870783 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:09:03.870786 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:03.870788 7f4f71beb700 20 allow all 2017-03-02 15:09:03.870796 7f4f71beb700 10 mon.0@0(leader).log v13566 prepare_update log(1 entries from seq 14132 at 2017-03-02 15:09:03.870591) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:09:03.870801 7f4f71beb700 10 mon.0@0(leader).log v13566 prepare_log log(1 entries from seq 14132 at 2017-03-02 15:09:03.870591) v1 from mon.0 2017-03-02 15:09:03.870805 7f4f71beb700 10 mon.0@0(leader).log v13566 logging 2017-03-02 15:09:03.870591 mon.0 172.21.15.17:6789/0 14132 : cluster [INF] pgmap v12984: 32 pgs: 32 active+clean; 69253 MB data, 132 GB used, 240 GB / 372 GB avail; 0 B/s rd, 3156 kB/s wr, 635 op/s 2017-03-02 15:09:03.870812 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13566) setting proposal_timer 0x558a1412ebc0 with delay of 0.998677 2017-03-02 15:09:04.869657 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13566) propose_pending 2017-03-02 15:09:04.869688 7f4f743f0700 10 mon.0@0(leader).log v13566 encode_full log v 13566 2017-03-02 15:09:04.869796 7f4f743f0700 10 mon.0@0(leader).log v13566 encode_pending v13567 2017-03-02 15:09:04.869810 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26858..27372) queue_pending_finisher 0x558a1412e690 2017-03-02 15:09:04.869814 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26858..27372) trigger_propose active, proposing now 2017-03-02 15:09:04.869852 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26858..27372) propose_pending 27373 10332 bytes 2017-03-02 15:09:04.869858 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26858..27372) begin for 27373 10332 bytes 2017-03-02 15:09:04.903133 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26858..27372) commit_start 27373 2017-03-02 15:09:04.936303 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26858..27372) commit_finish 27373 2017-03-02 15:09:04.936367 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:09:04.936393 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12984) refresh 2017-03-02 15:09:04.936413 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..739) refresh 2017-03-02 15:09:04.936431 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:09:04.936449 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13567) refresh 2017-03-02 15:09:04.936451 7f4f763f4700 10 mon.0@0(leader).log v13567 update_from_paxos 2017-03-02 15:09:04.936454 7f4f763f4700 10 mon.0@0(leader).log v13567 update_from_paxos version 13567 summary v 13566 2017-03-02 15:09:04.936459 7f4f763f4700 10 mon.0@0(leader).log v13567 update_from_paxos latest full 13566 2017-03-02 15:09:04.936471 7f4f763f4700 7 mon.0@0(leader).log v13567 update_from_paxos applying incremental log 13567 2017-03-02 15:09:03.870591 mon.0 172.21.15.17:6789/0 14132 : cluster [INF] pgmap v12984: 32 pgs: 32 active+clean; 69253 MB data, 132 GB used, 240 GB / 372 GB avail; 0 B/s rd, 3156 kB/s wr, 635 op/s 2017-03-02 15:09:04.936489 7f4f763f4700 20 mon.0@0(leader).log v13567 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:09:04.936509 7f4f763f4700 15 mon.0@0(leader).log v13567 update_from_paxos logging for 1 channels 2017-03-02 15:09:04.936512 7f4f763f4700 15 mon.0@0(leader).log v13567 update_from_paxos channel 'cluster' logging 198 bytes 2017-03-02 15:09:04.936587 7f4f763f4700 10 mon.0@0(leader).log v13567 check_subs 2017-03-02 15:09:04.936640 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:09:04.936667 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:09:04.936669 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:09:04.936688 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:09:04.936691 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12984) post_refresh 2017-03-02 15:09:04.936693 7f4f763f4700 10 mon.0@0(leader).pg v12984 post_paxos_update 2017-03-02 15:09:04.936696 7f4f763f4700 10 mon.0@0(leader).pg v12984 check_subs 2017-03-02 15:09:04.936698 7f4f763f4700 20 mon.0@0(leader).pg v12984 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:09:04.936705 7f4f763f4700 20 mon.0@0(leader).pg v12984 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:09:04.936708 7f4f763f4700 20 mon.0@0(leader).pg v12984 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:09:04.936711 7f4f763f4700 20 mon.0@0(leader).pg v12984 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:09:04.936714 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..739) post_refresh 2017-03-02 15:09:04.936716 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:09:04.936718 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13567) post_refresh 2017-03-02 15:09:04.936720 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:09:04.936721 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:09:04.936722 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:09:04.936725 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27373) commit_proposal 2017-03-02 15:09:04.936729 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13567) _active - not active 2017-03-02 15:09:04.936732 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27373) finish_round 2017-03-02 15:09:04.936734 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27373) finish_round waiting_for_acting 2017-03-02 15:09:04.936736 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13567) _active 2017-03-02 15:09:04.936738 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13567) _active creating new pending 2017-03-02 15:09:04.936747 7f4f763f4700 10 mon.0@0(leader).log v13567 create_pending v 13568 2017-03-02 15:09:04.936751 7f4f763f4700 7 mon.0@0(leader).log v13567 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:09:04.936757 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a16383600 0x558a15228760 log(last 14132) v1 2017-03-02 15:09:04.936762 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14132) v1 -- 0x558a15228760 con 0 2017-03-02 15:09:04.936789 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27373) finish_round waiting_for_readable 2017-03-02 15:09:04.936794 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27373) finish_round waiting_for_writeable 2017-03-02 15:09:04.936797 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26858..27373) finish_round done w/ waiters, state active 2017-03-02 15:09:04.936862 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14132) v1 ==== 0+0+0 (0 0 0) 0x558a15228760 con 0x558a14482000 2017-03-02 15:09:04.936900 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:09:04.936906 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:04.936910 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:09:04.936914 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:04.936915 7f4f71beb700 20 allow all 2017-03-02 15:09:06.013501 7f4f743f0700 11 mon.0@0(leader) e2 tick 2017-03-02 15:09:06.013527 7f4f743f0700 10 mon.0@0(leader).pg v12984 v12984: 32 pgs: 32 active+clean; 69253 MB data, 132 GB used, 240 GB / 372 GB avail; 0 B/s rd, 3156 kB/s wr, 635 op/s 2017-03-02 15:09:06.013543 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12984) maybe_trim trim_to 12484 would only trim 215 < paxos_service_trim_min 250 2017-03-02 15:09:06.013547 7f4f743f0700 10 mon.0@0(leader).mds e739 e739: 7/7/7 up {0=d=up:active(laggy or crashed),1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:09:06.013594 7f4f743f0700 10 mon.0@0(leader).mds e739 no beacon from 4121 172.21.15.22:6805/2203629154 mds.0.6 up:active since 2017-03-02 15:08:46.885540 2017-03-02 15:09:06.013608 7f4f743f0700 10 mon.0@0(leader).paxosservice(mdsmap 1..739) maybe_trim trim_to 239 would only trim 238 < paxos_service_trim_min 250 2017-03-02 15:09:06.013611 7f4f743f0700 10 mon.0@0(leader).osd e7 e7: 4 osds: 4 up, 4 in 2017-03-02 15:09:06.013616 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26858..27373) is_readable = 1 - now=2017-03-02 15:09:06.013616 lease_expire=0.000000 has v0 lc 27373 2017-03-02 15:09:06.013622 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26858..27373) is_readable = 1 - now=2017-03-02 15:09:06.013623 lease_expire=0.000000 has v0 lc 27373 2017-03-02 15:09:06.013627 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26858..27373) is_readable = 1 - now=2017-03-02 15:09:06.013628 lease_expire=0.000000 has v0 lc 27373 2017-03-02 15:09:06.013630 7f4f743f0700 10 mon.0@0(leader).osd e7 min_last_epoch_clean 6 2017-03-02 15:09:06.013632 7f4f743f0700 10 mon.0@0(leader).log v13567 log 2017-03-02 15:09:06.013634 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13567) maybe_trim trim_to 13067 would only trim 40 < paxos_service_trim_min 250 2017-03-02 15:09:06.013638 7f4f743f0700 10 mon.0@0(leader).auth v10 auth 2017-03-02 15:09:06.013642 7f4f743f0700 10 cephx keyserver: _check_rotating_secrets 2017-03-02 15:09:06.013650 7f4f743f0700 20 mon.0@0(leader) e2 sync_trim_providers 2017-03-02 15:09:06.871319 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.5 172.21.15.11:6804/1637448123 8796 ==== mdsbeacon(4116/e up:active seq 8508 v739) v7 ==== 418+0+0 (1368453353 0 0) 0x558a154b8680 con 0x558a1471d800 2017-03-02 15:09:06.871381 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b0f20 for mds.? 172.21.15.11:6804/1637448123 2017-03-02 15:09:06.871411 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:06.871416 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..739) dispatch 0x558a154b8680 mdsbeacon(4116/e up:active seq 8508 v739) v7 from mds.5 172.21.15.11:6804/1637448123 con 0x558a1471d800 2017-03-02 15:09:06.871424 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27373) is_readable = 1 - now=2017-03-02 15:09:06.871424 lease_expire=0.000000 has v0 lc 27373 2017-03-02 15:09:06.871439 7f4f71beb700 10 mon.0@0(leader).mds e739 preprocess_query mdsbeacon(4116/e up:active seq 8508 v739) v7 from mds.5 172.21.15.11:6804/1637448123 2017-03-02 15:09:06.871448 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:09:06.871451 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:06.871453 7f4f71beb700 20 allow all 2017-03-02 15:09:06.871454 7f4f71beb700 12 mon.0@0(leader).mds e739 preprocess_beacon mdsbeacon(4116/e up:active seq 8508 v739) v7 from mds.5 172.21.15.11:6804/1637448123 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:09:06.871464 7f4f71beb700 10 mon.0@0(leader).mds e739 preprocess_beacon: GID exists in map: 4116 2017-03-02 15:09:06.871470 7f4f71beb700 20 mon.0@0(leader).mds e739 preprocess_beacon health metrics for gid 4116 were updated 2017-03-02 15:09:06.871473 7f4f71beb700 15 mon.0@0(leader).mds e739 _note_beacon mdsbeacon(4116/e up:active seq 8508 v739) v7 noting time 2017-03-02 15:09:06.871478 7f4f71beb700 7 mon.0@0(leader).mds e739 prepare_update mdsbeacon(4116/e up:active seq 8508 v739) v7 2017-03-02 15:09:06.871482 7f4f71beb700 12 mon.0@0(leader).mds e739 prepare_beacon mdsbeacon(4116/e up:active seq 8508 v739) v7 from mds.5 172.21.15.11:6804/1637448123 2017-03-02 15:09:06.871486 7f4f71beb700 20 mon.0@0(leader).mds e739 prepare_beacon got health from gid 4116 with 1 metrics. 2017-03-02 15:09:06.871489 7f4f71beb700 10 mon.0@0(leader).mds e739 prepare_beacon mds.5 up:active -> up:active standby_for_rank=-1 2017-03-02 15:09:06.871493 7f4f71beb700 7 mon.0@0(leader).mds e739 prepare_beacon pending map now: 2017-03-02 15:09:06.871495 7f4f71beb700 7 mon.0@0(leader).mds e739 print_map 2017-03-02 15:09:06.871548 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..739) setting proposal_timer 0x558a1412cee0 with delay of 0.05 2017-03-02 15:09:06.884814 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.3 172.21.15.11:6805/1926402648 8571 ==== mdsbeacon(4118/f up:active seq 8508 v739) v7 ==== 406+0+0 (2430926773 0 0) 0x558a143e49c0 con 0x558a14905800 2017-03-02 15:09:06.884889 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b2260 for mds.? 172.21.15.11:6805/1926402648 2017-03-02 15:09:06.884896 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:06.884899 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..739) dispatch 0x558a143e49c0 mdsbeacon(4118/f up:active seq 8508 v739) v7 from mds.3 172.21.15.11:6805/1926402648 con 0x558a14905800 2017-03-02 15:09:06.884906 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27373) is_readable = 1 - now=2017-03-02 15:09:06.884906 lease_expire=0.000000 has v0 lc 27373 2017-03-02 15:09:06.884915 7f4f71beb700 10 mon.0@0(leader).mds e739 preprocess_query mdsbeacon(4118/f up:active seq 8508 v739) v7 from mds.3 172.21.15.11:6805/1926402648 2017-03-02 15:09:06.884923 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:09:06.884925 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:06.884927 7f4f71beb700 20 allow all 2017-03-02 15:09:06.884928 7f4f71beb700 12 mon.0@0(leader).mds e739 preprocess_beacon mdsbeacon(4118/f up:active seq 8508 v739) v7 from mds.3 172.21.15.11:6805/1926402648 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:09:06.884936 7f4f71beb700 10 mon.0@0(leader).mds e739 preprocess_beacon: GID exists in map: 4118 2017-03-02 15:09:06.884940 7f4f71beb700 10 mon.0@0(leader).mds e739 mds_beacon mdsbeacon(4118/f up:active seq 8508 v739) v7 ignoring requested state, because mds hasn't seen latest map 2017-03-02 15:09:06.884943 7f4f71beb700 15 mon.0@0(leader).mds e739 _note_beacon mdsbeacon(4118/f up:active seq 8508 v739) v7 noting time 2017-03-02 15:09:06.884947 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15f72800 0x558a15204d00 mdsbeacon(4118/f up:active seq 8508 v740) v7 2017-03-02 15:09:06.884951 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6805/1926402648 -- mdsbeacon(4118/f up:active seq 8508 v740) v7 -- 0x558a15204d00 con 0 2017-03-02 15:09:06.885937 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.2 172.21.15.22:6804/673088617 8570 ==== mdsbeacon(4119/c up:active seq 8508 v739) v7 ==== 365+0+0 (2514943476 0 0) 0x558a160c0700 con 0x558a14907000 2017-03-02 15:09:06.885973 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14916f20 for mds.? 172.21.15.22:6804/673088617 2017-03-02 15:09:06.885979 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:06.885982 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..739) dispatch 0x558a160c0700 mdsbeacon(4119/c up:active seq 8508 v739) v7 from mds.2 172.21.15.22:6804/673088617 con 0x558a14907000 2017-03-02 15:09:06.885989 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27373) is_readable = 1 - now=2017-03-02 15:09:06.885989 lease_expire=0.000000 has v0 lc 27373 2017-03-02 15:09:06.885998 7f4f71beb700 10 mon.0@0(leader).mds e739 preprocess_query mdsbeacon(4119/c up:active seq 8508 v739) v7 from mds.2 172.21.15.22:6804/673088617 2017-03-02 15:09:06.886005 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:09:06.886008 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:06.886009 7f4f71beb700 20 allow all 2017-03-02 15:09:06.886011 7f4f71beb700 12 mon.0@0(leader).mds e739 preprocess_beacon mdsbeacon(4119/c up:active seq 8508 v739) v7 from mds.2 172.21.15.22:6804/673088617 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:09:06.886019 7f4f71beb700 10 mon.0@0(leader).mds e739 preprocess_beacon: GID exists in map: 4119 2017-03-02 15:09:06.886022 7f4f71beb700 10 mon.0@0(leader).mds e739 mds_beacon mdsbeacon(4119/c up:active seq 8508 v739) v7 ignoring requested state, because mds hasn't seen latest map 2017-03-02 15:09:06.886025 7f4f71beb700 15 mon.0@0(leader).mds e739 _note_beacon mdsbeacon(4119/c up:active seq 8508 v739) v7 noting time 2017-03-02 15:09:06.886029 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15f72800 0x558a143e49c0 mdsbeacon(4119/c up:active seq 8508 v740) v7 2017-03-02 15:09:06.886033 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6804/673088617 -- mdsbeacon(4119/c up:active seq 8508 v740) v7 -- 0x558a143e49c0 con 0 2017-03-02 15:09:06.921657 7f4f743f0700 10 mon.0@0(leader).paxosservice(mdsmap 1..739) propose_pending 2017-03-02 15:09:06.921714 7f4f743f0700 10 mon.0@0(leader).mds e739 encode_pending e740 2017-03-02 15:09:06.921811 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26858..27373) queue_pending_finisher 0x558a1412cf00 2017-03-02 15:09:06.921879 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26858..27373) trigger_propose active, proposing now 2017-03-02 15:09:06.921895 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26858..27373) propose_pending 27374 2470 bytes 2017-03-02 15:09:06.921900 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26858..27373) begin for 27374 2470 bytes 2017-03-02 15:09:06.925249 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.1 172.21.15.17:6804/2576824570 8567 ==== mdsbeacon(4120/g up:active seq 8508 v739) v7 ==== 418+0+0 (3775426805 0 0) 0x558a154b8d00 con 0x558a14908800 2017-03-02 15:09:06.937758 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26858..27373) commit_start 27374 2017-03-02 15:09:06.937890 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14917600 for mds.? 172.21.15.17:6804/2576824570 2017-03-02 15:09:06.937921 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:06.937926 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..739) dispatch 0x558a154b8d00 mdsbeacon(4120/g up:active seq 8508 v739) v7 from mds.1 172.21.15.17:6804/2576824570 con 0x558a14908800 2017-03-02 15:09:06.937936 7f4f71beb700 5 mon.0@0(leader).paxos(paxos writing c 26858..27373) is_readable = 1 - now=2017-03-02 15:09:06.937937 lease_expire=0.000000 has v0 lc 27373 2017-03-02 15:09:06.937949 7f4f71beb700 10 mon.0@0(leader).mds e739 preprocess_query mdsbeacon(4120/g up:active seq 8508 v739) v7 from mds.1 172.21.15.17:6804/2576824570 2017-03-02 15:09:06.937958 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:09:06.937961 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:06.937963 7f4f71beb700 20 allow all 2017-03-02 15:09:06.937968 7f4f71beb700 12 mon.0@0(leader).mds e739 preprocess_beacon mdsbeacon(4120/g up:active seq 8508 v739) v7 from mds.1 172.21.15.17:6804/2576824570 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:09:06.937979 7f4f71beb700 10 mon.0@0(leader).mds e739 preprocess_beacon: GID exists in map: 4120 2017-03-02 15:09:06.937987 7f4f71beb700 10 mon.0@0(leader).mds e739 mds_beacon mdsbeacon(4120/g up:active seq 8508 v739) v7 ignoring requested state, because mds hasn't seen latest map 2017-03-02 15:09:06.937990 7f4f71beb700 15 mon.0@0(leader).mds e739 _note_beacon mdsbeacon(4120/g up:active seq 8508 v739) v7 noting time 2017-03-02 15:09:06.937995 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15f72800 0x558a160c0700 mdsbeacon(4120/g up:active seq 8508 v740) v7 2017-03-02 15:09:06.938000 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6804/2576824570 -- mdsbeacon(4120/g up:active seq 8508 v740) v7 -- 0x558a160c0700 con 0 2017-03-02 15:09:06.969493 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26858..27373) commit_finish 27374 2017-03-02 15:09:06.969548 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:09:06.969576 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12984) refresh 2017-03-02 15:09:06.969596 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..740) refresh 2017-03-02 15:09:06.969599 7f4f763f4700 10 mon.0@0(leader).mds e739 update_from_paxos version 740, my e 739 2017-03-02 15:09:06.969610 7f4f763f4700 10 mon.0@0(leader).mds e739 update_from_paxos got 740 2017-03-02 15:09:06.969667 7f4f763f4700 4 mon.0@0(leader).mds e740 new map 2017-03-02 15:09:06.969669 7f4f763f4700 0 mon.0@0(leader).mds e740 print_map 2017-03-02 15:09:06.969747 7f4f763f4700 20 mon.0@0(leader).mds e740 check_sub: mdsmap 2017-03-02 15:09:06.969755 7f4f763f4700 10 mon.0@0(leader).mds e740 check_sub: is_mds=1, fscid= 1 2017-03-02 15:09:06.969758 7f4f763f4700 10 mon.0@0(leader).mds e740 check_sub selected MDS map epoch 740 for namespace 1 for subscriber mds.? who wants epoch 740 2017-03-02 15:09:06.969785 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6804/2954782670 -- mdsmap(e 740) v1 -- 0x558a14d60480 con 0 2017-03-02 15:09:06.969856 7f4f763f4700 20 mon.0@0(leader).mds e740 check_sub: mdsmap 2017-03-02 15:09:06.969883 7f4f763f4700 10 mon.0@0(leader).mds e740 check_sub: is_mds=1, fscid= 1 2017-03-02 15:09:06.969889 7f4f763f4700 10 mon.0@0(leader).mds e740 check_sub selected MDS map epoch 740 for namespace 1 for subscriber mds.? who wants epoch 740 2017-03-02 15:09:06.969922 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6804/1637448123 -- mdsmap(e 740) v1 -- 0x558a14d618c0 con 0 2017-03-02 15:09:06.969966 7f4f763f4700 20 mon.0@0(leader).mds e740 check_sub: mdsmap 2017-03-02 15:09:06.969993 7f4f763f4700 10 mon.0@0(leader).mds e740 check_sub: is_mds=1, fscid= 1 2017-03-02 15:09:06.969999 7f4f763f4700 10 mon.0@0(leader).mds e740 check_sub selected MDS map epoch 740 for namespace 1 for subscriber mds.? who wants epoch 740 2017-03-02 15:09:06.970029 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6805/2033388711 -- mdsmap(e 740) v1 -- 0x558a14d61680 con 0 2017-03-02 15:09:06.970063 7f4f763f4700 20 mon.0@0(leader).mds e740 check_sub: mdsmap 2017-03-02 15:09:06.970073 7f4f763f4700 10 mon.0@0(leader).mds e740 check_sub: is_mds=1, fscid= 1 2017-03-02 15:09:06.970078 7f4f763f4700 10 mon.0@0(leader).mds e740 check_sub selected MDS map epoch 740 for namespace 1 for subscriber mds.? who wants epoch 740 2017-03-02 15:09:06.970108 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6805/1926402648 -- mdsmap(e 740) v1 -- 0x558a14d61b00 con 0 2017-03-02 15:09:06.970146 7f4f763f4700 20 mon.0@0(leader).mds e740 check_sub: mdsmap 2017-03-02 15:09:06.970169 7f4f763f4700 10 mon.0@0(leader).mds e740 check_sub: is_mds=1, fscid= 1 2017-03-02 15:09:06.970174 7f4f763f4700 10 mon.0@0(leader).mds e740 check_sub selected MDS map epoch 740 for namespace 1 for subscriber mds.? who wants epoch 740 2017-03-02 15:09:06.970206 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6804/673088617 -- mdsmap(e 740) v1 -- 0x558a14d61d40 con 0 2017-03-02 15:09:06.970250 7f4f763f4700 20 mon.0@0(leader).mds e740 check_sub: mdsmap 2017-03-02 15:09:06.970257 7f4f763f4700 10 mon.0@0(leader).mds e740 check_sub: is_mds=1, fscid= 1 2017-03-02 15:09:06.970261 7f4f763f4700 10 mon.0@0(leader).mds e740 check_sub selected MDS map epoch 740 for namespace 1 for subscriber mds.? who wants epoch 740 2017-03-02 15:09:06.970280 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6804/2576824570 -- mdsmap(e 740) v1 -- 0x558a14d61f80 con 0 2017-03-02 15:09:06.970302 7f4f763f4700 20 mon.0@0(leader).mds e740 check_sub: mdsmap 2017-03-02 15:09:06.970306 7f4f763f4700 10 mon.0@0(leader).mds e740 check_sub: is_mds=0, fscid= 1 2017-03-02 15:09:06.970310 7f4f763f4700 10 mon.0@0(leader).mds e740 check_sub selected MDS map epoch 740 for namespace 1 for subscriber unknown.0 who wants epoch 740 2017-03-02 15:09:06.970341 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.132:0/2597832251 -- mdsmap(e 740) v1 -- 0x558a14d621c0 con 0 2017-03-02 15:09:06.970376 7f4f763f4700 20 mon.0@0(leader).mds e740 check_sub: mdsmap 2017-03-02 15:09:06.970382 7f4f763f4700 10 mon.0@0(leader).mds e740 check_sub: is_mds=0, fscid= 1 2017-03-02 15:09:06.970387 7f4f763f4700 10 mon.0@0(leader).mds e740 check_sub selected MDS map epoch 740 for namespace 1 for subscriber unknown.0 who wants epoch 740 2017-03-02 15:09:06.970420 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.133:0/3792964262 -- mdsmap(e 740) v1 -- 0x558a14d62400 con 0 2017-03-02 15:09:06.970456 7f4f763f4700 10 mon.0@0(leader).mds e740 update_logger 2017-03-02 15:09:06.970502 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:09:06.970524 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13567) refresh 2017-03-02 15:09:06.970527 7f4f763f4700 10 mon.0@0(leader).log v13567 update_from_paxos 2017-03-02 15:09:06.970530 7f4f763f4700 10 mon.0@0(leader).log v13567 update_from_paxos version 13567 summary v 13567 2017-03-02 15:09:06.970550 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:09:06.970573 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:09:06.970575 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:09:06.970597 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:09:06.970599 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12984) post_refresh 2017-03-02 15:09:06.970602 7f4f763f4700 10 mon.0@0(leader).pg v12984 post_paxos_update 2017-03-02 15:09:06.970605 7f4f763f4700 10 mon.0@0(leader).pg v12984 check_subs 2017-03-02 15:09:06.970609 7f4f763f4700 20 mon.0@0(leader).pg v12984 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:09:06.970616 7f4f763f4700 20 mon.0@0(leader).pg v12984 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:09:06.970620 7f4f763f4700 20 mon.0@0(leader).pg v12984 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:09:06.970623 7f4f763f4700 20 mon.0@0(leader).pg v12984 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:09:06.970627 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..740) post_refresh 2017-03-02 15:09:06.970629 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:09:06.970631 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13567) post_refresh 2017-03-02 15:09:06.970632 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:09:06.970634 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:09:06.970636 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:09:06.970639 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27374) commit_proposal 2017-03-02 15:09:06.970645 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..740) _active - not active 2017-03-02 15:09:06.970649 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27374) finish_round 2017-03-02 15:09:06.970653 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27374) finish_round waiting_for_acting 2017-03-02 15:09:06.970657 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..740) _active 2017-03-02 15:09:06.970661 7f4f763f4700 7 mon.0@0(leader).paxosservice(mdsmap 1..740) _active creating new pending 2017-03-02 15:09:06.970686 7f4f763f4700 10 mon.0@0(leader).mds e740 create_pending e741 2017-03-02 15:09:06.970695 7f4f763f4700 10 mon.0@0(leader).mds e740 _updated mds.5 mdsbeacon(4116/e up:active seq 8508 v739) v7 2017-03-02 15:09:06.970715 7f4f763f4700 0 log_channel(cluster) log [INF] : mds.5 172.21.15.11:6804/1637448123 up:active 2017-03-02 15:09:06.970742 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14133 at 2017-03-02 15:09:06.970718) v1 -- 0x558a14d62880 con 0 2017-03-02 15:09:06.970856 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a1621cf00 0x558a143e7740 mdsbeacon(4116/e up:active seq 8508 v740) v7 2017-03-02 15:09:06.970866 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6804/1637448123 -- mdsbeacon(4116/e up:active seq 8508 v740) v7 -- 0x558a143e7740 con 0 2017-03-02 15:09:06.970815 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.4 172.21.15.26:6805/2033388711 8565 ==== mdsbeacon(4117/b up:active seq 8508 v739) v7 ==== 365+0+0 (2738925100 0 0) 0x558a160c0700 con 0x558a14904000 2017-03-02 15:09:06.970922 7f4f763f4700 10 mon.0@0(leader).mds e740 e740: 7/7/7 up {0=d=up:active(laggy or crashed),1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:09:06.970960 7f4f763f4700 10 mon.0@0(leader).mds e740 no beacon from 4121 172.21.15.22:6805/2203629154 mds.0.6 up:active since 2017-03-02 15:08:46.885540 2017-03-02 15:09:06.970973 7f4f763f4700 10 mon.0@0(leader).mds e740 update_logger 2017-03-02 15:09:06.970990 7f4f763f4700 0 log_channel(cluster) log [INF] : fsmap e740: 7/7/7 up {0=d=up:active(laggy or crashed),1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:09:06.971005 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14134 at 2017-03-02 15:09:06.970992) v1 -- 0x558a14d62640 con 0 2017-03-02 15:09:06.971028 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27374) finish_round waiting_for_readable 2017-03-02 15:09:06.971034 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27374) finish_round waiting_for_writeable 2017-03-02 15:09:06.971038 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26858..27374) finish_round done w/ waiters, state active 2017-03-02 15:09:06.971070 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a148b1600 for mds.? 172.21.15.26:6805/2033388711 2017-03-02 15:09:06.971084 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:06.971089 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..740) dispatch 0x558a160c0700 mdsbeacon(4117/b up:active seq 8508 v739) v7 from mds.4 172.21.15.26:6805/2033388711 con 0x558a14904000 2017-03-02 15:09:06.971100 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27374) is_readable = 1 - now=2017-03-02 15:09:06.971101 lease_expire=0.000000 has v0 lc 27374 2017-03-02 15:09:06.971114 7f4f71beb700 10 mon.0@0(leader).mds e740 preprocess_query mdsbeacon(4117/b up:active seq 8508 v739) v7 from mds.4 172.21.15.26:6805/2033388711 2017-03-02 15:09:06.971126 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:09:06.971130 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:06.971133 7f4f71beb700 20 allow all 2017-03-02 15:09:06.971135 7f4f71beb700 12 mon.0@0(leader).mds e740 preprocess_beacon mdsbeacon(4117/b up:active seq 8508 v739) v7 from mds.4 172.21.15.26:6805/2033388711 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:09:06.971149 7f4f71beb700 10 mon.0@0(leader).mds e740 preprocess_beacon: GID exists in map: 4117 2017-03-02 15:09:06.971154 7f4f71beb700 10 mon.0@0(leader).mds e740 mds_beacon mdsbeacon(4117/b up:active seq 8508 v739) v7 ignoring requested state, because mds hasn't seen latest map 2017-03-02 15:09:06.971158 7f4f71beb700 15 mon.0@0(leader).mds e740 _note_beacon mdsbeacon(4117/b up:active seq 8508 v739) v7 noting time 2017-03-02 15:09:06.971165 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a15f72800 0x558a154b8d00 mdsbeacon(4117/b up:active seq 8508 v740) v7 2017-03-02 15:09:06.971170 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6805/2033388711 -- mdsbeacon(4117/b up:active seq 8508 v740) v7 -- 0x558a154b8d00 con 0 2017-03-02 15:09:06.971227 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14133 at 2017-03-02 15:09:06.970718) v1 ==== 0+0+0 (0 0 0) 0x558a14d62880 con 0x558a14482000 2017-03-02 15:09:06.971251 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:09:06.971259 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:06.971263 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13567) dispatch 0x558a14d62880 log(1 entries from seq 14133 at 2017-03-02 15:09:06.970718) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:09:06.971276 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27374) is_readable = 1 - now=2017-03-02 15:09:06.971277 lease_expire=0.000000 has v0 lc 27374 2017-03-02 15:09:06.971287 7f4f71beb700 10 mon.0@0(leader).log v13567 preprocess_query log(1 entries from seq 14133 at 2017-03-02 15:09:06.970718) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:09:06.971299 7f4f71beb700 10 mon.0@0(leader).log v13567 preprocess_log log(1 entries from seq 14133 at 2017-03-02 15:09:06.970718) v1 from mon.0 2017-03-02 15:09:06.971305 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:09:06.971308 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:06.971309 7f4f71beb700 20 allow all 2017-03-02 15:09:06.971319 7f4f71beb700 10 mon.0@0(leader).log v13567 prepare_update log(1 entries from seq 14133 at 2017-03-02 15:09:06.970718) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:09:06.971326 7f4f71beb700 10 mon.0@0(leader).log v13567 prepare_log log(1 entries from seq 14133 at 2017-03-02 15:09:06.970718) v1 from mon.0 2017-03-02 15:09:06.971330 7f4f71beb700 10 mon.0@0(leader).log v13567 logging 2017-03-02 15:09:06.970718 mon.0 172.21.15.17:6789/0 14133 : cluster [INF] mds.5 172.21.15.11:6804/1637448123 up:active 2017-03-02 15:09:06.971339 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13567) setting proposal_timer 0x558a1412f6d0 with delay of 0.998178 2017-03-02 15:09:06.971366 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14134 at 2017-03-02 15:09:06.970992) v1 ==== 0+0+0 (0 0 0) 0x558a14d62640 con 0x558a14482000 2017-03-02 15:09:06.971378 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:09:06.971383 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:06.971385 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13567) dispatch 0x558a14d62640 log(1 entries from seq 14134 at 2017-03-02 15:09:06.970992) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:09:06.971392 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27374) is_readable = 1 - now=2017-03-02 15:09:06.971393 lease_expire=0.000000 has v0 lc 27374 2017-03-02 15:09:06.971398 7f4f71beb700 10 mon.0@0(leader).log v13567 preprocess_query log(1 entries from seq 14134 at 2017-03-02 15:09:06.970992) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:09:06.971403 7f4f71beb700 10 mon.0@0(leader).log v13567 preprocess_log log(1 entries from seq 14134 at 2017-03-02 15:09:06.970992) v1 from mon.0 2017-03-02 15:09:06.971407 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:09:06.971409 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:06.971410 7f4f71beb700 20 allow all 2017-03-02 15:09:06.971415 7f4f71beb700 10 mon.0@0(leader).log v13567 prepare_update log(1 entries from seq 14134 at 2017-03-02 15:09:06.970992) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:09:06.971424 7f4f71beb700 10 mon.0@0(leader).log v13567 prepare_log log(1 entries from seq 14134 at 2017-03-02 15:09:06.970992) v1 from mon.0 2017-03-02 15:09:06.971430 7f4f71beb700 10 mon.0@0(leader).log v13567 logging 2017-03-02 15:09:06.970992 mon.0 172.21.15.17:6789/0 14134 : cluster [INF] fsmap e740: 7/7/7 up {0=d=up:active(laggy or crashed),1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:09:06.971444 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13567) proposal_timer already set 2017-03-02 15:09:06.979951 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mds.6 172.21.15.26:6804/2954782670 8758 ==== mdsbeacon(4115/a up:active seq 8508 v740) v7 ==== 365+0+0 (2021342847 0 0) 0x558a15204d00 con 0x558a1471f000 2017-03-02 15:09:06.979994 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a14868d60 for mds.? 172.21.15.26:6804/2954782670 2017-03-02 15:09:06.980002 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:06.980006 7f4f71beb700 10 mon.0@0(leader).paxosservice(mdsmap 1..740) dispatch 0x558a15204d00 mdsbeacon(4115/a up:active seq 8508 v740) v7 from mds.6 172.21.15.26:6804/2954782670 con 0x558a1471f000 2017-03-02 15:09:06.980015 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27374) is_readable = 1 - now=2017-03-02 15:09:06.980016 lease_expire=0.000000 has v0 lc 27374 2017-03-02 15:09:06.980027 7f4f71beb700 10 mon.0@0(leader).mds e740 preprocess_query mdsbeacon(4115/a up:active seq 8508 v740) v7 from mds.6 172.21.15.26:6804/2954782670 2017-03-02 15:09:06.980036 7f4f71beb700 20 is_capable service=mds command= exec on cap allow * 2017-03-02 15:09:06.980039 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:06.980041 7f4f71beb700 20 allow all 2017-03-02 15:09:06.980043 7f4f71beb700 12 mon.0@0(leader).mds e740 preprocess_beacon mdsbeacon(4115/a up:active seq 8508 v740) v7 from mds.6 172.21.15.26:6804/2954782670 compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file layout v2} 2017-03-02 15:09:06.980053 7f4f71beb700 10 mon.0@0(leader).mds e740 preprocess_beacon: GID exists in map: 4115 2017-03-02 15:09:06.980059 7f4f71beb700 15 mon.0@0(leader).mds e740 _note_beacon mdsbeacon(4115/a up:active seq 8508 v740) v7 noting time 2017-03-02 15:09:06.980064 7f4f71beb700 2 mon.0@0(leader) e2 send_reply 0x558a166afe00 0x558a160c0700 mdsbeacon(4115/a up:active seq 8508 v740) v7 2017-03-02 15:09:06.980068 7f4f71beb700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6804/2954782670 -- mdsbeacon(4115/a up:active seq 8508 v740) v7 -- 0x558a160c0700 con 0 2017-03-02 15:09:07.969666 7f4f743f0700 10 mon.0@0(leader).paxosservice(logm 13027..13567) propose_pending 2017-03-02 15:09:07.969696 7f4f743f0700 10 mon.0@0(leader).log v13567 encode_full log v 13567 2017-03-02 15:09:07.969798 7f4f743f0700 10 mon.0@0(leader).log v13567 encode_pending v13568 2017-03-02 15:09:07.969811 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26858..27374) queue_pending_finisher 0x558a1412cee0 2017-03-02 15:09:07.969817 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26858..27374) trigger_propose active, proposing now 2017-03-02 15:09:07.969867 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26858..27374) propose_pending 27375 10477 bytes 2017-03-02 15:09:07.969872 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26858..27374) begin for 27375 10477 bytes 2017-03-02 15:09:08.002838 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26858..27374) commit_start 27375 2017-03-02 15:09:08.036370 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26858..27374) commit_finish 27375 2017-03-02 15:09:08.036440 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:09:08.036463 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12984) refresh 2017-03-02 15:09:08.036479 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..740) refresh 2017-03-02 15:09:08.036493 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:09:08.036507 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13568) refresh 2017-03-02 15:09:08.036509 7f4f763f4700 10 mon.0@0(leader).log v13568 update_from_paxos 2017-03-02 15:09:08.036512 7f4f763f4700 10 mon.0@0(leader).log v13568 update_from_paxos version 13568 summary v 13567 2017-03-02 15:09:08.036516 7f4f763f4700 10 mon.0@0(leader).log v13568 update_from_paxos latest full 13567 2017-03-02 15:09:08.036527 7f4f763f4700 7 mon.0@0(leader).log v13568 update_from_paxos applying incremental log 13568 2017-03-02 15:09:06.970718 mon.0 172.21.15.17:6789/0 14133 : cluster [INF] mds.5 172.21.15.11:6804/1637448123 up:active 2017-03-02 15:09:08.036541 7f4f763f4700 20 mon.0@0(leader).log v13568 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:09:08.036563 7f4f763f4700 7 mon.0@0(leader).log v13568 update_from_paxos applying incremental log 13568 2017-03-02 15:09:06.970992 mon.0 172.21.15.17:6789/0 14134 : cluster [INF] fsmap e740: 7/7/7 up {0=d=up:active(laggy or crashed),1=g=up:active,2=c=up:active,3=f=up:active,4=b=up:active,5=e=up:active,6=a=up:active} 2017-03-02 15:09:08.036568 7f4f763f4700 20 mon.0@0(leader).log v13568 update_from_paxos logging for channel 'cluster' to file '/var/log/ceph/ceph.log' 2017-03-02 15:09:08.036575 7f4f763f4700 15 mon.0@0(leader).log v13568 update_from_paxos logging for 1 channels 2017-03-02 15:09:08.036578 7f4f763f4700 15 mon.0@0(leader).log v13568 update_from_paxos channel 'cluster' logging 334 bytes 2017-03-02 15:09:08.036658 7f4f763f4700 10 mon.0@0(leader).log v13568 check_subs 2017-03-02 15:09:08.036701 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:09:08.036723 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:09:08.036725 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:09:08.036740 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:09:08.036742 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12984) post_refresh 2017-03-02 15:09:08.036743 7f4f763f4700 10 mon.0@0(leader).pg v12984 post_paxos_update 2017-03-02 15:09:08.036747 7f4f763f4700 10 mon.0@0(leader).pg v12984 check_subs 2017-03-02 15:09:08.036750 7f4f763f4700 20 mon.0@0(leader).pg v12984 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:09:08.036755 7f4f763f4700 20 mon.0@0(leader).pg v12984 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:09:08.036757 7f4f763f4700 20 mon.0@0(leader).pg v12984 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:09:08.036759 7f4f763f4700 20 mon.0@0(leader).pg v12984 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:09:08.036762 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..740) post_refresh 2017-03-02 15:09:08.036763 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:09:08.036764 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13568) post_refresh 2017-03-02 15:09:08.036766 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:09:08.036767 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:09:08.036768 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:09:08.036770 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27375) commit_proposal 2017-03-02 15:09:08.036773 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13568) _active - not active 2017-03-02 15:09:08.036776 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27375) finish_round 2017-03-02 15:09:08.036777 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27375) finish_round waiting_for_acting 2017-03-02 15:09:08.036779 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13568) _active 2017-03-02 15:09:08.036781 7f4f763f4700 7 mon.0@0(leader).paxosservice(logm 13027..13568) _active creating new pending 2017-03-02 15:09:08.036790 7f4f763f4700 10 mon.0@0(leader).log v13568 create_pending v 13569 2017-03-02 15:09:08.036793 7f4f763f4700 7 mon.0@0(leader).log v13568 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:09:08.036797 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a15f72800 0x558a1528c960 log(last 14133) v1 2017-03-02 15:09:08.036802 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14133) v1 -- 0x558a1528c960 con 0 2017-03-02 15:09:08.036863 7f4f763f4700 7 mon.0@0(leader).log v13568 _updated_log for mon.0 172.21.15.17:6789/0 2017-03-02 15:09:08.036868 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a162ca600 0x558a151d01c0 log(last 14134) v1 2017-03-02 15:09:08.036871 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(last 14134) v1 -- 0x558a151d01c0 con 0 2017-03-02 15:09:08.036877 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27375) finish_round waiting_for_readable 2017-03-02 15:09:08.036879 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27375) finish_round waiting_for_writeable 2017-03-02 15:09:08.036881 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26858..27375) finish_round done w/ waiters, state active 2017-03-02 15:09:08.036932 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14133) v1 ==== 0+0+0 (0 0 0) 0x558a1528c960 con 0x558a14482000 2017-03-02 15:09:08.036961 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:09:08.036970 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:08.036975 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:09:08.036979 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:08.036981 7f4f71beb700 20 allow all 2017-03-02 15:09:08.037005 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(last 14134) v1 ==== 0+0+0 (0 0 0) 0x558a151d01c0 con 0x558a14482000 2017-03-02 15:09:08.037015 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:09:08.037020 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:08.037023 7f4f71beb700 20 is_capable service=mon command= read on cap allow * 2017-03-02 15:09:08.037025 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:08.037027 7f4f71beb700 20 allow all 2017-03-02 15:09:08.268562 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.3 172.21.15.17:6800/27100 6842 ==== pg_stats(1 pgs tid 6806 v 0) v1 ==== 698+0+0 (2576299217 0 0) 0x558a15ae2a00 con 0x558a145d7000 2017-03-02 15:09:08.268612 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415f440 for osd.3 172.21.15.17:6800/27100 2017-03-02 15:09:08.268624 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:08.268629 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12984) dispatch 0x558a15ae2a00 pg_stats(1 pgs tid 6806 v 0) v1 from osd.3 172.21.15.17:6800/27100 con 0x558a145d7000 2017-03-02 15:09:08.268641 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27375) is_readable = 1 - now=2017-03-02 15:09:08.268642 lease_expire=0.000000 has v0 lc 27375 2017-03-02 15:09:08.268657 7f4f71beb700 10 mon.0@0(leader).pg v12984 preprocess_query pg_stats(1 pgs tid 6806 v 0) v1 from osd.3 172.21.15.17:6800/27100 2017-03-02 15:09:08.268670 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:09:08.268676 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:08.268678 7f4f71beb700 20 allow all 2017-03-02 15:09:08.268681 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27375) is_readable = 1 - now=2017-03-02 15:09:08.268682 lease_expire=0.000000 has v0 lc 27375 2017-03-02 15:09:08.268693 7f4f71beb700 10 mon.0@0(leader).pg v12984 prepare_update pg_stats(1 pgs tid 6806 v 0) v1 from osd.3 172.21.15.17:6800/27100 2017-03-02 15:09:08.268703 7f4f71beb700 10 mon.0@0(leader).pg v12984 prepare_pg_stats pg_stats(1 pgs tid 6806 v 0) v1 from osd.3 2017-03-02 15:09:08.268716 7f4f71beb700 10 mon.0@0(leader).pg v12984 got osd.3 osd_stat(33004 MB used, 62336 MB avail, 95341 MB total, peers [0,1,2] op hist []) (was osd_stat(33528 MB used, 61812 MB avail, 95341 MB total, peers [0,1,2] op hist [0,0,1])) 2017-03-02 15:09:08.268734 7f4f71beb700 15 mon.0@0(leader).pg v12984 got 2.6 reported at 7:965959 state active+clean -> active+clean 2017-03-02 15:09:08.268754 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12984) setting proposal_timer 0x558a1412ecd0 with delay of 0.767655 2017-03-02 15:09:08.469807 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.1 172.21.15.22:6800/26531 6842 ==== pg_stats(5 pgs tid 6806 v 0) v1 ==== 2974+0+0 (201263026 0 0) 0x558a155d4f80 con 0x558a14643800 2017-03-02 15:09:08.469881 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1463b1e0 for osd.1 172.21.15.22:6800/26531 2017-03-02 15:09:08.469892 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:08.469898 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12984) dispatch 0x558a155d4f80 pg_stats(5 pgs tid 6806 v 0) v1 from osd.1 172.21.15.22:6800/26531 con 0x558a14643800 2017-03-02 15:09:08.469908 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27375) is_readable = 1 - now=2017-03-02 15:09:08.469909 lease_expire=0.000000 has v0 lc 27375 2017-03-02 15:09:08.469922 7f4f71beb700 10 mon.0@0(leader).pg v12984 preprocess_query pg_stats(5 pgs tid 6806 v 0) v1 from osd.1 172.21.15.22:6800/26531 2017-03-02 15:09:08.469933 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:09:08.469938 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:08.469940 7f4f71beb700 20 allow all 2017-03-02 15:09:08.469943 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27375) is_readable = 1 - now=2017-03-02 15:09:08.469944 lease_expire=0.000000 has v0 lc 27375 2017-03-02 15:09:08.469953 7f4f71beb700 10 mon.0@0(leader).pg v12984 prepare_update pg_stats(5 pgs tid 6806 v 0) v1 from osd.1 172.21.15.22:6800/26531 2017-03-02 15:09:08.469962 7f4f71beb700 10 mon.0@0(leader).pg v12984 prepare_pg_stats pg_stats(5 pgs tid 6806 v 0) v1 from osd.1 2017-03-02 15:09:08.469973 7f4f71beb700 10 mon.0@0(leader).pg v12984 got osd.1 osd_stat(41404 MB used, 53937 MB avail, 95341 MB total, peers [0,2,3] op hist [1,0,1]) (was osd_stat(42170 MB used, 53170 MB avail, 95341 MB total, peers [0,2,3] op hist [])) 2017-03-02 15:09:08.469990 7f4f71beb700 15 mon.0@0(leader).pg v12984 got 1.1 reported at 7:587342 state active+clean -> active+clean 2017-03-02 15:09:08.470007 7f4f71beb700 15 mon.0@0(leader).pg v12984 got 1.5 reported at 7:568228 state active+clean -> active+clean 2017-03-02 15:09:08.470018 7f4f71beb700 15 mon.0@0(leader).pg v12984 got 1.6 reported at 7:503618 state active+clean -> active+clean 2017-03-02 15:09:08.470027 7f4f71beb700 15 mon.0@0(leader).pg v12984 got 2.4 reported at 7:1027782 state active+clean -> active+clean 2017-03-02 15:09:08.470037 7f4f71beb700 15 mon.0@0(leader).pg v12984 got 2.5 reported at 7:1131144 state active+clean -> active+clean 2017-03-02 15:09:08.470048 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12984) proposal_timer already set 2017-03-02 15:09:08.563522 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.2 172.21.15.11:6800/26536 6842 ==== pg_stats(8 pgs tid 6806 v 0) v1 ==== 4668+0+0 (883434016 0 0) 0x558a15f24000 con 0x558a145d5800 2017-03-02 15:09:08.563563 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a145c3700 for osd.2 172.21.15.11:6800/26536 2017-03-02 15:09:08.563576 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:08.563581 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12984) dispatch 0x558a15f24000 pg_stats(8 pgs tid 6806 v 0) v1 from osd.2 172.21.15.11:6800/26536 con 0x558a145d5800 2017-03-02 15:09:08.563590 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27375) is_readable = 1 - now=2017-03-02 15:09:08.563591 lease_expire=0.000000 has v0 lc 27375 2017-03-02 15:09:08.563602 7f4f71beb700 10 mon.0@0(leader).pg v12984 preprocess_query pg_stats(8 pgs tid 6806 v 0) v1 from osd.2 172.21.15.11:6800/26536 2017-03-02 15:09:08.563613 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:09:08.563617 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:08.563619 7f4f71beb700 20 allow all 2017-03-02 15:09:08.563622 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27375) is_readable = 1 - now=2017-03-02 15:09:08.563622 lease_expire=0.000000 has v0 lc 27375 2017-03-02 15:09:08.563630 7f4f71beb700 10 mon.0@0(leader).pg v12984 prepare_update pg_stats(8 pgs tid 6806 v 0) v1 from osd.2 172.21.15.11:6800/26536 2017-03-02 15:09:08.563639 7f4f71beb700 10 mon.0@0(leader).pg v12984 prepare_pg_stats pg_stats(8 pgs tid 6806 v 0) v1 from osd.2 2017-03-02 15:09:08.563650 7f4f71beb700 10 mon.0@0(leader).pg v12984 got osd.2 osd_stat(50038 MB used, 45302 MB avail, 95341 MB total, peers [0,1,3] op hist [0,1]) (was osd_stat(50727 MB used, 44613 MB avail, 95341 MB total, peers [0,1,3] op hist [0,0,1])) 2017-03-02 15:09:08.563666 7f4f71beb700 15 mon.0@0(leader).pg v12984 got 1.0 reported at 7:497067 state active+clean -> active+clean 2017-03-02 15:09:08.563682 7f4f71beb700 15 mon.0@0(leader).pg v12984 got 1.2 reported at 7:577943 state active+clean -> active+clean 2017-03-02 15:09:08.563692 7f4f71beb700 15 mon.0@0(leader).pg v12984 got 1.7 reported at 7:530203 state active+clean -> active+clean 2017-03-02 15:09:08.563701 7f4f71beb700 15 mon.0@0(leader).pg v12984 got 2.0 reported at 7:1058387 state active+clean -> active+clean 2017-03-02 15:09:08.563709 7f4f71beb700 15 mon.0@0(leader).pg v12984 got 2.1 reported at 7:1091132 state active+clean -> active+clean 2017-03-02 15:09:08.563718 7f4f71beb700 15 mon.0@0(leader).pg v12984 got 2.2 reported at 7:1164877 state active+clean -> active+clean 2017-03-02 15:09:08.563726 7f4f71beb700 15 mon.0@0(leader).pg v12984 got 2.3 reported at 7:1096908 state active+clean -> active+clean 2017-03-02 15:09:08.563734 7f4f71beb700 15 mon.0@0(leader).pg v12984 got 2.7 reported at 7:1137060 state active+clean -> active+clean 2017-03-02 15:09:08.563744 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12984) proposal_timer already set 2017-03-02 15:09:08.720284 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== osd.0 172.21.15.26:6800/26515 6842 ==== pg_stats(2 pgs tid 6806 v 0) v1 ==== 1272+0+0 (1523226401 0 0) 0x558a15ca2a00 con 0x558a145d8800 2017-03-02 15:09:08.720333 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1463af20 for osd.0 172.21.15.26:6800/26515 2017-03-02 15:09:08.720342 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:08.720346 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12984) dispatch 0x558a15ca2a00 pg_stats(2 pgs tid 6806 v 0) v1 from osd.0 172.21.15.26:6800/26515 con 0x558a145d8800 2017-03-02 15:09:08.720354 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27375) is_readable = 1 - now=2017-03-02 15:09:08.720355 lease_expire=0.000000 has v0 lc 27375 2017-03-02 15:09:08.720367 7f4f71beb700 10 mon.0@0(leader).pg v12984 preprocess_query pg_stats(2 pgs tid 6806 v 0) v1 from osd.0 172.21.15.26:6800/26515 2017-03-02 15:09:08.720376 7f4f71beb700 20 is_capable service=pg command= read on cap allow * 2017-03-02 15:09:08.720380 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:08.720381 7f4f71beb700 20 allow all 2017-03-02 15:09:08.720383 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27375) is_readable = 1 - now=2017-03-02 15:09:08.720384 lease_expire=0.000000 has v0 lc 27375 2017-03-02 15:09:08.720390 7f4f71beb700 10 mon.0@0(leader).pg v12984 prepare_update pg_stats(2 pgs tid 6806 v 0) v1 from osd.0 172.21.15.26:6800/26515 2017-03-02 15:09:08.720397 7f4f71beb700 10 mon.0@0(leader).pg v12984 prepare_pg_stats pg_stats(2 pgs tid 6806 v 0) v1 from osd.0 2017-03-02 15:09:08.720406 7f4f71beb700 10 mon.0@0(leader).pg v12984 got osd.0 osd_stat(8717 MB used, 86624 MB avail, 95341 MB total, peers [1,2,3] op hist [0,1]) (was osd_stat(8809 MB used, 86531 MB avail, 95341 MB total, peers [1,2,3] op hist [])) 2017-03-02 15:09:08.720419 7f4f71beb700 15 mon.0@0(leader).pg v12984 got 1.3 reported at 7:500654 state active+clean -> active+clean 2017-03-02 15:09:08.720432 7f4f71beb700 15 mon.0@0(leader).pg v12984 got 1.4 reported at 7:486575 state active+clean -> active+clean 2017-03-02 15:09:08.720440 7f4f71beb700 10 mon.0@0(leader).paxosservice(pgmap 12269..12984) proposal_timer already set 2017-03-02 15:09:09.036543 7f4f743f0700 10 mon.0@0(leader).paxosservice(pgmap 12269..12984) propose_pending 2017-03-02 15:09:09.036571 7f4f743f0700 10 mon.0@0(leader).pg v12984 encode_pending v 12985 2017-03-02 15:09:09.036741 7f4f743f0700 5 mon.0@0(leader).paxos(paxos active c 26858..27375) queue_pending_finisher 0x558a1412f6d0 2017-03-02 15:09:09.036755 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26858..27375) trigger_propose active, proposing now 2017-03-02 15:09:09.036787 7f4f743f0700 10 mon.0@0(leader).paxos(paxos active c 26858..27375) propose_pending 27376 10487 bytes 2017-03-02 15:09:09.036798 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26858..27375) begin for 27376 10487 bytes 2017-03-02 15:09:09.078110 7f4f743f0700 10 mon.0@0(leader).paxos(paxos updating c 26858..27375) commit_start 27376 2017-03-02 15:09:09.111631 7f4f763f4700 20 mon.0@0(leader).paxos(paxos writing c 26858..27375) commit_finish 27376 2017-03-02 15:09:09.111702 7f4f763f4700 10 mon.0@0(leader) e2 refresh_from_paxos 2017-03-02 15:09:09.111731 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12985) refresh 2017-03-02 15:09:09.111743 7f4f763f4700 10 mon.0@0(leader).pg v12984 update_from_paxos read_incremental 2017-03-02 15:09:09.111776 7f4f763f4700 20 mon.0@0(leader).pg v12984 refreshing pg 1.0 7:497067 active+clean 2017-03-02 15:09:09.111796 7f4f763f4700 20 mon.0@0(leader).pg v12984 refreshing pg 1.1 7:587342 active+clean 2017-03-02 15:09:09.111807 7f4f763f4700 20 mon.0@0(leader).pg v12984 refreshing pg 1.2 7:577943 active+clean 2017-03-02 15:09:09.111816 7f4f763f4700 20 mon.0@0(leader).pg v12984 refreshing pg 1.3 7:500654 active+clean 2017-03-02 15:09:09.111866 7f4f763f4700 20 mon.0@0(leader).pg v12984 refreshing pg 1.4 7:486575 active+clean 2017-03-02 15:09:09.111876 7f4f763f4700 20 mon.0@0(leader).pg v12984 refreshing pg 1.5 7:568228 active+clean 2017-03-02 15:09:09.111885 7f4f763f4700 20 mon.0@0(leader).pg v12984 refreshing pg 1.6 7:503618 active+clean 2017-03-02 15:09:09.111893 7f4f763f4700 20 mon.0@0(leader).pg v12984 refreshing pg 1.7 7:530203 active+clean 2017-03-02 15:09:09.111902 7f4f763f4700 20 mon.0@0(leader).pg v12984 refreshing pg 2.0 7:1058387 active+clean 2017-03-02 15:09:09.111911 7f4f763f4700 20 mon.0@0(leader).pg v12984 refreshing pg 2.1 7:1091132 active+clean 2017-03-02 15:09:09.111919 7f4f763f4700 20 mon.0@0(leader).pg v12984 refreshing pg 2.2 7:1164877 active+clean 2017-03-02 15:09:09.111927 7f4f763f4700 20 mon.0@0(leader).pg v12984 refreshing pg 2.3 7:1096908 active+clean 2017-03-02 15:09:09.111935 7f4f763f4700 20 mon.0@0(leader).pg v12984 refreshing pg 2.4 7:1027782 active+clean 2017-03-02 15:09:09.111942 7f4f763f4700 20 mon.0@0(leader).pg v12984 refreshing pg 2.5 7:1131144 active+clean 2017-03-02 15:09:09.111950 7f4f763f4700 20 mon.0@0(leader).pg v12984 refreshing pg 2.6 7:965959 active+clean 2017-03-02 15:09:09.111958 7f4f763f4700 20 mon.0@0(leader).pg v12984 refreshing pg 2.7 7:1137060 active+clean 2017-03-02 15:09:09.111961 7f4f763f4700 20 mon.0@0(leader).pg v12984 refreshing osd.0 2017-03-02 15:09:09.111971 7f4f763f4700 20 mon.0@0(leader).pg v12984 refreshing osd.1 2017-03-02 15:09:09.111977 7f4f763f4700 20 mon.0@0(leader).pg v12984 refreshing osd.2 2017-03-02 15:09:09.111982 7f4f763f4700 20 mon.0@0(leader).pg v12984 refreshing osd.3 2017-03-02 15:09:09.111995 7f4f763f4700 10 mon.0@0(leader).pg v12985 read_pgmap_meta 2017-03-02 15:09:09.112024 7f4f763f4700 10 mon.0@0(leader).pg v12985 update_logger 2017-03-02 15:09:09.112046 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..740) refresh 2017-03-02 15:09:09.112065 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) refresh 2017-03-02 15:09:09.112085 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13568) refresh 2017-03-02 15:09:09.112087 7f4f763f4700 10 mon.0@0(leader).log v13568 update_from_paxos 2017-03-02 15:09:09.112090 7f4f763f4700 10 mon.0@0(leader).log v13568 update_from_paxos version 13568 summary v 13568 2017-03-02 15:09:09.112108 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) refresh 2017-03-02 15:09:09.112127 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) refresh 2017-03-02 15:09:09.112129 7f4f763f4700 10 mon.0@0(leader).auth v10 update_from_paxos 2017-03-02 15:09:09.112148 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) refresh 2017-03-02 15:09:09.112150 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12985) post_refresh 2017-03-02 15:09:09.112154 7f4f763f4700 10 mon.0@0(leader).pg v12985 post_paxos_update 2017-03-02 15:09:09.112156 7f4f763f4700 10 mon.0@0(leader).pg v12985 check_subs 2017-03-02 15:09:09.112159 7f4f763f4700 20 mon.0@0(leader).pg v12985 check_subs .. osd.3 172.21.15.17:6800/27100 2017-03-02 15:09:09.112170 7f4f763f4700 20 mon.0@0(leader).pg v12985 check_subs .. osd.2 172.21.15.11:6800/26536 2017-03-02 15:09:09.112174 7f4f763f4700 20 mon.0@0(leader).pg v12985 check_subs .. osd.0 172.21.15.26:6800/26515 2017-03-02 15:09:09.112177 7f4f763f4700 20 mon.0@0(leader).pg v12985 check_subs .. osd.1 172.21.15.22:6800/26531 2017-03-02 15:09:09.112180 7f4f763f4700 10 mon.0@0(leader).paxosservice(mdsmap 1..740) post_refresh 2017-03-02 15:09:09.112183 7f4f763f4700 10 mon.0@0(leader).paxosservice(osdmap 1..7) post_refresh 2017-03-02 15:09:09.112185 7f4f763f4700 10 mon.0@0(leader).paxosservice(logm 13027..13568) post_refresh 2017-03-02 15:09:09.112186 7f4f763f4700 10 mon.0@0(leader).paxosservice(monmap 1..2) post_refresh 2017-03-02 15:09:09.112188 7f4f763f4700 10 mon.0@0(leader).paxosservice(auth 1..10) post_refresh 2017-03-02 15:09:09.112190 7f4f763f4700 10 mon.0@0(leader).paxosservice(mgr 1..1) post_refresh 2017-03-02 15:09:09.112192 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27376) commit_proposal 2017-03-02 15:09:09.112196 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12985) _active - not active 2017-03-02 15:09:09.112199 7f4f763f4700 10 mon.0@0(leader).paxos(paxos refresh c 26858..27376) finish_round 2017-03-02 15:09:09.112201 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27376) finish_round waiting_for_acting 2017-03-02 15:09:09.112204 7f4f763f4700 10 mon.0@0(leader).paxosservice(pgmap 12269..12985) _active 2017-03-02 15:09:09.112206 7f4f763f4700 7 mon.0@0(leader).paxosservice(pgmap 12269..12985) _active creating new pending 2017-03-02 15:09:09.112256 7f4f763f4700 10 mon.0@0(leader).pg v12985 create_pending v 12986 2017-03-02 15:09:09.112263 7f4f763f4700 7 mon.0@0(leader).pg v12985 _updated_stats for osd.3 172.21.15.17:6800/27100 2017-03-02 15:09:09.112269 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a166afe00 0x558a150e0200 pg_stats_ack(1 pgs tid 6806) v1 2017-03-02 15:09:09.112274 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6800/27100 -- pg_stats_ack(1 pgs tid 6806) v1 -- 0x558a150e0200 con 0 2017-03-02 15:09:09.112323 7f4f763f4700 7 mon.0@0(leader).pg v12985 _updated_stats for osd.1 172.21.15.22:6800/26531 2017-03-02 15:09:09.112329 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a15c72c00 0x558a153b7e00 pg_stats_ack(5 pgs tid 6806) v1 2017-03-02 15:09:09.112332 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.22:6800/26531 -- pg_stats_ack(5 pgs tid 6806) v1 -- 0x558a153b7e00 con 0 2017-03-02 15:09:09.112347 7f4f763f4700 7 mon.0@0(leader).pg v12985 _updated_stats for osd.2 172.21.15.11:6800/26536 2017-03-02 15:09:09.112352 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a15de1200 0x558a14f22a00 pg_stats_ack(8 pgs tid 6806) v1 2017-03-02 15:09:09.112354 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.11:6800/26536 -- pg_stats_ack(8 pgs tid 6806) v1 -- 0x558a14f22a00 con 0 2017-03-02 15:09:09.112369 7f4f763f4700 7 mon.0@0(leader).pg v12985 _updated_stats for osd.0 172.21.15.26:6800/26515 2017-03-02 15:09:09.112372 7f4f763f4700 2 mon.0@0(leader) e2 send_reply 0x558a163bd100 0x558a15706a00 pg_stats_ack(2 pgs tid 6806) v1 2017-03-02 15:09:09.112375 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.26:6800/26515 -- pg_stats_ack(2 pgs tid 6806) v1 -- 0x558a15706a00 con 0 2017-03-02 15:09:09.112409 7f4f763f4700 10 mon.0@0(leader).pg v12985 check_osd_map already seen 7 >= 7 2017-03-02 15:09:09.112415 7f4f763f4700 10 mon.0@0(leader).pg v12985 update_logger 2017-03-02 15:09:09.112444 7f4f763f4700 0 log_channel(cluster) log [INF] : pgmap v12985: 32 pgs: 32 active+clean; 67985 MB data, 130 GB used, 242 GB / 372 GB avail; 0 B/s rd, 2030 kB/s wr, 406 op/s 2017-03-02 15:09:09.112472 7f4f763f4700 1 -- 172.21.15.17:6789/0 --> 172.21.15.17:6789/0 -- log(1 entries from seq 14135 at 2017-03-02 15:09:09.112448) v1 -- 0x558a14d62880 con 0 2017-03-02 15:09:09.112566 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27376) finish_round waiting_for_readable 2017-03-02 15:09:09.112574 7f4f763f4700 20 mon.0@0(leader).paxos(paxos active c 26858..27376) finish_round waiting_for_writeable 2017-03-02 15:09:09.112577 7f4f763f4700 10 mon.0@0(leader).paxos(paxos active c 26858..27376) finish_round done w/ waiters, state active 2017-03-02 15:09:09.112583 7f4f71beb700 1 -- 172.21.15.17:6789/0 <== mon.0 172.21.15.17:6789/0 0 ==== log(1 entries from seq 14135 at 2017-03-02 15:09:09.112448) v1 ==== 0+0+0 (0 0 0) 0x558a14d62880 con 0x558a14482000 2017-03-02 15:09:09.112616 7f4f71beb700 20 mon.0@0(leader) e2 _ms_dispatch existing session 0x558a1415d340 for mon.0 172.21.15.17:6789/0 2017-03-02 15:09:09.112623 7f4f71beb700 20 mon.0@0(leader) e2 caps allow * 2017-03-02 15:09:09.112626 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13568) dispatch 0x558a14d62880 log(1 entries from seq 14135 at 2017-03-02 15:09:09.112448) v1 from mon.0 172.21.15.17:6789/0 con 0x558a14482000 2017-03-02 15:09:09.112639 7f4f71beb700 5 mon.0@0(leader).paxos(paxos active c 26858..27376) is_readable = 1 - now=2017-03-02 15:09:09.112640 lease_expire=0.000000 has v0 lc 27376 2017-03-02 15:09:09.112651 7f4f71beb700 10 mon.0@0(leader).log v13568 preprocess_query log(1 entries from seq 14135 at 2017-03-02 15:09:09.112448) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:09:09.112663 7f4f71beb700 10 mon.0@0(leader).log v13568 preprocess_log log(1 entries from seq 14135 at 2017-03-02 15:09:09.112448) v1 from mon.0 2017-03-02 15:09:09.112672 7f4f71beb700 20 is_capable service=log command= write on cap allow * 2017-03-02 15:09:09.112677 7f4f71beb700 20 allow so far , doing grant allow * 2017-03-02 15:09:09.112693 7f4f71beb700 20 allow all 2017-03-02 15:09:09.112705 7f4f71beb700 10 mon.0@0(leader).log v13568 prepare_update log(1 entries from seq 14135 at 2017-03-02 15:09:09.112448) v1 from mon.0 172.21.15.17:6789/0 2017-03-02 15:09:09.112716 7f4f71beb700 10 mon.0@0(leader).log v13568 prepare_log log(1 entries from seq 14135 at 2017-03-02 15:09:09.112448) v1 from mon.0 2017-03-02 15:09:09.112723 7f4f71beb700 10 mon.0@0(leader).log v13568 logging 2017-03-02 15:09:09.112448 mon.0 172.21.15.17:6789/0 14135 : cluster [INF] pgmap v12985: 32 pgs: 32 active+clean; 67985 MB data, 130 GB used, 242 GB / 372 GB avail; 0 B/s rd, 2030 kB/s wr, 406 op/s 2017-03-02 15:09:09.112739 7f4f71beb700 10 mon.0@0(leader).paxosservice(logm 13027..13568) setting proposal_timer 0x558a1412e580 with delay of 0.998919