Project

General

Profile

ceph-deploy-ceph.log

min-sheng Lin, 06/23/2016 08:57 AM

Download (34.6 KB)

 
1
[2016-06-23 08:42:07,282][ceph_deploy.conf][DEBUG ] found configuration file at: /home/vagrant/.cephdeploy.conf
2
[2016-06-23 08:42:07,283][ceph_deploy.cli][INFO  ] Invoked (1.5.34): /usr/bin/ceph-deploy new ceph-mon-1 ceph-mon-2 ceph-mon-3
3
[2016-06-23 08:42:07,283][ceph_deploy.cli][INFO  ] ceph-deploy options:
4
[2016-06-23 08:42:07,283][ceph_deploy.cli][INFO  ]  username                      : None
5
[2016-06-23 08:42:07,283][ceph_deploy.cli][INFO  ]  func                          : <function new at 0x7f903b6c8578>
6
[2016-06-23 08:42:07,283][ceph_deploy.cli][INFO  ]  verbose                       : False
7
[2016-06-23 08:42:07,283][ceph_deploy.cli][INFO  ]  overwrite_conf                : False
8
[2016-06-23 08:42:07,283][ceph_deploy.cli][INFO  ]  quiet                         : False
9
[2016-06-23 08:42:07,284][ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f903b25f8c0>
10
[2016-06-23 08:42:07,284][ceph_deploy.cli][INFO  ]  cluster                       : ceph
11
[2016-06-23 08:42:07,284][ceph_deploy.cli][INFO  ]  ssh_copykey                   : True
12
[2016-06-23 08:42:07,284][ceph_deploy.cli][INFO  ]  mon                           : ['ceph-mon-1', 'ceph-mon-2', 'ceph-mon-3']
13
[2016-06-23 08:42:07,284][ceph_deploy.cli][INFO  ]  public_network                : None
14
[2016-06-23 08:42:07,284][ceph_deploy.cli][INFO  ]  ceph_conf                     : None
15
[2016-06-23 08:42:07,284][ceph_deploy.cli][INFO  ]  cluster_network               : None
16
[2016-06-23 08:42:07,284][ceph_deploy.cli][INFO  ]  default_release               : False
17
[2016-06-23 08:42:07,285][ceph_deploy.cli][INFO  ]  fsid                          : None
18
[2016-06-23 08:42:07,285][ceph_deploy.new][DEBUG ] Creating new cluster named ceph
19
[2016-06-23 08:42:07,285][ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
20
[2016-06-23 08:42:07,305][ceph-mon-1][DEBUG ] connection detected need for sudo
21
[2016-06-23 08:42:07,317][ceph-mon-1][DEBUG ] connected to host: ceph-mon-1 
22
[2016-06-23 08:42:07,318][ceph-mon-1][DEBUG ] detect platform information from remote host
23
[2016-06-23 08:42:07,331][ceph-mon-1][DEBUG ] detect machine type
24
[2016-06-23 08:42:07,332][ceph-mon-1][DEBUG ] find the location of an executable
25
[2016-06-23 08:42:07,333][ceph-mon-1][INFO  ] Running command: sudo /sbin/initctl version
26
[2016-06-23 08:42:07,339][ceph-mon-1][DEBUG ] find the location of an executable
27
[2016-06-23 08:42:07,340][ceph-mon-1][INFO  ] Running command: sudo /bin/ip link show
28
[2016-06-23 08:42:07,344][ceph-mon-1][INFO  ] Running command: sudo /bin/ip addr show
29
[2016-06-23 08:42:07,348][ceph-mon-1][DEBUG ] IP addresses found: ['192.168.121.221', '192.168.20.11']
30
[2016-06-23 08:42:07,349][ceph_deploy.new][DEBUG ] Resolving host ceph-mon-1
31
[2016-06-23 08:42:07,349][ceph_deploy.new][DEBUG ] Monitor ceph-mon-1 at 192.168.20.11
32
[2016-06-23 08:42:07,349][ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
33
[2016-06-23 08:42:07,364][ceph-mon-2][DEBUG ] connected to host: ceph-mon-1 
34
[2016-06-23 08:42:07,367][ceph-mon-2][INFO  ] Running command: ssh -CT -o BatchMode=yes ceph-mon-2
35
[2016-06-23 08:42:07,432][ceph_deploy.new][WARNING] could not connect via SSH
36
[2016-06-23 08:42:07,433][ceph_deploy.new][INFO  ] creating a passwordless id_rsa.pub key file
37
[2016-06-23 08:42:07,443][ceph_deploy.new][DEBUG ] connected to host: ceph-mon-1 
38
[2016-06-23 08:42:07,446][ceph_deploy.new][INFO  ] Running command: ssh-keygen -t rsa -N  -f /home/vagrant/.ssh/id_rsa
39
[2016-06-23 08:42:07,763][ceph_deploy.new][DEBUG ] Generating public/private rsa key pair.
40
[2016-06-23 08:42:07,763][ceph_deploy.new][DEBUG ] Your identification has been saved in /home/vagrant/.ssh/id_rsa.
41
[2016-06-23 08:42:07,763][ceph_deploy.new][DEBUG ] Your public key has been saved in /home/vagrant/.ssh/id_rsa.pub.
42
[2016-06-23 08:42:07,764][ceph_deploy.new][DEBUG ] The key fingerprint is:
43
[2016-06-23 08:42:07,764][ceph_deploy.new][DEBUG ] 24:73:b8:59:f2:1d:bc:f2:c2:06:9b:f4:1f:c8:d7:a4 vagrant@ceph-mon-1
44
[2016-06-23 08:42:07,764][ceph_deploy.new][DEBUG ] The key's randomart image is:
45
[2016-06-23 08:42:07,764][ceph_deploy.new][DEBUG ] +--[ RSA 2048]----+
46
[2016-06-23 08:42:07,764][ceph_deploy.new][DEBUG ] |                 |
47
[2016-06-23 08:42:07,764][ceph_deploy.new][DEBUG ] |       . .       |
48
[2016-06-23 08:42:07,764][ceph_deploy.new][DEBUG ] |      = + o      |
49
[2016-06-23 08:42:07,765][ceph_deploy.new][DEBUG ] |       @ . o     |
50
[2016-06-23 08:42:07,765][ceph_deploy.new][DEBUG ] |      = S o .    |
51
[2016-06-23 08:42:07,765][ceph_deploy.new][DEBUG ] |     . B + +     |
52
[2016-06-23 08:42:07,765][ceph_deploy.new][DEBUG ] |      o B E .    |
53
[2016-06-23 08:42:07,765][ceph_deploy.new][DEBUG ] |       . + .     |
54
[2016-06-23 08:42:07,765][ceph_deploy.new][DEBUG ] |          .      |
55
[2016-06-23 08:42:07,765][ceph_deploy.new][DEBUG ] +-----------------+
56
[2016-06-23 08:42:07,766][ceph_deploy.new][INFO  ] will connect again with password prompt
57
[2016-06-23 08:42:09,713][ceph-mon-2][DEBUG ] connected to host: ceph-mon-2 
58
[2016-06-23 08:42:09,714][ceph-mon-2][DEBUG ] detect platform information from remote host
59
[2016-06-23 08:42:09,724][ceph-mon-2][DEBUG ] detect machine type
60
[2016-06-23 08:42:09,727][ceph-mon-2][DEBUG ] find the location of an executable
61
[2016-06-23 08:42:09,729][ceph-mon-2][INFO  ] Running command: /sbin/initctl version
62
[2016-06-23 08:42:09,738][ceph_deploy.new][INFO  ] adding public keys to authorized_keys
63
[2016-06-23 08:42:09,738][ceph-mon-2][DEBUG ] append contents to file
64
[2016-06-23 08:42:09,941][ceph-mon-2][DEBUG ] connection detected need for sudo
65
[2016-06-23 08:42:10,140][ceph-mon-2][DEBUG ] connected to host: ceph-mon-2 
66
[2016-06-23 08:42:10,141][ceph-mon-2][DEBUG ] detect platform information from remote host
67
[2016-06-23 08:42:10,151][ceph-mon-2][DEBUG ] detect machine type
68
[2016-06-23 08:42:10,154][ceph-mon-2][DEBUG ] find the location of an executable
69
[2016-06-23 08:42:10,156][ceph-mon-2][INFO  ] Running command: sudo /sbin/initctl version
70
[2016-06-23 08:42:10,164][ceph-mon-2][DEBUG ] find the location of an executable
71
[2016-06-23 08:42:10,166][ceph-mon-2][INFO  ] Running command: sudo /bin/ip link show
72
[2016-06-23 08:42:10,171][ceph-mon-2][INFO  ] Running command: sudo /bin/ip addr show
73
[2016-06-23 08:42:10,175][ceph-mon-2][DEBUG ] IP addresses found: ['192.168.121.43', '192.168.20.12']
74
[2016-06-23 08:42:10,175][ceph_deploy.new][DEBUG ] Resolving host ceph-mon-2
75
[2016-06-23 08:42:10,176][ceph_deploy.new][DEBUG ] Monitor ceph-mon-2 at 192.168.20.12
76
[2016-06-23 08:42:10,176][ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
77
[2016-06-23 08:42:10,186][ceph-mon-3][DEBUG ] connected to host: ceph-mon-1 
78
[2016-06-23 08:42:10,189][ceph-mon-3][INFO  ] Running command: ssh -CT -o BatchMode=yes ceph-mon-3
79
[2016-06-23 08:42:10,254][ceph_deploy.new][WARNING] could not connect via SSH
80
[2016-06-23 08:42:10,254][ceph_deploy.new][INFO  ] will connect again with password prompt
81
[2016-06-23 08:42:11,860][ceph-mon-3][DEBUG ] connected to host: ceph-mon-3 
82
[2016-06-23 08:42:11,861][ceph-mon-3][DEBUG ] detect platform information from remote host
83
[2016-06-23 08:42:11,871][ceph-mon-3][DEBUG ] detect machine type
84
[2016-06-23 08:42:11,873][ceph-mon-3][DEBUG ] find the location of an executable
85
[2016-06-23 08:42:11,875][ceph-mon-3][INFO  ] Running command: /sbin/initctl version
86
[2016-06-23 08:42:11,884][ceph_deploy.new][INFO  ] adding public keys to authorized_keys
87
[2016-06-23 08:42:11,884][ceph-mon-3][DEBUG ] append contents to file
88
[2016-06-23 08:42:12,084][ceph-mon-3][DEBUG ] connection detected need for sudo
89
[2016-06-23 08:42:12,283][ceph-mon-3][DEBUG ] connected to host: ceph-mon-3 
90
[2016-06-23 08:42:12,284][ceph-mon-3][DEBUG ] detect platform information from remote host
91
[2016-06-23 08:42:12,294][ceph-mon-3][DEBUG ] detect machine type
92
[2016-06-23 08:42:12,297][ceph-mon-3][DEBUG ] find the location of an executable
93
[2016-06-23 08:42:12,299][ceph-mon-3][INFO  ] Running command: sudo /sbin/initctl version
94
[2016-06-23 08:42:12,307][ceph-mon-3][DEBUG ] find the location of an executable
95
[2016-06-23 08:42:12,309][ceph-mon-3][INFO  ] Running command: sudo /bin/ip link show
96
[2016-06-23 08:42:12,313][ceph-mon-3][INFO  ] Running command: sudo /bin/ip addr show
97
[2016-06-23 08:42:12,318][ceph-mon-3][DEBUG ] IP addresses found: ['192.168.20.13', '192.168.121.62']
98
[2016-06-23 08:42:12,318][ceph_deploy.new][DEBUG ] Resolving host ceph-mon-3
99
[2016-06-23 08:42:12,319][ceph_deploy.new][DEBUG ] Monitor ceph-mon-3 at 192.168.20.13
100
[2016-06-23 08:42:12,319][ceph_deploy.new][DEBUG ] Monitor initial members are ['ceph-mon-1', 'ceph-mon-2', 'ceph-mon-3']
101
[2016-06-23 08:42:12,319][ceph_deploy.new][DEBUG ] Monitor addrs are ['192.168.20.11', '192.168.20.12', '192.168.20.13']
102
[2016-06-23 08:42:12,319][ceph_deploy.new][DEBUG ] Creating a random mon key...
103
[2016-06-23 08:42:12,319][ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
104
[2016-06-23 08:42:12,319][ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...
105
[2016-06-23 08:42:24,074][ceph_deploy.conf][DEBUG ] found configuration file at: /home/vagrant/.cephdeploy.conf
106
[2016-06-23 08:42:24,074][ceph_deploy.cli][INFO  ] Invoked (1.5.34): /usr/bin/ceph-deploy mon create-initial
107
[2016-06-23 08:42:24,074][ceph_deploy.cli][INFO  ] ceph-deploy options:
108
[2016-06-23 08:42:24,074][ceph_deploy.cli][INFO  ]  username                      : None
109
[2016-06-23 08:42:24,075][ceph_deploy.cli][INFO  ]  verbose                       : False
110
[2016-06-23 08:42:24,075][ceph_deploy.cli][INFO  ]  overwrite_conf                : False
111
[2016-06-23 08:42:24,075][ceph_deploy.cli][INFO  ]  subcommand                    : create-initial
112
[2016-06-23 08:42:24,075][ceph_deploy.cli][INFO  ]  quiet                         : False
113
[2016-06-23 08:42:24,075][ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7eff94d2ff80>
114
[2016-06-23 08:42:24,075][ceph_deploy.cli][INFO  ]  cluster                       : ceph
115
[2016-06-23 08:42:24,075][ceph_deploy.cli][INFO  ]  func                          : <function mon at 0x7eff95195b18>
116
[2016-06-23 08:42:24,075][ceph_deploy.cli][INFO  ]  ceph_conf                     : None
117
[2016-06-23 08:42:24,075][ceph_deploy.cli][INFO  ]  default_release               : False
118
[2016-06-23 08:42:24,076][ceph_deploy.cli][INFO  ]  keyrings                      : None
119
[2016-06-23 08:42:24,076][ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-mon-1 ceph-mon-2 ceph-mon-3
120
[2016-06-23 08:42:24,076][ceph_deploy.mon][DEBUG ] detecting platform for host ceph-mon-1 ...
121
[2016-06-23 08:42:24,099][ceph-mon-1][DEBUG ] connection detected need for sudo
122
[2016-06-23 08:42:24,111][ceph-mon-1][DEBUG ] connected to host: ceph-mon-1 
123
[2016-06-23 08:42:24,111][ceph-mon-1][DEBUG ] detect platform information from remote host
124
[2016-06-23 08:42:24,125][ceph-mon-1][DEBUG ] detect machine type
125
[2016-06-23 08:42:24,127][ceph-mon-1][DEBUG ] find the location of an executable
126
[2016-06-23 08:42:24,127][ceph-mon-1][INFO  ] Running command: sudo /sbin/initctl version
127
[2016-06-23 08:42:24,133][ceph-mon-1][DEBUG ] find the location of an executable
128
[2016-06-23 08:42:24,133][ceph_deploy.mon][INFO  ] distro info: Ubuntu 14.04 trusty
129
[2016-06-23 08:42:24,133][ceph-mon-1][DEBUG ] determining if provided host has same hostname in remote
130
[2016-06-23 08:42:24,134][ceph-mon-1][DEBUG ] get remote short hostname
131
[2016-06-23 08:42:24,134][ceph-mon-1][DEBUG ] deploying mon to ceph-mon-1
132
[2016-06-23 08:42:24,134][ceph-mon-1][DEBUG ] get remote short hostname
133
[2016-06-23 08:42:24,134][ceph-mon-1][DEBUG ] remote hostname: ceph-mon-1
134
[2016-06-23 08:42:24,135][ceph-mon-1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
135
[2016-06-23 08:42:24,135][ceph-mon-1][DEBUG ] create the mon path if it does not exist
136
[2016-06-23 08:42:24,136][ceph-mon-1][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-ceph-mon-1/done
137
[2016-06-23 08:42:24,136][ceph-mon-1][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-ceph-mon-1/done
138
[2016-06-23 08:42:24,136][ceph-mon-1][INFO  ] creating keyring file: /var/lib/ceph/tmp/ceph-ceph-mon-1.mon.keyring
139
[2016-06-23 08:42:24,137][ceph-mon-1][DEBUG ] create the monitor keyring file
140
[2016-06-23 08:42:24,137][ceph-mon-1][INFO  ] Running command: sudo ceph-mon --cluster ceph --mkfs -i ceph-mon-1 --keyring /var/lib/ceph/tmp/ceph-ceph-mon-1.mon.keyring
141
[2016-06-23 08:42:24,171][ceph-mon-1][DEBUG ] ceph-mon: mon.noname-a 192.168.20.11:6789/0 is local, renaming to mon.ceph-mon-1
142
[2016-06-23 08:42:24,171][ceph-mon-1][DEBUG ] ceph-mon: set fsid to 1bf3874b-1267-453b-9210-bb51e4aa3075
143
[2016-06-23 08:42:24,175][ceph-mon-1][DEBUG ] ceph-mon: created monfs at /var/lib/ceph/mon/ceph-ceph-mon-1 for mon.ceph-mon-1
144
[2016-06-23 08:42:24,175][ceph-mon-1][INFO  ] unlinking keyring file /var/lib/ceph/tmp/ceph-ceph-mon-1.mon.keyring
145
[2016-06-23 08:42:24,176][ceph-mon-1][DEBUG ] create a done file to avoid re-doing the mon deployment
146
[2016-06-23 08:42:24,176][ceph-mon-1][DEBUG ] create the init path if it does not exist
147
[2016-06-23 08:42:24,177][ceph-mon-1][INFO  ] Running command: sudo initctl emit ceph-mon cluster=ceph id=ceph-mon-1
148
[2016-06-23 08:42:26,230][ceph-mon-1][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-mon-1.asok mon_status
149
[2016-06-23 08:42:26,294][ceph-mon-1][DEBUG ] ********************************************************************************
150
[2016-06-23 08:42:26,294][ceph-mon-1][DEBUG ] status for monitor: mon.ceph-mon-1
151
[2016-06-23 08:42:26,294][ceph-mon-1][DEBUG ] {
152
[2016-06-23 08:42:26,295][ceph-mon-1][DEBUG ]   "election_epoch": 0, 
153
[2016-06-23 08:42:26,295][ceph-mon-1][DEBUG ]   "extra_probe_peers": [
154
[2016-06-23 08:42:26,295][ceph-mon-1][DEBUG ]     "192.168.20.12:6789/0", 
155
[2016-06-23 08:42:26,295][ceph-mon-1][DEBUG ]     "192.168.20.13:6789/0"
156
[2016-06-23 08:42:26,295][ceph-mon-1][DEBUG ]   ], 
157
[2016-06-23 08:42:26,295][ceph-mon-1][DEBUG ]   "monmap": {
158
[2016-06-23 08:42:26,295][ceph-mon-1][DEBUG ]     "created": "0.000000", 
159
[2016-06-23 08:42:26,295][ceph-mon-1][DEBUG ]     "epoch": 0, 
160
[2016-06-23 08:42:26,295][ceph-mon-1][DEBUG ]     "fsid": "1bf3874b-1267-453b-9210-bb51e4aa3075", 
161
[2016-06-23 08:42:26,295][ceph-mon-1][DEBUG ]     "modified": "0.000000", 
162
[2016-06-23 08:42:26,296][ceph-mon-1][DEBUG ]     "mons": [
163
[2016-06-23 08:42:26,296][ceph-mon-1][DEBUG ]       {
164
[2016-06-23 08:42:26,296][ceph-mon-1][DEBUG ]         "addr": "192.168.20.11:6789/0", 
165
[2016-06-23 08:42:26,296][ceph-mon-1][DEBUG ]         "name": "ceph-mon-1", 
166
[2016-06-23 08:42:26,296][ceph-mon-1][DEBUG ]         "rank": 0
167
[2016-06-23 08:42:26,296][ceph-mon-1][DEBUG ]       }, 
168
[2016-06-23 08:42:26,296][ceph-mon-1][DEBUG ]       {
169
[2016-06-23 08:42:26,296][ceph-mon-1][DEBUG ]         "addr": "0.0.0.0:0/1", 
170
[2016-06-23 08:42:26,296][ceph-mon-1][DEBUG ]         "name": "ceph-mon-2", 
171
[2016-06-23 08:42:26,296][ceph-mon-1][DEBUG ]         "rank": 1
172
[2016-06-23 08:42:26,297][ceph-mon-1][DEBUG ]       }, 
173
[2016-06-23 08:42:26,297][ceph-mon-1][DEBUG ]       {
174
[2016-06-23 08:42:26,297][ceph-mon-1][DEBUG ]         "addr": "0.0.0.0:0/2", 
175
[2016-06-23 08:42:26,297][ceph-mon-1][DEBUG ]         "name": "ceph-mon-3", 
176
[2016-06-23 08:42:26,297][ceph-mon-1][DEBUG ]         "rank": 2
177
[2016-06-23 08:42:26,297][ceph-mon-1][DEBUG ]       }
178
[2016-06-23 08:42:26,297][ceph-mon-1][DEBUG ]     ]
179
[2016-06-23 08:42:26,297][ceph-mon-1][DEBUG ]   }, 
180
[2016-06-23 08:42:26,297][ceph-mon-1][DEBUG ]   "name": "ceph-mon-1", 
181
[2016-06-23 08:42:26,297][ceph-mon-1][DEBUG ]   "outside_quorum": [
182
[2016-06-23 08:42:26,297][ceph-mon-1][DEBUG ]     "ceph-mon-1"
183
[2016-06-23 08:42:26,297][ceph-mon-1][DEBUG ]   ], 
184
[2016-06-23 08:42:26,298][ceph-mon-1][DEBUG ]   "quorum": [], 
185
[2016-06-23 08:42:26,298][ceph-mon-1][DEBUG ]   "rank": 0, 
186
[2016-06-23 08:42:26,298][ceph-mon-1][DEBUG ]   "state": "probing", 
187
[2016-06-23 08:42:26,298][ceph-mon-1][DEBUG ]   "sync_provider": []
188
[2016-06-23 08:42:26,298][ceph-mon-1][DEBUG ] }
189
[2016-06-23 08:42:26,298][ceph-mon-1][DEBUG ] ********************************************************************************
190
[2016-06-23 08:42:26,298][ceph-mon-1][INFO  ] monitor: mon.ceph-mon-1 is running
191
[2016-06-23 08:42:26,299][ceph-mon-1][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-mon-1.asok mon_status
192
[2016-06-23 08:42:26,371][ceph_deploy.mon][DEBUG ] detecting platform for host ceph-mon-2 ...
193
[2016-06-23 08:42:26,580][ceph-mon-2][DEBUG ] connection detected need for sudo
194
[2016-06-23 08:42:26,776][ceph-mon-2][DEBUG ] connected to host: ceph-mon-2 
195
[2016-06-23 08:42:26,776][ceph-mon-2][DEBUG ] detect platform information from remote host
196
[2016-06-23 08:42:26,787][ceph-mon-2][DEBUG ] detect machine type
197
[2016-06-23 08:42:26,790][ceph-mon-2][DEBUG ] find the location of an executable
198
[2016-06-23 08:42:26,792][ceph-mon-2][INFO  ] Running command: sudo /sbin/initctl version
199
[2016-06-23 08:42:26,800][ceph-mon-2][DEBUG ] find the location of an executable
200
[2016-06-23 08:42:26,801][ceph_deploy.mon][INFO  ] distro info: Ubuntu 14.04 trusty
201
[2016-06-23 08:42:26,801][ceph-mon-2][DEBUG ] determining if provided host has same hostname in remote
202
[2016-06-23 08:42:26,801][ceph-mon-2][DEBUG ] get remote short hostname
203
[2016-06-23 08:42:26,802][ceph-mon-2][DEBUG ] deploying mon to ceph-mon-2
204
[2016-06-23 08:42:26,802][ceph-mon-2][DEBUG ] get remote short hostname
205
[2016-06-23 08:42:26,802][ceph-mon-2][DEBUG ] remote hostname: ceph-mon-2
206
[2016-06-23 08:42:26,803][ceph-mon-2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
207
[2016-06-23 08:42:26,804][ceph-mon-2][DEBUG ] create the mon path if it does not exist
208
[2016-06-23 08:42:26,805][ceph-mon-2][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-ceph-mon-2/done
209
[2016-06-23 08:42:26,805][ceph-mon-2][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-ceph-mon-2/done
210
[2016-06-23 08:42:26,806][ceph-mon-2][INFO  ] creating keyring file: /var/lib/ceph/tmp/ceph-ceph-mon-2.mon.keyring
211
[2016-06-23 08:42:26,806][ceph-mon-2][DEBUG ] create the monitor keyring file
212
[2016-06-23 08:42:26,807][ceph-mon-2][INFO  ] Running command: sudo ceph-mon --cluster ceph --mkfs -i ceph-mon-2 --keyring /var/lib/ceph/tmp/ceph-ceph-mon-2.mon.keyring
213
[2016-06-23 08:42:26,825][ceph-mon-2][DEBUG ] ceph-mon: mon.noname-b 192.168.20.12:6789/0 is local, renaming to mon.ceph-mon-2
214
[2016-06-23 08:42:26,825][ceph-mon-2][DEBUG ] ceph-mon: set fsid to 1bf3874b-1267-453b-9210-bb51e4aa3075
215
[2016-06-23 08:42:26,841][ceph-mon-2][DEBUG ] ceph-mon: created monfs at /var/lib/ceph/mon/ceph-ceph-mon-2 for mon.ceph-mon-2
216
[2016-06-23 08:42:26,845][ceph-mon-2][INFO  ] unlinking keyring file /var/lib/ceph/tmp/ceph-ceph-mon-2.mon.keyring
217
[2016-06-23 08:42:26,845][ceph-mon-2][DEBUG ] create a done file to avoid re-doing the mon deployment
218
[2016-06-23 08:42:26,846][ceph-mon-2][DEBUG ] create the init path if it does not exist
219
[2016-06-23 08:42:26,847][ceph-mon-2][INFO  ] Running command: sudo initctl emit ceph-mon cluster=ceph id=ceph-mon-2
220
[2016-06-23 08:42:28,888][ceph-mon-2][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-mon-2.asok mon_status
221
[2016-06-23 08:42:28,956][ceph-mon-2][DEBUG ] ********************************************************************************
222
[2016-06-23 08:42:28,956][ceph-mon-2][DEBUG ] status for monitor: mon.ceph-mon-2
223
[2016-06-23 08:42:28,956][ceph-mon-2][DEBUG ] {
224
[2016-06-23 08:42:28,957][ceph-mon-2][DEBUG ]   "election_epoch": 1, 
225
[2016-06-23 08:42:28,957][ceph-mon-2][DEBUG ]   "extra_probe_peers": [
226
[2016-06-23 08:42:28,957][ceph-mon-2][DEBUG ]     "192.168.20.11:6789/0", 
227
[2016-06-23 08:42:28,957][ceph-mon-2][DEBUG ]     "192.168.20.13:6789/0"
228
[2016-06-23 08:42:28,957][ceph-mon-2][DEBUG ]   ], 
229
[2016-06-23 08:42:28,957][ceph-mon-2][DEBUG ]   "monmap": {
230
[2016-06-23 08:42:28,958][ceph-mon-2][DEBUG ]     "created": "0.000000", 
231
[2016-06-23 08:42:28,958][ceph-mon-2][DEBUG ]     "epoch": 0, 
232
[2016-06-23 08:42:28,958][ceph-mon-2][DEBUG ]     "fsid": "1bf3874b-1267-453b-9210-bb51e4aa3075", 
233
[2016-06-23 08:42:28,958][ceph-mon-2][DEBUG ]     "modified": "0.000000", 
234
[2016-06-23 08:42:28,958][ceph-mon-2][DEBUG ]     "mons": [
235
[2016-06-23 08:42:28,958][ceph-mon-2][DEBUG ]       {
236
[2016-06-23 08:42:28,959][ceph-mon-2][DEBUG ]         "addr": "192.168.20.11:6789/0", 
237
[2016-06-23 08:42:28,959][ceph-mon-2][DEBUG ]         "name": "ceph-mon-1", 
238
[2016-06-23 08:42:28,959][ceph-mon-2][DEBUG ]         "rank": 0
239
[2016-06-23 08:42:28,959][ceph-mon-2][DEBUG ]       }, 
240
[2016-06-23 08:42:28,959][ceph-mon-2][DEBUG ]       {
241
[2016-06-23 08:42:28,959][ceph-mon-2][DEBUG ]         "addr": "192.168.20.12:6789/0", 
242
[2016-06-23 08:42:28,960][ceph-mon-2][DEBUG ]         "name": "ceph-mon-2", 
243
[2016-06-23 08:42:28,960][ceph-mon-2][DEBUG ]         "rank": 1
244
[2016-06-23 08:42:28,960][ceph-mon-2][DEBUG ]       }, 
245
[2016-06-23 08:42:28,960][ceph-mon-2][DEBUG ]       {
246
[2016-06-23 08:42:28,960][ceph-mon-2][DEBUG ]         "addr": "0.0.0.0:0/2", 
247
[2016-06-23 08:42:28,960][ceph-mon-2][DEBUG ]         "name": "ceph-mon-3", 
248
[2016-06-23 08:42:28,960][ceph-mon-2][DEBUG ]         "rank": 2
249
[2016-06-23 08:42:28,961][ceph-mon-2][DEBUG ]       }
250
[2016-06-23 08:42:28,961][ceph-mon-2][DEBUG ]     ]
251
[2016-06-23 08:42:28,961][ceph-mon-2][DEBUG ]   }, 
252
[2016-06-23 08:42:28,961][ceph-mon-2][DEBUG ]   "name": "ceph-mon-2", 
253
[2016-06-23 08:42:28,961][ceph-mon-2][DEBUG ]   "outside_quorum": [], 
254
[2016-06-23 08:42:28,961][ceph-mon-2][DEBUG ]   "quorum": [], 
255
[2016-06-23 08:42:28,961][ceph-mon-2][DEBUG ]   "rank": 1, 
256
[2016-06-23 08:42:28,962][ceph-mon-2][DEBUG ]   "state": "electing", 
257
[2016-06-23 08:42:28,962][ceph-mon-2][DEBUG ]   "sync_provider": []
258
[2016-06-23 08:42:28,962][ceph-mon-2][DEBUG ] }
259
[2016-06-23 08:42:28,962][ceph-mon-2][DEBUG ] ********************************************************************************
260
[2016-06-23 08:42:28,962][ceph-mon-2][INFO  ] monitor: mon.ceph-mon-2 is running
261
[2016-06-23 08:42:28,963][ceph-mon-2][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-mon-2.asok mon_status
262
[2016-06-23 08:42:29,029][ceph_deploy.mon][DEBUG ] detecting platform for host ceph-mon-3 ...
263
[2016-06-23 08:42:29,226][ceph-mon-3][DEBUG ] connection detected need for sudo
264
[2016-06-23 08:42:29,429][ceph-mon-3][DEBUG ] connected to host: ceph-mon-3 
265
[2016-06-23 08:42:29,430][ceph-mon-3][DEBUG ] detect platform information from remote host
266
[2016-06-23 08:42:29,441][ceph-mon-3][DEBUG ] detect machine type
267
[2016-06-23 08:42:29,443][ceph-mon-3][DEBUG ] find the location of an executable
268
[2016-06-23 08:42:29,444][ceph-mon-3][INFO  ] Running command: sudo /sbin/initctl version
269
[2016-06-23 08:42:29,452][ceph-mon-3][DEBUG ] find the location of an executable
270
[2016-06-23 08:42:29,454][ceph_deploy.mon][INFO  ] distro info: Ubuntu 14.04 trusty
271
[2016-06-23 08:42:29,454][ceph-mon-3][DEBUG ] determining if provided host has same hostname in remote
272
[2016-06-23 08:42:29,454][ceph-mon-3][DEBUG ] get remote short hostname
273
[2016-06-23 08:42:29,455][ceph-mon-3][DEBUG ] deploying mon to ceph-mon-3
274
[2016-06-23 08:42:29,455][ceph-mon-3][DEBUG ] get remote short hostname
275
[2016-06-23 08:42:29,455][ceph-mon-3][DEBUG ] remote hostname: ceph-mon-3
276
[2016-06-23 08:42:29,457][ceph-mon-3][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
277
[2016-06-23 08:42:29,457][ceph-mon-3][DEBUG ] create the mon path if it does not exist
278
[2016-06-23 08:42:29,458][ceph-mon-3][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-ceph-mon-3/done
279
[2016-06-23 08:42:29,458][ceph-mon-3][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-ceph-mon-3/done
280
[2016-06-23 08:42:29,459][ceph-mon-3][INFO  ] creating keyring file: /var/lib/ceph/tmp/ceph-ceph-mon-3.mon.keyring
281
[2016-06-23 08:42:29,459][ceph-mon-3][DEBUG ] create the monitor keyring file
282
[2016-06-23 08:42:29,460][ceph-mon-3][INFO  ] Running command: sudo ceph-mon --cluster ceph --mkfs -i ceph-mon-3 --keyring /var/lib/ceph/tmp/ceph-ceph-mon-3.mon.keyring
283
[2016-06-23 08:42:29,482][ceph-mon-3][DEBUG ] ceph-mon: mon.noname-c 192.168.20.13:6789/0 is local, renaming to mon.ceph-mon-3
284
[2016-06-23 08:42:29,482][ceph-mon-3][DEBUG ] ceph-mon: set fsid to 1bf3874b-1267-453b-9210-bb51e4aa3075
285
[2016-06-23 08:42:29,546][ceph-mon-3][DEBUG ] ceph-mon: created monfs at /var/lib/ceph/mon/ceph-ceph-mon-3 for mon.ceph-mon-3
286
[2016-06-23 08:42:29,547][ceph-mon-3][INFO  ] unlinking keyring file /var/lib/ceph/tmp/ceph-ceph-mon-3.mon.keyring
287
[2016-06-23 08:42:29,547][ceph-mon-3][DEBUG ] create a done file to avoid re-doing the mon deployment
288
[2016-06-23 08:42:29,548][ceph-mon-3][DEBUG ] create the init path if it does not exist
289
[2016-06-23 08:42:29,550][ceph-mon-3][INFO  ] Running command: sudo initctl emit ceph-mon cluster=ceph id=ceph-mon-3
290
[2016-06-23 08:42:31,619][ceph-mon-3][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-mon-3.asok mon_status
291
[2016-06-23 08:42:31,684][ceph-mon-3][DEBUG ] ********************************************************************************
292
[2016-06-23 08:42:31,684][ceph-mon-3][DEBUG ] status for monitor: mon.ceph-mon-3
293
[2016-06-23 08:42:31,684][ceph-mon-3][DEBUG ] {
294
[2016-06-23 08:42:31,684][ceph-mon-3][DEBUG ]   "election_epoch": 1, 
295
[2016-06-23 08:42:31,685][ceph-mon-3][DEBUG ]   "extra_probe_peers": [
296
[2016-06-23 08:42:31,685][ceph-mon-3][DEBUG ]     "192.168.20.11:6789/0", 
297
[2016-06-23 08:42:31,685][ceph-mon-3][DEBUG ]     "192.168.20.12:6789/0"
298
[2016-06-23 08:42:31,685][ceph-mon-3][DEBUG ]   ], 
299
[2016-06-23 08:42:31,685][ceph-mon-3][DEBUG ]   "monmap": {
300
[2016-06-23 08:42:31,685][ceph-mon-3][DEBUG ]     "created": "0.000000", 
301
[2016-06-23 08:42:31,685][ceph-mon-3][DEBUG ]     "epoch": 0, 
302
[2016-06-23 08:42:31,685][ceph-mon-3][DEBUG ]     "fsid": "1bf3874b-1267-453b-9210-bb51e4aa3075", 
303
[2016-06-23 08:42:31,686][ceph-mon-3][DEBUG ]     "modified": "0.000000", 
304
[2016-06-23 08:42:31,686][ceph-mon-3][DEBUG ]     "mons": [
305
[2016-06-23 08:42:31,686][ceph-mon-3][DEBUG ]       {
306
[2016-06-23 08:42:31,686][ceph-mon-3][DEBUG ]         "addr": "192.168.20.11:6789/0", 
307
[2016-06-23 08:42:31,686][ceph-mon-3][DEBUG ]         "name": "ceph-mon-1", 
308
[2016-06-23 08:42:31,686][ceph-mon-3][DEBUG ]         "rank": 0
309
[2016-06-23 08:42:31,686][ceph-mon-3][DEBUG ]       }, 
310
[2016-06-23 08:42:31,687][ceph-mon-3][DEBUG ]       {
311
[2016-06-23 08:42:31,687][ceph-mon-3][DEBUG ]         "addr": "192.168.20.12:6789/0", 
312
[2016-06-23 08:42:31,687][ceph-mon-3][DEBUG ]         "name": "ceph-mon-2", 
313
[2016-06-23 08:42:31,687][ceph-mon-3][DEBUG ]         "rank": 1
314
[2016-06-23 08:42:31,687][ceph-mon-3][DEBUG ]       }, 
315
[2016-06-23 08:42:31,687][ceph-mon-3][DEBUG ]       {
316
[2016-06-23 08:42:31,687][ceph-mon-3][DEBUG ]         "addr": "192.168.20.13:6789/0", 
317
[2016-06-23 08:42:31,687][ceph-mon-3][DEBUG ]         "name": "ceph-mon-3", 
318
[2016-06-23 08:42:31,688][ceph-mon-3][DEBUG ]         "rank": 2
319
[2016-06-23 08:42:31,688][ceph-mon-3][DEBUG ]       }
320
[2016-06-23 08:42:31,688][ceph-mon-3][DEBUG ]     ]
321
[2016-06-23 08:42:31,688][ceph-mon-3][DEBUG ]   }, 
322
[2016-06-23 08:42:31,688][ceph-mon-3][DEBUG ]   "name": "ceph-mon-3", 
323
[2016-06-23 08:42:31,688][ceph-mon-3][DEBUG ]   "outside_quorum": [], 
324
[2016-06-23 08:42:31,688][ceph-mon-3][DEBUG ]   "quorum": [], 
325
[2016-06-23 08:42:31,688][ceph-mon-3][DEBUG ]   "rank": 2, 
326
[2016-06-23 08:42:31,689][ceph-mon-3][DEBUG ]   "state": "electing", 
327
[2016-06-23 08:42:31,689][ceph-mon-3][DEBUG ]   "sync_provider": []
328
[2016-06-23 08:42:31,689][ceph-mon-3][DEBUG ] }
329
[2016-06-23 08:42:31,689][ceph-mon-3][DEBUG ] ********************************************************************************
330
[2016-06-23 08:42:31,689][ceph-mon-3][INFO  ] monitor: mon.ceph-mon-3 is running
331
[2016-06-23 08:42:31,690][ceph-mon-3][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-mon-3.asok mon_status
332
[2016-06-23 08:42:31,806][ceph_deploy.mon][INFO  ] processing monitor mon.ceph-mon-1
333
[2016-06-23 08:42:31,820][ceph-mon-1][DEBUG ] connection detected need for sudo
334
[2016-06-23 08:42:31,833][ceph-mon-1][DEBUG ] connected to host: ceph-mon-1 
335
[2016-06-23 08:42:31,833][ceph-mon-1][DEBUG ] detect platform information from remote host
336
[2016-06-23 08:42:31,846][ceph-mon-1][DEBUG ] detect machine type
337
[2016-06-23 08:42:31,847][ceph-mon-1][DEBUG ] find the location of an executable
338
[2016-06-23 08:42:31,848][ceph-mon-1][INFO  ] Running command: sudo /sbin/initctl version
339
[2016-06-23 08:42:31,856][ceph-mon-1][DEBUG ] find the location of an executable
340
[2016-06-23 08:42:31,857][ceph-mon-1][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-mon-1.asok mon_status
341
[2016-06-23 08:42:31,922][ceph_deploy.mon][WARNING] mon.ceph-mon-1 monitor is not yet in quorum, tries left: 5
342
[2016-06-23 08:42:31,922][ceph_deploy.mon][WARNING] waiting 5 seconds before retrying
343
[2016-06-23 08:42:36,928][ceph-mon-1][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-mon-1.asok mon_status
344
[2016-06-23 08:42:36,994][ceph_deploy.mon][INFO  ] mon.ceph-mon-1 monitor has reached quorum!
345
[2016-06-23 08:42:36,994][ceph_deploy.mon][INFO  ] processing monitor mon.ceph-mon-2
346
[2016-06-23 08:42:37,197][ceph-mon-2][DEBUG ] connection detected need for sudo
347
[2016-06-23 08:42:37,398][ceph-mon-2][DEBUG ] connected to host: ceph-mon-2 
348
[2016-06-23 08:42:37,399][ceph-mon-2][DEBUG ] detect platform information from remote host
349
[2016-06-23 08:42:37,411][ceph-mon-2][DEBUG ] detect machine type
350
[2016-06-23 08:42:37,414][ceph-mon-2][DEBUG ] find the location of an executable
351
[2016-06-23 08:42:37,415][ceph-mon-2][INFO  ] Running command: sudo /sbin/initctl version
352
[2016-06-23 08:42:37,424][ceph-mon-2][DEBUG ] find the location of an executable
353
[2016-06-23 08:42:37,425][ceph-mon-2][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-mon-2.asok mon_status
354
[2016-06-23 08:42:37,490][ceph_deploy.mon][INFO  ] mon.ceph-mon-2 monitor has reached quorum!
355
[2016-06-23 08:42:37,490][ceph_deploy.mon][INFO  ] processing monitor mon.ceph-mon-3
356
[2016-06-23 08:42:37,698][ceph-mon-3][DEBUG ] connection detected need for sudo
357
[2016-06-23 08:42:37,905][ceph-mon-3][DEBUG ] connected to host: ceph-mon-3 
358
[2016-06-23 08:42:37,906][ceph-mon-3][DEBUG ] detect platform information from remote host
359
[2016-06-23 08:42:37,916][ceph-mon-3][DEBUG ] detect machine type
360
[2016-06-23 08:42:37,919][ceph-mon-3][DEBUG ] find the location of an executable
361
[2016-06-23 08:42:37,920][ceph-mon-3][INFO  ] Running command: sudo /sbin/initctl version
362
[2016-06-23 08:42:37,929][ceph-mon-3][DEBUG ] find the location of an executable
363
[2016-06-23 08:42:37,930][ceph-mon-3][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-mon-3.asok mon_status
364
[2016-06-23 08:42:37,995][ceph_deploy.mon][INFO  ] mon.ceph-mon-3 monitor has reached quorum!
365
[2016-06-23 08:42:37,995][ceph_deploy.mon][INFO  ] all initial monitors are running and have formed quorum
366
[2016-06-23 08:42:37,995][ceph_deploy.mon][INFO  ] Running gatherkeys...
367
[2016-06-23 08:42:37,996][ceph_deploy.gatherkeys][INFO  ] Storing keys in temp directory /tmp/tmpDoQGwl
368
[2016-06-23 08:42:38,010][ceph-mon-1][DEBUG ] connection detected need for sudo
369
[2016-06-23 08:42:38,023][ceph-mon-1][DEBUG ] connected to host: ceph-mon-1 
370
[2016-06-23 08:42:38,024][ceph-mon-1][DEBUG ] detect platform information from remote host
371
[2016-06-23 08:42:38,037][ceph-mon-1][DEBUG ] detect machine type
372
[2016-06-23 08:42:38,038][ceph-mon-1][DEBUG ] find the location of an executable
373
[2016-06-23 08:42:38,039][ceph-mon-1][INFO  ] Running command: sudo /sbin/initctl version
374
[2016-06-23 08:42:38,047][ceph-mon-1][DEBUG ] get remote short hostname
375
[2016-06-23 08:42:38,048][ceph-mon-1][DEBUG ] fetch remote file
376
[2016-06-23 08:42:38,049][ceph-mon-1][INFO  ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --admin-daemon=/var/run/ceph/ceph-mon.ceph-mon-1.asok mon_status
377
[2016-06-23 08:42:38,115][ceph-mon-1][INFO  ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-mon-1/keyring auth get-or-create client.admin osd allow * mds allow * mon allow *
378
[2016-06-23 08:42:38,229][ceph-mon-1][ERROR ] "ceph auth get-or-create for keytype admin returned 22
379
[2016-06-23 08:42:38,230][ceph-mon-1][DEBUG ] Error EINVAL: key for client.admin exists but cap mds does not match
380
[2016-06-23 08:42:38,434][ceph-mon-2][DEBUG ] connection detected need for sudo
381
[2016-06-23 08:42:38,631][ceph-mon-2][DEBUG ] connected to host: ceph-mon-2 
382
[2016-06-23 08:42:38,632][ceph-mon-2][DEBUG ] detect platform information from remote host
383
[2016-06-23 08:42:38,642][ceph-mon-2][DEBUG ] detect machine type
384
[2016-06-23 08:42:38,644][ceph-mon-2][DEBUG ] find the location of an executable
385
[2016-06-23 08:42:38,646][ceph-mon-2][INFO  ] Running command: sudo /sbin/initctl version
386
[2016-06-23 08:42:38,654][ceph-mon-2][DEBUG ] get remote short hostname
387
[2016-06-23 08:42:38,655][ceph-mon-2][DEBUG ] fetch remote file
388
[2016-06-23 08:42:38,657][ceph-mon-2][INFO  ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --admin-daemon=/var/run/ceph/ceph-mon.ceph-mon-2.asok mon_status
389
[2016-06-23 08:42:38,723][ceph-mon-2][INFO  ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-mon-2/keyring auth get-or-create client.admin osd allow * mds allow * mon allow *
390
[2016-06-23 08:42:38,837][ceph-mon-2][ERROR ] "ceph auth get-or-create for keytype admin returned 22
391
[2016-06-23 08:42:38,838][ceph-mon-2][DEBUG ] Error EINVAL: key for client.admin exists but cap mds does not match
392
[2016-06-23 08:42:39,036][ceph-mon-3][DEBUG ] connection detected need for sudo
393
[2016-06-23 08:42:39,234][ceph-mon-3][DEBUG ] connected to host: ceph-mon-3 
394
[2016-06-23 08:42:39,234][ceph-mon-3][DEBUG ] detect platform information from remote host
395
[2016-06-23 08:42:39,245][ceph-mon-3][DEBUG ] detect machine type
396
[2016-06-23 08:42:39,247][ceph-mon-3][DEBUG ] find the location of an executable
397
[2016-06-23 08:42:39,249][ceph-mon-3][INFO  ] Running command: sudo /sbin/initctl version
398
[2016-06-23 08:42:39,257][ceph-mon-3][DEBUG ] get remote short hostname
399
[2016-06-23 08:42:39,258][ceph-mon-3][DEBUG ] fetch remote file
400
[2016-06-23 08:42:39,260][ceph-mon-3][INFO  ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --admin-daemon=/var/run/ceph/ceph-mon.ceph-mon-3.asok mon_status
401
[2016-06-23 08:42:39,325][ceph-mon-3][INFO  ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-mon-3/keyring auth get-or-create client.admin osd allow * mds allow * mon allow *
402
[2016-06-23 08:42:39,490][ceph-mon-3][ERROR ] "ceph auth get-or-create for keytype admin returned 22
403
[2016-06-23 08:42:39,490][ceph-mon-3][DEBUG ] Error EINVAL: key for client.admin exists but cap mds does not match
404
[2016-06-23 08:42:39,492][ceph_deploy.gatherkeys][ERROR ] Failed to connect to host:ceph-mon-1, ceph-mon-2, ceph-mon-3
405
[2016-06-23 08:42:39,492][ceph_deploy.gatherkeys][INFO  ] Destroy temp directory /tmp/tmpDoQGwl
406
[2016-06-23 08:42:39,492][ceph_deploy][ERROR ] RuntimeError: Failed to connect any mon
407