Bug #6281
closedceph-deploy config.py write_conf throws away old config just because they are out of sync
0%
Description
in example
ceph-deploy new node-1:10.20.0.2
echo [global] >> /etc/ceph/ceph.conf
echo public address = 10.20.0.0/24 >> /etc/ceph/ceph.conf
echo cluster address = 10.20.1.0/24 >> /etc/ceph/ceph.conf
...
ceph-deploy mon create node-1:10.20.0.1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts node-1:10.20.0.2
[ceph_deploy.mon][DEBUG ] detecting platform for host node-1 ...
[ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection without sudo
[ceph_deploy.mon][INFO ] distro info: CentOS 6.4 Final
[node-1][DEBUG ] determining if provided host has same hostname in remote
[node-1][DEBUG ] deploying mon to node-1
[node-1][DEBUG ] remote hostname: node-1
[node-1][INFO ] write cluster configuration to /etc/ceph/{cluster}.conf
[node-1][ERROR ] Traceback (most recent call last):
[node-1][ERROR ] File "/usr/lib/python2.6/site-packages/ceph_deploy/util/decorators.py", line 10, in inner
[node-1][ERROR ] def inner(*args, **kwargs):
[node-1][ERROR ] File "/usr/lib/python2.6/site-packages/ceph_deploy/conf.py", line 12, in write_conf
[node-1][ERROR ] line = self.fp.readline()
[node-1][ERROR ] RuntimeError: config file /etc/ceph/ceph.conf exists with different content; use --overwrite-conf to overwrite
[node-1][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-node-6/done
[node-1][INFO ] create a done file to avoid re-doing the mon deployment
[node-1][INFO ] create the init path if it does not exist
[node-1][INFO ] locating `service` executable...
[node-1][INFO ] found `service` executable: /sbin/service
Warning: Permanently added 'node-1,10.0.0.130' (RSA) to the list of known hosts.
[node-1][INFO ] Running command: sudo /sbin/service ceph start mon.node-1
[node-1][DEBUG ] === mon.node-1 ===
[node-1][DEBUG ] Starting Ceph mon.node-1 on node-1...
[node-1][DEBUG ] Starting ceph-create-keys on node-1..
Warning: Permanently added 'node-1,10.0.0.130' (RSA) to the list of known hosts.
[node-1][INFO ] Running command: sudo ceph daemon mon.node-1 mon_status
[node-1][ERROR ] unrecognized command
[node-1][WARNIN] monitor: mon.node-1, might not be running yet
[root@node-1 /]# ceph -s
health HEALTH_ERR 392 pgs stuck inactive; 392 pgs stuck unclean; no osds
monmap e1: 1 mons at {node-1=10.20.0.2:6789/0}, election epoch 1, quorum 0 node-1
osdmap e3: 0 osds: 0 up, 0 in
pgmap v4: 392 pgs: 392 creating; 0 bytes data, 0 KB used, 0 KB / 0 KB avail
mdsmap e1: 0/0/1 up
using --overwrite-conf the error is gone
ceph-deploy --overwrite-conf mon create node-1:10.20.0.2
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts node-1:10.20.0.2
[ceph_deploy.mon][DEBUG ] detecting platform for host node-1 ...
[ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection without sudo
[ceph_deploy.mon][INFO ] distro info: CentOS 6.4 Final
[node-1][DEBUG ] determining if provided host has same hostname in remote
[node-1][DEBUG ] deploying mon to node-1
[node-1][DEBUG ] remote hostname: node-1
[node-1][INFO ] write cluster configuration to /etc/ceph/{cluster}.conf
[node-1][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-node-1/done
[node-1][INFO ] create a done file to avoid re-doing the mon deployment
[node-1][INFO ] create the init path if it does not exist
[node-1][INFO ] locating `service` executable...
[node-1][INFO ] found `service` executable: /sbin/service
Warning: Permanently added 'node-1,10.0.0.130' (RSA) to the list of known hosts.
[node-1][INFO ] Running command: sudo /sbin/service ceph start mon.node-1
[node-1][DEBUG ] === mon.node-1 ===
[node-1][DEBUG ] Starting Ceph mon.node-1 on node-1...
[node-1][DEBUG ] Starting ceph-create-keys on node-1...
Warning: Permanently added 'node-1,10.0.0.130' (RSA) to the list of known hosts.
[node-1][INFO ] Running command: sudo ceph daemon mon.node-1 mon_status
[node-1][ERROR ] unrecognized command
[node-1][WARNIN] monitor: mon.node-1, might not be running yet
[root@node-1 /]# ceph -s
health HEALTH_ERR 392 pgs stuck inactive; 392 pgs stuck unclean; no osds
monmap e1: 1 mons at {node-1=10.20.0.2:6789/0}, election epoch 1, quorum 0 node-1
osdmap e3: 0 osds: 0 up, 0 in
pgmap v4: 392 pgs: 392 creating; 0 bytes data, 0 KB used, 0 KB / 0 KB avail
mdsmap e1: 0/0/1 up
using --overwrite-conf throws out all changes in /etc/ceph/ceph.conf using ONLY the values in ceph.conf from the working directory. This is different from ceph-deploy-1.0.0 where it managed to merge the two files picking changes from cwd/ceph.conf
expected behavior, keys that are present in both ceph.conf files should be evaluated, if keys from both sources have any changes --overwrite-conf should be needed to update values from cwd/{cluster}.conf. un-matched keys in /etc/ceph/{cluster}.conf should remain. un-matched keys in cwd/{cluster}.conf should be added to /etc/ceph/ceph.conf
Updated by Alexey Sheplyakov almost 8 years ago
using --overwrite-conf throws out all changes in /etc/ceph/ceph.conf using ONLY the values
in ceph.conf from the working directory.
As far as I understand this is the intended behavior. The master copy of ceph.conf is stored
on the "admin" node (in the home directory of the user who invokes ceph-deploy), and ceph-deploy
distributes that file to the cluster nodes. The user is not supposed to directly modify the config
file on OSDs/monitors, instead one should modify the master copy and push it to the nodes.
As a side note "public address" is a per daemon setting and it shouldn't be set in the global section.
Thus you should have modified the ceph.conf which resides in the current directory instead
of that in /etc/ceph/ceph.conf:
ceph-deploy new node-1:10.20.0.2
cat >> ceph.conf <<-EOF
[global]
public network = 10.20.0.0/24
cluster network = 10.20.1.0/24
EOF
ceph-deploy mon create node-1:10.20.0.2
(pushing is not necessary here since the target node has no /etc/ceph/ceph.conf yet)
Updated by Alfredo Deza about 7 years ago
- Status changed from New to Closed
Closing as this is intended behavior