Feature #10904
closedlist all nodes in a ceph cluster
Added by Alfredo Deza about 9 years ago. Updated almost 9 years ago.
60%
Description
In a few parts of the documentation it is requested to make changes to "all nodes in the cluster".
By the glossary definition (http://ceph.com/docs/master/glossary/#term-ceph-node) it is 'any single machine or server in a ceph system'.
But there is no way to list those. After asking around I was able to gather a few commands to get nodes that are part of a ceph service (mon, osd, or mds).
Even if this is not enough (maybe I am missing something else I could run?) it would be helpful to have something that does this for a user:
For OSDs
$ sudo ceph osd tree | grep host | awk '{print $4}'
For MONs
$ sudo ceph mon dump | grep ^[0-9]
And for MDSs
$ sudo ceph mds dump | grep ^[0-9]
Updated by Kefu Chai about 9 years ago
Alfredo Deza wrote:
In a few parts of the documentation it is requested to make changes to "all nodes in the cluster".
could you be more specific ?
- what a user might want in this case? for example, the node name? in the case of OSD nodes, it could be host, or the osd.{osd-id}? the format of the output?
- when he/she need the list? how he/she will use this list?
they are basically the same question: a user story. so i can figure out what the feature really wants.
thanks.
Updated by Alfredo Deza about 9 years ago
There are a few places in the docs, I was specifically following the federated config docs for RGW.
For example in this section (as well as in a couple of others) http://ceph.com/docs/master/radosgw/federated-config/#restart-services
it would mention things like:
use the following on each Ceph Node:
Updated by Kefu Chai about 9 years ago
so we'd like to have a command which is able to list all nodes in the term of "http://ceph.com/docs/master/glossary/#term-ceph-node"
Any single machine or server in a Ceph System.
so it could be
$ ceph node list # list all nodes osd host1 host2 host3 mon host4 host5 host6 mds host10 host11 host12 $ ceph node list --osd # list all OSD nodes osd host1 host2 host3
Updated by Kefu Chai about 9 years ago
hi Alfredo, the sample usage and output looks like following, if no objection, i will post the PR for review.
$ ./ceph node ls mon: 127.0.0.1:6789 127.0.0.1:6790 127.0.0.1:6791 osd: 127.0.0.1:6800 127.0.0.1:6804 127.0.0.1:6815 mds: 127.0.0.1:6819 127.0.0.1:6821 127.0.0.1:6820 [kchai@rex003 src]$ ./ceph node ls all mon: 127.0.0.1:6789 127.0.0.1:6790 127.0.0.1:6791 osd: 127.0.0.1:6800 127.0.0.1:6804 127.0.0.1:6815 mds: 127.0.0.1:6819 127.0.0.1:6821 127.0.0.1:6820 [kchai@rex003 src]$ ./ceph node ls osd osd: 127.0.0.1:6800 127.0.0.1:6804 127.0.0.1:6815
Updated by Kefu Chai about 9 years ago
loïc wrote
ceph-deploy enforces constraints that do not match the constraints of ceph, which may make things easier or more difficult, depending. For instance, the mon name must be a hostname for ceph-deploy (although it's possible to write ceph-deploy create mon hostname:moname, but it's not consistently supported accross the command lines and won't do what you expect). So, if you're to list the nodes for mons (and not for osds), this constraints makes it trivial for ceph-deploy to figure out the hostname: it's the same as the mon name. For osds, ceph does not care at all about the host name and there may be situations where a perfectly valid osd IP/port will not mean anything to ceph-deploy.
so different commands are expecting different representation of node address.
alfredo deza wrote:
default to hostnames unless --ipaddr is passed in? If some node doesn't have a hostname (or is not known) then you can fallback
Updated by Kefu Chai about 9 years ago
talked with joao, seems we don't store the hostname anywhere in the monmap, osdmap or the mdsmap. but we can get_hostname
on each of the mon.
joao wrote:
one alternative though. extend 'mon_status' to supply the mon's hostname. have each mon outputting the mon's hostname somewhere in mon_status. it will still require a check on an individual monitor basis, but it's better than not knowing it I suppose.
i am not sure this is the way we should go.
Updated by Kefu Chai about 9 years ago
for the osd,
$ ./ceph osd tree *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -1 3.00000 root default -2 3.00000 host gen8 0 1.00000 osd.0 up 1.00000 1.00000 1 1.00000 osd.1 up 1.00000 1.00000 2 1.00000 osd.2 up 1.00000 1.00000
we should print the two level parent of device:
default gen8 osd.0 default gen8 osd.1 default gen8 osd.2
for the mds:
$ ./ceph mds dump dumped mdsmap epoch 10 epoch 10 flags 0 created 2015-03-18 17:35:30.947077 modified 2015-03-18 17:35:39.878665 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 last_failure 0 last_failure_osd_epoch 0 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4 =dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table} max_mds 3 in 0,1,2 up {0=4122,1=4120,2=4112} failed stopped data_pools 1 metadata_pool 2 inline_data disabled 4122: 127.0.0.1:6812/59748 'a' mds.0.1 up:active seq 3 4120: 127.0.0.1:6814/60103 'c' mds.1.1 up:active seq 2 4112: 127.0.0.1:6813/59932 'b' mds.2.1 up:active seq 3
we should print
a c b
for the mon,
dumped monmap epoch 1 epoch 1 fsid d75a486d-44de-47b5-885b-711ce03914a6 last_changed 2015-03-18 17:35:00.500817 created 2015-03-18 17:35:00.500817 0: 127.0.0.1:6789/0 mon.a 1: 127.0.0.1:6790/0 mon.b 2: 127.0.0.1:6791/0 mon.c
we should print
mon.a mon.b mon.c
Updated by Kefu Chai about 9 years ago
following is the sample output of the newly added "node ls" command:
$ ./ceph node ls mon a mon b mon c osd rex003 osd.0 osd rex003 osd.1 osd rex003 osd.2 mds a mds c mds b [kchai@rex003 src]$ ./ceph node ls mon mon a mon b mon c [kchai@rex003 src]$ ./ceph node ls osd osd rex003 osd.0 osd rex003 osd.1 osd rex003 osd.2 [kchai@rex003 src]$ ./ceph node ls mds mds a mds c mds b
Updated by Kefu Chai about 9 years ago
- Status changed from New to Fix Under Review
- % Done changed from 0 to 60
Updated by Kefu Chai about 9 years ago
- Status changed from Fix Under Review to In Progress
- % Done changed from 60 to 10
Updated by Kefu Chai about 9 years ago
will implement "mds metadata" and "mon medata" following the model of "osd metadata" first. and add "node ls" later on. just sent mail to the mailing list for comments.
Updated by Kefu Chai about 9 years ago
- Status changed from In Progress to Fix Under Review
- % Done changed from 10 to 60
Updated by Kefu Chai almost 9 years ago
- Status changed from Fix Under Review to Resolved