Ceph scales to 100's of hosts, 1000's of OSDs....can orchestrator?
This bug is intended to help us identify areas in mgr/orchestrator, mgr/cephadm, cephadm, mgr/prometheus and mgr/rook which represent potential scale bottlenecks.For example
- ceph_volume execution time
- orchestrator task parallelization
- lack of dashboard pagination
- dashboard prefetch strategy for 100's or 1000s of OSDs, RBD images, buckets
- mgr/prometheus reporting on 1000's OSDs or 100's hosts
- installation bandwidth (the demand image pull places on the network)
There have been trackers in the past that focus on a specific areas (https://tracker.ceph.com/issues/36451), but it would be great if we could look at scale issues holistically..
Please feel free to add information to this tracker which documents scale issues with the management layer, that need to be considered.
#1 Updated by Patrick Seidensal 5 months ago
A single Prometheus instance can, on a properly sized host, handle 1000 nodes. As an OSD is usually accompanied by other OSDs on a host, this is not an issue. Customers with 1000 OSD clusters haven't had any issues with Prometheus, though the Prometheus manager module made some difficulties. But in the meantime the cache of the Prometheus manager module has been overhauled as well as some patches have been contributed on the Ceph's side to improve performance. Since then there haven't been any issues that I'm aware of.