Project

General

Profile

Tasks #47369

Ceph scales to 100's of hosts, 1000's of OSDs....can orchestrator?

Added by Paul Cuzner 5 months ago. Updated 2 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
orchestrator
Target version:
% Done:

0%

Tags:
Reviewed:
Affected Versions:
Pull request ID:

Description

This bug is intended to help us identify areas in mgr/orchestrator, mgr/cephadm, cephadm, mgr/prometheus and mgr/rook which represent potential scale bottlenecks.

For example
  • list_daemons
  • ceph_volume execution time
  • orchestrator task parallelization
  • lack of dashboard pagination
  • dashboard prefetch strategy for 100's or 1000s of OSDs, RBD images, buckets
  • mgr/prometheus reporting on 1000's OSDs or 100's hosts
  • installation bandwidth (the demand image pull places on the network)

There have been trackers in the past that focus on a specific areas (https://tracker.ceph.com/issues/36451), but it would be great if we could look at scale issues holistically..

Please feel free to add information to this tracker which documents scale issues with the management layer, that need to be considered.


Related issues

Related to Orchestrator - Feature #47368: Provide a daemon mode for cephadm to handle host/daemon state requests Resolved
Related to mgr - Tasks #36451: mgr/dashboard: Scalability testing New 07/12/2019

History

#1 Updated by Patrick Seidensal 5 months ago

A single Prometheus instance can, on a properly sized host, handle 1000 nodes. As an OSD is usually accompanied by other OSDs on a host, this is not an issue. Customers with 1000 OSD clusters haven't had any issues with Prometheus, though the Prometheus manager module made some difficulties. But in the meantime the cache of the Prometheus manager module has been overhauled as well as some patches have been contributed on the Ceph's side to improve performance. Since then there haven't been any issues that I'm aware of.

#2 Updated by Sebastian Wagner 5 months ago

  • Related to Feature #47368: Provide a daemon mode for cephadm to handle host/daemon state requests added

#3 Updated by Sebastian Wagner 2 months ago

  • Related to Tasks #36451: mgr/dashboard: Scalability testing added

#4 Updated by Sebastian Wagner 2 months ago

  • Tracker changed from Bug to Tasks

Also available in: Atom PDF