Bug #47381: "ceph orch apply --dry-run" reports empty osdspec even though OSDs will be deployed - Orchestrator - Ceph

Actions

Copy link

Bug #47381

closed

"ceph orch apply --dry-run" reports empty osdspec even though OSDs will be deployed

Added by Nathan Cutler over 3 years ago. Updated about 3 years ago.

Status:

Can't reproduce

Priority:

Normal

Assignee:

Category:

cephadm/osd

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

If I precede "ceph orch apply --dry-run" with "ceph orch device ls --refresh", everything is fine.

But AFAICT we are not requiring users to issue a "ceph orch device ls --refresh" before running "ceph orch apply --dry-run".

Without "ceph orch device ls --refresh", I get the following output even though my spec file WILL deploy OSDs when applied:

    master: ++ ceph orch apply -i /root/service_spec_core.yml --dry-run
    master: WARNING! Dry-Runs are snapshots of a certain point in time and are bound
    master: to the current inventory setup. If any on these conditions changes, the
    master: preview will be invalid. Please make sure to have a minimal
    master: timeframe between planning and applying the specs.
    master: ####################
    master: SERVICESPEC PREVIEWS
    master: ####################
    master: +---------+------+-------------+-------------+
    master: |SERVICE  |NAME  |ADD_TO       |REMOVE_FROM  |
    master: +---------+------+-------------+-------------+
    master: |mon      |mon   |node2 node3  |             |
    master: |mgr      |mgr   |node2 node3  |             |
    master: +---------+------+-------------+-------------+
    master: ################
    master: OSDSPEC PREVIEWS
    master: ################
    master: +---------+------+------+------+----+-----+
    master: |SERVICE  |NAME  |HOST  |DATA  |DB  |WAL  |
    master: +---------+------+------+------+----+-----+
    master: +---------+------+------+------+----+-----+

Actions

Copy link

Updated by Joshua Schmid over 3 years ago

This is most likely a known timing issue of the `--dry-run` command.

You may verify this by running the provided command a couple of times in a row.

Actions

Copy link

Updated by Nathan Cutler over 3 years ago

Joshua Schmid wrote:

This is most likely a known timing issue of the `--dry-run` command.

Could you provide a link to where this known timing issue is tracked?

Actions

Copy link

Updated by Joshua Schmid over 3 years ago

But AFAICT we are not requiring users to issue a "ceph orch device ls --refresh" before running "ceph orch apply --dry-run".

When the mgr starts, cephadm goes out and starts gathering information about the hosts (including their disks). It's just not yet ready.
A `device ls --refresh` just re-triggers this action.

Without "ceph orch device ls --refresh", I get the following output even though my spec file WILL deploy OSDs when applied:

The osds will get deployed because the devices are potentially there. They just haven't been fully detected by cephadm. `apply` stores the specs internally and re-applies it frequently.

Imo this can be summarized to:

We need a way to track if a `refresh` is ongoing.

Apply (with or without the --dry-run mode) needs to detect that and defer/delay the execution until the refresh is done.
If --dry-run is used we bail out and ask the user to re-run the command when the refresh is done.

Actions

Copy link

Updated by Nathan Cutler over 3 years ago

Subject changed from Without preceding "ceph orch device ls --refresh", "ceph orch apply --dry-run" reports empty osdspec even though OSDs will be deployed to "ceph orch apply --dry-run" reports empty osdspec even though OSDs will be deployed

Actions

Copy link