Ceph » mgr » Dashboard

1. What version are you running?
   I built master a couple of days ago and this did not happen.
2. Did you try this on Pacific?
3. Did you try this on the gibba cluster Dashboard? You don't need to build anything, just access the Dashboard, and follow step 2.2 and on from the list in the description above.
4. How did you enable telemetry when you tried to reproduce this?

Nizamudeen reported experiencing this failure on the ceph-dev environment: https://github.com/rhcs-dashboard/ceph-dev

[root@ceph ceph]# ceph telemetry on --license sharing-1-0
2022-02-04T07:02:44.216+0000 7f587dea4700 -1 WARNING: all dangerous and experimental features are enabled.
2022-02-04T07:02:44.222+0000 7f587dea4700 -1 WARNING: all dangerous and experimental features are enabled.
Telemetry is on.
Some channels are disabled, please enable with:
`ceph telemetry enable channel perf`
[root@ceph ceph]# ceph telemetry status
2022-02-04T07:02:51.761+0000 7f5768191700 -1 WARNING: all dangerous and experimental features are enabled.
2022-02-04T07:02:51.764+0000 7f5768191700 -1 WARNING: all dangerous and experimental features are enabled.
Error EIO: Module 'telemetry' has experienced an error and cannot handle commands:

2022-02-04T07:02:45.697+0000 7f8ead583700  0 [telemetry INFO root] Sent report to https://telemetry.ceph.com/report
2022-02-04T07:02:45.697+0000 7f8ead583700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'telemetry' while running on mgr.x: 
2022-02-04T07:02:45.698+0000 7f8ead583700 -1 telemetry.serve:
2022-02-04T07:02:45.698+0000 7f8ead583700 -1 Traceback (most recent call last):
  File "/ceph/src/pybind/mgr/telemetry/module.py", line 1803, in serve
    self.send(self.last_report)
  File "/ceph/src/pybind/mgr/telemetry/module.py", line 1271, in send
    assert devices
AssertionError

Laura, Yaarit, ceph-dev basically runs vstart + master RPMs (usually the ones built last night). The issue happens when you start the cluster with vstart, log in to Dashboard and click "Accept" on the Telemetry notification to set it up from scratch.

I'll go on looking the issue you reported.

Hi,

I tried with ceph-dev and reproduced the behaviour described. In any case, if a user completes the wizard, telemetry will be eventually enabled. It'll only stay disabled if the user leaves the wizard before completion. IMHO, while not ideal, that makes the priority less urgent. If I'm not wrong, this is not a regression: this behaviour has been always there.

I assume that the issue was that the telemetry UI was mainly designed to gain adoption among new users (I cannot confirm because the contributor that worked on this is no longer with us), not with the upgrade path in mind.

1. Wizard starts
2. DASHBOARD fetches:
   • telemetry status (on or off),
   • enabled channels,
   • contact info,
3. DASHBOARD displays the first Wizard page with the above info updated accordingly.
4. The USER can modify:
   • telemetry status,
   • channels,
   • contact info
5. USER clicks "next" (but these changes are NOT applied yet)
6. USER is presented with the Review page:
   • the report will be displayed with the selected fields.
7. If USER clicks "Update" the changes are applied (enable disable telemetry, enable/disable changes, update contact info).
   • Otherwise, not changes are applied.

It seems we're not using the telemetry CLI API, but calling directly to the internal Telemetry methods via mgr.remote() interface.

The changes to the UI and REST shouldn't be too complex. Question: can you take this or you expect us to deal with it? I'm asking because we don't have plenty of hands lately...

Hi Ernesto,

Both Laura and I could not reproduce this issue.
I think we should avoid using 'assert devices', since if there is something wrong nothing will be reported (since the report will not be generated) and we will never know that it happened. We will have a fix for that.

Thanks for reporting this.

Thanks, Ernesto, Sarthak!
I have a few comments:

Ernesto Puerta wrote:

Hi,

I tried with ceph-dev and reproduced the behaviour described. In any case, if a user completes the wizard, telemetry will be eventually enabled. It'll only stay disabled if the user leaves the wizard before completion. IMHO, while not ideal, that makes the priority less urgent. If I'm not wrong, this is not a regression: this behaviour has been always there.

The thing is that we cannot rely on the user re-completing the wizard; users should be allowed just to look around, without applying any changes. It's a confusing and unintuitive behavior of the wizard. The "Next" button should not apply any changes to the configuration, just the "Update" button.
Thanks for the fix!

I assume that the issue was that the telemetry UI was mainly designed to gain adoption among new users (I cannot confirm because the contributor that worked on this is no longer with us), not with the upgrade path in mind.

Either case, the workflow could be like this:

1. Wizard starts
2. DASHBOARD fetches:
   • telemetry status (on or off),
   • enabled channels,
   • contact info,

There are additional settings (proxy, interval) that the wizard currently displays, they need to be included here too;
need to also fetch leaderboard, organization (which is part of the Contact Info section).

We should also display the output of `ceph telemetry collection ls` (introduced in Quincy), which will give the user more information about the available collections, whether they opt-in for the first time, or if they are upgrading. See https://docs.ceph.com/en/latest/mgr/telemetry/#collections for more details.

1. DASHBOARD displays the first Wizard page with the above info updated accordingly.
2. The USER can modify:
   • telemetry status,
   • channels,
   • contact info

- Contact info section (Contact, Description, Organization) should be visible only if the Ident channel is enabled. See related ticket (https://tracker.ceph.com/issues/54133).
- Leaderboard should also be displayed; the other Advanced Settings which are currently displayed should still be there.

1. USER clicks "next" (but these changes are NOT applied yet)
2. USER is presented with the Review page:
   • the report will be displayed with the selected fields.

In the CLI we use `ceph telemetry show` and `ceph telemetry preview` to differentiate between opted-in and opted-out states.
See https://docs.ceph.com/en/latest/mgr/telemetry/#sample-report for more details. Please let us know if you have any questions.

1. If USER clicks "Update" the changes are applied (enable disable telemetry, enable/disable changes, update contact info).

- Other settings should also be updated (proxy, interval, leaderboard)

• Otherwise, not changes are applied.

It seems we're not using the telemetry CLI API, but calling directly to the internal Telemetry methods via mgr.remote() interface.

The changes to the UI and REST shouldn't be too complex. Question: can you take this or you expect us to deal with it? I'm asking because we don't have plenty of hands lately...

We are also currently very busy on our end... we will be happy to assist whenever we can. I feel like we need to have a solid definition of the ideal GUI design first.

Is it possible to see the planned GUI blueprint prior to its implementation to see that we're on the same page? What tools do you use for that?

Thanks!