Project

General

Profile

Bug #41740

mgr/dashboard: After upgrade, dashboard still shows old version sometimes

Added by Charley Sheets 11 months ago. Updated 7 months ago.

Status:
Need More Info
Priority:
Normal
Assignee:
-
Category:
dashboard/general
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

I recently upgraded my cluster from 14.2.2 to 14.2.3, starting with my three monitor hosts. After the third monitor host had been upgraded, I noticed the /#/hosts page still showed it as running 14.2.2, as shown in the attached screenshot. Each mon host was upgraded using "apt update ; apt upgrade" and rebooted immediately after completion of the upgrade command.

root@ceph-mon0:~# ceph versions
{
    "mon": {
        "ceph version 14.2.3 (4e36ac4558b09ab0ace33a4d000e83fc802f91f5) nautilus (stable)": 3
    },
    "mgr": {
        "ceph version 14.2.3 (4e36ac4558b09ab0ace33a4d000e83fc802f91f5) nautilus (stable)": 2
    },
    "osd": {
        "ceph version 14.2.3 (4e36ac4558b09ab0ace33a4d000e83fc802f91f5) nautilus (stable)": 16
    },
    "mds": {
        "ceph version 14.2.3 (4e36ac4558b09ab0ace33a4d000e83fc802f91f5) nautilus (stable)": 3
    },
    "overall": {
        "ceph version 14.2.3 (4e36ac4558b09ab0ace33a4d000e83fc802f91f5) nautilus (stable)": 24
    }
}

note the version shown for ceph-mon2

Screenshot_2019-09-10 Ceph.png View - note the version shown for ceph-mon2 (27.5 KB) Charley Sheets, 09/10/2019 07:39 AM

Screenshot_2020-01-18 Ceph.png View (29.4 KB) Charley Sheets, 01/18/2020 11:55 PM


Related issues

Related to mgr - Bug #41790: mgr/dashboard: Only one version number per host displayed New 09/12/2019
Related to mgr - Bug #41878: mgr/prometheus: KeyError in mgr/prometheus/module.py", line 490, in get_mgr_status Resolved 09/17/2019

History

#1 Updated by Patrick Seidensal 11 months ago

  • Description updated (diff)

#2 Updated by Lenz Grimmer 11 months ago

  • Subject changed from After upgrade, dashboard still shows old version sometimes to mgr/dashboard: After upgrade, dashboard still shows old version sometimes
  • Description updated (diff)

#3 Updated by Lenz Grimmer 11 months ago

That "Version" information on the page is not very helpful to begin with. If multiple services are running on a single host, their versions might actually be different from each other - this needs to be improved.

Would it be possible for you to send us the output of the following two API GET requests?

https://<dashboard host IP>/api/host
https://<dashboard host IP>/api/summary

You can perform these API requests by clicking on the help icon (question mark) in the top right corner and selecting "API" from the dropdown.

#4 Updated by Lenz Grimmer 11 months ago

  • Related to Bug #41790: mgr/dashboard: Only one version number per host displayed added

#5 Updated by Lenz Grimmer 11 months ago

FWIW, I now submitted #41790 to request a change in how the version information is displayed.

#6 Updated by Sebastian Wagner 11 months ago

  • Related to Bug #41869: mgr/dashboard: api/mgr/module resource is returning an internal server error added

#7 Updated by Sebastian Wagner 11 months ago

  • Related to deleted (Bug #41869: mgr/dashboard: api/mgr/module resource is returning an internal server error)

#8 Updated by Sebastian Wagner 11 months ago

  • Related to Bug #41878: mgr/prometheus: KeyError in mgr/prometheus/module.py", line 490, in get_mgr_status added

#9 Updated by Sebastian Wagner 11 months ago

mgr_module.MgrModule#list_servers might return outdated information. This would be a good explanation for a similar issue #41878

#10 Updated by Charley Sheets 11 months ago

I haven't been able to provide the requested information because the problem went away after a few mgr reboots. However, now that I'll be upgrading to 14.2.4, I will see if I can reproduce the problem and then provide the requested information.

#11 Updated by Lenz Grimmer 10 months ago

  • Status changed from New to Need More Info

Charley Sheets wrote:

I haven't been able to provide the requested information because the problem went away after a few mgr reboots. However, now that I'll be upgrading to 14.2.4, I will see if I can reproduce the problem and then provide the requested information.

Thanks! Setting this to "Need more info" for the time being.

#12 Updated by Charley Sheets 7 months ago

After upgrading one of my three mon hosts from 14.2.5-1-g23e76c7aa6-1 to 14.2.6-1-gffd69200ad-1, it is still reporting its version as 14.2.5 on both the web and via the API. I've attached a screenshot. I'm using the Debian packages published by croit.io.

Here are the API call results that were previously requested:

Request URL: https://ceph.picote.ch/api/host

Response:

[
  {
    "services": [
      {
        "type": "mds",
        "id": "ceph-mon0" 
      },
      {
        "type": "mgr",
        "id": "ceph-mon0" 
      },
      {
        "type": "mon",
        "id": "ceph-mon0" 
      }
    ],
    "hostname": "ceph-mon0",
    "ceph_version": "ceph version 14.2.5-1-g23e76c7aa6 (23e76c7aa6e15817ffb6741aafbc95ca99f24cbb) nautilus (stable)" 
  },
  {
    "services": [
      {
        "type": "mds",
        "id": "ceph-mon1" 
      },
      {
        "type": "mon",
        "id": "ceph-mon1" 
      }
    ],
    "hostname": "ceph-mon1",
    "ceph_version": "ceph version 14.2.5-1-g23e76c7aa6 (23e76c7aa6e15817ffb6741aafbc95ca99f24cbb) nautilus (stable)" 
  },
  {
    "services": [
      {
        "type": "mds",
        "id": "ceph-mon2" 
      },
      {
        "type": "mgr",
        "id": "ceph-mon2" 
      },
      {
        "type": "mon",
        "id": "ceph-mon2" 
      }
    ],
    "hostname": "ceph-mon2",
    "ceph_version": "ceph version 14.2.5-1-g23e76c7aa6 (23e76c7aa6e15817ffb6741aafbc95ca99f24cbb) nautilus (stable)" 
  },
  {
    "services": [
      {
        "type": "osd",
        "id": "2" 
      },
      {
        "type": "osd",
        "id": "3" 
      }
    ],
    "hostname": "ceph-node1",
    "ceph_version": "ceph version 14.2.5-1-g23e76c7aa6 (23e76c7aa6e15817ffb6741aafbc95ca99f24cbb) nautilus (stable)" 
  },
  {
    "services": [
      {
        "type": "osd",
        "id": "4" 
      },
      {
        "type": "osd",
        "id": "5" 
      }
    ],
    "hostname": "ceph-node101",
    "ceph_version": "ceph version 14.2.5-1-g23e76c7aa6 (23e76c7aa6e15817ffb6741aafbc95ca99f24cbb) nautilus (stable)" 
  },
  {
    "services": [
      {
        "type": "osd",
        "id": "10" 
      },
      {
        "type": "osd",
        "id": "11" 
      },
      {
        "type": "osd",
        "id": "12" 
      },
      {
        "type": "osd",
        "id": "6" 
      },
      {
        "type": "osd",
        "id": "7" 
      },
      {
        "type": "osd",
        "id": "8" 
      },
      {
        "type": "osd",
        "id": "9" 
      }
    ],
    "hostname": "ceph-node2",
    "ceph_version": "ceph version 14.2.5-1-g23e76c7aa6 (23e76c7aa6e15817ffb6741aafbc95ca99f24cbb) nautilus (stable)" 
  },
  {
    "services": [
      {
        "type": "osd",
        "id": "13" 
      },
      {
        "type": "osd",
        "id": "15" 
      },
      {
        "type": "osd",
        "id": "16" 
      }
    ],
    "hostname": "ceph-node3",
    "ceph_version": "ceph version 14.2.5-1-g23e76c7aa6 (23e76c7aa6e15817ffb6741aafbc95ca99f24cbb) nautilus (stable)" 
  }
]

Request URL: https://ceph.picote.ch/api/summary

Response:

{
  "rbd_mirroring": {
    "errors": 0,
    "warnings": 0
  },
  "finished_tasks": [],
  "executing_tasks": [],
  "health_status": "HEALTH_WARN",
  "version": "ceph version 14.2.5-1-g23e76c7aa6 (23e76c7aa6e15817ffb6741aafbc95ca99f24cbb) nautilus (stable)",
  "mgr_id": "ceph-mon2",
  "have_mon_connection": true,
  "mgr_host": "https://ceph-mon2.picote.ch:8443/" 
}

Finally, ceph-mon0 (the host which was upgraded and then rebooted before collecting the above information) reports the following version information from the CLI:

root@ceph-mon0:~# /usr/bin/ceph-mgr --version
ceph version 14.2.6-1-gffd69200ad (ffd69200ad0fc59d919bacc5a0f2a09f24be4134) nautilus (stable)

#13 Updated by Charley Sheets 7 months ago

The versions reported by the ceph versions command are what I expect. This is after upgrading the second of three mon servers:

{
    "mon": {
        "ceph version 14.2.5-1-g23e76c7aa6 (23e76c7aa6e15817ffb6741aafbc95ca99f24cbb) nautilus (stable)": 1,
        "ceph version 14.2.6-1-gffd69200ad (ffd69200ad0fc59d919bacc5a0f2a09f24be4134) nautilus (stable)": 2
    },
    "mgr": {
        "ceph version 14.2.5-1-g23e76c7aa6 (23e76c7aa6e15817ffb6741aafbc95ca99f24cbb) nautilus (stable)": 1,
        "ceph version 14.2.6-1-gffd69200ad (ffd69200ad0fc59d919bacc5a0f2a09f24be4134) nautilus (stable)": 1
    },
    "osd": {
        "ceph version 14.2.5-1-g23e76c7aa6 (23e76c7aa6e15817ffb6741aafbc95ca99f24cbb) nautilus (stable)": 12
    },
    "mds": {
        "ceph version 14.2.5-1-g23e76c7aa6 (23e76c7aa6e15817ffb6741aafbc95ca99f24cbb) nautilus (stable)": 1,
        "ceph version 14.2.6-1-gffd69200ad (ffd69200ad0fc59d919bacc5a0f2a09f24be4134) nautilus (stable)": 1
    },
    "overall": {
        "ceph version 14.2.5-1-g23e76c7aa6 (23e76c7aa6e15817ffb6741aafbc95ca99f24cbb) nautilus (stable)": 15,
        "ceph version 14.2.6-1-gffd69200ad (ffd69200ad0fc59d919bacc5a0f2a09f24be4134) nautilus (stable)": 4
    }
}

Also available in: Atom PDF