Project

General

Profile

Actions
Copy link

Bug #52327

closed

Module 'devicehealth' has failed: can't subtract offset-naive and offset-aware datetimes mgr/dashboard

Added by Benno Lange over 2 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Yaarit Hatuka
Category:
devicehealth module
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
pacific, octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v16.2.5
ceph-qa-suite:
Pull request ID:
43412
Crash signature (v1):
Crash signature (v2):

Description

Description of problem

Since we updated to v16.2.5 (using cephadm since availability) we see this status message :-(

MGR_MODULE_ERROR: Module 'devicehealth' has failed: can't subtract offset-naive and offset-aware datetimes

ceph device get-health-metrics <devid>

also gives a similar message ...

We assume that the failure prediction does also not work anymore too ....

Environment

  • ceph version string: 16.2.5
  • Platform (OS/distro/release): Ubuntu 20.04.3 LTS
  • Cluster details (nodes, monitors, OSDs): 5 nodes, 5 monitors, 32 OSDs (hdd/ssd)
  • Browser used (e.g.: Version 86.0.4240.198 (Official Build) (64-bit)):

How reproducible

ceph device get-health-metrics <devid>

Actual results

Inferring fsid 2af6c250-ccb0-4fe8-80af-f2181098c0e4
Inferring config /var/lib/ceph/2af6c250-ccb0-4fe8-80af-f2181098c0e4/mon.ceph-a/config
Using recent ceph image ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a
Error EIO: Module 'devicehealth' has experienced an error and cannot handle commands: can't subtract offset-naive and offset-aware datetimes

Expected results

here

Additional info

here

Related issues 2 (0 open — 2 closed)

Copied to mgr - Backport #53620: octopus: Module 'devicehealth' has failed: can't subtract offset-naive and offset-aware datetimes mgr/dashboardResolvedActions
Copied to mgr - Backport #53621: pacific: Module 'devicehealth' has failed: can't subtract offset-naive and offset-aware datetimes mgr/dashboardResolvedKonstantin ShalyginActions
Actions
Copy link
 #1

Updated by Harry Coin over 2 years ago

Saw this for the first time just after an upgrade to 16.2.6

Actions
Copy link
 #2

Updated by Benno Lange over 2 years ago

Luckily, after the upgrade to v16.2.6 the error seems to be gone :-)

Actions
Copy link
 #3

Updated by Harry Coin over 2 years ago

Looks like you are Bilbo and I am Frodo... now I have the ring....

Actions
Copy link
 #4

Updated by Josh Durgin over 2 years ago

  • Assignee changed from Sage Weil to Yaarit Hatuka
Actions
Copy link
 #5

Updated by Yaarit Hatuka over 2 years ago

  • Status changed from New to Fix Under Review
  • Backport set to pacific, octopus
  • Pull request ID set to 43412

Thanks for reporting this, Benno and Harry.
The issue was caused due to a missing timezone in one of the objects.
We have a fix which will be backported.

Actions
Copy link
 #6

Updated by Harry Coin over 2 years ago

Still exists in 16.2.7

Actions
Copy link
 #7

Updated by Konstantin Shalygin over 2 years ago

Harry Coin wrote:

Still exists in 16.2.7

The fix still under review

Actions
Copy link
 #8

Updated by Neha Ojha over 2 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions
Copy link
 #9

Updated by Backport Bot over 2 years ago

  • Copied to Backport #53620: octopus: Module 'devicehealth' has failed: can't subtract offset-naive and offset-aware datetimes mgr/dashboard added
Actions
Copy link
 #10

Updated by Backport Bot over 2 years ago

  • Copied to Backport #53621: pacific: Module 'devicehealth' has failed: can't subtract offset-naive and offset-aware datetimes mgr/dashboard added
Actions
Copy link
 #11

Updated by Yaarit Hatuka almost 2 years ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link

Also available in: Atom PDF