Project

General

Profile

Actions
Copy link

Bug #51165

open

mgr/telegraf: telegraf plugin not starting and causing mgr process to crash

Added by Bastian Mäuser almost 3 years ago. Updated over 2 years ago.

Status:
New
Priority:
Normal
Assignee:
Wido den Hollander
Category:
ceph-mgr
Target version:
Ceph - v17.0.0
% Done:

0%

Source:
Community (user)
Tags:
Backport:
octopus, pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v15.2.13
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Description of problem

Cluster goes HEALTH_ERR after enabling telegraf plugin.

Environment

  • ceph version string: 15.2.13
  • Platform (OS/distro/release): Proxmox 6.4-8
  • Cluster details (nodes, monitors, OSDs): 5 nodes, 5mons, 20 OSD
  • Browser used (e.g.: Version 86.0.4240.198 (Official Build) (64-bit)): FF89

How reproducible

Run Octopus 15.2.13 (In earlier Versions an issue, too), enable telegraf plugin with:

ceph telegraf config-set address udp://:8094
ceph telegraf config-set interval 10
ceph mgr module enable telegraf

Actual results

Cluster goes HEALTH_ERR, message:

[ERR] [ERR] MGR_MODULE_ERROR: Module 'telegraf' has failed: str, bytes or bytearray expected, not NoneType

Expected results

Start broadcasting ceph telemetry via UDP to listening telegrafs

Actions
Copy link
 #1

Updated by Neha Ojha almost 3 years ago

  • Assignee set to Wido den Hollander
  • Backport set to octopus, pacific

Hi Wido, could you please help take a look at this.

Actions
Copy link
 #2

Updated by Wido den Hollander over 2 years ago

This only happens when using UDP broadcasting? It seems that the internal URL parsing inside the Telegraf module breaks

See: https://github.com/ceph/ceph/blob/master/src/pybind/mgr/telegraf/basesocket.py#L27

        self.sock = socket.socket(family=socket_family, type=socket_type)
        if self.sock.family == socket.AF_UNIX:
            self.address: Union[str, Tuple[str, int]] = self.url.path
        else:
            assert self.url.hostname
            assert self.url.port
            self.address = (self.url.hostname, self.url.port)
Actions
Copy link
 #3

Updated by Neha Ojha over 2 years ago

Wido den Hollander wrote:

This only happens when using UDP broadcasting? It seems that the internal URL parsing inside the Telegraf module breaks

See: https://github.com/ceph/ceph/blob/master/src/pybind/mgr/telegraf/basesocket.py#L27

[...]

We've seen another similar report, which is also using UDP broadcasting.

Actions
Copy link
 #4

Updated by Scott Hubbard over 2 years ago

Neha Ojha wrote:

We've seen another similar report, which is also using UDP broadcasting.

It works if you pass in a localhost IP address.

ceph telegraf config-set address udp://127.0.0.1:8094

Actions
Copy link

Also available in: Atom PDF