Project

General

Profile

Actions
Copy link

Bug #22944

open

Infiniband send_msg send returned error 32: (32) Broken pipe

Added by Radosław Piliszek about 6 years ago. Updated about 6 years ago.

Status:
New
Priority:
Normal
Assignee:
Haomai Wang
Category:
-
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
rdma, infiniband
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
v12.2.2
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Using CentOS 7.4, Mellanox OFED 4.2 on Connect-X 3 in Infiniband mode and Ceph 12.2.2 compiled with RDMA.

I've set:

ms type = async+rdma

I've fixed systemd units to allow RDMA device usage.

But I cannot get RDMA to work. Connections time out.

I sometimes get:

Infiniband send_msg send returned error 32: (32) Broken pipe

in monitor service journal.

RDMA by itself works just fine (verified with rping).

Ceph by itself also works just fine (verified without RDMA).

This happens even on one-node cluster when trying to access it from the very same node.

Please let me know how I can get you more info to debug it.

Files

Download all files
ceph-client.smp-016.log (46.4 KB) ceph-client.smp-016.log client log Radosław Piliszek, 02/13/2018 08:01 AM
ceph-mon.smp-016.log (54 KB) ceph-mon.smp-016.log mon log Radosław Piliszek, 02/13/2018 08:01 AM
ibdump.smp-016.pcap (3.46 KB) ibdump.smp-016.pcap ibdump capture Radosław Piliszek, 02/13/2018 08:01 AM
Actions
Copy link

Also available in: Atom PDF