Project

General

Profile

Actions
Copy link

Bug #14197

closed

Rados bench 4K write stops and reports pipe fault

Added by Jianjian Huo over 8 years ago. Updated over 8 years ago.

Status:
Rejected
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
v9.2.1
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Installed Infernalis 9.2.0 on one x64 Ubuntu 14.04 machine, setup one monitor and one OSD on same machine; ran radosbench 4MB or 16KB to fill the empty OSD to full, no problem; but ran radosbench 4KB to fill the empty OSD, it always stopped at near half capacity(total 55488 kobjects, 216 GB data), and reported below error:

2015-12-28 21:15:24.479928 7f566036b700 0 -- 10.1.10.102:0/2817697853 >> 10.1.10.102:6800/3804 pipe(0x7f5658002110 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f56580063b0).fault

$ceph -s
cluster ae29003b-8559-4896-8315-5e887eba1ede
health HEALTH_WARN
192 pgs stale
192 pgs stuck stale
1/1 in osds are down
monmap e1: 1 mons at {msl-lab-dsg02=10.1.10.102:6789/0}
election epoch 2, quorum 0 msl-lab-dsg02
osdmap e8: 1 osds: 0 up, 1 in
flags sortbitwise
pgmap v3731: 192 pgs, 2 pools, 216 GB data, 55488 kobjects
339 GB used, 101 GB / 441 GB avail
192 stale+active+clean

And ceph.log has this on bottom:
2015-12-28 21:30:19.459596 mon.0 10.1.10.102:6789/0 3779 : cluster [INF] osd.0 marked down after no pg stats for 901.104704seconds

Then I tried latest master branch on another machine, same issue, and it's easy to reproduce it. Have ran Hammer 0.94.2 a few months back, didn't see this issue.

Actions
Copy link

Also available in: Atom PDF