Project

General

Profile

Actions
Copy link

Bug #16981

closed

OSD crashes under moderate write load when using bluestore

Added by Yuri Gorshkov over 7 years ago. Updated about 7 years ago.

Status:
Won't Fix
Priority:
Low
Assignee:
-
Category:
OSD
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
jewel, osd, crash
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
v10.2.2
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Hi.

While testing out bluestore, one of our OSDs crashed suddenly outputting the following logs (see below):
After OSD restart it was able to get up again, so far we couldn't reproduce the bug reliably but I decided to file it as a bug anyway.

I will update the bug report if we happen to reproduce this again.

OSD output:

авг 10 16:16:51 stor01 ceph-osd[863226]: terminate called after throwing an instance of 'ceph::buffer::bad_alloc'
авг 10 16:16:51 stor01 ceph-osd[863226]: what():  buffer::bad_alloc
авг 10 16:16:51 stor01 ceph-osd[863226]: *** Caught signal (Aborted) **
авг 10 16:16:51 stor01 ceph-osd[863226]: in thread 7f62feb80700 thread_name:ms_pipe_read
авг 10 16:16:51 stor01 ceph-osd[863226]: terminate called recursively

Our osd config is now like this:

[osd]
osd mkfs type = xfs
osd mkfs options xfs = -f -i size=2048
osd mount options xfs = noatime,largeio,inode64,swalloc
osd journal size = 2600
osd op threads    = 4

osd objectstore = bluestore
bluestore block path = /dev/disk/by-partlabel/osd-device-$id-block
bluestore bluefs = false
bluestore fsck on mount = true
bluestore block db path = /var/lib/ceph/osd/$cluster-$id/block.db
bluestore block db create = true
bluestore block wal path = /var/lib/ceph/osd/$cluster-$id/block.wal
bluestore block wal create = true
bluestore rocksdb options = compression=kNoCompression,max_write_buffer_number=16,min_write_buffer_number_to_merge=3

osd recovery delay start = 10
osd recovery threads = 2
#osd recovery max active = 2

Actions
Copy link

Also available in: Atom PDF