Project

General

Profile

Bug #23526

"Message too long" error when appending journal

Added by Mykola Golub 12 months ago. Updated 11 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
Start date:
03/30/2018
Due date:
% Done:

0%

Source:
Tags:
Backport:
luminous,jewel
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

When appending to a journal object the number of appends sent in one rados operation is not limited and we may hit osd_max_write_size limit. It is more likely to observe when using very large rbd cache.

It looks like it is much harder to trigger on the master branch, but on luminous it is easily reproduced with this scenario:

rbd create test -s 100G
rbd feature enable test journaling
rbd-nbd --rbd_cache_max_dirty=1549551462 --rbd_cache_size=1610612736 --rbd_cache_target_dirty=1449551462 map test
mkfs.xfs -f /dev/nbd0
mount /dev/nbd0 /mnt
cd /mnt
fio --name=writefile --filesize=1G --bs=4M --nrfiles=16 --direct=1 --sync=0 --randrepeat=0 --rw=write --refill_buffers --end_fsync=1 --iodepth=16 --ioengine=libaio

Eventually it fails with:

  librbd::Journal: 0x7f52440037e0 handle_io_event_safe: failed to commit IO event: (90) Message too long


Related issues

Copied to rbd - Backport #23545: luminous: "Message too long" error when appending journal Resolved
Copied to rbd - Backport #23546: jewel: "Message too long" error when appending journal Resolved

History

#1 Updated by Mykola Golub 12 months ago

  • Status changed from In Progress to Need Review

#2 Updated by Jason Dillaman 12 months ago

  • Status changed from Need Review to Pending Backport

#3 Updated by Nathan Cutler 12 months ago

  • Copied to Backport #23545: luminous: "Message too long" error when appending journal added

#4 Updated by Nathan Cutler 12 months ago

  • Copied to Backport #23546: jewel: "Message too long" error when appending journal added

#7 Updated by Nathan Cutler 11 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF