Project

General

Profile

Actions

Bug #21542

open

luminous: index gets heavy read IOPS with index-less RGW pool

Added by Yuri Gorshkov over 6 years ago. Updated over 6 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
luminous, radosgw, index, blind buckets
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Hi All,

Recently, we've noticed a strange behaviour on one of our test clusters.

The cluster was configured to serve RGW and is running Luminous. Our standard procedure is to create blind (non-indexed) buckets so that our software manages the metadata by itself and we get less load on the index pool, with the zone configured as follows (notice the index type 1, which stands for blind buckets).

What we're seeing is periodical read spikes on the index pool, which are getting so bad the cluster starts seeing slow requests with barely any load. If I'm right in this configuration the index pool should not be getting any IOPS at all (at least on Kraken it worked this way).
Our Kraken sites don't have any objects in this pool, but for some reason our Luminous site has ~65K objects there - seems like RGW created them automatically?

Here's the relevant info from the cluster: https://pastebin.ca/3876293

Previously, we were running Kraken at multiple sites in the same configuration and never noticed this happening. Any ideas?


Files

index_stat.txt.xz (27 KB) index_stat.txt.xz per-second log of index pool I/O Yuri Gorshkov, 09/25/2017 08:01 PM
Actions #2

Updated by John Spray over 6 years ago

  • Project changed from Ceph to rgw
  • Category deleted (22)
Actions #3

Updated by Yehuda Sadeh over 6 years ago

Is this still happening? Can you list the pool, what type of objects do you have there?

Actions

Also available in: Atom PDF