Project

General

Profile

Bug #55410

[rgw-ms][dbr]:Large omap object found in the buckets.index pool, corresponding to resharded buckets

Added by Vidushi Mishra over 1 year ago. Updated over 1 year ago.

Status:
Won't Fix
Priority:
Normal
Assignee:
-
Target version:
Ceph - v17.0.0
% Done:

0%

Source:
Q/A
Tags:
multisite-reshard
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

We observe warning like large omap found in the index pool corresponding to resharded buckets that did not completely sync.

[WRN] LARGE_OMAP_OBJECTS: 2 large omap objects
2 large objects found in pool 'east.rgw.buckets.index'
Search the cluster log for 'Large omap object found' for more details.

---------------------------
snippet from the osd.log
---------------------------- 

grep -C 200 Large /var/log/ceph/e16ec086-bfb0-11ec-8537-002590fbd524/ceph-osd.*.log | egrep 'Large omap'
/var/log/ceph/e16ec086-bfb0-11ec-8537-002590fbd524/ceph-osd.2.log:2022-04-22T05:14:32.703+0000 7fcc52a8e700  0 log_channel(cluster) log [WRN] : Large omap object found. Object: 7:16ead4f9:::.dir.167fc967-e35e-4eb4-bc09-f71b324f77d8.31813.1.2:head PG: 7.9f2b5768 (7.8) Key count: 536197 Size (bytes): 141281185

History

#1 Updated by Vidushi Mishra over 1 year ago

1. ceph version 17.0.0-10783-ge38464a1 (e38464a10ae9e8c7b43bae5a9a7395eb2cbb2444) quincy (dev)

2. Steps to reproduce:

i. Create a multi-site with 14 rgws on each site [4 for ms sync and 10 for client IO.]
ii. The 4 rgws for ms sync are not behind any LB.
iii. create a bucket 'test-sync-no-lb-1' and upload 20M objects [10M from either site.]

Result:

We observe Large omap objects found in the index pool on the primary site.

#2 Updated by Casey Bodley over 1 year ago

Vidushi Mishra wrote:

1. ceph version 17.0.0-10783-ge38464a1 (e38464a10ae9e8c7b43bae5a9a7395eb2cbb2444) quincy (dev)

2. Steps to reproduce:

i. Create a multi-site with 14 rgws on each site [4 for ms sync and 10 for client IO.]
ii. The 4 rgws for ms sync are not behind any LB.
iii. create a bucket 'test-sync-no-lb-1' and upload 20M objects [10M from either site.]

Result:

We observe Large omap objects found in the index pool on the primary site.

this is probably just because bilog trim can't keep up with your 20M object workload. large omap can be expected here

#3 Updated by Casey Bodley over 1 year ago

  • Status changed from New to Won't Fix

Also available in: Atom PDF