Bug #61950: mds/OpenFileTable: match MAX_ITEMS_PER_OBJ does not honor osd_deep_scrub_large_omap_object_key_threshold - CephFS - Ceph

Actions

Copy link

Bug #61950

open

mds/OpenFileTable: match MAX_ITEMS_PER_OBJ does not honor osd_deep_scrub_large_omap_object_key_threshold

Added by Vikhyat Umrao 10 months ago. Updated 5 months ago.

Status:

Need More Info

Priority:

High

Assignee:

Category:

Correctness/Safety

Target version:

Ceph - v19.0.0

% Done:

Source:

Support

Tags:

Backport:

reef,quincy,pacific

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(FS):

MDS

Labels (FS):

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

The changes implemented in [1] should make sure that we never have an openfiletable objects omap keys above osd_deep_scrub_large_omap_object_key_threshold( default 200000) but in Ceph version: 14.2.22-110.el8cp (2e0d97dbe192cca7419bbf3f8ee6b7abb42965c4) nautilus

We see the following message in `ceph -s` and the following log in cluster logs:

LARGE_OMAP_OBJECTS

2023-07-09 10:42:34.177821 osd.123 (osd.123) 16631 : cluster [WRN] Large omap object found. Object: 3:2adbd678:::mds0_openfiles.9:head PG: 3.1e6bdb54 (3.154) Key count: 200001 Size (bytes): 12824872

In this situation the mds0_openfiles.9 object has just 1 key extra from the threshold(200000).

[1] Old Tracker and PR:
https://tracker.ceph.com/issues/42515
https://github.com/ceph/ceph/pull/31232

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Venky Shankar 10 months ago

Ok, so an off-by-one error - should be relatively easy to figure...

Actions

Copy link

Updated by Patrick Donnelly 10 months ago

Category set to Correctness/Safety
Status changed from New to In Progress
Assignee set to Patrick Donnelly
Priority changed from Normal to High
Target version set to v19.0.0
Source set to Support
Backport set to reef,quincy,pacific
Component(FS) MDS added

Actions

Copy link

Updated by Venky Shankar 10 months ago

Patrick - I can take this one if you haven't started on it yet.

Actions

Copy link

Updated by Patrick Donnelly 10 months ago

Venky Shankar wrote:

Patrick - I can take this one if you haven't started on it yet.

I have started on it. Should be quick.

Actions

Copy link

Updated by Patrick Donnelly 10 months ago

Related to Bug #42515: fs: OpenFileTable object shards have too many k/v pairs added

Actions

Copy link

Updated by Patrick Donnelly 10 months ago

Venky Shankar wrote:

Patrick - I can take this one if you haven't started on it yet.

https://github.com/ceph/ceph/pull/52415#issuecomment-1632803909

I haven't found the cause. If you'd like to have a stab at it be my guest. :)

Actions

Copy link

Updated by Patrick Donnelly 10 months ago

Pull request ID set to 52415

Actions

Copy link

Updated by Venky Shankar 8 months ago

Patrick Donnelly wrote:

Venky Shankar wrote:

Patrick - I can take this one if you haven't started on it yet.

https://github.com/ceph/ceph/pull/52415#issuecomment-1632803909

I haven't found the cause. If you'd like to have a stab at it be my guest. :)

I'll have a look. Assigning to myself.

Actions

Copy link