Project

General

Profile

Actions
Copy link

Bug #61600

open

dmClock client_map entries exceeding the erase age are not removed by the periodic clean-up job in some cases.

Added by Sridhar Seshasayee 12 months ago.

Status:
New
Priority:
Normal
Assignee:
Sridhar Seshasayee
Category:
Correctness/Safety
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
dmclock
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

The issue was originally observed in https://tracker.ceph.com/issues/61594. It was reproduced locally.
The test involved just running client I/O and dumping the queue state after the erase age has elapsed.

The output below shows one of the OSD shards still showing client entries:

$ ceph tell osd.0 dump_op_pq_state | grep "clients" 
            "clients": "{ PriorityQueue:: HEAPS-EMPTY }" 
            "clients": "{ PriorityQueue::  { client:{ class_id: 3 client_id: 4149 profile_id: 0 }, record:{ ClientRec:: client:{ class_id: 3 client_id: 4149 profile_id: 0 } prev_tag:{ RequestTag:: ready:false r:56147.0570 p:666547.0191 l:0.0000 } req_count:0 top_req:none } } { reservation_top:{ ClientRec:: client:{ class_id: 3 client_id: 4149 profile_id: 0 } prev_tag:{ RequestTag:: ready:false r:56147.0570 p:666547.0191 l:0.0000 } req_count:0 top_req:none } } { ready_top:{ ClientRec:: client:{ class_id: 3 client_id: 4149 profile_id: 0 } prev_tag:{ RequestTag:: ready:false r:56147.0570 p:666547.0191 l:0.0000 } req_count:0 top_req:none } } { limit_top:{ ClientRec:: client:{ class_id: 3 client_id: 4149 profile_id: 0 } prev_tag:{ RequestTag:: ready:false r:56147.0570 p:666547.0191 l:0.0000 } req_count:0 top_req:none } } }" 
            "clients": "{ PriorityQueue:: HEAPS-EMPTY }" 
            "clients": "{ PriorityQueue:: HEAPS-EMPTY }" 
            "clients": "{ PriorityQueue:: HEAPS-EMPTY }" 
            "clients": "{ PriorityQueue:: HEAPS-EMPTY }" 
            "clients": "{ PriorityQueue:: HEAPS-EMPTY }" 
            "clients": "{ PriorityQueue:: HEAPS-EMPTY }"

The client entries are maintained by the dmclock server within the 'client_map' data structure.
This map is periodically cleaned up by a thread running in the background based on the erase
age of an entry which is currently set to 600 secs.

Further investigation into why this is happening is in-progress.

No data to display

Actions
Copy link

Also available in: Atom PDF