Bug #9078: Removing an RBD is very slow whenever there is write's in other RBD which also belongs to same Pool - rbd - Ceph

Actions

Copy link

Bug #9078

closed

Removing an RBD is very slow whenever there is write's in other RBD which also belongs to same Pool

Added by Ramakrishnan P over 9 years ago. Updated over 9 years ago.

Status:

Rejected

Priority:

Normal

Assignee:

Sage Weil

Target version:

% Done:

Source:

other

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

Configuration:
3 node with mon and 3 node with OSD connected via Enclosure/jbod, total 15 OSD's

Steps followed:
1. Create Cluster and ensure Cluster health "ok"
2. Create 2 Pools and 3 RBD's in each pool(total 6 RBD) with different order size (created each RBD of 1TB)
3. Map all 6 RBD to client
4. Stated IO's on all 6 RBD with write/randwrite/readwrite
5. After 1 hr of IO, stop IO on one RBD, and let IO's continue in remaining RBD's
6. unmap RBD from client which will succeed, ensure no IO's are continuing in particular RBD
7. remove/delete RBD from pool which seems like hung but it is very slow, after 4 min can observe % of RBD removal

Actions

Copy link

Updated by Sage Weil over 9 years ago

Status changed from New to Need More Info

it sounds like the cluster is just under heavy load. can you confirm how many ops ceph -w shows before and during the remove?

the remove probably should have the lowest priority anyway...

Actions

Copy link

Updated by Ramakrishnan P over 9 years ago

setup is not available, unable to check "ceph -w", below are information based on IO tool(fio)
before rbd remove: iops=818
while rbd remove: iops=614

Observations:
1. if IO write's are going on rbd remove will be slower (check this after min 1 hr of IO)
2. if it is newly created rbd, rbd remove is fine
3. checked for IO read's, rbd remove is fine but IO ran for only 10 mins (after 1 hr not verified)

Actions

Copy link

Updated by Ramakrishnan P over 9 years ago

RBD's are created with different order parameter

Actions

Copy link

Updated by Sage Weil over 9 years ago

This seems like more or less what I would expect. What do you suggest should happen? Either rbd remove is slow and client io is impacted less, or remove is fast and client io is impacted more ...

Actions

Copy link

Updated by Ramakrishnan P over 9 years ago

Either rbd remove will get any impact on different orders(12-24) or it will be same behaviour ?, because for lesser order the created objects will be more so it will take time to remove rbd.

Actions

Copy link