Bug #9078
closed
Removing an RBD is very slow whenever there is write's in other RBD which also belongs to same Pool
Added by Ramakrishnan P over 9 years ago.
Updated over 9 years ago.
Description
Configuration:
3 node with mon and 3 node with OSD connected via Enclosure/jbod, total 15 OSD's
Steps followed:
1. Create Cluster and ensure Cluster health "ok"
2. Create 2 Pools and 3 RBD's in each pool(total 6 RBD) with different order size (created each RBD of 1TB)
3. Map all 6 RBD to client
4. Stated IO's on all 6 RBD with write/randwrite/readwrite
5. After 1 hr of IO, stop IO on one RBD, and let IO's continue in remaining RBD's
6. unmap RBD from client which will succeed, ensure no IO's are continuing in particular RBD
7. remove/delete RBD from pool which seems like hung but it is very slow, after 4 min can observe % of RBD removal
- Status changed from New to Need More Info
it sounds like the cluster is just under heavy load. can you confirm how many ops ceph -w shows before and during the remove?
the remove probably should have the lowest priority anyway...
setup is not available, unable to check "ceph -w", below are information based on IO tool(fio)
before rbd remove: iops=818
while rbd remove: iops=614
Observations:
1. if IO write's are going on rbd remove will be slower (check this after min 1 hr of IO)
2. if it is newly created rbd, rbd remove is fine
3. checked for IO read's, rbd remove is fine but IO ran for only 10 mins (after 1 hr not verified)
RBD's are created with different order parameter
This seems like more or less what I would expect. What do you suggest should happen? Either rbd remove is slow and client io is impacted less, or remove is fast and client io is impacted more ...
Either rbd remove will get any impact on different orders(12-24) or it will be same behaviour ?, because for lesser order the created objects will be more so it will take time to remove rbd.
Yes, smaller order means more objects to remove...
- Status changed from Need More Info to Rejected
Also available in: Atom
PDF