Bug #24744
open
rgw: index wrongly deleted when put raced with list
Added by Tianshan Qu almost 6 years ago.
Updated almost 3 years ago.
Backport:
octopus, pacific
Description
like the issue http://tracker.ceph.com/issues/22555 , a special sequence can cause this new situation.
IO sequence:
1.put index prepare
2.list, get stale index
3.check_disk_state, find the head obj not exist
4.write head obj
5.index complete
6.aio_operate dir_suggest_changes CEPH_RGW_REMOVE
step 6 will delete the index
Files
- Status changed from New to Fix Under Review
- Status changed from Fix Under Review to 7
- Assignee set to J. Eric Ivancich
This race still exists in latest mimic (v13.2.6), we are hitting it several times a day in a production setup. The sequence of events seems to be:
1. PUT does bucket_prepare_op, creates index entry with exists=false and pending_map with CLS_RGW_OP_ADD
2. LIST sees in-flight index entry with both head object and index entry exists=false, suggests CEPH_RGW_REMOVE
3. PUT does bucket_complete_op, index entry updated, pending_map cleared
4. rgw_dir_suggest_changes deletes index entry
It seems that rgw_dir_suggest_changes tries to avoid interfering with in-flight ops by checking that the pending_map empty first: "if (cur_disk.pending_map.empty()) { ... }", but that does not account for ops that have just completed successfully.
I think you are right, and the issue still exists in master, will repush the original fix.
- Status changed from 7 to Fix Under Review
- Backport set to nautilus,mimic
- Pull request ID set to 28654
Its fairly easy to repro the bug, for what its worth: I attached a repro. Copying the file to itself makes it orders of magnitude harder to hit, but still you can hit it. The orphan can be found using rgw-orphan-list, which can be used as a workaround.
- Pull request ID changed from 28654 to 41978
- Target version set to v17.0.0
- Backport changed from nautilus,mimic to octopus, pacific
Also available in: Atom
PDF