Ceph » rbd

hi jason, sorry for late response, i can reproduce this on the latest master, it seems that this only happens with async messenger, the 'rados ls' output is greater than 'rbd du' output, but with simple messenger i can not reproduce this, the 'rados ls' output is lower than or equal to 'rbd du' output, i think this is reasonable since librbd always updates object map before writing the data to ceph cluser

however in both cases, inconsistency state always occurs, my first thought is to update the object map and set the object map as invalid or other transient state, object map is only set to valid until the write completes, does this make sense to you

I am still not following the issue and will require more information. To answer your question, you cannot set the object map to valid after the block is updated because that would mean the object map doesn't reflect reality (i.e. if you were to crash after adding the object but before updating the object map, you would have an orphaned object).

m thought is as follows

1. update object map, set object map state to invalid or transient state
2. write data to rados
3. clear invalid state or change transient state to valid state

if rbd crashed after step 2, object map is updated but it is not a valid state, since we provide object map rebuild api, the user should rebuild objectmap if the objectmap is invalid.

You really need to explain to me the problem you are attempting to solve because I am not understanding it.

i want to fix the inconsistency between 'rbd df' and 'rados ls', as this tracker state, 'rados ls' show that there are 26 objects in this rbd, 'rbd du' should return 106496K, but actually it shows 24576K, only 6 objects has been written in this rbd, this is the inconsistenc of these two commands, i investigate write flow, found that librbd always update objectmap before writing data, if librbd crashes after updating objectmap, but before writing data, these two commands's output is inconsistent.

OK, I think I am still not understanding the issue. If `rados ls` shows 26 objects (104MB of used space) but `rbd du` shows 24MB of used space (6 objects), how is what you are suggesting going to address this issue? Since the object map is updated before the object is written, if anything, `rbd du` should say 104MB of space is used and `rados ls` would only show 24MB.

it looks to me like the only possibility is random write instead of sequential write was used to fill the drive, rbd du report 24MB doesnt necessary means 6 objects, it just iter the diff and sum up.

The easiest way to make this "issue" clear, @Xinxin, could you pls add output in https://github.com/ceph/ceph/blob/master/src/tools/rbd/action/DiskUsage.cc#L27? just log out <offset, len, exists> , which will be clear enough to show your disk structure to everybody.

@Xinxin: Thanks for the debug logs. I will try to take a look at this today or tomorrow.