Project

General

Profile

Feature #6338

rgw: better manifest scaling

Added by Yehuda Sadeh over 10 years ago. Updated about 10 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
% Done:

0%

Source:
other
Tags:
Backport:
Reviewed:
Affected Versions:
Pull request ID:

History

#1 Updated by Ian Colle about 10 years ago

  • Target version set to v0.76

#2 Updated by Ian Colle about 10 years ago

  • Target version changed from v0.76 to v0.78

#3 Updated by Ian Colle about 10 years ago

  • Status changed from New to In Progress
  • Assignee set to Yehuda Sadeh

#4 Updated by Ian Colle about 10 years ago

  • translation missing: en.field_story_points set to 5.0

#5 Updated by Yehuda Sadeh about 10 years ago

Currently a gateway object includes in the manifest a list of all the rados objects that it uses. Every time we access the object we need to read the entire list, which can grow extremely large for larger object, which means that access time to really large objects can be too long, potentially lead to timeouts, overloading the osds (due to retries snowballing). Also, it might be impossible to store the large manifest with certain underlying osd filesystem.

The solution we discussed was instead of keeping a list of all the object, keep a list of rules on how to generate the object names. This means that for regular upload objects we're only going to have a single rule. And for multipart upload we'll have a single rule in most cases -- except for the case of client sending different sized parts, in which case it's going to be O(num of different sizes).

#6 Updated by Ian Colle about 10 years ago

  • Status changed from In Progress to 7

#7 Updated by Yehuda Sadeh about 10 years ago

  • Status changed from 7 to Resolved

Merged, commit:c5d3bdf7fa23990171024cc0a4480d50a3951f41

Also available in: Atom PDF