Project

General

Profile

Bug #16188

rgw: Swift API returns double space usage and objects of account metadata

Added by Albert Tu about 3 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Target version:
-
Start date:
06/08/2016
Due date:
% Done:

0%

Source:
other
Tags:
Backport:
jewel
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

When user upload file and run `stat` command, rgw returns correct account metadata immediately.

StorageURL: http://192.168.62.50:8080/swift/v1
Auth Token: AUTH_rgwtk0b0000006d656c6f6e3a7377696674da9bd854682bd73095e75857cd14fb39bd80bae4fc7af565839cdd2075b34cef4bdaff71
Account: v1
Containers: 1
Objects: 1
Bytes: 10M
Connection: Keep-Alive
X-Timestamp: 1465357845.97638
X-Account-Bytes-Used-Actual: 10485760
X-Trans-Id: tx000000000000000000077-0057579615-5e32-default
Content-Type: text/plain; charset=utf-8
Accept-Ranges: bytes

But after a few mintues rgw returns double space usage and objects of account metadata.
StorageURL: http://192.168.62.50:8080/swift/v1
Auth Token: AUTH_rgwtk0b0000006d656c6f6e3a7377696674258a5e645be0252461e85857b32bf422ce1df90fff606d86a86798d2b89ab4fff626a0fe
Account: v1
Containers: 1
Objects: 2
Bytes: 20M
Connection: Keep-Alive
X-Timestamp: 1465358049.58936
X-Account-Bytes-Used-Actual: 20971520
X-Trans-Id: tx000000000000000000079-00575796e1-5e32-default
Content-Type: text/plain; charset=utf-8
Accept-Ranges: bytes

Related issues

Copied to rgw - Backport #16599: jewel: rgw: Swift API returns double space usage and objects of account metadata Resolved

History

#2 Updated by Orit Wasserman about 3 years ago

In the second stat command you have two objects, maybe someone uploaded another object?
can you do reproduce and also run list buckets?

#3 Updated by Albert Tu about 3 years ago

This bug can reproduce on development environment.

$ swift -A http://localhost:8000/auth -U test:tester -K 'testing' stat --lh melon

Account: v1
Containers: 1
Objects: 2
Bytes: 20M
Connection: Keep-Alive
X-Timestamp: 1465895380.67945
X-Account-Bytes-Used-Actual: 20971520
X-Trans-Id: tx00000000000000000000c-00575fc9d4-101d-default
Content-Type: text/plain; charset=utf-8
Accept-Ranges: bytes

$ swift -A http://localhost:8000/auth -U test:tester -K 'testing' list --lh melon
10M 2016-06-14 09:04:04 tmp/10M
10M

The same issue before: https://github.com/ceph/ceph/pull/6047

#4 Updated by Pritha Srivastava about 3 years ago

I wasnt able to reproduce the problem. I tried the following:

1. In my dev environment, created a container - container1
2. Uploaded an object of 16 bytes to container1
3. Ran the stat command more than once - in each run I could see the correct number of objects which is 1 and its size which is 16.
4. Ran the list command - the ouput of it lists the object once. The size which is listed at the end after the objects is the total size of all objects in the container like

$ swift -A http://localhost:8000/auth/1.0 -U test:tester -K testing list --lh container1
21 2016-06-17 07:05:10 None sample.txt
16 2016-06-17 07:03:58 None sample1.txt
21 2016-06-17 07:05:19 None sample2.txt
58

I also checked the PR mentioned in the comment above: If the needs_stats is set to false, then it supresseses the stats related to objects output by the swift 'stat' command. I am not sure how that will solve the problem that is mentioned in the ticket.

#5 Updated by Pritha Srivastava about 3 years ago

Albert Tu wrote:

This bug can reproduce on development environment.
[...]

[...]
The same issue before: https://github.com/ceph/ceph/pull/6047

Hi Albert, which branch are you working off? I tried the above using the master branch and couldn't see the issue in my dev environment.

#6 Updated by Albert Tu almost 3 years ago

Pritha Srivastava wrote:

Albert Tu wrote:

This bug can reproduce on development environment.
[...]

[...]
The same issue before: https://github.com/ceph/ceph/pull/6047

Hi Albert, which branch are you working off? I tried the above using the master branch and couldn't see the issue in my dev environment.

I'm using the master branch (87b5a65). Sorry for typo, the correct stat command shouldn't append container name.

#7 Updated by Albert Tu almost 3 years ago

The stat command returns double space usage because rgw_read_user_buckets() get bucket stats and then count based on it when parameter `need_stats` is true.

Bucket stats are synced by BucketsSyncThread every 3 minutes. That's why we need wait a few minutes to reproduce this bug.

Code links:
  1. RGWStatAccount::execute() https://github.com/ceph/ceph/blob/master/src/rgw/rgw_op.cc#L1582
  2. rgw_read_user_buckets() https://github.com/ceph/ceph/blob/master/src/rgw/rgw_bucket.cc#L99
  3. BucketsSyncThread https://github.com/ceph/ceph/blob/master/src/rgw/rgw_quota.cc#L398

#8 Updated by Albert Tu almost 3 years ago

#9 Updated by Albert Tu almost 3 years ago

Need to backport to hammer. Do I need to create issue or pull request?

#10 Updated by Nathan Cutler almost 3 years ago

  • Status changed from New to Pending Backport
  • Backport set to jewel

#11 Updated by Nathan Cutler almost 3 years ago

  • Copied to Backport #16599: jewel: rgw: Swift API returns double space usage and objects of account metadata added

#12 Updated by Nathan Cutler almost 3 years ago

  • Subject changed from jewel: rgw: Swift API returns double space usage and objects of account metadata to rgw: Swift API returns double space usage and objects of account metadata

#13 Updated by Nathan Cutler almost 3 years ago

jewel backport has been staged. Albert, the bug says affected version is 10.2.1 which is jewel. Are you sure this needs to be backported to hammer (0.94.x)?

#14 Updated by Albert Tu almost 3 years ago

Sure, because #13140 fixes this bug by don't call update_container_stats(). This made radosgw doesn't return immediate stats.

And after backported this, the list command gets fixed too.

#15 Updated by Loic Dachary almost 3 years ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF