Project

General

Profile

Actions

Bug #62051

open

rgw usage cannot retrieve some information (1H period)

Added by jielei zhou 10 months ago. Updated 7 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
admin usage
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
ceph-deploy
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

when i use rgw usage to get user usage. I found if i request 1h info. There is some info lost in it.
like this. list_buckets field will not be display. But if i change period to 2H list_bucket field will be display.
example like this
2023-07-15 00:00:00~01:00:00 have not list_buckets field
2023-07-15 01:00:00~02:00:00 have not list_buckets field
2023-07-15 00:00:00~02:00:00 have list_buckets field

1H period by 00~01 hours
```
root@dev-ceph-y13004:~# bash test.sh GET usage "uid=f4e0b6c3eba8417d95781c1136265036%24f4e0b6c3eba8417d95781c1136265036&start=2023-07-15%2000:00:00&end=2023-07-15%2001:00:00" |jq .Note: Unnecessary use of -X or --request, GET is already inferred.
  • Trying ****:80...
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
    Connected to 172.22.35.94 (172.22.35.94) port 80 (#0)

GET /admin/usage?uid=f4e0b6c3eba8417d95781c1136265036%24f4e0b6c3eba8417d95781c1136265036&start=2023-07-15%2000:00:00&end=2023-07-15%2001:00:00 HTTP/1.1
Host:
User-Agent: curl/7.81.0
Accept: */

Content-Type: application/x-compressed-tar
Date: Tue, 18 Jul 2023 00:56:28 +0000
Authorization: AWS admin:Bhmf79WOa66PwheuNEeFWbIblpU=

  • Mark bundle as not supporting multiuse
    < HTTP/1.1 200 OK
    < x-amz-request-id: tx00000a3db35ea89f74a8c-0064b5e33c-4ad68d-dev-c3j1
    < Content-Length: 633
    < Date: Tue, 18 Jul 2023 00:56:28 GMT
    < Connection: Keep-Alive
    < { [633 bytes data]
    100 633 100 633 0 0 24641 0 --:--:-- --:--:-- --:--:-- 25320
  • Connection #0 to host * left intact {
    "entries": [ {
    "user": "f4e0b6c3eba8417d95781c1136265036$f4e0b6c3eba8417d95781c1136265036",
    "buckets": [ {
    "bucket": "test",
    "time": "2023-07-15T00:00:00.000000Z",
    "epoch": 1689379200,
    "owner": "f4e0b6c3eba8417d95781c1136265036$f4e0b6c3eba8417d95781c1136265036",
    "categories": [ {
    "category": "list_bucket",
    "bytes_sent": 2590,
    "bytes_received": 0,
    "ops": 20,
    "successful_ops": 20
    }
    ]
    }
    ]
    }
    ],
    "summary": [ {
    "user": "f4e0b6c3eba8417d95781c1136265036$f4e0b6c3eba8417d95781c1136265036",
    "categories": [ {
    "category": "list_bucket",
    "bytes_sent": 2590,
    "bytes_received": 0,
    "ops": 20,
    "successful_ops": 20
    }
    ],
    "total": {
    "bytes_sent": 2590,
    "bytes_received": 0,
    "ops": 20,
    "successful_ops": 20
    }
    }
    ]
    }
    ```
1H period by 01~02 hours
```
root@dev-ceph-y13004:~# bash test.sh GET usage "uid=f4e0b6c3eba8417d95781c1136265036%24f4e0b6c3eba8417d95781c1136265036&start=2023-07-15%2001:00:00&end=2023-07-15%2002:00:00" |jq .Note: Unnecessary use of -X or --request, GET is already inferred.
  • Trying :80...
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
    Connected to 172.22.35.94 (172.22.35.94) port 80 (#0)

GET /admin/usage?uid=f4e0b6c3eba8417d95781c1136265036%24f4e0b6c3eba8417d95781c1136265036&start=2023-07-15%2001:00:00&end=2023-07-15%2002:00:00 HTTP/1.1
Host:
User-Agent: curl/7.81.0
Accept: */

Content-Type: application/x-compressed-tar
Date: Tue, 18 Jul 2023 01:00:13 +0000
Authorization: AWS admin:wPJGOd/ELfnT4FgV7rZjm3rzndU=

  • Mark bundle as not supporting multiuse
    < HTTP/1.1 200 OK
    < x-amz-request-id: tx000007c53f9ba6912672e-0064b5e41d-4ad68d-dev-c3j1
    < Content-Length: 633
    < Date: Tue, 18 Jul 2023 01:00:13 GMT
    < Connection: Keep-Alive
    < { [633 bytes data]
    100 633 100 633 0 0 26886 0 --:--:-- --:--:-- --:--:-- 27521
  • Connection #0 to host 172.22.35.94 left intact {
    "entries": [ {
    "user": "f4e0b6c3eba8417d95781c1136265036$f4e0b6c3eba8417d95781c1136265036",
    "buckets": [ {
    "bucket": "test",
    "time": "2023-07-15T01:00:00.000000Z",
    "epoch": 1689382800,
    "owner": "f4e0b6c3eba8417d95781c1136265036$f4e0b6c3eba8417d95781c1136265036",
    "categories": [ {
    "category": "list_bucket",
    "bytes_sent": 2849,
    "bytes_received": 0,
    "ops": 22,
    "successful_ops": 22
    }
    ]
    }
    ]
    }
    ],
    "summary": [ {
    "user": "f4e0b6c3eba8417d95781c1136265036$f4e0b6c3eba8417d95781c1136265036",
    "categories": [ {
    "category": "list_bucket",
    "bytes_sent": 2849,
    "bytes_received": 0,
    "ops": 22,
    "successful_ops": 22
    }
    ],
    "total": {
    "bytes_sent": 2849,
    "bytes_received": 0,
    "ops": 22,
    "successful_ops": 22
    }
    }
    ]
    }
    ```

2H period

```
root@dev-ceph-y13004:~# bash test.sh GET usage "uid=f4e0b6c3eba8417d95781c1136265036%24f4e0b6c3eba8417d95781c1136265036&start=2023-07-15%2000:00:00&end=2023-07-15%2002:00:00" |jq .Note: Unnecessary use of -X or --request, GET is already inferred.
  • Trying :80...
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
    Connected to 172.22.35.94 (172.22.35.94) port 80 (#0)

GET /admin/usage?uid=f4e0b6c3eba8417d95781c1136265036%24f4e0b6c3eba8417d95781c1136265036&start=2023-07-15%2000:00:00&end=2023-07-15%2002:00:00 HTTP/1.1
Host: 172.22.35.94
User-Agent: curl/7.81.0
Accept: */*
Content-Type: application/x-compressed-tar
Date: Tue, 18 Jul 2023 00:58:48 +0000
Authorization: AWS admin:hSmo6Yo7FQd725WIpEqEKMhbt2s=

  • Mark bundle as not supporting multiuse
    < HTTP/1.1 200 OK
    < x-amz-request-id: tx000007419d662c82e60f3-0064b5e3c8-4ad68d-dev-c3j1
    < Content-Length: 982
    < Date: Tue, 18 Jul 2023 00:58:48 GMT
    < Connection: Keep-Alive
    < { [982 bytes data]
    100 982 100 982 0 0 12055 0 --:--:-- --:--:-- --:--:-- 12123
  • Connection #0 to host 172.22.35.94 left intact {
    "entries": [ {
    "user": "f4e0b6c3eba8417d95781c1136265036$f4e0b6c3eba8417d95781c1136265036",
    "buckets": [ {
    "bucket": "",
    "time": "2023-07-15T01:00:00.000000Z",
    "epoch": 1689382800,
    "owner": "f4e0b6c3eba8417d95781c1136265036$f4e0b6c3eba8417d95781c1136265036",
    "categories": [ {
    "category": "list_buckets",
    "bytes_sent": 1150,
    "bytes_received": 0,
    "ops": 23,
    "successful_ops": 23
    }
    ]
    }, {
    "bucket": "test",
    "time": "2023-07-15T00:00:00.000000Z",
    "epoch": 1689379200,
    "owner": "f4e0b6c3eba8417d95781c1136265036$f4e0b6c3eba8417d95781c1136265036",
    "categories": [ {
    "category": "list_bucket",
    "bytes_sent": 5439,
    "bytes_received": 0,
    "ops": 42,
    "successful_ops": 42
    }
    ]
    }
    ]
    }
    ],
    "summary": [ {
    "user": "f4e0b6c3eba8417d95781c1136265036$f4e0b6c3eba8417d95781c1136265036",
    "categories": [ {
    "category": "list_bucket",
    "bytes_sent": 5439,
    "bytes_received": 0,
    "ops": 42,
    "successful_ops": 42
    }, {
    "category": "list_buckets",
    "bytes_sent": 1150,
    "bytes_received": 0,
    "ops": 23,
    "successful_ops": 23
    }
    ],
    "total": {
    "bytes_sent": 6589,
    "bytes_received": 0,
    "ops": 65,
    "successful_ops": 65
    }
    }
    ]
    }
    ```
Actions #1

Updated by Casey Bodley 10 months ago

  • Tags set to admin usage
Actions #2

Updated by Shreyansh Sancheti 7 months ago

jielei zhou wrote:

when i use rgw usage to get user usage. I found if i request 1h info. There is some info lost in it.
like this. list_buckets field will not be display. But if i change period to 2H list_bucket field will be display.
example like this
2023-07-15 00:00:00~01:00:00 have not list_buckets field
2023-07-15 01:00:00~02:00:00 have not list_buckets field
2023-07-15 00:00:00~02:00:00 have list_buckets field

1H period by 00~01 hours
```
root@dev-ceph-y13004:~# bash test.sh GET usage "uid=f4e0b6c3eba8417d95781c1136265036%24f4e0b6c3eba8417d95781c1136265036&start=2023-07-15%2000:00:00&end=2023-07-15%2001:00:00" |jq .Note: Unnecessary use of -X or --request, GET is already inferred.
  • Trying ****:80...
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
    Connected to 172.22.35.94 (172.22.35.94) port 80 (#0)

GET /admin/usage?uid=f4e0b6c3eba8417d95781c1136265036%24f4e0b6c3eba8417d95781c1136265036&start=2023-07-15%2000:00:00&end=2023-07-15%2001:00:00 HTTP/1.1
Host:
User-Agent: curl/7.81.0
Accept: */

Content-Type: application/x-compressed-tar
Date: Tue, 18 Jul 2023 00:56:28 +0000
Authorization: AWS admin:Bhmf79WOa66PwheuNEeFWbIblpU=

  • Mark bundle as not supporting multiuse
    < HTTP/1.1 200 OK
    < x-amz-request-id: tx00000a3db35ea89f74a8c-0064b5e33c-4ad68d-dev-c3j1
    < Content-Length: 633
    < Date: Tue, 18 Jul 2023 00:56:28 GMT
    < Connection: Keep-Alive
    < { [633 bytes data]
    100 633 100 633 0 0 24641 0 --:--:-- --:--:-- --:--:-- 25320
  • Connection #0 to host * left intact {
    "entries": [ {
    "user": "f4e0b6c3eba8417d95781c1136265036$f4e0b6c3eba8417d95781c1136265036",
    "buckets": [ {
    "bucket": "test",
    "time": "2023-07-15T00:00:00.000000Z",
    "epoch": 1689379200,
    "owner": "f4e0b6c3eba8417d95781c1136265036$f4e0b6c3eba8417d95781c1136265036",
    "categories": [ {
    "category": "list_bucket",
    "bytes_sent": 2590,
    "bytes_received": 0,
    "ops": 20,
    "successful_ops": 20
    }
    ]
    }
    ]
    }
    ],
    "summary": [ {
    "user": "f4e0b6c3eba8417d95781c1136265036$f4e0b6c3eba8417d95781c1136265036",
    "categories": [ {
    "category": "list_bucket",
    "bytes_sent": 2590,
    "bytes_received": 0,
    "ops": 20,
    "successful_ops": 20
    }
    ],
    "total": {
    "bytes_sent": 2590,
    "bytes_received": 0,
    "ops": 20,
    "successful_ops": 20
    }
    }
    ]
    }
    ```
1H period by 01~02 hours
```
root@dev-ceph-y13004:~# bash test.sh GET usage "uid=f4e0b6c3eba8417d95781c1136265036%24f4e0b6c3eba8417d95781c1136265036&start=2023-07-15%2001:00:00&end=2023-07-15%2002:00:00" |jq .Note: Unnecessary use of -X or --request, GET is already inferred.
  • Trying :80...
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
    Connected to 172.22.35.94 (172.22.35.94) port 80 (#0)

GET /admin/usage?uid=f4e0b6c3eba8417d95781c1136265036%24f4e0b6c3eba8417d95781c1136265036&start=2023-07-15%2001:00:00&end=2023-07-15%2002:00:00 HTTP/1.1
Host:
User-Agent: curl/7.81.0
Accept: */

Content-Type: application/x-compressed-tar
Date: Tue, 18 Jul 2023 01:00:13 +0000
Authorization: AWS admin:wPJGOd/ELfnT4FgV7rZjm3rzndU=

  • Mark bundle as not supporting multiuse
    < HTTP/1.1 200 OK
    < x-amz-request-id: tx000007c53f9ba6912672e-0064b5e41d-4ad68d-dev-c3j1
    < Content-Length: 633
    < Date: Tue, 18 Jul 2023 01:00:13 GMT
    < Connection: Keep-Alive
    < { [633 bytes data]
    100 633 100 633 0 0 26886 0 --:--:-- --:--:-- --:--:-- 27521
  • Connection #0 to host 172.22.35.94 left intact {
    "entries": [ {
    "user": "f4e0b6c3eba8417d95781c1136265036$f4e0b6c3eba8417d95781c1136265036",
    "buckets": [ {
    "bucket": "test",
    "time": "2023-07-15T01:00:00.000000Z",
    "epoch": 1689382800,
    "owner": "f4e0b6c3eba8417d95781c1136265036$f4e0b6c3eba8417d95781c1136265036",
    "categories": [ {
    "category": "list_bucket",
    "bytes_sent": 2849,
    "bytes_received": 0,
    "ops": 22,
    "successful_ops": 22
    }
    ]
    }
    ]
    }
    ],
    "summary": [ {
    "user": "f4e0b6c3eba8417d95781c1136265036$f4e0b6c3eba8417d95781c1136265036",
    "categories": [ {
    "category": "list_bucket",
    "bytes_sent": 2849,
    "bytes_received": 0,
    "ops": 22,
    "successful_ops": 22
    }
    ],
    "total": {
    "bytes_sent": 2849,
    "bytes_received": 0,
    "ops": 22,
    "successful_ops": 22
    }
    }
    ]
    }
    ```

2H period

```
root@dev-ceph-y13004:~# bash test.sh GET usage "uid=f4e0b6c3eba8417d95781c1136265036%24f4e0b6c3eba8417d95781c1136265036&start=2023-07-15%2000:00:00&end=2023-07-15%2002:00:00" |jq .Note: Unnecessary use of -X or --request, GET is already inferred.
  • Trying :80...
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
    Connected to 172.22.35.94 (172.22.35.94) port 80 (#0)

GET /admin/usage?uid=f4e0b6c3eba8417d95781c1136265036%24f4e0b6c3eba8417d95781c1136265036&start=2023-07-15%2000:00:00&end=2023-07-15%2002:00:00 HTTP/1.1
Host: 172.22.35.94
User-Agent: curl/7.81.0
Accept: */*
Content-Type: application/x-compressed-tar
Date: Tue, 18 Jul 2023 00:58:48 +0000
Authorization: AWS admin:hSmo6Yo7FQd725WIpEqEKMhbt2s=

  • Mark bundle as not supporting multiuse
    < HTTP/1.1 200 OK
    < x-amz-request-id: tx000007419d662c82e60f3-0064b5e3c8-4ad68d-dev-c3j1
    < Content-Length: 982
    < Date: Tue, 18 Jul 2023 00:58:48 GMT
    < Connection: Keep-Alive
    < { [982 bytes data]
    100 982 100 982 0 0 12055 0 --:--:-- --:--:-- --:--:-- 12123
  • Connection #0 to host 172.22.35.94 left intact {
    "entries": [ {
    "user": "f4e0b6c3eba8417d95781c1136265036$f4e0b6c3eba8417d95781c1136265036",
    "buckets": [ {
    "bucket": "",
    "time": "2023-07-15T01:00:00.000000Z",
    "epoch": 1689382800,
    "owner": "f4e0b6c3eba8417d95781c1136265036$f4e0b6c3eba8417d95781c1136265036",
    "categories": [ {
    "category": "list_buckets",
    "bytes_sent": 1150,
    "bytes_received": 0,
    "ops": 23,
    "successful_ops": 23
    }
    ]
    }, {
    "bucket": "test",
    "time": "2023-07-15T00:00:00.000000Z",
    "epoch": 1689379200,
    "owner": "f4e0b6c3eba8417d95781c1136265036$f4e0b6c3eba8417d95781c1136265036",
    "categories": [ {
    "category": "list_bucket",
    "bytes_sent": 5439,
    "bytes_received": 0,
    "ops": 42,
    "successful_ops": 42
    }
    ]
    }
    ]
    }
    ],
    "summary": [ {
    "user": "f4e0b6c3eba8417d95781c1136265036$f4e0b6c3eba8417d95781c1136265036",
    "categories": [ {
    "category": "list_bucket",
    "bytes_sent": 5439,
    "bytes_received": 0,
    "ops": 42,
    "successful_ops": 42
    }, {
    "category": "list_buckets",
    "bytes_sent": 1150,
    "bytes_received": 0,
    "ops": 23,
    "successful_ops": 23
    }
    ],
    "total": {
    "bytes_sent": 6589,
    "bytes_received": 0,
    "ops": 65,
    "successful_ops": 65
    }
    }
    ]
    }
    ```

Need more info, I am unclear about the problem with this explanation. Also, please provide the `test.sh` script which you have mentioned in the description.

Actions

Also available in: Atom PDF