Project

General

Profile

Actions

Bug #54607

open

Rate limits not working on tenanted users

Added by Tejas C about 2 years ago. Updated about 1 year ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
low-hanging-fruit ratelimit
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

I have a tenanted users having rate limits set

/]# radosgw-admin ratelimit get --ratelimit-scope user --uid ten1/user1 {
"user_ratelimit": {
"max_read_ops": 5,
"max_write_ops": 5,
"max_read_bytes": 102400,
"max_write_bytes": 10240,
"enabled": true
}
}

  1. /usr/local/bin/s3cmd put 110m s3://ten1/110
    upload: '110m' -> 's3://ten1/110' [part 1 of 8, 15MB] [1 of 1]
    15728640 of 15728640 100% in 0s 30.75 MB/s done

]# /usr/local/bin/s3cmd put 110m s3://ten1/111
upload: '110m' -> 's3://ten1/111' [part 1 of 8, 15MB] [1 of 1]
15728640 of 15728640 100% in 0s 44.45 MB/s done
It is allowing consecutive 110M writes

Same with get , consecutive 110M read operations are being allowed.

  1. /usr/local/bin/s3cmd get s3://ten1/110 --force
    download: 's3://ten1/110' -> './110' [1 of 1]
    115343360 of 115343360 100% in 0s 232.92 MB/s done

]# /usr/local/bin/s3cmd get s3://ten1/110 --force
download: 's3://ten1/110' -> './110' [1 of 1]
115343360 of 115343360 100% in 0s 263.33 MB/s done

Actions #1

Updated by Casey Bodley about 2 years ago

  • Assignee set to Or Friedmann
Actions #2

Updated by Casey Bodley almost 2 years ago

bump

Actions #3

Updated by Casey Bodley about 1 year ago

  • Assignee deleted (Or Friedmann)
  • Tags set to low-hanging-fruit ratelimit
Actions #4

Updated by Yuval Lifshitz about 1 year ago

Tejas,
i have an issue with the way you access the teneted user.
when you create a rate limiter for a user:

bin/radosgw-admin ratelimit set --ratelimit-scope user --uid user1 --tenant ten1 --max-read-bytes 100000 --max-write-bytes 100000 --max-read-ops 5 --max-write-ops 5

you cannot access it using the following syntax (with slash):
bin/radosgw-admin ratelimit get --uid ten1/user1 --ratelimit-scope=user

you have to use:
bin/radosgw-admin ratelimit get --uid user1 --tenant ten1 --ratelimit-scope=user

Actions #5

Updated by Tejas C about 1 year ago

hi Yuval,
Acknowledge that the syntax of the get command is wrong.
I tried the scenario again:

radosgw-admin ratelimit set --ratelimit-scope user --uid user1 --tenant ten1 --max-read-bytes 100 --max-write-bytes 100 --max-read-ops 2 --max-write-ops 2

]# venv/bin/s3cmd put io_info_test_bucket_stats.yaml s3://test1/f1
upload: 'io_info_test_bucket_stats.yaml' -> 's3://test1/f1' [1 of 1]
149 of 149 100% in 0s 5.73 KB/s done
[root@ceph-tj1-ujn2rl-node6 cephuser]# venv/bin/s3cmd put io_info_test_bucket_stats.yaml s3://test1/f12
upload: 'io_info_test_bucket_stats.yaml' -> 's3://test1/f12' [1 of 1]
149 of 149 100% in 0s 9.36 KB/s done
[root@ceph-tj1-ujn2rl-node6 cephuser]# venv/bin/s3cmd put io_info_test_bucket_stats.yaml s3://test1/f13
upload: 'io_info_test_bucket_stats.yaml' -> 's3://test1/f13' [1 of 1]
149 of 149 100% in 0s 7.09 KB/s done
[root@ceph-tj1-ujn2rl-node6 cephuser]# venv/bin/s3cmd put io_info_test_bucket_stats.yaml s3://test1/f14
upload: 'io_info_test_bucket_stats.yaml' -> 's3://test1/f14' [1 of 1]
149 of 149 100% in 0s 7.37 KB/s done
[root@ceph-tj1-ujn2rl-node6 cephuser]# venv/bin/s3cmd put io_info_test_bucket_stats.yaml s3://test1/f15
upload: 'io_info_test_bucket_stats.yaml' -> 's3://test1/f15' [1 of 1]
149 of 149 100% in 0s 7.45 KB/s done

The rate limit of 2 write ops is not getting enforced.

Actions #6

Updated by Yuval Lifshitz about 1 year ago

can you please show the output of the "get" command, to verify that the limiter is enabled?

Actions #7

Updated by Tejas C about 1 year ago

]# radosgw-admin ratelimit get --uid user1 --tenant ten1 --ratelimit-scope user {
"user_ratelimit": {
"max_read_ops": 2,
"max_write_ops": 2,
"max_read_bytes": 100,
"max_write_bytes": 100,
"enabled": false
}
}

Actions #8

Updated by Tejas C about 1 year ago

radosgw-admin ratelimit enable --uid user1 --tenant ten1 --ratelimit-scope user

]# radosgw-admin ratelimit get --uid user1 --tenant ten1 --ratelimit-scope user {
"user_ratelimit": {
"max_read_ops": 2,
"max_write_ops": 2,
"max_read_bytes": 100,
"max_write_bytes": 100,
"enabled": true
}
}

root@ceph-tj1-ujn2rl-node6 cephuser]# venv/bin/s3cmd put io_info_test_bucket_stats.yaml s3://test1/f15
upload: 'io_info_test_bucket_stats.yaml' -> 's3://test1/f15' [1 of 1]
149 of 149 100% in 0s 3.91 KB/s done
[root@ceph-tj1-ujn2rl-node6 cephuser]# venv/bin/s3cmd put io_info_test_bucket_stats.yaml s3://test1/f16
upload: 'io_info_test_bucket_stats.yaml' -> 's3://test1/f16' [1 of 1]
149 of 149 100% in 0s 23.14 KB/s done
WARNING: Upload failed: /f16 (503 (SlowDown))
WARNING: Retrying on lower speed (throttle=0.01)
WARNING: Waiting 3 sec...
^CSee ya!

It was not enabled , after enabling seems to be working .

Actions #9

Updated by Tejas C about 1 year ago

this format of get command is not working now :
/]# radosgw-admin ratelimit get --ratelimit-scope user --uid ten1/user1 {
"user_ratelimit": {
"max_read_ops": 5,
"max_write_ops": 5,
"max_read_bytes": 102400,
"max_write_bytes": 10240,
"enabled": true
}
}

Actions #10

Updated by Tejas C about 1 year ago

however from my automation script executed from a client , this format says the limits are enabled , but not enforced:

2023-03-20 07:40:17,430 INFO: executing cmd: radosgw-admin ratelimit set --ratelimit-scope=user --uid=tenant1$marjoriey.470 --max-read-ops=2 --max-read-bytes=4096 --max-write-bytes=4096 --max-write-ops=2
2023-03-20 07:40:17,672 INFO: cmd excuted
2023-03-20 07:40:17,672 INFO:
2023-03-20 07:40:17,672 INFO: Rate limits set on user tenant1$marjoriey.470
2023-03-20 07:40:17,672 INFO: executing cmd: radosgw-admin ratelimit enable --ratelimit-scope=user --uid=tenant1$marjoriey.470
2023-03-20 07:40:17,907 INFO: cmd excuted
2023-03-20 07:40:17,907 INFO:
2023-03-20 07:40:17,907 INFO: executing cmd: radosgw-admin ratelimit get --ratelimit-scope=user --uid=tenant1$marjoriey.470
2023-03-20 07:40:18,133 INFO: cmd excuted
2023-03-20 07:40:18,133 INFO: {
"user_ratelimit": {
"max_read_ops": 2,
"max_write_ops": 2,
"max_read_bytes": 4096,
"max_write_bytes": 4096,
"enabled": true
}
}

2023-03-20 07:40:18,134 INFO: Rate limits enabled on bucket : {
"user_ratelimit": {
"max_read_ops": 2,
"max_write_ops": 2,
"max_read_bytes": 4096,
"max_write_bytes": 4096,
"enabled": true
}
}

Actions #11

Updated by Ali Masarwa about 1 year ago

I think that the format is still wrong, if you don't want to use the --tenant parameter then you need to like this

radosgw-admin ratelimit set --ratelimit-scope=user --uid='tenant1$marjoriey.470' --max-read-ops=2 --max-read-bytes=4096 --max-write-bytes=4096 --max-write-ops=2

--uid='TENANT$USERID'

Actions

Also available in: Atom PDF