Bug #54607
openRate limits not working on tenanted users
0%
Description
I have a tenanted users having rate limits set
/]# radosgw-admin ratelimit get --ratelimit-scope user --uid ten1/user1
{
"user_ratelimit": {
"max_read_ops": 5,
"max_write_ops": 5,
"max_read_bytes": 102400,
"max_write_bytes": 10240,
"enabled": true
}
}
- /usr/local/bin/s3cmd put 110m s3://ten1/110
upload: '110m' -> 's3://ten1/110' [part 1 of 8, 15MB] [1 of 1]
15728640 of 15728640 100% in 0s 30.75 MB/s done
]# /usr/local/bin/s3cmd put 110m s3://ten1/111
upload: '110m' -> 's3://ten1/111' [part 1 of 8, 15MB] [1 of 1]
15728640 of 15728640 100% in 0s 44.45 MB/s done
It is allowing consecutive 110M writes
Same with get , consecutive 110M read operations are being allowed.
- /usr/local/bin/s3cmd get s3://ten1/110 --force
download: 's3://ten1/110' -> './110' [1 of 1]
115343360 of 115343360 100% in 0s 232.92 MB/s done
]# /usr/local/bin/s3cmd get s3://ten1/110 --force
download: 's3://ten1/110' -> './110' [1 of 1]
115343360 of 115343360 100% in 0s 263.33 MB/s done
Updated by Casey Bodley about 1 year ago
- Assignee deleted (
Or Friedmann) - Tags set to low-hanging-fruit ratelimit
Updated by Yuval Lifshitz about 1 year ago
Tejas,
i have an issue with the way you access the teneted user.
when you create a rate limiter for a user:
bin/radosgw-admin ratelimit set --ratelimit-scope user --uid user1 --tenant ten1 --max-read-bytes 100000 --max-write-bytes 100000 --max-read-ops 5 --max-write-ops 5
you cannot access it using the following syntax (with slash):
bin/radosgw-admin ratelimit get --uid ten1/user1 --ratelimit-scope=user
you have to use:
bin/radosgw-admin ratelimit get --uid user1 --tenant ten1 --ratelimit-scope=user
Updated by Tejas C about 1 year ago
hi Yuval,
Acknowledge that the syntax of the get command is wrong.
I tried the scenario again:
radosgw-admin ratelimit set --ratelimit-scope user --uid user1 --tenant ten1 --max-read-bytes 100 --max-write-bytes 100 --max-read-ops 2 --max-write-ops 2
]# venv/bin/s3cmd put io_info_test_bucket_stats.yaml s3://test1/f1
upload: 'io_info_test_bucket_stats.yaml' -> 's3://test1/f1' [1 of 1]
149 of 149 100% in 0s 5.73 KB/s done
[root@ceph-tj1-ujn2rl-node6 cephuser]# venv/bin/s3cmd put io_info_test_bucket_stats.yaml s3://test1/f12
upload: 'io_info_test_bucket_stats.yaml' -> 's3://test1/f12' [1 of 1]
149 of 149 100% in 0s 9.36 KB/s done
[root@ceph-tj1-ujn2rl-node6 cephuser]# venv/bin/s3cmd put io_info_test_bucket_stats.yaml s3://test1/f13
upload: 'io_info_test_bucket_stats.yaml' -> 's3://test1/f13' [1 of 1]
149 of 149 100% in 0s 7.09 KB/s done
[root@ceph-tj1-ujn2rl-node6 cephuser]# venv/bin/s3cmd put io_info_test_bucket_stats.yaml s3://test1/f14
upload: 'io_info_test_bucket_stats.yaml' -> 's3://test1/f14' [1 of 1]
149 of 149 100% in 0s 7.37 KB/s done
[root@ceph-tj1-ujn2rl-node6 cephuser]# venv/bin/s3cmd put io_info_test_bucket_stats.yaml s3://test1/f15
upload: 'io_info_test_bucket_stats.yaml' -> 's3://test1/f15' [1 of 1]
149 of 149 100% in 0s 7.45 KB/s done
The rate limit of 2 write ops is not getting enforced.
Updated by Yuval Lifshitz about 1 year ago
can you please show the output of the "get" command, to verify that the limiter is enabled?
Updated by Tejas C about 1 year ago
]# radosgw-admin ratelimit get --uid user1 --tenant ten1 --ratelimit-scope user
{
"user_ratelimit": {
"max_read_ops": 2,
"max_write_ops": 2,
"max_read_bytes": 100,
"max_write_bytes": 100,
"enabled": false
}
}
Updated by Tejas C about 1 year ago
radosgw-admin ratelimit enable --uid user1 --tenant ten1 --ratelimit-scope user
]# radosgw-admin ratelimit get --uid user1 --tenant ten1 --ratelimit-scope user
{
"user_ratelimit": {
"max_read_ops": 2,
"max_write_ops": 2,
"max_read_bytes": 100,
"max_write_bytes": 100,
"enabled": true
}
}
root@ceph-tj1-ujn2rl-node6 cephuser]# venv/bin/s3cmd put io_info_test_bucket_stats.yaml s3://test1/f15
upload: 'io_info_test_bucket_stats.yaml' -> 's3://test1/f15' [1 of 1]
149 of 149 100% in 0s 3.91 KB/s done
[root@ceph-tj1-ujn2rl-node6 cephuser]# venv/bin/s3cmd put io_info_test_bucket_stats.yaml s3://test1/f16
upload: 'io_info_test_bucket_stats.yaml' -> 's3://test1/f16' [1 of 1]
149 of 149 100% in 0s 23.14 KB/s done
WARNING: Upload failed: /f16 (503 (SlowDown))
WARNING: Retrying on lower speed (throttle=0.01)
WARNING: Waiting 3 sec...
^CSee ya!
It was not enabled , after enabling seems to be working .
Updated by Tejas C about 1 year ago
this format of get command is not working now :
/]# radosgw-admin ratelimit get --ratelimit-scope user --uid ten1/user1 {
"user_ratelimit": {
"max_read_ops": 5,
"max_write_ops": 5,
"max_read_bytes": 102400,
"max_write_bytes": 10240,
"enabled": true
}
}
Updated by Tejas C about 1 year ago
however from my automation script executed from a client , this format says the limits are enabled , but not enforced:
2023-03-20 07:40:17,430 INFO: executing cmd: radosgw-admin ratelimit set --ratelimit-scope=user --uid=tenant1$marjoriey.470 --max-read-ops=2 --max-read-bytes=4096 --max-write-bytes=4096 --max-write-ops=2
2023-03-20 07:40:17,672 INFO: cmd excuted
2023-03-20 07:40:17,672 INFO:
2023-03-20 07:40:17,672 INFO: Rate limits set on user tenant1$marjoriey.470
2023-03-20 07:40:17,672 INFO: executing cmd: radosgw-admin ratelimit enable --ratelimit-scope=user --uid=tenant1$marjoriey.470
2023-03-20 07:40:17,907 INFO: cmd excuted
2023-03-20 07:40:17,907 INFO:
2023-03-20 07:40:17,907 INFO: executing cmd: radosgw-admin ratelimit get --ratelimit-scope=user --uid=tenant1$marjoriey.470
2023-03-20 07:40:18,133 INFO: cmd excuted
2023-03-20 07:40:18,133 INFO: {
"user_ratelimit": {
"max_read_ops": 2,
"max_write_ops": 2,
"max_read_bytes": 4096,
"max_write_bytes": 4096,
"enabled": true
}
}
2023-03-20 07:40:18,134 INFO: Rate limits enabled on bucket : {
"user_ratelimit": {
"max_read_ops": 2,
"max_write_ops": 2,
"max_read_bytes": 4096,
"max_write_bytes": 4096,
"enabled": true
}
}
Updated by Ali Masarwa about 1 year ago
I think that the format is still wrong, if you don't want to use the --tenant parameter then you need to like this
radosgw-admin ratelimit set --ratelimit-scope=user --uid='tenant1$marjoriey.470' --max-read-ops=2 --max-read-bytes=4096 --max-write-bytes=4096 --max-write-ops=2
--uid='TENANT$USERID'