Project

General

Profile

Actions

Feature #63563

open

rgw: add s3select bytes processed and bytes returned to usage

Added by Seena Fallah 5 months ago. Updated 2 months ago.

Status:
Pending Backport
Priority:
Normal
Assignee:
Target version:
-
% Done:

0%

Source:
Tags:
s3select backport_processed
Backport:
quincy reef squid
Reviewed:
Affected Versions:
Pull request ID:

Description

Expose bytes processed and bytes returned for s3select requests to log usage.


Related issues 3 (3 open0 closed)

Copied to rgw - Backport #64497: reef: rgw: add s3select bytes processed and bytes returned to usageNewActions
Copied to rgw - Backport #64498: quincy: rgw: add s3select bytes processed and bytes returned to usageNewActions
Copied to rgw - Backport #64499: squid: rgw: add s3select bytes processed and bytes returned to usageNewActions
Actions #2

Updated by Casey Bodley 5 months ago

  • Status changed from New to In Progress
  • Tags set to s3select
  • Pull request ID set to 54554
Actions #3

Updated by Gal Salomon 3 months ago

Hi

can you explain what it should count? for what purpose?

the s3select request is a get-request with additional processing (SQL computation)
upon TXT objects such as CSV & JSON, it scanned the whole objects (thus processed-bytes = object-size, like a get-request)

as for the return bytes, it depends on the SQL result.

it must be noted that RGW/s3select contains counters, and it returns them to the client side(boto3 client).
the `aws_response_handler` object responsible for that.

as for Paequet, it is quite different, the s3select engine is processing only parts of the object according to the SQL statement.
thus the processed-bytes is something the engine "knows" (in practice, it is an s3-get-range-request)

Actions #4

Updated by Seena Fallah 3 months ago

Regarding the AWS docs (https://aws.amazon.com/s3/pricing/ - tab "Transform & query") there needs to be an option to know the total "Data scanned" and "Data returned" per bucket and user for billing purposes (the rgw usage log purpose basically).
So regardless of the response body, we need to count them in the cluster for later usage reports.

Actions #5

Updated by Casey Bodley 2 months ago

  • Status changed from In Progress to Pending Backport
  • Assignee set to Seena Fallah
  • Backport changed from reef, quincy to quincy reef squid
Actions #6

Updated by Backport Bot 2 months ago

  • Copied to Backport #64497: reef: rgw: add s3select bytes processed and bytes returned to usage added
Actions #7

Updated by Backport Bot 2 months ago

  • Copied to Backport #64498: quincy: rgw: add s3select bytes processed and bytes returned to usage added
Actions #8

Updated by Backport Bot 2 months ago

  • Copied to Backport #64499: squid: rgw: add s3select bytes processed and bytes returned to usage added
Actions #9

Updated by Backport Bot 2 months ago

  • Tags changed from s3select to s3select backport_processed
Actions

Also available in: Atom PDF