Project

General

Profile

Actions
Copy link

Bug #44508

closed

RGW prefetches data for range requests

Added by Or Friedmann about 4 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Target version:
Ceph - v15.0.0
% Done:

0%

Source:
Tags:
Backport:
nautilus, octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v15.0.0
ceph-qa-suite:
rgw
Pull request ID:
33807
Crash signature (v1):
Crash signature (v2):

Description

The problem:

When user is sending GET request with range header bytes=0-1 for example the RGW is trying to fetch the all head object (usually it is 4M).
Reproducer

dd if=/dev/zero of=/tmp/zeros count=1 bs=4M
aws s3 cp /tmp/zeros s3://mybucket --endpoint-url http://rgw_url
aws s3api get-object --bucket mybucket --key zeros --range bytes=0-1 --endpoint-url http://rgw_url output_filename
The RGW logs showing that it reads 4M and not 2 bytes:

2020-03-08T14:25:45.387+0200 7f0088f09700 1 -- 10.46.10.199:0/325646556 --> [v2:10.46.10.199:6800/31564,v1:10.46.10.199:6801/31564] -- osd_op(unknown.0.0:12051 6.4 6:232cb0ff:::f2816f20-5421-4ade-a31f-dde020a68a0c.4157.1_zeros:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e644) v8 -- 0x7f031401a690 con 0x2355340
2020-03-08T14:25:45.470+0200 7f032ffff700 1 -- 10.46.10.199:0/325646556 <== osd.0 v2:10.46.10.199:6800/31564 12052 ==== osd_op_reply(12051 f2816f20-5421-4ade-a31f-dde020a68a0c.4157.1_zeros [getxattrs out=902b,stat out=16b,read 0~4194304 out=4194304b]

Related issues 2 (0 open — 2 closed)

Copied to rgw - Backport #45052: octopus: RGW prefetches data for range requestsResolvedNathan CutlerActions
Copied to rgw - Backport #48576: nautilus: RGW prefetches data for range requestsResolvedNathan CutlerActions
Actions
Copy link
 #1

Updated by Mark Kogan about 4 years ago

  • Pull request ID set to 33807
Actions
Copy link
 #2

Updated by Mark Kogan about 4 years ago

Testing results of reproducer on PR https://github.com/ceph/ceph/pull/33807

at peak load of 863 simul;tanious connections:

./bin/ceph --admin-daemon ./out/radosgw.8000.asok perf dump | jq -C '.rgw' | rg 'qlen|qactive'

*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***                                                                              
  "qlen": 863,                                                                                                                                    
  "qactive": 863,

Before the fix PR bandwidth from dstat --bits 16Gbit at peak

----total-usage---- -dsk/total- -net/ens4f0----net/lo----net/total- ---paging-- ---system--
usr sys idl wai stl| read  writ| recv  send: recv  send: recv  send|  in   out | int   csw 
 48  12  40   0   0|   0  1084M|3584b   17k:  15G   15G|   0     0 |  60k  115k
 57  13  29   0   0|   0  1031M|2639b   17k:  13G   13G|   0     0 |  60k  108k
 73  14  12   0   0|   0  1069M|2097b   14k:  13G   13G|   0     0 |  63k   84k
 85  15   1   0   0|   0   992M|2112b   12k:  11G   11G|   0     0 |  67k   83k
 77  16   8   0   0|   0  2156M|3168b   16k:  12G   12G|   0     0 |  62k   59k
 54  14  32   0   0|   0  1586M|3104b   17k:  14G   14G|   0     0 |  57k   84k
 63  14  23   0   0|   0  1099M|1584b   14k:  14G   14G|   0     0 |  60k   84k
 71  15  14   0   0|   0  2164M|4210b   15k:  13G   13G|   0     0 |  62k   73k
 60  13  26   0   0|   0  1615M|1583b   14k:  14G   14G|   0     0 |  60k   76k
 83  16   1   0   0|   0  3212M|2624b   16k:  11G   11G|   0     0 |  66k   54k
 56  13  30   0   0|   0  2457M|2640b   20k:  13G   13G|   0     0 |  60k   83k
 16   8  76   0   0|   0   352M|2097b   15k:  16G   16G|   0     0 |  49k  101k
 19   7  73   0   0|   0  3783k|2111b   18k:  16G   16G|   0     0 |  50k  101k

After the fix PR bandwidth in 4 Gbit at peak

----total-usage---- -dsk/total- -net/ens4f0----net/lo----net/total- ---paging-- ---system--
usr sys idl wai stl| read  writ| recv  send: recv  send: recv  send|  in   out | int   csw 
 88  12   0   0   0|   0    46M|2639b   21k:4292M 4292M|   0     0 |  40k   45k
 87  13   0   0   0|   0   174M|2110b   29k:4138M 4138M|   0     0 |  40k   45k
 88  12   0   0   0|   0   284M|1586b   19k:3909M 3909M|   0     0 |  39k   44k
 87  13   0   0   0|   0   389M|3072b   21k:4241M 4241M|   0     0 |  42k   48k
 87  12   0   0   0|   0   479M|1584b   20k:4208M 4208M|   0     0 |  41k   45k
 88  12   0   0   0|   0   651M|3168b   27k:3085M 3085M|   0     0 |  39k   42k
 88  12   0   0   0|   0   704M|  18k   27k:4006M 4006M|   0     0 |  41k   46k            
 87  13   0   0   0|   0  1022M|1584b   23k:3270M 3270M|   0     0 |  41k   44k

Actions
Copy link
 #3

Updated by Mark Kogan about 4 years ago

  • Status changed from New to Triaged
Actions
Copy link
 #4

Updated by Casey Bodley about 4 years ago

  • Status changed from Triaged to In Progress
  • Assignee set to Marcus Watts
  • Backport set to octopus
Actions
Copy link
 #5

Updated by Casey Bodley about 4 years ago

  • Status changed from In Progress to Pending Backport
  • Assignee deleted (Marcus Watts)
Actions
Copy link
 #6

Updated by Nathan Cutler about 4 years ago

  • Copied to Backport #45052: octopus: RGW prefetches data for range requests added
Actions
Copy link
 #7

Updated by Nathan Cutler almost 4 years ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Actions
Copy link
 #8

Updated by Dan Hill over 3 years ago

  • Backport changed from octopus to nautilus, octopus

This issue is also present in nautilus. I've added it to the backport list for consideration.

Actions
Copy link
 #9

Updated by Dan Hill over 3 years ago

  • Status changed from Resolved to Pending Backport
Actions
Copy link
 #10

Updated by Backport Bot over 3 years ago

  • Copied to Backport #48576: nautilus: RGW prefetches data for range requests added
Actions
Copy link
 #11

Updated by Nathan Cutler over 3 years ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Actions
Copy link

Also available in: Atom PDF