Project

General

Profile

Bug #23489

[rgw] civetweb behind haproxy doesn't work with absolute URI

Added by Aleksandr Rudenko over 1 year ago. Updated 6 months ago.

Status:
In Progress
Priority:
Normal
Target version:
Start date:
03/28/2018
Due date:
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

I'm sorry, maybe it isn't bug, but i don't know how to solve this problem.

I know that absolute URIs are supported in civetweb and it works fine for me without haproxy in the middle.

But if client send absolute URIs through reverse proxy(haproxy) to civetweb, civetweb breaks connection without responce.

i set:

debug rgw = 20
debug civetweb = 10

but no any messgaes in civetweb logs(access, error) and in rgw logs.
in tcpdump i only see as rgw closes connection after request with absolute URI. Relative URIs in requests work fine with haproxy.

Client:
Docker registry v2.6.2, s3 driver based on aws-sdk-go/1.2.4 (go1.7.6; linux; amd64) uses absolute URI in requests.

s3 driver options of docker registry:

  s3:
    region: us-east-1
    bucket: docker
    accesskey: 'access_key'
    secretkey: 'secret_key'
    regionendpoint: http://storage.my-domain.ru
    secure: false
    v4auth: true

ceph.conf for rgw instance:

[client]
    rgw dns name = storage.my-domain.ru
    rgw enable apis = s3, admin
    rgw dynamic resharding = false
    rgw enable usage log = true
    rgw num rados handles = 8
    rgw thread pool size = 256

[client.rgw.a]
    host = aj15
    keyring = /var/lib/ceph/radosgw/rgw.a.keyring
    rgw enable static website = true
    rgw frontends = civetweb authentication_domain=storage.my-domain.ru num_threads=128 port=0.0.0.0:7480 access_log_file=/var/log/ceph/civetweb.rgw.access.log error_log_file=/var/log/ceph/civetweb.rgw.error.log
    debug rgw = 20
    debug civetweb = 10

very simple haproxy.cfg:

global
    chroot /var/empty
    # /log is chroot path
    log /haproxy-log local2

    pidfile /var/run/haproxy.pid

    user haproxy
    group haproxy
    daemon

    ssl-default-bind-options no-sslv3
    ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
    ssl-dh-param-file /etc/pki/tls/dhparams.pem

defaults
    mode http
    log global

frontend s3

    bind *:80
    bind *:443 ssl crt /etc/pki/tls/certs/s3.pem crt /etc/pki/tls/certs/s3-buckets.pem

    use_backend rgw

backend rgw

    balance roundrobin

    server a aj15:7480 check fall 1
    server a aj16:7480 check fall 1

http haeder from tcpdump before and after haproxy:

GET http://storage.my-domain.ru/docker?max-keys=1&prefix= HTTP/1.1
Host: storage.my-domain.ru
User-Agent: aws-sdk-go/1.2.4 (go1.7.6; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=user:/20180328/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=10043867bbb2833d50f9fe16a6991436a5c328adc5042556ce1ddf1101ee2cb9
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20180328T111255Z
Accept-Encoding: gzip

i don't understand how use haproxy and absolute URIs in requests(

History

#1 Updated by Abhishek Lekshmanan over 1 year ago

What happens when authentication_domain is not set in civetweb options? Since we disable authentication domain check in civetweb by default, passing this option may cause different behaviours

#2 Updated by Abhishek Lekshmanan over 1 year ago

  • Status changed from New to Need More Info

#3 Updated by Abhishek Lekshmanan over 1 year ago

  • Assignee set to Abhishek Lekshmanan

#4 Updated by Aleksandr Rudenko over 1 year ago

I try without "authentication_domain=" but it's not helped.
I try without "rgw dns name"(rgw option) and the same.
I try many parameters but had no success.
Civetweb or rgw(i don't know) dropped tcp connection and i don't have any logs for debug(.

#5 Updated by Abhishek Lekshmanan over 1 year ago

  • Status changed from Need More Info to In Progress

#6 Updated by Abhishek Lekshmanan over 1 year ago

When using absolute urls, following things need to be in order:

rgw_dns_name must correspond to the host part of url the client is requesting... so in this case the url used in the clinet storage.my-domain.ru must correspond to rgw dns name and this request much reach rgw as such. What happens without the haproxy? are the requests being honoured correctly?

#7 Updated by Aleksandr Rudenko over 1 year ago

Yes, you can see in issue description that rgw_dns_name corresponded to the host part of client url.

It's easy to reproduce. Steps:

# docker pull registry
cat /root/config.yaml
version: 0.1
log:
  fields:
    service: registry
storage:
  cache:
    blobdescriptor: inmemory
  s3:
    region: us-east-1
    bucket: docker
    accesskey: 'access_key'
    secretkey: 'secret_key'
    regionendpoint: http://your-s3-endpoint
    secure: false
    v4auth: true
http:
    addr: :5000
    headers:
        X-Content-Type-Options: [nosniff]
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3
# docker run --name registry --rm --network host -e DEBUG=true -v /root/config.yaml:/etc/docker/registry/config.yml registry:latest

in my custom haproxy logs i see:

haproxy[3743089]: xxx.xxx.xxx.xxx:58129 [31/May/2018:10:21:31.599] s3 rgw/aj15 0/0/0/-1/0 502 204 - - SH-- 1/1/0/0/0 0/0 {|http|storage.my-domain.ru||aws-sdk-go/1.2.4 (go1.7.6; linux; amd64)} "GET http://storage.my-domain.ru/docker?max-keys=1&prefix= HTTP/1.1" 

SH in haproxy log mean:

     SH   The server aborted before sending its full HTTP response headers, or
          it crashed while processing the request. Since a server aborting at
          this moment is very rare, it would be wise to inspect its logs to
          control whether it crashed and why. The logged request may indicate a
          small set of faulty requests, demonstrating bugs in the application.
          Sometimes this might also be caused by an IDS killing the connection
          between haproxy and the server.

it's realy true, i see full http header sended to rgw:

GET http://storage.my-domain.ru/docker?max-keys=1&prefix= HTTP/1.1
Host: storage.my-domain.ru
User-Agent: aws-sdk-go/1.2.4 (go1.7.6; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=user:user@cloud.croc.ru/20180531/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=df96ad46ea127f3b4e0815c92b9f865c910aff6b0bb844a9225b5fc7be4036bb
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20180531T071711Z
Accept-Encoding: gzip
X-Forwarded-Proto: http
X-Forwarded-For: xxx.xxx.xxx.xxx

and i see that rgw drop the connection without any logs.
As you can see header is correct.

#8 Updated by Dieter Roels about 1 year ago

should be fixed by this PR: https://github.com/ceph/civetweb/pull/27

#9 Updated by Dieter Roels about 1 year ago

Sorry, I was under the impression this was on mimic but its luminous. AbsoluteURI and thus docker registry works on luminous starting from 12.2.2 up to at least 12.2.7 as thats what we use our ceph clusters for. Civetweb 1.10 broke this in 13.2, but there is a PR that will fix this, see https://tracker.ceph.com/issues/27048

#10 Updated by Abhishek Lekshmanan 6 months ago

Yeah absolute urls should be fixed in 13.2.5

Also available in: Atom PDF