Project

General

Profile

Actions
Copy link

Bug #7796

closed

RGW Keystone token auth fails with '411 Length Required' when Keystone using Apache/WSGI

Added by Michael Kidd about 10 years ago. Updated over 9 years ago.

Status:
Resolved
Priority:
High
Assignee:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
firefly, giant
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

When using keystone behind apache/wsgi in the following config, radosgw is unable to authenticate tokens:

cat /etc/apache2/site-enabled/keystone.conf

<VirtualHost *:5000>
    # WSGIDaemonProcess keystone-main user=keystone group=keystone processes=$(facter processorcount) threads=10
    WSGIDaemonProcess keystone-main user=keystone group=keystone processes=10 threads=10
    WSGIScriptAlias / /var/www/cgi-bin/keystone/main
    # WSGIScriptAlias /main /var/www/cgi-bin/keystone/main

    <Directory /var/www/cgi-bin/keystone>
        WSGIProcessGroup keystone-main
        WSGIApplicationGroup %{GLOBAL}
        Order deny,allow
        Allow from all
    </Directory>
</VirtualHost>

<VirtualHost *:35357>
    # WSGIDaemonProcess keystone-admin user=keystone group=keystone processes=$(facter processorcount) threads=10
    WSGIDaemonProcess keystone-admin user=keystone group=keystone processes=10 threads=10
    WSGIScriptAlias / /var/www/cgi-bin/keystone/admin
    # WSGIScriptAlias /admin /var/www/cgi-bin/keystone/admin

    <Directory /var/www/cgi-bin/keystone>
        WSGIProcessGroup keystone-admin
        WSGIApplicationGroup %{GLOBAL}
        Order deny,allow
        Allow from all
    </Directory>
</VirtualHost>

debug rgw=20 logs:

2014-03-17 17:53:06.754621 7f05537fe700 20 RGWWQ: empty
2014-03-17 17:53:06.754638 7f05537fe700  1 ====== starting new request req=0x1c90660 =====
2014-03-17 17:53:06.754746 7f05537fe700  2 req 1:0.000107::HEAD /swift/v1/test::initializing
2014-03-17 17:53:06.754842 7f05537fe700 10 ver=v1 first=test req=
2014-03-17 17:53:06.754863 7f05537fe700 10 s->object=<NULL> s->bucket=test
2014-03-17 17:53:06.754880 7f05537fe700 20 FCGI_ROLE=RESPONDER
2014-03-17 17:53:06.754881 7f05537fe700 20 SCRIPT_URL=/swift/v1/test
2014-03-17 17:53:06.754882 7f05537fe700 20 SCRIPT_URI=http://us-dfw-storage-1.cisco.com/swift/v1/test
2014-03-17 17:53:06.754884 7f05537fe700 20 HTTP_AUTHORIZATION=
2014-03-17 17:53:06.754884 7f05537fe700 20 HTTP_HOST=us-dfw-storage-1.cisco.com
2014-03-17 17:53:06.754885 7f05537fe700 20 HTTP_ACCEPT_ENCODING=identity
2014-03-17 17:53:06.754886 7f05537fe700 20 HTTP_X_AUTH_TOKEN=6c87a7d2351e4003a0fe7997430458b4
2014-03-17 17:53:06.754887 7f05537fe700 20 PATH=/usr/local/bin:/usr/bin:/bin
2014-03-17 17:53:06.754888 7f05537fe700 20 SERVER_SIGNATURE=
2014-03-17 17:53:06.754889 7f05537fe700 20 SERVER_SOFTWARE=Apache/2.2.22 (Ubuntu)
2014-03-17 17:53:06.754895 7f05537fe700 20 SERVER_NAME=us-dfw-storage-1.cisco.com
2014-03-17 17:53:06.754896 7f05537fe700 20 SERVER_ADDR=10.202.1.12
2014-03-17 17:53:06.754897 7f05537fe700 20 SERVER_PORT=80
2014-03-17 17:53:06.754898 7f05537fe700 20 REMOTE_ADDR=10.202.4.10
2014-03-17 17:53:06.754899 7f05537fe700 20 DOCUMENT_ROOT=/var/www
2014-03-17 17:53:06.754899 7f05537fe700 20 SERVER_ADMIN=[no address given]
2014-03-17 17:53:06.754900 7f05537fe700 20 SCRIPT_FILENAME=/var/www/s3gw.fcgi
2014-03-17 17:53:06.754901 7f05537fe700 20 REMOTE_PORT=54735
2014-03-17 17:53:06.754902 7f05537fe700 20 GATEWAY_INTERFACE=CGI/1.1
2014-03-17 17:53:06.754903 7f05537fe700 20 SERVER_PROTOCOL=HTTP/1.1
2014-03-17 17:53:06.754904 7f05537fe700 20 REQUEST_METHOD=HEAD
2014-03-17 17:53:06.754905 7f05537fe700 20 QUERY_STRING=page=swift&params=/v1/test
2014-03-17 17:53:06.754906 7f05537fe700 20 REQUEST_URI=/swift/v1/test
2014-03-17 17:53:06.754907 7f05537fe700 20 SCRIPT_NAME=/swift/v1/test
2014-03-17 17:53:06.754908 7f05537fe700  2 req 1:0.000270:swift:HEAD /swift/v1/test::getting op
2014-03-17 17:53:06.754918 7f05537fe700  2 req 1:0.000280:swift:HEAD /swift/v1/test:stat_bucket:authorizing
2014-03-17 17:53:06.754925 7f05537fe700 20 token_id=6c87a7d2351e4003a0fe7997430458b4
2014-03-17 17:53:06.755014 7f05537fe700 20 sending request to https://us-dfw-1.cisco.com:35357/v2.0/tokens/6c87a7d2351e4003a0fe7997430458b4
2014-03-17 17:53:06.856331 7f05537fe700 20 received response: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>411 Length Required</title>
</head><body>
<h1>Length Required</h1>
<p>A request of the requested method GET requires a valid Content-length.<br />
</p>
<hr>
<address>Apache/2.2.22 (Ubuntu) Server at us-dfw-1.cisco.com Port 35357</address>
</body></html>

2014-03-17 17:53:06.856463 7f05537fe700  0 Keystone token parse error: malformed json
2014-03-17 17:53:06.858035 7f05537fe700 10 failed to authorize request
2014-03-17 17:53:06.858123 7f05537fe700  2 req 1:0.103485:swift:HEAD /swift/v1/test:stat_bucket:http status=401
2014-03-17 17:53:06.858271 7f05537fe700  1 ====== req done req=0x1c90660 http_status=401 ======

All other keystone authenticated processes functioned normally.

Review of rgw code shows that the rgw_http_client.cc process function expects 'has_send_len' flag to be set in order to set the CURLOPT_INFILESIZE libcurl option. Without CURLOPT_INFILESIZE set, libcurl defaults to chunked transfer encoding. It appears that apache is stripping the chunked encoding header, but not adding the content-length header which Keystone is requiring.

  if (has_send_len) {
    curl_easy_setopt(curl_handle, CURLOPT_INFILESIZE, (void *)send_len);
  }

Workaround:
As a workaround, we added 'WSGIChunkedRequest On' to the apache config so the resulting section for port 35357 is as follows:

<VirtualHost *:35357>
    # WSGIDaemonProcess keystone-admin user=keystone group=keystone processes=$(facter processorcount) threads=10
    WSGIDaemonProcess keystone-admin user=keystone group=keystone processes=10 threads=10
    WSGIScriptAlias / /var/www/cgi-bin/keystone/admin
    # WSGIScriptAlias /admin /var/www/cgi-bin/keystone/admin

    <Directory /var/www/cgi-bin/keystone>
        WSGIProcessGroup keystone-admin
        WSGIApplicationGroup %{GLOBAL}
        WSGIChunkedRequest On
        Order deny,allow
        Allow from all
    </Directory>
</VirtualHost>

This allows normal RGW token authentication.

Related issues 1 (0 open — 1 closed)

Has duplicate rgw - Bug #9043: rgw:Cannot add object to Ceph using Openstack Dashboard(Horizon) in fireflyDuplicateJosh Durgin08/08/2014

Actions
Actions
Copy link
 #1

Updated by Sage Weil about 10 years ago

  • Priority changed from Normal to High
Actions
Copy link
 #2

Updated by Abhishek Lekshmanan almost 10 years ago

I also ran into this while trying to set up a test cluster. Never could figure out what went wrong until I finally stumbled upon keystone logs and setting 

WSGIChunkedRequest On

Actions
Copy link
 #3

Updated by Yehuda Sadeh over 9 years ago

  • Status changed from New to Won't Fix

The recommendation is to work around the issue using the afformentioned apache configuration.

Actions
Copy link
 #4

Updated by Yehuda Sadeh over 9 years ago

  • Status changed from Won't Fix to Fix Under Review
  • Backport set to firefly, giant
Actions
Copy link
 #5

Updated by Sage Weil over 9 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions
Copy link
 #6

Updated by Yehuda Sadeh over 9 years ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link

Also available in: Atom PDF