Project

General

Profile

Bug #11419

DragonDisk fails to create directories via S3: MissingContentLength

Added by Robin Johnson almost 9 years ago. Updated about 8 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Target version:
% Done:

0%

Spent time:
Source:
Community (user)
Tags:
Backport:
hammer, firefly
Regression:
No
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
rgw
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

We recently upgraded from Firefly (0.80.9) to Hammer (0.94.1); in doing so, it seems that DragonDisk now fails to create new directories. If I do an upload with s3cmd, then delete the file, the directory does remain visible to the system until it is deleted. It fails with an error of MissingContentLength. The same operations DO still work correctly with Amazon S3.

I tried to review the diff for RGW of v0.80.9..v0.94.1, but nothing stands out as the cause of the breakage; DragonDisk is sending 'Content-Length: 0' in both cases.

Below is a debug capture from radosgw for the create directory request from DragonDisk.

2015-04-17 03:22:07.435844 7facb37fe700 20 enqueued request req=0x7faca8017b80
2015-04-17 03:22:07.435856 7facb37fe700 20 RGWWQ:
2015-04-17 03:22:07.435858 7facb37fe700 20 req: 0x7faca8017b80
2015-04-17 03:22:07.435868 7facb37fe700 10 allocated request req=0x7faca801c330
2015-04-17 03:22:07.435896 7fac80fc1700 20 dequeued request req=0x7faca8017b80
2015-04-17 03:22:07.435913 7fac80fc1700 20 RGWWQ: empty
2015-04-17 03:22:07.435968 7fac80fc1700 20 DOCUMENT_ROOT=/var/www
2015-04-17 03:22:07.435979 7fac80fc1700 20 FCGI_ROLE=RESPONDER
2015-04-17 03:22:07.435980 7fac80fc1700 20 GATEWAY_INTERFACE=CGI/1.1
2015-04-17 03:22:07.435982 7fac80fc1700 20 HTTP_AUTHORIZATION=AWS 1GYH1195XZL5HCV0N275:OHFDKKZmHAJJY/lFFaFFL4Qe+vQ=
2015-04-17 03:22:07.435984 7fac80fc1700 20 HTTP_CONNECTION=Keep-Alive
2015-04-17 03:22:07.435987 7fac80fc1700 20 HTTP_CONTENT_LENGTH=0
2015-04-17 03:22:07.435988 7fac80fc1700 20 HTTP_DATE=Fri, 17 Apr 2015 03:22:07 GMT
2015-04-17 03:22:07.435989 7fac80fc1700 20 HTTP_HOST=download.s.libraries.coop
2015-04-17 03:22:07.435990 7fac80fc1700 20 HTTP_USER_AGENT=DragonDisk 1.05 ( http://www.dragondisk.com )
2015-04-17 03:22:07.435991 7fac80fc1700 20 PATH_INFO=/test2/
2015-04-17 03:22:07.435992 7fac80fc1700 20 PATH_TRANSLATED=/var/www/test2/
2015-04-17 03:22:07.435993 7fac80fc1700 20 QUERY_STRING=
2015-04-17 03:22:07.435995 7fac80fc1700 20 REDIRECT_STATUS=200
2015-04-17 03:22:07.435996 7fac80fc1700 20 REMOTE_ADDR=24.85.176.56
2015-04-17 03:22:07.435996 7fac80fc1700 20 REMOTE_PORT=59458
2015-04-17 03:22:07.435998 7fac80fc1700 20 REQUEST_METHOD=PUT
2015-04-17 03:22:07.435998 7fac80fc1700 20 REQUEST_URI=/test/test2/
2015-04-17 03:22:07.435999 7fac80fc1700 20 SCRIPT_FILENAME=/var/www/test
2015-04-17 03:22:07.436000 7fac80fc1700 20 SCRIPT_NAME=/test
2015-04-17 03:22:07.436001 7fac80fc1700 20 SERVER_ADDR=10.77.0.40
2015-04-17 03:22:07.436002 7fac80fc1700 20 SERVER_NAME=download.s.libraries.coop
2015-04-17 03:22:07.436003 7fac80fc1700 20 SERVER_PORT=80
2015-04-17 03:22:07.436004 7fac80fc1700 20 SERVER_PROTOCOL=HTTP/1.1
2015-04-17 03:22:07.436004 7fac80fc1700 20 SERVER_SOFTWARE=lighttpd/1.4.33
2015-04-17 03:22:07.436005 7fac80fc1700 1 ====== starting new request req=0x7faca8017b80 =====
2015-04-17 03:22:07.436016 7fac80fc1700 2 req 22868:0.000010::PUT /test/test2/::initializing
2015-04-17 03:22:07.436025 7fac80fc1700 10 host=download.s.libraries.coop
2015-04-17 03:22:07.436029 7fac80fc1700 20 subdomain=download domain=s.libraries.coop in_hosted_domain=1
2015-04-17 03:22:07.436053 7fac80fc1700 10 s->object=test/test2/ s->bucket=download
2015-04-17 03:22:07.436059 7fac80fc1700 2 req 22868:0.000053:s3:PUT /test/test2/::getting op
2015-04-17 03:22:07.436063 7fac80fc1700 2 req 22868:0.000057:s3:PUT /test/test2/:put_obj:authorizing
2015-04-17 03:22:07.436092 7fac80fc1700 10 get_canon_resource(): dest=/download/test/test2/
2015-04-17 03:22:07.436095 7fac80fc1700 10 auth_hdr:
PUT

Fri, 17 Apr 2015 03:22:07 GMT
/download/test/test2/
2015-04-17 03:22:07.436148 7fac80fc1700 15 calculated digest=OHFDKKZmHAJJY/lFFaFFL4Qe+vQ=
2015-04-17 03:22:07.436151 7fac80fc1700 15 auth_sign=OHFDKKZmHAJJY/lFFaFFL4Qe+vQ=
2015-04-17 03:22:07.436152 7fac80fc1700 15 compare=0
2015-04-17 03:22:07.436155 7fac80fc1700 2 req 22868:0.000149:s3:PUT /test/test2/:put_obj:reading permissions
2015-04-17 03:22:07.436178 7fac80fc1700 15 Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>downloads</ID><DisplayName>Downloads</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>downloads</ID><DisplayName>Downloads</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
2015-04-17 03:22:07.436187 7fac80fc1700 2 req 22868:0.000181:s3:PUT /test/test2/:put_obj:init op
2015-04-17 03:22:07.436191 7fac80fc1700 2 req 22868:0.000185:s3:PUT /test/test2/:put_obj:verifying op mask
2015-04-17 03:22:07.436193 7fac80fc1700 20 required_mask= 2 user.op_mask=7
2015-04-17 03:22:07.436194 7fac80fc1700 2 req 22868:0.000188:s3:PUT /test/test2/:put_obj:verifying op permissions
2015-04-17 03:22:07.436197 7fac80fc1700 5 Searching permissions for uid=downloads mask=50
2015-04-17 03:22:07.436199 7fac80fc1700 5 Found permission: 15
2015-04-17 03:22:07.436200 7fac80fc1700 5 Searching permissions for group=1 mask=50
2015-04-17 03:22:07.436202 7fac80fc1700 5 Permissions for group not found
2015-04-17 03:22:07.436203 7fac80fc1700 5 Searching permissions for group=2 mask=50
2015-04-17 03:22:07.436205 7fac80fc1700 5 Permissions for group not found
2015-04-17 03:22:07.436205 7fac80fc1700 5 Getting permissions id=downloads owner=downloads perm=2
2015-04-17 03:22:07.436207 7fac80fc1700 10 uid=downloads requested perm (type)=2, policy perm=2, user_perm_mask=2, acl perm=2
2015-04-17 03:22:07.436209 7fac80fc1700 2 req 22868:0.000203:s3:PUT /test/test2/:put_obj:verifying op params
2015-04-17 03:22:07.436211 7fac80fc1700 2 req 22868:0.000205:s3:PUT /test/test2/:put_obj:executing
2015-04-17 03:22:07.436332 7fac80fc1700 2 req 22868:0.000326:s3:PUT /test/test2/:put_obj:http status=411
2015-04-17 03:22:07.436337 7fac80fc1700 1 ====== req done req=0x7faca8017b80 http_status=411 ======


Related issues

Copied to rgw - Backport #12041: DragonDisk fails to create directories via S3: MissingContentLength Rejected 04/17/2015
Copied to rgw - Backport #12042: DragonDisk fails to create directories via S3: MissingContentLength Resolved 04/17/2015

Associated revisions

Revision 3e38eab4 (diff)
Added by Robin Johnson almost 9 years ago

rgw: improve content-length env var handling

The FastCGI specification, section 6.3 on Authorizers, describes a case
where HTTP_CONTENT_LENGTH will be set in the environment and
CONTENT_LENGTH will NOT be set.

Further documention in the code.

Fixes: #11419
Signed-off-by: Robin H. Johnson <>

Revision 0260abd5 (diff)
Added by Robin Johnson over 8 years ago

rgw: improve content-length env var handling

The FastCGI specification, section 6.3 on Authorizers, describes a case
where HTTP_CONTENT_LENGTH will be set in the environment and
CONTENT_LENGTH will NOT be set.

Further documention in the code.

Fixes: #11419
Signed-off-by: Robin H. Johnson <>
(cherry picked from commit 3e38eab44bfb082fdd2b6f29b8b0357f8f5c11bb)

History

#1 Updated by Yehuda Sadeh almost 9 years ago

Note that the content length header is passed as HTTP_CONTENT_LENGTH and not as CONTENT_LENGTH as required. Maybe there's some lighttpd configuration that is missing?

#2 Updated by Robin Johnson almost 9 years ago

I went through the lighttpd 1.4.33 codebase, and found the details below. I think in this case, the rgw codebase should be looking at env:HTTP_CONTENT_LENGTH if env:CONTENT_LENGTH is not present. I'll work up a quick patch next.

mod_fastcgi.c: ===
if (con->request.content_length > 0 && host->mode != FCGI_AUTHORIZER) {
/* CGI-SPEC 6.1.2 and FastCGI spec 6.3 */ ===

/* request.content_length < SSIZE_MAX, see request.c */
LI_ltostr(buf, con->request.content_length);
FCGI_ENV_ADD_CHECK(fcgi_env_add(p->fcgi_env, CONST_STR_LEN("CONTENT_LENGTH"), buf, strlen(buf)),con)
}

http://www.fastcgi.com/devkit/doc/fcgi-spec.html#S6.3 ===
6.3 Authorizer:
...
The Authorizer application receives HTTP request information from the Web server on the FCGI_PARAMS stream, in the same format as a Responder. The Web server does not send CONTENT_LENGTH, PATH_INFO, PATH_TRANSLATED, and SCRIPT_NAME headers. ===

#4 Updated by Sage Weil almost 9 years ago

  • Status changed from New to Pending Backport
  • Backport set to hammer, firefly
  • Regression set to No

#5 Updated by Yehuda Sadeh over 8 years ago

  • Assignee set to Loïc Dachary

#7 Updated by Loïc Dachary over 8 years ago

  • Assignee changed from Loïc Dachary to Robin Johnson

Assigning Robin to recall who was originally taking care of the issue (I still care for the backport ;-)

#8 Updated by Loïc Dachary over 8 years ago

  • Status changed from Pending Backport to Resolved

#9 Updated by Max Grobecker over 8 years ago

I'm facing the same problem on version 0.94.5 too by using RGW with nginx/fastcgi.
Should this fix have been backported to this version or is it a new bug?

#10 Updated by Nathan Cutler over 8 years ago

The three commits from https://github.com/ceph/ceph/pull/5118 were released in 0.94.3:

$ git describe 0260abd
v0.94.2-27-g0260abd
$ git describe d9bbef3
v0.94.2-28-gd9bbef3
$ git describe 56c2688
v0.94.2-29-g56c2688

So they should be present in 0.94.5 as well. Matt, if you're sure this bug is present in 0.94.5, that would mean it's a regression.

#11 Updated by Robin Johnson over 8 years ago

Max Grobecker wrote:

I'm facing the same problem on version 0.94.5 too by using RGW with nginx/fastcgi.
Should this fix have been backported to this version or is it a new bug?

Max:
can I get your nginx config and details to check? I tested on lighttpd & apache.
nginx version esp, so I can eyeball it's code for which variable it's using.

Lastly, if you could test just rgw+nginx with 0.94.3 (don't downgrade the rest) that would be handy to know if it's a regression.

#12 Updated by Max Grobecker over 8 years ago

Whenever I try to PUT a new file (tested DragonDisk and the officiall S3 SDK) I'm getting back HTTP/411 with error message "MissingContentLength".
I can confirm that my client is sending a Content-Length header and that nginx is passing it into the fcgi socket of RGW.

In nginx im using the socket file instead of the TCP socket - maybe that's causing the problem. I could test if the problem still is persistent when using a TCP socket to RGW, but I can't test it until tomorrow.

#13 Updated by Max Grobecker over 8 years ago

Robin Johnson wrote:

Max:
can I get your nginx config and details to check? I tested on lighttpd & apache.
nginx version esp, so I can eyeball it's code for which variable it's using.

That's the nginx config (Version nginx/1.6.2):

location /
{
    fastcgi_pass unix:/var/run/ceph/radosgw-s3-cust.socket;
    include fastcgi_params;
    fastcgi_keep_conn off;

    fastcgi_buffering off;
    fastcgi_cache off;

    client_max_body_size 0;
    fastcgi_read_timeout 300s;

    ### Für PUT benötigt
    error_page 405 = $uri;
}

Lastly, if you could test just rgw+nginx with 0.94.3 (don't downgrade the rest) that would be handy to know if it's a regression.

I will test that, but I need to wait until tomorrow. I'll give you an update then.

#14 Updated by Max Grobecker about 8 years ago

Sorry for my late response.
Finally, I were able to figure out what happens:

When connecting nginx to RadosGW via FastCGI backend, there are a few "headers" transmitted over the FastCGI connection. I sniffed this backend connection and found out, that there are basically two key/value pairs transmitted which can cause this effect.

- CONTENT_TYPE
- CONTENT_LENGTH
- HTTP_CONTENT_TYPE
- HTTP_CONTENT_LENGTH

If...
CONTENT_TYPE: present, but empty
CONTENT_LENGTH: present, value "0"
then: RGW responds with HTTP 411, "MissingContentLength".

If...
CONTENT_TYPE: not present
CONTENT_LENGTH: present, value "0"
then it works and (for example) new buckets can be created.

If...
CONTENT_LENGTH: present, value > 0
HTTP_CONTENT_LENGTH: present, same value
then: RGW responds with HTTP 403, "SignatureMismatch".

If...
CONTENT_LENGTH: not present
HTTP_CONTENT_LENGTH: present, value > 0
then it works and also files can be uploaded

With CONTENT_TYPE and HTTP_CONTENT_TYPE there seems to be no confusion to RGW so far.
This seems to be an issue with confusion between CONTENT_... and HTTP_CONTENT_... parameters.

In nginx this can be circumvented with the following configuration (tested on Debian 8.2):

-- in file /etc/nginx/fastcgi_params --
## Remove (or disable) the line:
fastcgi_param  CONTENT_LENGTH     $content_length;

## Modify the line:
fastcgi_param  CONTENT_TYPE       $content_type;
## to
fastcgi_param  CONTENT_TYPE       $content_type if_not_empty;
----------------------------------------

Also available in: Atom PDF