https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2015-04-21T17:43:57ZCeph rgw - Bug #11419: DragonDisk fails to create directories via S3: MissingContentLengthhttps://tracker.ceph.com/issues/11419?journal_id=508702015-04-21T17:43:57ZYehuda Sadehyehuda@redhat.com
<ul></ul><p>Note that the content length header is passed as HTTP_CONTENT_LENGTH and not as CONTENT_LENGTH as required. Maybe there's some lighttpd configuration that is missing?</p> rgw - Bug #11419: DragonDisk fails to create directories via S3: MissingContentLengthhttps://tracker.ceph.com/issues/11419?journal_id=509732015-04-22T17:53:06ZRobin Johnsonrobbat2@gentoo.org
<ul></ul><p>I went through the lighttpd 1.4.33 codebase, and found the details below. I think in this case, the rgw codebase should be looking at env:HTTP_CONTENT_LENGTH if env:CONTENT_LENGTH is not present. I'll work up a quick patch next.</p>
<p>mod_fastcgi.c:
===<br /> if (con->request.content_length > 0 && host->mode != FCGI_AUTHORIZER) {<br /> /* CGI-SPEC 6.1.2 and FastCGI spec 6.3 */
===</p>
<pre><code>/* request.content_length < SSIZE_MAX, see request.c */<br /> LI_ltostr(buf, con->request.content_length);<br /> FCGI_ENV_ADD_CHECK(fcgi_env_add(p->fcgi_env, CONST_STR_LEN("CONTENT_LENGTH"), buf, strlen(buf)),con)<br /> }</code></pre>
<p><a class="external" href="http://www.fastcgi.com/devkit/doc/fcgi-spec.html#S6.3">http://www.fastcgi.com/devkit/doc/fcgi-spec.html#S6.3</a>
===<br />6.3 Authorizer:<br />...<br />The Authorizer application receives HTTP request information from the Web server on the FCGI_PARAMS stream, in the same format as a Responder. The Web server does not send CONTENT_LENGTH, PATH_INFO, PATH_TRANSLATED, and SCRIPT_NAME headers.
===</p> rgw - Bug #11419: DragonDisk fails to create directories via S3: MissingContentLengthhttps://tracker.ceph.com/issues/11419?journal_id=509792015-04-22T20:25:22ZRobin Johnsonrobbat2@gentoo.org
<ul></ul><p><a class="external" href="https://github.com/ceph/ceph/pull/4436">https://github.com/ceph/ceph/pull/4436</a></p> rgw - Bug #11419: DragonDisk fails to create directories via S3: MissingContentLengthhttps://tracker.ceph.com/issues/11419?journal_id=536832015-06-16T18:10:29ZSage Weilsage@newdream.net
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Pending Backport</i></li><li><strong>Backport</strong> set to <i>hammer, firefly</i></li><li><strong>Regression</strong> set to <i>No</i></li></ul> rgw - Bug #11419: DragonDisk fails to create directories via S3: MissingContentLengthhttps://tracker.ceph.com/issues/11419?journal_id=543372015-06-30T18:11:07ZYehuda Sadehyehuda@redhat.com
<ul><li><strong>Assignee</strong> set to <i>Loïc Dachary</i></li></ul> rgw - Bug #11419: DragonDisk fails to create directories via S3: MissingContentLengthhttps://tracker.ceph.com/issues/11419?journal_id=543852015-07-01T17:14:51ZAbhishek Lekshmananabhishek.lekshmanan@gmail.com
<ul></ul><p>Hammer Backport: <a class="external" href="https://github.com/ceph/ceph/pull/5118">https://github.com/ceph/ceph/pull/5118</a></p> rgw - Bug #11419: DragonDisk fails to create directories via S3: MissingContentLengthhttps://tracker.ceph.com/issues/11419?journal_id=574142015-08-31T14:12:52ZLoïc Dacharyloic@dachary.org
<ul><li><strong>Assignee</strong> changed from <i>Loïc Dachary</i> to <i>Robin Johnson</i></li></ul><p>Assigning Robin to recall who was originally taking care of the issue (I still care for the backport ;-)</p> rgw - Bug #11419: DragonDisk fails to create directories via S3: MissingContentLengthhttps://tracker.ceph.com/issues/11419?journal_id=600622015-10-12T19:50:29ZLoïc Dacharyloic@dachary.org
<ul><li><strong>Status</strong> changed from <i>Pending Backport</i> to <i>Resolved</i></li></ul> rgw - Bug #11419: DragonDisk fails to create directories via S3: MissingContentLengthhttps://tracker.ceph.com/issues/11419?journal_id=627912015-12-10T17:36:00ZMax Grobecker
<ul></ul><p>I'm facing the same problem on version 0.94.5 too by using RGW with nginx/fastcgi.<br />Should this fix have been backported to this version or is it a new bug?</p> rgw - Bug #11419: DragonDisk fails to create directories via S3: MissingContentLengthhttps://tracker.ceph.com/issues/11419?journal_id=627952015-12-10T20:03:19ZNathan Cutlerncutler@suse.cz
<ul></ul><p>The three commits from <a class="external" href="https://github.com/ceph/ceph/pull/5118">https://github.com/ceph/ceph/pull/5118</a> were released in 0.94.3:</p>
<pre>
$ git describe 0260abd
v0.94.2-27-g0260abd
$ git describe d9bbef3
v0.94.2-28-gd9bbef3
$ git describe 56c2688
v0.94.2-29-g56c2688
</pre>
<p>So they should be present in 0.94.5 as well. Matt, if you're sure this bug is present in 0.94.5, that would mean it's a regression.</p> rgw - Bug #11419: DragonDisk fails to create directories via S3: MissingContentLengthhttps://tracker.ceph.com/issues/11419?journal_id=627962015-12-10T20:06:51ZRobin Johnsonrobbat2@gentoo.org
<ul></ul><p>Max Grobecker wrote:</p>
<blockquote>
<p>I'm facing the same problem on version 0.94.5 too by using RGW with nginx/fastcgi.<br />Should this fix have been backported to this version or is it a new bug?</p>
</blockquote>
<p>Max:<br />can I get your nginx config and details to check? I tested on lighttpd & apache.<br />nginx version esp, so I can eyeball it's code for which variable it's using.</p>
<p>Lastly, if you could test just rgw+nginx with 0.94.3 (don't downgrade the rest) that would be handy to know if it's a regression.</p> rgw - Bug #11419: DragonDisk fails to create directories via S3: MissingContentLengthhttps://tracker.ceph.com/issues/11419?journal_id=627972015-12-10T20:09:21ZMax Grobecker
<ul></ul><p>Whenever I try to PUT a new file (tested DragonDisk and the officiall S3 SDK) I'm getting back HTTP/411 with error message "MissingContentLength".<br />I can confirm that my client is sending a Content-Length header and that nginx is passing it into the fcgi socket of RGW.</p>
<p>In nginx im using the socket file instead of the TCP socket - maybe that's causing the problem. I could test if the problem still is persistent when using a TCP socket to RGW, but I can't test it until tomorrow.</p> rgw - Bug #11419: DragonDisk fails to create directories via S3: MissingContentLengthhttps://tracker.ceph.com/issues/11419?journal_id=627982015-12-10T20:12:14ZMax Grobecker
<ul></ul><p>Robin Johnson wrote:</p>
<blockquote>
<p>Max:<br />can I get your nginx config and details to check? I tested on lighttpd & apache.<br />nginx version esp, so I can eyeball it's code for which variable it's using.</p>
</blockquote>
<p>That's the nginx config (Version nginx/1.6.2):<br /><pre>
location /
{
fastcgi_pass unix:/var/run/ceph/radosgw-s3-cust.socket;
include fastcgi_params;
fastcgi_keep_conn off;
fastcgi_buffering off;
fastcgi_cache off;
client_max_body_size 0;
fastcgi_read_timeout 300s;
### Für PUT benötigt
error_page 405 = $uri;
}
</pre></p>
<blockquote>
<p>Lastly, if you could test just rgw+nginx with 0.94.3 (don't downgrade the rest) that would be handy to know if it's a regression.</p>
</blockquote>
<p>I will test that, but I need to wait until tomorrow. I'll give you an update then.</p> rgw - Bug #11419: DragonDisk fails to create directories via S3: MissingContentLengthhttps://tracker.ceph.com/issues/11419?journal_id=643912016-01-20T16:30:26ZMax Grobecker
<ul></ul><p>Sorry for my late response.<br />Finally, I were able to figure out what happens:</p>
<p>When connecting nginx to RadosGW via FastCGI backend, there are a few "headers" transmitted over the FastCGI connection. I sniffed this backend connection and found out, that there are basically two key/value pairs transmitted which can cause this effect.</p>
<p>- CONTENT_TYPE<br />- CONTENT_LENGTH<br />- HTTP_CONTENT_TYPE<br />- HTTP_CONTENT_LENGTH</p>
<p>If...<br /> CONTENT_TYPE: present, but empty<br /> CONTENT_LENGTH: present, value "0" <br />then: RGW responds with HTTP 411, "MissingContentLength".</p>
<p>If...<br /> CONTENT_TYPE: not present<br /> CONTENT_LENGTH: present, value "0" <br />then it works and (for example) new buckets can be created.</p>
<p>If...<br /> CONTENT_LENGTH: present, value > 0<br /> HTTP_CONTENT_LENGTH: present, same value<br />then: RGW responds with HTTP 403, "SignatureMismatch".</p>
<p>If...<br /> CONTENT_LENGTH: not present<br /> HTTP_CONTENT_LENGTH: present, value > 0<br />then it works and also files can be uploaded</p>
<p>With CONTENT_TYPE and HTTP_CONTENT_TYPE there seems to be no confusion to RGW so far.<br />This seems to be an issue with confusion between CONTENT_... and HTTP_CONTENT_... parameters.</p>
<p>In nginx this can be circumvented with the following configuration (tested on Debian 8.2):</p>
<pre>
-- in file /etc/nginx/fastcgi_params --
## Remove (or disable) the line:
fastcgi_param CONTENT_LENGTH $content_length;
## Modify the line:
fastcgi_param CONTENT_TYPE $content_type;
## to
fastcgi_param CONTENT_TYPE $content_type if_not_empty;
----------------------------------------
</pre>