Ceph : Matt Benjamin
https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2024-03-27T22:36:55Z
Ceph
Redmine
rgw - Bug #65188 (Fix Under Review): rgwlc: Executing radosgw-admin lc process --bucket <bkt-name...
https://tracker.ceph.com/issues/65188#change-258505
2024-03-27T22:36:55Z
Matt Benjamin
mbenjamin@redhat.com
rgw - Bug #65188 (Fix Under Review): rgwlc: Executing radosgw-admin lc process --bucket <bkt-name...
https://tracker.ceph.com/issues/65188
2024-03-27T22:30:11Z
Matt Benjamin
mbenjamin@redhat.com
<p>[LC-Process]: Executing radosgw-admin lc process --bucket <bkt-name> without setting lc rule results in Segmentation fault</p>
<p>Description of problem:<br />[LC-Process]: Executing radosgw-admin lc process --bucket <bkt-name> without setting lc rule results in Segmentation fault</p>
<p>Version-Release number of selected component (if applicable):<br />ceph version 18.2.1-73.el9cp</p>
<p>How reproducible:<br />3/3</p>
<p>Steps to Reproduce:<br />1. Deploy cluster with: ceph version 18.2.1-73.el9cp<br />2. Create a bucket: <bkt_name><br />3. Upload object to the bucket<br />4. Perform: radosgw-admin lc process --bucket <bkt_name></p>
<p>Actual results:<br />Throwing error:</p>
<ul>
<li>Caught signal (Segmentation fault) <b><br /> in thread 7f74eed29800 thread_name:radosgw-admin<br /> ceph version 18.2.1-73.el9cp (16d1bc4bed21ede5993c301b4626fa21cbe97cff) reef (stable)<br /> 1: /lib64/libc.so.6(+0x54db0) [0x7f74ef254db0]<br /> 2: (RGWLC::process_bucket(int, int, RGWLC::LCWorker*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)+0x2b6) [0x556e82e69626]<br /> 3: (RGWLC::process(RGWLC::LCWorker*, std::unique_ptr<rgw::sal::Bucket, std::default_delete<rgw::sal::Bucket> > const&, bool)+0xb7) [0x556e82e6d1a7]<br /> 4: (RGWRados::process_lc(std::unique_ptr<rgw::sal::Bucket, std::default_delete<rgw::sal::Bucket> > const&)+0xdd) [0x556e831d409d]<br /> 5: main()<br /> 6: /lib64/libc.so.6(+0x3feb0) [0x7f74ef23feb0]<br /> 7: __libc_start_main()<br /> 8: _start()<br />2024-03-20T02:17:03.968-0400 7f74eed29800 -1 <strong></b> Caught signal (Segmentation fault) *</strong><br /> in thread 7f74eed29800 thread_name:radosgw-admin</li>
</ul>
rgw - Feature #65131: perf counters for CreateMultipartUpload, AbortMultipartUpload, CompleteMult...
https://tracker.ceph.com/issues/65131#change-258289
2024-03-27T02:13:23Z
Matt Benjamin
mbenjamin@redhat.com
<p>Oh, duh, this is doing multipart upload.</p>
<pre>
debug 2024-03-27T00:58:53.834+0000 7fe541aed700 1 beast: 0x7fe4e0a2a730: 10.242.8.219 - warp [27/Mar/2024:00:58:53.682 +0000] "PUT /boto-2/object_311?uploadId=2~gDSKylGfy-fsnVNgQQFw-HPOWsg0Loa&partNumber=8 HTTP/1.1" 200 8388608 - "Boto3/
1.9.253 Python/3.8.10 Linux/5.4.0-169-generic Botocore/1.16.19 Resource" - latency=0.152004898s
debug 2024-03-27T00:58:53.834+0000 7fe56db45700 1 ====== req done req=0x7fe4e0928730 op status=0 http_status=200 latency=0.156005025s ======
</pre>
<p>The l_rgw_put will be incremented once for each UploadPart attempted--perhaps each object is being sent in 4 (er, 16) MB parts?</p>
rgw - Feature #65131: perf counters for CreateMultipartUpload, AbortMultipartUpload, CompleteMult...
https://tracker.ceph.com/issues/65131#change-258287
2024-03-27T01:42:59Z
Matt Benjamin
mbenjamin@redhat.com
<p>The rgw counter we should be sampling, I think, is l_rgw put. The logic to increment the counter, as I recalled in slack, is<br />in fact in the op layer--which tracks actual ops, not (e.g.) the 4MB chunks we might be writing out to RADOS.</p>
<p>OTOH, it seems like this counter is being incremented when the op starts executing, not when it has committed. So while it seems<br />implausible that you would see a discrete multiple of the number of objects warp says it uploaded, it does seem like this<br />would induce the counter to reflect the number of attempts to upload each object, not just the ones that succeeded:</p>
<pre><code class="cpp syntaxhl"><span class="CodeRay"><span class="directive">void</span> RGWPutObj::execute(optional_yield y)
{
<span class="predefined-type">char</span> supplied_md5_bin[CEPH_CRYPTO_MD5_DIGESTSIZE + <span class="integer">1</span>];
<span class="predefined-type">char</span> supplied_md5[CEPH_CRYPTO_MD5_DIGESTSIZE * <span class="integer">2</span> + <span class="integer">1</span>];
<span class="predefined-type">char</span> calc_md5[CEPH_CRYPTO_MD5_DIGESTSIZE * <span class="integer">2</span> + <span class="integer">1</span>];
<span class="predefined-type">unsigned</span> <span class="predefined-type">char</span> m[CEPH_CRYPTO_MD5_DIGESTSIZE];
MD5 hash;
<span class="comment">// Allow use of MD5 digest in FIPS mode for non-cryptographic purposes</span>
hash.SetFlags(EVP_MD_CTX_FLAG_NON_FIPS_ALLOW);
bufferlist bl, aclbl, bs;
<span class="predefined-type">int</span> len;
off_t fst;
off_t lst;
<span class="directive">auto</span> counters = rgw::op_counters::get(s);
<span class="predefined-type">bool</span> need_calc_md5 = (dlo_manifest == <span class="predefined-constant">NULL</span>) && (slo_info == <span class="predefined-constant">NULL</span>);
rgw::op_counters::inc(counters, l_rgw_op_put_obj, <span class="integer">1</span>);
perfcounter->inc(l_rgw_put);
</span></code></pre>
<p>(this is downstream reef)</p>
rgw - Bug #58879: S3 CompleteMultipartUploadResult has empty ETag element
https://tracker.ceph.com/issues/58879#change-257852
2024-03-22T17:33:02Z
Matt Benjamin
mbenjamin@redhat.com
<p>This seems to be helping, review appreciated:</p>
<pre><code class="diff syntaxhl"><span class="CodeRay"><span class="line comment">diff --git a/src/rgw/rgw_op.cc b/src/rgw/rgw_op.cc</span>
<span class="line comment">index 3c4c59fc70c..f3f4ce859cb 100644</span>
<span class="line head"><span class="head">--- </span><span class="filename">a/src/rgw/rgw_op.cc</span></span>
<span class="line head"><span class="head">+++ </span><span class="filename">b/src/rgw/rgw_op.cc</span></span>
<span class="change"><span class="change">@@</span> -6809,10 +6809,17 <span class="change">@@</span></span> <span class="directive">void</span> RGWCompleteMultipart::execute(optional_yield y)
ldpp_dout(<span class="local-variable">this</span>, <span class="integer">0</span>) << <span class="string"><span class="delimiter">"</span><span class="content">WARNING: failed to remove object </span><span class="delimiter">"</span></span> << meta_obj << dendl;
}
<span class="line insert"><span class="insert">+</span> r = s->object->get_obj_attrs(s->yield, <span class="local-variable">this</span>);</span>
<span class="line insert"><span class="insert">+</span> <span class="keyword">if</span> (r < <span class="integer">0</span>) {</span>
<span class="line insert"><span class="insert">+</span> ldpp_dout(<span class="local-variable">this</span>, <span class="integer">0</span>) << __func__ << <span class="string"><span class="delimiter">"</span><span class="content">() ERROR: get_obj_attrs() returned ret=</span><span class="delimiter">"</span></span> << r << dendl;</span>
<span class="line insert"><span class="insert">+</span> }</span>
<span class="line insert"><span class="insert">+</span></span>
<span class="line insert"><span class="insert">+</span> etag = s->object->get_attrs()[RGW_ATTR_ETAG].to_str();</span>
<span class="line insert"><span class="insert">+</span></span>
<span class="comment">// send request to notification manager</span>
<span class="line delete"><span class="delete">-</span> <span class="eyecatcher"><span class="predefined-type">int</span> ret</span> = res->publish_commit(<span class="local-variable">this</span>, ofs, upload->get_mtime(), etag, target_obj->get_instance());</span>
<span class="line delete"><span class="delete">-</span> <span class="keyword">if</span> (r<span class="eyecatcher">et</span> < <span class="integer">0</span>) {</span>
<span class="line delete"><span class="delete">-</span> ldpp_dout(<span class="local-variable">this</span>, <span class="integer">1</span>) << <span class="string"><span class="delimiter">"</span><span class="content">ERROR: publishing notification failed, with error: </span><span class="delimiter">"</span></span> << r<span class="eyecatcher">et</span> << dendl;</span>
<span class="line insert"><span class="insert">+</span> <span class="eyecatcher">r</span> = res->publish_commit(<span class="local-variable">this</span>, ofs, upload->get_mtime(), etag, target_obj->get_instance());</span>
<span class="line insert"><span class="insert">+</span> <span class="keyword">if</span> (r < <span class="integer">0</span>) {</span>
<span class="line insert"><span class="insert">+</span> ldpp_dout(<span class="local-variable">this</span>, <span class="integer">1</span>) << <span class="string"><span class="delimiter">"</span><span class="content">ERROR: publishing notification failed, with error: </span><span class="delimiter">"</span></span> << r << dendl;</span>
<span class="comment">// too late to rollback operation, hence op_ret is not set here</span>
}
} <span class="comment">// RGWCompleteMultipart::execute</span>
<span class="change"><span class="change">@@</span> -6866,8 +6873,6 <span class="change">@@</span></span> <span class="directive">void</span> RGWCompleteMultipart::complete()
}
}
<span class="line delete"><span class="delete">-</span> etag = s->object->get_attrs()[RGW_ATTR_ETAG].to_str();</span>
<span class="line delete"><span class="delete">-</span></span>
send_response();
}
</span></code></pre>
rgw - Bug #64950: rgw-nfs: various file mv (rename) operations fail
https://tracker.ceph.com/issues/64950#change-257122
2024-03-15T14:32:30Z
Matt Benjamin
mbenjamin@redhat.com
<p>update: not quincy, reef</p>
rgw - Bug #64950 (Fix Under Review): rgw-nfs: various file mv (rename) operations fail
https://tracker.ceph.com/issues/64950#change-257118
2024-03-15T14:08:23Z
Matt Benjamin
mbenjamin@redhat.com
rgw - Bug #64950 (In Progress): rgw-nfs: various file mv (rename) operations fail
https://tracker.ceph.com/issues/64950#change-257116
2024-03-15T13:57:19Z
Matt Benjamin
mbenjamin@redhat.com
rgw - Bug #64950 (Pending Backport): rgw-nfs: various file mv (rename) operations fail
https://tracker.ceph.com/issues/64950
2024-03-15T13:56:58Z
Matt Benjamin
mbenjamin@redhat.com
<p>This is a regression, probably dating back to Quincy. The initial regression was caused by zipper integration.</p>
rgw - Bug #64676 (Fix Under Review): rgw: awssigv4: new trailer boundary case
https://tracker.ceph.com/issues/64676#change-255857
2024-03-02T19:36:06Z
Matt Benjamin
mbenjamin@redhat.com
rgw - Bug #64676 (Pending Backport): rgw: awssigv4: new trailer boundary case
https://tracker.ceph.com/issues/64676
2024-03-02T19:34:30Z
Matt Benjamin
mbenjamin@redhat.com
<p>I observed an environment in which the maven test suite, using http, generated a trailer chunk<br />boundary of "0;" rather than the expected "\r\n0;"</p>
<p>This only arose during upload of a 0-length object.</p>
<p>The failing test has obviously run successfully at and since merge, this was seen only when the<br />current sigv4 parsing logic was taken for our downstream 7.0z1.</p>