https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2017-05-05T17:10:51ZCeph rgw - Bug #19870: rgw: bytes_send and bytes_recv in the msg of usage show returning is 0 in master branchhttps://tracker.ceph.com/issues/19870?journal_id=908412017-05-05T17:10:51ZGreg Farnumgfarnum@redhat.com
<ul><li><strong>Project</strong> changed from <i>Ceph</i> to <i>rgw</i></li></ul> rgw - Bug #19870: rgw: bytes_send and bytes_recv in the msg of usage show returning is 0 in master branchhttps://tracker.ceph.com/issues/19870?journal_id=916622017-05-25T17:38:25ZOrit Wassermanowasserm@redhat.com
<ul><li><strong>Assignee</strong> set to <i>Marcus Watts</i></li></ul> rgw - Bug #19870: rgw: bytes_send and bytes_recv in the msg of usage show returning is 0 in master branchhttps://tracker.ceph.com/issues/19870?journal_id=916672017-05-25T17:45:56ZCasey Bodleycbodley@redhat.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-10 priority-5 priority-high3 closed" href="/issues/19194">Bug #19194</a>: after s3cmd put file to bucket, 'radosgw-admin usage show --categories=put_obj' bytes_received=0</i> added</li></ul> rgw - Bug #19870: rgw: bytes_send and bytes_recv in the msg of usage show returning is 0 in master branchhttps://tracker.ceph.com/issues/19870?journal_id=916692017-05-25T17:46:54ZOrit Wassermanowasserm@redhat.com
<ul></ul><p>duplicated of <a class="external" href="http://tracker.ceph.com/issues/19194">http://tracker.ceph.com/issues/19194</a></p> rgw - Bug #19870: rgw: bytes_send and bytes_recv in the msg of usage show returning is 0 in master branchhttps://tracker.ceph.com/issues/19870?journal_id=922572017-06-06T17:53:45ZYehuda Sadehyehuda@redhat.com
<ul><li><strong>Subject</strong> changed from <i>bytes_send and bytes_recv in the msg of usage show returning is 0 in master branch</i> to <i>rgw: bytes_send and bytes_recv in the msg of usage show returning is 0 in master branch</i></li></ul> rgw - Bug #19870: rgw: bytes_send and bytes_recv in the msg of usage show returning is 0 in master branchhttps://tracker.ceph.com/issues/19870?journal_id=922582017-06-06T17:57:02ZYehuda Sadehyehuda@redhat.com
<ul></ul><p>while we're at it, we need to make sure that there's sufficient teuthology coverage for this issue. I think we do test usage, but maybe not look at the bytes_received / bytes_sent fields.</p> rgw - Bug #19870: rgw: bytes_send and bytes_recv in the msg of usage show returning is 0 in master branchhttps://tracker.ceph.com/issues/19870?journal_id=923002017-06-07T07:15:09ZMarcus Wattsmwatts@redhat.com
<ul></ul><p>I may have found the teuthology tests. in ceph, master branch, "qa/tasks/radosgw_admin.py" and "qa/tasks/radosgw_admin_rest.py". Is that the current thing? Are there any other places I should look? It looks like that just exercises radosgw-admin interfaces, so may not fully cover all the s3/swift possible accounting paths. There's no logic there to look at bytes_sent|bytes_received. Looks like it's mainly checking existance and "successful_ops".</p>
<p>jewel has relatively straightforward code for this. Does it calculate bytes_sent and bytes_received correctly?</p>
<p>Looking at master, it appears it has significantly more complex code using c++ classes, templates, and such. One of the classes involved is "BufferingFilter" which appears to have the property of shifting byte counts from send_body() to a later complete_request() call. So far my gdb experiments suggest that this later count is not wrapped in AccountingFilter and so winds up not being accumulated.</p> rgw - Bug #19870: rgw: bytes_send and bytes_recv in the msg of usage show returning is 0 in master branchhttps://tracker.ceph.com/issues/19870?journal_id=924652017-06-08T20:28:49ZMarcus Wattsmwatts@redhat.com
<ul></ul><p>cbodley has a fix for a similar problem with swift:<br /> <a class="external" href="https://github.com/ceph/ceph/pull/14042">https://github.com/ceph/ceph/pull/14042</a></p>
<p>I believe it may be correct that "create_bucket" lists 0 bytes sent/received. The count appears to be intended to cover the body of the request/response only, not the headers. create_bucket may not have any data for either. I'll verify that this is the case with jewel/boto.</p>
<p>"list_buckets" should definitely have a nz sent_bytes count. Jewel certainly shows a nz count. I've been able to confirm with debug code that the count is being lost in process_request() when it calls complete_request.</p>
<p>In my experiments with s3cmd, I believe get/put object do update byte counts right.</p>
<p>I'm looking at what it takes to run teuthology's radmin_admin command standalone. I appear to have managed this once upon a time, so it should be just a matter of updating my process. Once I can do that, adding code to test out usage should be much more feasible.</p> rgw - Bug #19870: rgw: bytes_send and bytes_recv in the msg of usage show returning is 0 in master branchhttps://tracker.ceph.com/issues/19870?journal_id=926932017-06-13T07:58:04ZMarcus Wattsmwatts@redhat.com
<ul></ul><p>I've been able to run a modified version of radosgw_admin.py against jewel & master. Modifications included instrumenting boto calls to track sent & received bytes, moving the "usage show" logic to near the end (after all the boto) calls, and added a few more boto calls to exercise more interfaces.</p>
<p>Both jewel & master seem to have various oddities in counts.</p>
<p>jewel,<br />list_buckets: actually received 299 bytes, but ceph claims to have sent 320.<br />create_bucket: in 9 ops, 8 successes returned 0 bytes, one failed returning 218 bytes. Ceph claims to have sent 152 bytes in 8 ops.<br />get_acls: got 1026 bytes back, but ceph claims to have sent 1068.<br />list_bucket: got 484 bytes, but ceph claims to have sent 505.<br />put_acls: got 0 bytes back, but ceph claims to have sent 38.</p>
<p>master,<br />create_bucket: received 220 bytes in 1 failure of 9 ops total, but ceph claims to have sent 0 bytes in 8 ops.<br />get_acls: received 1026 bytes, but ceph claism to have sent 0 bytes.<br />list_bucket: received 484 butes, but ceph claims to have sent 0 bytes.<br />list_buckets: received 299 bytes, but ceph claims to have sent 0 bytes.</p>
<p>My build of ceph master doesn't have casey's patch. I'll have to look at that to see if it fixes s3 too.<br />Ideally these tests should be doing swift too, and be capable of testing casey's patch more directly.</p> rgw - Bug #19870: rgw: bytes_send and bytes_recv in the msg of usage show returning is 0 in master branchhttps://tracker.ceph.com/issues/19870?journal_id=930792017-06-17T07:31:28ZMarcus Wattsmwatts@redhat.com
<ul></ul><p>AccountingFilter is missing "complete_request". So that's why master drops "bytes_sent" in most cases. Implementing that function in reveals another problem: BufferingFilter does some trickery of its own. The big problem is that in complete_request, it calls send_content_length/complete_header - which return counts that should not be counted. The mechanism that's supposed to control when bytes are counted, "AccountingFilter::set_account()", is only supposed to be called after complete_header() is called, but BufferingFilter cleverly defers calling that. I have a not very elegant fix for this, which seems to be giving me mostly correct counts.</p> rgw - Bug #19870: rgw: bytes_send and bytes_recv in the msg of usage show returning is 0 in master branchhttps://tracker.ceph.com/issues/19870?journal_id=931662017-06-18T22:29:08ZMarcus Wattsmwatts@redhat.com
<ul></ul><p>github.com:mdw-at-linuxbox/ceph-1 policy is a copy of my current changes to master to do this. Lots of temp debug messages and deadend code get_pending_data_count() to remove--needs significant cleanup before making a PR. Also there's a dmclock build fix which at the very least has a different sha1 in master.</p> rgw - Bug #19870: rgw: bytes_send and bytes_recv in the msg of usage show returning is 0 in master branchhttps://tracker.ceph.com/issues/19870?journal_id=936122017-06-22T17:44:49ZMatt Benjaminmbenjamin@redhat.com
<ul></ul><p>per Marcus 6/21, he has a candidate fix with some "ugly" aspects, is working on revised version</p> rgw - Bug #19870: rgw: bytes_send and bytes_recv in the msg of usage show returning is 0 in master branchhttps://tracker.ceph.com/issues/19870?journal_id=955652017-07-20T18:01:45ZMatt Benjaminmbenjamin@redhat.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> rgw - Bug #19870: rgw: bytes_send and bytes_recv in the msg of usage show returning is 0 in master branchhttps://tracker.ceph.com/issues/19870?journal_id=959552017-07-27T17:40:32ZCasey Bodleycbodley@redhat.com
<ul></ul><p>can Marcus propose a simple fix so other developers can review and suggest alternatives?</p> rgw - Bug #19870: rgw: bytes_send and bytes_recv in the msg of usage show returning is 0 in master branchhttps://tracker.ceph.com/issues/19870?journal_id=965322017-08-04T23:29:17ZMarcus Wattsmwatts@redhat.com
<ul></ul><p>I've posted a fix against master for the bytes usage fix.<br /> <a class="external" href="https://github.com/ceph/ceph/pull/16834">https://github.com/ceph/ceph/pull/16834</a></p>
<p>I got rid of some unnecessary detours for fixing this bug (fixes based on theories of the bug that turned out to be wrong.)</p>
<p>This passes modified teuthology test code that I'll shortly post as a PR against teuthology.</p> rgw - Bug #19870: rgw: bytes_send and bytes_recv in the msg of usage show returning is 0 in master branchhttps://tracker.ceph.com/issues/19870?journal_id=965352017-08-05T01:23:57ZMarcus Wattsmwatts@redhat.com
<ul></ul><p>Turns out the actual teuthologic logic to test usage now lives in the ceph repo! So</p>
<p><a class="external" href="https://github.com/ceph/ceph/pull/16835">https://github.com/ceph/ceph/pull/16835</a></p>
<p>This points to a changed radosgw_admin.py file that checks usage a lot more thoroughly.</p>
<p>I tested this logic out against a jewel without any special patches - it passes this test as well. So we could use this to catch future regressions in jewel if we want.</p> rgw - Bug #19870: rgw: bytes_send and bytes_recv in the msg of usage show returning is 0 in master branchhttps://tracker.ceph.com/issues/19870?journal_id=967612017-08-09T18:24:53ZMatt Benjaminmbenjamin@redhat.com
<ul></ul><p>does this need backport (L?)</p> rgw - Bug #19870: rgw: bytes_send and bytes_recv in the msg of usage show returning is 0 in master branchhttps://tracker.ceph.com/issues/19870?journal_id=967622017-08-09T18:25:10ZMatt Benjaminmbenjamin@redhat.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Pending Backport</i></li></ul> rgw - Bug #19870: rgw: bytes_send and bytes_recv in the msg of usage show returning is 0 in master branchhttps://tracker.ceph.com/issues/19870?journal_id=967742017-08-09T18:36:33ZNathan Cutlerncutler@suse.cz
<ul><li><strong>Backport</strong> set to <i>luminous</i></li></ul> rgw - Bug #19870: rgw: bytes_send and bytes_recv in the msg of usage show returning is 0 in master branchhttps://tracker.ceph.com/issues/19870?journal_id=967882017-08-09T19:31:01ZNathan Cutlerncutler@suse.cz
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-4 priority-default closed" href="/issues/20968">Backport #20968</a>: luminous: rgw: bytes_send and bytes_recv in the msg of usage show returning is 0 in master branch</i> added</li></ul> rgw - Bug #19870: rgw: bytes_send and bytes_recv in the msg of usage show returning is 0 in master branchhttps://tracker.ceph.com/issues/19870?journal_id=988292017-09-11T19:50:14ZNathan Cutlerncutler@suse.cz
<ul><li><strong>Status</strong> changed from <i>Pending Backport</i> to <i>Resolved</i></li></ul>