Project

General

Profile

Activity

From 11/18/2020 to 12/17/2020

12/17/2020

06:19 PM Backport #48127: octopus: Unnecessary bilogs are left in sync-disabled buckets
Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/38338
merged
Yuri Weinstein
06:18 PM Bug #48571: Multi part upload abort on first put with unknown error -2009
Casey Bodley wrote:
> what is the s3 client? does it support multipart uploads?
The S3 client, as stated at the b...
Simon Pierre Desrosiers
04:50 PM Bug #48571: Multi part upload abort on first put with unknown error -2009
I am not sure what exactly you are looking for, but here are the logs for the two rgw server, both in debug log level... Simon Pierre Desrosiers
03:16 PM Bug #48571 (Need More Info): Multi part upload abort on first put with unknown error -2009
Casey Bodley
03:16 PM Bug #48571: Multi part upload abort on first put with unknown error -2009
what is the s3 client? does it support multipart uploads? Casey Bodley
03:13 PM Bug #48571: Multi part upload abort on first put with unknown error -2009
Can you provide an rgw log segment which shows the required init-multipart-upload step which needs to run before any ... Matt Benjamin
03:10 PM Bug #48571: Multi part upload abort on first put with unknown error -2009
rgw_common.h:#define ERR_NO_SUCH_UPLOAD 2009
rgw_common.cc: { ERR_NO_SUCH_UPLOAD, {404, "NoSuchUpload" }},
Casey Bodley
06:05 PM Backport #48660 (Rejected): octopus: notifications: only bucket owner is allowed to add/remove no...
Backport Bot
06:05 PM Backport #48659 (Resolved): octopus: notification: radosgw-admin hangs on while closing
https://github.com/ceph/ceph/pull/40381 Backport Bot
06:04 PM Bug #48250 (Pending Backport): notifications: only bucket owner is allowed to add/remove notifica...
Yuval Lifshitz
06:03 PM Bug #46243 (Resolved): notification: add more examples notification boto3 README
Yuval Lifshitz
06:00 PM Backport #48658 (Rejected): octopus: notification: cannot delete a notification from a deleted bu...
Backport Bot
06:00 PM Bug #48293 (Pending Backport): notification: radosgw-admin hangs on while closing
Yuval Lifshitz
06:00 PM Backport #48657 (Rejected): octopus: notifications: tag based filtering works only for object del...
Backport Bot
05:58 PM Bug #48321 (Pending Backport): notifications: tag based filtering works only for object deletions
Yuval Lifshitz
05:57 PM Bug #46128 (Pending Backport): notification: cannot delete a notification from a deleted bucket
Yuval Lifshitz
03:28 PM Bug #48351: object-lock enabled is not working on 14.2.9 - files still deletable
Matt to reassign Casey Bodley
03:24 PM Bug #48472 (Need More Info): rgw crash when get none exist role policy
Casey Bodley
03:23 PM Bug #48554 (Fix Under Review): When copying an encrypted object, the result object is empty.
Casey Bodley
10:45 AM Bug #48646 (Closed): Bucket operations an issue with C# AWSSDK.S3 client
## Description
Bucket operations an issue with C# AWSSDK.S3 client
We are having issues with bucket operations with...
Ist Gab

12/16/2020

06:35 PM Backport #48426 (In Progress): nautilus: Put policy should return 204 instead of 200
Nathan Cutler
06:34 PM Backport #48423 (Need More Info): nautilus: Able to circumvent S3 Object Lock using deleteobjects...
complicated backport Nathan Cutler
04:18 PM Bug #48632 (Resolved): S3 error: 404 (NoSuchBucket) due to distribute cache is not being invoked
Reproducer:
aws s3 cp s3://hello/hosts abc --endpoint-url http://<rgw1>
aws s3 mb s3://hello --endpoint-url http:...
Or Friedmann

12/15/2020

02:22 PM Backport #48346 (In Progress): nautilus: rgw: unnecessary payload is added at the end of the message
Nathan Cutler
02:19 PM Backport #48128 (In Progress): nautilus: Unnecessary bilogs are left in sync-disabled buckets
Nathan Cutler
02:18 PM Backport #48100 (In Progress): nautilus: Admin API returns 200 instead of 404 for Get Bucket Info
Nathan Cutler
01:46 PM Bug #48571: Multi part upload abort on first put with unknown error -2009
I am sorry, the end of my message was lost during edition.
We can see in the rgw output that an unknown error is h...
Simon Pierre Desrosiers

12/14/2020

08:30 AM Bug #47910: radosgw crash on objecter operations
Hi Mauricio,
I planned to test backport this week, I will be back with results.
Rafal
Rafal Wadolowski
12:43 AM Bug #48554: When copying an encrypted object, the result object is empty.
https://github.com/ceph/ceph/pull/38537 Ilsoo Byun
12:42 AM Bug #48556: UploadPartCopy uses the wrong headers
https://github.com/ceph/ceph/pull/38541 Ilsoo Byun

12/13/2020

10:32 AM Backport #48576 (In Progress): nautilus: RGW prefetches data for range requests
Nathan Cutler

12/12/2020

02:39 PM Backport #48238 (Resolved): nautilus: list object versions returned multiple 'IsLatest true' entries
This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/38085
m...
Nathan Cutler
01:06 AM Bug #47910: radosgw crash on objecter operations
Hi Rafal,
Please excuse this message if you're already out for vacation/holiday season.
Just following up if yo...
Mauricio Oliveira
01:00 AM Backport #48576 (Resolved): nautilus: RGW prefetches data for range requests
https://github.com/ceph/ceph/pull/38556 Backport Bot
12:58 AM Bug #44508 (Pending Backport): RGW prefetches data for range requests
Dan Hill
12:57 AM Bug #44508: RGW prefetches data for range requests
This issue is also present in nautilus. I've added it to the backport list for consideration. Dan Hill

12/11/2020

09:19 PM Bug #48571 (Can't reproduce): Multi part upload abort on first put with unknown error -2009
Hello,
We are trying to configure IBM Cloud Tape Connector to do backup on our Cephs<s S3. Initial Connection hap...
Simon Pierre Desrosiers
07:59 PM Backport #48238: nautilus: list object versions returned multiple 'IsLatest true' entries
https://github.com/ceph/ceph/pull/38085 merged Yuri Weinstein
07:21 AM Bug #48556 (New): UploadPartCopy uses the wrong headers
to create an part from an encrypted object, RadosGW uses the following headers:
* x-amz-server-side-encryption-custo...
Ilsoo Byun
05:31 AM Bug #48472: rgw crash when get none exist role policy
Hi @joke lee,
I tried the following:
radosgw-admin user create --tenant t1tenant --uid t1user --display-name t1...
Pritha Srivastava
12:55 AM Bug #48554 (Resolved): When copying an encrypted object, the result object is empty.
When I use UploadPartCopyRequest to copy an encrypted object without a key, the request succeeds, but the result obje... Ilsoo Byun

12/10/2020

03:16 PM Backport #48548 (Rejected): octopus: Access denied for multi-object-delete by non-bucket-owner
Backport Bot
03:16 PM Backport #48547 (Rejected): nautilus: Access denied for multi-object-delete by non-bucket-owner
Backport Bot
03:15 PM Backport #48546 (Resolved): octopus: rgwlc: shard-index vector short by 1?
https://github.com/ceph/ceph/pull/38824 Backport Bot
03:15 PM Backport #48545 (Rejected): nautilus: rgwlc: shard-index vector short by 1?
Backport Bot
03:15 PM Backport #48544 (Resolved): octopus: rgw_file: common_prefixes returned out of lexical order
https://github.com/ceph/ceph/pull/38829 Backport Bot
03:15 PM Backport #48543 (Resolved): nautilus: rgw_file: common_prefixes returned out of lexical order
https://github.com/ceph/ceph/pull/38828 Backport Bot
03:12 PM Bug #48410 (Pending Backport): rgw_file: common_prefixes returned out of lexical order
Matt Benjamin
03:08 PM Bug #46567 (Pending Backport): Access denied for multi-object-delete by non-bucket-owner
Casey Bodley
03:06 PM Bug #48255 (Pending Backport): rgwlc: shard-index vector short by 1?
Matt Benjamin
01:04 AM Bug #48363: rgw: concurrent object delete can leak bucket index instance entries
Matt Benjamin wrote:
> Hi Yehuda,
>
> What Ceph versions are affected?
>
> Matt
we ran into the same proble...
joke lee

12/09/2020

11:35 AM Feature #48513: uses librgw2 to directly access the rados cluster for hadoop
The adapter invokes the librgw2 interface through JNI. The following interfaces need to be implemented:
static const...
chunsong feng
11:32 AM Feature #48513 (New): uses librgw2 to directly access the rados cluster for hadoop
Hadoop S3 accesses Ceph through the object gateway. The radosgw daemon process needs to be deployed. Messages are sen... chunsong feng

12/08/2020

06:47 PM Bug #47074: rgw: bucket radoslist looping endlessly while processing a bucket
I'm experiencing the same issue in my environment. I recently attempted to run the ‘rgw-orphan-list’ tool against our... James McEwan
04:23 PM Fix #48499 (Resolved): nautilus: rgw: in ordered bucket listing skip namespaced entries internall...
When doing an ordered bucket listing, such as from s3, we're listing all the entries that are not in a namespace. Non... J. Eric Ivancich

12/07/2020

01:26 AM Bug #48472 (Need More Info): rgw crash when get none exist role policy
rgw will crash when get put delete pocliy to a none exist role
import boto3, json
import botocore
botocore.session...
joke lee

12/04/2020

09:04 PM Bug #47910: radosgw crash on objecter operations
Hi Rafal, Casey,
I've completed the backport on the nautilus branch, and tested with the steps from 43739 [1] usin...
Mauricio Oliveira
11:08 AM Backport #48462 (Resolved): octopus: notifications: support GetTopicAttributes API
https://github.com/ceph/ceph/pull/40812 Nathan Cutler
11:08 AM Backport #48461 (Resolved): octopus: notifications not sent when non-owner modifies the bucket
https://github.com/ceph/ceph/pull/40029 Nathan Cutler

12/03/2020

11:32 AM Backport #48429 (In Progress): octopus: rgw: expiration is triggered in advance because of an ove...
Nathan Cutler
11:31 AM Backport #48427 (In Progress): octopus: Put policy should return 204 instead of 200
Nathan Cutler
11:31 AM Backport #48424 (Need More Info): octopus: Able to circumvent S3 Object Lock using deleteobjects ...
feature backport, presumed non-trivial Nathan Cutler
07:11 AM Feature #46296 (Pending Backport): notifications: support GetTopicAttributes API
Yuval Lifshitz
07:10 AM Bug #47904 (Pending Backport): notifications not sent when non-owner modifies the bucket
Yuval Lifshitz

12/02/2020

11:10 AM Backport #48429 (Resolved): octopus: rgw: expiration is triggered in advance because of an overfl...
https://github.com/ceph/ceph/pull/38421 Nathan Cutler
11:10 AM Backport #48428 (Resolved): nautilus: rgw: expiration is triggered in advance because of an overf...
https://github.com/ceph/ceph/pull/38823 Nathan Cutler
11:08 AM Backport #48427 (Resolved): octopus: Put policy should return 204 instead of 200
https://github.com/ceph/ceph/pull/38420 Nathan Cutler
11:08 AM Backport #48426 (Resolved): nautilus: Put policy should return 204 instead of 200
https://github.com/ceph/ceph/pull/38623 Nathan Cutler
11:07 AM Backport #48424 (Resolved): octopus: Able to circumvent S3 Object Lock using deleteobjects command
https://github.com/ceph/ceph/pull/41031 Nathan Cutler
11:07 AM Backport #48423 (Resolved): nautilus: Able to circumvent S3 Object Lock using deleteobjects command
https://github.com/ceph/ceph/pull/41164 Nathan Cutler

12/01/2020

10:09 PM Bug #48418 (New): rgw-admin: segfault during sync pipe modification at diff_sets()
Yehuda Sadeh
10:01 PM Bug #47586 (Pending Backport): Able to circumvent S3 Object Lock using deleteobjects command
J. Eric Ivancich
10:00 PM Bug #45467 (Pending Backport): Put policy should return 204 instead of 200
J. Eric Ivancich
09:58 PM Bug #42634 (Pending Backport): rgw: expiration is triggered in advance because of an overflow pro...
J. Eric Ivancich
09:57 PM Bug #47919 (Pending Backport): list object versions returned multiple 'IsLatest true' entries
J. Eric Ivancich
09:56 PM Bug #44987 (Resolved): rgw: anonymous object should belong to the bucket owner
J. Eric Ivancich
08:43 PM Bug #48416 (Fix Under Review): rgw: bucket sync fails if bucket policy specifies different target...
Yehuda Sadeh
07:01 PM Bug #48415 (New): radosgw-admin: segfault on sync group pipe modify and source/dest zones not spe...
Yehuda Sadeh
01:31 PM Bug #48410 (Fix Under Review): rgw_file: common_prefixes returned out of lexical order
Matt Benjamin
12:53 PM Bug #48410 (Resolved): rgw_file: common_prefixes returned out of lexical order
Since inception RGWReaddirRequest has sent all leaf objects first (in lexical order), then common_prefixes (in lexica... Matt Benjamin
10:19 AM Feature #48402: multisite option to enable keepalive
I wonder if https://tracker.ceph.com/issues/47961 would fix my issue as well... If the connections are closed after ... Dieter Roels
07:17 AM Bug #48001: Brocken SwiftAPI anonymous access
maybe related to #48382 Pietari Hyvärinen

11/30/2020

03:28 PM Feature #48402 (Fix Under Review): multisite option to enable keepalive
We have a multisite setup with firewalls between the two sites. The firewalls silently drop all connections that are... Dieter Roels
02:13 PM Bug #46128 (Fix Under Review): notification: cannot delete a notification from a deleted bucket
Yuval Lifshitz
11:51 AM Backport #48345 (In Progress): octopus: rgw: unnecessary payload is added at the end of the message
Nathan Cutler
11:40 AM Backport #48127 (In Progress): octopus: Unnecessary bilogs are left in sync-disabled buckets
Nathan Cutler

11/29/2020

04:03 PM Bug #48369: notification: get/delete of specific notification does not work
this is working and tested: https://github.com/ceph/ceph/blob/master/src/test/rgw/rgw_multi/tests_ps.py#L953
faili...
Yuval Lifshitz

11/27/2020

10:22 PM Bug #47910: radosgw crash on objecter operations
I completed a first pass of the backport,
and should look at build/test next week.
Mauricio Oliveira
04:42 PM Bug #47910: radosgw crash on objecter operations
Casey, thanks for the pointers!
Not sure how I missed that one.
So, the stack size was increased from 128k to 512...
Mauricio Oliveira
04:32 PM Bug #47910: radosgw crash on objecter operations
this was fixed for octopus as part of https://tracker.ceph.com/issues/43739
it looks like the nautilus backport ht...
Casey Bodley
02:16 PM Bug #47910: radosgw crash on objecter operations
Hi Rafal,
Thanks for testing!
So, the issue indeed does seem to be stack corruption/overflow.
Unfortunately ...
Mauricio Oliveira
01:24 PM Bug #47910: radosgw crash on objecter operations
Or, Mauricio,
I tested radosgw with increased coroutine stack sized to 1M. Everything is working fine.
I think, w...
Rafal Wadolowski
04:29 PM Bug #47866: Object not found on healthy cluster
Janek Bevendorff wrote:
> Is there a roadmap when the fix will be available? Luckily, we haven't lost any more objec...
J. Eric Ivancich

11/26/2020

11:55 AM Bug #48382 (Duplicate): Broken public Swift bucket access with Keystone integration
Public swift bucket access is broken. Prevents upgrading towards 14.2.12 or newer.
In reference to:
https://lis...
Pietari Hyvärinen
11:16 AM Bug #47866 (Resolved): Object not found on healthy cluster
While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ... Nathan Cutler
08:20 AM Bug #47866: Object not found on healthy cluster
Is there a roadmap when the fix will be available? Luckily, we haven't lost any more objects so far (not sure if that... Janek Bevendorff
11:10 AM Bug #48369 (Fix Under Review): notification: get/delete of specific notification does not work
note that this is an extension to the AWS API, for the next steps to work, follow the instructions here:
https://git...
Yuval Lifshitz

11/25/2020

09:47 PM Bug #48363: rgw: concurrent object delete can leak bucket index instance entries
Hi Yehuda,
What Ceph versions are affected?
Matt
Matt Benjamin
07:44 PM Bug #48363 (New): rgw: concurrent object delete can leak bucket index instance entries
This happens when there are concurrent operations, where at least one succeeds, whereas at least another one is cance... Yehuda Sadeh
07:55 PM Bug #48364 (New): rgw: bucket index delete marker instance entry might leak
at rgw_bucket_unlink_instance(), if olh.start_modify() returns false (which means that op.epoch is specified and is l... Yehuda Sadeh
07:53 PM Backport #48331 (Resolved): octopus: Object not found on healthy cluster
This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/38249
m...
Nathan Cutler
04:12 PM Backport #48331: octopus: Object not found on healthy cluster
J. Eric Ivancich wrote:
> https://github.com/ceph/ceph/pull/38249
merged
Yuri Weinstein
01:38 PM Bug #48358: rgw: qlen and qactive perf counters leak
> We'll send a separate PR to expose the throttle `outstanding_requests` values in a new perf counter
https://gith...
Dan van der Ster
12:51 PM Bug #48358 (New): rgw: qlen and qactive perf counters leak
In our environment the rgw qlen and qactive perf counters seem to trend slowly upwards. See the plot attached.
I sus...
Dan van der Ster
09:52 AM Bug #48351: object-lock enabled is not working on 14.2.9 - files still deletable
aws s3api get-object-lock-configuration --bucket locktest --endpoint http://our-s3
{
"ObjectLockConfiguration...
Torsten Ennenbach
09:49 AM Bug #48351: object-lock enabled is not working on 14.2.9 - files still deletable
Added Screenshot of my testfile within this bucket to show the headers
Torsten Ennenbach
09:41 AM Bug #48351 (Closed): object-lock enabled is not working on 14.2.9 - files still deletable
the object lock mentioned in https://github.com/ceph/ceph/pull/29905 - objects within buckets, created with object-lo... Torsten Ennenbach
05:54 AM Bug #48350 (Can't reproduce): [rgw]some multipart object will become orphan objects in data pool
OS: CentOS 7.6
Ceph Version: 14.2.5
Procedure:
1, Use s3 browser upload a big file with multipart upload.
2, Set...
haitao chen

11/24/2020

11:24 PM Bug #45033: S3 multipart copy broken by multi tenant
Still a problem on 14.2.15.
Matt: I just realized I might have misinterpreted you answer, just to clarify, you bel...
Asbjørn Sannes
03:02 PM Backport #48346 (Resolved): nautilus: rgw: unnecessary payload is added at the end of the message
https://github.com/ceph/ceph/pull/38590 Nathan Cutler
03:02 PM Backport #48345 (Resolved): octopus: rgw: unnecessary payload is added at the end of the message
https://github.com/ceph/ceph/pull/38339 Nathan Cutler
08:50 AM Bug #47866: Object not found on healthy cluster
Thank you for sharing that insight with us Pritha and Eric, it clarifies a lot. Denis Krienbühl

11/23/2020

09:42 PM Bug #46257 (In Progress): FAIL: s3tests_boto3.functional.test_s3select
this was fixed in https://github.com/ceph/s3select/pull/14, but the ceph submodule was never updated to include these... Casey Bodley
07:17 PM Bug #48332 (New): rgw: null versioned delete markers removal doesn't sync
null versioned delete marker can exist in suspended versioning buckets. Yehuda Sadeh
06:58 PM Backport #48331 (In Progress): octopus: Object not found on healthy cluster
J. Eric Ivancich
06:52 PM Backport #48331 (Resolved): octopus: Object not found on healthy cluster
https://github.com/ceph/ceph/pull/38249 J. Eric Ivancich
06:22 PM Bug #47866: Object not found on healthy cluster
I exchanged email with Pritha, who is the true expert on modern GC in RGW, to clarify my own questions. I'm pasting t... J. Eric Ivancich
03:24 PM Bug #47866: Object not found on healthy cluster
Denis Krienbühl wrote:
> Pritha Srivastava wrote:
> > Denis Krienbühl wrote:
> > > Pritha Srivastava wrote:
> > >...
Pritha Srivastava
03:10 PM Bug #47866: Object not found on healthy cluster
Pritha Srivastava wrote:
> Denis Krienbühl wrote:
> > Pritha Srivastava wrote:
> > > Please note that this will re...
Denis Krienbühl
02:47 PM Bug #47866: Object not found on healthy cluster
Denis Krienbühl wrote:
> Pritha Srivastava wrote:
> > Please note that this will remove the gc entries before their...
Pritha Srivastava
01:29 PM Bug #47866: Object not found on healthy cluster
Pritha Srivastava wrote:
> Please note that this will remove the gc entries before their expiration time.
What do...
Denis Krienbühl
04:09 AM Bug #47866: Object not found on healthy cluster
If the affected users have increased the value of rgw_gc_obj_min_wait, then before restoring the original expiration ... Pritha Srivastava
06:02 PM Bug #48330 (New): rgw: fix GC defer process and clean up from immediate fix
We had a temporary fix to avert a data loss bug here:
https://tracker.ceph.com/issues/47866
But now GC defe...
J. Eric Ivancich
05:00 PM Bug #48329 (New): cls_rgw_gc_queue_remove_entries() doesn't remove entries from xattr urgent data
cls_rgw_gc_queue_remove_entries() has to scan the entries it's removing from the queue in order to also remove any ur... Casey Bodley
04:48 PM Bug #48328 (New): inefficient check for xattr urgent data in cls_rgw_gc_queue_list_entries()
cls_rgw_gc_queue_list_entries() reads the urgent data map at the beginning as part of read_queue_head(). but if urgen... Casey Bodley
04:22 PM Bug #48321: notifications: tag based filtering works only for object deletions
in "nautilus" only "copy" does not work. so, this should be fixed in nautilus as well, however, since nautilus does n... Yuval Lifshitz
03:48 PM Bug #48327 (Fix Under Review): Infinite loop in old reset-stats
Casey Bodley
03:30 PM Bug #48327 (Resolved): Infinite loop in old reset-stats
from-index was not updated causing a inf loop for buckets > 1000 Abhishek Lekshmanan
03:29 AM Bug #48322: rgw copy obj should check the object size
https://github.com/ceph/ceph/pull/38234 yunqing wang
03:19 AM Bug #48322 (New): rgw copy obj should check the object size
As the doc of boto3 says, copy obj should also check the object size and reject the request when the object is large ... yunqing wang

11/22/2020

07:28 PM Bug #48321 (Pending Backport): notifications: tag based filtering works only for object deletions
tags filtering fails for object "put" and "copy".
tags are not populated into req_state when filtering is done
Yuval Lifshitz
03:37 PM Bug #47866 (Pending Backport): Object not found on healthy cluster
J. Eric Ivancich
08:48 AM Bug #47866: Object not found on healthy cluster
This actually makes a lot of sense.
Here are what we have set in our config:
* rgw_gc_obj_min_wait: 300
* rgw_gc_p...
Julien Demais
09:59 AM Bug #48319 (New): rgw add x-amz-version-id and x-amz-copy-source-version-id for copyobj
joke lee
08:55 AM Bug #48293 (Fix Under Review): notification: radosgw-admin hangs on while closing
Yuval Lifshitz

11/21/2020

09:50 PM Bug #47866: Object not found on healthy cluster
Denis Krienbühl wrote:
> Eric Ivancich wrote:
> > Do you have the means to rebuild your RGWs with the PR attached t...
J. Eric Ivancich
09:40 PM Bug #47866: Object not found on healthy cluster
Gaudenz Steinlin wrote:
> I made some tests in our lab and can confirm the theory. I was able to reproduce this bug....
J. Eric Ivancich
09:31 PM Bug #47866: Object not found on healthy cluster
I made some tests in our lab and can confirm the theory. I was able to reproduce this bug. I used the following setti... Gaudenz Steinlin
08:28 PM Bug #47866: Object not found on healthy cluster
Eric Ivancich wrote:
> Do you have the means to rebuild your RGWs with the PR attached to this tracker? If so that w...
Denis Krienbühl
08:21 PM Bug #47866: Object not found on healthy cluster
Denis Krienbühl wrote:
> Your theory aligns perfectly with our request logs. Here are the longest measured GET reque...
J. Eric Ivancich
07:58 PM Bug #47866: Object not found on healthy cluster
Your theory aligns perfectly with our request logs. Here are the longest measured GET requests for all of the S3 obje... Denis Krienbühl
07:23 PM Bug #47866: Object not found on healthy cluster
Please note that there is now a PR attached to this Tracker. I still have not been able to reproduce, as using my com... J. Eric Ivancich
06:59 PM Bug #47866 (Fix Under Review): Object not found on healthy cluster
J. Eric Ivancich
05:07 PM Bug #47866: Object not found on healthy cluster
IMPORTANT
rgw_gc_obj_min_wait is set in seconds.
So 2 hours, the default, is 7200 seconds.
5 hours is 18000 seco...
J. Eric Ivancich
04:28 PM Bug #47866: Object not found on healthy cluster
Denis Krienbühl wrote:
> Your theory sounds plausible to me. Though if I understand correctly we might be able to ve...
J. Eric Ivancich
04:26 PM Bug #47866: Object not found on healthy cluster
Your theory sounds plausible to me. Though if I understand correctly we might be able to verify this by looking at ou... Denis Krienbühl
04:09 PM Bug #47866: Object not found on healthy cluster
Based on a review of code paths, I have a theory as to why this happens. There's an immediate configuration change th... J. Eric Ivancich

11/20/2020

10:08 PM Bug #48316 (New): rgw: bi entry mtime discrepancy after incremental sync of versioned object
Can happen when the object creation is followed by attribute change (such as acl) on the same object version. The obj... Yehuda Sadeh
09:34 PM Bug #47866: Object not found on healthy cluster
I see, I'll try to run a search for case "a" next week. We have plenty of access logs going back a few months detaili... Denis Krienbühl
09:29 PM Bug #47866: Object not found on healthy cluster
Denis Krienbühl wrote:
> Eric Ivancich wrote:
>
> > a. if you have a bucket that doesn't get deletes then using t...
J. Eric Ivancich
09:26 PM Bug #47866: Object not found on healthy cluster
Sorry to update again, but I want to cover both newer and older gc implementations. The egrep discussed above is now:... J. Eric Ivancich
09:14 PM Bug #47866: Object not found on healthy cluster
Eric Ivancich wrote:
> a. if you have a bucket that doesn't get deletes then using the egrep command above will he...
Denis Krienbühl
08:55 PM Bug #47866: Object not found on healthy cluster
Denis Krienbühl wrote:
> Eric Ivancich wrote:
> > If you have a bucket where things get uploaded and maybe read but...
J. Eric Ivancich
07:43 PM Bug #47866: Object not found on healthy cluster
Eric Ivancich wrote:
> If you have a bucket where things get uploaded and maybe read but rarely deleted, and if you'...
Denis Krienbühl
07:30 PM Bug #47866: Object not found on healthy cluster
Just to clarify on why this is helpful: "If you have a bucket where things get uploaded and maybe read but rarely del... J. Eric Ivancich
07:21 PM Bug #47866: Object not found on healthy cluster
We realize not everyone is able to do this, but for those who can, here is one approach we're using to try to nail th... J. Eric Ivancich
07:14 PM Bug #47866: Object not found on healthy cluster
If the multipart components are themselves larger than the striping size, they two can have shadow tail objects, as s... J. Eric Ivancich
03:04 PM Bug #47866: Object not found on healthy cluster
Update: forget what I said. When I retrieve the ETag via boto3, I get 57ba90377073200f91b5dbee7aaece6-64 instead. I a... Janek Bevendorff
03:03 PM Bug #47866: Object not found on healthy cluster
Thank you, I'm glad to see the @__multipart@ files in there, as that's what I'm counting on.
And just in case some...
Denis Krienbühl
02:56 PM Bug #47866: Object not found on healthy cluster
I crated a bucket and uploaded two 1GB multipart objects. Here's what the radoslist looks like:... Janek Bevendorff
01:55 PM Bug #47866: Object not found on healthy cluster
Update: It seems like @refcount.put@ is the call that logs the @__multipart@ file about to be deleted. Denis Krienbühl
01:18 PM Bug #47866: Object not found on healthy cluster
Janek Bevendorff wrote:
> Yes, it is uploading multiple parts and the resulting RADOS objects live in the multipart ...
Denis Krienbühl
12:55 PM Bug #47866: Object not found on healthy cluster
Yes, it is uploading multiple parts and the resulting RADOS objects live in the multipart namespace, not the shadow n... Janek Bevendorff
12:39 PM Bug #47866: Object not found on healthy cluster
Janek Bevendorff wrote:
> I wanted to reupload our multipart objects as single parts and noticed something very stra...
Denis Krienbühl
12:03 PM Bug #47866: Object not found on healthy cluster
I wanted to reupload our multipart objects as single parts and noticed something very strange: Not all multipart obje... Janek Bevendorff
08:55 AM Bug #47866: Object not found on healthy cluster
Sorry if that was unclear. The majority of all objects we have in total are single-part objects. The affected objects... Janek Bevendorff
08:34 AM Bug #47866: Object not found on healthy cluster
Janek Bevendorff wrote:
> I don't think the issue is packet loss per se. The objects in our affected buckets were al...
Denis Krienbühl
08:34 AM Bug #47866: Object not found on healthy cluster
Janek Bevendorff wrote:
> I don't think the issue is packet loss per se. The objects in our affected buckets were al...
Denis Krienbühl
08:30 AM Bug #47866: Object not found on healthy cluster
I don't think the issue is packet loss per se. The objects in our affected buckets were all filled by a Hadoop job ru... Janek Bevendorff
08:12 AM Bug #47866: Object not found on healthy cluster
Another update from our side: We have two sites, a larger and a smaller one. The smaller site was successfully scanne... Denis Krienbühl
04:56 PM Bug #47800 (Resolved): rgw/tempest test failure in test_object_upload_in_segments
Casey Bodley
04:56 PM Bug #47801 (Resolved): rgw/tempest test failure in test_get_object_with_x_object_manifest
Casey Bodley
12:43 PM Bug #47910: radosgw crash on objecter operations
Thanks for confirming on large buckets/EC pools.
Correct; that patch is the only change in the ceph packages in th...
Mauricio Oliveira
06:00 AM Bug #47910: radosgw crash on objecter operations
Both of them.
I checked your links. I should implement https://github.com/cbodley/ceph/commit/d23507bd1295a29ccae3...
Rafal Wadolowski
12:23 AM Bug #47910: radosgw crash on objecter operations
Hey Rafal,
Thanks!
By the way, does your ceph cluster have:
1) bucket(s) with a large number of objects?
2)...
Mauricio Oliveira
02:01 AM Bug #48289 (Fix Under Review): prefetch in rgw_file is done 3 times per read
Matt Benjamin

11/19/2020

05:29 PM Documentation #23027: document rgw_multipart_min_part_size

There are over 1500 options in the source as of today. `rgw_multipart_min_part_size` is marked as `ADVANCED`, and I...
Anthony D'Atri
04:57 PM Bug #47588 (Resolved): sts test failures
Casey Bodley
04:02 PM Bug #48289: prefetch in rgw_file is done 3 times per read
please backport the fix to Nautilus Kajetan Janiak
09:48 AM Bug #48289 (Resolved): prefetch in rgw_file is done 3 times per read
Each call to rgw_read (rgw_file.cc) invokes three calls to RGWRados::get_obj_state with RGWObjState::prefetch_data = ... Kajetan Janiak
03:30 PM Bug #47984: rgw: filter marker in list vesions
lei cao wrote:
> https://github.com/ceph/ceph/pull/37848
This PR has been closed as a result of another PR furth...
J. Eric Ivancich
03:29 PM Bug #47984 (In Progress): rgw: filter marker in list vesions
A different PR is close to a solution, so the PR is updated.
https://github.com/ceph/ceph/pull/37790
J. Eric Ivancich
03:24 PM Bug #48064 (Pending Backport): rgw: unnecessary payload is added at the end of the message
Or Friedmann
02:41 PM Bug #48293 (Resolved): notification: radosgw-admin hangs on while closing
when "data sync run" command is executed on a pubsub zone, the code creates the rabbitmq manager (as part initializin... Yuval Lifshitz
01:23 PM Bug #48241: librgw double read
Keep in mind that if you're seeing the same read dispatched from nfs-ganesha twice, it most likely was issued twice. ... Matt Benjamin
01:10 PM Bug #48241: librgw double read
Please be advised, librgw is somewhat broken on master, mostly due to zipper changes. The known issues are fixed in ... Matt Benjamin
10:14 AM Bug #48241: librgw double read
Or Friedmann wrote:
> Is it possible to add the osd.log with debug_objclass 20 and debug_ms 1? (I want to see the cl...
chunsong feng
11:12 AM Backport #47836: octopus: Add OIDC provider support in RGW STS
ok, thank you for the clarification. Pritha Srivastava
11:10 AM Backport #47836: octopus: Add OIDC provider support in RGW STS
Pritha Srivastava wrote:
> The target version for this PR is v15.2.6, but when I check the v15.2.6 release I do not ...
Nathan Cutler
06:57 AM Backport #47836: octopus: Add OIDC provider support in RGW STS
Hi @Nathan,
The target version for this PR is v15.2.6, but when I check the v15.2.6 release I do not see the relat...
Pritha Srivastava
10:26 AM Bug #47866: Object not found on healthy cluster
An update on our efforts: We enabled @debug_ms 1@ and we are trying to find an object that vanishes while these logs ... Denis Krienbühl
10:21 AM Backport #48101 (In Progress): octopus: Admin API returns 200 instead of 404 for Get Bucket Info
Nathan Cutler
07:43 AM Bug #47910: radosgw crash on objecter operations
Hi Mauricio,
I am using package from download.ceph.com.
I can test the fix, I will back with the answers until en...
Rafal Wadolowski
02:19 AM Documentation #25061: Wrong figure in multisite documentation
Which figure? What's wrong with it? Anthony D'Atri

11/18/2020

05:11 PM Feature #46296 (Fix Under Review): notifications: support GetTopicAttributes API
Yuval Lifshitz
05:11 PM Bug #47910: radosgw crash on objecter operations
Hi Rafal and Or,
Rafal,
Could you please confirm if you are using ceph packages from Ubuntu?
(say, Ubuntu Clou...
Mauricio Oliveira
04:19 PM Bug #47866: Object not found on healthy cluster
We're hitting this problem in our 15.2.5 cluster as well. So far we know of one S3 that went missing, in a typical Do... Denis Krienbühl
01:07 PM Bug #47866: Object not found on healthy cluster
I believe client-rgw is doing the job. The default here seems to be 0/0 and when I turn it back off, I have nothing i... Janek Bevendorff
12:59 PM Bug #47866: Object not found on healthy cluster
Janek Bevendorff wrote:
> The config says the default for rgw is 1/5, that's why I was wondering. https://docs.ceph....
Dan van der Ster
12:54 PM Bug #47866: Object not found on healthy cluster
The config says the default for rgw is 1/5, that's why I was wondering. https://docs.ceph.com/en/latest/rados/trouble... Janek Bevendorff
10:21 AM Bug #47866: Object not found on healthy cluster
Janek Bevendorff wrote:
> I have enabled the setting via
>
> [...]
>
> I will check the logs when the next obj...
Dan van der Ster
10:20 AM Bug #47866: Object not found on healthy cluster
Janek Bevendorff wrote:
> Short question here: isn't 1 the default log level anyway? I couldn't find any such messag...
Dan van der Ster
08:33 AM Bug #47866: Object not found on healthy cluster
Short question here: isn't 1 the default log level anyway? I couldn't find any such messages in previous logs at least. Janek Bevendorff
08:28 AM Bug #47866: Object not found on healthy cluster
I have enabled the setting via... Janek Bevendorff
04:25 AM Bug #47866: Object not found on healthy cluster
In Comment #11 I asked: Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'?
Let me fol...
J. Eric Ivancich
09:38 AM Bug #48241: librgw double read
Or Friedmann wrote:
> Is it possible to add the osd.log with debug_objclass 20 and debug_ms 1? (I want to see the cl...
chunsong feng
 

Also available in: Atom