Activity
From 11/18/2020 to 12/17/2020
12/17/2020
- 06:19 PM Backport #48127: octopus: Unnecessary bilogs are left in sync-disabled buckets
- Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/38338
merged - 06:18 PM Bug #48571: Multi part upload abort on first put with unknown error -2009
- Casey Bodley wrote:
> what is the s3 client? does it support multipart uploads?
The S3 client, as stated at the b... - 04:50 PM Bug #48571: Multi part upload abort on first put with unknown error -2009
- I am not sure what exactly you are looking for, but here are the logs for the two rgw server, both in debug log level...
- 03:16 PM Bug #48571 (Need More Info): Multi part upload abort on first put with unknown error -2009
- 03:16 PM Bug #48571: Multi part upload abort on first put with unknown error -2009
- what is the s3 client? does it support multipart uploads?
- 03:13 PM Bug #48571: Multi part upload abort on first put with unknown error -2009
- Can you provide an rgw log segment which shows the required init-multipart-upload step which needs to run before any ...
- 03:10 PM Bug #48571: Multi part upload abort on first put with unknown error -2009
- rgw_common.h:#define ERR_NO_SUCH_UPLOAD 2009
rgw_common.cc: { ERR_NO_SUCH_UPLOAD, {404, "NoSuchUpload" }}, - 06:05 PM Backport #48660 (Rejected): octopus: notifications: only bucket owner is allowed to add/remove no...
- 06:05 PM Backport #48659 (Resolved): octopus: notification: radosgw-admin hangs on while closing
- https://github.com/ceph/ceph/pull/40381
- 06:04 PM Bug #48250 (Pending Backport): notifications: only bucket owner is allowed to add/remove notifica...
- 06:03 PM Bug #46243 (Resolved): notification: add more examples notification boto3 README
- 06:00 PM Backport #48658 (Rejected): octopus: notification: cannot delete a notification from a deleted bu...
- 06:00 PM Bug #48293 (Pending Backport): notification: radosgw-admin hangs on while closing
- 06:00 PM Backport #48657 (Rejected): octopus: notifications: tag based filtering works only for object del...
- 05:58 PM Bug #48321 (Pending Backport): notifications: tag based filtering works only for object deletions
- 05:57 PM Bug #46128 (Pending Backport): notification: cannot delete a notification from a deleted bucket
- 03:28 PM Bug #48351: object-lock enabled is not working on 14.2.9 - files still deletable
- Matt to reassign
- 03:24 PM Bug #48472 (Need More Info): rgw crash when get none exist role policy
- 03:23 PM Bug #48554 (Fix Under Review): When copying an encrypted object, the result object is empty.
- 10:45 AM Bug #48646 (Closed): Bucket operations an issue with C# AWSSDK.S3 client
- ## Description
Bucket operations an issue with C# AWSSDK.S3 client
We are having issues with bucket operations with...
12/16/2020
- 06:35 PM Backport #48426 (In Progress): nautilus: Put policy should return 204 instead of 200
- 06:34 PM Backport #48423 (Need More Info): nautilus: Able to circumvent S3 Object Lock using deleteobjects...
- complicated backport
- 04:18 PM Bug #48632 (Resolved): S3 error: 404 (NoSuchBucket) due to distribute cache is not being invoked
- Reproducer:
aws s3 cp s3://hello/hosts abc --endpoint-url http://<rgw1>
aws s3 mb s3://hello --endpoint-url http:...
12/15/2020
- 02:22 PM Backport #48346 (In Progress): nautilus: rgw: unnecessary payload is added at the end of the message
- 02:19 PM Backport #48128 (In Progress): nautilus: Unnecessary bilogs are left in sync-disabled buckets
- 02:18 PM Backport #48100 (In Progress): nautilus: Admin API returns 200 instead of 404 for Get Bucket Info
- 01:46 PM Bug #48571: Multi part upload abort on first put with unknown error -2009
- I am sorry, the end of my message was lost during edition.
We can see in the rgw output that an unknown error is h...
12/14/2020
- 08:30 AM Bug #47910: radosgw crash on objecter operations
- Hi Mauricio,
I planned to test backport this week, I will be back with results.
Rafal
- 12:43 AM Bug #48554: When copying an encrypted object, the result object is empty.
- https://github.com/ceph/ceph/pull/38537
- 12:42 AM Bug #48556: UploadPartCopy uses the wrong headers
- https://github.com/ceph/ceph/pull/38541
12/13/2020
12/12/2020
- 02:39 PM Backport #48238 (Resolved): nautilus: list object versions returned multiple 'IsLatest true' entries
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/38085
m... - 01:06 AM Bug #47910: radosgw crash on objecter operations
- Hi Rafal,
Please excuse this message if you're already out for vacation/holiday season.
Just following up if yo... - 01:00 AM Backport #48576 (Resolved): nautilus: RGW prefetches data for range requests
- https://github.com/ceph/ceph/pull/38556
- 12:58 AM Bug #44508 (Pending Backport): RGW prefetches data for range requests
- 12:57 AM Bug #44508: RGW prefetches data for range requests
- This issue is also present in nautilus. I've added it to the backport list for consideration.
12/11/2020
- 09:19 PM Bug #48571 (Can't reproduce): Multi part upload abort on first put with unknown error -2009
- Hello,
We are trying to configure IBM Cloud Tape Connector to do backup on our Cephs<s S3. Initial Connection hap... - 07:59 PM Backport #48238: nautilus: list object versions returned multiple 'IsLatest true' entries
- https://github.com/ceph/ceph/pull/38085 merged
- 07:21 AM Bug #48556 (New): UploadPartCopy uses the wrong headers
- to create an part from an encrypted object, RadosGW uses the following headers:
* x-amz-server-side-encryption-custo... - 05:31 AM Bug #48472: rgw crash when get none exist role policy
- Hi @joke lee,
I tried the following:
radosgw-admin user create --tenant t1tenant --uid t1user --display-name t1... - 12:55 AM Bug #48554 (Resolved): When copying an encrypted object, the result object is empty.
- When I use UploadPartCopyRequest to copy an encrypted object without a key, the request succeeds, but the result obje...
12/10/2020
- 03:16 PM Backport #48548 (Rejected): octopus: Access denied for multi-object-delete by non-bucket-owner
- 03:16 PM Backport #48547 (Rejected): nautilus: Access denied for multi-object-delete by non-bucket-owner
- 03:15 PM Backport #48546 (Resolved): octopus: rgwlc: shard-index vector short by 1?
- https://github.com/ceph/ceph/pull/38824
- 03:15 PM Backport #48545 (Rejected): nautilus: rgwlc: shard-index vector short by 1?
- 03:15 PM Backport #48544 (Resolved): octopus: rgw_file: common_prefixes returned out of lexical order
- https://github.com/ceph/ceph/pull/38829
- 03:15 PM Backport #48543 (Resolved): nautilus: rgw_file: common_prefixes returned out of lexical order
- https://github.com/ceph/ceph/pull/38828
- 03:12 PM Bug #48410 (Pending Backport): rgw_file: common_prefixes returned out of lexical order
- 03:08 PM Bug #46567 (Pending Backport): Access denied for multi-object-delete by non-bucket-owner
- 03:06 PM Bug #48255 (Pending Backport): rgwlc: shard-index vector short by 1?
- 01:04 AM Bug #48363: rgw: concurrent object delete can leak bucket index instance entries
- Matt Benjamin wrote:
> Hi Yehuda,
>
> What Ceph versions are affected?
>
> Matt
we ran into the same proble...
12/09/2020
- 11:35 AM Feature #48513: uses librgw2 to directly access the rados cluster for hadoop
- The adapter invokes the librgw2 interface through JNI. The following interfaces need to be implemented:
static const... - 11:32 AM Feature #48513 (New): uses librgw2 to directly access the rados cluster for hadoop
- Hadoop S3 accesses Ceph through the object gateway. The radosgw daemon process needs to be deployed. Messages are sen...
12/08/2020
- 06:47 PM Bug #47074: rgw: bucket radoslist looping endlessly while processing a bucket
- I'm experiencing the same issue in my environment. I recently attempted to run the ‘rgw-orphan-list’ tool against our...
- 04:23 PM Fix #48499 (Resolved): nautilus: rgw: in ordered bucket listing skip namespaced entries internall...
- When doing an ordered bucket listing, such as from s3, we're listing all the entries that are not in a namespace. Non...
12/07/2020
- 01:26 AM Bug #48472 (Need More Info): rgw crash when get none exist role policy
- rgw will crash when get put delete pocliy to a none exist role
import boto3, json
import botocore
botocore.session...
12/04/2020
- 09:04 PM Bug #47910: radosgw crash on objecter operations
- Hi Rafal, Casey,
I've completed the backport on the nautilus branch, and tested with the steps from 43739 [1] usin... - 11:08 AM Backport #48462 (Resolved): octopus: notifications: support GetTopicAttributes API
- https://github.com/ceph/ceph/pull/40812
- 11:08 AM Backport #48461 (Resolved): octopus: notifications not sent when non-owner modifies the bucket
- https://github.com/ceph/ceph/pull/40029
12/03/2020
- 11:32 AM Backport #48429 (In Progress): octopus: rgw: expiration is triggered in advance because of an ove...
- 11:31 AM Backport #48427 (In Progress): octopus: Put policy should return 204 instead of 200
- 11:31 AM Backport #48424 (Need More Info): octopus: Able to circumvent S3 Object Lock using deleteobjects ...
- feature backport, presumed non-trivial
- 07:11 AM Feature #46296 (Pending Backport): notifications: support GetTopicAttributes API
- 07:10 AM Bug #47904 (Pending Backport): notifications not sent when non-owner modifies the bucket
12/02/2020
- 11:10 AM Backport #48429 (Resolved): octopus: rgw: expiration is triggered in advance because of an overfl...
- https://github.com/ceph/ceph/pull/38421
- 11:10 AM Backport #48428 (Resolved): nautilus: rgw: expiration is triggered in advance because of an overf...
- https://github.com/ceph/ceph/pull/38823
- 11:08 AM Backport #48427 (Resolved): octopus: Put policy should return 204 instead of 200
- https://github.com/ceph/ceph/pull/38420
- 11:08 AM Backport #48426 (Resolved): nautilus: Put policy should return 204 instead of 200
- https://github.com/ceph/ceph/pull/38623
- 11:07 AM Backport #48424 (Resolved): octopus: Able to circumvent S3 Object Lock using deleteobjects command
- https://github.com/ceph/ceph/pull/41031
- 11:07 AM Backport #48423 (Resolved): nautilus: Able to circumvent S3 Object Lock using deleteobjects command
- https://github.com/ceph/ceph/pull/41164
12/01/2020
- 10:09 PM Bug #48418 (New): rgw-admin: segfault during sync pipe modification at diff_sets()
- 10:01 PM Bug #47586 (Pending Backport): Able to circumvent S3 Object Lock using deleteobjects command
- 10:00 PM Bug #45467 (Pending Backport): Put policy should return 204 instead of 200
- 09:58 PM Bug #42634 (Pending Backport): rgw: expiration is triggered in advance because of an overflow pro...
- 09:57 PM Bug #47919 (Pending Backport): list object versions returned multiple 'IsLatest true' entries
- 09:56 PM Bug #44987 (Resolved): rgw: anonymous object should belong to the bucket owner
- 08:43 PM Bug #48416 (Fix Under Review): rgw: bucket sync fails if bucket policy specifies different target...
- 07:01 PM Bug #48415 (New): radosgw-admin: segfault on sync group pipe modify and source/dest zones not spe...
- 01:31 PM Bug #48410 (Fix Under Review): rgw_file: common_prefixes returned out of lexical order
- 12:53 PM Bug #48410 (Resolved): rgw_file: common_prefixes returned out of lexical order
- Since inception RGWReaddirRequest has sent all leaf objects first (in lexical order), then common_prefixes (in lexica...
- 10:19 AM Feature #48402: multisite option to enable keepalive
- I wonder if https://tracker.ceph.com/issues/47961 would fix my issue as well... If the connections are closed after ...
- 07:17 AM Bug #48001: Brocken SwiftAPI anonymous access
- maybe related to #48382
11/30/2020
- 03:28 PM Feature #48402 (Fix Under Review): multisite option to enable keepalive
- We have a multisite setup with firewalls between the two sites. The firewalls silently drop all connections that are...
- 02:13 PM Bug #46128 (Fix Under Review): notification: cannot delete a notification from a deleted bucket
- 11:51 AM Backport #48345 (In Progress): octopus: rgw: unnecessary payload is added at the end of the message
- 11:40 AM Backport #48127 (In Progress): octopus: Unnecessary bilogs are left in sync-disabled buckets
11/29/2020
- 04:03 PM Bug #48369: notification: get/delete of specific notification does not work
- this is working and tested: https://github.com/ceph/ceph/blob/master/src/test/rgw/rgw_multi/tests_ps.py#L953
faili...
11/27/2020
- 10:22 PM Bug #47910: radosgw crash on objecter operations
- I completed a first pass of the backport,
and should look at build/test next week. - 04:42 PM Bug #47910: radosgw crash on objecter operations
- Casey, thanks for the pointers!
Not sure how I missed that one.
So, the stack size was increased from 128k to 512... - 04:32 PM Bug #47910: radosgw crash on objecter operations
- this was fixed for octopus as part of https://tracker.ceph.com/issues/43739
it looks like the nautilus backport ht... - 02:16 PM Bug #47910: radosgw crash on objecter operations
- Hi Rafal,
Thanks for testing!
So, the issue indeed does seem to be stack corruption/overflow.
Unfortunately ... - 01:24 PM Bug #47910: radosgw crash on objecter operations
- Or, Mauricio,
I tested radosgw with increased coroutine stack sized to 1M. Everything is working fine.
I think, w... - 04:29 PM Bug #47866: Object not found on healthy cluster
- Janek Bevendorff wrote:
> Is there a roadmap when the fix will be available? Luckily, we haven't lost any more objec...
11/26/2020
- 11:55 AM Bug #48382 (Duplicate): Broken public Swift bucket access with Keystone integration
- Public swift bucket access is broken. Prevents upgrading towards 14.2.12 or newer.
In reference to:
https://lis... - 11:16 AM Bug #47866 (Resolved): Object not found on healthy cluster
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 08:20 AM Bug #47866: Object not found on healthy cluster
- Is there a roadmap when the fix will be available? Luckily, we haven't lost any more objects so far (not sure if that...
- 11:10 AM Bug #48369 (Fix Under Review): notification: get/delete of specific notification does not work
- note that this is an extension to the AWS API, for the next steps to work, follow the instructions here:
https://git...
11/25/2020
- 09:47 PM Bug #48363: rgw: concurrent object delete can leak bucket index instance entries
- Hi Yehuda,
What Ceph versions are affected?
Matt - 07:44 PM Bug #48363 (New): rgw: concurrent object delete can leak bucket index instance entries
- This happens when there are concurrent operations, where at least one succeeds, whereas at least another one is cance...
- 07:55 PM Bug #48364 (New): rgw: bucket index delete marker instance entry might leak
- at rgw_bucket_unlink_instance(), if olh.start_modify() returns false (which means that op.epoch is specified and is l...
- 07:53 PM Backport #48331 (Resolved): octopus: Object not found on healthy cluster
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/38249
m... - 04:12 PM Backport #48331: octopus: Object not found on healthy cluster
- J. Eric Ivancich wrote:
> https://github.com/ceph/ceph/pull/38249
merged - 01:38 PM Bug #48358: rgw: qlen and qactive perf counters leak
- > We'll send a separate PR to expose the throttle `outstanding_requests` values in a new perf counter
https://gith... - 12:51 PM Bug #48358 (New): rgw: qlen and qactive perf counters leak
- In our environment the rgw qlen and qactive perf counters seem to trend slowly upwards. See the plot attached.
I sus... - 09:52 AM Bug #48351: object-lock enabled is not working on 14.2.9 - files still deletable
- aws s3api get-object-lock-configuration --bucket locktest --endpoint http://our-s3
{
"ObjectLockConfiguration... - 09:49 AM Bug #48351: object-lock enabled is not working on 14.2.9 - files still deletable
- Added Screenshot of my testfile within this bucket to show the headers
- 09:41 AM Bug #48351 (Closed): object-lock enabled is not working on 14.2.9 - files still deletable
- the object lock mentioned in https://github.com/ceph/ceph/pull/29905 - objects within buckets, created with object-lo...
- 05:54 AM Bug #48350 (Can't reproduce): [rgw]some multipart object will become orphan objects in data pool
- OS: CentOS 7.6
Ceph Version: 14.2.5
Procedure:
1, Use s3 browser upload a big file with multipart upload.
2, Set...
11/24/2020
- 11:24 PM Bug #45033: S3 multipart copy broken by multi tenant
- Still a problem on 14.2.15.
Matt: I just realized I might have misinterpreted you answer, just to clarify, you bel... - 03:02 PM Backport #48346 (Resolved): nautilus: rgw: unnecessary payload is added at the end of the message
- https://github.com/ceph/ceph/pull/38590
- 03:02 PM Backport #48345 (Resolved): octopus: rgw: unnecessary payload is added at the end of the message
- https://github.com/ceph/ceph/pull/38339
- 08:50 AM Bug #47866: Object not found on healthy cluster
- Thank you for sharing that insight with us Pritha and Eric, it clarifies a lot.
11/23/2020
- 09:42 PM Bug #46257 (In Progress): FAIL: s3tests_boto3.functional.test_s3select
- this was fixed in https://github.com/ceph/s3select/pull/14, but the ceph submodule was never updated to include these...
- 07:17 PM Bug #48332 (New): rgw: null versioned delete markers removal doesn't sync
- null versioned delete marker can exist in suspended versioning buckets.
- 06:58 PM Backport #48331 (In Progress): octopus: Object not found on healthy cluster
- 06:52 PM Backport #48331 (Resolved): octopus: Object not found on healthy cluster
- https://github.com/ceph/ceph/pull/38249
- 06:22 PM Bug #47866: Object not found on healthy cluster
- I exchanged email with Pritha, who is the true expert on modern GC in RGW, to clarify my own questions. I'm pasting t...
- 03:24 PM Bug #47866: Object not found on healthy cluster
- Denis Krienbühl wrote:
> Pritha Srivastava wrote:
> > Denis Krienbühl wrote:
> > > Pritha Srivastava wrote:
> > >... - 03:10 PM Bug #47866: Object not found on healthy cluster
- Pritha Srivastava wrote:
> Denis Krienbühl wrote:
> > Pritha Srivastava wrote:
> > > Please note that this will re... - 02:47 PM Bug #47866: Object not found on healthy cluster
- Denis Krienbühl wrote:
> Pritha Srivastava wrote:
> > Please note that this will remove the gc entries before their... - 01:29 PM Bug #47866: Object not found on healthy cluster
- Pritha Srivastava wrote:
> Please note that this will remove the gc entries before their expiration time.
What do... - 04:09 AM Bug #47866: Object not found on healthy cluster
- If the affected users have increased the value of rgw_gc_obj_min_wait, then before restoring the original expiration ...
- 06:02 PM Bug #48330 (New): rgw: fix GC defer process and clean up from immediate fix
- We had a temporary fix to avert a data loss bug here:
https://tracker.ceph.com/issues/47866
But now GC defe... - 05:00 PM Bug #48329 (New): cls_rgw_gc_queue_remove_entries() doesn't remove entries from xattr urgent data
- cls_rgw_gc_queue_remove_entries() has to scan the entries it's removing from the queue in order to also remove any ur...
- 04:48 PM Bug #48328 (New): inefficient check for xattr urgent data in cls_rgw_gc_queue_list_entries()
- cls_rgw_gc_queue_list_entries() reads the urgent data map at the beginning as part of read_queue_head(). but if urgen...
- 04:22 PM Bug #48321: notifications: tag based filtering works only for object deletions
- in "nautilus" only "copy" does not work. so, this should be fixed in nautilus as well, however, since nautilus does n...
- 03:48 PM Bug #48327 (Fix Under Review): Infinite loop in old reset-stats
- 03:30 PM Bug #48327 (Resolved): Infinite loop in old reset-stats
- from-index was not updated causing a inf loop for buckets > 1000
- 03:29 AM Bug #48322: rgw copy obj should check the object size
- https://github.com/ceph/ceph/pull/38234
- 03:19 AM Bug #48322 (New): rgw copy obj should check the object size
- As the doc of boto3 says, copy obj should also check the object size and reject the request when the object is large ...
11/22/2020
- 07:28 PM Bug #48321 (Pending Backport): notifications: tag based filtering works only for object deletions
- tags filtering fails for object "put" and "copy".
tags are not populated into req_state when filtering is done - 03:37 PM Bug #47866 (Pending Backport): Object not found on healthy cluster
- 08:48 AM Bug #47866: Object not found on healthy cluster
- This actually makes a lot of sense.
Here are what we have set in our config:
* rgw_gc_obj_min_wait: 300
* rgw_gc_p... - 09:59 AM Bug #48319 (New): rgw add x-amz-version-id and x-amz-copy-source-version-id for copyobj
- 08:55 AM Bug #48293 (Fix Under Review): notification: radosgw-admin hangs on while closing
11/21/2020
- 09:50 PM Bug #47866: Object not found on healthy cluster
- Denis Krienbühl wrote:
> Eric Ivancich wrote:
> > Do you have the means to rebuild your RGWs with the PR attached t... - 09:40 PM Bug #47866: Object not found on healthy cluster
- Gaudenz Steinlin wrote:
> I made some tests in our lab and can confirm the theory. I was able to reproduce this bug.... - 09:31 PM Bug #47866: Object not found on healthy cluster
- I made some tests in our lab and can confirm the theory. I was able to reproduce this bug. I used the following setti...
- 08:28 PM Bug #47866: Object not found on healthy cluster
- Eric Ivancich wrote:
> Do you have the means to rebuild your RGWs with the PR attached to this tracker? If so that w... - 08:21 PM Bug #47866: Object not found on healthy cluster
- Denis Krienbühl wrote:
> Your theory aligns perfectly with our request logs. Here are the longest measured GET reque... - 07:58 PM Bug #47866: Object not found on healthy cluster
- Your theory aligns perfectly with our request logs. Here are the longest measured GET requests for all of the S3 obje...
- 07:23 PM Bug #47866: Object not found on healthy cluster
- Please note that there is now a PR attached to this Tracker. I still have not been able to reproduce, as using my com...
- 06:59 PM Bug #47866 (Fix Under Review): Object not found on healthy cluster
- 05:07 PM Bug #47866: Object not found on healthy cluster
- IMPORTANT
rgw_gc_obj_min_wait is set in seconds.
So 2 hours, the default, is 7200 seconds.
5 hours is 18000 seco... - 04:28 PM Bug #47866: Object not found on healthy cluster
- Denis Krienbühl wrote:
> Your theory sounds plausible to me. Though if I understand correctly we might be able to ve... - 04:26 PM Bug #47866: Object not found on healthy cluster
- Your theory sounds plausible to me. Though if I understand correctly we might be able to verify this by looking at ou...
- 04:09 PM Bug #47866: Object not found on healthy cluster
- Based on a review of code paths, I have a theory as to why this happens. There's an immediate configuration change th...
11/20/2020
- 10:08 PM Bug #48316 (New): rgw: bi entry mtime discrepancy after incremental sync of versioned object
- Can happen when the object creation is followed by attribute change (such as acl) on the same object version. The obj...
- 09:34 PM Bug #47866: Object not found on healthy cluster
- I see, I'll try to run a search for case "a" next week. We have plenty of access logs going back a few months detaili...
- 09:29 PM Bug #47866: Object not found on healthy cluster
- Denis Krienbühl wrote:
> Eric Ivancich wrote:
>
> > a. if you have a bucket that doesn't get deletes then using t... - 09:26 PM Bug #47866: Object not found on healthy cluster
- Sorry to update again, but I want to cover both newer and older gc implementations. The egrep discussed above is now:...
- 09:14 PM Bug #47866: Object not found on healthy cluster
- Eric Ivancich wrote:
> a. if you have a bucket that doesn't get deletes then using the egrep command above will he... - 08:55 PM Bug #47866: Object not found on healthy cluster
- Denis Krienbühl wrote:
> Eric Ivancich wrote:
> > If you have a bucket where things get uploaded and maybe read but... - 07:43 PM Bug #47866: Object not found on healthy cluster
- Eric Ivancich wrote:
> If you have a bucket where things get uploaded and maybe read but rarely deleted, and if you'... - 07:30 PM Bug #47866: Object not found on healthy cluster
- Just to clarify on why this is helpful: "If you have a bucket where things get uploaded and maybe read but rarely del...
- 07:21 PM Bug #47866: Object not found on healthy cluster
- We realize not everyone is able to do this, but for those who can, here is one approach we're using to try to nail th...
- 07:14 PM Bug #47866: Object not found on healthy cluster
- If the multipart components are themselves larger than the striping size, they two can have shadow tail objects, as s...
- 03:04 PM Bug #47866: Object not found on healthy cluster
- Update: forget what I said. When I retrieve the ETag via boto3, I get 57ba90377073200f91b5dbee7aaece6-64 instead. I a...
- 03:03 PM Bug #47866: Object not found on healthy cluster
- Thank you, I'm glad to see the @__multipart@ files in there, as that's what I'm counting on.
And just in case some... - 02:56 PM Bug #47866: Object not found on healthy cluster
- I crated a bucket and uploaded two 1GB multipart objects. Here's what the radoslist looks like:...
- 01:55 PM Bug #47866: Object not found on healthy cluster
- Update: It seems like @refcount.put@ is the call that logs the @__multipart@ file about to be deleted.
- 01:18 PM Bug #47866: Object not found on healthy cluster
- Janek Bevendorff wrote:
> Yes, it is uploading multiple parts and the resulting RADOS objects live in the multipart ... - 12:55 PM Bug #47866: Object not found on healthy cluster
- Yes, it is uploading multiple parts and the resulting RADOS objects live in the multipart namespace, not the shadow n...
- 12:39 PM Bug #47866: Object not found on healthy cluster
- Janek Bevendorff wrote:
> I wanted to reupload our multipart objects as single parts and noticed something very stra... - 12:03 PM Bug #47866: Object not found on healthy cluster
- I wanted to reupload our multipart objects as single parts and noticed something very strange: Not all multipart obje...
- 08:55 AM Bug #47866: Object not found on healthy cluster
- Sorry if that was unclear. The majority of all objects we have in total are single-part objects. The affected objects...
- 08:34 AM Bug #47866: Object not found on healthy cluster
- Janek Bevendorff wrote:
> I don't think the issue is packet loss per se. The objects in our affected buckets were al... - 08:34 AM Bug #47866: Object not found on healthy cluster
- Janek Bevendorff wrote:
> I don't think the issue is packet loss per se. The objects in our affected buckets were al... - 08:30 AM Bug #47866: Object not found on healthy cluster
- I don't think the issue is packet loss per se. The objects in our affected buckets were all filled by a Hadoop job ru...
- 08:12 AM Bug #47866: Object not found on healthy cluster
- Another update from our side: We have two sites, a larger and a smaller one. The smaller site was successfully scanne...
- 04:56 PM Bug #47800 (Resolved): rgw/tempest test failure in test_object_upload_in_segments
- 04:56 PM Bug #47801 (Resolved): rgw/tempest test failure in test_get_object_with_x_object_manifest
- 12:43 PM Bug #47910: radosgw crash on objecter operations
- Thanks for confirming on large buckets/EC pools.
Correct; that patch is the only change in the ceph packages in th... - 06:00 AM Bug #47910: radosgw crash on objecter operations
- Both of them.
I checked your links. I should implement https://github.com/cbodley/ceph/commit/d23507bd1295a29ccae3... - 12:23 AM Bug #47910: radosgw crash on objecter operations
- Hey Rafal,
Thanks!
By the way, does your ceph cluster have:
1) bucket(s) with a large number of objects?
2)... - 02:01 AM Bug #48289 (Fix Under Review): prefetch in rgw_file is done 3 times per read
11/19/2020
- 05:29 PM Documentation #23027: document rgw_multipart_min_part_size
There are over 1500 options in the source as of today. `rgw_multipart_min_part_size` is marked as `ADVANCED`, and I...- 04:57 PM Bug #47588 (Resolved): sts test failures
- 04:02 PM Bug #48289: prefetch in rgw_file is done 3 times per read
- please backport the fix to Nautilus
- 09:48 AM Bug #48289 (Resolved): prefetch in rgw_file is done 3 times per read
- Each call to rgw_read (rgw_file.cc) invokes three calls to RGWRados::get_obj_state with RGWObjState::prefetch_data = ...
- 03:30 PM Bug #47984: rgw: filter marker in list vesions
- lei cao wrote:
> https://github.com/ceph/ceph/pull/37848
This PR has been closed as a result of another PR furth... - 03:29 PM Bug #47984 (In Progress): rgw: filter marker in list vesions
- A different PR is close to a solution, so the PR is updated.
https://github.com/ceph/ceph/pull/37790 - 03:24 PM Bug #48064 (Pending Backport): rgw: unnecessary payload is added at the end of the message
- 02:41 PM Bug #48293 (Resolved): notification: radosgw-admin hangs on while closing
- when "data sync run" command is executed on a pubsub zone, the code creates the rabbitmq manager (as part initializin...
- 01:23 PM Bug #48241: librgw double read
- Keep in mind that if you're seeing the same read dispatched from nfs-ganesha twice, it most likely was issued twice. ...
- 01:10 PM Bug #48241: librgw double read
- Please be advised, librgw is somewhat broken on master, mostly due to zipper changes. The known issues are fixed in ...
- 10:14 AM Bug #48241: librgw double read
- Or Friedmann wrote:
> Is it possible to add the osd.log with debug_objclass 20 and debug_ms 1? (I want to see the cl... - 11:12 AM Backport #47836: octopus: Add OIDC provider support in RGW STS
- ok, thank you for the clarification.
- 11:10 AM Backport #47836: octopus: Add OIDC provider support in RGW STS
- Pritha Srivastava wrote:
> The target version for this PR is v15.2.6, but when I check the v15.2.6 release I do not ... - 06:57 AM Backport #47836: octopus: Add OIDC provider support in RGW STS
- Hi @Nathan,
The target version for this PR is v15.2.6, but when I check the v15.2.6 release I do not see the relat... - 10:26 AM Bug #47866: Object not found on healthy cluster
- An update on our efforts: We enabled @debug_ms 1@ and we are trying to find an object that vanishes while these logs ...
- 10:21 AM Backport #48101 (In Progress): octopus: Admin API returns 200 instead of 404 for Get Bucket Info
- 07:43 AM Bug #47910: radosgw crash on objecter operations
- Hi Mauricio,
I am using package from download.ceph.com.
I can test the fix, I will back with the answers until en... - 02:19 AM Documentation #25061: Wrong figure in multisite documentation
- Which figure? What's wrong with it?
11/18/2020
- 05:11 PM Feature #46296 (Fix Under Review): notifications: support GetTopicAttributes API
- 05:11 PM Bug #47910: radosgw crash on objecter operations
- Hi Rafal and Or,
Rafal,
Could you please confirm if you are using ceph packages from Ubuntu?
(say, Ubuntu Clou... - 04:19 PM Bug #47866: Object not found on healthy cluster
- We're hitting this problem in our 15.2.5 cluster as well. So far we know of one S3 that went missing, in a typical Do...
- 01:07 PM Bug #47866: Object not found on healthy cluster
- I believe client-rgw is doing the job. The default here seems to be 0/0 and when I turn it back off, I have nothing i...
- 12:59 PM Bug #47866: Object not found on healthy cluster
- Janek Bevendorff wrote:
> The config says the default for rgw is 1/5, that's why I was wondering. https://docs.ceph.... - 12:54 PM Bug #47866: Object not found on healthy cluster
- The config says the default for rgw is 1/5, that's why I was wondering. https://docs.ceph.com/en/latest/rados/trouble...
- 10:21 AM Bug #47866: Object not found on healthy cluster
- Janek Bevendorff wrote:
> I have enabled the setting via
>
> [...]
>
> I will check the logs when the next obj... - 10:20 AM Bug #47866: Object not found on healthy cluster
- Janek Bevendorff wrote:
> Short question here: isn't 1 the default log level anyway? I couldn't find any such messag... - 08:33 AM Bug #47866: Object not found on healthy cluster
- Short question here: isn't 1 the default log level anyway? I couldn't find any such messages in previous logs at least.
- 08:28 AM Bug #47866: Object not found on healthy cluster
- I have enabled the setting via...
- 04:25 AM Bug #47866: Object not found on healthy cluster
- In Comment #11 I asked: Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'?
Let me fol... - 09:38 AM Bug #48241: librgw double read
- Or Friedmann wrote:
> Is it possible to add the osd.log with debug_objclass 20 and debug_ms 1? (I want to see the cl...
Also available in: Atom