Activity
From 10/31/2020 to 11/29/2020
11/29/2020
- 04:03 PM Bug #48369: notification: get/delete of specific notification does not work
- this is working and tested: https://github.com/ceph/ceph/blob/master/src/test/rgw/rgw_multi/tests_ps.py#L953
faili...
11/27/2020
- 10:22 PM Bug #47910: radosgw crash on objecter operations
- I completed a first pass of the backport,
and should look at build/test next week. - 04:42 PM Bug #47910: radosgw crash on objecter operations
- Casey, thanks for the pointers!
Not sure how I missed that one.
So, the stack size was increased from 128k to 512... - 04:32 PM Bug #47910: radosgw crash on objecter operations
- this was fixed for octopus as part of https://tracker.ceph.com/issues/43739
it looks like the nautilus backport ht... - 02:16 PM Bug #47910: radosgw crash on objecter operations
- Hi Rafal,
Thanks for testing!
So, the issue indeed does seem to be stack corruption/overflow.
Unfortunately ... - 01:24 PM Bug #47910: radosgw crash on objecter operations
- Or, Mauricio,
I tested radosgw with increased coroutine stack sized to 1M. Everything is working fine.
I think, w... - 04:29 PM Bug #47866: Object not found on healthy cluster
- Janek Bevendorff wrote:
> Is there a roadmap when the fix will be available? Luckily, we haven't lost any more objec...
11/26/2020
- 11:55 AM Bug #48382 (Duplicate): Broken public Swift bucket access with Keystone integration
- Public swift bucket access is broken. Prevents upgrading towards 14.2.12 or newer.
In reference to:
https://lis... - 11:16 AM Bug #47866 (Resolved): Object not found on healthy cluster
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 08:20 AM Bug #47866: Object not found on healthy cluster
- Is there a roadmap when the fix will be available? Luckily, we haven't lost any more objects so far (not sure if that...
- 11:10 AM Bug #48369 (Fix Under Review): notification: get/delete of specific notification does not work
- note that this is an extension to the AWS API, for the next steps to work, follow the instructions here:
https://git...
11/25/2020
- 09:47 PM Bug #48363: rgw: concurrent object delete can leak bucket index instance entries
- Hi Yehuda,
What Ceph versions are affected?
Matt - 07:44 PM Bug #48363 (New): rgw: concurrent object delete can leak bucket index instance entries
- This happens when there are concurrent operations, where at least one succeeds, whereas at least another one is cance...
- 07:55 PM Bug #48364 (New): rgw: bucket index delete marker instance entry might leak
- at rgw_bucket_unlink_instance(), if olh.start_modify() returns false (which means that op.epoch is specified and is l...
- 07:53 PM Backport #48331 (Resolved): octopus: Object not found on healthy cluster
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/38249
m... - 04:12 PM Backport #48331: octopus: Object not found on healthy cluster
- J. Eric Ivancich wrote:
> https://github.com/ceph/ceph/pull/38249
merged - 01:38 PM Bug #48358: rgw: qlen and qactive perf counters leak
- > We'll send a separate PR to expose the throttle `outstanding_requests` values in a new perf counter
https://gith... - 12:51 PM Bug #48358 (New): rgw: qlen and qactive perf counters leak
- In our environment the rgw qlen and qactive perf counters seem to trend slowly upwards. See the plot attached.
I sus... - 09:52 AM Bug #48351: object-lock enabled is not working on 14.2.9 - files still deletable
- aws s3api get-object-lock-configuration --bucket locktest --endpoint http://our-s3
{
"ObjectLockConfiguration... - 09:49 AM Bug #48351: object-lock enabled is not working on 14.2.9 - files still deletable
- Added Screenshot of my testfile within this bucket to show the headers
- 09:41 AM Bug #48351 (Closed): object-lock enabled is not working on 14.2.9 - files still deletable
- the object lock mentioned in https://github.com/ceph/ceph/pull/29905 - objects within buckets, created with object-lo...
- 05:54 AM Bug #48350 (Can't reproduce): [rgw]some multipart object will become orphan objects in data pool
- OS: CentOS 7.6
Ceph Version: 14.2.5
Procedure:
1, Use s3 browser upload a big file with multipart upload.
2, Set...
11/24/2020
- 11:24 PM Bug #45033: S3 multipart copy broken by multi tenant
- Still a problem on 14.2.15.
Matt: I just realized I might have misinterpreted you answer, just to clarify, you bel... - 03:02 PM Backport #48346 (Resolved): nautilus: rgw: unnecessary payload is added at the end of the message
- https://github.com/ceph/ceph/pull/38590
- 03:02 PM Backport #48345 (Resolved): octopus: rgw: unnecessary payload is added at the end of the message
- https://github.com/ceph/ceph/pull/38339
- 08:50 AM Bug #47866: Object not found on healthy cluster
- Thank you for sharing that insight with us Pritha and Eric, it clarifies a lot.
11/23/2020
- 09:42 PM Bug #46257 (In Progress): FAIL: s3tests_boto3.functional.test_s3select
- this was fixed in https://github.com/ceph/s3select/pull/14, but the ceph submodule was never updated to include these...
- 07:17 PM Bug #48332 (New): rgw: null versioned delete markers removal doesn't sync
- null versioned delete marker can exist in suspended versioning buckets.
- 06:58 PM Backport #48331 (In Progress): octopus: Object not found on healthy cluster
- 06:52 PM Backport #48331 (Resolved): octopus: Object not found on healthy cluster
- https://github.com/ceph/ceph/pull/38249
- 06:22 PM Bug #47866: Object not found on healthy cluster
- I exchanged email with Pritha, who is the true expert on modern GC in RGW, to clarify my own questions. I'm pasting t...
- 03:24 PM Bug #47866: Object not found on healthy cluster
- Denis Krienbühl wrote:
> Pritha Srivastava wrote:
> > Denis Krienbühl wrote:
> > > Pritha Srivastava wrote:
> > >... - 03:10 PM Bug #47866: Object not found on healthy cluster
- Pritha Srivastava wrote:
> Denis Krienbühl wrote:
> > Pritha Srivastava wrote:
> > > Please note that this will re... - 02:47 PM Bug #47866: Object not found on healthy cluster
- Denis Krienbühl wrote:
> Pritha Srivastava wrote:
> > Please note that this will remove the gc entries before their... - 01:29 PM Bug #47866: Object not found on healthy cluster
- Pritha Srivastava wrote:
> Please note that this will remove the gc entries before their expiration time.
What do... - 04:09 AM Bug #47866: Object not found on healthy cluster
- If the affected users have increased the value of rgw_gc_obj_min_wait, then before restoring the original expiration ...
- 06:02 PM Bug #48330 (New): rgw: fix GC defer process and clean up from immediate fix
- We had a temporary fix to avert a data loss bug here:
https://tracker.ceph.com/issues/47866
But now GC defe... - 05:00 PM Bug #48329 (New): cls_rgw_gc_queue_remove_entries() doesn't remove entries from xattr urgent data
- cls_rgw_gc_queue_remove_entries() has to scan the entries it's removing from the queue in order to also remove any ur...
- 04:48 PM Bug #48328 (New): inefficient check for xattr urgent data in cls_rgw_gc_queue_list_entries()
- cls_rgw_gc_queue_list_entries() reads the urgent data map at the beginning as part of read_queue_head(). but if urgen...
- 04:22 PM Bug #48321: notifications: tag based filtering works only for object deletions
- in "nautilus" only "copy" does not work. so, this should be fixed in nautilus as well, however, since nautilus does n...
- 03:48 PM Bug #48327 (Fix Under Review): Infinite loop in old reset-stats
- 03:30 PM Bug #48327 (Resolved): Infinite loop in old reset-stats
- from-index was not updated causing a inf loop for buckets > 1000
- 03:29 AM Bug #48322: rgw copy obj should check the object size
- https://github.com/ceph/ceph/pull/38234
- 03:19 AM Bug #48322 (New): rgw copy obj should check the object size
- As the doc of boto3 says, copy obj should also check the object size and reject the request when the object is large ...
11/22/2020
- 07:28 PM Bug #48321 (Pending Backport): notifications: tag based filtering works only for object deletions
- tags filtering fails for object "put" and "copy".
tags are not populated into req_state when filtering is done - 03:37 PM Bug #47866 (Pending Backport): Object not found on healthy cluster
- 08:48 AM Bug #47866: Object not found on healthy cluster
- This actually makes a lot of sense.
Here are what we have set in our config:
* rgw_gc_obj_min_wait: 300
* rgw_gc_p... - 09:59 AM Bug #48319 (New): rgw add x-amz-version-id and x-amz-copy-source-version-id for copyobj
- 08:55 AM Bug #48293 (Fix Under Review): notification: radosgw-admin hangs on while closing
11/21/2020
- 09:50 PM Bug #47866: Object not found on healthy cluster
- Denis Krienbühl wrote:
> Eric Ivancich wrote:
> > Do you have the means to rebuild your RGWs with the PR attached t... - 09:40 PM Bug #47866: Object not found on healthy cluster
- Gaudenz Steinlin wrote:
> I made some tests in our lab and can confirm the theory. I was able to reproduce this bug.... - 09:31 PM Bug #47866: Object not found on healthy cluster
- I made some tests in our lab and can confirm the theory. I was able to reproduce this bug. I used the following setti...
- 08:28 PM Bug #47866: Object not found on healthy cluster
- Eric Ivancich wrote:
> Do you have the means to rebuild your RGWs with the PR attached to this tracker? If so that w... - 08:21 PM Bug #47866: Object not found on healthy cluster
- Denis Krienbühl wrote:
> Your theory aligns perfectly with our request logs. Here are the longest measured GET reque... - 07:58 PM Bug #47866: Object not found on healthy cluster
- Your theory aligns perfectly with our request logs. Here are the longest measured GET requests for all of the S3 obje...
- 07:23 PM Bug #47866: Object not found on healthy cluster
- Please note that there is now a PR attached to this Tracker. I still have not been able to reproduce, as using my com...
- 06:59 PM Bug #47866 (Fix Under Review): Object not found on healthy cluster
- 05:07 PM Bug #47866: Object not found on healthy cluster
- IMPORTANT
rgw_gc_obj_min_wait is set in seconds.
So 2 hours, the default, is 7200 seconds.
5 hours is 18000 seco... - 04:28 PM Bug #47866: Object not found on healthy cluster
- Denis Krienbühl wrote:
> Your theory sounds plausible to me. Though if I understand correctly we might be able to ve... - 04:26 PM Bug #47866: Object not found on healthy cluster
- Your theory sounds plausible to me. Though if I understand correctly we might be able to verify this by looking at ou...
- 04:09 PM Bug #47866: Object not found on healthy cluster
- Based on a review of code paths, I have a theory as to why this happens. There's an immediate configuration change th...
11/20/2020
- 10:08 PM Bug #48316 (New): rgw: bi entry mtime discrepancy after incremental sync of versioned object
- Can happen when the object creation is followed by attribute change (such as acl) on the same object version. The obj...
- 09:34 PM Bug #47866: Object not found on healthy cluster
- I see, I'll try to run a search for case "a" next week. We have plenty of access logs going back a few months detaili...
- 09:29 PM Bug #47866: Object not found on healthy cluster
- Denis Krienbühl wrote:
> Eric Ivancich wrote:
>
> > a. if you have a bucket that doesn't get deletes then using t... - 09:26 PM Bug #47866: Object not found on healthy cluster
- Sorry to update again, but I want to cover both newer and older gc implementations. The egrep discussed above is now:...
- 09:14 PM Bug #47866: Object not found on healthy cluster
- Eric Ivancich wrote:
> a. if you have a bucket that doesn't get deletes then using the egrep command above will he... - 08:55 PM Bug #47866: Object not found on healthy cluster
- Denis Krienbühl wrote:
> Eric Ivancich wrote:
> > If you have a bucket where things get uploaded and maybe read but... - 07:43 PM Bug #47866: Object not found on healthy cluster
- Eric Ivancich wrote:
> If you have a bucket where things get uploaded and maybe read but rarely deleted, and if you'... - 07:30 PM Bug #47866: Object not found on healthy cluster
- Just to clarify on why this is helpful: "If you have a bucket where things get uploaded and maybe read but rarely del...
- 07:21 PM Bug #47866: Object not found on healthy cluster
- We realize not everyone is able to do this, but for those who can, here is one approach we're using to try to nail th...
- 07:14 PM Bug #47866: Object not found on healthy cluster
- If the multipart components are themselves larger than the striping size, they two can have shadow tail objects, as s...
- 03:04 PM Bug #47866: Object not found on healthy cluster
- Update: forget what I said. When I retrieve the ETag via boto3, I get 57ba90377073200f91b5dbee7aaece6-64 instead. I a...
- 03:03 PM Bug #47866: Object not found on healthy cluster
- Thank you, I'm glad to see the @__multipart@ files in there, as that's what I'm counting on.
And just in case some... - 02:56 PM Bug #47866: Object not found on healthy cluster
- I crated a bucket and uploaded two 1GB multipart objects. Here's what the radoslist looks like:...
- 01:55 PM Bug #47866: Object not found on healthy cluster
- Update: It seems like @refcount.put@ is the call that logs the @__multipart@ file about to be deleted.
- 01:18 PM Bug #47866: Object not found on healthy cluster
- Janek Bevendorff wrote:
> Yes, it is uploading multiple parts and the resulting RADOS objects live in the multipart ... - 12:55 PM Bug #47866: Object not found on healthy cluster
- Yes, it is uploading multiple parts and the resulting RADOS objects live in the multipart namespace, not the shadow n...
- 12:39 PM Bug #47866: Object not found on healthy cluster
- Janek Bevendorff wrote:
> I wanted to reupload our multipart objects as single parts and noticed something very stra... - 12:03 PM Bug #47866: Object not found on healthy cluster
- I wanted to reupload our multipart objects as single parts and noticed something very strange: Not all multipart obje...
- 08:55 AM Bug #47866: Object not found on healthy cluster
- Sorry if that was unclear. The majority of all objects we have in total are single-part objects. The affected objects...
- 08:34 AM Bug #47866: Object not found on healthy cluster
- Janek Bevendorff wrote:
> I don't think the issue is packet loss per se. The objects in our affected buckets were al... - 08:34 AM Bug #47866: Object not found on healthy cluster
- Janek Bevendorff wrote:
> I don't think the issue is packet loss per se. The objects in our affected buckets were al... - 08:30 AM Bug #47866: Object not found on healthy cluster
- I don't think the issue is packet loss per se. The objects in our affected buckets were all filled by a Hadoop job ru...
- 08:12 AM Bug #47866: Object not found on healthy cluster
- Another update from our side: We have two sites, a larger and a smaller one. The smaller site was successfully scanne...
- 04:56 PM Bug #47800 (Resolved): rgw/tempest test failure in test_object_upload_in_segments
- 04:56 PM Bug #47801 (Resolved): rgw/tempest test failure in test_get_object_with_x_object_manifest
- 12:43 PM Bug #47910: radosgw crash on objecter operations
- Thanks for confirming on large buckets/EC pools.
Correct; that patch is the only change in the ceph packages in th... - 06:00 AM Bug #47910: radosgw crash on objecter operations
- Both of them.
I checked your links. I should implement https://github.com/cbodley/ceph/commit/d23507bd1295a29ccae3... - 12:23 AM Bug #47910: radosgw crash on objecter operations
- Hey Rafal,
Thanks!
By the way, does your ceph cluster have:
1) bucket(s) with a large number of objects?
2)... - 02:01 AM Bug #48289 (Fix Under Review): prefetch in rgw_file is done 3 times per read
11/19/2020
- 05:29 PM Documentation #23027: document rgw_multipart_min_part_size
There are over 1500 options in the source as of today. `rgw_multipart_min_part_size` is marked as `ADVANCED`, and I...- 04:57 PM Bug #47588 (Resolved): sts test failures
- 04:02 PM Bug #48289: prefetch in rgw_file is done 3 times per read
- please backport the fix to Nautilus
- 09:48 AM Bug #48289 (Resolved): prefetch in rgw_file is done 3 times per read
- Each call to rgw_read (rgw_file.cc) invokes three calls to RGWRados::get_obj_state with RGWObjState::prefetch_data = ...
- 03:30 PM Bug #47984: rgw: filter marker in list vesions
- lei cao wrote:
> https://github.com/ceph/ceph/pull/37848
This PR has been closed as a result of another PR furth... - 03:29 PM Bug #47984 (In Progress): rgw: filter marker in list vesions
- A different PR is close to a solution, so the PR is updated.
https://github.com/ceph/ceph/pull/37790 - 03:24 PM Bug #48064 (Pending Backport): rgw: unnecessary payload is added at the end of the message
- 02:41 PM Bug #48293 (Resolved): notification: radosgw-admin hangs on while closing
- when "data sync run" command is executed on a pubsub zone, the code creates the rabbitmq manager (as part initializin...
- 01:23 PM Bug #48241: librgw double read
- Keep in mind that if you're seeing the same read dispatched from nfs-ganesha twice, it most likely was issued twice. ...
- 01:10 PM Bug #48241: librgw double read
- Please be advised, librgw is somewhat broken on master, mostly due to zipper changes. The known issues are fixed in ...
- 10:14 AM Bug #48241: librgw double read
- Or Friedmann wrote:
> Is it possible to add the osd.log with debug_objclass 20 and debug_ms 1? (I want to see the cl... - 11:12 AM Backport #47836: octopus: Add OIDC provider support in RGW STS
- ok, thank you for the clarification.
- 11:10 AM Backport #47836: octopus: Add OIDC provider support in RGW STS
- Pritha Srivastava wrote:
> The target version for this PR is v15.2.6, but when I check the v15.2.6 release I do not ... - 06:57 AM Backport #47836: octopus: Add OIDC provider support in RGW STS
- Hi @Nathan,
The target version for this PR is v15.2.6, but when I check the v15.2.6 release I do not see the relat... - 10:26 AM Bug #47866: Object not found on healthy cluster
- An update on our efforts: We enabled @debug_ms 1@ and we are trying to find an object that vanishes while these logs ...
- 10:21 AM Backport #48101 (In Progress): octopus: Admin API returns 200 instead of 404 for Get Bucket Info
- 07:43 AM Bug #47910: radosgw crash on objecter operations
- Hi Mauricio,
I am using package from download.ceph.com.
I can test the fix, I will back with the answers until en... - 02:19 AM Documentation #25061: Wrong figure in multisite documentation
- Which figure? What's wrong with it?
11/18/2020
- 05:11 PM Feature #46296 (Fix Under Review): notifications: support GetTopicAttributes API
- 05:11 PM Bug #47910: radosgw crash on objecter operations
- Hi Rafal and Or,
Rafal,
Could you please confirm if you are using ceph packages from Ubuntu?
(say, Ubuntu Clou... - 04:19 PM Bug #47866: Object not found on healthy cluster
- We're hitting this problem in our 15.2.5 cluster as well. So far we know of one S3 that went missing, in a typical Do...
- 01:07 PM Bug #47866: Object not found on healthy cluster
- I believe client-rgw is doing the job. The default here seems to be 0/0 and when I turn it back off, I have nothing i...
- 12:59 PM Bug #47866: Object not found on healthy cluster
- Janek Bevendorff wrote:
> The config says the default for rgw is 1/5, that's why I was wondering. https://docs.ceph.... - 12:54 PM Bug #47866: Object not found on healthy cluster
- The config says the default for rgw is 1/5, that's why I was wondering. https://docs.ceph.com/en/latest/rados/trouble...
- 10:21 AM Bug #47866: Object not found on healthy cluster
- Janek Bevendorff wrote:
> I have enabled the setting via
>
> [...]
>
> I will check the logs when the next obj... - 10:20 AM Bug #47866: Object not found on healthy cluster
- Janek Bevendorff wrote:
> Short question here: isn't 1 the default log level anyway? I couldn't find any such messag... - 08:33 AM Bug #47866: Object not found on healthy cluster
- Short question here: isn't 1 the default log level anyway? I couldn't find any such messages in previous logs at least.
- 08:28 AM Bug #47866: Object not found on healthy cluster
- I have enabled the setting via...
- 04:25 AM Bug #47866: Object not found on healthy cluster
- In Comment #11 I asked: Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'?
Let me fol... - 09:38 AM Bug #48241: librgw double read
- Or Friedmann wrote:
> Is it possible to add the osd.log with debug_objclass 20 and debug_ms 1? (I want to see the cl...
11/17/2020
- 11:54 PM Bug #48268 (New): rgw: when removing a current versioned object, the new current object doesn't g...
- E.g., we have an object with two versions, older one has epoch=2, new one has epoch=3. When removing the epoch=3 obje...
- 02:59 PM Bug #47904 (Fix Under Review): notifications not sent when non-owner modifies the bucket
- 02:39 PM Bug #48262 (Resolved): rgw: RGWListMultipart show Owner ID and DisplayName
- the owner ID and DisplayName should not be empty when list parts
<?xml version="1.0" encoding="UTF-8"?>
<ListPart... - 01:09 PM Bug #48255 (Fix Under Review): rgwlc: shard-index vector short by 1?
- 01:25 AM Bug #48255 (Resolved): rgwlc: shard-index vector short by 1?
- The new parallel lc code relies on a vector of indices to seed
random-order shard processing. This vector was dimen... - 08:20 AM Bug #47866: Object not found on healthy cluster
- I waited for the first few minor releases and upgraded from Nautilus directly to 15.2.5, so yes.
- 02:11 AM Bug #47866: Object not found on healthy cluster
- Dan van der Ster wrote:
> Has #42670 been excluded?
So the first version of Octopus that that fix appears in is 1... - 06:54 AM Bug #48208 (Fix Under Review): rgw:usage:when trim usage by user,it is possible to trim other use...
11/16/2020
- 09:33 PM Bug #47866: Object not found on healthy cluster
- Janek Bevendorff wrote:
> What I do remember is error messages about compression during OSD upgrades (i.e., compre... - 05:20 PM Bug #47866: Object not found on healthy cluster
- Interesting issue, but it doesn't really look like it's the same thing for two reasons:
1) We never saw this kind ... - 08:56 AM Bug #47866: Object not found on healthy cluster
- Has #42670 been excluded?
- 08:40 AM Bug #47866: Object not found on healthy cluster
- So far, the issue has only affected multipart objects. I have already suspected that this might be part of the issue,...
- 03:48 PM Bug #48250 (Resolved): notifications: only bucket owner is allowed to add/remove notifications
- instead, anyone that has permission on the bucket should be allowed to add/remove notifications.
"verify_bucket_owne... - 03:45 PM Bug #48236 (Fix Under Review): rgw: fix auth when date is RFC-822/ISO 8601 standard timezone spec...
- 02:53 PM Bug #47799 (Resolved): rgw/crypt suite with Barbican fails with "Cannot create secret"
- 10:15 AM Bug #47799 (Fix Under Review): rgw/crypt suite with Barbican fails with "Cannot create secret"
- 11:52 AM Bug #48241: librgw double read
- Is it possible to add the osd.log with debug_objclass 20 and debug_ms 1? (I want to see the cls call logs and the obj...
- 02:06 AM Bug #48241 (Need More Info): librgw double read
- I test read a 3MB object with this rgw: Fix librgw end define patch,According to the logs, there are still two ops.
... - 08:01 AM Bug #48208: rgw:usage:when trim usage by user,it is possible to trim other users’ usage.
- chengwu liang wrote:
> when trim usage by specifying user,it is possible to trim other users’ usage.
> Steps:
> 1、...
11/15/2020
- 05:57 PM Backport #48239: octopus: list object versions returned multiple 'IsLatest true' entries
- h3. original description
backport to octopus,
parent tracker: https://tracker.ceph.com/issues/47919 - 05:56 PM Backport #48239 (In Progress): octopus: list object versions returned multiple 'IsLatest true' en...
- 02:48 PM Backport #48239 (Resolved): octopus: list object versions returned multiple 'IsLatest true' entries
- https://github.com/ceph/ceph/pull/38086
- 05:54 PM Backport #48238 (In Progress): nautilus: list object versions returned multiple 'IsLatest true' e...
- 02:44 PM Backport #48238: nautilus: list object versions returned multiple 'IsLatest true' entries
- https://github.com/ceph/ceph/pull/38085
- 02:37 PM Backport #48238 (Resolved): nautilus: list object versions returned multiple 'IsLatest true' entries
- https://github.com/ceph/ceph/pull/38085
---
backport to nautilus: https://tracker.ceph.com/issues/47919 - 11:19 AM Bug #47866: Object not found on healthy cluster
- Janek Bevendorff wrote:
> Indeed. I just noticed the other object isn't a multipart object. I attached the radosgw-a... - 11:09 AM Bug #47866: Object not found on healthy cluster
- Julien Demais wrote:
> No one is using my ceph cluster so far. I am in the process of evaluating ceph so I am the on...
11/14/2020
- 12:59 PM Bug #47866: Object not found on healthy cluster
- Please find my answers below:
1. It seems like those reporting this issue are seeing it strictly after upgrading t... - 04:33 AM Bug #48236 (Fix Under Review): rgw: fix auth when date is RFC-822/ISO 8601 standard timezone spec...
- in amazon s3, it is support date format with timezone
> PUT /yuliyangwebsite/1G-16 HTTP/1.1
> Host: s3.amazonaws....
11/13/2020
- 07:19 PM Bug #47866: Object not found on healthy cluster
- @Julien Demais and ANYONE ELSE experiencing this issue,
I posted these questions on ceph-users but wanted to see i... - 07:16 PM Bug #47866: Object not found on healthy cluster
- I'm copying in answers Janek gave to questions I posted on ceph-users.
1. It seems like those reporting this iss... - 08:18 AM Bug #47866: Object not found on healthy cluster
- Indeed. I just noticed the other object isn't a multipart object. I attached the radosgw-admin stat dump of a working...
- 06:08 PM Bug #47801 (Fix Under Review): rgw/tempest test failure in test_get_object_with_x_object_manifest
- 02:03 PM Bug #48049 (Resolved): rgw: getting S3 website root without last slash crashes rgw
11/12/2020
- 09:13 PM Bug #47866: Object not found on healthy cluster
- I did some experiments, and I don't think the head object having a size of 0 is necessarily an indicator of a problem...
- 05:11 PM Bug #47866: Object not found on healthy cluster
- I have not set any lifecycle policy on the bucket and I double checked that there is indeed none. s3cmd getlifecycle ...
- 05:04 PM Bug #47866: Object not found on healthy cluster
- I am not running Ceph in Kubernetes.
I attached a config dump and a stat dump of an OK object. All sensitive infor... - 03:45 PM Bug #47866: Object not found on healthy cluster
- May we also please ask to attach the ceph.conf
and `radosgw-admin object stat` of a similarly sized OK object from t... - 03:17 PM Bug #47866 (Triaged): Object not found on healthy cluster
- 03:12 PM Bug #47866: Object not found on healthy cluster
- Do you have any other storage classes configured? Is there a lifecycle policy on the bucket?
- 01:05 PM Bug #47866: Object not found on healthy cluster
- Can you check that the RADOS object is larger than 0 bytes? This is a creeping data loss that has been reported on Oc...
- 03:23 PM Bug #46567: Access denied for multi-object-delete by non-bucket-owner
- included in https://github.com/ceph/ceph/pull/37933
- 03:19 PM Bug #47799 (In Progress): rgw/crypt suite with Barbican fails with "Cannot create secret"
- 02:11 AM Bug #48208 (Fix Under Review): rgw:usage:when trim usage by user,it is possible to trim other use...
- when trim usage by user,it is possible to trim other users’ usage.
Steps:
1、set rgw enable usage log = true and rgw...
11/10/2020
- 09:31 AM Bug #47919: list object versions returned multiple 'IsLatest true' entries
- Matt Benjamin wrote:
> Is target version Nautilus really correct (linked fix is against master). I.e., do we need b...
11/09/2020
- 03:04 PM Bug #48034 (Fix Under Review): Should rgw::auth::Strategy::apply be noexcept?
- 02:50 PM Bug #46439: librgw: fhcache deadlock
- Also, Xuehan Xu, the call stack provided with this report is from librgw running under FUSE. I wasn't aware of an op...
- 02:36 PM Bug #46439: librgw: fhcache deadlock
- Thanks for the detailed analysis, which seems potentially valid.
Could you report status after merge of the fix at... - 02:27 PM Bug #47919: list object versions returned multiple 'IsLatest true' entries
- Is target version Nautilus really correct (linked fix is against master). I.e., do we need backports for Octopus and...
11/08/2020
- 11:43 AM Cleanup #48146 (In Progress): fix compilation error: declaration conflicts with target of using d...
- when compiling with clang the following error occurs:...
- 09:43 AM Bug #48073: radosgw_admin_rest failure: bucket info API gets NoSuchKey, expects NoSuchBucket
- (This tracker issue is a followup to #45193)
11/06/2020
- 05:47 PM Bug #48136 (Resolved): rgw_file: fix Zipper integration in RGWLibContinuedReq
- These fixes aren't needed by any current Ceph release, so there's nothing to backport.
- 09:58 AM Bug #48139 (Duplicate): Ceph Dashboard Object Gateway InvalidRange bucket exception
- We've setup Ceph Octopus 15.2.5 cluster in CentOS8. Ceph cluster is up and able to view the dashboard
1) when we t...
11/05/2020
- 05:18 PM Bug #48136 (Fix Under Review): rgw_file: fix Zipper integration in RGWLibContinuedReq
- 05:15 PM Bug #48136 (Resolved): rgw_file: fix Zipper integration in RGWLibContinuedReq
- 03:35 PM Bug #48034 (In Progress): Should rgw::auth::Strategy::apply be noexcept?
- 03:24 PM Bug #48049 (Fix Under Review): rgw: getting S3 website root without last slash crashes rgw
- 03:16 PM Bug #48064 (Fix Under Review): rgw: unnecessary payload is added at the end of the message
- 03:14 PM Bug #48115 (Closed): rgw stops responding correctly, error log full of messages: ceph s3:get_obj ...
- 03:11 PM Bug #48115: rgw stops responding correctly, error log full of messages: ceph s3:get_obj Schedulin...
- Hi,
2218 is ERR_RATE_LIMITED, by default each rgw can handle up to 1000, configured by this parameter rgw max_conc... - 03:05 PM Bug #45193: Admin API returns 200 instead of 404 for Get Bucket Info
- please include the followup fix from https://tracker.ceph.com/issues/48073
- 03:04 PM Bug #48073 (Resolved): radosgw_admin_rest failure: bucket info API gets NoSuchKey, expects NoSuch...
- 03:03 PM Bug #48073 (Fix Under Review): radosgw_admin_rest failure: bucket info API gets NoSuchKey, expect...
- 02:38 PM Backport #48128 (Resolved): nautilus: Unnecessary bilogs are left in sync-disabled buckets
- https://github.com/ceph/ceph/pull/38589
- 02:37 PM Backport #48127 (Resolved): octopus: Unnecessary bilogs are left in sync-disabled buckets
- https://github.com/ceph/ceph/pull/38338
11/04/2020
- 09:08 PM Bug #48122: rgw cannot find keyring after config file is minimized
- Moved to "Ceph" project since it's a bug in the implementation of "ceph config generate-minimal-conf".
- 08:25 PM Bug #48122 (Won't Fix): rgw cannot find keyring after config file is minimized
- Given a ceph.conf file like this:...
- 03:08 PM Bug #47882 (Resolved): rgw: crash on non S3/swift requests (eg. admin api requests)
- No backports are listed, so I moved this to resolved. If backports are appropriate then please update the Backport an...
- 03:07 PM Bug #47472 (Resolved): rgw: lifecycle Filter clauses with Prefix and one Tag are valid
- 03:07 PM Bug #47472: rgw: lifecycle Filter clauses with Prefix and one Tag are valid
- No backports are listed, so I'm switching to resolved. If backports are appropriate, please update Backport and Statu...
- 03:04 PM Bug #48037 (Pending Backport): Unnecessary bilogs are left in sync-disabled buckets
- 02:44 PM Bug #48115 (Closed): rgw stops responding correctly, error log full of messages: ceph s3:get_obj ...
- rgw stops responding correctly, error log full of messages: ceph s3:get_obj Scheduling request failed with -2218.
... - 09:27 AM Bug #46614 (Resolved): rgw lc expiration header returns although it should not
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 09:26 AM Bug #47235 (Resolved): rgw/rgw_file: incorrect lru object eviction in lookup_fh
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 09:26 AM Bug #47408 (Resolved): rgw: orphan list teuthology test uses `dnf`, which may not always be avail...
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 09:25 AM Bug #47710 (Resolved): rgw_file: avoid long delay on shutdown
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 09:25 AM Bug #47720 (Resolved): rgw: radosgw-admin does not paginate internally when listing bucket
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 09:25 AM Fix #48000 (Resolved): rgw: fix S3 API KeyCount incorrect return
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 09:24 AM Backport #48004 (Resolved): nautilus: rgw: fix S3 API KeyCount incorrect return
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37810
m... - 09:23 AM Backport #47959 (Resolved): nautilus: rgw lc expiration header returns although it should not
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37806
m... - 09:23 AM Backport #47897 (Resolved): nautilus: rgw: orphan list teuthology test uses `dnf`, which may not ...
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37805
m... - 09:23 AM Backport #47851 (Resolved): nautilus: rgw/rgw_file: incorrect lru object eviction in lookup_fh
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37804
m... - 09:23 AM Backport #47818 (Resolved): nautilus: rgw: radosgw-admin does not paginate internally when listin...
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37802
m... - 09:22 AM Backport #47735 (Resolved): nautilus: rgw_file: avoid long delay on shutdown
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37552
m...
11/03/2020
- 07:48 PM Backport #48004: nautilus: rgw: fix S3 API KeyCount incorrect return
- Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/37810
merged - 07:48 PM Backport #47959: nautilus: rgw lc expiration header returns although it should not
- Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/37806
merged - 07:47 PM Backport #47897: nautilus: rgw: orphan list teuthology test uses `dnf`, which may not always be a...
- Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/37805
merged - 07:46 PM Backport #47851: nautilus: rgw/rgw_file: incorrect lru object eviction in lookup_fh
- Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/37804
merged - 07:46 PM Backport #47818: nautilus: rgw: radosgw-admin does not paginate internally when listing bucket
- Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/37802
merged - 07:41 PM Backport #47735: nautilus: rgw_file: avoid long delay on shutdown
- Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/37552
merged - 03:48 PM Bug #48103 (New): Bilogs automatic trimming fails (multisite, ceph nautilus 14.2.9)
- Issue
=====
We're seeing a problem in our multisite Ceph deployment, where automatic bilog trimming has stopped acr... - 03:09 PM Cleanup #47848 (Closed): use std::unique_ptr in cls_fifo_legacy.cc
- this is already fixed here: https://github.com/ceph/ceph/pull/37660
- 02:46 PM Bug #16767: RadosGW Multipart Cleanup Failure
- I've encountered this bug in Ceph 14 and 15 and it's a pretty big problem for us for the same reason Brian Felton men...
- 12:54 PM Bug #47586 (Fix Under Review): Able to circumvent S3 Object Lock using deleteobjects command
- 11:49 AM Bug #47586: Able to circumvent S3 Object Lock using deleteobjects command
- Pull request to fix this: https://github.com/ceph/ceph/pull/37933
Updates to S3 tests: https://github.com/ceph/s3-te... - 11:28 AM Backport #48101 (Resolved): octopus: Admin API returns 200 instead of 404 for Get Bucket Info
- https://github.com/ceph/ceph/pull/38184
- 11:28 AM Backport #48100 (Resolved): nautilus: Admin API returns 200 instead of 404 for Get Bucket Info
- https://github.com/ceph/ceph/pull/38588
- 11:28 AM Backport #48099 (Rejected): mimic: Admin API returns 200 instead of 404 for Get Bucket Info
- 11:27 AM Bug #46400 (Resolved): Space usage accounting overestimated
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 11:25 AM Bug #47771 (Resolved): rgw: allow rgw-orphan-list to note when rados objects in namespace
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 11:25 AM Bug #47780 (Resolved): rgw: fix setting of namespace in ordered and unordered bucket listing
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 11:25 AM Bug #47812 (Resolved): rgw: rgw-orphan-list should use "plain" formatted `rados ls` output
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 11:25 AM Bug #47908 (Resolved): GC perfcounter fails to update when deletion occurs
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 11:25 AM Bug #47909 (Resolved): list pending GCs is very slow
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 10:19 AM Backport #48003 (Resolved): octopus: rgw: fix S3 API KeyCount incorrect return
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37849
m... - 10:19 AM Backport #47956 (Resolved): octopus: GC perfcounter fails to update when deletion occurs
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37847
m... - 10:19 AM Backport #47955 (Resolved): octopus: list pending GCs is very slow
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37846
m... - 09:58 AM Backport #47896 (Resolved): octopus: rgw: orphan list teuthology test uses `dnf`, which may not a...
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37845
m... - 09:57 AM Backport #47962 (Resolved): octopus: beast frontend option to set the request_timeout_ms
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37809
m... - 09:57 AM Backport #47960 (Resolved): octopus: rgw lc expiration header returns although it should not
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37807
m... - 09:57 AM Backport #47819 (Resolved): octopus: rgw: radosgw-admin does not paginate internally when listing...
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37803
m... - 09:57 AM Backport #47938 (Resolved): octopus: rgw: rgw-orphan-list should use "plain" formatted `rados ls`...
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37800
m... - 09:56 AM Backport #47817 (Resolved): octopus: rgw: allow rgw-orphan-list to note when rados objects in nam...
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37800
m... - 09:56 AM Backport #47037 (Resolved): octopus: rgw: Space usage accounting overestimated
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37779
m... - 09:18 AM Bug #47910: radosgw crash on objecter operations
- Right.
I want to make sure this is not a tcmalloc bug (on one of the nodes, you are using for testing this error). - 06:48 AM Bug #48073: radosgw_admin_rest failure: bucket info API gets NoSuchKey, expects NoSuchBucket
- https://github.com/ceph/ceph/pull/37929
11/02/2020
- 08:13 PM rgw-testing Bug #47795 (Resolved): rgw: s3-tests: when I try to run locally I get 'Your config file is missin...
- 08:09 PM rgw-testing Bug #47795: rgw: s3-tests: when I try to run locally I get 'Your config file is missing the "iam"...
- Eric Ivancich wrote:
> Re-opened at request of Ali Maredia. He does not think the tests should fail if there is no i... - 05:44 PM Bug #47910: radosgw crash on objecter operations
- This one from focal 2.7? Right now I'm testing on bionic with 2.5-2.2ubuntu3
- 05:28 PM Bug #47910: radosgw crash on objecter operations
- The swift healthcheck is not making a RADOS call so this is the reason I guess.
Can you please update this rgw to a ... - 05:19 PM Bug #47910: radosgw crash on objecter operations
- Or,
focal is not supported for nautilus, I tested with bionic version and this same effect.
I can't understand, why... - 12:44 PM Bug #47910: radosgw crash on objecter operations
- Thanks for the fast response,
Is it possible to install on the specific RGW newer version of tcmalloc(https://laun... - 11:04 AM Bug #47910: radosgw crash on objecter operations
- Here you are ...
- 10:17 AM Bug #47910: radosgw crash on objecter operations
- Thank you Rafal,
Can you please provide this command output:
ceph versions and ceph.conf - 05:17 PM Bug #47801 (Pending Backport): rgw/tempest test failure in test_get_object_with_x_object_manifest
- 05:17 PM Bug #47800 (Fix Under Review): rgw/tempest test failure in test_object_upload_in_segments
- 04:50 PM Bug #48073 (Resolved): radosgw_admin_rest failure: bucket info API gets NoSuchKey, expects NoSuch...
- ...
- 07:41 AM Bug #48064: rgw: unnecessary payload is added at the end of the message
- I made a related PR: https://github.com/ceph/ceph/pull/37914
- 07:10 AM Bug #48064 (Resolved): rgw: unnecessary payload is added at the end of the message
- Configured s3website as follows:...
Also available in: Atom