Project

General

Profile

Activity

From 10/31/2020 to 11/29/2020

11/29/2020

04:03 PM Bug #48369: notification: get/delete of specific notification does not work
this is working and tested: https://github.com/ceph/ceph/blob/master/src/test/rgw/rgw_multi/tests_ps.py#L953
faili...
Yuval Lifshitz

11/27/2020

10:22 PM Bug #47910: radosgw crash on objecter operations
I completed a first pass of the backport,
and should look at build/test next week.
Mauricio Oliveira
04:42 PM Bug #47910: radosgw crash on objecter operations
Casey, thanks for the pointers!
Not sure how I missed that one.
So, the stack size was increased from 128k to 512...
Mauricio Oliveira
04:32 PM Bug #47910: radosgw crash on objecter operations
this was fixed for octopus as part of https://tracker.ceph.com/issues/43739
it looks like the nautilus backport ht...
Casey Bodley
02:16 PM Bug #47910: radosgw crash on objecter operations
Hi Rafal,
Thanks for testing!
So, the issue indeed does seem to be stack corruption/overflow.
Unfortunately ...
Mauricio Oliveira
01:24 PM Bug #47910: radosgw crash on objecter operations
Or, Mauricio,
I tested radosgw with increased coroutine stack sized to 1M. Everything is working fine.
I think, w...
Rafal Wadolowski
04:29 PM Bug #47866: Object not found on healthy cluster
Janek Bevendorff wrote:
> Is there a roadmap when the fix will be available? Luckily, we haven't lost any more objec...
J. Eric Ivancich

11/26/2020

11:55 AM Bug #48382 (Duplicate): Broken public Swift bucket access with Keystone integration
Public swift bucket access is broken. Prevents upgrading towards 14.2.12 or newer.
In reference to:
https://lis...
Pietari Hyvärinen
11:16 AM Bug #47866 (Resolved): Object not found on healthy cluster
While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ... Nathan Cutler
08:20 AM Bug #47866: Object not found on healthy cluster
Is there a roadmap when the fix will be available? Luckily, we haven't lost any more objects so far (not sure if that... Janek Bevendorff
11:10 AM Bug #48369 (Fix Under Review): notification: get/delete of specific notification does not work
note that this is an extension to the AWS API, for the next steps to work, follow the instructions here:
https://git...
Yuval Lifshitz

11/25/2020

09:47 PM Bug #48363: rgw: concurrent object delete can leak bucket index instance entries
Hi Yehuda,
What Ceph versions are affected?
Matt
Matt Benjamin
07:44 PM Bug #48363 (New): rgw: concurrent object delete can leak bucket index instance entries
This happens when there are concurrent operations, where at least one succeeds, whereas at least another one is cance... Yehuda Sadeh
07:55 PM Bug #48364 (New): rgw: bucket index delete marker instance entry might leak
at rgw_bucket_unlink_instance(), if olh.start_modify() returns false (which means that op.epoch is specified and is l... Yehuda Sadeh
07:53 PM Backport #48331 (Resolved): octopus: Object not found on healthy cluster
This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/38249
m...
Nathan Cutler
04:12 PM Backport #48331: octopus: Object not found on healthy cluster
J. Eric Ivancich wrote:
> https://github.com/ceph/ceph/pull/38249
merged
Yuri Weinstein
01:38 PM Bug #48358: rgw: qlen and qactive perf counters leak
> We'll send a separate PR to expose the throttle `outstanding_requests` values in a new perf counter
https://gith...
Dan van der Ster
12:51 PM Bug #48358 (New): rgw: qlen and qactive perf counters leak
In our environment the rgw qlen and qactive perf counters seem to trend slowly upwards. See the plot attached.
I sus...
Dan van der Ster
09:52 AM Bug #48351: object-lock enabled is not working on 14.2.9 - files still deletable
aws s3api get-object-lock-configuration --bucket locktest --endpoint http://our-s3
{
"ObjectLockConfiguration...
Torsten Ennenbach
09:49 AM Bug #48351: object-lock enabled is not working on 14.2.9 - files still deletable
Added Screenshot of my testfile within this bucket to show the headers
Torsten Ennenbach
09:41 AM Bug #48351 (Closed): object-lock enabled is not working on 14.2.9 - files still deletable
the object lock mentioned in https://github.com/ceph/ceph/pull/29905 - objects within buckets, created with object-lo... Torsten Ennenbach
05:54 AM Bug #48350 (Can't reproduce): [rgw]some multipart object will become orphan objects in data pool
OS: CentOS 7.6
Ceph Version: 14.2.5
Procedure:
1, Use s3 browser upload a big file with multipart upload.
2, Set...
haitao chen

11/24/2020

11:24 PM Bug #45033: S3 multipart copy broken by multi tenant
Still a problem on 14.2.15.
Matt: I just realized I might have misinterpreted you answer, just to clarify, you bel...
Asbjørn Sannes
03:02 PM Backport #48346 (Resolved): nautilus: rgw: unnecessary payload is added at the end of the message
https://github.com/ceph/ceph/pull/38590 Nathan Cutler
03:02 PM Backport #48345 (Resolved): octopus: rgw: unnecessary payload is added at the end of the message
https://github.com/ceph/ceph/pull/38339 Nathan Cutler
08:50 AM Bug #47866: Object not found on healthy cluster
Thank you for sharing that insight with us Pritha and Eric, it clarifies a lot. Denis Krienbühl

11/23/2020

09:42 PM Bug #46257 (In Progress): FAIL: s3tests_boto3.functional.test_s3select
this was fixed in https://github.com/ceph/s3select/pull/14, but the ceph submodule was never updated to include these... Casey Bodley
07:17 PM Bug #48332 (New): rgw: null versioned delete markers removal doesn't sync
null versioned delete marker can exist in suspended versioning buckets. Yehuda Sadeh
06:58 PM Backport #48331 (In Progress): octopus: Object not found on healthy cluster
J. Eric Ivancich
06:52 PM Backport #48331 (Resolved): octopus: Object not found on healthy cluster
https://github.com/ceph/ceph/pull/38249 J. Eric Ivancich
06:22 PM Bug #47866: Object not found on healthy cluster
I exchanged email with Pritha, who is the true expert on modern GC in RGW, to clarify my own questions. I'm pasting t... J. Eric Ivancich
03:24 PM Bug #47866: Object not found on healthy cluster
Denis Krienbühl wrote:
> Pritha Srivastava wrote:
> > Denis Krienbühl wrote:
> > > Pritha Srivastava wrote:
> > >...
Pritha Srivastava
03:10 PM Bug #47866: Object not found on healthy cluster
Pritha Srivastava wrote:
> Denis Krienbühl wrote:
> > Pritha Srivastava wrote:
> > > Please note that this will re...
Denis Krienbühl
02:47 PM Bug #47866: Object not found on healthy cluster
Denis Krienbühl wrote:
> Pritha Srivastava wrote:
> > Please note that this will remove the gc entries before their...
Pritha Srivastava
01:29 PM Bug #47866: Object not found on healthy cluster
Pritha Srivastava wrote:
> Please note that this will remove the gc entries before their expiration time.
What do...
Denis Krienbühl
04:09 AM Bug #47866: Object not found on healthy cluster
If the affected users have increased the value of rgw_gc_obj_min_wait, then before restoring the original expiration ... Pritha Srivastava
06:02 PM Bug #48330 (New): rgw: fix GC defer process and clean up from immediate fix
We had a temporary fix to avert a data loss bug here:
https://tracker.ceph.com/issues/47866
But now GC defe...
J. Eric Ivancich
05:00 PM Bug #48329 (New): cls_rgw_gc_queue_remove_entries() doesn't remove entries from xattr urgent data
cls_rgw_gc_queue_remove_entries() has to scan the entries it's removing from the queue in order to also remove any ur... Casey Bodley
04:48 PM Bug #48328 (New): inefficient check for xattr urgent data in cls_rgw_gc_queue_list_entries()
cls_rgw_gc_queue_list_entries() reads the urgent data map at the beginning as part of read_queue_head(). but if urgen... Casey Bodley
04:22 PM Bug #48321: notifications: tag based filtering works only for object deletions
in "nautilus" only "copy" does not work. so, this should be fixed in nautilus as well, however, since nautilus does n... Yuval Lifshitz
03:48 PM Bug #48327 (Fix Under Review): Infinite loop in old reset-stats
Casey Bodley
03:30 PM Bug #48327 (Resolved): Infinite loop in old reset-stats
from-index was not updated causing a inf loop for buckets > 1000 Abhishek Lekshmanan
03:29 AM Bug #48322: rgw copy obj should check the object size
https://github.com/ceph/ceph/pull/38234 yunqing wang
03:19 AM Bug #48322 (New): rgw copy obj should check the object size
As the doc of boto3 says, copy obj should also check the object size and reject the request when the object is large ... yunqing wang

11/22/2020

07:28 PM Bug #48321 (Pending Backport): notifications: tag based filtering works only for object deletions
tags filtering fails for object "put" and "copy".
tags are not populated into req_state when filtering is done
Yuval Lifshitz
03:37 PM Bug #47866 (Pending Backport): Object not found on healthy cluster
J. Eric Ivancich
08:48 AM Bug #47866: Object not found on healthy cluster
This actually makes a lot of sense.
Here are what we have set in our config:
* rgw_gc_obj_min_wait: 300
* rgw_gc_p...
Julien Demais
09:59 AM Bug #48319 (New): rgw add x-amz-version-id and x-amz-copy-source-version-id for copyobj
joke lee
08:55 AM Bug #48293 (Fix Under Review): notification: radosgw-admin hangs on while closing
Yuval Lifshitz

11/21/2020

09:50 PM Bug #47866: Object not found on healthy cluster
Denis Krienbühl wrote:
> Eric Ivancich wrote:
> > Do you have the means to rebuild your RGWs with the PR attached t...
J. Eric Ivancich
09:40 PM Bug #47866: Object not found on healthy cluster
Gaudenz Steinlin wrote:
> I made some tests in our lab and can confirm the theory. I was able to reproduce this bug....
J. Eric Ivancich
09:31 PM Bug #47866: Object not found on healthy cluster
I made some tests in our lab and can confirm the theory. I was able to reproduce this bug. I used the following setti... Gaudenz Steinlin
08:28 PM Bug #47866: Object not found on healthy cluster
Eric Ivancich wrote:
> Do you have the means to rebuild your RGWs with the PR attached to this tracker? If so that w...
Denis Krienbühl
08:21 PM Bug #47866: Object not found on healthy cluster
Denis Krienbühl wrote:
> Your theory aligns perfectly with our request logs. Here are the longest measured GET reque...
J. Eric Ivancich
07:58 PM Bug #47866: Object not found on healthy cluster
Your theory aligns perfectly with our request logs. Here are the longest measured GET requests for all of the S3 obje... Denis Krienbühl
07:23 PM Bug #47866: Object not found on healthy cluster
Please note that there is now a PR attached to this Tracker. I still have not been able to reproduce, as using my com... J. Eric Ivancich
06:59 PM Bug #47866 (Fix Under Review): Object not found on healthy cluster
J. Eric Ivancich
05:07 PM Bug #47866: Object not found on healthy cluster
IMPORTANT
rgw_gc_obj_min_wait is set in seconds.
So 2 hours, the default, is 7200 seconds.
5 hours is 18000 seco...
J. Eric Ivancich
04:28 PM Bug #47866: Object not found on healthy cluster
Denis Krienbühl wrote:
> Your theory sounds plausible to me. Though if I understand correctly we might be able to ve...
J. Eric Ivancich
04:26 PM Bug #47866: Object not found on healthy cluster
Your theory sounds plausible to me. Though if I understand correctly we might be able to verify this by looking at ou... Denis Krienbühl
04:09 PM Bug #47866: Object not found on healthy cluster
Based on a review of code paths, I have a theory as to why this happens. There's an immediate configuration change th... J. Eric Ivancich

11/20/2020

10:08 PM Bug #48316 (New): rgw: bi entry mtime discrepancy after incremental sync of versioned object
Can happen when the object creation is followed by attribute change (such as acl) on the same object version. The obj... Yehuda Sadeh
09:34 PM Bug #47866: Object not found on healthy cluster
I see, I'll try to run a search for case "a" next week. We have plenty of access logs going back a few months detaili... Denis Krienbühl
09:29 PM Bug #47866: Object not found on healthy cluster
Denis Krienbühl wrote:
> Eric Ivancich wrote:
>
> > a. if you have a bucket that doesn't get deletes then using t...
J. Eric Ivancich
09:26 PM Bug #47866: Object not found on healthy cluster
Sorry to update again, but I want to cover both newer and older gc implementations. The egrep discussed above is now:... J. Eric Ivancich
09:14 PM Bug #47866: Object not found on healthy cluster
Eric Ivancich wrote:
> a. if you have a bucket that doesn't get deletes then using the egrep command above will he...
Denis Krienbühl
08:55 PM Bug #47866: Object not found on healthy cluster
Denis Krienbühl wrote:
> Eric Ivancich wrote:
> > If you have a bucket where things get uploaded and maybe read but...
J. Eric Ivancich
07:43 PM Bug #47866: Object not found on healthy cluster
Eric Ivancich wrote:
> If you have a bucket where things get uploaded and maybe read but rarely deleted, and if you'...
Denis Krienbühl
07:30 PM Bug #47866: Object not found on healthy cluster
Just to clarify on why this is helpful: "If you have a bucket where things get uploaded and maybe read but rarely del... J. Eric Ivancich
07:21 PM Bug #47866: Object not found on healthy cluster
We realize not everyone is able to do this, but for those who can, here is one approach we're using to try to nail th... J. Eric Ivancich
07:14 PM Bug #47866: Object not found on healthy cluster
If the multipart components are themselves larger than the striping size, they two can have shadow tail objects, as s... J. Eric Ivancich
03:04 PM Bug #47866: Object not found on healthy cluster
Update: forget what I said. When I retrieve the ETag via boto3, I get 57ba90377073200f91b5dbee7aaece6-64 instead. I a... Janek Bevendorff
03:03 PM Bug #47866: Object not found on healthy cluster
Thank you, I'm glad to see the @__multipart@ files in there, as that's what I'm counting on.
And just in case some...
Denis Krienbühl
02:56 PM Bug #47866: Object not found on healthy cluster
I crated a bucket and uploaded two 1GB multipart objects. Here's what the radoslist looks like:... Janek Bevendorff
01:55 PM Bug #47866: Object not found on healthy cluster
Update: It seems like @refcount.put@ is the call that logs the @__multipart@ file about to be deleted. Denis Krienbühl
01:18 PM Bug #47866: Object not found on healthy cluster
Janek Bevendorff wrote:
> Yes, it is uploading multiple parts and the resulting RADOS objects live in the multipart ...
Denis Krienbühl
12:55 PM Bug #47866: Object not found on healthy cluster
Yes, it is uploading multiple parts and the resulting RADOS objects live in the multipart namespace, not the shadow n... Janek Bevendorff
12:39 PM Bug #47866: Object not found on healthy cluster
Janek Bevendorff wrote:
> I wanted to reupload our multipart objects as single parts and noticed something very stra...
Denis Krienbühl
12:03 PM Bug #47866: Object not found on healthy cluster
I wanted to reupload our multipart objects as single parts and noticed something very strange: Not all multipart obje... Janek Bevendorff
08:55 AM Bug #47866: Object not found on healthy cluster
Sorry if that was unclear. The majority of all objects we have in total are single-part objects. The affected objects... Janek Bevendorff
08:34 AM Bug #47866: Object not found on healthy cluster
Janek Bevendorff wrote:
> I don't think the issue is packet loss per se. The objects in our affected buckets were al...
Denis Krienbühl
08:34 AM Bug #47866: Object not found on healthy cluster
Janek Bevendorff wrote:
> I don't think the issue is packet loss per se. The objects in our affected buckets were al...
Denis Krienbühl
08:30 AM Bug #47866: Object not found on healthy cluster
I don't think the issue is packet loss per se. The objects in our affected buckets were all filled by a Hadoop job ru... Janek Bevendorff
08:12 AM Bug #47866: Object not found on healthy cluster
Another update from our side: We have two sites, a larger and a smaller one. The smaller site was successfully scanne... Denis Krienbühl
04:56 PM Bug #47800 (Resolved): rgw/tempest test failure in test_object_upload_in_segments
Casey Bodley
04:56 PM Bug #47801 (Resolved): rgw/tempest test failure in test_get_object_with_x_object_manifest
Casey Bodley
12:43 PM Bug #47910: radosgw crash on objecter operations
Thanks for confirming on large buckets/EC pools.
Correct; that patch is the only change in the ceph packages in th...
Mauricio Oliveira
06:00 AM Bug #47910: radosgw crash on objecter operations
Both of them.
I checked your links. I should implement https://github.com/cbodley/ceph/commit/d23507bd1295a29ccae3...
Rafal Wadolowski
12:23 AM Bug #47910: radosgw crash on objecter operations
Hey Rafal,
Thanks!
By the way, does your ceph cluster have:
1) bucket(s) with a large number of objects?
2)...
Mauricio Oliveira
02:01 AM Bug #48289 (Fix Under Review): prefetch in rgw_file is done 3 times per read
Matt Benjamin

11/19/2020

05:29 PM Documentation #23027: document rgw_multipart_min_part_size

There are over 1500 options in the source as of today. `rgw_multipart_min_part_size` is marked as `ADVANCED`, and I...
Anthony D'Atri
04:57 PM Bug #47588 (Resolved): sts test failures
Casey Bodley
04:02 PM Bug #48289: prefetch in rgw_file is done 3 times per read
please backport the fix to Nautilus Kajetan Janiak
09:48 AM Bug #48289 (Resolved): prefetch in rgw_file is done 3 times per read
Each call to rgw_read (rgw_file.cc) invokes three calls to RGWRados::get_obj_state with RGWObjState::prefetch_data = ... Kajetan Janiak
03:30 PM Bug #47984: rgw: filter marker in list vesions
lei cao wrote:
> https://github.com/ceph/ceph/pull/37848
This PR has been closed as a result of another PR furth...
J. Eric Ivancich
03:29 PM Bug #47984 (In Progress): rgw: filter marker in list vesions
A different PR is close to a solution, so the PR is updated.
https://github.com/ceph/ceph/pull/37790
J. Eric Ivancich
03:24 PM Bug #48064 (Pending Backport): rgw: unnecessary payload is added at the end of the message
Or Friedmann
02:41 PM Bug #48293 (Resolved): notification: radosgw-admin hangs on while closing
when "data sync run" command is executed on a pubsub zone, the code creates the rabbitmq manager (as part initializin... Yuval Lifshitz
01:23 PM Bug #48241: librgw double read
Keep in mind that if you're seeing the same read dispatched from nfs-ganesha twice, it most likely was issued twice. ... Matt Benjamin
01:10 PM Bug #48241: librgw double read
Please be advised, librgw is somewhat broken on master, mostly due to zipper changes. The known issues are fixed in ... Matt Benjamin
10:14 AM Bug #48241: librgw double read
Or Friedmann wrote:
> Is it possible to add the osd.log with debug_objclass 20 and debug_ms 1? (I want to see the cl...
chunsong feng
11:12 AM Backport #47836: octopus: Add OIDC provider support in RGW STS
ok, thank you for the clarification. Pritha Srivastava
11:10 AM Backport #47836: octopus: Add OIDC provider support in RGW STS
Pritha Srivastava wrote:
> The target version for this PR is v15.2.6, but when I check the v15.2.6 release I do not ...
Nathan Cutler
06:57 AM Backport #47836: octopus: Add OIDC provider support in RGW STS
Hi @Nathan,
The target version for this PR is v15.2.6, but when I check the v15.2.6 release I do not see the relat...
Pritha Srivastava
10:26 AM Bug #47866: Object not found on healthy cluster
An update on our efforts: We enabled @debug_ms 1@ and we are trying to find an object that vanishes while these logs ... Denis Krienbühl
10:21 AM Backport #48101 (In Progress): octopus: Admin API returns 200 instead of 404 for Get Bucket Info
Nathan Cutler
07:43 AM Bug #47910: radosgw crash on objecter operations
Hi Mauricio,
I am using package from download.ceph.com.
I can test the fix, I will back with the answers until en...
Rafal Wadolowski
02:19 AM Documentation #25061: Wrong figure in multisite documentation
Which figure? What's wrong with it? Anthony D'Atri

11/18/2020

05:11 PM Feature #46296 (Fix Under Review): notifications: support GetTopicAttributes API
Yuval Lifshitz
05:11 PM Bug #47910: radosgw crash on objecter operations
Hi Rafal and Or,
Rafal,
Could you please confirm if you are using ceph packages from Ubuntu?
(say, Ubuntu Clou...
Mauricio Oliveira
04:19 PM Bug #47866: Object not found on healthy cluster
We're hitting this problem in our 15.2.5 cluster as well. So far we know of one S3 that went missing, in a typical Do... Denis Krienbühl
01:07 PM Bug #47866: Object not found on healthy cluster
I believe client-rgw is doing the job. The default here seems to be 0/0 and when I turn it back off, I have nothing i... Janek Bevendorff
12:59 PM Bug #47866: Object not found on healthy cluster
Janek Bevendorff wrote:
> The config says the default for rgw is 1/5, that's why I was wondering. https://docs.ceph....
Dan van der Ster
12:54 PM Bug #47866: Object not found on healthy cluster
The config says the default for rgw is 1/5, that's why I was wondering. https://docs.ceph.com/en/latest/rados/trouble... Janek Bevendorff
10:21 AM Bug #47866: Object not found on healthy cluster
Janek Bevendorff wrote:
> I have enabled the setting via
>
> [...]
>
> I will check the logs when the next obj...
Dan van der Ster
10:20 AM Bug #47866: Object not found on healthy cluster
Janek Bevendorff wrote:
> Short question here: isn't 1 the default log level anyway? I couldn't find any such messag...
Dan van der Ster
08:33 AM Bug #47866: Object not found on healthy cluster
Short question here: isn't 1 the default log level anyway? I couldn't find any such messages in previous logs at least. Janek Bevendorff
08:28 AM Bug #47866: Object not found on healthy cluster
I have enabled the setting via... Janek Bevendorff
04:25 AM Bug #47866: Object not found on healthy cluster
In Comment #11 I asked: Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'?
Let me fol...
J. Eric Ivancich
09:38 AM Bug #48241: librgw double read
Or Friedmann wrote:
> Is it possible to add the osd.log with debug_objclass 20 and debug_ms 1? (I want to see the cl...
chunsong feng

11/17/2020

11:54 PM Bug #48268 (New): rgw: when removing a current versioned object, the new current object doesn't g...
E.g., we have an object with two versions, older one has epoch=2, new one has epoch=3. When removing the epoch=3 obje... Yehuda Sadeh
02:59 PM Bug #47904 (Fix Under Review): notifications not sent when non-owner modifies the bucket
Yuval Lifshitz
02:39 PM Bug #48262 (Resolved): rgw: RGWListMultipart show Owner ID and DisplayName
the owner ID and DisplayName should not be empty when list parts
<?xml version="1.0" encoding="UTF-8"?>
<ListPart...
joke lee
01:09 PM Bug #48255 (Fix Under Review): rgwlc: shard-index vector short by 1?
Matt Benjamin
01:25 AM Bug #48255 (Resolved): rgwlc: shard-index vector short by 1?
The new parallel lc code relies on a vector of indices to seed
random-order shard processing. This vector was dimen...
Matt Benjamin
08:20 AM Bug #47866: Object not found on healthy cluster
I waited for the first few minor releases and upgraded from Nautilus directly to 15.2.5, so yes. Janek Bevendorff
02:11 AM Bug #47866: Object not found on healthy cluster
Dan van der Ster wrote:
> Has #42670 been excluded?
So the first version of Octopus that that fix appears in is 1...
J. Eric Ivancich
06:54 AM Bug #48208 (Fix Under Review): rgw:usage:when trim usage by user,it is possible to trim other use...
Kefu Chai

11/16/2020

09:33 PM Bug #47866: Object not found on healthy cluster
Janek Bevendorff wrote:
> What I do remember is error messages about compression during OSD upgrades (i.e., compre...
J. Eric Ivancich
05:20 PM Bug #47866: Object not found on healthy cluster
Interesting issue, but it doesn't really look like it's the same thing for two reasons:
1) We never saw this kind ...
Janek Bevendorff
08:56 AM Bug #47866: Object not found on healthy cluster
Has #42670 been excluded? Dan van der Ster
08:40 AM Bug #47866: Object not found on healthy cluster
So far, the issue has only affected multipart objects. I have already suspected that this might be part of the issue,... Janek Bevendorff
03:48 PM Bug #48250 (Resolved): notifications: only bucket owner is allowed to add/remove notifications
instead, anyone that has permission on the bucket should be allowed to add/remove notifications.
"verify_bucket_owne...
Yuval Lifshitz
03:45 PM Bug #48236 (Fix Under Review): rgw: fix auth when date is RFC-822/ISO 8601 standard timezone spec...
Casey Bodley
02:53 PM Bug #47799 (Resolved): rgw/crypt suite with Barbican fails with "Cannot create secret"
Casey Bodley
10:15 AM Bug #47799 (Fix Under Review): rgw/crypt suite with Barbican fails with "Cannot create secret"
Or Friedmann
11:52 AM Bug #48241: librgw double read
Is it possible to add the osd.log with debug_objclass 20 and debug_ms 1? (I want to see the cls call logs and the obj... Or Friedmann
02:06 AM Bug #48241 (Need More Info): librgw double read
I test read a 3MB object with this rgw: Fix librgw end define patch,According to the logs, there are still two ops.
...
chunsong feng
08:01 AM Bug #48208: rgw:usage:when trim usage by user,it is possible to trim other users’ usage.
chengwu liang wrote:
> when trim usage by specifying user,it is possible to trim other users’ usage.
> Steps:
> 1、...
chengwu liang

11/15/2020

05:57 PM Backport #48239: octopus: list object versions returned multiple 'IsLatest true' entries
h3. original description
backport to octopus,
parent tracker: https://tracker.ceph.com/issues/47919
Nathan Cutler
05:56 PM Backport #48239 (In Progress): octopus: list object versions returned multiple 'IsLatest true' en...
Nathan Cutler
02:48 PM Backport #48239 (Resolved): octopus: list object versions returned multiple 'IsLatest true' entries
https://github.com/ceph/ceph/pull/38086 Zitao Ruan
05:54 PM Backport #48238 (In Progress): nautilus: list object versions returned multiple 'IsLatest true' e...
Nathan Cutler
02:44 PM Backport #48238: nautilus: list object versions returned multiple 'IsLatest true' entries
https://github.com/ceph/ceph/pull/38085 Zitao Ruan
02:37 PM Backport #48238 (Resolved): nautilus: list object versions returned multiple 'IsLatest true' entries
https://github.com/ceph/ceph/pull/38085
---
backport to nautilus: https://tracker.ceph.com/issues/47919
Zitao Ruan
11:19 AM Bug #47866: Object not found on healthy cluster
Janek Bevendorff wrote:
> Indeed. I just noticed the other object isn't a multipart object. I attached the radosgw-a...
Mark Kogan
11:09 AM Bug #47866: Object not found on healthy cluster
Julien Demais wrote:
> No one is using my ceph cluster so far. I am in the process of evaluating ceph so I am the on...
Mark Kogan

11/14/2020

12:59 PM Bug #47866: Object not found on healthy cluster
Please find my answers below:
1. It seems like those reporting this issue are seeing it strictly after upgrading t...
Julien Demais
04:33 AM Bug #48236 (Fix Under Review): rgw: fix auth when date is RFC-822/ISO 8601 standard timezone spec...
in amazon s3, it is support date format with timezone
> PUT /yuliyangwebsite/1G-16 HTTP/1.1
> Host: s3.amazonaws....
joke lee

11/13/2020

07:19 PM Bug #47866: Object not found on healthy cluster
@Julien Demais and ANYONE ELSE experiencing this issue,
I posted these questions on ceph-users but wanted to see i...
J. Eric Ivancich
07:16 PM Bug #47866: Object not found on healthy cluster
I'm copying in answers Janek gave to questions I posted on ceph-users.
1. It seems like those reporting this iss...
J. Eric Ivancich
08:18 AM Bug #47866: Object not found on healthy cluster
Indeed. I just noticed the other object isn't a multipart object. I attached the radosgw-admin stat dump of a working... Janek Bevendorff
06:08 PM Bug #47801 (Fix Under Review): rgw/tempest test failure in test_get_object_with_x_object_manifest
Nathan Cutler
02:03 PM Bug #48049 (Resolved): rgw: getting S3 website root without last slash crashes rgw
Daniel Gryniewicz

11/12/2020

09:13 PM Bug #47866: Object not found on healthy cluster
I did some experiments, and I don't think the head object having a size of 0 is necessarily an indicator of a problem... J. Eric Ivancich
05:11 PM Bug #47866: Object not found on healthy cluster
I have not set any lifecycle policy on the bucket and I double checked that there is indeed none. s3cmd getlifecycle ... Janek Bevendorff
05:04 PM Bug #47866: Object not found on healthy cluster
I am not running Ceph in Kubernetes.
I attached a config dump and a stat dump of an OK object. All sensitive infor...
Janek Bevendorff
03:45 PM Bug #47866: Object not found on healthy cluster
May we also please ask to attach the ceph.conf
and `radosgw-admin object stat` of a similarly sized OK object from t...
Mark Kogan
03:17 PM Bug #47866 (Triaged): Object not found on healthy cluster
Casey Bodley
03:12 PM Bug #47866: Object not found on healthy cluster
Do you have any other storage classes configured? Is there a lifecycle policy on the bucket? Casey Bodley
01:05 PM Bug #47866: Object not found on healthy cluster
Can you check that the RADOS object is larger than 0 bytes? This is a creeping data loss that has been reported on Oc... Janek Bevendorff
03:23 PM Bug #46567: Access denied for multi-object-delete by non-bucket-owner
included in https://github.com/ceph/ceph/pull/37933 Abhishek Lekshmanan
03:19 PM Bug #47799 (In Progress): rgw/crypt suite with Barbican fails with "Cannot create secret"
Or Friedmann
02:11 AM Bug #48208 (Fix Under Review): rgw:usage:when trim usage by user,it is possible to trim other use...
when trim usage by user,it is possible to trim other users’ usage.
Steps:
1、set rgw enable usage log = true and rgw...
chengwu liang

11/10/2020

09:31 AM Bug #47919: list object versions returned multiple 'IsLatest true' entries
Matt Benjamin wrote:
> Is target version Nautilus really correct (linked fix is against master). I.e., do we need b...
Honggang Yang

11/09/2020

03:04 PM Bug #48034 (Fix Under Review): Should rgw::auth::Strategy::apply be noexcept?
Casey Bodley
02:50 PM Bug #46439: librgw: fhcache deadlock
Also, Xuehan Xu, the call stack provided with this report is from librgw running under FUSE. I wasn't aware of an op... Matt Benjamin
02:36 PM Bug #46439: librgw: fhcache deadlock
Thanks for the detailed analysis, which seems potentially valid.
Could you report status after merge of the fix at...
Matt Benjamin
02:27 PM Bug #47919: list object versions returned multiple 'IsLatest true' entries
Is target version Nautilus really correct (linked fix is against master). I.e., do we need backports for Octopus and... Matt Benjamin

11/08/2020

11:43 AM Cleanup #48146 (In Progress): fix compilation error: declaration conflicts with target of using d...
when compiling with clang the following error occurs:... Mark Kogan
09:43 AM Bug #48073: radosgw_admin_rest failure: bucket info API gets NoSuchKey, expects NoSuchBucket
(This tracker issue is a followup to #45193) Mark Kogan

11/06/2020

05:47 PM Bug #48136 (Resolved): rgw_file: fix Zipper integration in RGWLibContinuedReq
These fixes aren't needed by any current Ceph release, so there's nothing to backport. Matt Benjamin
09:58 AM Bug #48139 (Duplicate): Ceph Dashboard Object Gateway InvalidRange bucket exception
We've setup Ceph Octopus 15.2.5 cluster in CentOS8. Ceph cluster is up and able to view the dashboard
1) when we t...
kasi viswanadh

11/05/2020

05:18 PM Bug #48136 (Fix Under Review): rgw_file: fix Zipper integration in RGWLibContinuedReq
Matt Benjamin
05:15 PM Bug #48136 (Resolved): rgw_file: fix Zipper integration in RGWLibContinuedReq
Matt Benjamin
03:35 PM Bug #48034 (In Progress): Should rgw::auth::Strategy::apply be noexcept?
Casey Bodley
03:24 PM Bug #48049 (Fix Under Review): rgw: getting S3 website root without last slash crashes rgw
Casey Bodley
03:16 PM Bug #48064 (Fix Under Review): rgw: unnecessary payload is added at the end of the message
Casey Bodley
03:14 PM Bug #48115 (Closed): rgw stops responding correctly, error log full of messages: ceph s3:get_obj ...
Casey Bodley
03:11 PM Bug #48115: rgw stops responding correctly, error log full of messages: ceph s3:get_obj Schedulin...
Hi,
2218 is ERR_RATE_LIMITED, by default each rgw can handle up to 1000, configured by this parameter rgw max_conc...
Or Friedmann
03:05 PM Bug #45193: Admin API returns 200 instead of 404 for Get Bucket Info
please include the followup fix from https://tracker.ceph.com/issues/48073 Casey Bodley
03:04 PM Bug #48073 (Resolved): radosgw_admin_rest failure: bucket info API gets NoSuchKey, expects NoSuch...
Casey Bodley
03:03 PM Bug #48073 (Fix Under Review): radosgw_admin_rest failure: bucket info API gets NoSuchKey, expect...
Casey Bodley
02:38 PM Backport #48128 (Resolved): nautilus: Unnecessary bilogs are left in sync-disabled buckets
https://github.com/ceph/ceph/pull/38589 Nathan Cutler
02:37 PM Backport #48127 (Resolved): octopus: Unnecessary bilogs are left in sync-disabled buckets
https://github.com/ceph/ceph/pull/38338 Nathan Cutler

11/04/2020

09:08 PM Bug #48122: rgw cannot find keyring after config file is minimized
Moved to "Ceph" project since it's a bug in the implementation of "ceph config generate-minimal-conf". Nathan Cutler
08:25 PM Bug #48122 (Won't Fix): rgw cannot find keyring after config file is minimized
Given a ceph.conf file like this:... Wyllys Ingersoll
03:08 PM Bug #47882 (Resolved): rgw: crash on non S3/swift requests (eg. admin api requests)
No backports are listed, so I moved this to resolved. If backports are appropriate then please update the Backport an... J. Eric Ivancich
03:07 PM Bug #47472 (Resolved): rgw: lifecycle Filter clauses with Prefix and one Tag are valid
J. Eric Ivancich
03:07 PM Bug #47472: rgw: lifecycle Filter clauses with Prefix and one Tag are valid
No backports are listed, so I'm switching to resolved. If backports are appropriate, please update Backport and Statu... J. Eric Ivancich
03:04 PM Bug #48037 (Pending Backport): Unnecessary bilogs are left in sync-disabled buckets
J. Eric Ivancich
02:44 PM Bug #48115 (Closed): rgw stops responding correctly, error log full of messages: ceph s3:get_obj ...
rgw stops responding correctly, error log full of messages: ceph s3:get_obj Scheduling request failed with -2218.
...
Matthew Darwin
09:27 AM Bug #46614 (Resolved): rgw lc expiration header returns although it should not
While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ... Nathan Cutler
09:26 AM Bug #47235 (Resolved): rgw/rgw_file: incorrect lru object eviction in lookup_fh
While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ... Nathan Cutler
09:26 AM Bug #47408 (Resolved): rgw: orphan list teuthology test uses `dnf`, which may not always be avail...
While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ... Nathan Cutler
09:25 AM Bug #47710 (Resolved): rgw_file: avoid long delay on shutdown
While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ... Nathan Cutler
09:25 AM Bug #47720 (Resolved): rgw: radosgw-admin does not paginate internally when listing bucket
While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ... Nathan Cutler
09:25 AM Fix #48000 (Resolved): rgw: fix S3 API KeyCount incorrect return
While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ... Nathan Cutler
09:24 AM Backport #48004 (Resolved): nautilus: rgw: fix S3 API KeyCount incorrect return
This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37810
m...
Nathan Cutler
09:23 AM Backport #47959 (Resolved): nautilus: rgw lc expiration header returns although it should not
This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37806
m...
Nathan Cutler
09:23 AM Backport #47897 (Resolved): nautilus: rgw: orphan list teuthology test uses `dnf`, which may not ...
This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37805
m...
Nathan Cutler
09:23 AM Backport #47851 (Resolved): nautilus: rgw/rgw_file: incorrect lru object eviction in lookup_fh
This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37804
m...
Nathan Cutler
09:23 AM Backport #47818 (Resolved): nautilus: rgw: radosgw-admin does not paginate internally when listin...
This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37802
m...
Nathan Cutler
09:22 AM Backport #47735 (Resolved): nautilus: rgw_file: avoid long delay on shutdown
This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37552
m...
Nathan Cutler

11/03/2020

07:48 PM Backport #48004: nautilus: rgw: fix S3 API KeyCount incorrect return
Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/37810
merged
Yuri Weinstein
07:48 PM Backport #47959: nautilus: rgw lc expiration header returns although it should not
Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/37806
merged
Yuri Weinstein
07:47 PM Backport #47897: nautilus: rgw: orphan list teuthology test uses `dnf`, which may not always be a...
Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/37805
merged
Yuri Weinstein
07:46 PM Backport #47851: nautilus: rgw/rgw_file: incorrect lru object eviction in lookup_fh
Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/37804
merged
Yuri Weinstein
07:46 PM Backport #47818: nautilus: rgw: radosgw-admin does not paginate internally when listing bucket
Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/37802
merged
Yuri Weinstein
07:41 PM Backport #47735: nautilus: rgw_file: avoid long delay on shutdown
Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/37552
merged
Yuri Weinstein
03:48 PM Bug #48103 (New): Bilogs automatic trimming fails (multisite, ceph nautilus 14.2.9)
Issue
=====
We're seeing a problem in our multisite Ceph deployment, where automatic bilog trimming has stopped acr...
David Piper
03:09 PM Cleanup #47848 (Closed): use std::unique_ptr in cls_fifo_legacy.cc
this is already fixed here: https://github.com/ceph/ceph/pull/37660 Yuval Lifshitz
02:46 PM Bug #16767: RadosGW Multipart Cleanup Failure
I've encountered this bug in Ceph 14 and 15 and it's a pretty big problem for us for the same reason Brian Felton men... Vicki Good
12:54 PM Bug #47586 (Fix Under Review): Able to circumvent S3 Object Lock using deleteobjects command
Nathan Cutler
11:49 AM Bug #47586: Able to circumvent S3 Object Lock using deleteobjects command
Pull request to fix this: https://github.com/ceph/ceph/pull/37933
Updates to S3 tests: https://github.com/ceph/s3-te...
Mark Houghton
11:28 AM Backport #48101 (Resolved): octopus: Admin API returns 200 instead of 404 for Get Bucket Info
https://github.com/ceph/ceph/pull/38184 Nathan Cutler
11:28 AM Backport #48100 (Resolved): nautilus: Admin API returns 200 instead of 404 for Get Bucket Info
https://github.com/ceph/ceph/pull/38588 Nathan Cutler
11:28 AM Backport #48099 (Rejected): mimic: Admin API returns 200 instead of 404 for Get Bucket Info
Nathan Cutler
11:27 AM Bug #46400 (Resolved): Space usage accounting overestimated
While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ... Nathan Cutler
11:25 AM Bug #47771 (Resolved): rgw: allow rgw-orphan-list to note when rados objects in namespace
While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ... Nathan Cutler
11:25 AM Bug #47780 (Resolved): rgw: fix setting of namespace in ordered and unordered bucket listing
While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ... Nathan Cutler
11:25 AM Bug #47812 (Resolved): rgw: rgw-orphan-list should use "plain" formatted `rados ls` output
While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ... Nathan Cutler
11:25 AM Bug #47908 (Resolved): GC perfcounter fails to update when deletion occurs
While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ... Nathan Cutler
11:25 AM Bug #47909 (Resolved): list pending GCs is very slow
While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ... Nathan Cutler
10:19 AM Backport #48003 (Resolved): octopus: rgw: fix S3 API KeyCount incorrect return
This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37849
m...
Nathan Cutler
10:19 AM Backport #47956 (Resolved): octopus: GC perfcounter fails to update when deletion occurs
This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37847
m...
Nathan Cutler
10:19 AM Backport #47955 (Resolved): octopus: list pending GCs is very slow
This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37846
m...
Nathan Cutler
09:58 AM Backport #47896 (Resolved): octopus: rgw: orphan list teuthology test uses `dnf`, which may not a...
This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37845
m...
Nathan Cutler
09:57 AM Backport #47962 (Resolved): octopus: beast frontend option to set the request_timeout_ms
This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37809
m...
Nathan Cutler
09:57 AM Backport #47960 (Resolved): octopus: rgw lc expiration header returns although it should not
This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37807
m...
Nathan Cutler
09:57 AM Backport #47819 (Resolved): octopus: rgw: radosgw-admin does not paginate internally when listing...
This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37803
m...
Nathan Cutler
09:57 AM Backport #47938 (Resolved): octopus: rgw: rgw-orphan-list should use "plain" formatted `rados ls`...
This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37800
m...
Nathan Cutler
09:56 AM Backport #47817 (Resolved): octopus: rgw: allow rgw-orphan-list to note when rados objects in nam...
This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37800
m...
Nathan Cutler
09:56 AM Backport #47037 (Resolved): octopus: rgw: Space usage accounting overestimated
This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37779
m...
Nathan Cutler
09:18 AM Bug #47910: radosgw crash on objecter operations
Right.
I want to make sure this is not a tcmalloc bug (on one of the nodes, you are using for testing this error).
Or Friedmann
06:48 AM Bug #48073: radosgw_admin_rest failure: bucket info API gets NoSuchKey, expects NoSuchBucket
https://github.com/ceph/ceph/pull/37929 lei cao

11/02/2020

08:13 PM rgw-testing Bug #47795 (Resolved): rgw: s3-tests: when I try to run locally I get 'Your config file is missin...
J. Eric Ivancich
08:09 PM rgw-testing Bug #47795: rgw: s3-tests: when I try to run locally I get 'Your config file is missing the "iam"...
Eric Ivancich wrote:
> Re-opened at request of Ali Maredia. He does not think the tests should fail if there is no i...
Kalpesh Pandya
05:44 PM Bug #47910: radosgw crash on objecter operations
This one from focal 2.7? Right now I'm testing on bionic with 2.5-2.2ubuntu3 Rafal Wadolowski
05:28 PM Bug #47910: radosgw crash on objecter operations
The swift healthcheck is not making a RADOS call so this is the reason I guess.
Can you please update this rgw to a ...
Or Friedmann
05:19 PM Bug #47910: radosgw crash on objecter operations
Or,
focal is not supported for nautilus, I tested with bionic version and this same effect.
I can't understand, why...
Rafal Wadolowski
12:44 PM Bug #47910: radosgw crash on objecter operations
Thanks for the fast response,
Is it possible to install on the specific RGW newer version of tcmalloc(https://laun...
Or Friedmann
11:04 AM Bug #47910: radosgw crash on objecter operations
Here you are ... Rafal Wadolowski
10:17 AM Bug #47910: radosgw crash on objecter operations
Thank you Rafal,
Can you please provide this command output:
ceph versions and ceph.conf
Or Friedmann
05:17 PM Bug #47801 (Pending Backport): rgw/tempest test failure in test_get_object_with_x_object_manifest
Casey Bodley
05:17 PM Bug #47800 (Fix Under Review): rgw/tempest test failure in test_object_upload_in_segments
Casey Bodley
04:50 PM Bug #48073 (Resolved): radosgw_admin_rest failure: bucket info API gets NoSuchKey, expects NoSuch...
... Casey Bodley
07:41 AM Bug #48064: rgw: unnecessary payload is added at the end of the message
I made a related PR: https://github.com/ceph/ceph/pull/37914 Ilsoo Byun
07:10 AM Bug #48064 (Resolved): rgw: unnecessary payload is added at the end of the message
Configured s3website as follows:... Ilsoo Byun
 

Also available in: Atom