Activity
From 10/06/2022 to 11/04/2022
11/04/2022
- 03:00 PM Bug #57911 (In Progress): Segmentation fault when uploading file with bucket policy on Quincy
11/03/2022
- 07:11 PM Bug #57562: multisite replication issue on Quincy
- We are still testing the latest evidence (HEAD at https://github.com/ceph/ceph/commit/cfc3bde36dbc9c6e0b7182bbb325390...
- 07:02 PM Bug #57936 (Fix Under Review): 'radosgw-admin bucket chown' doesn't set bucket instance owner or ...
- 02:16 PM Bug #57936 (In Progress): 'radosgw-admin bucket chown' doesn't set bucket instance owner or unlin...
- 02:13 PM Bug #57968 (New): Partial fix for XML responses returning different order of XML elements
- Hi
This is a follow up on original problem reported here
https://tracker.ceph.com/issues/52027
I've added my com... - 02:13 PM Bug #57951 (Fix Under Review): rgw: lc: lc for a single large bucket can run too long
- 02:03 PM Bug #57724 (In Progress): Keys returned by Admin API during user creation on secondary zone not v...
- 08:50 AM Bug #44660: Multipart re-uploads cause orphan data
- As it was discussed in [1] there is already a wip PR with more generic solution [2].
[1] https://github.com/ceph/c...
11/02/2022
- 08:27 PM Feature #57965 (Resolved): Add new zone option to control whether an object's first data stripe i...
- Delete requests are quite slow on clusters that have a data pool backed by HDDs, especially with an EC pool. For exam...
- 06:58 PM Bug #57562: multisite replication issue on Quincy
- Yeah, both those commits are gone, make sure you only have the newest one.
- 06:33 PM Bug #57562: multisite replication issue on Quincy
- Adam Emerson wrote:
> Pushed a new version with what should be a fix for multi-thread and multi-client races.
We ... - 07:22 AM Bug #57562: multisite replication issue on Quincy
- Pushed a new version with what should be a fix for multi-thread and multi-client races.
- 08:07 AM Bug #57942: rgw leaks rados objects when a part is submitted multiple times in a multipart upload
- FI pull-request https://github.com/ceph/ceph/pull/48704
11/01/2022
- 08:12 PM Bug #57942: rgw leaks rados objects when a part is submitted multiple times in a multipart upload
- FI Working on https://github.com/pgoron/ceph/commits/fix_rgw_rados_leaks_57942 to fix both issues (index entry leaks ...
- 07:25 PM Bug #57562: multisite replication issue on Quincy
- Agree as you mentioned, the other solution could be, secondary not limited to just listening on to orpan part, but co...
- 06:37 PM Bug #57562: multisite replication issue on Quincy
- Ah, I see, I need to update the async lister.
- 06:36 PM Bug #57562: multisite replication issue on Quincy
- That's the point of the commit `rgw/fifo: `part_full` is not a reliable indicator`. There is no 'orphan part' in that...
- 05:39 PM Bug #57562: multisite replication issue on Quincy
- Hey Adam,
Just a heads-up we tested with latest commit and we still see the issue.
The issue is seen when running M... - 02:11 PM Bug #57562: multisite replication issue on Quincy
- Thank you Adam. We'll test with the latest change.
- 04:27 PM Bug #44660 (Fix Under Review): Multipart re-uploads cause orphan data
- Actually it looks like there is a simpler solution to this problem, which uses the meta object lock when checking if ...
10/31/2022
- 06:56 PM Bug #57562: multisite replication issue on Quincy
- Pushed a new version that should make listing list all the objects reliably.
- 04:40 PM Bug #57951 (Pending Backport): rgw: lc: lc for a single large bucket can run too long
- If this happens, other lc hosts/threads can attempt to process the same bucket, which inflates overhead without any c...
- 09:44 AM Feature #57947 (Pending Backport): Improve performance of multi-object delete by handling individ...
- Multi-object deletes are currently quite slow. The handler for this method currently just loops through the list of o...
- 08:10 AM Bug #57942: rgw leaks rados objects when a part is submitted multiple times in a multipart upload
- After digging more on the issue, I think the root cause is linked to following code:
https://github.com/ceph/ceph/...
10/29/2022
- 01:32 AM Bug #57853: multisite sync process block after long time running
- I think something wrong with rgw-coroutine,please check the above PR
- 01:30 AM Bug #57853: multisite sync process block after long time running
- PR https://github.com/ceph/ceph/pull/48626
10/28/2022
- 09:10 PM Bug #57770: RGW (pacific) misplaces index entries after dynamically resharding bucket
- J. Eric Ivancich wrote:
> Nick,
>
> I don't know that I have a cluster at my fingertips that might be necessary t... - 07:23 PM Bug #57770: RGW (pacific) misplaces index entries after dynamically resharding bucket
- Nick,
I don't know that I have a cluster at my fingertips that might be necessary to test this potential fix. How ... - 07:21 PM Bug #57770 (Fix Under Review): RGW (pacific) misplaces index entries after dynamically resharding...
- 12:36 PM Bug #57942 (Duplicate): rgw leaks rados objects when a part is submitted multiple times in a mult...
- Hello,
Issue presented below affects all ceph versions at least since 14.2 (reproducer tested on 14.2, 15.2, 16.2,...
10/27/2022
- 09:21 PM Bug #57562: multisite replication issue on Quincy
- Pushed a newer, newer fix that guards all calls to _prepare_new_head behind check/set of preparing.
- 04:15 PM Bug #57562: multisite replication issue on Quincy
- Pushed a newer fix that does the check in need_new_head()
- 02:01 PM Bug #57562: multisite replication issue on Quincy
- Hi Adam,
We obtained the extra logging with the fix in place.
I think the contention is not within _prepare_ne... - 01:09 AM Bug #57562: multisite replication issue on Quincy
- I expect there are multiple problems with sync in Quincy, so I don't expect this to actually make sync work.
But i... - 12:15 AM Bug #57562: multisite replication issue on Quincy
- Pulled the changes in on top of the commit _9056dbcdeaa7f4350b54a69f669982358ec5448e_ (on main branch). Unfortunately...
- 02:31 PM Bug #57928 (Duplicate): Octopus:multisite sync process block after long time running
- 02:31 PM Bug #57927 (Duplicate): pacific:multisite sync process block after long time running
- 07:36 AM Cleanup #57938 (Pending Backport): relying on boost flatmap emplace behavior is risky
- see coverity issue: http://folio07.front.sepia.ceph.com/main/ceph-main-98d41855/cov-main-html/3/2253rgw_trim_bilog.cc...
10/26/2022
- 10:01 PM Bug #57936 (Pending Backport): 'radosgw-admin bucket chown' doesn't set bucket instance owner or ...
- steps to reproduce:
1. start a vstart cluster and create a bucket as user 'testid'... - 05:08 PM Bug #57562: multisite replication issue on Quincy
- Awesome! Thanks for the quick turn around! Will pull and test.
- 04:49 PM Bug #57562 (Fix Under Review): multisite replication issue on Quincy
- I have a candidate fix at https://github.com/ceph/ceph/pull/48632
- 02:14 PM Bug #57562: multisite replication issue on Quincy
- FYI: We pulled in the 2 PRs Casey posted in the tracker https://tracker.ceph.com/issues/57783, and tested again with ...
- 12:31 PM Bug #57562: multisite replication issue on Quincy
- FWIW, below provides some log snippets with enhanced events. To be specific, some existing log events are added addit...
- 03:48 AM Bug #57853: multisite sync process block after long time running
- Quincy、Pacific、Octopus、 Nautilus has same issue
- 03:27 AM Bug #57928 (Duplicate): Octopus:multisite sync process block after long time running
- 1、deploy RADOSGW multisite
2、put lot of objects
3、keep it runing for a long time - 03:25 AM Bug #57927: pacific:multisite sync process block after long time running
- same as https://tracker.ceph.com/issues/57853
- 03:24 AM Bug #57927 (Duplicate): pacific:multisite sync process block after long time running
- 1、deploy RADOSGW multisite
2、put lot of objects
3、keep it runing for a long time
10/25/2022
- 10:20 PM Bug #57562 (In Progress): multisite replication issue on Quincy
- Small reproducer turned out to not be, but fixing that.
- 04:51 PM Bug #57562: multisite replication issue on Quincy
- Thank you.
- 04:34 PM Bug #57562: multisite replication issue on Quincy
- Please see the following FIFO log snippets. And please let me know if you need more.
The creation of data_log.34.n... - 03:53 PM Bug #57562: multisite replication issue on Quincy
- Can we get a more complete log snippet? All the FIFO logging with the relevant TIDs would make tracing what's going o...
- 03:12 PM Bug #57562: multisite replication issue on Quincy
- thanks, that's very interesting
- 02:59 PM Bug #57562: multisite replication issue on Quincy
- We pretty much narrowed down what the problem is: a race condition has been identified in FIFO::_prepare_new_head(..)...
- 07:18 AM Bug #57919 (New): bucket can not be resharded after cancelling prior reshard process
- Hi,
we run a multisite setup where only the metadata get synced, but not the actual data.
I wanted to reshard a b... - 05:52 AM Bug #56248: crash: rgw::ARN::ARN(rgw_bucket const&)
- Fixed in https://tracker.ceph.com/issues/55765 and https://github.com/ceph/ceph/pull/47194/commits is waiting for rel...
- 05:47 AM Bug #56248: crash: rgw::ARN::ARN(rgw_bucket const&)
- We had a RGW crash on this as well some hours ago....
10/24/2022
- 03:52 PM Bug #19988 (Resolved): RGW: can't stack compression and encryption filters
- 11:37 AM Bug #44660: Multipart re-uploads cause orphan data
- Looking at the code. In `MultipartObjectProcessor::process_first_chunk`, if writing the multipart object first chunk ...
10/23/2022
10/21/2022
- 10:39 AM Bug #57911 (Pending Backport): Segmentation fault when uploading file with bucket policy on Quincy
- RGW crashes when a file is uploaded and a bucket policy has been set up.
The crash has been "reproduced for latest...
10/20/2022
- 02:24 PM Bug #57770 (Triaged): RGW (pacific) misplaces index entries after dynamically resharding bucket
- 02:24 PM Bug #57770 (New): RGW (pacific) misplaces index entries after dynamically resharding bucket
- 02:21 PM Bug #57783: multisite: data sync reports shards behind after source zone fully trims datalog
- related work in https://github.com/ceph/ceph/pull/47682 and https://github.com/ceph/ceph/pull/48397
- 02:20 PM Bug #57804: Enabling sync on bucket not working
- i can only recommend running the command until it succeeds
- 02:18 PM Bug #57853 (Need More Info): multisite sync process block after long time running
- 02:16 PM Bug #57901 (Fix Under Review): s3:ListBuckets response limited to 1000 buckets (by default) since...
- 02:11 PM Bug #57231 (Resolved): Valgrind: jump on unitialized in s3select
- 01:30 PM Bug #57905 (Pending Backport): multisite: terminate called after throwing an instance of 'ceph::b...
- example from rgw/multisite suite: http://qa-proxy.ceph.com/teuthology/cbodley-2022-10-19_23:28:37-rgw-wip-cbodley-tes...
- 05:57 AM Bug #57562: multisite replication issue on Quincy
- We have an example scenario here where one of the objects in a bucket failed to be synced to the secondary.
* Mdlog...
10/19/2022
- 09:28 PM Bug #57901 (Resolved): s3:ListBuckets response limited to 1000 buckets (by default) since Octopus
- Since Octopus, s3:ListBuckets is limited to rgw_list_buckets_max_chunk buckets in its response due to loss of truncat...
- 03:05 PM Bug #16767 (In Progress): RadosGW Multipart Cleanup Failure
- 02:55 PM Bug #16767: RadosGW Multipart Cleanup Failure
- Vicki Good wrote:
> I've encountered this bug in Ceph 14 and 15 and it's a pretty big problem for us for the same re... - 01:20 PM rgw-testing Bug #54104: test_rgw_datacache.py: s3cmd fails with '403 (SignatureDoesNotMatch)' in ubuntu
- ping @Mark, this remains a blocker for enabling ubuntu in the rgw/verify suite. that subsuite contains most of our fu...
- 01:11 PM Bug #57899 (Pending Backport): admin: cannot use tenant with notification topic
- issue was a regression introduced in: 200f71a90c9e77c91452cec128c2c8be0d3d6f1f
topic notification commands should be...
10/18/2022
- 04:13 PM Backport #57889 (Rejected): pacific: amqp: rgw crash when ca location is used for amqp connections
- 04:12 PM Backport #57888 (In Progress): quincy: amqp: rgw crash when ca location is used for amqp connections
- https://github.com/ceph/ceph/pull/54170
- 04:08 PM Bug #57850 (Pending Backport): amqp: rgw crash when ca location is used for amqp connections
- 03:39 PM Bug #57881 (Fix Under Review): LDAP invalid password resource leak fix
- 09:56 AM Bug #57881: LDAP invalid password resource leak fix
- I created a pull request for a possible fix:
https://github.com/ceph/ceph/pull/48509 - 01:02 PM Bug #57877 (Fix Under Review): rgw: some operations may not have a valid bucket object
10/17/2022
- 12:30 PM Bug #57881 (Pending Backport): LDAP invalid password resource leak fix
- I have noticed that in the case a User tries to log in using LDAP with a wrong password, two new LDAP sessions will b...
- 09:19 AM Bug #57877 (Resolved): rgw: some operations may not have a valid bucket object
- Some codepaths may not always have a valid bucket, so add checks to detect this.
10/14/2022
- 09:17 PM Bug #52027: XML responses return different order of XML elements
- Hi
I think this is not fully addressed.
I've added a comment to pull request https://github.com/ceph/ceph/pull/42... - 06:54 PM Backport #57430: quincy: key is used after move in RGWGetObj_ObjStore_S3::override_range_hdr
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/48228
merged - 01:42 PM Bug #44660: Multipart re-uploads cause orphan data
- Writing on behalf of Ulrich Klein <Ulrich.Klein@ulrichklein.de>, he wanted to add some info to this tracker, below is...
- 08:15 AM Bug #57804: Enabling sync on bucket not working
- Hello Casey,
The init command ended after 60 minutes running.
Unfortunately the two errors are returned constan...
10/13/2022
- 02:18 PM Bug #57783 (In Progress): multisite: data sync reports shards behind after source zone fully trim...
- 02:17 PM Bug #57804 (Need More Info): Enabling sync on bucket not working
- did the `bucket sync enable` command finish? i imagine it would take a while with 60000 index shards..
- 02:15 PM Bug #57807 (Duplicate): The cloud sync module does not work starting with Pacific
- 02:01 PM Bug #57724 (Triaged): Keys returned by Admin API during user creation on secondary zone not valid
- 12:44 PM Bug #57770: RGW (pacific) misplaces index entries after dynamically resharding bucket
- Nick Janus wrote:
> J. Eric Ivancich wrote:
> > The theory is that the bucket index shard does not exist at this mo...
10/12/2022
- 05:00 PM Bug #57770: RGW (pacific) misplaces index entries after dynamically resharding bucket
- J. Eric Ivancich wrote:
> The theory is that the bucket index shard does not exist at this moment, as it was deleted... - 04:11 AM Backport #57197 (Resolved): pacific: x-amz-date protocol change breaks aws v4 signature logic: wa...
- 04:11 AM Bug #47527 (Resolved): Ceph returns s3 incompatible xml response for listMultipartUploads
- 04:10 AM Backport #53148 (Rejected): octopus: Ceph returns s3 incompatible xml response for listMultipartU...
- Octopus is EOL
- 04:10 AM Backport #53149 (Resolved): pacific: Ceph returns s3 incompatible xml response for listMultipartU...
- 02:51 AM Bug #57853 (Pending Backport): multisite sync process block after long time running
- 1、deploy RADOSGW multisite
2、put lot of objects
3、keep it runing for a long time
10/11/2022
- 10:23 PM Backport #57649: pacific: rgw: fix bool/int logic error when calling get_obj_head_ioctx
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/48230
merged - 10:22 PM Backport #57429: pacific: key is used after move in RGWGetObj_ObjStore_S3::override_range_hdr
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/48229
merged - 10:20 PM Backport #57753: pacific: Log status of individual object deletions for multi-object delete reque...
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/48348
merged - 10:19 PM Backport #57197: pacific: x-amz-date protocol change breaks aws v4 signature logic: was rfc 2616....
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/48313
merged - 10:18 PM Backport #55918: pacific: Bucket sync policy core dumped
- https://github.com/ceph/ceph/pull/47994 merged
- 10:17 PM Backport #57450: pacific: 'radosgw-admin sync flow create' cmd crashes if flow-type omitted
- https://github.com/ceph/ceph/pull/47994 merged
- 10:15 PM Backport #55245: pacific: rgwlc: ordinary expiration can remove delete-markers at end of current...
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/47231
merged - 10:14 PM Backport #56185: pacific: rgw crash when use swift api
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/47230
merged - 10:14 PM Backport #55135: pacific: multisite: data sync only spawns one bucket sync at a time
- Casey Bodley wrote:
> https://github.com/ceph/ceph/pull/45713
merged - 10:13 PM Backport #54144: pacific: bilog trim: segfault in RGWRadosBILogTrimCR::send_request if bucket sha...
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/44907
merged - 10:11 PM Backport #53149: pacific: Ceph returns s3 incompatible xml response for listMultipartUploads
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/44558
merged - 06:04 PM Bug #57770: RGW (pacific) misplaces index entries after dynamically resharding bucket
- Here is the code that does this:...
- 06:02 PM Bug #57770 (Need More Info): RGW (pacific) misplaces index entries after dynamically resharding b...
- So I looked at the code in 16.2.9 to try to understand how this might happen. The final step in adding an object to t...
- 01:50 PM Bug #57850 (Fix Under Review): amqp: rgw crash when ca location is used for amqp connections
- 01:45 PM Bug #57850 (Pending Backport): amqp: rgw crash when ca location is used for amqp connections
- ca location value is stored as a reference, and the original string may already be destroyed when ca location is used
10/10/2022
- 08:54 PM Bug #57807: The cloud sync module does not work starting with Pacific
- Related issue: https://tracker.ceph.com/issues/55310
- 12:48 PM Bug #57807 (Duplicate): The cloud sync module does not work starting with Pacific
- We have a cluster running Ceph Pacific storing objects with S3 and we want to sync the objects with an external endpo...
- 07:19 PM Bug #57562: multisite replication issue on Quincy
- Are there any suggestions/tips on how we can debug this type of multisite/replication issues?
- 06:47 AM Bug #57804 (Need More Info): Enabling sync on bucket not working
- Hello,
I'm having a problem when trying to enable sync on one of our buckets (multi-site) from master zone.
Her...
10/08/2022
- 07:27 AM Bug #56992: rgw_op.cc:Deleting a non-existent object also generates a delete marker
- Because I do not fully understand the PR process, I need to close the old PR, and the new one in https://github.com/c...
10/07/2022
- 09:01 PM Bug #57562: multisite replication issue on Quincy
- Hi,
Here's some extra data from another test which used increased rgw debugging levels by feeding in the options _... - 02:11 PM Bug #51919 (Duplicate): crash: ceph::common::PerfCounters::inc(int, unsigned long) (in RGWAsyncFe...
- Changed status from Resolved to Duplicate since this issue duplicates https://tracker.ceph.com/issues/49666.
- 07:41 AM cleanup Tasks #57569: implement chown admin rest entrypoint
- This item has been repurposed to: *implement chown admin rest entrypoint*.
After a chat with Daniel Gryniewicz we ag... - 07:33 AM Bug #57784: beast frontend crashes on exception from socket.local_endpoint()
- Hey,
here is a full stack trace from the RGW daemon. I removed bucket/file/user names.
The host is:
Ubuntu 20.04...
10/06/2022
- 05:25 PM Bug #57784 (Fix Under Review): beast frontend crashes on exception from socket.local_endpoint()
- 05:19 PM Bug #57784 (Pending Backport): beast frontend crashes on exception from socket.local_endpoint()
- reported on ceph-users in https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/FSXGU7WVFJTPHW5S5A63IN4AEOV...
- 04:44 PM Bug #57562: multisite replication issue on Quincy
- The difference between this issue and Bug #57783 is that in our case, the buckets/objects are NOT synced.
I tried a... - 03:28 PM Bug #57783 (In Progress): multisite: data sync reports shards behind after source zone fully trim...
- workload tests have been producing cases where data sync shows shards behind, although the source zone has fully trim...
Also available in: Atom