Ceph » rgw

Hello, JS Landry, I'd really like to help you resolve this for your customers. I'd really like to be able to reproduce this so I can work on it, and what would really help is a `radosgw-admin bi list --bucket=XYZ --max-entries=9999999`. We need max-entries to be greater than the number of actual entries in order to capture every one.

It's not lost on me that this is a BIG ask. I realize there's a possibility (probability?) that the object names may leak proprietary information. But I want to ask as it would really help speed up the process.

Additionally you provided some rgw logs. Would it be possible to get them at level 20? Please let me know when you see this, so I might know what to expect and when.

Thanks!!

Hi, sure, I will provide you that in a few hours tonight. Thanks!

Hi, I have the files you asked for and I'm waiting for the owner agreement. It shouldn't take long / next week.
Thanks!

JS Landry wrote:

Hi, I have the files you asked for and I'm waiting for the owner agreement. It shouldn't take long / next week.
Thanks!

Thanks!

JS Landry wrote:

Hi Eric, here's the files. I did anonymize some data about the bucket name, owner, server name, ip address, otherwise everything is "as-is".

The bi-list file is about 2GB, ~76MB bzipped, I can't attach it here, the attachment size limit is 1MB. Where should I send it?

Thanks!

Thank you!! I'll take a look.

Eric, please note that I will be out of the office starting Monday, August 9, through August 23. Tell me if you need the bi-list file and where I can send it.
Thanks!

JS Landry,

Sorry my previous response did't catch that you weren't able to upload the bi list given the limitations here on the tracker.

Do you have an online storage account (e.g., Google Drive) that you can upload it to and give me read permissions?

Eric

JS Landry wrote:

Hi Eric, here's the files. I did anonymize some data about the bucket name, owner, server name, ip address, otherwise everything is "as-is".

The bi-list file is about 2GB, ~76MB bzipped, I can't attach it here, the attachment size limit is 1MB. Where should I send it?

Thanks!

I examined at the logs. I was looking for log lines that contain "cls_bucket_list_ordered" and "list_objects_ordered", but I couldn't find any. So somehow those logs don't capture the listing operation. They would be helpful, although the bi list would be the most helpful as I can then recreate such a bucket and perform various tests.

Thanks!!

JS Landry wrote:

Eric, please note that I will be out of the office starting Monday, August 9, through August 23. Tell me if you need the bi-list file and where I can send it.
Thanks!

If you don't have another option, a colleague recommended https://wormhole.app/ as a way to get me the data. It will expire after 24 hours.

JS Landry wrote:

Hi, you can download the bi-list file here: https://f001.backblazeb2.com/file/9455b1e05809/bi.list.10shards.txt.anon.bz2

About the "list_objects_ordered" and "cls_bucket_list_ordered", strange enough I have about 8000 events in the central syslog server for July 30 between 9h48 and 9h50,
a day after the logfiles I already sent you. Anyhow, here's an export from the syslog server of all radosgw* events for this short period.

Tell me if you need anything else.

I grabbed the two files. Thank you! I'll keep you posted.

Thanks!

I wanted to provide an update. I've been trying to recreate the bucket (via the index) in order to reproduce. Ran into some hiccups on the machine due to limited disk space. It's currently running, although with 1.5 million entries to add it might take a day or two.

Excellent! Thanks for the followup.

JS Landry wrote:

Excellent! Thanks for the followup.

I have a bucket loaded with the ids from your bi list -- bi.list.10shards.txt.anon. And I wasn't able to produce listings and then I realized that there's no mention of "carto/DATA/site/Works/vq" in that bi list. What am I missing?

Do you have example problematic bucket listings based on the bucket index entries listed in bi.list.10shards.txt.anon?

J. Eric Ivancich wrote:

I have a bucket loaded with the ids from your bi list -- bi.list.10shards.txt.anon. And I wasn't able to produce listings and then I realized that there's no mention of "carto/DATA/site/Works/vq" in that bi list. What am I missing?

Hi, sorry about that, it's related to the anonymizing task. The path is "carto/DATA/ulaval/Rasters/vq/"

JS Landry wrote:

J. Eric Ivancich wrote:

I have a bucket loaded with the ids from your bi list -- bi.list.10shards.txt.anon. And I wasn't able to produce listings and then I realized that there's no mention of "carto/DATA/site/Works/vq" in that bi list. What am I missing?

Hi, sorry about that, it's related to the anonymizing task. The path is "carto/DATA/ulaval/Rasters/vq/"

Thanks for the info. So when I tested it with a recent master with 1, 10, 11, 13, and 17 shards, the listings were consistent. That could mean one of two things a) it's been fixed between 15.2.13 and master or b) the anonymizing step altered something crucial to exposing the bug. I'm now loading up a v15.2.13 test cluster with the 1,780,433 bucket index entries and I'll re-test. I will keep you posted.

As I try to load up the bucket index, my osd keeps crashing. For example:

-1256> 2021-09-21T20:37:53.900-0400 7f8c51a82640 10 osd.0 18 dequeue_op 0x5621d4fea340 finish
 -1255> 2021-09-21T20:37:53.900-0400 7f8c51a82640 20 osd.0 op_wq(4) _process empty q, waiting
 -1254> 2021-09-21T20:37:53.900-0400 7f8c55a8a640 20 osd.0 op_wq(4) _process empty q, waiting
 -1253> 2021-09-21T20:37:53.902-0400 7f8c712c1640 -1 ** Caught signal (Segmentation fault) *
 in thread 7f8c712c1640 thread_name:msgr-worker-2

I'm going to switch from v15.2.13 to v15.2.14 and see if that resolves anything. I'll also look to see if that's a known bug.

The osd crash I'm experiencing is reported here: https://tracker.ceph.com/issues/51527

It also happens on v15.2.14. I'm going to try to add automation to restart my osd when it crashes in order to load up the bucket index.

Hi, I will try to find the time next week to add & test a v15.2.14 radosgw with my v15.2.13 cluster.
Is it possible, or a bad idea, to run/test a Pacific radosgw on a Octopus cluster?
Thanks!

JS Landry wrote:

Hi, I will try to find the time next week to add & test a v15.2.14 radosgw with my v15.2.13 cluster.
Is it possible, or a bad idea, to run/test a Pacific radosgw on a Octopus cluster?
Thanks!

It's not considered supported. But as long as it's only used for reading, it should not create an issue. Once you test it with your bucket listing, it would probably be best to take it down to prevent it from causing issues.

[I'm at around 700K of 1.8M entries loaded. I hope to be able to test it Monday.]

So I was finally able to reproduce on v15.2.13. However it does not reproduce on v15.2.14.

So I'd be curious, JS Landry, to know if you experience the bug with v15.2.14?

• https://github.com/ceph/ceph/pull/41141
• https://github.com/ceph/ceph/pull/40975

• https://github.com/ceph/ceph/pull/41413
• https://github.com/ceph/ceph/pull/41775

Let e know what you find with v15.2.14 and we'll go from there. Thanks!

J. Eric Ivancich wrote:

So I was finally able to reproduce on v15.2.13. However it does not reproduce on v15.2.14.
So I'd be curious, JS Landry, to know if you experience the bug with v15.2.14?

Hi! That's great news. I'm working on it today and I let you know.
Thanks!

Indeed, 15.2.14 fix it!
I should have test it earlier. I hit the bug before the release of 15.2.14 and I didn't check all the elements of the latest changelog, sorry about that.
Anyhow, thank you, everything is working as expected now.
We can close this ticket as fixed.