Project

General

Profile

Bug #58053

bucket is list in s3cmd but can not be queried after deleting

Added by Max Gao 2 months ago. Updated about 2 months ago.

Status:
Need More Info
Priority:
Normal
Assignee:
Target version:
-
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
ceph-ansible
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

ceph version 16.2.10

There will be a race between s3.DeleteBucket and s3.DeleteBucketPolicy. When the race happens, the deleted bucket remains in the "s3cmd ls”output, but its not listed in “radosgw-admin bucket list”output. At that state, doing get/delete of that bucket via s3 will return “noSuchBucket” error.. In order to delete the bucket, I have to manually create and delele that bucket with s3cmd.

History

#1 Updated by Max Gao 2 months ago

The bug can be reproduced with the following script:

#2 Updated by Max Gao 2 months ago

Prepare a policy.json file that contains the bucket policy for testing.


#!/bin/sh

alias aws='aws --endpoint-url http://10.200.0.2:8989'

# create the bucket bkt1
aws s3api create-bucket --bucket bkt1

# set the bucket policy for bkt1
aws s3api put-bucket-policy --bucket bkt1 --policy file://policy.json

# run delete-bucket and delete-bucket-policy in parallel
aws s3api delete-bucket --bucket bkt1 &
aws s3api delete-bucket-policy --bucket bkt1 &

# wait until above two commands are completed.
wait

# Bucket bkt1 still shows up in the bucket list
aws s3api list-buckets --query "Buckets[].Name" 

# But, the bucket can't be fetched via HEAD request
aws s3api head-bucket --bucket bkt1

# As expected, doesn't show up in radosgw-admin listing:
radosgw-admin bucket list

After running the above script, the output of "s3cmd ls && radosgw-admin bucket list" is as following:

2022-11-07 17:51 s3://bhb3d-inventory-export
2022-11-22 06:17 s3://bkt1
2022-11-18 07:54 s3://qg18v-bkt2
2022-11-18 19:32 s3://qg18v-bkt6
[
"bhb3d-inventory-export",
"qg18v-bkt6",
"qg18v-bkt2"
]

The outputs from two clients are not matched.

#3 Updated by Max Gao 2 months ago

ceph-qa-suite should be RGW

#4 Updated by Casey Bodley about 2 months ago

  • Assignee set to Casey Bodley

#5 Updated by Casey Bodley about 2 months ago

  • Status changed from New to Need More Info

hey Max, thanks for the reproducer script. i ran this against main for ~50 buckets and wasn't able to reproduce the issue - 's3api list-buckets' and 'radosagw-admin bucket list' both return empty listings every time

what ceph version are you using here?

Also available in: Atom PDF