Bug #16408: Creating Bucket failed and can't be deleted. - rgw - Ceph

Custom queries

Bug queue
Bug triage
Crash queue
Crash triage
Feedback
My issues
Need Review
outstanding non-trivial backports
Pending backports
Product Backlog Scrub
rgw-easy-first-bug
rgw-multisite-backlog

Actions

Copy link

Bug #16408

open

Creating Bucket failed and can't be deleted.

Added by dongping lan almost 8 years ago. Updated over 6 years ago.

Status:

In Progress

Priority:

Normal

Assignee:

Orit Wasserman

Target version:

% Done:

Source:

other

Tags:

Backport:

jewel, kraken

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

Ceph - v0.94.5

ceph-qa-suite:

rgw

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

For some reasons, bucket instance is not created, but bucket meta is OK, after creating bucket. Like this below:

[root@CNSH232046 ~]# radosgw-admin bucket list | grep "im-core-prd-aa-s100-5\""
"im-core-prd-aa-s100-5",
[root@CNSH232046 ~]# radosgw-admin bucket stats --bucket=im-core-prd-aa-s100-5
2016-06-22 17:45:35.088267 2b05ea29b8c0 0 could not get bucket info for bucket=im-core-prd-aa-s100-5
[root@CNSH232046 ~]#

Now, I cant't put object to bucket im-core-prd-aa-s100-5, even I can't delete this bucket. How can I do? Thanks~

Files

bucket.rm.bucket.log.gz (111 KB) bucket.rm.bucket.log.gz

Robin Johnson, 11/15/2016 12:04 AM

History
Notes
Property changes

Actions

Copy link

Updated by dongping lan almost 8 years ago

More details:
when I recreate this bucket, error logs like below:
2016-06-22 14:07:21.702293 2b0681c47700 10 cache put: name=.rgw+.bucket.meta.im-core-prd-aa-s100-5:szone.917714.226
2016-06-22 14:07:21.702301 2b0681c47700 10 adding .rgw+.bucket.meta.im-core-prd-aa-s100-5:szone.917714.226 to cache LRU end
2016-06-22 14:07:21.702302 2b0681c47700 10 updating xattr: name=user.rgw.acl bl.length()=163
2016-06-22 14:07:21.702312 2b0681c47700 10 distributing notification oid=notify.1 bl.length()=647
2016-06-22 14:07:21.702907 2b064d200700 10 RGWWatcher::handle_notify() notify_id 23746874180895 cookie 47306007202176 notifier 917714 bl.length()=647
2016-06-22 14:07:21.702928 2b064d200700 10 cache put: name=.rgw+.bucket.meta.im-core-prd-aa-s100-5:szone.917714.226
2016-06-22 14:07:21.702932 2b064d200700 10 moving .rgw+.bucket.meta.im-core-prd-aa-s100-5:szone.917714.226 to cache LRU end
2016-06-22 14:07:21.702942 2b064d200700 10 updating xattr: name=user.rgw.acl bl.length()=163
2016-06-22 14:07:21.704320 2b0681c47700 10 removing .rgw+im-core-prd-aa-s100-5 from cache
2016-06-22 14:07:21.704359 2b0681c47700 20 get_obj_state: rctx=0x2b0681c45860 obj=.rgw:im-core-prd-aa-s100-5 state=0x2b065252e690 s->prefetch_data=0
2016-06-22 14:07:21.704370 2b0681c47700 10 cache get: name=.rgw+im-core-prd-aa-s100-5 : miss
2016-06-22 14:07:21.704948 2b0681c47700 10 cache put: name=.rgw+im-core-prd-aa-s100-5
2016-06-22 14:07:21.704959 2b0681c47700 10 adding .rgw+im-core-prd-aa-s100-5 to cache LRU end
2016-06-22 14:07:21.704978 2b0681c47700 20 get_obj_state: s->obj_tag was set empty
2016-06-22 14:07:21.704984 2b0681c47700 10 cache get: name=.rgw+im-core-prd-aa-s100-5 : type miss (requested=17, cached=22)
2016-06-22 14:07:21.704989 2b0681c47700 20 get_obj_state: rctx=0x2b0681c45860 obj=.rgw:im-core-prd-aa-s100-5 state=0x2b065252e690 s->prefetch_data=0
2016-06-22 14:07:21.705015 2b0681c47700 20 rados->read ofs=0 len=524288
2016-06-22 14:07:21.705655 2b0681c47700 20 rados->read r=0 bl.length=165
2016-06-22 14:07:21.705662 2b0681c47700 10 cache put: name=.rgw+im-core-prd-aa-s100-5
2016-06-22 14:07:21.705663 2b0681c47700 10 moving .rgw+im-core-prd-aa-s100-5 to cache LRU end
2016-06-22 14:07:21.705669 2b0681c47700 20 rgw_get_bucket_info: bucket instance: im-core-prd-aa-s100-5({i=.rgw.buckets.index,e=.rgw.buckets.extra}.rgw.buckets[mzone.1460317.2611]) 2016-06-22 14:07:21.705672 2b0681c47700 20 reading from .rgw:.bucket.meta.im-core-prd-aa-s100-5:mzone.1460317.2611 2016-06-22 14:07:21.705677 2b0681c47700 20 get_obj_state: rctx=0x2b0681c45860 obj=.rgw:.bucket.meta.im-core-prd-aa-s100-5:mzone.1460317.2611 state=0x2b065252e090 s->prefetch_data=0 2016-06-22 14:07:21.705681 2b0681c47700 10 cache get: name=.rgw+.bucket.meta.im-core-prd-aa-s100-5:mzone.1460317.2611 : type miss (requested=22, cached=0) 2016-06-22 14:07:21.706237 2b0681c47700 10 cache put: name=.rgw+.bucket.meta.im-core-prd-aa-s100-5:mzone.1460317.2611 2016-06-22 14:07:21.706241 2b0681c47700 10 moving .rgw+.bucket.meta.im-core-prd-aa-s100-5:mzone.1460317.2611 to cache LRU end 2016-06-22 14:07:21.706248 2b0681c47700 0 ERROR: could not create bucket, continuously raced with bucket creation and removal 2016-06-22 14:07:21.706249 2b0681c47700 20 rgw_create_bucket returned ret=-2 bucket=im-core-prd-aa-s100-5({i=.rgw.buckets.index,e=.rgw.buckets.extra}.rgw.buckets[szone.917714.226])
2016-06-22 14:07:21.706268 2b0681c47700 2 req 876:2.556914:s3:PUT /im-core-prd-aa-s100-5:create_bucket:http status=404
2016-06-22 14:07:21.706273 2b0681c47700 1 ====== req done req=0x2b065cb0bd80 http_status=404 ======

Actions

Copy link

Updated by Yao Ning almost 8 years ago

~~Same reason in issue: http://tracker.ceph.com/issues/15318?~~

Actions

Copy link

Updated by dongping lan almost 8 years ago

Only bucket meta is created would lead to this issue. We can rm bucket meta in .rgw pool, then it works. Thanks anyway.

Actions

Copy link

Updated by Yehuda Sadeh almost 8 years ago

Assignee set to Orit Wasserman

Actions

Copy link

Updated by Yehuda Sadeh over 7 years ago

Orit, ping?

Actions

Copy link

Updated by Orit Wasserman over 7 years ago

Status changed from New to In Progress

Actions

Copy link

Updated by Robin Johnson over 7 years ago

@owasserm:
Do you have progress/insights on this? I'm seeing it on a lot of Dreamhost buckets, but it looks like it's been slowly building up over time on the stale buckets (esp those generated by s3-tests and not cleaning up properly).

Actions

Copy link

Updated by Orit Wasserman over 7 years ago

Can you add a log at least for failed delete?

Actions

Copy link

Updated by Robin Johnson over 7 years ago

File bucket.rm.bucket.log.gz bucket.rm.bucket.log.gz added

Attached is the log from:
b=bucket sudo radosgw-admin --bucket $b bucket rm --debug-rgw=20 --debug-ms 20 2>&1 | tee bucket.rm.bucket.log sed -r 's/${IPV6_NET_PREFIX}/2001:db8/g' bucket.rm.bucket.log -i

Yes, the bucket is literally named bucket.

Actions

Copy link

#10

Updated by Yehuda Sadeh over 7 years ago

This seems broken:

2016-11-14 23:43:33.290542 7fd5fc6d8900  1 -- [2001:db8::5752]:0/2991243590 --> [2001:db8::8851]:6820/3585 -- osd_op(client.291883026.0:133 13.c9dbc1cb .bucket.meta. [call version.read,getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e1020137) v7 -- ?+0 0x560d01781b90 con 0x560d0177faa0
2016-11-14 23:43:33.290572 7fd5fc6d8900 20 -- [2001:db8::5752]:0/2991243590 submit_message osd_op(client.291883026.0:133 13.c9dbc1cb .bucket.meta. [call version.read,getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e1020137) v7 remote, [2001:db8::8851]:6820/3585, have pipe.
2016-11-14 23:43:33.291473 7fd4c288b700 20 -- [2001:db8::5752]:0/2991243590 >> [2001:db8::8851]:6820/3585 pipe(0x560d017808d0 sd=83 :33274 s=2 pgs=28188 cs=1 l=1 c=0x560d0177faa0).writer encoding 1 features 576460752032874495 0x560d01781b90 osd_op(client.291883026.0:133 13.c9dbc1cb .bucket.meta. [call version.read,getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e1020137) v7
2016-11-14 23:43:33.293727 7fd4c278a700 10 -- [2001:db8::5752]:0/2991243590 >> [2001:db8::8851]:6820/3585 pipe(0x560d017808d0 sd=83 :33274 s=2 pgs=28188 cs=1 l=1 c=0x560d0177faa0).reader got message 1 0x7fd5d0000ff0 osd_op_reply(133 .bucket.meta. [call,getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7
2016-11-14 23:43:33.293755 7fd4c278a700  1 -- [2001:db8::5752]:0/2991243590 <== osd.946 [2001:db8::8851]:6820/3585 1 ==== osd_op_reply(133 .bucket.meta. [call,getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 217+0+0 (200346745 0 0) 0x7fd5d0000ff0 con 0x560d0177faa0

Specifically, it looks like a broken bucket instance metadata oid (.bucket.meta.). What does the following return?

$ radosgw-admin metadata get bucket:bucket

Actions

Copy link

#11

Updated by Robin Johnson over 7 years ago

rjohnson@peon5752:~$ sudo radosgw-admin metadata get bucket:bucket
{
    "key": "bucket:bucket",
    "ver": {
        "tag": "tWNQSiM9zfnGpgp7vzkIUD6w",
        "ver": 5
    },
    "mtime": "2012-09-12 23:43:46.861813Z",
    "data": {
        "bucket": {
            "name": "",
            "pool": "",
            "data_extra_pool": "",
            "index_pool": "",
            "marker": "",
            "bucket_id": "",
            "tenant": "" 
        },
        "owner": "",
        "creation_time": "0.000000",
        "linked": "false",
        "has_bucket_info": "false" 
    }
}

Actions

Copy link

#12

Updated by Yehuda Sadeh over 7 years ago

Yeah, bucket entrypoint is broken. Does it happen also with newer buckets? This one seems a bit old. We can try identifying these cases (e.g., data.bucket.name is empty). You should be able to remove this one manually: `radosgw-admin metadata rm bucket:bucket`

Actions

Copy link

#13