Project

General

Profile

Actions
Copy link

Bug #16408

open

Creating Bucket failed and can't be deleted.

Added by dongping lan almost 8 years ago. Updated over 6 years ago.

Status:
In Progress
Priority:
Normal
Assignee:
Orit Wasserman
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
jewel, kraken
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v0.94.5
ceph-qa-suite:
rgw
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

For some reasons, bucket instance is not created, but bucket meta is OK, after creating bucket. Like this below:

[root@CNSH232046 ~]# radosgw-admin bucket list | grep "im-core-prd-aa-s100-5\""
"im-core-prd-aa-s100-5",
[root@CNSH232046 ~]# radosgw-admin bucket stats --bucket=im-core-prd-aa-s100-5
2016-06-22 17:45:35.088267 2b05ea29b8c0 0 could not get bucket info for bucket=im-core-prd-aa-s100-5
[root@CNSH232046 ~]#

Now, I cant't put object to bucket im-core-prd-aa-s100-5, even I can't delete this bucket. How can I do? Thanks~

Files

bucket.rm.bucket.log.gz (111 KB) bucket.rm.bucket.log.gz Robin Johnson, 11/15/2016 12:04 AM
Actions
Copy link
 #1

Updated by dongping lan almost 8 years ago

More details:
when I recreate this bucket, error logs like below:
2016-06-22 14:07:21.702293 2b0681c47700 10 cache put: name=.rgw+.bucket.meta.im-core-prd-aa-s100-5:szone.917714.226
2016-06-22 14:07:21.702301 2b0681c47700 10 adding .rgw+.bucket.meta.im-core-prd-aa-s100-5:szone.917714.226 to cache LRU end
2016-06-22 14:07:21.702302 2b0681c47700 10 updating xattr: name=user.rgw.acl bl.length()=163
2016-06-22 14:07:21.702312 2b0681c47700 10 distributing notification oid=notify.1 bl.length()=647
2016-06-22 14:07:21.702907 2b064d200700 10 RGWWatcher::handle_notify() notify_id 23746874180895 cookie 47306007202176 notifier 917714 bl.length()=647
2016-06-22 14:07:21.702928 2b064d200700 10 cache put: name=.rgw+.bucket.meta.im-core-prd-aa-s100-5:szone.917714.226
2016-06-22 14:07:21.702932 2b064d200700 10 moving .rgw+.bucket.meta.im-core-prd-aa-s100-5:szone.917714.226 to cache LRU end
2016-06-22 14:07:21.702942 2b064d200700 10 updating xattr: name=user.rgw.acl bl.length()=163
2016-06-22 14:07:21.704320 2b0681c47700 10 removing .rgw+im-core-prd-aa-s100-5 from cache
2016-06-22 14:07:21.704359 2b0681c47700 20 get_obj_state: rctx=0x2b0681c45860 obj=.rgw:im-core-prd-aa-s100-5 state=0x2b065252e690 s->prefetch_data=0
2016-06-22 14:07:21.704370 2b0681c47700 10 cache get: name=.rgw+im-core-prd-aa-s100-5 : miss
2016-06-22 14:07:21.704948 2b0681c47700 10 cache put: name=.rgw+im-core-prd-aa-s100-5
2016-06-22 14:07:21.704959 2b0681c47700 10 adding .rgw+im-core-prd-aa-s100-5 to cache LRU end
2016-06-22 14:07:21.704978 2b0681c47700 20 get_obj_state: s->obj_tag was set empty
2016-06-22 14:07:21.704984 2b0681c47700 10 cache get: name=.rgw+im-core-prd-aa-s100-5 : type miss (requested=17, cached=22)
2016-06-22 14:07:21.704989 2b0681c47700 20 get_obj_state: rctx=0x2b0681c45860 obj=.rgw:im-core-prd-aa-s100-5 state=0x2b065252e690 s->prefetch_data=0
2016-06-22 14:07:21.705015 2b0681c47700 20 rados->read ofs=0 len=524288
2016-06-22 14:07:21.705655 2b0681c47700 20 rados->read r=0 bl.length=165
2016-06-22 14:07:21.705662 2b0681c47700 10 cache put: name=.rgw+im-core-prd-aa-s100-5
2016-06-22 14:07:21.705663 2b0681c47700 10 moving .rgw+im-core-prd-aa-s100-5 to cache LRU end
2016-06-22 14:07:21.705669 2b0681c47700 20 rgw_get_bucket_info: bucket instance: im-core-prd-aa-s100-5({i=.rgw.buckets.index,e=.rgw.buckets.extra}.rgw.buckets[mzone.1460317.2611])
2016-06-22 14:07:21.705672 2b0681c47700 20 reading from .rgw:.bucket.meta.im-core-prd-aa-s100-5:mzone.1460317.2611
2016-06-22 14:07:21.705677 2b0681c47700 20 get_obj_state: rctx=0x2b0681c45860 obj=.rgw:.bucket.meta.im-core-prd-aa-s100-5:mzone.1460317.2611 state=0x2b065252e090 s->prefetch_data=0
2016-06-22 14:07:21.705681 2b0681c47700 10 cache get: name=.rgw+.bucket.meta.im-core-prd-aa-s100-5:mzone.1460317.2611 : type miss (requested=22, cached=0)
2016-06-22 14:07:21.706237 2b0681c47700 10 cache put: name=.rgw+.bucket.meta.im-core-prd-aa-s100-5:mzone.1460317.2611
2016-06-22 14:07:21.706241 2b0681c47700 10 moving .rgw+.bucket.meta.im-core-prd-aa-s100-5:mzone.1460317.2611 to cache LRU end
2016-06-22 14:07:21.706248 2b0681c47700 0 ERROR: could not create bucket, continuously raced with bucket creation and removal
2016-06-22 14:07:21.706249 2b0681c47700 20 rgw_create_bucket returned ret=-2 bucket=im-core-prd-aa-s100-5({i=.rgw.buckets.index,e=.rgw.buckets.extra}.rgw.buckets[szone.917714.226])
2016-06-22 14:07:21.706268 2b0681c47700 2 req 876:2.556914:s3:PUT /im-core-prd-aa-s100-5:create_bucket:http status=404
2016-06-22 14:07:21.706273 2b0681c47700 1 ====== req done req=0x2b065cb0bd80 http_status=404 ======

Actions
Copy link
 #2

Updated by Yao Ning almost 8 years ago

Actions
Copy link
 #3

Updated by dongping lan almost 8 years ago

Only bucket meta is created would lead to this issue. We can rm bucket meta in .rgw pool, then it works. Thanks anyway.

Actions
Copy link
 #4

Updated by Yehuda Sadeh almost 8 years ago

  • Assignee set to Orit Wasserman
Actions
Copy link
 #5

Updated by Yehuda Sadeh over 7 years ago

Orit, ping?

Actions
Copy link
 #6

Updated by Orit Wasserman over 7 years ago

  • Status changed from New to In Progress
Actions
Copy link
 #7

Updated by Robin Johnson over 7 years ago

@owasserm:
Do you have progress/insights on this? I'm seeing it on a lot of Dreamhost buckets, but it looks like it's been slowly building up over time on the stale buckets (esp those generated by s3-tests and not cleaning up properly).

Actions
Copy link
 #8

Updated by Orit Wasserman over 7 years ago

Can you add a log at least for failed delete?

Actions
Copy link
 #9

Updated by Robin Johnson over 7 years ago

Attached is the log from:

b=bucket
sudo radosgw-admin --bucket $b bucket rm --debug-rgw=20 --debug-ms 20 2>&1 | tee bucket.rm.bucket.log
sed -r 's/${IPV6_NET_PREFIX}/2001:db8/g' bucket.rm.bucket.log -i

Yes, the bucket is literally named bucket.

Actions
Copy link
 #10

Updated by Yehuda Sadeh over 7 years ago

This seems broken:

2016-11-14 23:43:33.290542 7fd5fc6d8900  1 -- [2001:db8::5752]:0/2991243590 --> [2001:db8::8851]:6820/3585 -- osd_op(client.291883026.0:133 13.c9dbc1cb .bucket.meta. [call version.read,getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e1020137) v7 -- ?+0 0x560d01781b90 con 0x560d0177faa0
2016-11-14 23:43:33.290572 7fd5fc6d8900 20 -- [2001:db8::5752]:0/2991243590 submit_message osd_op(client.291883026.0:133 13.c9dbc1cb .bucket.meta. [call version.read,getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e1020137) v7 remote, [2001:db8::8851]:6820/3585, have pipe.
2016-11-14 23:43:33.291473 7fd4c288b700 20 -- [2001:db8::5752]:0/2991243590 >> [2001:db8::8851]:6820/3585 pipe(0x560d017808d0 sd=83 :33274 s=2 pgs=28188 cs=1 l=1 c=0x560d0177faa0).writer encoding 1 features 576460752032874495 0x560d01781b90 osd_op(client.291883026.0:133 13.c9dbc1cb .bucket.meta. [call version.read,getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e1020137) v7
2016-11-14 23:43:33.293727 7fd4c278a700 10 -- [2001:db8::5752]:0/2991243590 >> [2001:db8::8851]:6820/3585 pipe(0x560d017808d0 sd=83 :33274 s=2 pgs=28188 cs=1 l=1 c=0x560d0177faa0).reader got message 1 0x7fd5d0000ff0 osd_op_reply(133 .bucket.meta. [call,getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7
2016-11-14 23:43:33.293755 7fd4c278a700  1 -- [2001:db8::5752]:0/2991243590 <== osd.946 [2001:db8::8851]:6820/3585 1 ==== osd_op_reply(133 .bucket.meta. [call,getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 217+0+0 (200346745 0 0) 0x7fd5d0000ff0 con 0x560d0177faa0

Specifically, it looks like a broken bucket instance metadata oid (.bucket.meta.). What does the following return?

$ radosgw-admin metadata get bucket:bucket
Actions
Copy link
 #11

Updated by Robin Johnson over 7 years ago 

rjohnson@peon5752:~$ sudo radosgw-admin metadata get bucket:bucket
{
    "key": "bucket:bucket",
    "ver": {
        "tag": "tWNQSiM9zfnGpgp7vzkIUD6w",
        "ver": 5
    },
    "mtime": "2012-09-12 23:43:46.861813Z",
    "data": {
        "bucket": {
            "name": "",
            "pool": "",
            "data_extra_pool": "",
            "index_pool": "",
            "marker": "",
            "bucket_id": "",
            "tenant": "" 
        },
        "owner": "",
        "creation_time": "0.000000",
        "linked": "false",
        "has_bucket_info": "false" 
    }
}
Actions
Copy link
 #12

Updated by Yehuda Sadeh over 7 years ago

Yeah, bucket entrypoint is broken. Does it happen also with newer buckets? This one seems a bit old. We can try identifying these cases (e.g., data.bucket.name is empty). You should be able to remove this one manually: `radosgw-admin metadata rm bucket:bucket`

Actions
Copy link
 #13

Updated by Robin Johnson over 7 years ago

I haven't seen it with newer buckets, just old buckets.

I do confirm in this case, that your fix worked: radosgw-admin metadata rm bucket:bucket

Actions
Copy link
 #14

Updated by Orit Wasserman over 7 years ago

  • Status changed from In Progress to Fix Under Review
  • Backport set to jewel
Actions
Copy link
 #15

Updated by Orit Wasserman over 7 years ago

  • Backport changed from jewel to jewel, kraken
Actions
Copy link
 #16

Updated by Nathan Cutler over 7 years ago

kraken backporting will not start for some time - this patch will get into kraken via manual merge

Actions
Copy link
 #17

Updated by Orit Wasserman over 7 years ago

  • Status changed from Fix Under Review to In Progress
Actions
Copy link
 #18

Updated by Yehuda Sadeh over 7 years ago

@Orit Wasserman any progress here?

Actions
Copy link
 #19

Updated by Matt Benjamin over 6 years ago

  • Priority changed from High to Normal
Actions
Copy link

Also available in: Atom PDF