Bug #16408
openCreating Bucket failed and can't be deleted.
Added by dongping lan almost 8 years ago. Updated over 6 years ago.
0%
Description
For some reasons, bucket instance is not created, but bucket meta is OK, after creating bucket. Like this below:
[root@CNSH232046 ~]# radosgw-admin bucket list | grep "im-core-prd-aa-s100-5\""
"im-core-prd-aa-s100-5",
[root@CNSH232046 ~]# radosgw-admin bucket stats --bucket=im-core-prd-aa-s100-5
2016-06-22 17:45:35.088267 2b05ea29b8c0 0 could not get bucket info for bucket=im-core-prd-aa-s100-5
[root@CNSH232046 ~]#
Now, I cant't put object to bucket im-core-prd-aa-s100-5, even I can't delete this bucket. How can I do? Thanks~
Files
bucket.rm.bucket.log.gz (111 KB) bucket.rm.bucket.log.gz | Robin Johnson, 11/15/2016 12:04 AM |
Updated by dongping lan almost 8 years ago
More details:
when I recreate this bucket, error logs like below:
2016-06-22 14:07:21.702293 2b0681c47700 10 cache put: name=.rgw+.bucket.meta.im-core-prd-aa-s100-5:szone.917714.226
2016-06-22 14:07:21.702301 2b0681c47700 10 adding .rgw+.bucket.meta.im-core-prd-aa-s100-5:szone.917714.226 to cache LRU end
2016-06-22 14:07:21.702302 2b0681c47700 10 updating xattr: name=user.rgw.acl bl.length()=163
2016-06-22 14:07:21.702312 2b0681c47700 10 distributing notification oid=notify.1 bl.length()=647
2016-06-22 14:07:21.702907 2b064d200700 10 RGWWatcher::handle_notify() notify_id 23746874180895 cookie 47306007202176 notifier 917714 bl.length()=647
2016-06-22 14:07:21.702928 2b064d200700 10 cache put: name=.rgw+.bucket.meta.im-core-prd-aa-s100-5:szone.917714.226
2016-06-22 14:07:21.702932 2b064d200700 10 moving .rgw+.bucket.meta.im-core-prd-aa-s100-5:szone.917714.226 to cache LRU end
2016-06-22 14:07:21.702942 2b064d200700 10 updating xattr: name=user.rgw.acl bl.length()=163
2016-06-22 14:07:21.704320 2b0681c47700 10 removing .rgw+im-core-prd-aa-s100-5 from cache
2016-06-22 14:07:21.704359 2b0681c47700 20 get_obj_state: rctx=0x2b0681c45860 obj=.rgw:im-core-prd-aa-s100-5 state=0x2b065252e690 s->prefetch_data=0
2016-06-22 14:07:21.704370 2b0681c47700 10 cache get: name=.rgw+im-core-prd-aa-s100-5 : miss
2016-06-22 14:07:21.704948 2b0681c47700 10 cache put: name=.rgw+im-core-prd-aa-s100-5
2016-06-22 14:07:21.704959 2b0681c47700 10 adding .rgw+im-core-prd-aa-s100-5 to cache LRU end
2016-06-22 14:07:21.704978 2b0681c47700 20 get_obj_state: s->obj_tag was set empty
2016-06-22 14:07:21.704984 2b0681c47700 10 cache get: name=.rgw+im-core-prd-aa-s100-5 : type miss (requested=17, cached=22)
2016-06-22 14:07:21.704989 2b0681c47700 20 get_obj_state: rctx=0x2b0681c45860 obj=.rgw:im-core-prd-aa-s100-5 state=0x2b065252e690 s->prefetch_data=0
2016-06-22 14:07:21.705015 2b0681c47700 20 rados->read ofs=0 len=524288
2016-06-22 14:07:21.705655 2b0681c47700 20 rados->read r=0 bl.length=165
2016-06-22 14:07:21.705662 2b0681c47700 10 cache put: name=.rgw+im-core-prd-aa-s100-5
2016-06-22 14:07:21.705663 2b0681c47700 10 moving .rgw+im-core-prd-aa-s100-5 to cache LRU end
2016-06-22 14:07:21.705669 2b0681c47700 20 rgw_get_bucket_info: bucket instance: im-core-prd-aa-s100-5({i=.rgw.buckets.index,e=.rgw.buckets.extra}.rgw.buckets[mzone.1460317.2611])
{i=.rgw.buckets.index,e=.rgw.buckets.extra}.rgw.buckets[szone.917714.226])
2016-06-22 14:07:21.705672 2b0681c47700 20 reading from .rgw:.bucket.meta.im-core-prd-aa-s100-5:mzone.1460317.2611
2016-06-22 14:07:21.705677 2b0681c47700 20 get_obj_state: rctx=0x2b0681c45860 obj=.rgw:.bucket.meta.im-core-prd-aa-s100-5:mzone.1460317.2611 state=0x2b065252e090 s->prefetch_data=0
2016-06-22 14:07:21.705681 2b0681c47700 10 cache get: name=.rgw+.bucket.meta.im-core-prd-aa-s100-5:mzone.1460317.2611 : type miss (requested=22, cached=0)
2016-06-22 14:07:21.706237 2b0681c47700 10 cache put: name=.rgw+.bucket.meta.im-core-prd-aa-s100-5:mzone.1460317.2611
2016-06-22 14:07:21.706241 2b0681c47700 10 moving .rgw+.bucket.meta.im-core-prd-aa-s100-5:mzone.1460317.2611 to cache LRU end
2016-06-22 14:07:21.706248 2b0681c47700 0 ERROR: could not create bucket, continuously raced with bucket creation and removal
2016-06-22 14:07:21.706249 2b0681c47700 20 rgw_create_bucket returned ret=-2 bucket=im-core-prd-aa-s100-5(
2016-06-22 14:07:21.706268 2b0681c47700 2 req 876:2.556914:s3:PUT /im-core-prd-aa-s100-5:create_bucket:http status=404
2016-06-22 14:07:21.706273 2b0681c47700 1 ====== req done req=0x2b065cb0bd80 http_status=404 ======
Updated by Yao Ning almost 8 years ago
Same reason in issue: http://tracker.ceph.com/issues/15318?
Updated by dongping lan almost 8 years ago
Only bucket meta is created would lead to this issue. We can rm bucket meta in .rgw pool, then it works. Thanks anyway.
Updated by Orit Wasserman over 7 years ago
- Status changed from New to In Progress
Updated by Robin Johnson over 7 years ago
@owasserm:
Do you have progress/insights on this? I'm seeing it on a lot of Dreamhost buckets, but it looks like it's been slowly building up over time on the stale buckets (esp those generated by s3-tests and not cleaning up properly).
Updated by Orit Wasserman over 7 years ago
Can you add a log at least for failed delete?
Updated by Robin Johnson over 7 years ago
- File bucket.rm.bucket.log.gz bucket.rm.bucket.log.gz added
Attached is the log from:
b=bucket
sudo radosgw-admin --bucket $b bucket rm --debug-rgw=20 --debug-ms 20 2>&1 | tee bucket.rm.bucket.log
sed -r 's/${IPV6_NET_PREFIX}/2001:db8/g' bucket.rm.bucket.log -i
Yes, the bucket is literally named bucket.
Updated by Yehuda Sadeh over 7 years ago
This seems broken:
2016-11-14 23:43:33.290542 7fd5fc6d8900 1 -- [2001:db8::5752]:0/2991243590 --> [2001:db8::8851]:6820/3585 -- osd_op(client.291883026.0:133 13.c9dbc1cb .bucket.meta. [call version.read,getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e1020137) v7 -- ?+0 0x560d01781b90 con 0x560d0177faa0 2016-11-14 23:43:33.290572 7fd5fc6d8900 20 -- [2001:db8::5752]:0/2991243590 submit_message osd_op(client.291883026.0:133 13.c9dbc1cb .bucket.meta. [call version.read,getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e1020137) v7 remote, [2001:db8::8851]:6820/3585, have pipe. 2016-11-14 23:43:33.291473 7fd4c288b700 20 -- [2001:db8::5752]:0/2991243590 >> [2001:db8::8851]:6820/3585 pipe(0x560d017808d0 sd=83 :33274 s=2 pgs=28188 cs=1 l=1 c=0x560d0177faa0).writer encoding 1 features 576460752032874495 0x560d01781b90 osd_op(client.291883026.0:133 13.c9dbc1cb .bucket.meta. [call version.read,getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e1020137) v7 2016-11-14 23:43:33.293727 7fd4c278a700 10 -- [2001:db8::5752]:0/2991243590 >> [2001:db8::8851]:6820/3585 pipe(0x560d017808d0 sd=83 :33274 s=2 pgs=28188 cs=1 l=1 c=0x560d0177faa0).reader got message 1 0x7fd5d0000ff0 osd_op_reply(133 .bucket.meta. [call,getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 2016-11-14 23:43:33.293755 7fd4c278a700 1 -- [2001:db8::5752]:0/2991243590 <== osd.946 [2001:db8::8851]:6820/3585 1 ==== osd_op_reply(133 .bucket.meta. [call,getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 217+0+0 (200346745 0 0) 0x7fd5d0000ff0 con 0x560d0177faa0
Specifically, it looks like a broken bucket instance metadata oid (.bucket.meta.). What does the following return?
$ radosgw-admin metadata get bucket:bucket
Updated by Robin Johnson over 7 years ago
rjohnson@peon5752:~$ sudo radosgw-admin metadata get bucket:bucket { "key": "bucket:bucket", "ver": { "tag": "tWNQSiM9zfnGpgp7vzkIUD6w", "ver": 5 }, "mtime": "2012-09-12 23:43:46.861813Z", "data": { "bucket": { "name": "", "pool": "", "data_extra_pool": "", "index_pool": "", "marker": "", "bucket_id": "", "tenant": "" }, "owner": "", "creation_time": "0.000000", "linked": "false", "has_bucket_info": "false" } }
Updated by Yehuda Sadeh over 7 years ago
Yeah, bucket entrypoint is broken. Does it happen also with newer buckets? This one seems a bit old. We can try identifying these cases (e.g., data.bucket.name is empty). You should be able to remove this one manually: `radosgw-admin metadata rm bucket:bucket`
Updated by Robin Johnson over 7 years ago
I haven't seen it with newer buckets, just old buckets.
I do confirm in this case, that your fix worked: radosgw-admin metadata rm bucket:bucket
Updated by Orit Wasserman over 7 years ago
- Status changed from In Progress to Fix Under Review
- Backport set to jewel
Updated by Orit Wasserman over 7 years ago
- Backport changed from jewel to jewel, kraken
Updated by Nathan Cutler over 7 years ago
kraken backporting will not start for some time - this patch will get into kraken via manual merge
Updated by Orit Wasserman over 7 years ago
- Status changed from Fix Under Review to In Progress