Project

General

Profile

Bug #15937

radosgw-admin: failure for user create after upgrade from hammer to jewel

Added by Abhishek Lekshmanan over 2 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Target version:
-
Start date:
05/19/2016
Due date:
% Done:

0%

Source:
other
Tags:
Backport:
jewel
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

trying to create user on a node fails in the following way:

host-44-0-2-59:~ # radosgw-admin user create --uid=foobar --display-name=foobar
2016-05-19 12:13:07.059868 7fac0a44e9c0  0 RGWZoneParams::create(): error creating default zone params: (17
could not create user: unable to create user, unable to store user info
2016-05-19 12:13:15.835415 7fac0a44e9c0  0 ERROR: rgw_put_system_obj() oid=.meta:user:foobar:_buATuy1E2MFEX
-22
2016-05-19 12:13:15.835427 7fac0a44e9c0  0 ERROR: put_entry: store_in_heap() key=foobar returned ret=-22

Looks like the metadata_heap is not populated at all

host-44-0-2-59:~ # radosgw-admin zone get --rgw-zone=default
{   
    "id": "default",
    "name": "default",
    "domain_root": ".rgw",
    "control_pool": ".rgw.control",
    "gc_pool": ".rgw.gc",
    "log_pool": ".log",
    "intent_log_pool": ".intent-log",
    "usage_log_pool": ".usage",
    "user_keys_pool": ".users",
    "user_email_pool": ".users.email",
    "user_swift_pool": ".users.swift",
    "user_uid_pool": ".users.uid",
    "system_key": {
        "access_key": "",
        "secret_key": "" 
    },
    "placement_pools": [
        {   
            "key": "default-placement",
            "val": {
                "index_pool": ".rgw.buckets.index",
                "data_pool": ".rgw.buckets",
                "data_extra_pool": ".rgw.buckets.extra",
                "index_type": 0
            }
        }
    ],
    "metadata_heap": "",
    "realm_id": "" 
}

ceph-client.rgw.logs.zip (543 KB) Charles Alva, 05/24/2016 05:16 PM


Related issues

Copied to rgw - Backport #16009: radosgw-admin: failure for user create after upgrade from hammer to jewel Resolved

History

#1 Updated by Abhishek Lekshmanan over 2 years ago

basically manually editing the zone and populating `metadata_heap` does the trick.. but not sure why it is empty in the first place

#2 Updated by Abhishek Lekshmanan over 2 years ago

More updates, this hammer cluster did not have any rgw data as such (so .rgw.meta pool might not have been created yet) and upgraded to jewel, not sure how the default zone was populated in hammer, if so this may be the reason for the failure

#3 Updated by Orit Wasserman over 2 years ago

  • Assignee set to Orit Wasserman
  • Backport set to jewel

#4 Updated by Orit Wasserman over 2 years ago

  • Status changed from New to Need Review

#5 Updated by Abhishek Lekshmanan over 2 years ago

  • Status changed from Need Review to Pending Backport

#6 Updated by Abhishek Lekshmanan over 2 years ago

  • Copied to Backport #16009: radosgw-admin: failure for user create after upgrade from hammer to jewel added

#7 Updated by Charles Alva over 2 years ago

Even when we manually created .rgw.meta and default realm to get its UUID for metadata_heap and realm_id values and put it back in Ceph RGW zone, and finally be able to create S3 user, we are still having HTTP 500 Internal Server Error when testing the s3boto.py test.

We deployed Ceph RGW using ceph-deploy tool in Jewel 10.1. The RGWs are running on CivetWeb port 80.

I'm at lost how to fix this manually. Hope you guys can shed us a light?

#8 Updated by Orit Wasserman over 2 years ago

can you provide the logs with debug_rgw=20 and debug_ms=1?

#9 Updated by Charles Alva over 2 years ago

Hi Orit,

The attached is the log.

The RGWs sit below Nginx HTTP load balancer. I haven't figured out how to do S3test with HTTPS yet, but even on port 80 I could not make it work. The RGW login page with anonymous user works just fine though.

Hope this helps.

#10 Updated by Orit Wasserman over 2 years ago

can you get the output of "radosgw-admin zonegroup get"?

#11 Updated by Charles Alva over 2 years ago

It seemed ceph-deploy didn't create the zonegroup.

Below is the output:
# radosgw-admin zonegroup get
failed to init zonegroup: (2) No such file or directory

#12 Updated by Orit Wasserman over 2 years ago

try "radosgw-admin zonegroup get --rgw-zonegroup=default"

#13 Updated by Charles Alva over 2 years ago

Hi Orit,

Oh, it's missing the realm_id too.

@# radosgw-admin zonegroup get --rgw-zonegroup=default {
"id": "default",
"name": "default",
"api_name": "",
"is_master": "true",
"endpoints": [],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "",
"zones": [ {
"id": "default",
"name": "default",
"endpoints": [],
"log_meta": "false",
"log_data": "false",
"bucket_index_max_shards": 0,
"read_only": "false"
}
],
"placement_targets": [ {
"name": "default-placement",
"tags": []
}
],
"default_placement": "default-placement",
"realm_id": ""
}
@

#14 Updated by Nathan Cutler over 2 years ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF