Project

General

Profile

Actions
Copy link

Bug #15937

closed

radosgw-admin: failure for user create after upgrade from hammer to jewel

Added by Abhishek Lekshmanan almost 8 years ago. Updated almost 8 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Orit Wasserman
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
jewel
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

trying to create user on a node fails in the following way:

host-44-0-2-59:~ # radosgw-admin user create --uid=foobar --display-name=foobar
2016-05-19 12:13:07.059868 7fac0a44e9c0  0 RGWZoneParams::create(): error creating default zone params: (17
could not create user: unable to create user, unable to store user info
2016-05-19 12:13:15.835415 7fac0a44e9c0  0 ERROR: rgw_put_system_obj() oid=.meta:user:foobar:_buATuy1E2MFEX
-22
2016-05-19 12:13:15.835427 7fac0a44e9c0  0 ERROR: put_entry: store_in_heap() key=foobar returned ret=-22

Looks like the metadata_heap is not populated at all 

host-44-0-2-59:~ # radosgw-admin zone get --rgw-zone=default
{   
    "id": "default",
    "name": "default",
    "domain_root": ".rgw",
    "control_pool": ".rgw.control",
    "gc_pool": ".rgw.gc",
    "log_pool": ".log",
    "intent_log_pool": ".intent-log",
    "usage_log_pool": ".usage",
    "user_keys_pool": ".users",
    "user_email_pool": ".users.email",
    "user_swift_pool": ".users.swift",
    "user_uid_pool": ".users.uid",
    "system_key": {
        "access_key": "",
        "secret_key": "" 
    },
    "placement_pools": [
        {   
            "key": "default-placement",
            "val": {
                "index_pool": ".rgw.buckets.index",
                "data_pool": ".rgw.buckets",
                "data_extra_pool": ".rgw.buckets.extra",
                "index_type": 0
            }
        }
    ],
    "metadata_heap": "",
    "realm_id": "" 
}

Files

ceph-client.rgw.logs.zip (543 KB) ceph-client.rgw.logs.zip Charles Alva, 05/24/2016 05:16 PM

Related issues 1 (0 open — 1 closed)

Copied to rgw - Backport #16009: radosgw-admin: failure for user create after upgrade from hammer to jewelResolvedAbhishek LekshmananActions
Actions
Copy link
 #1

Updated by Abhishek Lekshmanan almost 8 years ago

basically manually editing the zone and populating `metadata_heap` does the trick.. but not sure why it is empty in the first place

Actions
Copy link
 #2

Updated by Abhishek Lekshmanan almost 8 years ago

More updates, this hammer cluster did not have any rgw data as such (so .rgw.meta pool might not have been created yet) and upgraded to jewel, not sure how the default zone was populated in hammer, if so this may be the reason for the failure

Actions
Copy link
 #3

Updated by Orit Wasserman almost 8 years ago

  • Assignee set to Orit Wasserman
  • Backport set to jewel
Actions
Copy link
 #4

Updated by Orit Wasserman almost 8 years ago

  • Status changed from New to Fix Under Review
Actions
Copy link
 #5

Updated by Abhishek Lekshmanan almost 8 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions
Copy link
 #6

Updated by Abhishek Lekshmanan almost 8 years ago

  • Copied to Backport #16009: radosgw-admin: failure for user create after upgrade from hammer to jewel added
Actions
Copy link
 #7

Updated by Charles Alva almost 8 years ago

Even when we manually created .rgw.meta and default realm to get its UUID for metadata_heap and realm_id values and put it back in Ceph RGW zone, and finally be able to create S3 user, we are still having HTTP 500 Internal Server Error when testing the s3boto.py test.

We deployed Ceph RGW using ceph-deploy tool in Jewel 10.1. The RGWs are running on CivetWeb port 80.

I'm at lost how to fix this manually. Hope you guys can shed us a light?

Actions
Copy link
 #8

Updated by Orit Wasserman almost 8 years ago

can you provide the logs with debug_rgw=20 and debug_ms=1?

Actions
Copy link
 #9

Updated by Charles Alva almost 8 years ago

Hi Orit,

The attached is the log.

The RGWs sit below Nginx HTTP load balancer. I haven't figured out how to do S3test with HTTPS yet, but even on port 80 I could not make it work. The RGW login page with anonymous user works just fine though.

Hope this helps.

Actions
Copy link
 #10

Updated by Orit Wasserman almost 8 years ago

can you get the output of "radosgw-admin zonegroup get"?

Actions
Copy link
 #11

Updated by Charles Alva almost 8 years ago

It seemed ceph-deploy didn't create the zonegroup.

Below is the output:
# radosgw-admin zonegroup get
failed to init zonegroup: (2) No such file or directory

Actions
Copy link
 #12

Updated by Orit Wasserman almost 8 years ago

try "radosgw-admin zonegroup get --rgw-zonegroup=default"

Actions
Copy link
 #13

Updated by Charles Alva almost 8 years ago

Hi Orit,

Oh, it's missing the realm_id too.

@# radosgw-admin zonegroup get --rgw-zonegroup=default {
"id": "default",
"name": "default",
"api_name": "",
"is_master": "true",
"endpoints": [],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "",
"zones": [ {
"id": "default",
"name": "default",
"endpoints": [],
"log_meta": "false",
"log_data": "false",
"bucket_index_max_shards": 0,
"read_only": "false"
}
],
"placement_targets": [ {
"name": "default-placement",
"tags": []
}
],
"default_placement": "default-placement",
"realm_id": ""
}
@

Actions
Copy link
 #14

Updated by Nathan Cutler almost 8 years ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link

Also available in: Atom PDF