Project

General

Profile

Actions

Bug #15597

closed

rgw: when upgrading to jewel, a default zone setup is is not upgraded correctly

Added by Yehuda Sadeh almost 8 years ago. Updated almost 8 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
jewel
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

- hammer / infernalis
- create a default setup (don't set up zone)
- create user + bucket
- stop + upgrade to jewel
- radosgw-admin user info --uid=<uid>


Related issues 1 (0 open1 closed)

Copied to rgw - Backport #15803: jewel: rgw: when upgrading to jewel, a default zone setup is is not upgraded correctlyResolvedYehuda SadehActions
Actions #1

Updated by Yehuda Sadeh almost 8 years ago

  • Priority changed from Normal to Urgent
Actions #2

Updated by Yehuda Sadeh almost 8 years ago

  • Backport set to jewel
Actions #3

Updated by Yehuda Sadeh almost 8 years ago

  • Project changed from Ceph to rgw
Actions #5

Updated by Nathan Cutler almost 8 years ago

  • Status changed from New to Fix Under Review
Actions #6

Updated by Herve Rousseau almost 8 years ago

Got also hit by this on a brand new Jewel cluster (so nothing to upgrade).

I have uploaded radosgw logs with debug_rgw set to 20 at

ceph-post-file: 0161c5aa-a161-4c9c-94e5-89b4b0644141

If you have a gitbuilder rpm I'd be happy to test it !

Actions #7

Updated by Yehuda Sadeh almost 8 years ago

Herve, maybe you see a different issue? This bug is specifically about failing to upgrade. Looking at your log it seems that your radosgw is processing requests. What are you actually seeing? Can you also run it with 'debug ms = 1'? Thanks

Actions #8

Updated by Herve Rousseau almost 8 years ago

Indeed, it could be a different problem. As the symptom is very similar I thought they would be somewhat related.

After some more testing, I confirm that I can do all operations, it's just very annoying to get these errors each time I instantiate radosgw-admin and very slow as it tries to list/create/set the zone.

The request that you see is actually just some health check script that parses the HTTP return code (of GET /), no S3 operation here.

2016-04-28 21:50:53.230504 7fb087875a40  0 RGWZoneParams::create(): error creating default zone params: (17) File exists

Here's the radosgw log with a clean cluster and debug_ms=1 in addition to debug_rgw:

ceph-post-file: 3bf4fcf2-8a3b-425c-9c30-e6ad01743f73

Actions #9

Updated by Abhishek Lekshmanan almost 8 years ago

Herve Rousseau wrote:

Indeed, it could be a different problem. As the symptom is very similar I thought they would be somewhat related.

After some more testing, I confirm that I can do all operations, it's just very annoying to get these errors each time I instantiate radosgw-admin and very slow as it tries to list/create/set the zone.

The request that you see is actually just some health check script that parses the HTTP return code (of GET /), no S3 operation here.

2016-04-28 21:50:53.230504 7fb087875a40 0 RGWZoneParams::create(): error creating default zone params: (17) File exists

We could probably increase the verbosity for error messages like these so that they dont get printed for the regular case so that it is less confusing for the user?

[...]

Here's the radosgw log with a clean cluster and debug_ms=1 in addition to debug_rgw:
[...]

Actions #10

Updated by Herve Rousseau almost 8 years ago

Abhishek Lekshmanan wrote:

We could probably increase the verbosity for error messages like these so that they dont get printed for the regular case so that it is less confusing for the user?

Hi,

I think more work is needed than just changing the loglevel of the message, the cluster has no buckets and still it takes more than 1 minute to get the empty list back:

$ time radosgw-admin --id radosgw bucket list
2016-04-29 16:53:01.012824 7f90e224ba40  0 RGWZoneParams::create(): error creating default zone params: (17) File exists
[]

real    1m2.578s
user    0m0.651s
sys     0m0.415s

Actions #11

Updated by Yehuda Sadeh almost 8 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions #12

Updated by Nathan Cutler almost 8 years ago

  • Copied to Backport #15803: jewel: rgw: when upgrading to jewel, a default zone setup is is not upgraded correctly added
Actions #13

Updated by Abhishek Lekshmanan almost 8 years ago

@Herve Rosseau,

can you open a different issue to track the radosgw-admin operation slowness, are you seeing this only for bucket listing or for user ops etc as well?

Actions #14

Updated by Nathan Cutler almost 8 years ago

  • Status changed from Pending Backport to Resolved
Actions

Also available in: Atom PDF