Project

General

Profile

Actions
Copy link

Bug #50113

closed

Upgrading to v16 breaks rgw_frontends setting

Added by 玮文 胡 about 3 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
High
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v16.2.0
ceph-qa-suite:
Pull request ID:
41400
Crash signature (v1):
Crash signature (v2):

Description

We are upgrading our cluster to v16 today with cephadm.

We have rgw daemons set up and the "rgw_frontends" config is left at its default (beast port=7480)

However, when upgrading, cephadm seems want to redeploy all rgw daemons. It set "rgw_frontends" config for new daemons to "beast port=80", which of course breaks our existing applications. Besides, We have other daemon listening on port 80. so we continuously get errors like:

mgr.gpu024.bapbcz (mgr.6544553) 113 : cephadm [INF] Deploying daemon rgw.smil.b7-1.gpu013.zshphp on gpu013
mgr.gpu024.bapbcz (mgr.6544553) 114 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon rgw.smil.b7-1.gpu013.zshphp ...
Verifying port 80 ...
Cannot bind to IP 0.0.0.0 port 80: [Errno 98] Address already in use
ERROR: TCP Port(s) '80' required for rgw already in use
Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/serve.py", line 1172, in _remote_connection
yield (conn, connr)
File "/usr/share/ceph/mgr/cephadm/serve.py", line 1087, in _run_cephadm
code, '\n'.join(err)))
orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon rgw.smil.b7-1.gpu013.zshphp ...
Verifying port 80 ...
Cannot bind to IP 0.0.0.0 port 80: [Errno 98] Address already in use
ERROR: TCP Port(s) '80' required for rgw already in use
mgr.gpu024.bapbcz (mgr.6544553) 115 : cephadm [INF] Removing key for client.rgw.smil.b7-1.gpu013.zshphp

and the deployment of rgw cannot proceed.

We ended up stopping our daemon on 80 port to let it proceed, then fix the config and restart all rgw daemons manually.

Actions
Copy link
 #1

Updated by Deepika Upadhyay almost 3 years ago 

2021-05-06T12:25:12.927 INFO:journalctl@ceph.mgr.x.smithi169.stdout:May 06 12:25:12 smithi169 bash[21973]: Verifying port 8000 ...
2021-05-06T12:25:12.928 INFO:journalctl@ceph.mgr.x.smithi169.stdout:May 06 12:25:12 smithi169 bash[21973]: Cannot bind to IP 0.0.0.0 port 8000: [Errno 98] Address already in use
2021-05-06T12:25:12.928 INFO:journalctl@ceph.mgr.x.smithi169.stdout:May 06 12:25:12 smithi169 bash[21973]: ERROR: TCP Port(s) '8000' required for rgw already in use
2021-05-06T12:25:12.928 INFO:journalctl@ceph.mgr.x.smithi169.stdout:May 06 12:25:12 smithi169 bash[21973]: Traceback (most recent call last):
2021-05-06T12:25:12.929 INFO:journalctl@ceph.mgr.x.smithi169.stdout:May 06 12:25:12 smithi169 bash[21973]:   File "/usr/share/ceph/mgr/cephadm/serve.py", line 1223, in _remote_connection
2021-05-06T12:25:12.929 INFO:journalctl@ceph.mgr.x.smithi169.stdout:May 06 12:25:12 smithi169 bash[21973]:     yield (conn, connr)
2021-05-06T12:25:12.930 INFO:journalctl@ceph.mgr.x.smithi169.stdout:May 06 12:25:12 smithi169 bash[21973]:   File "/usr/share/ceph/mgr/cephadm/serve.py", line 1120, in _run_cephadm
2021-05-06T12:25:12.930 INFO:journalctl@ceph.mgr.x.smithi169.stdout:May 06 12:25:12 smithi169 bash[21973]:     code, '\n'.join(err)))
2021-05-06T12:25:12.930 INFO:journalctl@ceph.mgr.x.smithi169.stdout:May 06 12:25:12 smithi169 bash[21973]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon rgw.r.z.smithi169.bjprov ...
2021-05-06T12:25:12.931 INFO:journalctl@ceph.mgr.x.smithi169.stdout:May 06 12:25:12 smithi169 bash[21973]: Verifying port 8000 ...
2021-05-06T12:25:12.931 INFO:journalctl@ceph.mgr.x.smithi169.stdout:May 06 12:25:12 smithi169 bash[21973]: Cannot bind to IP 0.0.0.0 port 8000: [Errno 98] Address already in use
2021-05-06T12:25:12.932 INFO:journalctl@ceph.mgr.x.smithi169.stdout:May 06 12:25:12 smithi169 bash[21973]: ERROR: TCP Port(s) '8000' required for rgw already in use

/ceph/teuthology-archive/ideepika-2021-05-06_05:00:22-rados-wip-deepika-testing-2021-05-05-1704-distro-basic-smithi/6100272/teuthology.log

Actions
Copy link
 #3

Updated by Sebastian Wagner almost 3 years ago

workaround is to manually set the port:

 ceph orch apply rgw .... --port ..
Actions
Copy link
 #4

Updated by Sage Weil almost 3 years ago

Before you set the port (if it's not too late), can you attach the rgw portion of the 'ceph orch ls --export' output?

Actions
Copy link
 #5

Updated by Sage Weil almost 3 years ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 41400
Actions
Copy link
 #6

Updated by Sebastian Wagner almost 3 years ago

  • Status changed from Fix Under Review to Pending Backport
  • Backport set to pacific
Actions
Copy link
 #7

Updated by Sebastian Wagner almost 3 years ago

  • Status changed from Pending Backport to Resolved

turns out doc/releases/pacific.rst is completely missing in pacific

Actions
Copy link

Also available in: Atom PDF