Project

General

Profile

Bug #44661

radosgw can't bind to reserved port (443)

Added by Chris Durham 9 months ago. Updated 7 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Target version:
% Done:

0%

Source:
Community (user)
Tags:
Backport:
octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

Using ceph-radosgw from http://http://download.ceph.com/rpm-octopus/el8/x86_64/ceph-radosgw-15.1.1-0.el8.x86_64.rpm on CentOS 8.1

I have the following setup in /etc/ceph/ceph.conf for radosgw:

[client.rgw.node1]
host = node1
rgw frontends = beast ssl_endpoint=0.0.0.0 ssl_certificate=/etc/ceph/cert.pem

The gateway does not startup. I have, in the log (debug rgw = 20) the following:

2020-03-17T14:10:11.249-0400 7f96f9c4e280 -1 failed to bind address 0.0.0.0:443: Permission denied

If I change to a port > 1024, such as:

rgw frontends = beast ssl_endpoint=0.0.0.0:7480 ssl_certificate=/etc/ceph/cert.pem

That works fine.

Same error if use the local IP address in ssl_endpoint instead of 0.0.0.0. The same error occurs if I use civetweb instead

This has worked fine on earlier releases with the exact same configuration file.

No, nothing else is running on 443 :-).

selinux is ceph-selinux-15.1.1-0.el8.x86_64 from the same repo if that matters


Related issues

Related to Orchestrator - Bug #45197: cephadm: rgw: failed to bind address 0.0.0.0:80 Duplicate
Related to Orchestrator - Feature #43911: test cephadm rgw deployment Resolved
Copied to rgw - Backport #45215: octopus: radosgw can't bind to reserved port (443) Resolved

History

#1 Updated by Chris Durham 9 months ago

same error if I explicitly set port 443, i.e ssl_endpoint = 0.0.0.0:443 or the localIP:443

#2 Updated by Chris Durham 8 months ago

It may be that you are doing setuid()/setgid() or seteuid()/setegid() before bind(), which would preclude you from binding to a port < 1024. Normally radosgw runs s ceph:ceph after starting up.

If I run it by hand (not systemctl) with ssl_endpoint=0.0.0.0 (so it uses 443 by default):

/usr/bin/radosgw -f --cluster ceph --name client.rgw.server-name --setuser root --setgroup root

It binds to port 443 and starts up, which leads me to think that the setuid() is in the wrong place.

#3 Updated by Casey Bodley 8 months ago

radosgw is expected to /start/ as a privileged user to bind these ports, and then setuid to a non-privileged user. for example, systemd will run it as root and setuid to 'ceph'. we bind ports before the setuid to make this work

#4 Updated by Casey Bodley 8 months ago

  • Status changed from New to Triaged

#5 Updated by Chris Durham 8 months ago

Thanks Casey, exactly my point.

When I run it from systemd, it cannot bind to 443 if I have ssl_endpoint set to use 443. If I run it by hand as I specified, it works fine. If I set ssl_endpoint to use 7480, that works too in systemd. Thus it appears you are doing bind() after setuid() or seteuid(), as systemd has it configured to run with --setuser ceph and --setgroup ceph

#6 Updated by Casey Bodley 8 months ago

Not sure what broke there. Radosgw hasn't changed any of this setuid code in either beast or civetweb. We have test coverage that runs radosgw as root with '--setuid ceph --setgid ceph', and it succeeds in binding ports 80 and 443.

#7 Updated by Chris Durham 8 months ago

I've now verified this with 15.2.0 as well, (currently available on download.ceph.com, on two separate centos8.1 installations

If I disable systemd and run it by hand as follows as root....

  1. /usr/bin/radosgw -f --cluster ceph --name client-rgw.servername --setuser ceph --setgroup ceph

It immediately comes back with:

-1 failed to bind address a.b.c.d:443 : Permission denied
-1 ERROR: failed initializing frontend

This is with ceph.conf having:

[client.rgw,servername]
host = servername
rgw frontends = beast ssl_endpoint=a.b.c.d ssl_certificate=.....

If I add :7480 to the above line in ceph,conf, all is well

I've gotten around this for now by adding a local port redirect from 443 to 7480

I agree that I may have done something silly, but this doesn't make sense

#8 Updated by Frode Nordahl 8 months ago

I see this change in Ceph RadosGW behavior on Ubuntu too, bug reference: https://bugs.launchpad.net/ceph/+bug/1869324

#9 Updated by James Page 8 months ago

https://github.com/ceph/ceph/commit/e28718eaa18e49c770db45820b591088ea92846b moves the creation of the global ceph context to before the determination of the frontend for the rgw so the flag that defers the privileged drop in the context setup is not set at the right point in time:

https://github.com/ceph/ceph/blob/master/src/rgw/rgw_main.cc#L234

so privs are dropped before the frontend has been determined and the appropriate defer flag set.

#10 Updated by Casey Bodley 8 months ago

  • Assignee set to Casey Bodley

#11 Updated by Casey Bodley 8 months ago

  • Status changed from Triaged to Fix Under Review
  • Pull request ID set to 34613

#12 Updated by Nathan Cutler 8 months ago

  • Backport set to octopus
$ git branch --contains 966c050 | egrep '^..master|^..octopus'
  master
* octopus

#13 Updated by Casey Bodley 8 months ago

  • Priority changed from Normal to Urgent

#14 Updated by Casey Bodley 7 months ago

  • Status changed from Fix Under Review to Pending Backport

#15 Updated by Sebastian Wagner 7 months ago

  • Related to Bug #45197: cephadm: rgw: failed to bind address 0.0.0.0:80 added

#16 Updated by Sebastian Wagner 7 months ago

#17 Updated by Nathan Cutler 7 months ago

  • Copied to Backport #45215: octopus: radosgw can't bind to reserved port (443) added

#18 Updated by Nathan Cutler 7 months ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Also available in: Atom PDF