Project

General

Profile

Bug #52867

pick_address.cc prints: unable to find any IPv4 address in networks 'fd00:fd00:fd00:3000::/64' interfaces

Added by John Fulton over 2 years ago. Updated over 2 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Administration/Usability
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

When using IPv6 for my public and cluster network my mon is able to bootstrap (because I have [1]) but I end up with 0 in OSDs when I try to add them. I see the following error message (from [2]).

"unable to find any IPv4 address in networks 'fd00:fd00:fd00:3000::/64' interfaces"

Why is it trying to find an IPv4 address in an IPv6 network?

Here's [3] the output from my bootstrap. I'll attach the full journalctl output from the OSD.

[1] https://github.com/ceph/ceph/pull/43029/commits/306b7e195c5141f895f1e11a4e263ad22c00930c

[2]

# cephadm ls
...
   {
        "style": "cephadm:v1",
        "fsid": "313c101e-9947-43ee-b60a-348a3aa357fd",
        "systemd_unit": "ceph-313c101e-9947-43ee-b60a-348a3aa357fd@osd.8",
...
<pre>
</pre>
# journalctl -u  ceph-313c101e-9947-43ee-b60a-348a3aa357fd@osd.8  | curl -F 'f:1=<-' ix.io
http://ix.io/3B1y


The ix.io URL contains the following lines from journalctl

Oct 06 14:51:37 ceph-0 conmon[39128]: debug 2021-10-06T14:51:37.804+0000 7fd3bb3f6080  1 bdev(0x5650f3c20c00 /var/lib/ceph/osd/ceph-8/block) close
Oct 06 14:51:38 ceph-0 conmon[39128]: debug 2021-10-06T14:51:38.050+0000 7fd3bb3f6080  0 starting osd.8 osd_data /var/lib/ceph/osd/ceph-8 /var/lib/ceph/osd/ceph-8/journal
Oct 06 14:51:38 ceph-0 conmon[39128]: debug 2021-10-06T14:51:38.051+0000 7fd3bb3f6080 -1 unable to find any IPv4 address in networks 'fd00:fd00:fd00:3000::/64' interfaces ''
Oct 06 14:51:38 ceph-0 conmon[39128]: debug 2021-10-06T14:51:38.051+0000 7fd3bb3f6080 -1 Failed to pick public address.

[3]

            "Verifying podman|docker is present...",
            "Verifying lvm2 is present...",
            "Verifying time synchronization is in place...",
            "Unit chronyd.service is enabled and running",
            "Repeating the final host check...",
            "podman|docker (/bin/podman) is present",
            "systemctl is present",
            "lvcreate is present",
            "Unit chronyd.service is enabled and running",
            "Host looks OK",
            "Cluster fsid: 35eb0397-6a2a-4595-bcec-be8caf6e7081",
            "Verifying IP [fd00:fd00:fd00:3000::397] port 3300 ...",
            "Verifying IP [fd00:fd00:fd00:3000::397] port 6789 ...",
            "Mon IP [fd00:fd00:fd00:3000::397] is in CIDR network fd00:fd00:fd00:3000::/64",
            "Pulling container image undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhceph:5-12...",
            "Ceph version: ceph version 16.2.0-72.el8cp (1e802193e0b4084ffcdb2338dd09f08bbea54a1a) pacific (stable)",
            "Extracting ceph user uid/gid from container image...",
            "Creating initial keys...",
            "Creating initial monmap...",
            "Creating mon...",
            "Waiting for mon to start...",
            "Waiting for mon...",
            "mon is available",
            "Assimilating anything we can from ceph.conf...",
            "Generating new minimal ceph.conf...",
            "Restarting the monitor...",
            "Setting mon public_network to fd00:fd00:fd00:3000::/64",
            "Setting cluster_network to fd00:fd00:fd00:4000::/64",
            "Enabling IPv6 (ms_bind_ipv6) binding",
            "Wrote config to /etc/ceph/ceph.conf",
            "Wrote keyring to /etc/ceph/ceph.client.admin.keyring",
            "Creating mgr...",
            "Verifying port 9283 ...",
            "Waiting for mgr to start...",
            "Waiting for mgr...",
            "mgr not available, waiting (1/15)...",
            "mgr not available, waiting (2/15)...",
            "mgr not available, waiting (3/15)...",
            "mgr is available",
            "Enabling cephadm module...",
            "Waiting for the mgr to restart...",
            "Waiting for mgr epoch 5...",
            "mgr epoch 5 is available",
            "Setting orchestrator backend to cephadm...",
            "Using provided ssh keys...",
            "Adding host controller-0...",
            "Deploying mon service with default placement...",
            "Deploying mgr service with default placement...",
            "Deploying crash service with default placement...",
            "You can access the Ceph CLI with:",
            "",
            "\tsudo /usr/sbin/cephadm shell --fsid 35eb0397-6a2a-4595-bcec-be8caf6e7081 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring",
            "",
            "Please consider enabling telemetry to help improve Ceph:",
            "",
            "\tceph telemetry on",
            "",
            "For more information see:",
            "",
            "\thttps://docs.ceph.com/docs/pacific/mgr/telemetry/",
            "",
            "Bootstrap complete." 

journalctl_output.log View - output of 'journalctl -u ceph-35eb0397-6a2a-4595-bcec-be8caf6e7081@osd.2' (147 KB) John Fulton, 10/07/2021 08:58 PM


Related issues

Related to RADOS - Bug #46845: Newly orchestrated OSD fails with 'unable to find any IPv4 address in networks '2001:db8:11d::/120' with ms_bind_ipv6=true Resolved
Related to Messengers - Bug #39711: "unable to find any IPv4 address in networks <ipv6-network>" after upgrade to nautilus on osd and mds Duplicate

History

#1 Updated by Daniel Pivonka over 2 years ago

  • Related to Bug #46845: Newly orchestrated OSD fails with 'unable to find any IPv4 address in networks '2001:db8:11d::/120' with ms_bind_ipv6=true added

#2 Updated by Daniel Pivonka over 2 years ago

  • Related to Bug #39711: "unable to find any IPv4 address in networks <ipv6-network>" after upgrade to nautilus on osd and mds added

#3 Updated by John Fulton over 2 years ago

I set the following after bootstrap and before adding any OSDs but I got the same error.

`ceph config set mon ms_bind_ipv4 false`
`ceph config set mon ms_bind_ipv6 true`

#4 Updated by Sebastian Wagner over 2 years ago

  • Assignee set to Daniel Pivonka

#5 Updated by Kefu Chai over 2 years ago

@John,

per the logging message pasted at http://ix.io/3B1y


Oct 06 14:51:02 ceph-0 conmon[29607]: debug 2021-10-06T14:51:02.669+0000 7f9162fc1080  0 ceph version 16.2.0-72.el8cp (1e802193e0b4084ffcdb2338dd09f08bbea54a1a) pacific (stable), process ceph-osd, pid 6
...
Oct 06 14:51:02 ceph-0 conmon[29607]: debug 2021-10-06T14:51:02.945+0000 7f9162fc1080  0 starting osd.8 osd_data /var/lib/ceph/osd/ceph-8 /var/lib/ceph/osd/ceph-8/journal
Oct 06 14:51:02 ceph-0 conmon[29607]: debug 2021-10-06T14:51:02.945+0000 7f9162fc1080 -1 unable to find any IPv4 address in networks 'fd00:fd00:fd00:3000::/64' interfaces ''
Oct 06 14:51:02 ceph-0 conmon[29607]: debug 2021-10-06T14:51:02.945+0000 7f9162fc1080 -1 Failed to pick public address.

it seems that the osd being tested was a downstream pacific version, as i am not able to find the commit of 1e802193e0b4084ffcdb2338dd09f08bbea54a1a in upstream's pacific branch. is it possible that the pacific commit you were testing failed to include a backport (for instance, in pick_address.cc) from upstream?

#6 Updated by John Fulton over 2 years ago

As per comment #3 I was on the right path but I should have set an OSD setting, not a mon setting. If I run the following after cephadm bootstrap but before I add any OSD nodes then the OSDs are added without the issue.

ceph config set osd ms_bind_ipv4 false

Question:

Should cephadm detect that it was passed IPv6 IPs and set the above for the user?

or

Should pick_address.cc detect that it has an IPv6 address and do the right thing without requiring the user to set ms_bind_ipv4 false?

#7 Updated by Neha Ojha over 2 years ago

  • Status changed from New to Need More Info

#8 Updated by Neha Ojha over 2 years ago

John Fulton wrote:

As per comment #3 I was on the right path but I should have set an OSD setting, not a mon setting. If I run the following after cephadm bootstrap but before I add any OSD nodes then the OSDs are added without the issue.

ceph config set osd ms_bind_ipv4 false

Question:

Should cephadm detect that it was passed IPv6 IPs and set the above for the user?

or

Should pick_address.cc detect that it has an IPv6 address and do the right thing without requiring the user to set ms_bind_ipv4 false?

John, could you please help answer Kefu's question https://tracker.ceph.com/issues/52867#note-5?

#9 Updated by John Fulton over 2 years ago

Kefu Chai wrote:

@John,

per the logging message pasted at http://ix.io/3B1y

[...]

it seems that the osd being tested was a downstream pacific version, as i am not able to find the commit of 1e802193e0b4084ffcdb2338dd09f08bbea54a1a in upstream's pacific branch. is it possible that the pacific commit you were testing failed to include a backport (for instance, in pick_address.cc) from upstream?

Kefu,

Yes, I hit this bug with a downstream pacific version. I re-ran the test with a newer downstream pacific version (scratch build of 16.2.6-24) and didn't see that exact error [1]. The OSD did not go in unless I set `ceph config set osd ms_bind_ipv4 false`. As it's a downstream build I have a RH BZ for it https://bugzilla.redhat.com/show_bug.cgi?id=2016496.

John
[1]
  1. journalctl u | curl -F 'f:1=<' ix.io
    http://ix.io/3FbH

#10 Updated by Sebastian Wagner over 2 years ago

  • Description updated (diff)

#11 Updated by Sebastian Wagner over 2 years ago

  • Project changed from Orchestrator to RADOS
  • Subject changed from cephadm unable to bring OSDs in when using IPv6 to pick_address.cc prints: unable to find any IPv4 address in networks 'fd00:fd00:fd00:3000::/64' interfaces
  • Category changed from cephadm to Administration/Usability
  • Status changed from Need More Info to New
  • Assignee deleted (Daniel Pivonka)

moving over to rados

Also available in: Atom PDF