Project

General

Profile

Tasks #40937

Problem "open vSwitch" networkbond set_numa_affinity

Added by Mario Hosse 26 days ago. Updated 21 days ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
Start date:
07/25/2019
Due date:
% Done:

0%

Tags:
set_numa_affinity open vSwitch
Reviewed:
Affected Versions:
Component(RADOS):
Pull request ID:

Description

Hello,

after installing ceph 14.2.1 (Proxmox 6.0-4-6.0) i have the following message example in syslog when starting osd:

Jul 22 11:53:48 prox1a ceph-osd[2731]: 2019-07-22 11:53:48.124 7f5e02576700 -1 osd.2 20115 set_numa_affinity unable to identify public interface 'ceph_public' numa node: (2) No such file or directory

After consultation in the proxmoxforum under:
[[https://forum.proxmox.com/threads/fragen-zu-ceph-nach-upgrade-5-4-auf-6-0.56209/]] in German.

Here it was determined that in the file:

ceph/src/common/pick_address.cc

to a path and a file which is not created by "open vSwitch" as a virtual network card:
int get_iface_numa_node(                                                    
  const std::string& iface,                                                
  int *node)                                                               
{                                                                        
  string fn = std::string("/sys/class/net/") + iface + "/device/numa_node";

Under /sys/class/net/ceph_public (my virtuell Bond) the path /device and file /device/numa_node is not created from "open vSwitch".

The network card is a virtual bond device over "open vSwitch" with two mellanox 40GB network cards.
So the osd can't find the file numa_node at startup and the error message is generated accordingly.
If you need more information, please do not hesitate to contact me.

History

#1 Updated by Yuri Weinstein 26 days ago

  • Project changed from Stable releases to Ceph

#2 Updated by Greg Farnum 22 days ago

  • Project changed from Ceph to RADOS

Does this visibly break anything or is it just a message in the logs?

#3 Updated by Mario Hosse 21 days ago

It's all working so far. The question is, does the cpu assignment work? To do this I manually set the following command for each osd (example):

ceph config set osd.0 osd_numa_node 0

Result:

 ceph osd numa status
OSD HOST     NETWORK STORAGE AFFINITY   CPUS
  0 prox1a     -       1         0       0-11.24-35
  1 prox1a     -       1         0       0-11,24-35
  2 prox1a     -       1         0       0-11,24-35
  3 prox1a     -       1         1      12-23,36-47
  4 prox1a     -       1         1      12-23,36-47
  5 prox2b     -       1         0       0-11.24-35
  6 prox2b     -       1         0       0-11.24-35
  7 prox2b     -       1         0       0-11.24-35
  8 prox2b     -       1         1      12-23,36-47
  9 prox2b     -       1         1      12-23,36-47
 10 prox3c     -       1         0       0-11.24-35
 11 prox3c     -       1         0       0-11.24-35
 12 prox3c     -       1         0       0-11.24-35
 13 prox3c     -       1         1      12-23,36-47
 14 prox3c     -       1         1      12-23,36-47
 15 prox4d     -       1         0       0-11.24-35
 16 prox4d     -       1         0       0-11.24-35
 17 prox4d     -       1         0       0-11.24-35
 18 prox4d     -       1         1      12-23,36-47
 19 prox4d     -       1         1      12-23,36-47
 20 prox1a     -       1         1      12-23,36-47
 21 prox2b     -       1         1      12-23,36-47
 22 prox3c     -       1         1      12-23,36-47
 23 prox4d     -       1         1      12-23,36-47

As you can see, the network cannot be set at startup. I can't judge what effects this will have except for the error messages in the monitors and osds.

#4 Updated by Mario Hosse 21 days ago

During the reboot the following messages are in the log:

Jul 29 18:53:56 prox1a sh[1206]: Running command: /usr/sbin/ceph-volume simple trigger 4-d9f5c57d-3c13-4b6c-9a05-0df3a7bdbbda
Jul 29 18:53:56 prox1a systemd[1]: Started Ceph object storage daemon osd.2.
Jul 29 18:53:56 prox1a sh[1186]: Running command: /usr/sbin/ceph-volume simple trigger 20-0e506159-0e6c-4d8d-9443-cb2b85f43f06
Jul 29 18:53:56 prox1a sh[1195]: Running command: /usr/sbin/ceph-volume simple trigger 0-3c5adc6f-a126-4665-88d2-7b5cf026aecd
Jul 29 18:53:56 prox1a sh[1194]: Running command: /usr/sbin/ceph-volume simple trigger 3-97ae2b94-2a17-442f-b5a6-8a8408abb705
Jul 29 18:53:56 prox1a sh[1182]: Running command: /usr/sbin/ceph-volume simple trigger 1-e6a63cbc-21af-4157-8c28-1b4b93a49998
Jul 29 18:53:56 prox1a sh[1189]: Running command: /usr/sbin/ceph-volume simple trigger 2-da071614-5f04-4c91-b172-69710f6094b4
Jul 29 18:53:56 prox1a systemd[1]: ceph-volume@simple-20-0e506159-0e6c-4d8d-9443-cb2b85f43f06.service: Succeeded.
Jul 29 18:53:56 prox1a systemd[1]: Started Ceph Volume activation: simple-20-0e506159-0e6c-4d8d-9443-cb2b85f43f06.
Jul 29 18:53:56 prox1a systemd[1]: ceph-volume@simple-0-3c5adc6f-a126-4665-88d2-7b5cf026aecd.service: Succeeded.
Jul 29 18:53:56 prox1a systemd[1]: Started Ceph Volume activation: simple-0-3c5adc6f-a126-4665-88d2-7b5cf026aecd.
Jul 29 18:53:56 prox1a systemd[1]: ceph-volume@simple-1-e6a63cbc-21af-4157-8c28-1b4b93a49998.service: Succeeded.
Jul 29 18:53:56 prox1a systemd[1]: Started Ceph Volume activation: simple-1-e6a63cbc-21af-4157-8c28-1b4b93a49998.
Jul 29 18:53:56 prox1a systemd[1]: ceph-volume@simple-4-d9f5c57d-3c13-4b6c-9a05-0df3a7bdbbda.service: Succeeded.
Jul 29 18:53:56 prox1a systemd[1]: Started Ceph Volume activation: simple-4-d9f5c57d-3c13-4b6c-9a05-0df3a7bdbbda.
Jul 29 18:53:56 prox1a systemd[1]: ceph-volume@simple-3-97ae2b94-2a17-442f-b5a6-8a8408abb705.service: Succeeded.
Jul 29 18:53:56 prox1a systemd[1]: Started Ceph Volume activation: simple-3-97ae2b94-2a17-442f-b5a6-8a8408abb705.
Jul 29 18:53:56 prox1a systemd[1]: ceph-volume@simple-2-da071614-5f04-4c91-b172-69710f6094b4.service: Succeeded.
Jul 29 18:53:56 prox1a systemd[1]: Started Ceph Volume activation: simple-2-da071614-5f04-4c91-b172-69710f6094b4.

Jul 29 18:54:24 prox1a ceph-mon[2636]: 2019-07-29 18:54:24.824 7febc2d8c700 -1 mon.prox1a@0(electing) e8 failed to get devid for : fallback method has serial ''but no model

Jul 29 18:54:30 prox1a ceph-osd[2671]: 2019-07-29 18:54:30.420 7fd5ed208f80 -1 unable to find any IPv4 address in networks '192.xxx.xx3.0/24' interfaces ''
Jul 29 18:54:30 prox1a ceph-osd[2676]: 2019-07-29 18:54:30.420 7f47d21fcf80 -1 unable to find any IPv4 address in networks '192.xxx.xx3.0/24' interfaces ''
Jul 29 18:54:30 prox1a ceph-osd[2680]: 2019-07-29 18:54:30.420 7fc266544f80 -1 unable to find any IPv4 address in networks '192.xxx.xx3.0/24' interfaces ''
Jul 29 18:54:30 prox1a ceph-osd[2669]: 2019-07-29 18:54:30.420 7f443f653f80 -1 unable to find any IPv4 address in networks '192.xxx.xx3.0/24' interfaces ''
Jul 29 18:54:30 prox1a ceph-osd[2671]: 2019-07-29 18:54:30.424 7fd5ed208f80 -1 unable to find any IPv4 address in networks '192.xxx.xx4.0/24' interfaces ''
Jul 29 18:54:30 prox1a ceph-osd[2669]: 2019-07-29 18:54:30.424 7f443f653f80 -1 unable to find any IPv4 address in networks '192.xxx.xx4.0/24' interfaces ''
Jul 29 18:54:30 prox1a ceph-osd[2676]: 2019-07-29 18:54:30.424 7f47d21fcf80 -1 unable to find any IPv4 address in networks '192.xxx.xx4.0/24' interfaces ''
Jul 29 18:54:30 prox1a ceph-osd[2680]: 2019-07-29 18:54:30.424 7fc266544f80 -1 unable to find any IPv4 address in networks '192.xxx.xx4.0/24' interfaces ''
Jul 29 18:54:30 prox1a ceph-osd[2680]: 2019-07-29 18:54:30.424 7fc266306700 -1 PosixStack listen unable to listen on v2:192.xxx.xx3.1:6803/0: (98) Address already in use
Jul 29 18:54:30 prox1a ceph-osd[2666]: 2019-07-29 18:54:30.436 7fd5c4db6f80 -1 unable to find any IPv4 address in networks '192.xxx.xx3.0/24' interfaces ''
Jul 29 18:54:30 prox1a ceph-osd[2666]: 2019-07-29 18:54:30.436 7fd5c4db6f80 -1 unable to find any IPv4 address in networks '192.xxx.xx4.0/24' interfaces ''

Jul 29 18:54:31 prox1a ceph-osd[2657]: 2019-07-29 18:54:31.470 7fe6cbcb6f80 -1 unable to find any IPv4 address in networks '192.xxx.xx3.0/24' interfaces ''
Jul 29 18:54:31 prox1a ceph-osd[2657]: 2019-07-29 18:54:31.470 7fe6cbcb6f80 -1 unable to find any IPv4 address in networks '192.xxx.xx4.0/24' interfaces ''

Jul 29 18:54:33 prox1a ceph-mon[2636]: 2019-07-29 18:54:33.901 7febc2d8c700 -1 mon.prox1a@0(electing) e8 failed to get devid for : fallback method has serial ''but no model

Jul 29 18:54:39 prox1a ceph-mon[2636]: 2019-07-29 18:54:39.856 7febc2d8c700 -1 mon.prox1a@0(electing) e8 failed to get devid for : fallback method has serial ''but no model

Jul 29 18:55:52 prox1a ceph-osd[2666]: 2019-07-29 18:55:52.163 7fd5c4db6f80 -1 osd.3 20264 log_to_monitors {default=true}
Jul 29 18:55:52 prox1a ceph-osd[2666]: 2019-07-29 18:55:52.171 7fd5be0a9700 -1 osd.3 20264 set_numa_affinity unable to identify public interface 'ceph_public' numa node: (2) No such file or directory
Jul 29 18:55:52 prox1a ceph-osd[2671]: 2019-07-29 18:55:52.871 7fd5ed208f80 -1 osd.0 20264 log_to_monitors {default=true}
Jul 29 18:55:52 prox1a ceph-osd[2671]: 2019-07-29 18:55:52.879 7fd5e64fb700 -1 osd.0 20264 set_numa_affinity unable to identify public interface 'ceph_public' numa node: (2) No such file or directory
Jul 29 18:55:55 prox1a ceph-osd[2680]: 2019-07-29 18:55:55.779 7fc266544f80 -1 osd.2 20264 log_to_monitors {default=true}
Jul 29 18:55:55 prox1a ceph-osd[2680]: 2019-07-29 18:55:55.783 7fc25f837700 -1 osd.2 20264 set_numa_affinity unable to identify public interface 'ceph_public' numa node: (2) No such file or directory
Jul 29 18:55:56 prox1a ceph-osd[2676]: 2019-07-29 18:55:56.723 7f47d21fcf80 -1 osd.1 20264 log_to_monitors {default=true}
Jul 29 18:55:56 prox1a ceph-osd[2676]: 2019-07-29 18:55:56.731 7f47cb4ef700 -1 osd.1 20264 set_numa_affinity unable to identify public interface 'ceph_public' numa node: (2) No such file or directory
Jul 29 18:55:58 prox1a ceph-osd[2669]: 2019-07-29 18:55:58.411 7f443f653f80 -1 osd.20 20264 log_to_monitors {default=true}
Jul 29 18:55:58 prox1a ceph-osd[2669]: 2019-07-29 18:55:58.419 7f4438946700 -1 osd.20 20264 set_numa_affinity unable to identify public interface 'ceph_public' numa node: (2) No such file or directory

Jul 29 18:55:58 prox1a ceph-osd[2666]: 2019-07-29 18:55:58.723 7fd5c3b76700 -1 --2- [v2:192.xxx.xx4.1:6816/2666,v1:192.xxx.xx4.1:6817/2666] >> [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] conn(0x562a88a9b680 0x562a88aa9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] is using msgr V1 protocol
Jul 29 18:55:58 prox1a ceph-osd[2676]: 2019-07-29 18:55:58.723 7f47d17bd700 -1 --2- [v2:192.xxx.xx4.1:6802/2676,v1:192.xxx.xx4.1:6803/2676] >> [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] conn(0x557e6c10b680 0x557e6c066100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] is using msgr V1 protocol
Jul 29 18:55:58 prox1a ceph-osd[2680]: 2019-07-29 18:55:58.723 7fc265304700 -1 --2- [v2:192.xxx.xx4.1:6807/2680,v1:192.xxx.xx4.1:6809/2680] >> [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] conn(0x564249d15680 0x564249d2bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] is using msgr V1 protocol
Jul 29 18:55:58 prox1a ceph-osd[2671]: 2019-07-29 18:55:58.723 7fd5ebfc8700 -1 --2- [v2:192.xxx.xx4.1:6800/2671,v1:192.xxx.xx4.1:6801/2671] >> [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] conn(0x55bbac483a80 0x55bbac4c4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] is using msgr V1 protocol
Jul 29 18:55:58 prox1a ceph-osd[2657]: 2019-07-29 18:55:58.759 7fe6cbcb6f80 -1 osd.4 20264 log_to_monitors {default=true}
Jul 29 18:55:58 prox1a ceph-osd[2657]: 2019-07-29 18:55:58.767 7fe6c4fa9700 -1 osd.4 20264 set_numa_affinity unable to identify public interface 'ceph_public' numa node: (2) No such file or directory
Jul 29 18:55:58 prox1a ceph-osd[2676]: 2019-07-29 18:55:58.923 7f47d17bd700 -1 --2- [v2:192.xxx.xx4.1:6802/2676,v1:192.xxx.xx4.1:6803/2676] >> [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] conn(0x557e6c10b680 0x557e6c066100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] is using msgr V1 protocol
Jul 29 18:55:58 prox1a ceph-osd[2680]: 2019-07-29 18:55:58.923 7fc265304700 -1 --2- [v2:192.xxx.xx4.1:6807/2680,v1:192.xxx.xx4.1:6809/2680] >> [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] conn(0x564249d15680 0x564249d2bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] is using msgr V1 protocol
Jul 29 18:55:58 prox1a ceph-osd[2671]: 2019-07-29 18:55:58.923 7fd5ebfc8700 -1 --2- [v2:192.xxx.xx4.1:6800/2671,v1:192.xxx.xx4.1:6801/2671] >> [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] conn(0x55bbac483a80 0x55bbac4c4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] is using msgr V1 protocol
Jul 29 18:55:58 prox1a ceph-osd[2666]: 2019-07-29 18:55:58.923 7fd5c3b76700 -1 --2- [v2:192.xxx.xx4.1:6816/2666,v1:192.xxx.xx4.1:6817/2666] >> [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] conn(0x562a88a9b680 0x562a88aa9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] is using msgr V1 protocol
Jul 29 18:55:59 prox1a ceph-osd[2680]: 2019-07-29 18:55:59.323 7fc265304700 -1 --2- [v2:192.xxx.xx4.1:6807/2680,v1:192.xxx.xx4.1:6809/2680] >> [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] conn(0x564249d15680 0x564249d2bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] is using msgr V1 protocol
Jul 29 18:55:59 prox1a ceph-osd[2666]: 2019-07-29 18:55:59.323 7fd5c3b76700 -1 --2- [v2:192.xxx.xx4.1:6816/2666,v1:192.xxx.xx4.1:6817/2666] >> [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] conn(0x562a88a9b680 0x562a88aa9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] is using msgr V1 protocol
Jul 29 18:55:59 prox1a ceph-osd[2676]: 2019-07-29 18:55:59.323 7f47d17bd700 -1 --2- [v2:192.xxx.xx4.1:6802/2676,v1:192.xxx.xx4.1:6803/2676] >> [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] conn(0x557e6c10b680 0x557e6c066100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] is using msgr V1 protocol
Jul 29 18:55:59 prox1a ceph-osd[2671]: 2019-07-29 18:55:59.327 7fd5ebfc8700 -1 --2- [v2:192.xxx.xx4.1:6800/2671,v1:192.xxx.xx4.1:6801/2671] >> [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] conn(0x55bbac483a80 0x55bbac4c4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] is using msgr V1 protocol
Jul 29 18:55:59 prox1a ceph-osd[2669]: 2019-07-29 18:55:59.499 7f443f415700 -1 --2- [v2:192.xxx.xx4.1:6805/2669,v1:192.xxx.xx4.1:6808/2669] >> [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] conn(0x561cf1f17b00 0x561cf1ed5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] is using msgr V1 protocol
Jul 29 18:55:59 prox1a ceph-osd[2669]: 2019-07-29 18:55:59.699 7f443f415700 -1 --2- [v2:192.xxx.xx4.1:6805/2669,v1:192.xxx.xx4.1:6808/2669] >> [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] conn(0x561cf1f17b00 0x561cf1ed5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2671]: 2019-07-29 18:56:00.035 7fd5ec7c9700 -1 --2- 192.xxx.xx3.1:0/2671 >> [v2:192.xxx.xx3.2:6819/2752,v1:192.xxx.xx3.2:6822/2752] conn(0x55bbac482400 0x55bbac499700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx3.2:6819/2752,v1:192.xxx.xx3.2:6822/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2666]: 2019-07-29 18:56:00.035 7fd5c3b76700 -1 --2- 192.xxx.xx3.1:0/2666 >> [v2:192.xxx.xx3.2:6819/2752,v1:192.xxx.xx3.2:6822/2752] conn(0x562a88a4c900 0x562a88a31700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx3.2:6819/2752,v1:192.xxx.xx3.2:6822/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2676]: 2019-07-29 18:56:00.035 7f47d0fbc700 -1 --2- 192.xxx.xx3.1:0/2676 >> [v2:192.xxx.xx3.2:6819/2752,v1:192.xxx.xx3.2:6822/2752] conn(0x557e6c038900 0x557e6c035600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx3.2:6819/2752,v1:192.xxx.xx3.2:6822/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2680]: 2019-07-29 18:56:00.035 7fc265304700 -1 --2- 192.xxx.xx3.1:0/2680 >> [v2:192.xxx.xx3.2:6819/2752,v1:192.xxx.xx3.2:6822/2752] conn(0x564249cbe400 0x564249ce0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx3.2:6819/2752,v1:192.xxx.xx3.2:6822/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2666]: 2019-07-29 18:56:00.035 7fd5c4377700 -1 --2- 192.xxx.xx4.1:0/2666 >> [v2:192.xxx.xx4.2:6819/2752,v1:192.xxx.xx4.2:6822/2752] conn(0x562a88a4c480 0x562a88a31180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6819/2752,v1:192.xxx.xx4.2:6822/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2671]: 2019-07-29 18:56:00.035 7fd5ecfca700 -1 --2- 192.xxx.xx4.1:0/2671 >> [v2:192.xxx.xx4.2:6819/2752,v1:192.xxx.xx4.2:6822/2752] conn(0x55bbac481f80 0x55bbac499180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6819/2752,v1:192.xxx.xx4.2:6822/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2676]: 2019-07-29 18:56:00.035 7f47d17bd700 -1 --2- 192.xxx.xx4.1:0/2676 >> [v2:192.xxx.xx4.2:6819/2752,v1:192.xxx.xx4.2:6822/2752] conn(0x557e6c038480 0x557e6c035080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6819/2752,v1:192.xxx.xx4.2:6822/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2680]: 2019-07-29 18:56:00.035 7fc265b05700 -1 --2- 192.xxx.xx4.1:0/2680 >> [v2:192.xxx.xx4.2:6819/2752,v1:192.xxx.xx4.2:6822/2752] conn(0x564249cbdf80 0x564249ce0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6819/2752,v1:192.xxx.xx4.2:6822/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2669]: 2019-07-29 18:56:00.099 7f443f415700 -1 --2- [v2:192.xxx.xx4.1:6805/2669,v1:192.xxx.xx4.1:6808/2669] >> [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] conn(0x561cf1f17b00 0x561cf1ed5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2666]: 2019-07-29 18:56:00.123 7fd5c3b76700 -1 --2- [v2:192.xxx.xx4.1:6816/2666,v1:192.xxx.xx4.1:6817/2666] >> [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] conn(0x562a88a9b680 0x562a88aa9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2676]: 2019-07-29 18:56:00.123 7f47d17bd700 -1 --2- [v2:192.xxx.xx4.1:6802/2676,v1:192.xxx.xx4.1:6803/2676] >> [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] conn(0x557e6c10b680 0x557e6c066100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2680]: 2019-07-29 18:56:00.123 7fc265304700 -1 --2- [v2:192.xxx.xx4.1:6807/2680,v1:192.xxx.xx4.1:6809/2680] >> [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] conn(0x564249d15680 0x564249d2bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] is using msgr V1 protocol

Jul 29 18:56:00 prox1a ceph-osd[2671]: 2019-07-29 18:56:00.127 7fd5ebfc8700 -1 --2- [v2:192.xxx.xx4.1:6800/2671,v1:192.xxx.xx4.1:6801/2671] >> [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] conn(0x55bbac483a80 0x55bbac4c4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] is using msgr V1 protocol

Jul 29 18:56:00 prox1a ceph-osd[2669]: 2019-07-29 18:56:00.435 7f443e413700 -1 --2- 192.xxx.xx3.1:0/2669 >> [v2:192.xxx.xx3.2:6819/2752,v1:192.xxx.xx3.2:6822/2752] conn(0x561cf1f6bf80 0x561cf1f75b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx3.2:6819/2752,v1:192.xxx.xx3.2:6822/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2669]: 2019-07-29 18:56:00.435 7f443ec14700 -1 --2- 192.xxx.xx4.1:0/2669 >> [v2:192.xxx.xx4.2:6819/2752,v1:192.xxx.xx4.2:6822/2752] conn(0x561cf1f6bb00 0x561cf1f75600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6819/2752,v1:192.xxx.xx4.2:6822/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2657]: 2019-07-29 18:56:00.507 7fe6caa76700 -1 --2-  >> [v2:192.xxx.xx4.2:6819/2752,v1:192.xxx.xx4.2:6822/2752] conn(0x559573bfc900 0x559573bfa100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6819/2752,v1:192.xxx.xx4.2:6822/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2657]: 2019-07-29 18:56:00.507 7fe6cba78700 -1 --2-  >> [v2:192.xxx.xx3.2:6819/2752,v1:192.xxx.xx3.2:6822/2752] conn(0x559573bfcd80 0x559573bfa680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx3.2:6819/2752,v1:192.xxx.xx3.2:6822/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2657]: 2019-07-29 18:56:00.507 7fe6cb277700 -1 --2- [v2:192.xxx.xx4.1:6820/2657,v1:192.xxx.xx4.1:6821/2657] >> [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] conn(0x559573cb7b00 0x559573cab600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2669]: 2019-07-29 18:56:00.635 7f443ec14700 -1 --2- 192.xxx.xx4.1:0/2669 >> [v2:192.xxx.xx4.2:6819/2752,v1:192.xxx.xx4.2:6822/2752] conn(0x561cf1f6bb00 0x561cf1f75600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6819/2752,v1:192.xxx.xx4.2:6822/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2669]: 2019-07-29 18:56:00.635 7f443e413700 -1 --2- 192.xxx.xx3.1:0/2669 >> [v2:192.xxx.xx3.2:6819/2752,v1:192.xxx.xx3.2:6822/2752] conn(0x561cf1f6bf80 0x561cf1f75b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx3.2:6819/2752,v1:192.xxx.xx3.2:6822/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2671]: 2019-07-29 18:56:00.635 7fd5ec7c9700 -1 --2- 192.xxx.xx3.1:0/2671 >> [v2:192.xxx.xx3.2:6819/2752,v1:192.xxx.xx3.2:6822/2752] conn(0x55bbac482400 0x55bbac499700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx3.2:6819/2752,v1:192.xxx.xx3.2:6822/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2676]: 2019-07-29 18:56:00.635 7f47d17bd700 -1 --2- 192.xxx.xx4.1:0/2676 >> [v2:192.xxx.xx4.2:6819/2752,v1:192.xxx.xx4.2:6822/2752] conn(0x557e6c038480 0x557e6c035080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6819/2752,v1:192.xxx.xx4.2:6822/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2680]: 2019-07-29 18:56:00.635 7fc265304700 -1 --2- 192.xxx.xx3.1:0/2680 >> [v2:192.xxx.xx3.2:6819/2752,v1:192.xxx.xx3.2:6822/2752] conn(0x564249cbe400 0x564249ce0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx3.2:6819/2752,v1:192.xxx.xx3.2:6822/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2666]: 2019-07-29 18:56:00.635 7fd5c3b76700 -1 --2- 192.xxx.xx3.1:0/2666 >> [v2:192.xxx.xx3.2:6819/2752,v1:192.xxx.xx3.2:6822/2752] conn(0x562a88a4c900 0x562a88a31700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx3.2:6819/2752,v1:192.xxx.xx3.2:6822/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2666]: 2019-07-29 18:56:00.635 7fd5c4377700 -1 --2- 192.xxx.xx4.1:0/2666 >> [v2:192.xxx.xx4.2:6819/2752,v1:192.xxx.xx4.2:6822/2752] conn(0x562a88a4c480 0x562a88a31180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6819/2752,v1:192.xxx.xx4.2:6822/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2680]: 2019-07-29 18:56:00.635 7fc265b05700 -1 --2- 192.xxx.xx4.1:0/2680 >> [v2:192.xxx.xx4.2:6819/2752,v1:192.xxx.xx4.2:6822/2752] conn(0x564249cbdf80 0x564249ce0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6819/2752,v1:192.xxx.xx4.2:6822/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2671]: 2019-07-29 18:56:00.635 7fd5ecfca700 -1 --2- 192.xxx.xx4.1:0/2671 >> [v2:192.xxx.xx4.2:6819/2752,v1:192.xxx.xx4.2:6822/2752] conn(0x55bbac481f80 0x55bbac499180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6819/2752,v1:192.xxx.xx4.2:6822/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2676]: 2019-07-29 18:56:00.635 7f47d0fbc700 -1 --2- 192.xxx.xx3.1:0/2676 >> [v2:192.xxx.xx3.2:6819/2752,v1:192.xxx.xx3.2:6822/2752] conn(0x557e6c038900 0x557e6c035600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx3.2:6819/2752,v1:192.xxx.xx3.2:6822/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2657]: 2019-07-29 18:56:00.707 7fe6cba78700 -1 --2- 192.xxx.xx3.1:0/2657 >> [v2:192.xxx.xx3.2:6819/2752,v1:192.xxx.xx3.2:6822/2752] conn(0x559573bfcd80 0x559573bfa680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx3.2:6819/2752,v1:192.xxx.xx3.2:6822/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2657]: 2019-07-29 18:56:00.707 7fe6caa76700 -1 --2- 192.xxx.xx4.1:0/2657 >> [v2:192.xxx.xx4.2:6819/2752,v1:192.xxx.xx4.2:6822/2752] conn(0x559573bfc900 0x559573bfa100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6819/2752,v1:192.xxx.xx4.2:6822/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2657]: 2019-07-29 18:56:00.711 7fe6cb277700 -1 --2- [v2:192.xxx.xx4.1:6820/2657,v1:192.xxx.xx4.1:6821/2657] >> [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] conn(0x559573cb7b00 0x559573cab600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] is using msgr V1 protocol
Jul 29 18:56:00 prox1a ceph-osd[2669]: 2019-07-29 18:56:00.899 7f443f415700 -1 --2- [v2:192.xxx.xx4.1:6805/2669,v1:192.xxx.xx4.1:6808/2669] >> [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] conn(0x561cf1f17b00 0x561cf1ed5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] is using msgr V1 protocol
Jul 29 18:56:01 prox1a ceph-osd[2669]: 2019-07-29 18:56:01.035 7f443e413700 -1 --2- 192.xxx.xx3.1:0/2669 >> [v2:192.xxx.xx3.2:6819/2752,v1:192.xxx.xx3.2:6822/2752] conn(0x561cf1f6bf80 0x561cf1f75b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx3.2:6819/2752,v1:192.xxx.xx3.2:6822/2752] is using msgr V1 protocol
Jul 29 18:56:01 prox1a ceph-osd[2669]: 2019-07-29 18:56:01.035 7f443ec14700 -1 --2- 192.xxx.xx4.1:0/2669 >> [v2:192.xxx.xx4.2:6819/2752,v1:192.xxx.xx4.2:6822/2752] conn(0x561cf1f6bb00 0x561cf1f75600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6819/2752,v1:192.xxx.xx4.2:6822/2752] is using msgr V1 protocol
Jul 29 18:56:01 prox1a ceph-osd[2657]: 2019-07-29 18:56:01.107 7fe6cba78700 -1 --2- 192.xxx.xx3.1:0/2657 >> [v2:192.xxx.xx3.2:6819/2752,v1:192.xxx.xx3.2:6822/2752] conn(0x559573bfcd80 0x559573bfa680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx3.2:6819/2752,v1:192.xxx.xx3.2:6822/2752] is using msgr V1 protocol
Jul 29 18:56:01 prox1a ceph-osd[2657]: 2019-07-29 18:56:01.107 7fe6caa76700 -1 --2- 192.xxx.xx4.1:0/2657 >> [v2:192.xxx.xx4.2:6819/2752,v1:192.xxx.xx4.2:6822/2752] conn(0x559573bfc900 0x559573bfa100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6819/2752,v1:192.xxx.xx4.2:6822/2752] is using msgr V1 protocol
Jul 29 18:56:01 prox1a ceph-osd[2657]: 2019-07-29 18:56:01.111 7fe6cb277700 -1 --2- [v2:192.xxx.xx4.1:6820/2657,v1:192.xxx.xx4.1:6821/2657] >> [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] conn(0x559573cb7b00 0x559573cab600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner peer [v2:192.xxx.xx4.2:6813/2752,v1:192.xxx.xx4.2:6816/2752] is using msgr V1 protocol

Also available in: Atom PDF