Infrastructure » sepia

5) Paste your SSH public key(s) between the pre tags

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkS3qZAsTQMObkUS9QBEpYlL7A3/4ypeRiVQil7UXrZkXVhYd8RAlvSBomOHtslz6OMuvm1GtP53apQQbLb2ziJXlogua/2ju17iyBP8KfyGUzx45S3XF6GaSi3C4gyviQ0hLhgg0HSaO92qw7lbjf731hXYBWXDwkXz/6wbYecykKHTM2xeIhTqCSDIx+fdrHmYW8tXp5ZIGxmCXgLhyNLaMXlNrsGHvLixrnUb3abJebdi+diQxWn4fx2Q+WwoAnY1uwawyeFJzmLEs3qjFWmuYSKcDJ94gHHedXyDhZN1mZOs0a0E319jtXG88/SQrX3+/OypS92Hr0FF6PQwhT root@lxbceph0

6) Paste your hashed VPN credentials between the pre tags (Format: user@hostname 22CharacterSalt 65CharacterHashedPassword)

root@lxbceph0 9x6Xh0EZtdtgzsY03HPKUg 1fd387c9c9260813fedf2347c7d479a0037e1027cd27e70fd67082182fe27b1f

Hey Xiubo Li, Is this replacing the old credentials you have - https://tracker.ceph.com/issues/43518
Or it's an additional credentials you need ?

Cheers

Adam

adam kraitman wrote:

Hey Xiubo Li, Is this replacing the old credentials you have - https://tracker.ceph.com/issues/43518
Or it's an additional credentials you need ?

An addtional credentials.

Thanks very much.
BRs

Cheers

Adam

Hi Xiubo Li,

You should have access to the Sepia lab now. Please verify you're able to connect to the vpn and ssh xiubli@teuthology.front.sepia.ceph.com using the private key matching the pubkey you provided.

Be sure to check out the following links for final workstation setup steps:
https://wiki.sepia.ceph.com/doku.php?id=vpnaccess#vpn_client_access
https://wiki.sepia.ceph.com/doku.php?id=testnodeaccess#ssh_config

Most developers choose to schedule runs from the shared teuthology VM. For information on that, see http://docs.ceph.com/teuthology/docs/intro_testers.html

Thanks.

Hi Adam

I am getting:

[root@lxbceph0 ~]# ssh xiubli@teuthology.front.sepia.ceph.com
ssh: connect to host teuthology.front.sepia.ceph.com port 22: Connection timed out
[root@lxbceph0 ~]#

[root@lxbceph0 build]# systemctl status openvpn-client@sepia.service
● openvpn-client@sepia.service - OpenVPN tunnel for sepia
   Loaded: loaded (/usr/lib/systemd/system/openvpn-client@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2020-04-30 19:10:20 EDT; 10min ago
     Docs: man:openvpn(8)
           https://community.openvpn.net/openvpn/wiki/Openvpn24ManPage
           https://community.openvpn.net/openvpn/wiki/HOWTO
 Main PID: 1191 (openvpn)
   Status: "Pre-connection initialization successful" 
   CGroup: /system.slice/system-openvpn\x2dclient.slice/openvpn-client@sepia.service
           └─1191 /usr/sbin/openvpn --suppress-timestamps --nobind --config sepia.conf

Apr 30 19:17:12 lxbceph0 openvpn[1191]: UDP link remote: [AF_INET]8.43.84.129:1194
Apr 30 19:18:12 lxbceph0 openvpn[1191]: TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Apr 30 19:18:12 lxbceph0 openvpn[1191]: TLS Error: TLS handshake failed
Apr 30 19:18:12 lxbceph0 openvpn[1191]: SIGUSR1[soft,tls-error] received, process restarting
Apr 30 19:18:52 lxbceph0 openvpn[1191]: TCP/UDP: Preserving recently used remote address: [AF_INET]8.43.84.129:1194
Apr 30 19:18:52 lxbceph0 openvpn[1191]: UDP link local: (not bound)
Apr 30 19:18:52 lxbceph0 openvpn[1191]: UDP link remote: [AF_INET]8.43.84.129:1194
Apr 30 19:19:52 lxbceph0 openvpn[1191]: TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Apr 30 19:19:52 lxbceph0 openvpn[1191]: TLS Error: TLS handshake failed
Apr 30 19:19:52 lxbceph0 openvpn[1191]: SIGUSR1[soft,tls-error] received, process restarting
Hint: Some lines were ellipsized, use -l to show in full.
[root@lxbceph0 build]#

For the old one also get the same error suddenly.

Thanks

Hey Please run:
rm -rf /etc/openvpn/*sepia* /etc/openvpn-client/*sepia*

And do the process again https://wiki.sepia.ceph.com/doku.php?id=vpnaccess
then paste your new hashed VPN credentials

Thanks

Hi Adam,

Please see:

6) Paste your hashed VPN credentials between the pre tags (Format: user@hostname 22CharacterSalt 65CharacterHashedPassword)

xiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3

Thanks.

You should have access to the Sepia lab now

Hi Adam,

It seems still the same.

[root@lxbceph0 ceph]# ssh xiubli@teuthology.front.sepia.ceph.com -vvvv
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017
debug1: Reading configuration data /root/.ssh/config
debug1: /root/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug2: resolving "teuthology.front.sepia.ceph.com" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to teuthology.front.sepia.ceph.com [172.21.0.51] port 22.
debug1: connect to address 172.21.0.51 port 22: Connection timed out
ssh: connect to host teuthology.front.sepia.ceph.com port 22: Connection timed out
[root@lxbceph0 ceph]# 

[root@lxbceph0 ceph]# systemctl status openvpn-client@sepia.service
● openvpn-client@sepia.service - OpenVPN tunnel for sepia
   Loaded: loaded (/usr/lib/systemd/system/openvpn-client@.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2020-05-05 06:43:24 EDT; 18h ago
     Docs: man:openvpn(8)
           https://community.openvpn.net/openvpn/wiki/Openvpn24ManPage
           https://community.openvpn.net/openvpn/wiki/HOWTO
 Main PID: 2995 (openvpn)
   Status: "Pre-connection initialization successful" 
   CGroup: /system.slice/system-openvpn\x2dclient.slice/openvpn-client@sepia.service
           └─2995 /usr/sbin/openvpn --suppress-timestamps --nobind --config sepia.conf

May 06 01:11:00 lxbceph0 openvpn[2995]: UDP link remote: [AF_INET]8.43.84.129:1194
May 06 01:12:00 lxbceph0 openvpn[2995]: TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
May 06 01:12:00 lxbceph0 openvpn[2995]: TLS Error: TLS handshake failed
May 06 01:12:00 lxbceph0 openvpn[2995]: SIGUSR1[soft,tls-error] received, process restarting
May 06 01:17:00 lxbceph0 openvpn[2995]: TCP/UDP: Preserving recently used remote address: [AF_INET]8.43.84.129:1194
May 06 01:17:00 lxbceph0 openvpn[2995]: UDP link local: (not bound)
May 06 01:17:00 lxbceph0 openvpn[2995]: UDP link remote: [AF_INET]8.43.84.129:1194
May 06 01:18:00 lxbceph0 openvpn[2995]: TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
May 06 01:18:00 lxbceph0 openvpn[2995]: TLS Error: TLS handshake failed
May 06 01:18:00 lxbceph0 openvpn[2995]: SIGUSR1[soft,tls-error] received, process restarting
Hint: Some lines were ellipsized, use -l to show in full.
[root@lxbceph0 ceph]#

Thanks

Please do the Troubleshooting steps under https://wiki.sepia.ceph.com/doku.php?id=vpnaccess#linux
You can paste the output you see there from the openvpn command if you have question

adam kraitman wrote:

Please do the Troubleshooting steps under https://wiki.sepia.ceph.com/doku.php?id=vpnaccess#linux
You can paste the output you see there from the openvpn command if you have question

Hi Adam,

[root@lxbceph0 ceph-client]# systemctl restart openvpn-client@sepia.service
[root@lxbceph0 ceph-client]# openvpn --config /etc/openvpn/client/sepia.conf --cd /etc/openvpn/client --verb 5
Wed May  6 20:43:35 2020 us=237798 Current Parameter Settings:
Wed May  6 20:43:35 2020 us=238055   config = '/etc/openvpn/client/sepia.conf'
Wed May  6 20:43:35 2020 us=238107   mode = 0
Wed May  6 20:43:35 2020 us=238137   persist_config = DISABLED
Wed May  6 20:43:35 2020 us=239561   persist_mode = 1
Wed May  6 20:43:35 2020 us=239590   show_ciphers = DISABLED
Wed May  6 20:43:35 2020 us=239642   show_digests = DISABLED
Wed May  6 20:43:35 2020 us=239681   show_engines = DISABLED
Wed May  6 20:43:35 2020 us=239716   genkey = DISABLED
Wed May  6 20:43:35 2020 us=239789   key_pass_file = '[UNDEF]'
Wed May  6 20:43:35 2020 us=239843 NOTE: --mute triggered...
Wed May  6 20:43:35 2020 us=239910 273 variation(s) on previous 10 message(s) suppressed by --mute
Wed May  6 20:43:35 2020 us=239980 OpenVPN 2.4.8 x86_64-redhat-linux-gnu [Fedora EPEL patched] [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Nov  1 2019
Wed May  6 20:43:35 2020 us=240041 library versions: OpenSSL 1.0.2k-fips  26 Jan 2017, LZO 2.06
Wed May  6 20:43:35 2020 us=241242 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Wed May  6 20:43:35 2020 us=241315 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Wed May  6 20:43:35 2020 us=241404 LZO compression initializing
Wed May  6 20:43:35 2020 us=241557 Control Channel MTU parms [ L:1622 D:1184 EF:66 EB:0 ET:0 EL:3 ]
Wed May  6 20:43:35 2020 us=247032 Data Channel MTU parms [ L:1622 D:1450 EF:122 EB:406 ET:0 EL:3 ]
Wed May  6 20:43:35 2020 us=247263 Local Options String (VER=V4): 'V4,dev-type tun,link-mtu 1542,tun-mtu 1500,proto UDPv4,comp-lzo,keydir 1,cipher BF-CBC,auth SHA1,keysize 128,tls-auth,key-method 2,tls-client'
Wed May  6 20:43:35 2020 us=247346 Expected Remote Options String (VER=V4): 'V4,dev-type tun,link-mtu 1542,tun-mtu 1500,proto UDPv4,comp-lzo,keydir 0,cipher BF-CBC,auth SHA1,keysize 128,tls-auth,key-method 2,tls-server'
Wed May  6 20:43:35 2020 us=250979 TCP/UDP: Preserving recently used remote address: [AF_INET]8.43.84.129:1194
Wed May  6 20:43:35 2020 us=251646 Socket Buffers: R=[212992->212992] S=[212992->212992]
Wed May  6 20:43:35 2020 us=251780 UDP link local: (not bound)
Wed May  6 20:43:35 2020 us=251853 UDP link remote: [AF_INET]8.43.84.129:1194
Wed May  6 20:43:35 2020 us=251944 NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
WWWWWWed May  6 20:44:36 2020 us=11398 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Wed May  6 20:44:36 2020 us=11528 TLS Error: TLS handshake failed
Wed May  6 20:44:36 2020 us=11859 TCP/UDP: Closing socket
Wed May  6 20:44:36 2020 us=11999 SIGUSR1[soft,tls-error] received, process restarting
Wed May  6 20:44:36 2020 us=12094 Restart pause, 5 second(s)
Wed May  6 20:44:41 2020 us=12297 Re-using SSL/TLS context
Wed May  6 20:44:41 2020 us=12385 LZO compression initializing
Wed May  6 20:44:41 2020 us=12536 Control Channel MTU parms [ L:1622 D:1184 EF:66 EB:0 ET:0 EL:3 ]
Wed May  6 20:44:41 2020 us=14234 Data Channel MTU parms [ L:1622 D:1450 EF:122 EB:406 ET:0 EL:3 ]
Wed May  6 20:44:41 2020 us=14341 Local Options String (VER=V4): 'V4,dev-type tun,link-mtu 1542,tun-mtu 1500,proto UDPv4,comp-lzo,keydir 1,cipher BF-CBC,auth SHA1,keysize 128,tls-auth,key-method 2,tls-client'
Wed May  6 20:44:41 2020 us=14399 Expected Remote Options String (VER=V4): 'V4,dev-type tun,link-mtu 1542,tun-mtu 1500,proto UDPv4,comp-lzo,keydir 0,cipher BF-CBC,auth SHA1,keysize 128,tls-auth,key-method 2,tls-server'
Wed May  6 20:44:41 2020 us=14456 TCP/UDP: Preserving recently used remote address: [AF_INET]8.43.84.129:1194
Wed May  6 20:44:41 2020 us=14845 Socket Buffers: R=[212992->212992] S=[212992->212992]
Wed May  6 20:44:41 2020 us=14926 UDP link local: (not bound)
Wed May  6 20:44:41 2020 us=14979 UDP link remote: [AF_INET]8.43.84.129:1194

[root@lxbceph0 client]# ping 8.43.84.129 -c 3
PING 8.43.84.129 (8.43.84.129) 56(84) bytes of data.
64 bytes from 8.43.84.129: icmp_seq=1 ttl=33 time=236 ms
64 bytes from 8.43.84.129: icmp_seq=2 ttl=33 time=236 ms
64 bytes from 8.43.84.129: icmp_seq=3 ttl=33 time=236 ms

--- 8.43.84.129 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 236.523/236.655/236.838/0.419 ms
[root@lxbceph0 client]# 

It is still failed when doing the TLS key negotiation ? Both my two VMs have the same issue as above, and for one of them worked well.

Thanks

It seems to me that maybe you have a network connectivity issue on your side it could be a firewall or selinux on your side that is blocking you from creating the tcp connection

adam kraitman wrote:

It seems to me that maybe you have a network connectivity issue on your side it could be a firewall or selinux on your side that is blocking you from creating the tcp connection

Both the firewall and selinux are disabled locally.

[root@lxbceph0 ~]# getenforce
Disabled
[root@lxbceph0 ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
[root@lxbceph0 ~]#

I am thinking what's the reason for my old node, on which I didn't touch anything except to ssh to the sepia and suddenly disconnected.

Thanks

It could also be some other network device that is blocking you since you say that also you old node is not connecting any more maybe it's a good idea to open a ticket in the IT about it and they could check what is blocking your nodes

Sure, will do it. Thanks @Adam DC949.

Hi Adam,

Have checked this with IT guys, they didn't see any network issue for these two nodes.

The old node is a local VM on my laptop, and the new one is running in RedHat Virtualization from remote. And both are hitting the same issue. Is it possible that there has something or setting blocking them from the Sepia VM ?

Thanks
BRs

Here's what I see on the VPN server. You have two credentials.

Apr 22 14:16:35 gw openvpn: ERROR:auth-openvpn:auth-openvpn: invalid auth for user 'root@lxbceph0'.
Apr 22 14:16:35 gw openvpn: ERROR:auth-openvpn:User not found: 'root@lxbceph0'

Apr 25 00:35:41 gw openvpn: Sat Apr 25 00:35:41 2020 us=188683 xiubli@fedora1/112.23.104.XXX:1053 TLS: Username/Password authentication succeeded for username 'xiubli@fedora1' [CN SET]

Those are the last two connection attempts I see for either credential. The credential on the Sepia server for the first one is xiubli@lxbceph0, however. If you changed the username, you must re-run the new-client script and give us the new output. You can't just change the secret text file on your machine.

Are you in China? My understanding is the Chinese government blocks outgoing VPN connections. Could that be related? I believe Red Hat gets around this by having VPN servers listen on port 443 (HTTPS) instead of the OpenVPN default 1194 but that's not feasible for us.

Hi David,

Thanks very much for your info.

The xiubli@fedora1 is an old node I am using, and xiubli@lxbceph0 is the new node which Adam helped me add the new credential recently in May.

@Apr 22 14:16:35, I just copied the config from xiubli@fedora1 node to xiubli@lxbceph0 and tried it, didn't touch anything on xiubli@fedora1 node. But @Apr 25, the ssh connection suddenly got lost on xiubli@fedora1 node and this month I have tried many times and still couldn't success. If you didn't see any connection attempts after Apr 25, it might be the packages didn't reach the openvpn server ever ?

The odd thing is that Kefu doesn't hit any issue about this, we are all in Shanghai. So odd...

BRs
Xiubo

Hey Adam,

Please help remove the previous old ones and add the following new one for me, let's see will it could work. This is another new VM in Beijing office.

5) Paste your SSH public key(s) between the pre tags

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0

6) Paste your hashed VPN credentials between the pre tags (Format: user@hostname 22CharacterSalt 65CharacterHashedPassword)

root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3

Thanks,
BRs

Hi Xiubo Li,

You should have access to the Sepia lab now. Please verify you're able to connect to the vpn and ssh xiubli@teuthology.front.sepia.ceph.com using the private key matching the pubkey you provided.

Be sure to check out the following links for final workstation setup steps:
https://wiki.sepia.ceph.com/doku.php?id=vpnaccess#vpn_client_access
https://wiki.sepia.ceph.com/doku.php?id=testnodeaccess#ssh_config

Most developers choose to schedule runs from the shared teuthology VM. For information on that, see http://docs.ceph.com/teuthology/docs/intro_testers.html

Thanks.