Support #45347
closedSepia Lab Access Request
Added by Xiubo Li about 4 years ago. Updated almost 4 years ago.
0%
Description
1) Do you just need VPN access or will you also be running teuthology jobs?
VPN access
2) Desired Username:
xiubli
3) Alternate e-mail address(es) we can reach you at:
xiubli@redhat.com
4) If you don't already have an established history of code contributions to Ceph, is there an existing community or core developer you've worked with who has reviewed your work and can vouch for your access request?
If you answered "No" to # 4, please answer the following (paste directly below the question to keep indentation):
4a) Paste a link to a Blueprint or planning doc of yours that was reviewed at a Ceph Developer Monthly.
4b) Paste a link to an accepted pull request for a major patch or feature.
4c) If applicable, include a link to the current project (planning doc, dev branch, or pull request) that you are looking to test.
5) Paste your SSH public key(s) between the pre tags
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkS3qZAsTQMObkUS9QBEpYlL7A3/4ypeRiVQil7UXrZkXVhYd8RAlvSBomOHtslz6OMuvm1GtP53apQQbLb2ziJXlogua/2ju17iyBP8KfyGUzx45S3XF6GaSi3C4gyviQ0hLhgg0HSaO92qw7lbjf731hXYBWXDwkXz/6wbYecykKHTM2xeIhTqCSDIx+fdrHmYW8tXp5ZIGxmCXgLhyNLaMXlNrsGHvLixrnUb3abJebdi+diQxWn4fx2Q+WwoAnY1uwawyeFJzmLEs3qjFWmuYSKcDJ94gHHedXyDhZN1mZOs0a0E319jtXG88/SQrX3+/OypS92Hr0FF6PQwhT root@lxbceph0
6) Paste your hashed VPN credentials between the pre tags (Format: user@hostname 22CharacterSalt 65CharacterHashedPassword)
root@lxbceph0 9x6Xh0EZtdtgzsY03HPKUg 1fd387c9c9260813fedf2347c7d479a0037e1027cd27e70fd67082182fe27b1f
Updated by Xiubo Li about 4 years ago
5) Paste your SSH public key(s) between the pre tags
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkS3qZAsTQMObkUS9QBEpYlL7A3/4ypeRiVQil7UXrZkXVhYd8RAlvSBomOHtslz6OMuvm1GtP53apQQbLb2ziJXlogua/2ju17iyBP8KfyGUzx45S3XF6GaSi3C4gyviQ0hLhgg0HSaO92qw7lbjf731hXYBWXDwkXz/6wbYecykKHTM2xeIhTqCSDIx+fdrHmYW8tXp5ZIGxmCXgLhyNLaMXlNrsGHvLixrnUb3abJebdi+diQxWn4fx2Q+WwoAnY1uwawyeFJzmLEs3qjFWmuYSKcDJ94gHHedXyDhZN1mZOs0a0E319jtXG88/SQrX3+/OypS92Hr0FF6PQwhT root@lxbceph0
6) Paste your hashed VPN credentials between the pre tags (Format: user@hostname 22CharacterSalt 65CharacterHashedPassword)
root@lxbceph0 9x6Xh0EZtdtgzsY03HPKUg 1fd387c9c9260813fedf2347c7d479a0037e1027cd27e70fd67082182fe27b1f
Updated by adam kraitman about 4 years ago
- Category set to User access
- Status changed from New to In Progress
- Assignee set to adam kraitman
Updated by adam kraitman about 4 years ago
Hey Xiubo Li, Is this replacing the old credentials you have - https://tracker.ceph.com/issues/43518
Or it's an additional credentials you need ?
Cheers
Adam
Updated by Xiubo Li about 4 years ago
adam kraitman wrote:
Hey Xiubo Li, Is this replacing the old credentials you have - https://tracker.ceph.com/issues/43518
Or it's an additional credentials you need ?
An addtional credentials.
Thanks very much.
BRs
Cheers
Adam
Updated by adam kraitman about 4 years ago
Hi Xiubo Li,
You should have access to the Sepia lab now. Please verify you're able to connect to the vpn and ssh xiubli@teuthology.front.sepia.ceph.com using the private key matching the pubkey you provided.
Be sure to check out the following links for final workstation setup steps:
https://wiki.sepia.ceph.com/doku.php?id=vpnaccess#vpn_client_access
https://wiki.sepia.ceph.com/doku.php?id=testnodeaccess#ssh_config
Most developers choose to schedule runs from the shared teuthology VM. For information on that, see http://docs.ceph.com/teuthology/docs/intro_testers.html
Thanks.
Updated by Xiubo Li about 4 years ago
Hi Adam
I am getting:
[root@lxbceph0 ~]# ssh xiubli@teuthology.front.sepia.ceph.com
ssh: connect to host teuthology.front.sepia.ceph.com port 22: Connection timed out
[root@lxbceph0 ~]#
[root@lxbceph0 build]# systemctl status openvpn-client@sepia.service
● openvpn-client@sepia.service - OpenVPN tunnel for sepia
Loaded: loaded (/usr/lib/systemd/system/openvpn-client@.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2020-04-30 19:10:20 EDT; 10min ago
Docs: man:openvpn(8)
https://community.openvpn.net/openvpn/wiki/Openvpn24ManPage
https://community.openvpn.net/openvpn/wiki/HOWTO
Main PID: 1191 (openvpn)
Status: "Pre-connection initialization successful"
CGroup: /system.slice/system-openvpn\x2dclient.slice/openvpn-client@sepia.service
└─1191 /usr/sbin/openvpn --suppress-timestamps --nobind --config sepia.conf
Apr 30 19:17:12 lxbceph0 openvpn[1191]: UDP link remote: [AF_INET]8.43.84.129:1194
Apr 30 19:18:12 lxbceph0 openvpn[1191]: TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Apr 30 19:18:12 lxbceph0 openvpn[1191]: TLS Error: TLS handshake failed
Apr 30 19:18:12 lxbceph0 openvpn[1191]: SIGUSR1[soft,tls-error] received, process restarting
Apr 30 19:18:52 lxbceph0 openvpn[1191]: TCP/UDP: Preserving recently used remote address: [AF_INET]8.43.84.129:1194
Apr 30 19:18:52 lxbceph0 openvpn[1191]: UDP link local: (not bound)
Apr 30 19:18:52 lxbceph0 openvpn[1191]: UDP link remote: [AF_INET]8.43.84.129:1194
Apr 30 19:19:52 lxbceph0 openvpn[1191]: TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Apr 30 19:19:52 lxbceph0 openvpn[1191]: TLS Error: TLS handshake failed
Apr 30 19:19:52 lxbceph0 openvpn[1191]: SIGUSR1[soft,tls-error] received, process restarting
Hint: Some lines were ellipsized, use -l to show in full.
[root@lxbceph0 build]#
For the old one also get the same error suddenly.
Thanks
Updated by adam kraitman about 4 years ago
Hey Please run:
rm -rf /etc/openvpn/*sepia* /etc/openvpn-client/*sepia*
And do the process again https://wiki.sepia.ceph.com/doku.php?id=vpnaccess
then paste your new hashed VPN credentials
Thanks
Updated by Xiubo Li almost 4 years ago
Hi Adam,
Please see:
6) Paste your hashed VPN credentials between the pre tags (Format: user@hostname 22CharacterSalt 65CharacterHashedPassword)
xiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3
Thanks.
Updated by adam kraitman almost 4 years ago
You should have access to the Sepia lab now
Updated by Xiubo Li almost 4 years ago
Hi Adam,
It seems still the same.
[root@lxbceph0 ceph]# ssh xiubli@teuthology.front.sepia.ceph.com -vvvv
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: Reading configuration data /root/.ssh/config
debug1: /root/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug2: resolving "teuthology.front.sepia.ceph.com" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to teuthology.front.sepia.ceph.com [172.21.0.51] port 22.
debug1: connect to address 172.21.0.51 port 22: Connection timed out
ssh: connect to host teuthology.front.sepia.ceph.com port 22: Connection timed out
[root@lxbceph0 ceph]#
[root@lxbceph0 ceph]# systemctl status openvpn-client@sepia.service
● openvpn-client@sepia.service - OpenVPN tunnel for sepia
Loaded: loaded (/usr/lib/systemd/system/openvpn-client@.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2020-05-05 06:43:24 EDT; 18h ago
Docs: man:openvpn(8)
https://community.openvpn.net/openvpn/wiki/Openvpn24ManPage
https://community.openvpn.net/openvpn/wiki/HOWTO
Main PID: 2995 (openvpn)
Status: "Pre-connection initialization successful"
CGroup: /system.slice/system-openvpn\x2dclient.slice/openvpn-client@sepia.service
└─2995 /usr/sbin/openvpn --suppress-timestamps --nobind --config sepia.conf
May 06 01:11:00 lxbceph0 openvpn[2995]: UDP link remote: [AF_INET]8.43.84.129:1194
May 06 01:12:00 lxbceph0 openvpn[2995]: TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
May 06 01:12:00 lxbceph0 openvpn[2995]: TLS Error: TLS handshake failed
May 06 01:12:00 lxbceph0 openvpn[2995]: SIGUSR1[soft,tls-error] received, process restarting
May 06 01:17:00 lxbceph0 openvpn[2995]: TCP/UDP: Preserving recently used remote address: [AF_INET]8.43.84.129:1194
May 06 01:17:00 lxbceph0 openvpn[2995]: UDP link local: (not bound)
May 06 01:17:00 lxbceph0 openvpn[2995]: UDP link remote: [AF_INET]8.43.84.129:1194
May 06 01:18:00 lxbceph0 openvpn[2995]: TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
May 06 01:18:00 lxbceph0 openvpn[2995]: TLS Error: TLS handshake failed
May 06 01:18:00 lxbceph0 openvpn[2995]: SIGUSR1[soft,tls-error] received, process restarting
Hint: Some lines were ellipsized, use -l to show in full.
[root@lxbceph0 ceph]#
Thanks
Updated by adam kraitman almost 4 years ago
Please do the Troubleshooting steps under https://wiki.sepia.ceph.com/doku.php?id=vpnaccess#linux
You can paste the output you see there from the openvpn command if you have question
Updated by Xiubo Li almost 4 years ago
adam kraitman wrote:
Please do the Troubleshooting steps under https://wiki.sepia.ceph.com/doku.php?id=vpnaccess#linux
You can paste the output you see there from the openvpn command if you have question
Hi Adam,
[root@lxbceph0 ceph-client]# systemctl restart openvpn-client@sepia.service
[root@lxbceph0 ceph-client]# openvpn --config /etc/openvpn/client/sepia.conf --cd /etc/openvpn/client --verb 5
Wed May 6 20:43:35 2020 us=237798 Current Parameter Settings:
Wed May 6 20:43:35 2020 us=238055 config = '/etc/openvpn/client/sepia.conf'
Wed May 6 20:43:35 2020 us=238107 mode = 0
Wed May 6 20:43:35 2020 us=238137 persist_config = DISABLED
Wed May 6 20:43:35 2020 us=239561 persist_mode = 1
Wed May 6 20:43:35 2020 us=239590 show_ciphers = DISABLED
Wed May 6 20:43:35 2020 us=239642 show_digests = DISABLED
Wed May 6 20:43:35 2020 us=239681 show_engines = DISABLED
Wed May 6 20:43:35 2020 us=239716 genkey = DISABLED
Wed May 6 20:43:35 2020 us=239789 key_pass_file = '[UNDEF]'
Wed May 6 20:43:35 2020 us=239843 NOTE: --mute triggered...
Wed May 6 20:43:35 2020 us=239910 273 variation(s) on previous 10 message(s) suppressed by --mute
Wed May 6 20:43:35 2020 us=239980 OpenVPN 2.4.8 x86_64-redhat-linux-gnu [Fedora EPEL patched] [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Nov 1 2019
Wed May 6 20:43:35 2020 us=240041 library versions: OpenSSL 1.0.2k-fips 26 Jan 2017, LZO 2.06
Wed May 6 20:43:35 2020 us=241242 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Wed May 6 20:43:35 2020 us=241315 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Wed May 6 20:43:35 2020 us=241404 LZO compression initializing
Wed May 6 20:43:35 2020 us=241557 Control Channel MTU parms [ L:1622 D:1184 EF:66 EB:0 ET:0 EL:3 ]
Wed May 6 20:43:35 2020 us=247032 Data Channel MTU parms [ L:1622 D:1450 EF:122 EB:406 ET:0 EL:3 ]
Wed May 6 20:43:35 2020 us=247263 Local Options String (VER=V4): 'V4,dev-type tun,link-mtu 1542,tun-mtu 1500,proto UDPv4,comp-lzo,keydir 1,cipher BF-CBC,auth SHA1,keysize 128,tls-auth,key-method 2,tls-client'
Wed May 6 20:43:35 2020 us=247346 Expected Remote Options String (VER=V4): 'V4,dev-type tun,link-mtu 1542,tun-mtu 1500,proto UDPv4,comp-lzo,keydir 0,cipher BF-CBC,auth SHA1,keysize 128,tls-auth,key-method 2,tls-server'
Wed May 6 20:43:35 2020 us=250979 TCP/UDP: Preserving recently used remote address: [AF_INET]8.43.84.129:1194
Wed May 6 20:43:35 2020 us=251646 Socket Buffers: R=[212992->212992] S=[212992->212992]
Wed May 6 20:43:35 2020 us=251780 UDP link local: (not bound)
Wed May 6 20:43:35 2020 us=251853 UDP link remote: [AF_INET]8.43.84.129:1194
Wed May 6 20:43:35 2020 us=251944 NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
WWWWWWed May 6 20:44:36 2020 us=11398 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Wed May 6 20:44:36 2020 us=11528 TLS Error: TLS handshake failed
Wed May 6 20:44:36 2020 us=11859 TCP/UDP: Closing socket
Wed May 6 20:44:36 2020 us=11999 SIGUSR1[soft,tls-error] received, process restarting
Wed May 6 20:44:36 2020 us=12094 Restart pause, 5 second(s)
Wed May 6 20:44:41 2020 us=12297 Re-using SSL/TLS context
Wed May 6 20:44:41 2020 us=12385 LZO compression initializing
Wed May 6 20:44:41 2020 us=12536 Control Channel MTU parms [ L:1622 D:1184 EF:66 EB:0 ET:0 EL:3 ]
Wed May 6 20:44:41 2020 us=14234 Data Channel MTU parms [ L:1622 D:1450 EF:122 EB:406 ET:0 EL:3 ]
Wed May 6 20:44:41 2020 us=14341 Local Options String (VER=V4): 'V4,dev-type tun,link-mtu 1542,tun-mtu 1500,proto UDPv4,comp-lzo,keydir 1,cipher BF-CBC,auth SHA1,keysize 128,tls-auth,key-method 2,tls-client'
Wed May 6 20:44:41 2020 us=14399 Expected Remote Options String (VER=V4): 'V4,dev-type tun,link-mtu 1542,tun-mtu 1500,proto UDPv4,comp-lzo,keydir 0,cipher BF-CBC,auth SHA1,keysize 128,tls-auth,key-method 2,tls-server'
Wed May 6 20:44:41 2020 us=14456 TCP/UDP: Preserving recently used remote address: [AF_INET]8.43.84.129:1194
Wed May 6 20:44:41 2020 us=14845 Socket Buffers: R=[212992->212992] S=[212992->212992]
Wed May 6 20:44:41 2020 us=14926 UDP link local: (not bound)
Wed May 6 20:44:41 2020 us=14979 UDP link remote: [AF_INET]8.43.84.129:1194
[root@lxbceph0 client]# ping 8.43.84.129 -c 3
PING 8.43.84.129 (8.43.84.129) 56(84) bytes of data.
64 bytes from 8.43.84.129: icmp_seq=1 ttl=33 time=236 ms
64 bytes from 8.43.84.129: icmp_seq=2 ttl=33 time=236 ms
64 bytes from 8.43.84.129: icmp_seq=3 ttl=33 time=236 ms
--- 8.43.84.129 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 236.523/236.655/236.838/0.419 ms
[root@lxbceph0 client]#
It is still failed when doing the TLS key negotiation ? Both my two VMs have the same issue as above, and for one of them worked well.
Thanks
Updated by adam kraitman almost 4 years ago
It seems to me that maybe you have a network connectivity issue on your side it could be a firewall or selinux on your side that is blocking you from creating the tcp connection
Updated by Xiubo Li almost 4 years ago
adam kraitman wrote:
It seems to me that maybe you have a network connectivity issue on your side it could be a firewall or selinux on your side that is blocking you from creating the tcp connection
Both the firewall and selinux are disabled locally.
[root@lxbceph0 ~]# getenforce
Disabled
[root@lxbceph0 ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
[root@lxbceph0 ~]#
I am thinking what's the reason for my old node, on which I didn't touch anything except to ssh to the sepia and suddenly disconnected.
Thanks
Updated by adam kraitman almost 4 years ago
It could also be some other network device that is blocking you since you say that also you old node is not connecting any more maybe it's a good idea to open a ticket in the IT about it and they could check what is blocking your nodes
Updated by Xiubo Li almost 4 years ago
Hi Adam,
Have checked this with IT guys, they didn't see any network issue for these two nodes.
The old node is a local VM on my laptop, and the new one is running in RedHat Virtualization from remote. And both are hitting the same issue. Is it possible that there has something or setting blocking them from the Sepia VM ?
Thanks
BRs
Updated by David Galloway almost 4 years ago
Here's what I see on the VPN server. You have two credentials.
Apr 22 14:16:35 gw openvpn: ERROR:auth-openvpn:auth-openvpn: invalid auth for user 'root@lxbceph0'. Apr 22 14:16:35 gw openvpn: ERROR:auth-openvpn:User not found: 'root@lxbceph0'
Apr 25 00:35:41 gw openvpn: Sat Apr 25 00:35:41 2020 us=188683 xiubli@fedora1/112.23.104.XXX:1053 TLS: Username/Password authentication succeeded for username 'xiubli@fedora1' [CN SET]
Those are the last two connection attempts I see for either credential. The credential on the Sepia server for the first one is xiubli@lxbceph0, however. If you changed the username, you must re-run the new-client script and give us the new output. You can't just change the secret text file on your machine.
Are you in China? My understanding is the Chinese government blocks outgoing VPN connections. Could that be related? I believe Red Hat gets around this by having VPN servers listen on port 443 (HTTPS) instead of the OpenVPN default 1194 but that's not feasible for us.
Updated by Xiubo Li almost 4 years ago
Hi David,
Thanks very much for your info.
The xiubli@fedora1 is an old node I am using, and xiubli@lxbceph0 is the new node which Adam helped me add the new credential recently in May.
@Apr 22 14:16:35, I just copied the config from xiubli@fedora1 node to xiubli@lxbceph0 and tried it, didn't touch anything on xiubli@fedora1 node. But @Apr 25, the ssh connection suddenly got lost on xiubli@fedora1 node and this month I have tried many times and still couldn't success. If you didn't see any connection attempts after Apr 25, it might be the packages didn't reach the openvpn server ever ?
The odd thing is that Kefu doesn't hit any issue about this, we are all in Shanghai. So odd...
BRs
Xiubo
Updated by Xiubo Li almost 4 years ago
Hey Adam,
Please help remove the previous old ones and add the following new one for me, let's see will it could work. This is another new VM in Beijing office.
5) Paste your SSH public key(s) between the pre tags
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0
6) Paste your hashed VPN credentials between the pre tags (Format: user@hostname 22CharacterSalt 65CharacterHashedPassword)
root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3
Thanks,
BRs
Updated by adam kraitman almost 4 years ago
Hi Xiubo Li,
You should have access to the Sepia lab now. Please verify you're able to connect to the vpn and ssh xiubli@teuthology.front.sepia.ceph.com using the private key matching the pubkey you provided.
Be sure to check out the following links for final workstation setup steps:
https://wiki.sepia.ceph.com/doku.php?id=vpnaccess#vpn_client_access
https://wiki.sepia.ceph.com/doku.php?id=testnodeaccess#ssh_config
Most developers choose to schedule runs from the shared teuthology VM. For information on that, see http://docs.ceph.com/teuthology/docs/intro_testers.html
Thanks.
Updated by Xiubo Li almost 4 years ago
- Status changed from In Progress to Resolved
It works for me now. Thanks very much @Adam DC949.