Infrastructure » sepia

Does anyone handle this request?

Mer Xuanyi is there an existing community or core developer you've worked with who has reviewed your work and can vouch for your access request?

adam kraitman wrote:

Mer Xuanyi is there an existing community or core developer you've worked with who has reviewed your work and can vouch for your access request?

temporarily no, I've been worked with @Venky Shankar and @lxbsz, but I didn't talk to them before this request, so no vouch.

Okay I will add your username but can you please choose a username without a dot?

adam kraitman wrote:

Okay I will add your username but can you please choose a username without a dot?

Thank you, is "mer" ok?

Yes, I am adding it now

Hey Mer Xuanyi,

You should have access to the Sepia lab now. Please verify you're able to connect to the vpn and ssh mer@teuthology.front.sepia.ceph.com using the private key matching the pubkey you provided.

Be sure to check out the following links for final workstation setup steps:
https://wiki.sepia.ceph.com/doku.php?id=vpnaccess#vpn_client_access
https://wiki.sepia.ceph.com/doku.php?id=testnodeaccess#ssh_config

Most developers choose to schedule runs from the shared teuthology VM. For information on that, see http://docs.ceph.com/teuthology/docs/intro_testers.html

If you plan on scheduling tests, one of the options you'll need to set with teuthology-suite is -p, --priority. Please refrain from using a priority lower than 101 (lower number = higher priority). When a high priority is used, it locks up too many testnodes at once and prevents other developers from testing changes.

Thanks.

adam kraitman wrote:

Hey Mer Xuanyi,

You should have access to the Sepia lab now. Please verify you're able to connect to the vpn and ssh mer@teuthology.front.sepia.ceph.com using the private key matching the pubkey you provided.

Be sure to check out the following links for final workstation setup steps:
https://wiki.sepia.ceph.com/doku.php?id=vpnaccess#vpn_client_access
https://wiki.sepia.ceph.com/doku.php?id=testnodeaccess#ssh_config

Most developers choose to schedule runs from the shared teuthology VM. For information on that, see http://docs.ceph.com/teuthology/docs/intro_testers.html

If you plan on scheduling tests, one of the options you'll need to set with teuthology-suite is -p, --priority. Please refrain from using a priority lower than 101 (lower number = higher priority). When a high priority is used, it locks up too many testnodes at once and prevents other developers from testing changes.

Thanks.

I can't start openvpn-client@sepia serivce, maybe it's because the user changed before?

➜  ~ systemctl status openvpn-client@sepia.service
× openvpn-client@sepia.service - OpenVPN tunnel for sepia
     Loaded: loaded (/usr/lib/systemd/system/openvpn-client@.service; disabled; vendor preset: disabled)
     Active: failed (Result: exit-code) since Mon 2023-02-13 14:15:23 CST; 9s ago
       Docs: man:openvpn(8)
             https://community.openvpn.net/openvpn/wiki/Openvpn24ManPage
             https://community.openvpn.net/openvpn/wiki/HOWTO
    Process: 255672 ExecStart=/usr/sbin/openvpn --suppress-timestamps --nobind --config sepia.conf (code=exited,>
   Main PID: 255672 (code=exited, status=1/FAILURE)

Feb 13 14:15:23 DESKTOP-VVPDIRP systemd[1]: Starting OpenVPN tunnel for sepia...
Feb 13 14:15:23 DESKTOP-VVPDIRP openvpn[255672]: Options error: In [CMD-LINE]:1: Error opening configuration fil>
Feb 13 14:15:23 DESKTOP-VVPDIRP openvpn[255672]: Use --help for more information.
Feb 13 14:15:23 DESKTOP-VVPDIRP systemd[1]: openvpn-client@sepia.service: Main process exited, code=exited, stat>
Feb 13 14:15:23 DESKTOP-VVPDIRP systemd[1]: openvpn-client@sepia.service: Failed with result 'exit-code'.
Feb 13 14:15:23 DESKTOP-VVPDIRP systemd[1]: Failed to start OpenVPN tunnel for sepia.

the sepia secret is xuanyi.meng (I've tried change that to mer but still can't boot)

➜  ~ cat /etc/openvpn/sepia/secret
xuanyi.meng@xtaotech
...

Yes maybe it's the user change, please re-run the new-client script with the "mer" user and paste the output

adam kraitman wrote:

Yes maybe it's the user change, please re-run the new-client script with the "mer" user and paste the output

Here is:

➜  openvpn sepia/new-client mer@xtaotech

!!!!! DO NOT RUN THIS SCRIPT MORE THAN ONCE !!!!!

Please paste the following line in your Sepia Lab Access Request tracker ticket:

mer@xtaotech VUjwFfr6OGjDlF0L1Lrvcw 839b438aaeb36eddce541ccdc8a1a0d9934953b83425aaddfc15ec77f773af1e

Run this before you restart the OpenVPN service
sudo sed -i 's/nogroup/openvpn/g' /etc/openvpn/client/sepia/client.conf
and then try to ssh

I realized I've the different dir struct than you, like this:

➜  ~ tree /etc/openvpn
/etc/openvpn
├── client
├── sepia
│   ├── ca.crt
│   ├── client.conf
│   ├── new-client
│   ├── secret
│   └── tlsauth
├── sepia.conf -> sepia/client.conf
├── sepia-vpn-client.tar.gz
└── server
3 directories, 7 files

So I made a symbol link for sepia and now in /etc/openvpn:

➜  openvpn tree
.
├── client
│   ├── sepia -> ../sepia
│   └── sepia.conf -> ../sepia.conf
├── sepia
│   ├── ca.crt
│   ├── client.conf
│   ├── new-client
│   ├── secret
│   └── tlsauth
├── sepia.conf -> sepia/client.conf
├── sepia-vpn-client.tar.gz
└── server

The client.conf after execute sed:

script-security 1
client
remote vpn.sepia.ceph.com 1194
dev tun
remote-random
resolv-retry infinite
nobind
user nobody
group openvpn
persist-tun
persist-key
comp-lzo
verb 2
mute 10
remote-cert-tls server
tls-auth sepia/tlsauth 1
ca sepia/ca.crt
auth-user-pass sepia/secret

Now the openvpn service can boot successful (the manual on this page should be update https://ceph.github.io/sepia/adding_users/)

➜  ~ systemctl status openvpn-client@sepia
● openvpn-client@sepia.service - OpenVPN tunnel for sepia
     Loaded: loaded (/usr/lib/systemd/system/openvpn-client@.service; disabled; vendor preset: disabled)
     Active: active (running) since Wed 2023-02-15 11:05:28 CST; 15min ago
       Docs: man:openvpn(8)
             https://community.openvpn.net/openvpn/wiki/Openvpn24ManPage
             https://community.openvpn.net/openvpn/wiki/HOWTO
   Main PID: 347497 (openvpn)
     Status: "Pre-connection initialization successful" 
      Tasks: 1 (limit: 308305)
     Memory: 2.6M
     CGroup: /system.slice/system-openvpn\x2dclient.slice/openvpn-client@sepia.service
             └─347497 /usr/sbin/openvpn --suppress-timestamps --nobind --config sepia.conf

But ssh still failed:

➜  client ssh mer@teuthology.front.sepia.ceph.com
ssh: connect to host teuthology.front.sepia.ceph.com port 22: Connection timed out

journal of openvpn:

➜  ~ journalctl -u openvpn-client@sepia.service
Feb 15 11:05:28 DESKTOP-VVPDIRP systemd[1]: Started OpenVPN tunnel for sepia.
Feb 15 11:05:28 DESKTOP-VVPDIRP openvpn[347497]: Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Feb 15 11:05:28 DESKTOP-VVPDIRP openvpn[347497]: Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Feb 15 11:05:29 DESKTOP-VVPDIRP openvpn[347497]: TCP/UDP: Preserving recently used remote address: [AF_INET]8.43.84.129:1194
Feb 15 11:05:29 DESKTOP-VVPDIRP openvpn[347497]: UDP link local: (not bound)
Feb 15 11:05:29 DESKTOP-VVPDIRP openvpn[347497]: UDP link remote: [AF_INET]8.43.84.129:1194
Feb 15 11:05:29 DESKTOP-VVPDIRP openvpn[347497]: NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Feb 15 11:06:30 DESKTOP-VVPDIRP openvpn[347497]: TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Feb 15 11:06:30 DESKTOP-VVPDIRP openvpn[347497]: TLS Error: TLS handshake failed
Feb 15 11:06:30 DESKTOP-VVPDIRP openvpn[347497]: SIGUSR1[soft,tls-error] received, process restarting

Please change the "group openvpn" to "group nobody" in the client.conf and try to restart the service

adam kraitman wrote:

Please change the "group openvpn" to "group nobody" in the client.conf and try to restart the service

still timeout :(

Can you please paste the output of "ls -l" of both directories where the files exist? I want to see the permissions

adam kraitman wrote:

Can you please paste the output of "ls -l" of both directories where the files exist? I want to see the permissions

➜  openvpn ls -lhart
total 24K
-rw-r--r--   1 root root    2.8K Oct 28  2020 sepia-vpn-client.tar.gz
drwxr-x---   2 root openvpn 4.0K Nov  2 00:00 server
lrwxrwxrwx   1 root root      17 Feb 14 11:27 sepia.conf -> sepia/client.conf
drwxr-xr-x   5 root root    4.0K Feb 14 11:27 .
drwxr-xr-x 103 root root    4.0K Feb 15 10:54 ..
drwxr-x---   2 root openvpn 4.0K Feb 15 11:05 client
drwxrwxr-x   2 1106    1106 4.0K Feb 15 14:39 sepia
➜  openvpn ls -lhart sepia
total 28K
-r--r----- 1 1106 1106  636 Mar 23  2015 tlsauth
-rw-r--r-- 1 1106 1106 1.3K Mar 23  2015 ca.crt
-rwxr-xr-x 1 1106 1106 2.0K Oct 28  2020 new-client
drwxr-xr-x 5 root root 4.0K Feb 14 11:27 ..
drwxrwxr-x 2 1106 1106 4.0K Feb 15 14:39 .
-rw------- 1 root root  100 Feb 15 14:39 secret
-rw-r--r-- 1 1106 1106  273 Feb 15 18:03 client.conf

Change the top folder to look like this

ls l /etc/openvpn/client/
total 8
drwxr-xr-x 2 akraitma akraitma 4096 Jan 25 18:51 sepia
lrwxrwxrwx 1 root root 17 Jan 8 18:34 sepia.conf -> sepia/client.conf
-rw-r--r- 1 root root 2737 Jan 8 12:04 sepia-vpn-client.tar.gz

adam kraitman wrote:

Change the top folder to look like this

ls l /etc/openvpn/client/
total 8
drwxr-xr-x 2 akraitma akraitma 4096 Jan 25 18:51 sepia
lrwxrwxrwx 1 root root 17 Jan 8 18:34 sepia.conf -> sepia/client.conf
-rw-r--r- 1 root root 2737 Jan 8 12:04 sepia-vpn-client.tar.gz

Looks like no different, my config dir now looks like this:

➜  ~openvpn
➜  ~openvpn tree
.
├── client
│   ├── sepia
│   │   ├── ca.crt
│   │   ├── client.conf
│   │   ├── new-client
│   │   ├── secret
│   │   └── tlsauth
│   └── sepia.conf -> sepia/client.conf
├── sepia-vpn-client.tar.gz
└── server

3 directories, 7 files
➜  ~openvpn ls -lhart client
total 12K
lrwxrwxrwx 1 root root      17 Feb 14 11:27 sepia.conf -> sepia/client.conf
drwxrwxr-x 2 root root    4.0K Feb 15 14:39 sepia
drwxr-xr-x 4 root root    4.0K Feb 20 11:11 ..
drwxr-x--- 3 root openvpn 4.0K Feb 20 11:11 .

And I've tried to modify the uid&gid of sepia/ to some other combinations like root:root, root:openvpn, openvpn:openvpn, even mer:mer (I don't have this user before, created by useradd).

In all of these scenes I can start openvpn-client@sepia service, but the ssh is all failed.

So maybe the problem is not with the owner, could you please re-confirm if I use the correct user or command:

ssh  mer@teuthology.front.sepia.ceph.com

Thanks

Or maybe I should delete the sepia dir and try again.

Can you ping teuthology.front.sepia.ceph.com ?

adam kraitman wrote:

Can you ping teuthology.front.sepia.ceph.com ?

I can't, the DNS looks like works, but all packs were dropped.

➜  ~ ping teuthology.front.sepia.ceph.com
PING teuthology.front.sepia.ceph.com (172.21.0.51) 56(84) bytes of data.
^C
--- teuthology.front.sepia.ceph.com ping statistics ---
15 packets transmitted, 0 received, 100% packet loss, time 14582ms

I know there is a firewall in china maybe this is what's blocking it?

adam kraitman wrote:

I know there is a firewall in china maybe this is what's blocking it?

I'm already configured http-proxy in sepia.conf

proto tcp-client
http-proxy proxy-ip proxy-port

And I can ping the vpn.sepia.ceph.com instead of the teuthology.front.sepia.ceph.com

➜  ~ ping vpn.sepia.ceph.com -c5
PING gw.sepia.ceph.com (8.43.84.129) 56(84) bytes of data.
64 bytes from gw.sepia.ceph.com (8.43.84.129): icmp_seq=1 ttl=45 time=266 ms
64 bytes from gw.sepia.ceph.com (8.43.84.129): icmp_seq=2 ttl=45 time=283 ms
64 bytes from gw.sepia.ceph.com (8.43.84.129): icmp_seq=3 ttl=45 time=278 ms
64 bytes from gw.sepia.ceph.com (8.43.84.129): icmp_seq=4 ttl=45 time=285 ms
64 bytes from gw.sepia.ceph.com (8.43.84.129): icmp_seq=5 ttl=45 time=290 ms

--- gw.sepia.ceph.com ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4003ms
rtt min/avg/max/mdev = 265.614/280.241/290.022/8.301 ms
➜  ~ ping teuthology.front.sepia.ceph.com -c5
PING teuthology.front.sepia.ceph.com (172.21.0.51) 56(84) bytes of data.

--- teuthology.front.sepia.ceph.com ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4184ms

I know that Yingxin Cheng also had this issue with the firewall and configured a proxy maybe you should ask him how he did it ?

adam kraitman wrote:

I know that Yingxin Cheng also had this issue with the firewall and configured a proxy maybe you should ask him how he did it ?

Thanks, I don't have his contact information, maybe you could give me an address.

cyx1231st@gmail.com