VPN Routing and Security
I was working abroad temporarily and needed to use a shopping mall’s Internet service. This mall offered open Wi-Fi that anyone could use after going through a captive portal.
After filling out the necessary form, I gained access to the Internet through the network. Then, I activated the VPN based on the OpenVPN client. All traffic will go through the VPN, so I should be safe, right?
Well, not exactly!
I was using this shopping mall’s Internet without any issues until I had to connect to an internal IP through SSH. Strangely, it was timing out.
$ ssh my-host
ssh: connect to host 10.2.1.30 port 22: Operation timed out
I started to try to understand what was happening. At first, I thought it was a problem with the server itself or the internal network. But it sounded more like my packets were not reaching the internal network.
So, what’s happening to the packets on my laptop? I’ve used tcpdump to watch the movement of packets in the network interface.
$ tcpdump -lvvnp port 22
tcpdump: data link type PKTAP
tcpdump: listening on pktap, link-type PKTAP (Apple DLT_PKTAP), capture size 262144 bytes
15:17:39.392720 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
10.199.125.146.59021 > 10.2.1.30.22: Flags [SEW], cksum 0x7d95 (correct), seq 1625334601, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 719456825 ecr 0,sackOK,eol], length 0
15:17:40.379066 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
10.199.125.146.59021 > 10.2.1.30.22: Flags [S], cksum 0x7a6d (correct), seq 1625334601, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 719457825 ecr 0,sackOK,eol], length 0
15:17:41.382076 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
10.199.125.146.59021 > 10.2.1.30.22: Flags [S], cksum 0x7685 (correct), seq 1625334601, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 719458825 ecr 0,sackOK,eol], length 0
15:17:42.384117 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
10.199.125.146.59021 > 10.2.1.30.22: Flags [S], cksum 0x729d (correct), seq 1625334601, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 719459825 ecr 0,sackOK,eol], length 0
The source IP of the packets is set to the gateway of the Wifi AP and not from the VPN tunnel interface. Why? I’d expect that if I have the VPN active and with adequate VPN configuration, all packets would go through the VPN tunnel…
The next step was to look at the route tables. In macOS we can use netstat -rn command. An excerpt of the output is:
:::plaintext
$ netstat -rn
Routing tables
Internet:
Destination Gateway Flags Netif Expire
0/1 x.x.x.x UGSc utun2
default 10.128.128.128 UGSc en0
10 link#4 UCS en0 !
10.128.128.128/32 link#4 UCS en0 !
10.128.128.128 x.x.x.x UHLWIir en0 1145
10.199.125.146/32 link#4 UCS en0 !
10.248.5.109 x.x.x.x. UHLWI en0 1194
10.255.255.255 ff:ff:ff:ff:ff:ff UHLWbI en0 !
80.1.1.1/29 x.x.x.x UGSc utun2
80.1.1.1/32 10.128.128.128 UGSc en0
127 127.0.0.1 UCS lo0
127.0.0.1 127.0.0.1 UH lo0
...
The 80.1.1.1 is the public IP of the VPN gateway, and the 10.128.128.128 is the internal IP of the Wifi AP gateway. So, the VPN configuration adds a default destination, 0/1, which should have more priority than the “default” gateway of the Wifi AP.
Why is the packet routed from the en0 interface if the routing table looks good? 🤔
If we look more carefully, we see an additional entry for the destination 10.0.0.0/8:
:::plaintext
...
default 10.128.128.128 UGSc en0
10 link#4 UCS en0 <---
10.128.128.128/32 link#4 UCS en0
...
Since the network stack prioritizes routes with a longer prefix, instead of forwarding the packet through the default gateway, it will use this route with a longer prefix: 10.0.0.0/8 (instead of 0/1) if the destination IP is within the range 10.0.0.0/8, which is the case!
So even though the VPN configuration sets a default gateway, network-provided routes for specific IP ranges can’t override it!
Off-topic, but if it happens to you also, we’ve to remove the conflicting route to fix the connectivity issue. In macOS, that can be done with:
# route delete -host 10.0.0.0/8 -link 4
Comments? Feel free to send me an email or reach out through Twitter.