Site-to-Site VPN: Manual IPSec

Despite residing in the heart of Silicon Valley here in California, I have exactly one ISP offering speeds greater than 25 Mbps - Xfinity. For the last few years, I had been running shop with a 100 Mbps down / 5 Mbps up cable plan (which Xfinity has graciously upgraded recently to 400 Mbps / 20 Mbps). As mentioned in the previous section, I do run a relatively heavy network, thanks to the lab infrastructure in place for evaluating systems and storage devices in addition to serving the needs of a typical family of four.

Back in India, there is a lot more competition among ISPs to serve consumers. There are multiple fiber-to-the-home (FTTH) service providers with a wide variety of symmetric speed options. For the new home, my parents opted to go with Airtel's symmetric 100 Mbps plan (costing approximately US $12 inclusive of taxes). Their network demands were not too heavy - a smart TV, couple of mobile phones, a desktop, and a notebook - with only a couple of clients being simultaneously active.

While I had multiple VLANs at home, with a specific subnet for guests isolated from the rest (automatically created when a guest Wi-Fi network is configured), the configuration for the UniFi Dream Machine had only one primary network and another guest network.

During the initial configuration of the UniFi Dream Machine, Airtel had provided a public-facing WAN IP for the UDM to pick up. There was a necessity to call up the ISP to put their gateway (FTTH terminal / Wi-Fi AP) in bridge mode, but that is outside the scope of this article. The UDM was configured with the appropriate credentials to authenticate over PPPoE and pick up the WAN IP via the bridge connection. With the public-facing WAN IPs of both sites at my disposal, configuring the site-to-site VPN was a breeze.

On the US side, activating the site-to-site VPN network creation prompted for the required details - network name, VPN protocol, the pre-shared key, and the server address. The USG Pro 4 supports manual IPSec and OpenVPN, with the former capable of getting hardware-accelerated. A random pre-shared key can be generated and copied over. The server address was set to the WAN IP of the USG Pro 4. Under the 'Remote Device Configurations' section, it was required to specify the remote subnets desired to be made visible locally, along with the WAN IP of the UDM.

Similar information was entered in the UDM, with the pre-shared key generated on the USG Pro 4 placed in the PSK field. Ubiquiti has slightly reworked the UI in the UDM's Network application (Network 7.2.94 vs. 7.1.68 in the USG Pro 4), with the 'server address' tag being replaced by 'UniFi Gateway IP', making things slightly more user friendly. The remote device configuration section is filled with the required subnets from the US side, along with the USG Pro 4's WAN IP.

Upon adding the new VPN network on both ends, there was a handshake between the two devices and I was able to access the devices in the Indian network from the US and vice-versa. The web UI configuration transparently handles all the port openings required on either end.

The site-to-site VPN setup was further augmented with an old NUC connected to the UDM. The PC was set up to run a squid proxy server. In the US, an Android tablet was dedicated to accessing the Indian OTT services and set up to access the Internet using the NUC as a proxy. This configuration worked fine for more than a month. There were a few interruptions due to power failures and DHCP WAN IP changes on the UDM side. The latter had to be reflected in the site-to-site VPN setup and resulted in some downtime, but was not a cause for major concern.

I was fairly happy with the setup and would have left it as-is, if not for waking up one fine morning and finding the VPN link down. Expecting the customary WAN IP change, I fired up the UniFi Network app and tried to figure out the new IP assigned to the UDM.

Using the 100.107.xx.xx IP in the site-to-site setup was not helpful in re-activating the VPN link. Given my lack of formal network administration skills, this ended up being my introduction to the nitty-gritty details of carrier-grade network-address translation (CGNAT) - a term I had only encountered in passing earlier.

Introduction The CGNAT Spanner in the Works
Comments Locked

35 Comments

View All Comments

  • prophet001 - Wednesday, December 21, 2022 - link

    Not really a ubiquiti fan.
  • Threska - Wednesday, December 21, 2022 - link

    Ubiquiti vacuum.
  • OddballSix - Wednesday, December 21, 2022 - link

    There's no point in even talking about Ubiquiti, you can't buy most of their products. Some of them have been out of stock in the entire channel for months.

    Entire parts and lines of products gone. You can't buy them. One breaks? You're screwed. Need to upgrade the firewall? Tough.
  • HalcyonDays - Wednesday, December 21, 2022 - link

    I actually went down a similar path as you did. Years ago, when I moved out, I needed away to troubleshoot my parents network remotely when I inevitably get the dreaded phone call "internet is not working".

    My requirements for this setup are as follows:
    1. Bidirectional encrypted tunnel(s) - preferably peer-to-peer
    2. No third-party cloud services
    3. Each site access internet through their its own ISP
    4. Router at each site will handle the VPN connection - no additional hardware

    After attempting and investigating multiple methods, I eventually settled on "tinc" based on the suggestion from the openwrt forums.

    "tinc" is a peer-to-peer VPN supported by Tomato, Openwrt, and asuswrt-merlin. It doesn't need all sites to have public IP to work. It just need one site to have public IP (I think). To handle dynamic IPs, I use a free DDNS service and assign a domain name to each of the site.

    Since then, I have expanded VPN network to include the in-laws and parents' home in Taiwan. It just required the router at each site to have the public key of at least one other site and it'll be able to see all sites. This means that I can be at any of these sites and still see every site.

    Some caveats: I am uncertain of the performance. From what I can tell, "tinc" is pretty lightweight but not as performant as wireguard. Because I don't stream anything over tinc tunnels, I can't vouch for how well it works for for that.

    Give it a try.
  • Samus - Thursday, December 22, 2022 - link

    Amazing hardware and stability totally ruined by crap software. The controller is trash. Relying on Java is already a red flag but the way the controller database functions is bazaar and totally insecure. Inheriting\adopting hardware into a new instance results in a mandatory config wipe. No fortune 500 or enterprise network would use this so what really separates it from a $100 consumer product? A consumer product that often has more basic functionality; Ubiquiti has to this day failed to implement MAC cloning, axing it from consideration to anybody who has AT&T or Verizon fiber that need to emulate their gateway from the ONT\media converter. Such a basic feature dating back to the Linksys routers of the 90's missing from a $300 prosumer product is embarrassing and should alone put the company underwater. I mean why?
  • Hamm Burger - Thursday, December 22, 2022 - link

    My ISP provides CGNAT by default, but one can pay extra (€1.95/month afair) for a non-fixed but routable address. Which I do. Of course, you have to to know that you can ask, because they don't advertise this feature.
  • Samus - Saturday, December 24, 2022 - link

    That is cheap. Commercial block IP's are rarely offered in the US to residential subscribers, and even 'business' internet plans find a way to screw you out of $15 minimum for a "usable" static address.

    It's worth noting over the years I've seen most IP addresses - even for residential internet - have become statically assigned to subscribers, but they are non-routable.
  • ballsystemlord - Thursday, December 22, 2022 - link

    @Ganesh , why not just contact the ISP and tell them that you were paying for an IP address that is *not* behind CGNAT? I mean, if you're spending the money for the IP you should get it.
  • Jorgp2 - Thursday, December 22, 2022 - link

    Yup, or just pay for a /29 or something.
  • coburn_c - Thursday, December 22, 2022 - link

    ipv6 is dead and rightly so

Log in

Don't have an account? Sign up now