Let’s Set The Mood
Lately I’ve been getting into more video games. I happen to prefer co-op video gaming with friends. I’m also a big fan of first person perspective games.
There is one game in particular that’s been nothing but problematic for myself and my core group of friends that play co-op games with each other. We do the whole ‘just a few of us who are in it for fun’ gaming and prefer to have something that doesn’t require a hell of a lot of thought.
Then we started playing this one game…
To say we’ve had nothing short of pure pain trying to get co-op sessions going is wild understatement. I spent 4 hours one night with a close friend trying to figure out why we just could NOT connect to each others game sessions. We both have FreeBSD based routers (Mine is pfSense, theirs is opnSense). They are real computers running real operating systems and networking stacks that are used in very critical environments. These are routers that don’t mess around. Mine has an octo core Atom CPU with 8Gb RAM, 4x Intel Gigabit ethernet and full iKVM and IPMI. My friends box is an off lease HP enterprise machine with dual Intel Gigabit ethernet, 4Gb of ram and a proper Intel i3 CPU.
These are REAL routers.
Imagine our surprise when the best we could ascertain was absolutely nothing. We port forwarded, turned off QoS, re-jiggered our internal LAN subnets, turned on IDS features to see what traffic may be flowing and more.
4 Hours later we just tossed in the towel and gave the fuck up and resigned ourselves to probably not being able to play co-op together.
A week later another friend of mine, a network engineer for an ISP, picked up the game and had the EXACT SAME PROBLEMS as me and my other friend. This network engineer happens to have a FULL UniFi stack, from the USG all the way out to the switching and wireless APs. We’re talking pure enterprise hardware that just works always.
That’s 3 people now, all who own competent, quality hardware. That’s a sign the problem is definitely not our hardware. Most likely the network stack and code that the game uses under the hood. But how do you figure out the root problem?
This time it’s me and a network engineer.
We ran Wireshark, tcpdump, overhauled LANs, created VLANs and did all of the magic voodoo network engineers, dev ops and systems integration folks work with regularly.
It’s the game code. 100% the game code.
Specifically the way they handle the NAT traversal piece via coturn/stun on the game developers match making service and NAT traversal endpoints. Turns out if you have the same subnet they use internally you’re going to leak your private IP address. They also have a web socket in play that helps with the upstream hand off stuff but it’s prone to failures. They also have some fall back stuff in place that rarely works.
Run wireshark and poke around connecting to a friends game if you’re curious to see the traffic.
Getting A Fix Working
It took about 6 hours to go through the magical dance of networking doom to find the actual problem and once we found it, it was clear what needed to be done.
AVOID THEIR SERVERS AND SERVICES
So we deployed a VPN and got to more testing and integration. Turns out we could NOT get local LAN play to work, like at all. We could see traffic flowing through 2 different VPN solutions (Wireguard and ZeroTier) but nothing would allow our games to see each other.
That’s a head scratcher.
Turns out the games net code is trash too.
If you use the ‘online play’ option in the game AND you have a VPN in play that’ll let two computers see each other (even if it’s a non-default network or a local only VPN network) you can reliably connect to the other game.
So you start the VPN so you can see the other person’s computer from yours, fire up the game, watch the game’s UDP traffic start flowing because it can see the other instance (it still won’t work yet, I promise) and then JOIN VIA THE ONLINE SYSTEM to connect to the other game session. This works flawless and you’ll see the games talking directly to each other over the VPN. You don’t even need broadcast support on the VPN for this approach to work.
We did a bunch of connect, disconnect, re-connect kinds of testing and the game has been reliable since.
We have a fix!
Ranty Ranterson
Let’s say this was one of the worst online gaming experiences I’ve ever experienced in my 20+ years of playing video games on a computer. I’m old enough to know what it’s like to use a dial up modem to connect to a gaming server, go through the match making progress and then enjoy an online game with 300ms of lag due to the tech of the time and network speeds of the day. I also remember having LAN parties back when you had to use a network hub because switches were too expensive.
This modern game is terrible and I have no idea what is causing such tremendous problems but you can go online to various forums, support sites and more to see myself and my friends are not alone with this problem.
The Full Fix Setup
Now that I’ve set the tone and laid out a basic overview of the fix, lets walk through how to put this together in a useful manner that’s production grade.
The core of this is basically a properly configured Wireguard VPN server that everyone can connect to for ensuring the game can connect to other instances. It also sets up a VPN so you can play other games that only support local LAN play.
Good stuff.
Hardware
This guide assumes you have a Linux box running Docker. I used a Friendly arm m4 board (similar to a Raspberry Pi 4 but more powerful) that I also have pi-hole deployed on.
You can use anything with a single CPU core and 512Mb of RAM or better. Wireguard doesn’t need much hardware for small deployments and scales nicely.
The configs below can be adapted for other deployment strategies but I leave that as an exercise to the enterprising readers of this blog.
Router
On your router you’ll need to setup the following.
- A static IP address for the host machine running Docker
- A NAT rule for the Wireguard UDP port. By default it’s
51820. The configuration below assumes this port. - Any QoS tagging/queuing needed. I used the Gaming queue on my QoS setup
Docker Host OS
In order for the container to work reliably I opted to setup the wireguard.ko kernel module on my base Debian install instead of letting the container manage the kernel module installation.
I installed the wireguard-dkms package that will build the wireguard module for all the kernels on the machine and I added wireguard to /etc/modules so it auto starts on boot.
The kernel module can load safely and won’t cause Wireguard to activate. It will only activate if the container is setup and running (see below).
Wireguard Config Prep
The Docker container we will be deploying is fairly automagical and works off templates to generate configuration files for the server and peers. We’re going to go ahead and front load these templates with some adjustments.
The default templates the container creates are good, generic ones but we want this to be a dedicated LAN that won’t allow people to see into each others local networks, route their traffic through someone elses internet or anything else potentially hazardous.
The container will use /var/wireguard for the storage of configs and persistent data so we’re going to work off that directory.
You’ll need to run mkdir -p /var/wireguard/templates and then create the following files in that directory.
peer.conf
[Interface]
Address = ${INTERFACE}.$(( $i + 1 ))
PrivateKey = $(cat /config/peer${i}/privatekey-peer${i})
ListenPort = 51820
DNS = ${PEERDNS}
[Peer]
PublicKey = $(cat /config/server/publickey-server)
Endpoint = ${SERVERURL}:${SERVERPORT}
AllowedIPs = 192.168.254.0/24
PersistentKeepAlive = 25
server.conf
[Interface]
Address = ${INTERFACE}.1
ListenPort = 51820
PrivateKey = $(cat /config/server/privatekey-server)
Table = off
PostUp = ip route add 192.168.254.0/24 via 192.168.254.1 dev wg0 ; iptables -t nat -A PREROUTING -s 192.168.254.0/24 -d 192.168.254.0/24 -j ACCEPT ; iptables -A FORWARD -i wg0 -s 192.168.254.0/24 -d 192.168.254.0/24 -j ACCEPT ; iptables -A FORWARD -i wg0 -s 192.168.254.0/24 -d 0.0.0.0/0 -j DROP ; iptables -A INPUT -i wg0 -s 192.168.254.0/24 -d 192.168.254.0/24 -j ACCEPT ; iptables -A INPUT -i wg0 -s 192.168.254.0/24 -d 0.0.0.0/0 -j DROP ;
PostDown = ip route del 192.168.254.0/24 via 192.168.254.1 dev wg0 ; iptables -D -t nat -A PREROUTING -s 192.168.254.0/24 -d 192.168.254.0/24 -j ACCEPT ; iptables -D -A FORWARD -i wg0 -s 192.168.254.0/24 -d 192.168.254.0/24 -j ACCEPT ; iptables -D -A FORWARD -i wg0 -s 192.168.254.0/24 -d 0.0.0.0/0 -j DROP ; iptables -D -A INPUT -i wg0 -s 192.168.254.0/24 -d 192.168.254.0/24 -j ACCEPT ; iptables -D -A INPUT -i wg0 -s 192.168.254.0/24 -d 0.0.0.0/0 -j DROP ;
The above configs will setup a basic server with keep alive support (helps with stability) and a TON of iptables magic to ensure that the Wireguard VPN traffic is isolated to the Wireguard container. No LAN or routing leaks. All you’re going to see on the Wireguard network is the other computers connected to the VPN, exactly what we want.
The peer config will only allow the LAN as a valid network for use via the Wireguard endpoint. This can be modified by clients but the server networking rules will prevent people from screwing around trying to get the Wireguard server to leak data (hint: it won’t, we tried during our tests).
Docker Container
The below chunk of shell script code will setup and run a Wireguard VPN Docker container. I used the LinuxServer.io (link) Wireguard container. They know what they are doing, have been around for many years and are a great source of Docker containers that will ‘just work’ on Raspberry Pi’s and similar hardware.
Please note this container run script will delete the container when you press Ctrl-C to kill the active docker run command that’s in the script. That’s intentional. If I stop the Wireguard service I don’t want it popping back up again. Remove the docker rm -f line at the end of the script to prevent the container being removed upon exit.
You’ll also notice the SERVERURL parameter. You need to change this to something. I have my own DNS records I can manage so I went with one of my own DNS records as a CNAME. If you don’t have a domain, there are plenty of DynamicDNS providers online you can use to get a proper DNS record that’ll make outside connections easier to setup and more stable.
Change the PEERS parameter if you need more than 9 or less than 9 people supported by your Wireguard deployment.
#!/bin/bash
docker pull linuxserver/wireguard:latest
docker rm -f wireguard
docker run -it \
--restart unless-stopped \
--name wireguard \
--cap-add=NET_ADMIN \
--cap-add=SYS_MODULE \
-p 51820:51820/udp \
-e TZ="UTC" \
-e SERVERPORT=51820 \
-e PEERS=9 \
-e INTERNAL_SUBNET=192.168.254.0 \
-e SERVERURL=[setthistosomething] \
-v /var/wireguard:/config \
-v /lib/modules:/lib/modules:ro \
linuxserver/wireguard:latest
docker rm -f wireguard
Running The Setup
Simply run the script above and it’ll deploy the Wireguard container and will auto-generate the peer configurations in /var/wireguard on the Docker host.
Give each person connecting to the Wireguard VPN a different peer config from that folder so they can setup Wireguard on their local machines and connect to the VPN.
Once everyone is connected you should be able to game happily and freely without relying on broken upstream services.