Firewalling Revisited

Dealing with low level firewalling (ufw, iptables, nftables) is something that I haven’t done in a while, but had to revisit recently for the following scenarios:

1) Building a minimal custom confidential VM image 2) Migrating workloads from public cloud to bare metal cloud

This post will focus more on point two. While I did have to wrangle a bit with conntrack, I intend to focus this post more on stateless firewalls.

With public cloud, Cloud Service Providers (CSP) often provide a web UI or a cli command for you to easily whitelist specific ports - 22 for ssh is typically enabled for you by default since the only way you can access your VM instance is via SSH. And so, very often, you don’t really need to put too much thought into actually fiddling with the “low level” firewall in your VM itself. Everything is very simple and just works.

However, it’s a different story when switching to bare metal cloud - while the CSP I used does have an interface for setting some rules at their Edge Router, I believe there’s a limit to the number of rules we can set, which meant that fiddling with low level firewalling settings was a much better option.

However, the initial tests were rather messy and some things did not quite work out the way I planned.

For example, this is a scenario which happened: I had two different docker containers running some service on the same port, 8545. I set the docker compose to forward host port 9001 to the first container’s port 8545, and 9002 to the second container’s port 8545.

In this case, what should be the correct ufw rule to whitelist traffic going into these two docker containers, assuming I already have a rule to drop all other traffic by default, except for SSH? The answer is actually this:

-A INPUT -p tcp -s 1.2.3.4 --dport 8545 -j ACCEPT

The reason is this: iptables processes rules in a certain order. The order is roughly something like this:

1. RAW Chain
2. MANGLE Chain: You can use this to manipulate the packet contents.
3. PREROUTING Chain: This is used by rules that are listed as DNAT.
4. INPUT Chain: For packets destined for this VM.
5. OUTPUT Chain: For packets generated by this VM.
6. FORWARD Chain: For packets routed through this VM.
7. POSTROUTING Chain: You can apply SNAT rules to outgoing packets.

What’s actually happening is that when you spin up the container, docker creates some firewall rules, something like this:

-A DOCKER -d 0.0.0.0/0 -p tcp --dport 9001 -j DNAT --to-destination <container-ip>:8545

This rule means to forward any incoming TCP traffic on the host’s port 9001 to the container’s IP address on port 8545.

Since this rule happens on the PREROUTING chain, it executes before our rule, which happens on the INPUT chain. This means that if you set your input rule to allow traffic to port 9001 or 9002, it will not have any matches, as your packet has already been re-written, and so your packet gets dropped instead. The way to confirm this is to first check if you are receiving any traffic on the host NIC interface at port 9001, which you should definitely see your incoming packets, then check if you are receiving any traffic on the bridge interface used by your docker container at port 8545, which should basically show you no packets being received.