Firecracker VMs That Reach the Internet and Nothing Else

A coding agent runs pnpm install, test suites and whatever else a task calls for, and until recently it did all of that as me, on the machine with my SSH keys and logins. stedding gives each session its own Firecracker microVM on a Linux box in my homelab. The agent loop stays where it was, in my case aludra, and only the tool calls go into the VM, over SSH. aludra’s post covers that side: templates per project, getting commits back and shipping them.

stedding is Bash, nftables and systemd. The network policy, the base image, the systemd units, pause and resume, and templates went in as five commits on the evening of November 3.

One VM, one systemd unit

Each sandbox is a transient systemd unit, stedding-sandbox-N, running one Firecracker process as a regular user in the kvm group, not root. When the unit stops, an ExecStopPost removes its tap device as root. Every operation on a sandbox takes a lock on that sandbox first, so a resume and a pause can’t cross.

Nothing in the guest manages its network. The address goes on the kernel command line:

console=ttyS0 reboot=k panic=1 quiet loglevel=3 clocksource=kvm-clock root=/dev/vda rw ip=$guest_ip::$host_ip:255.255.255.252:sandbox:eth0:off

The base image is Ubuntu 26.04 minbase built with mmdebstrap: systemd, Node 24 and pnpm from their release tarballs, git, build tools, Python, Rust through rustup with clippy and rustfmt, and the libraries Tauri builds against on Linux. The versions and download checksums are pinned in versions.env. There’s one user, user, root is locked, and SSH takes keys only. Units a sandbox has no use for, like networkd, resolved, timesyncd and the apt timers, are masked. A cold boot to SSH takes 2.7 seconds.

The image has no agent, no credentials and no browser. Chromium and its libraries would add about 400 MB to every template, so a project that uses Playwright installs it in its own setup and only its templates carry it. Rust added about 0.7 GB, and the image uses 1.6 GB.

A rebuild keeps the SSH host keys from the last one, so clients that pinned the sandboxes’ host key keep working. Adding a client’s key to a new sandbox doesn’t need a mount either: stedding new --key appends it to authorized_keys inside the disk image with debugfs.

Internet and nothing else

Sandbox N gets a tap device, stN, on the Nth /30 of 172.16.0.0/16, with the host at .1 and the guest at .2. That’s room for 16,384 sandboxes. The guest’s MAC encodes its address, which is the convention Firecracker’s CI images use.

The policy is one nftables table:

set blocked4 {
	type ipv4_addr
	flags interval
	elements = {
		0.0.0.0/8,          # "this network"
		10.0.0.0/8,         # private
		100.64.0.0/10,      # CGNAT, which Tailscale uses, 100.100.100.100 included
		127.0.0.0/8,
		169.254.0.0/16,     # link-local, cloud metadata
		172.16.0.0/12,      # private, other guests included
		192.168.0.0/16,     # private: the LAN
		224.0.0.0/3,        # multicast and reserved
	}
}

chain forward {
	type filter hook forward priority filter - 10; policy accept;
	iifname "st*" meta nfproto ipv6 counter drop comment "guests get no IPv6"
	iifname "st*" ip daddr @blocked4 counter drop comment "no LAN, tailnet or other guests"
	iifname "st*" oifname != $UPLINK counter drop comment "only out through the uplink"
}

An input chain drops anything a guest sends to the host unless it’s a reply, and a NAT table masquerades 172.16.0.0/16 out the uplink. Since the host’s and the LAN’s resolvers are both blocked, the image points DNS at public resolvers.

The box also runs ufw and Tailscale, and each has its own tables. The reason this works alongside them is how nftables evaluates: every table’s chain on a hook runs, and a drop in any of them is final. Another table accepting a packet can’t undo stedding’s drop. stedding’s chains also run at filter - 10, ahead of the others. The one thing ufw needs is a route rule letting st+ out the uplink, since its forward policy is drop, and the install script adds it.

bench/nettest.sh checks the policy from inside a guest. It boots two sandboxes, then probes from the first one: 1.1.1.1, github.com and the npm registry on 443 should be open, and the host over its tap, the host’s LAN address, the LAN gateway, the other guest and Tailscale’s 100.100.100.100 should be blocked.

Paused, with its processes running

A sandbox is running, paused or stopped. Pausing one pauses the VM, writes a full snapshot with its memory, and stops the unit. The snapshot goes into a new directory that’s renamed into place after Firecracker exits, because the running VM still has the old memory file mapped.

Resuming maps the memory file and loads pages as the guest touches them, so a resume takes about as long whatever the sandbox’s memory size: under a quarter second to SSH. Processes keep running across a pause. The guest uses kvm-clock, and the snapshot loads with clock_realtime, so the guest’s clock jumps forward to now.

The piece that makes pausing invisible to a client is connect:

connect)
	n=$(num "$1")
	( lock "$n"; up "$n" ) >&2
	net "$n"; exec nc -q 0 "$guest_ip" 22 ;;

up resumes a paused sandbox or boots a stopped one, then connect relays stdin and stdout to the guest’s SSH port. A client uses it as its SSH ProxyCommand, so connecting to a sleeping sandbox wakes it. In aludra’s live check, a background process started in one turn was still running with the same pid in the next turn after a pause.

A reaper on a 30-second timer pauses sandboxes nobody is using. A client connected through connect holds an nc relay, so the reaper looks for one and leaves that sandbox alone. Once a sandbox has gone a minute without one, it’s paused.

down is the clean shutdown. It sends Ctrl-Alt-Del through Firecracker’s API, the guest’s systemd reboots, and with reboot=k on the command line that ends Firecracker. The guest writes its page cache to disk on the way down, which killing the process wouldn’t allow.

Templates

stedding save N T shuts sandbox N down that way and keeps its disk as template T. stedding new --template T makes a new sandbox with cp --sparse=always, which takes about a quarter second for a 1 GB template. Before its first boot, new grows the disk to 32 GB (or anything from 8 to 256 with --disk) with truncate, e2fsck and resize2fs. The file stays sparse, so the host pays only for what the guest writes, and a template saved from a grown sandbox is already that size, so its sandboxes skip the resize.

A template holds only a disk. Each sandbox boots once from its template, then pauses and resumes with its own memory. stedding never deletes a template itself. The client that makes them knows when one is stale, and calls stedding drop.

Sizes are per sandbox: 512 MB to 32 GB of memory and 1 to 8 CPUs, with 4 GB and 2 CPUs by default.

One key, a few commands

A client’s key is restricted on the host like this:

restrict,command="/usr/local/bin/stedding gate" ssh-ed25519 AAAA... client

restrict turns off forwarding and terminals, and the forced command means whatever the client asked to run arrives as $SSH_ORIGINAL_COMMAND for stedding gate to judge. Every word has to match ^[A-Za-z0-9._-]+$, and the first has to be connect, new, save, rm, status, templates or drop. Pausing and shutting down aren’t on the list. Anything else exits with an error, so the most a client can do on the host is make, use and remove sandboxes.

In the Wheel of Time, a stedding is an Ogier refuge where the One Power can’t be touched.