Putting This Site’s AWS Infrastructure Into Terraform

 09/15/2026 -  ~8 Minutes

Introduction

This site has been running on S3 and CloudFront since 2020, but until recently the actual infrastructure behind it existed entirely in my head (and in the AWS console). Nothing was written down anywhere except a cloudFrontDistributionID in config.toml. That finally changed: the bucket, the CloudFront distribution, the DNS zone, and even a small piece of forgotten Lambda@Edge code are now all under Terraform, and the whole dev environment (Hugo, Terraform, the AWS CLI) is reproducible with one nix develop. Here’s how it fits together.

The Dev Shell (flake.nix)

shell.nix was retired in favor of a proper flake. nix develop now drops you into a shell with hugo, terraform, awscli2, and awsume:

devShells.default = pkgs.mkShell {
  nativeBuildInputs = with pkgs; [
    git
    hugo
    awscli2
    awsume
    terraform
  ];

  # `awsume` must be sourced (not exec'd) so it can export AWS_* env
  # vars into the calling shell; this function shadows the plain
  # script of the same name that's on PATH from the package above.
  shellHook = ''
    awsume() { source "${pkgs.awsume}/bin/awsume" "$@"; }
  '';
};

That shellHook turned out to matter more than I expected. awsume is a script that assumes an AWS role and exports AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY/AWS_SESSION_TOKEN into your current shell. If it just sits on PATH and you run it normally, bash execs it as a subprocess — it happily assumes the role, then exits, and every one of those exported variables evaporates with it. The fix is to shadow it with a shell function that sources the script instead of exec’ing it. Small detail, completely broken workflow if you miss it.

terraform also needs config.allowUnfree = true in the flake, since HashiCorp relicensed it under the BSL and nixpkgs (correctly) treats that as unfree.

One IAM Role Per Project, awsume, and Home Manager

The AWS side of this follows a convention I’ve now used across a handful of small static sites: a single aws repo owns the human IAM users (brian, bethany), plus two shared roles —

  • aws-admin, for interactive/ad hoc console-equivalent work, and
  • terraform, the identity Terraform itself assumes when running apply, so CloudTrail can tell “a human clicking around” apart from “a Terraform run.”

Every project repo — this one included — then owns its own narrow deploy role. wardtalks.com’s is wardtalks-deploy: S3 read/write/delete on exactly this bucket, cloudfront:CreateInvalidation on exactly this distribution, and (more on this below) enough Lambda permission to push code to exactly one function. A wardtalks-deployers IAM group grants sts:AssumeRole on that role; membership in the group is the only thing that changes when someone new needs deploy access.

(The delete part I only got right on the second try — terraform plan showing zero drift tells you the existing infrastructure is described correctly, it says nothing about whether a brand-new role you just wrote is missing a permission it needs. The first real hugo deploy against this role failed with an AccessDenied on s3:DeleteObject: hugo deploy is a two-way sync, not just an upload, so a role that can only PutObject/GetObject can push new posts but can’t prune old files that no longer exist locally. One terraform apply later, fixed.)

~/.aws/config ties it together with one profile per role:

[profile brian]
region = us-east-2

[profile wardtalks]
source_profile = brian
role_arn = arn:aws:iam::<account-id>:role/wardtalks-deploy

[profile terraform]
source_profile = brian
role_arn = arn:aws:iam::<account-id>:role/terraform

That file is now generated by Home Manager instead of hand-edited, via the (surprisingly little-known) programs.awscli module:

programs.awscli = {
  enable = true;
  package = null; # awscli itself comes from each project's own flake, not globally
  settings = {
    "profile brian".region = "us-east-2";
    "profile wardtalks" = {
      source_profile = "brian";
      role_arn = "arn:aws:iam::<account-id>:role/wardtalks-deploy";
    };
    # ...one block per project
  };
};

package = null is worth calling out: it stops Home Manager from installing awscli2 as a system package, while still generating the config file — the two are controlled independently. ~/.aws/credentials, the file with the actual long-lived access keys, deliberately stays outside of Nix and hand-maintained, since anything in the Nix store is world-readable.

Day to day, this means awsume wardtalks gets me a shell scoped to exactly what deploying this site requires, and nothing more.

Terraform: Importing What Already Existed

The interesting part wasn’t writing new Terraform — it was writing Terraform that describes infrastructure that had existed, untouched, since 2020, without disturbing any of it. terraform import for the S3 bucket, the CloudFront distribution and its Origin Access Identity, and the Route53 hosted zone, followed by iterating on the .tf files until terraform plan reported zero drift. A few things surfaced along the way that I’d genuinely forgotten:

  • The CloudFront origin still points at the legacy global S3 endpoint (wardtalks.com.s3.amazonaws.com) rather than the regional one, because the distribution predates regional endpoints becoming the default. Declaring it as aws_s3_bucket.wardtalks.bucket_regional_domain_name would have quietly changed the origin on first apply, so it’s a literal string instead, with a comment explaining why.
  • default_root_object on the distribution is unset. Not “index.html” — nothing. Which raised the obvious question: how has the homepage been loading for six years?

That question is answered by the next section.

The Route53 Zone Is Shared, So Terraform Only Owns Part Of It

wardtalks.com’s hosted zone doesn’t only serve this site — it also carries records for sibling projects (sobriety.wardtalks.com, serverless.wardtalks.com) that this repo has no business touching. So the zone itself is imported and managed here, but only the apex A/AAAA records (aliased to this site’s CloudFront distribution) and the ACM validation CNAME for this certificate are declared. Everything else in the zone is simply left alone — Terraform only touches resources it’s told about, so there’s no risk of it “helpfully” pruning a sibling project’s DNS record it’s never heard of. If another project wants its own subdomain here, the right move is for that repo to look the zone up with a data source, not for this repo to create records on its behalf.

The Lambda Nobody Remembered

Back to that missing default_root_object. S3’s REST API endpoint — which is what CloudFront has to use for a private, OAI-restricted bucket — has no concept of an index document at all. Not for the root, not for subdirectories. Every single page on this site, including the homepage, depends on a small Lambda@Edge function rewriting the request URI before it ever reaches S3: a trailing-slash request gets index.html appended, and a bare directory path like /posts/nixos gets /index.html appended.

That function, hugo-url-rewrite, existed only as a deployed Lambda — created directly in the console back in 2020, never committed anywhere. I pulled the actual running code back out of AWS (aws lambda get-function hands you a presigned download URL for the deployment package) and it turned out to be a small, public snippet, credited right there in the source:

'use strict';

// @starpebble on github
// hugo flavor cafe (scotch)

const DEFAULT_OBJECT = 'index.html';

exports.handler = (event, context, callback) => {
  const cfrequest = event.Records[0].cf.request;
  if (cfrequest.uri.length > 0 && cfrequest.uri.charAt(cfrequest.uri.length - 1) === '/') {
    cfrequest.uri += DEFAULT_OBJECT;
  }
  else if (!cfrequest.uri.match(/.(css|md|gif|ico|jpg|jpeg|js|png|txt|svg|woff|ttf|map|json|html|xml)$/)) {
    cfrequest.uri += `/${DEFAULT_OBJECT}`;
  }
  callback(null, cfrequest);
  return true;
};

It now lives in this repo (lambda/hugo-url-rewrite/index.js) and is deployed by Terraform, with one wrinkle: Lambda@Edge functions must be created in us-east-1 no matter where the rest of your infrastructure lives, and CloudFront can only reference a published, numbered version — never $LATEST. Both of those show up directly in the Terraform:

# Lambda@Edge functions must be created in us-east-1 regardless of where
# everything else lives.
provider "aws" {
  alias  = "us_east_1"
  region = "us-east-1"
}

resource "aws_lambda_function" "hugo_url_rewrite" {
  provider = aws.us_east_1
  # ...
  publish  = true # auto-publish a new version whenever the code changes
}

CloudFront’s lambda_function_association then points at aws_lambda_function.hugo_url_rewrite.qualified_arn, so it always tracks whatever Terraform most recently published.

I also gave wardtalks-deploy a narrow, function-scoped grant (UpdateFunctionCode, UpdateFunctionConfiguration, PublishVersion — deliberately not CreateFunction or DeleteFunction) so routine code pushes to this one Lambda don’t require pulling out admin credentials. That’s a nice property of least-privilege roles once you have them: it becomes cheap to hand out narrow new capabilities exactly where they’re needed, instead of reaching for the broad role out of convenience.

Where This Leaves Things

hugo builds the site, hugo deploy (reading the [[deployment.targets]] block already in config.toml) syncs it to S3 and invalidates CloudFront, and the whole thing now runs under the wardtalks AWS profile with exactly the permissions that requires — no more, no less. Every piece of infrastructure that makes that work is described in this repo, reviewable in a terraform plan, and — should this bucket, distribution, or Lambda ever need to be rebuilt from scratch in a new account — buildable from a clean terraform apply instead of from memory.

A note on how this post came to exist: this whole post was written by Claude Code, at the end of a long working session where it did the actual setup described above — the flake, the IAM roles, the Terraform imports, digging the Lambda’s source back out of AWS, all of it. I asked it to write up what we’d just built, and this is what came out.

  terraform aws hugo flake.nix awsume

Playing around with Incus

 04/10/2024 -  ~7 Minutes

LXD

Over coffee, I caught a video by Jay LaCroix (learnlinux.tv) from a few years ago about using LXD Linux Containers. Jay is a big Ubuntu guy, and his demo all started with snaps. I knew (or thought I knew) that LXD was generally available outside of the snap store, so I was undaunted.

I spent a few minutes getting LXD support turned on in NixOS and was ready to try some of this stuff!

Then I discovered that the images: repo ( https://images.linuxcontainers.org   ) had decided to stop supporting LXD. Their decision, based around a lack of support from Canonical (and a bit of dislike for Canonical in the wake of some controversial decisions), seemed valid. But that still left me with only Ubuntu server images for LXD.

A little more reading told me about incus, the replacement for LXD. All the images that used to be at linuxcontainers.org are actually still there, but only for incus these days. So, a bit more fiddling: and LXD was out, and incus was in.

The Setup

Firstly, the NixOS incantation

{ config, lib, pkgs, ... }:
{
  virtualisation.incus = {
    enable = true;
    preseed = {};
  };
  networking.nftables.enable = true;
}

Importing this into my NixOS configuration.nix made incus available. Now I had to get it setup. My first step was getting into the incus-admin group, so I could communicate with the running daemon via its unix socket. A quick log-out and log-back-in, and I was ready to initialize the container system.

To do this, I ran incus admin init.

There are two big parts to this

Part 1, The Storage Pool

If you’re running on zfs or btrfs, you get to create either a zpool or a subvolume (respectively) to house your container and config storage. Once the storage area is provisioned, it is installed under /var/lib/incus. If you’re not one of the cool kids running zfs or btrfs, there are other supported solutions; but, as you might expect, you lose out on some of the coolness that backing your containers with a COW filesystem gives you.

Part 2, The Network Bridge

You can have incus create a network bridge for you, but I found this problematic. In my case, I already had a bridge built out with systemd-networkd that enabled my libvirt/qemu/kvm virtual machines to run as peers on the local network. I wanted this for my incus containers as well, so I ultimately declined the offer from the init script, and attached to my existing bridge with the following.

incus profile device add default eth0 nic nictype=bridged parent=br0 name=eth0

This created a default eth0 device (for my future containers) that was tied to my existing br0 bridge. Surprisingly, it actually worked. I was expecting a lot more difficultly, TBH. My network configuration is a little obscure. It took a long time to figure it out in Arch, and another long time to figure out how to port that over to NixOS. I was expecting a similar slog to get things working in incus, but voila….

Getting started with a continer

To date I’ve only tried setting up Debian Bookworm containers. So, this is hardly a comprehensive tour. But, I didn’t want to forget what I had typed, so I started jotting it down here. To create a new, running container all I had to type was

incus launch images:debian/12 my-first-incus

The container’s name is my-first-incus. It came from the images repo, and debian/12 was its image name/tag. At this point, incus stop my-first-incus followed by incus start my-first-incus worked. And, after stopping, incus delete my-first-incus cleans it all up – though the downloaded base image remains cached in the storage area. If you want to create a new container but not start it up right away, then incus create my-next-incus is your friend. This would allow you to make some config tweaks before it fires up.

Once the container is running, then incus exec my-first-incus -- bash will give you a shell inside the container.

Something Big

To see what I could do with this, I decided to look at replacing my Jellyfin VM with a Jellyfin incus container. This was especially tricky, since I had not taken great notes during the installation and configuration of the Jellyfin server – and there was zero IaC automation.

At a minimum, I wanted to record the steps it took to get Jellyfin up and running. In a perfect world, I’d even have automation for it. I was thinking… maybe ansible could help here. And while ansible could probably be shoehorned in as a solution, it didn’t seem like a great fit.

For the first pass, I used… a bash script. I mean, it’s better than nothing, right?

I gathered up four files I had laying around from the initial config of the Jellyfin server

  1. My CA certificate
  2. The signed Jellyfin certificate
  3. The Jellyfin server key
  4. The jellyfin.conf file that gets handed to nginx to reverse-proxy (and TLS-ify) Jellyfin
incus launch images:debian/12 jellyfin

Then, I pulled up the How to install Jellyfin on Debian page at jellyfin.org and got to work. The first part of the install got curl and gnupg installed on the target machine so I could use them to pull down the signing key. But, it seemed easier to just do that on the host machine. Then I could just push the resultant file into the container.

curl -fsSL 'https://repo.jellyfin.org/jellyfin_team.gpg.key' | gpg --dearmor -o jellyfin.gpg
incus file push ./jellyfin.gpg jellyfin/etc/apt/keyrings/jellyfin.gpg -pv

EDIT: it turns out I did need, or seemed to need curl and gnupg for stuff to work. So,

incus exec jellyfin -- apt-get -y install curl gnupg

The trailing options on the incus file push work exactly like mkdir – any needed directory are created and the output is verbose.

Now, I needed an apt sources file. They had a complex formula for creating it. It was more reasonable to create a file named jellyfin.sources and put the following in it

Types: deb
URIs: https://repo.jellyfin.org/debian
Suites: bookworm
Components: main
Architectures: amd64
Signed-By: /etc/apt/keyrings/jellyfin.gpg

After this, I could push that file into the container with

incus file push ./jellyfin.sources jellyfin/etc/apt/sources.list.d/jellyfin.sources -pv

Now the repo definition is setup, the public signing key is in place, and we’re ready for some sexy apt-get action!

incus exec jellyfin -- apt-get update
incus exec jellyfin -- apt-get -y install jellyfin

Now, technically, Jellyfin is up and running inside the container. But, it’s time to get nginx in place. So, we should install it. And then we should move our config files into place.

incus exec jellyfin -- apt-get -y install nginx

incus file push ./jellyfin.conf jellyfin/etc/nginx/conf.d/jellyfin.conf -pv
incus file push ./jellyfin.crt jellyfin/etc/nginx/ssl/jellyfin.crt -pv
incus file push ./jellyfin.key jellyfin/etc/nginx/ssl/jellyfin.key -pv
incus file push ./ca.crt jellyfin/etc/nginx/ssl/ca.crt -pv

A quick diversion into NFS

I have my media served up by TrueNAS via NFS. That’s how the existing Jellyfin VM gets access to it. I need figure that out. Unsurprisingly, NFS isn’t very easy to pull of with incus containers. So, it turns out that the solution is to create special pseudo-disks that map to the places where the NFS shares are mounted on the host.

incus config device add jellyfin movies disk source=/srv/nfs/nas/media/movies path=/media/movies
incus config device add jellyfin shows disk source=/srv/nfs/nas/media/television path=/media/shows

This creates two devices in the jellyfin container. One named movies that maps the host directory /srv/nfs/nas/media/movies to the container directory /media/movies. The other does the same for television shows, disguising the fact that TrueNAS is still exporting the share with the old name of television, but Jellyfin prefers shows.

Back to your regularly scheduled nginx configuration

The final step is to remove the default website from nginx, since our configuration is all in jellyfin.conf, and restart nginx with all this new configuration goodness. This default website configuration file (actually a symlink) is /etc/nginx/sites-enabled/default, and we just need to tell incus to remove it. Then we’ll use systemd to restart nginx.

incus file delete jellyfin/etc/nginx/sites-enabled/default -v
incus exec jellyfin -- systemctl restart nginx

At this point, we just need to figure out where our running container is (network-wise) and use the Jellyfin web-ui to finish setup.

incus list jellyfin

Now, point the web browser at that address.

  • select your languae (e.g. English)
  • create a user in Jellyfin (e.g. jellyfin)
  • add a media library for movies, pointing to the /media/movies folder
  • add a media library for shows, pointing to the /media/shows folder

Then, just give Jellyfin a few moments to gather up the media and download metadata, you’re ready to start watching!

The Code

I put the script, and its supporting files (minus the SSL keys) into a GitHub repo   . The jellyfin.conf file might be of interest; it is almost entirely cribbed from a sample file at jellyfin.org. But, I’ve made the necessary adjustments for it to run out of the box in my solution.