Playing around with Incus

 04/10/2024 -  ~7 Minutes

LXD

Over coffee, I caught a video by Jay LaCroix (learnlinux.tv) from a few years ago about using LXD Linux Containers. Jay is a big Ubuntu guy, and his demo all started with snaps. I knew (or thought I knew) that LXD was generally available outside of the snap store, so I was undaunted.

I spent a few minutes getting LXD support turned on in NixOS and was ready to try some of this stuff!

Then I discovered that the images: repo ( https://images.linuxcontainers.org   ) had decided to stop supporting LXD. Their decision, based around a lack of support from Canonical (and a bit of dislike for Canonical in the wake of some controversial decisions), seemed valid. But that still left me with only Ubuntu server images for LXD.

A little more reading told me about incus, the replacement for LXD. All the images that used to be at linuxcontainers.org are actually still there, but only for incus these days. So, a bit more fiddling: and LXD was out, and incus was in.

The Setup

Firstly, the NixOS incantation

{ config, lib, pkgs, ... }:
{
  virtualisation.incus = {
    enable = true;
    preseed = {};
  };
  networking.nftables.enable = true;
}

Importing this into my NixOS configuration.nix made incus available. Now I had to get it setup. My first step was getting into the incus-admin group, so I could communicate with the running daemon via its unix socket. A quick log-out and log-back-in, and I was ready to initialize the container system.

To do this, I ran incus admin init.

There are two big parts to this

Part 1, The Storage Pool

If you’re running on zfs or btrfs, you get to create either a zpool or a subvolume (respectively) to house your container and config storage. Once the storage area is provisioned, it is installed under /var/lib/incus. If you’re not one of the cool kids running zfs or btrfs, there are other supported solutions; but, as you might expect, you lose out on some of the coolness that backing your containers with a COW filesystem gives you.

Part 2, The Network Bridge

You can have incus create a network bridge for you, but I found this problematic. In my case, I already had a bridge built out with systemd-networkd that enabled my libvirt/qemu/kvm virtual machines to run as peers on the local network. I wanted this for my incus containers as well, so I ultimately declined the offer from the init script, and attached to my existing bridge with the following.

incus profile device add default eth0 nic nictype=bridged parent=br0 name=eth0

This created a default eth0 device (for my future containers) that was tied to my existing br0 bridge. Surprisingly, it actually worked. I was expecting a lot more difficultly, TBH. My network configuration is a little obscure. It took a long time to figure it out in Arch, and another long time to figure out how to port that over to NixOS. I was expecting a similar slog to get things working in incus, but voila….

Getting started with a continer

To date I’ve only tried setting up Debian Bookworm containers. So, this is hardly a comprehensive tour. But, I didn’t want to forget what I had typed, so I started jotting it down here. To create a new, running container all I had to type was

incus launch images:debian/12 my-first-incus

The container’s name is my-first-incus. It came from the images repo, and debian/12 was its image name/tag. At this point, incus stop my-first-incus followed by incus start my-first-incus worked. And, after stopping, incus delete my-first-incus cleans it all up – though the downloaded base image remains cached in the storage area. If you want to create a new container but not start it up right away, then incus create my-next-incus is your friend. This would allow you to make some config tweaks before it fires up.

Once the container is running, then incus exec my-first-incus -- bash will give you a shell inside the container.

Something Big

To see what I could do with this, I decided to look at replacing my Jellyfin VM with a Jellyfin incus container. This was especially tricky, since I had not taken great notes during the installation and configuration of the Jellyfin server – and there was zero IaC automation.

At a minimum, I wanted to record the steps it took to get Jellyfin up and running. In a perfect world, I’d even have automation for it. I was thinking… maybe ansible could help here. And while ansible could probably be shoehorned in as a solution, it didn’t seem like a great fit.

For the first pass, I used… a bash script. I mean, it’s better than nothing, right?

I gathered up four files I had laying around from the initial config of the Jellyfin server

  1. My CA certificate
  2. The signed Jellyfin certificate
  3. The Jellyfin server key
  4. The jellyfin.conf file that gets handed to nginx to reverse-proxy (and TLS-ify) Jellyfin
incus launch images:debian/12 jellyfin

Then, I pulled up the How to install Jellyfin on Debian page at jellyfin.org and got to work. The first part of the install got curl and gnupg installed on the target machine so I could use them to pull down the signing key. But, it seemed easier to just do that on the host machine. Then I could just push the resultant file into the container.

curl -fsSL 'https://repo.jellyfin.org/jellyfin_team.gpg.key' | gpg --dearmor -o jellyfin.gpg
incus file push ./jellyfin.gpg jellyfin/etc/apt/keyrings/jellyfin.gpg -pv

EDIT: it turns out I did need, or seemed to need curl and gnupg for stuff to work. So,

incus exec jellyfin -- apt-get -y install curl gnupg

The trailing options on the incus file push work exactly like mkdir – any needed directory are created and the output is verbose.

Now, I needed an apt sources file. They had a complex formula for creating it. It was more reasonable to create a file named jellyfin.sources and put the following in it

Types: deb
URIs: https://repo.jellyfin.org/debian
Suites: bookworm
Components: main
Architectures: amd64
Signed-By: /etc/apt/keyrings/jellyfin.gpg

After this, I could push that file into the container with

incus file push ./jellyfin.sources jellyfin/etc/apt/sources.list.d/jellyfin.sources -pv

Now the repo definition is setup, the public signing key is in place, and we’re ready for some sexy apt-get action!

incus exec jellyfin -- apt-get update
incus exec jellyfin -- apt-get -y install jellyfin

Now, technically, Jellyfin is up and running inside the container. But, it’s time to get nginx in place. So, we should install it. And then we should move our config files into place.

incus exec jellyfin -- apt-get -y install nginx

incus file push ./jellyfin.conf jellyfin/etc/nginx/conf.d/jellyfin.conf -pv
incus file push ./jellyfin.crt jellyfin/etc/nginx/ssl/jellyfin.crt -pv
incus file push ./jellyfin.key jellyfin/etc/nginx/ssl/jellyfin.key -pv
incus file push ./ca.crt jellyfin/etc/nginx/ssl/ca.crt -pv

A quick diversion into NFS

I have my media served up by TrueNAS via NFS. That’s how the existing Jellyfin VM gets access to it. I need figure that out. Unsurprisingly, NFS isn’t very easy to pull of with incus containers. So, it turns out that the solution is to create special pseudo-disks that map to the places where the NFS shares are mounted on the host.

incus config device add jellyfin movies disk source=/srv/nfs/nas/media/movies path=/media/movies
incus config device add jellyfin shows disk source=/srv/nfs/nas/media/television path=/media/shows

This creates two devices in the jellyfin container. One named movies that maps the host directory /srv/nfs/nas/media/movies to the container directory /media/movies. The other does the same for television shows, disguising the fact that TrueNAS is still exporting the share with the old name of television, but Jellyfin prefers shows.

Back to your regularly scheduled nginx configuration

The final step is to remove the default website from nginx, since our configuration is all in jellyfin.conf, and restart nginx with all this new configuration goodness. This default website configuration file (actually a symlink) is /etc/nginx/sites-enabled/default, and we just need to tell incus to remove it. Then we’ll use systemd to restart nginx.

incus file delete jellyfin/etc/nginx/sites-enabled/default -v
incus exec jellyfin -- systemctl restart nginx

At this point, we just need to figure out where our running container is (network-wise) and use the Jellyfin web-ui to finish setup.

incus list jellyfin

Now, point the web browser at that address.

  • select your languae (e.g. English)
  • create a user in Jellyfin (e.g. jellyfin)
  • add a media library for movies, pointing to the /media/movies folder
  • add a media library for shows, pointing to the /media/shows folder

Then, just give Jellyfin a few moments to gather up the media and download metadata, you’re ready to start watching!

The Code

I put the script, and its supporting files (minus the SSL keys) into a GitHub repo   . The jellyfin.conf file might be of interest; it is almost entirely cribbed from a sample file at jellyfin.org. But, I’ve made the necessary adjustments for it to run out of the box in my solution.

NixOS Demo Script

 03/27/2024 -  ~5 Minutes

Install NixOS via the graphical installer

Nothing really special; just remember to select Allow Non-Free near the end of the (Calamares) installation wizard. It will be easier if the user created is named “demo”, otherwise there is a deviation from the script later on. Choose the graphical desktop you’d like to use.

Add a new package to the

Since the default installation does not include vim, let’s use nix-shell -p vim to temporarily make it available.

Add the google-chrome package to the user’s package list (sudo vim /etc/nixos/configuration.nix). While you’re in there, rename the system via networking.hostname = "demo"; in the config. Also, it is important to add the settings that enable flakes (‘cause we’re getting there shortly); add nix.settings.experimental-features = [ "nix-command" "flakes" ]; to the bottom of the configuration.nix file.

Then perform a sudo nixos-rebuild switch. Show it running!

You will have to reboot to see the new hostname, and you should since we’ll use it below….

Exit the nix-shell and show that vim goes away from the user environment; remind everyone that the package and its dependencies still exist in /nix/store.

A quick summary of this step:

  1. nix-shell -p vim
  2. set hostname to demo by modifying existing declaration to read networking.hostname = "demo"; in the configuration.nix file
  3. add google-chrome to the user’s package list in the configuration.nix file
  4. add nix.settings.experimental-features = [ "flakes" "nix-command" ]; to the bottom of the configuration.nix file
  5. exit
  6. nixos-rebuild switch
  7. [optional] reboot

Get Flakey!

Change directory into /etc/nixos and run sudo nix flake init to create a new instance of a basic flake template. Everything in the inputs section is good; the entirety of the outputs section can be replaced with the following:

nixosConfigurations.demo = nixpkgs.lib.nixosSystem { modules = [ ./configuration.nix ]; };

In the above, the demo token represents the hostname (which we changed from nixos to demo in the previous section).

Now, run sudo nixos-rebuild boot and a lot will happen! If you have not yet rebooted, be sure to add --flake .#demo to override the hostname to flake configuration mapping. We’re using nixos-rebuild boot since the drastic level of change implies we should reboot (likely an update kernel, for instance).

  1. The flake will lock in the current git hash of the nixos-unstable branch, and download the nixpkgs metadata for that revision
  2. All packages will be updated to the current unstable version
  3. The flake.lock file will be created to preserve this version of the system.

Now would be a good time to show off the lock file and display how it “follows” the nixos-unstable branch, but “locks” the particular rev.

Might be a good idea to reboot now (since our changes are waiting for us in the next generation).

There’s no place like Home (Manager)

After this big reboot, login as the unprivileged user. Let’s install home-manager! I chose the standalone configuration to underscore the distinction between user configuration and system configuration. There is another way (home-manager as a nixos module) which is left as an exercise for the reader.

As the user, we first need to add a nix-channel. This is something that happened (for the system user) behind the scenes during the install. It is also something that will be quickly surplanted by the flake we will create. Nonetheless

nix-channel --add https://github.com/nix-community/home-manager/archive/master.tar.gz home-manager
nix-channel --update

For whatever reason (still haven’t quite sorted this out, honestly), you have to log out and back in again. Something needs to be sourced into the environment, but I haven’t found it.

Once you’re back in you can install home-manager with the following incantation

nix-shell '<home-manager>' -A install

This actually creates our first home-manager generation. It also makes the home-manager tool generally available to us. From now on out, new generations of home-manager can be created with home-manager switch.

Customizing our user environment

Though the magic of nix, we can configure our user environment, and be reasonable isolated from the system environment. Let’s add and configure vim so we have access to it in our user environment. First let’s do our nix-shell -p vim trick from earlier so we don’t have to fumble through using nano. Then let’s add the following into our home.nix file via vim ~/.config/home-manager/home.nix

  programs.vim = {
    enable = true;
    settings = {
      expandtab = true;
      tabstop = 2;
      number = true;
      relativenumber = true;
      shiftwidth = 2;
    };
  };

Without even needing to add the package to the list, we will have vim and it will be configured to our (well, my) liking. A quick home-manager switch will make this a reality.

A more complicated Flake for Home-Manager

The home-manager flake is more complicated than the nixos flake, because home-manager has multiple inputs (home-manager and nixpkgs), and we want to keep the two closely tied to each other. We can start with the template, but here’s the full flake.nix file after modifications… or you can just write if from scratch using vim

The occurrence of “demo” below corresponds to the presumed username demo. If you have created a different unprivileged user, then replace the existing token with your new name.

{
  description = "Demo Flake for Home-Manager (Standalone)";

  inputs =  {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    home-manager = {
      url = "github:nix-community/home-manager";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { self, nixpkgs, home-manager, ... }:
  let
    system = "x86_64-linux";
    pkgs = import nixpkgs { system = "${system}"; config.allowUnfree = true; };
  in {
    homeConfigurations."demo" = home-manager.lib.homeManagerConfiguration {
      inherit pkgs;

      modules = [ ./home.nix ];
    };
  };
}

As I understand it, the follows line indicates the the nixpkgs supporting home-manager should follow the main nixpkages input (e.g. nixos-unstable). Again as I understand it, the is designed primarily to reduce system size by not duplicating packages that differ ever so slightly. However, Nix is fully capable of supporting (successfully) these inputs being disconnected. Any package used by one will pull in the correct versions of its dependencies, while the same package (different version) included in by the other input will pull its dependencies.

Regardless, now is the time to implement!

home-manager switch

Watch all the fun!

Additional Materials/Content

The slides   are available online.

NixOS Configuration Search/Wiki  

Home Manager Options Search