NixOS with snowfall-lib and deploy-rs

🔖 nixos

This post covers setting up a NixOS configuration using snowfall-lib for structured configuration and deploy-rs for deployment. By the end, you’ll have a solid baseline for managing your configuration.

The documentation only takes you so far. It assumes that you’re more comfortable with Nix than I initially was. Although I’m a big fan of NixOS I do not find reading Nix easy. The type of talented people that NixOS attracts often results in creative Nix configurations. I didn’t let this put me off jumping into it.

I have been using NixOS for about 2 years and my configuration has become very messy. I have 7 systems that share much of the same configuration. Currently I make use of lots of imports to enable the functionality I need. E.g. imports [./users/philip.nix].

Here I will outline the basic structure to have a NixOS system with snowfall-lib which is deployed by deploy-rs. Although the docs the cover creation of the NixOS system, they do not complete a guide with a functioning build that can be deployed.

If at any point you're confused why your changes aren't being seen, ensure that you've done a `git add .`. Flakes need files to be added to git to be used.

The top-level flake takes a standard set of configurations with inputs, outputs, etc. There is a sprinkle of snowfall-lib mixed.

Part 1: Setup

My folder tree.

./flake.lock
./flake.nix
./systems/x86_64-linux/test/default.nix
./systems/x86_64-linux/test/hardware.nix

We will cover the flake.nix file in parts to comment on it with a final example showing the full file. Commands may not work with partially complete configuration so please continue to the end.

Below we have a standard inputs section of a flake file. We define that we want to use nixpkgs, snowfall-lib, and deploy-rs.

# ./flake.nix (partial)
{
	inputs = {
		nixpkgs.url = "github:nixos/nixpkgs/nixos-24.05";

		snowfall-lib = {
			url = "github:snowfallorg/lib";
			inputs.nixpkgs.follows = "nixpkgs";
		};

		deploy-rs.url = "github:serokell/deploy-rs";
	};
}

Below we have the outputs as per the snowfall-lib documentation. We define where the src root is and some information for snowfall itself.

# ./flake.nix (partial)
{
	# ... inputs ...

	outputs = inputs:
		inputs.snowfall-lib.mkFlake {
			inherit inputs;

			src = ./.;

			snowfall = {
				namespace = "mypkgs";
				meta = {
					name = "my-awesome-flake";
					title = "My Awesome Flake";
				};
			};
		};
}

At this point with inputs and outputs, we would see our NixOS configuration with the nix flake show command. This shows that snowfall-lib has found our nixosConfiguration and it’s become available for us to use in the flake. (If you want to test this command, ensure you’ve merged the 2 partials above into 1.)

$ nix flake show
git+file:///home/philip/src/nix-config
├───darwinModules: unknown
├───deploy: unknown
├───homeModules: unknown
├───lib: unknown
├───nixosConfigurations
│   └───test: NixOS configuration
├───overlays
│   └───default: Nixpkgs overlay
├───pkgs: unknown
├───snowfall: unknown
└───templates

We can also check the configuration is valid.

$ nix flake check
warning: unknown flake output 'snowfall'
warning: unknown flake output 'deploy'
warning: unknown flake output 'pkgs'
warning: The check omitted these incompatible systems: aarch64-darwin, aarch64-linux, x86_64-darwin
Use '--all-systems' to check all.

If the configuration is not valid, you will get the typical Nix error. In this example I’ve temporarily removed the boot.loader.grub.device configuration to demonstrate an error.

$ nix flake check
warning: Git tree '/home/philip/src/nix-config' is dirty
warning: unknown flake output 'snowfall'
warning: unknown flake output 'deploy'
error:
       … while checking flake output 'nixosConfigurations'

       … while checking the NixOS configuration 'nixosConfigurations.test'
         at /nix/store/84m9nfg7kamv5va1fkk2iszbn5q2mavh-source/lib/mkFlake.nix:132:7:
          131|     {
          132|       ${host.output}.${reverseDomainName} = host.builder ({
             |       ^
          133|         inherit (host) system;

       … while calling the 'head' builtin
         at /nix/store/lv9bmgm6v1wc3fiz00v29gi4rk13ja6l-source/lib/attrsets.nix:1575:11:
         1574|         || pred here (elemAt values 1) (head values) then
         1575|           head values
             |           ^
         1576|         else

       … while evaluating the option `system.build.toplevel':

       … while evaluating definitions from `/nix/store/lv9bmgm6v1wc3fiz00v29gi4rk13ja6l-source/nixos/modules/system/activation/top-level.nix':

       (stack trace truncated; use '--show-trace' to show the full, detailed trace)

       error:
       Failed assertions:
       - You must set the option ‘boot.loader.grub.devices’ or 'boot.loader.grub.mirroredBoots' to make the system bootable.

The next part of the configuration is to make use of that NixOS configuration. I’m going to use deploy-rs for this, which is a tool that lets us easily manage remote (and local) systems configurations.

Below under the deploy keyword is the definition for deploy-rs. We are giving it a node called test with the configuration. The hostname is also test, a remote hostname to my local machine so that deploy-rs can SSH to that system. The configuration it deploys is the system profile.

# ./flake.nix (partial)
{
	# ... inputs ...

	outputs = inputs:
		inputs.snowfall-lib.mkFlake {

			# ... snowfall config ...

			deploy.nodes.test = {
				hostname = "test";
                interactiveSudo = true;
				profiles.system = {
					user = "root";
					path = inputs.deploy-rs.lib.x86_64-linux.activate.nixos inputs.self.nixosConfigurations.test;
				};
			};

			checks = builtins.mapAttrs (system: deployLib: deployLib.deployChecks inputs.self.deploy) inputs.deploy-rs.lib;
		};
}

The key part here is the path value inputs.self.nixosConfigurations.test. As shown with the nix flake show command, we have a NixOS configuration called test within nixosConfigurations. To pass this to deploy-rs, we specify this value. We do not need to explicitly import the files for test because snowfall-lib is doing that automatically for us. Let’s break it down some more.

inputs. We use this because we’re not defining every input at the outputs section.
- If we did not group inputs into a single inputs keyword then we’d have to define nixpkgs, snowfall-lib, deploy-rs, and self above.
- This grouping approach reduces the management of input keywords. It is common to have many imports.
self. This flake has the nixosConfiguration pulled in by snowfall-lib and is not in another flake.
nixosConfiguration. We’re using a NixOS configuration not a package or other.
test This is the folder name of the system.

Additionally, the deploy-rs documentation assumes it’s available at the keyword deploy-rs. However, due to the grouping of inputs under the inputs keyword, we have to prefix inputs. where we see deploy-rs.

It’s important that the version of deploy-rs CLI used matches the same deploy-rs within the flake configuration. This is easy to handle using Nix dev shells. As the dev shell is defined within the same flake it will use the same version of deploy-rs. To enter a dev shell simply run nix develop. I also alias deploy to simply d for convenience. If you see this error it’s probably a version mismatch error: Found argument '/nix/var/nix/profiles/system' which wasn't expected, or isn't valid in this context.

# ./flake.nix (partial)
{
	# ... inputs ...

	outputs = inputs:
		inputs.snowfall-lib.mkFlake {

			# ... snowfall config ...

            # ... system config ...

            # ... checks ...

            devShells.x86_64-linux.default = let
				pkgs = import inputs.nixpkgs {system = "x86_64-linux";};
			in
				pkgs.mkShell {
					packages = [
						inputs.deploy-rs.packages.${pkgs.system}.deploy-rs
					];
					shellHook = ''alias d="deploy"'';
				};
		};
}

Great. Now let’s pull all the bits of flake.nix together.

# ./flake.nix
{
	inputs = {
		nixpkgs.url = "github:nixos/nixpkgs/nixos-24.05";

		snowfall-lib = {
			url = "github:snowfallorg/lib";
			inputs.nixpkgs.follows = "nixpkgs";
		};

		deploy-rs.url = "github:serokell/deploy-rs";
	};

	outputs = inputs:
		inputs.snowfall-lib.mkFlake {
			inherit inputs;

			src = ./.;

			snowfall = {
				namespace = "mypkgs";
				meta = {
					name = "my-awesome-flake";
					title = "My Awesome Flake";
				};
			};

			deploy.nodes.test = {
				hostname = "test";
                interactiveSudo = true;
				profiles.system = {
					user = "root";
					path = inputs.deploy-rs.lib.x86_64-linux.activate.nixos inputs.self.nixosConfigurations.test;
				};
			};

			checks = builtins.mapAttrs (system: deployLib: deployLib.deployChecks inputs.self.deploy) inputs.deploy-rs.lib;

            devShells.x86_64-linux.default = let
				pkgs = import inputs.nixpkgs {system = "x86_64-linux";};
			in
				pkgs.mkShell {
					packages = [
						inputs.deploy-rs.packages.${pkgs.system}.deploy-rs
					];
					shellHook = ''alias d="deploy"'';
				};
		};
}

Part 2: Setup VM

This post does not cover how to set up a VM or other target system. I have manually created a new VM using virt-manager. It is customised with a network bridge name on the host (br0) and the vdisk serial set to os for easier identification. I performed a NixOS installation using a live CD and installed with no desktop environment. Using the console access provided by virt-manager I have enabled some settings. NixOS by default installs with the configuration files in /etc/nixos/.

# /etc/nixos/configuration.nix
{
	# .., existing config ...
	networking.hostName = "test";

	services.openssh.enable = true;

	nix.settings.trusted-users = ["root" "@wheel"];
	
	security.sudo.extraRules = [
		{
			users = ["philip"];
			commands = [
				{
					command = "ALL";
					options = ["NOPASSWD" "SETENV"];
				}
			];
		}
	];
	# .., existing config ...
}

After an apply with nixos-rebuild switch I can successfully SSH and sudo without repeated password prompts to the hostname of test. (My home network is configured to create DNS records for new clients. If yours does not, create one or identify the IP address of the VM.)

It is possible to generate bootable images with custom configuration with nixos-generators. I've used this for generating raspberry pi images as they nave no remote console. However, for this post we are not at that stage yet and nixos-generators will not be covered.

Part 3: Copy baseline config

Before getting creative we want to ensure that we can deploy to the VM without any problems. We will copy the /etc/nixos/hardware-configuration.nix and /etc/nixos/configuration.nix into our config and deploy that.

# ./systems/x86_64-linux/test/hardware.nix
{
	lib,
	modulesPath,
	...
}: {
	imports = [
		(modulesPath + "/profiles/qemu-guest.nix")
	];

	boot.initrd.availableKernelModules = ["ahci" "xhci_pci" "virtio_pci" "sr_mod" "virtio_blk"];
	boot.initrd.kernelModules = [];
	boot.kernelModules = ["kvm-intel"];
	boot.extraModulePackages = [];

	fileSystems."/" = {
		device = "/dev/disk/by-id/virtio-os-part1";
		fsType = "ext4";
	};

	swapDevices = [];

	networking.useDHCP = lib.mkDefault true;

	nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";
}

# ./systems/x86_64-linux/test/default.nix
{
	lib,
	namespace,
	...
}:
with lib;
with lib.${namespace}; {
	imports = [./hardware.nix];

	boot.loader.grub.enable = true;
	boot.loader.grub.device = "/dev/disk/by-id/virtio-os";
	boot.loader.grub.useOSProber = true;

	networking.networkmanager.enable = true;

	time.timeZone = "Europe/London";

	i18n.defaultLocale = "en_GB.UTF-8";
	i18n.extraLocaleSettings = {
		LC_ADDRESS = "en_GB.UTF-8";
		LC_IDENTIFICATION = "en_GB.UTF-8";
		LC_MEASUREMENT = "en_GB.UTF-8";
		LC_MONETARY = "en_GB.UTF-8";
		LC_NAME = "en_GB.UTF-8";
		LC_NUMERIC = "en_GB.UTF-8";
		LC_PAPER = "en_GB.UTF-8";
		LC_TELEPHONE = "en_GB.UTF-8";
		LC_TIME = "en_GB.UTF-8";
	};

	services.xserver.xkb = {
		layout = "gb";
		variant = "";
	};
	console.keyMap = "uk";

	users.users.philip = {
		isNormalUser = true;
		description = "Philip";
		extraGroups = ["networkmanager" "wheel"];
	};

	networking.hostName = "test";

	services.openssh.enable = true;

	nix.settings.trusted-users = ["root" "@wheel"];
	security.sudo.extraRules = [
		{
			users = ["philip"];
			commands = [
				{
					command = "ALL";
					options = ["NOPASSWD" "SETENV"];
				}
			];
		}
	];

	system.stateVersion = "24.11";
}

Part 4: Deploy

We can now deploy this configuration to confirm it can be done remotely with deploy-rs.

$ nix develop
$
$ deploy
🚀 ℹ️ [deploy] [INFO] Running checks for flake in .
warning: Git tree '/home/philip/src/nix-config' is dirty
warning: unknown flake output 'snowfall'
warning: unknown flake output 'deploy'
warning: unknown flake output 'pkgs'
warning: The check omitted these incompatible systems: aarch64-darwin, aarch64-linux, x86_64-darwin
Use '--all-systems' to check all.
🚀 ℹ️ [deploy] [INFO] Evaluating flake in .
warning: Git tree '/home/philip/src/nix-config' is dirty
🚀 ⚠️ [deploy] [WARN] Interactive sudo is enabled! Using a sudo password is less secure than correctly configured SSH keys.
Please use keys in production environments.
🚀 ℹ️ [deploy] [INFO] You will now be prompted for the sudo password for test.
(sudo for test) Password:
🚀 ℹ️ [deploy] [INFO] The following profiles are going to be deployed:
[test.system]
user = "root"
ssh_user = "philip"
path = "/nix/store/1a9jxc82a28x4dwk82bsghhnlihpmbii-activatable-nixos-system-test-24.05.20241230.b134951"
hostname = "test"
ssh_opts = []

🚀 ℹ️ [deploy] [INFO] Building profile `system` for node `test`
🚀 ℹ️ [deploy] [INFO] Copying profile `system` to node `test`
🚀 ℹ️ [deploy] [INFO] Activating profile `system` for node `test`
🚀 ℹ️ [deploy] [INFO] Creating activation waiter
⭐ ℹ️ [activate] [INFO] Activating profile
👀 ℹ️ [wait] [INFO] Waiting for confirmation event...
updating GRUB 2 menu...
Warning: os-prober will be executed to detect other bootable partitions.
Its output will be used to detect bootable binaries on them and create new boot entries.
lsblk: /dev/mapper/no*[0-9]: not a block device
lsblk: /dev/mapper/block*[0-9]: not a block device
lsblk: /dev/mapper/devices*[0-9]: not a block device
lsblk: /dev/mapper/found*[0-9]: not a block device
installing the GRUB 2 boot loader on /dev/disk/by-id/virtio-os...
Installing for i386-pc platform.
Installation finished. No error reported.
activating the configuration...
setting up /etc...
reloading user units for philip...
restarting sysinit-reactivation.target
⭐ ℹ️ [activate] [INFO] Activation succeeded!
⭐ ℹ️ [activate] [INFO] Magic rollback is enabled, setting up confirmation hook...
👀 ℹ️ [wait] [INFO] Found canary file, done waiting!
⭐ ℹ️ [activate] [INFO] Waiting for confirmation event...
🚀 ℹ️ [deploy] [INFO] Success activating, attempting to confirm activation
🚀 ℹ️ [deploy] [INFO] Deployment confirmed.

Run deploy a second time. For the first run the VM has the configuration.nix applied, which was a success. If you did not copy configuration over properly this apply may have undone a needed option. Running a second time tests with the configuration defined by our flake applied.

I hope that this helps you jump ahead in your journey with these tools. My next steps are to start converting my configuration over into here and point deploy-rs at a real system. I’m a big fan of Nix, and in particular NixOS.

Please check out these resources which I found very helpful.