Impermanence on NixOS with ZFS and tmpfs

Tags: Impermanence, ZFS, Guide

What is Impermanence

It wipes your /root on reboot and your startup is a blank canvas, but you can persist mounts and bind mount directories from it in your normal root to save stuff like cache and tokens. So you wipe all the junk and save actually useful stuff.

For example you can install full KDE Plasma session, run it, and if you get bored. Just disable it and no KDE junk left.

Important to note: That impermanence of my setup uses tmpfs, so it writes /root to RAM, so nothing actually gets erased on the Disk. Meaning no continuous I/O rewrites wearing out your Drive(not that it would mater in practice). But the state isn't saved between reboots, as with anything on RAM

Also when I refer to /root, it's actually the whole /, not just root user directory.

Why did you set it up

I was bored. I don't find benefits of impermanence so crucial to completely overhaul how your system behaves and I don't trust myself to maintain it.

But there are some benefits to it:

  • I only backup important files, no cache, no states, only files and media;
  • I always know what's on my system because it's declared in the config;
  • It opens up possibilities to experiment more with my system, because if I could setup impermanence and not loose all my files, I am unstoppable;

What's the meaning of writing this page?

It's not that hard to setup impermanence, but to requires reading a lot of stuff, and if you don't use ZFS or BTRFS even full reinstall for rearranging partitions. I have read several articles, watched videos, and stole code from many GitHub repos.

Plus most guides just go to the wipe stage right away, without saying how to persist, or how it practically works for the user to not loose their files. I will try to compete in these aspects.

What is your current setup?

I got ZFS with tmpfs, with 2 persistence datasets. /cache and /persist. And a plaint vFAT /boot partition, where GRUB or Systemd-boot will put generation images.

Most people assume you gotta have 64Gb or RAM for tmpfs to be convenient, but actually you just need a well structured disk layout. Then the tmpfs usually only takes 50MB to 100MB of RAM.

Template structure of ZFS datasets that will be essential:

  • /cache is for rust targets, everything in ~/.cache, .local states, etc.
  • /persist is for Media, browser profiles, Projects, etc. This is the only datasets that get's backed up by sanoid.
  • /nix for /nix/store. NixOS won't boot without it. All the files that aren't persisted, but appear on my system are symlinked from /nix. That includes config files and services.
  • /tmp for /tmp. yeah, anyways it's to not overload tmpfs when downloading something on browser. with boot.tmp.cleanOnBoot = true; it is cleared on boot anyways.

tmpfs is erased on reboot, so / and everything below it, including /home is gone, unless put into /cache or /persist datasets. tmpfs is on RAM, so it can overload if exceeds certain size, to prevent that I got several more zfs datasets, that aren't persisted, meaning they don't have connection to files in other datasets, but aren't erased by default.

What we need?

A ZFS setup

This isn't a ZFS guide so, unless you have ZFS setup on your NixOS, you can kiss this Guide goodbye. Go read OpenZFS documentation. But here is my installation script in case you'd like more guiding. But seriously, go read docs, real engineers know a lot more than I do.

permalink in the github repo, please insure the branch is up to date before checking in, it contains actual descriptions to each line as a Norg file format that you can tangle. Like bible in org mode.

sudo zpool create -f \
-o ashift=12 \
-o autotrim=on \
-O compression=zstd \
-O acltype=posixacl \
-O atime=off \
-O xattr=sa \
-O normalization=formD \
-O mountpoint=none \
zroot "/dev/sda2"

sudo zfs create -o mountpoint=legacy zroot/root
sudo mount -t zfs zroot/root /mnt

sudo mount --mkdir "$BOOTDISK" /mnt/boot
# All the stuff below will be explained later
sudo zfs create -o mountpoint=legacy zroot/nix
sudo mount --mkdir -t zfs zroot/nix /mnt/nix

sudo zfs create -o mountpoint=legacy zroot/tmp
sudo mount --mkdir -t zfs zroot/tmp /mnt/tmp

sudo zfs create -o mountpoint=legacy zroot/cache
sudo mount --mkdir -t zfs zroot/cache /mnt/cache

sudo zfs create -o mountpoint=legacy zroot/persist
sudo zfs snapshot zroot/persist@blank
sudo mount --mkdir -t zfs zroot/persist /mnt/persist
# All the stuff above will be explained later
sudo nixos-install --no-root-password --flake "github:Ladas552/Nix-Is-Unbreakable#NixVM"

Partitions

A new way to manage your system. NixOS.

Tho you probably already use NixOS if you are reading this, if you don't then get out while you can.

On a more serious note, you need ZFS setup, with 2 particular datasets.

fileSystems = {
  "/nix" = {
    device = "zroot/nix";
    fsType = "zfs";
  };
  "/tmp" = {
    device = "zroot/tmp";
    fsType = "zfs";
  };
};

If you don't have them, but have ZFS installed, just create them using commands

sudo zfs create -o mountpoint=legacy zroot/tmp
sudo zfs create -o mountpoint=legacy zroot/nix

This will insure that you won't delete your /nix/store and it stays intact between reboots. And for this particular setup the tmp dataset will be used so our tmpfs root will insure that it won't randomly overload.

Impermanence module

The Impermanence module is a NixOS flake that creates mount binds. The main purpose of it is to just put stuff in special /persist dataset, and still be able to access it from /root and /home. More about technicality of bind mounts later

Basically you define certain directories names in it, and it creates them, then binds them to specific relevant locations, like ".config/nvim" will be located in ~/.config/nvim. And if you put your Neovim config there, neovim will still follow the config, but it will be located on different dataset, and won't be wiped on boot.

Neat right? Not really, because if directory already exists, Impermanence will override that old directory with new empty one. Don't panic. Data isn't lost, it was just reallocated, you can delete the directory from impermanence module and it will comeback.

That's the main reason why most people reinstall their OS if they want to use Impermanence, because it's a pain in the glands to move the files from directories before persisting it and moving things back. There are projects that circumvent that, but I didn't use them. For example: Persist-retro.

Also to persist an individual file, you need to move the file, and manually copy it to persist directory. Otherwise it complains about the original file being in the way of a mount bind.

You forgot to tell installation instructions

It's nix so here is just a snippet of code. Works for flakes.

#flake.nix
{
  inputs.impermanence.url = "github:nix-community/impermanence";
}

And then just import the module, like:

imports = [
  inputs.impermanence.nixosModules.impermanence
];

We will only use the nixosModule because I don't have standalone Home-Manager and not planning to adopt impermanence for distros outside of NixOS.

I have seen people use impermanence module on non flake setups, but I am not so interested in them to find and link a good one.

Immutable users

As we delete everything in /root, it means passwords for users, and most importantly root user will be deleted.

So just make them immutable. You can store the password file in sops, or just provide raw path from /persist directory.

{
  # setup immutable users for impermanence

  # silence warning about setting multiple user password options
  # https://github.com/NixOS/nixpkgs/pull/287506#issuecomment-1950958990
  # Stolen from Iynaix https://github.com/iynaix/dotfiles/blob/4880969e7797451f4adc3475cf33f33cc3ceb86e/nixos/users.nix#L18-L24
  options = {
    warnings = lib.mkOption {
      apply = lib.filter (
        w: !(lib.hasInfix "If multiple of these password options are set at the same time" w)
      );
    };
  };

  config = {
    # disabling user mutability
    users.mutableUsers = false;

    # defining regular user, ME!
    users.users.ladas552 = {
      isNormalUser = true;
      description = "Ladas552";
      extraGroups = [
        "networkmanager"
        "wheel"
      ];
      initialPassword = "pass";
      # Use a path or your  encryption method here
      hashedPasswordFile = config.sops.secrets."mystuff/host_pwd".path;
    };

    nix.settings.trusted-users = [ "ladas552" ];

    # Setting root user
    users.users.root = {
      initialPassword = "pass";
      hashedPasswordFile = config.sops.secrets."mystuff/host_pwd".path;
    };
  };
}

Other features for immutable users:

  • Can use --no-root-password flag in nixos-install command. Meaning you don't ever have to monitor it, it will install password automatically.
  • Can't use passwd <user> command. So if you mess up your password path the first time, you have to reboot to previous generation to set it correctly.

The initialPassword is set as plain text because it suppose to be a backup if sops decryption failed, so you won't leave with useless system state. Otherwise, it's unused and won't have security implications for your host.

When do we start deleting stuff?

Not so fast bakaru, we first need to save our stuff.

So you need to create persisted directories

sudo zfs create -o mountpoint=legacy zroot/persist
sudo zfs create -o mountpoint=legacy zroot/cache

And now we add them to be mounted on boot

# persist mount
fileSystems."/persist" = {
  device = "zroot/persist";
  fsType = "zfs";
  # so it's required to boot, and you won't reboot into empty desktop
  neededForBoot = true;
};

# cache are files that should be persisted, but not to snapshot
# e.g. npm, cargo cache etc, that could always be redownload
"/cache" = {
  device = "zroot/cache";
  fsType = "zfs";
  neededForBoot = true;
};

I also recommend setting up backups with sanoid if you didn't already.

services.sanoid = {
  enable = true;
  # if you have sanoid options somewhere else, lib.mkForce
  # will override anything, so you only have snapshots that matter
  datasets = lib.mkForce {
    "zroot/persist" = {
      hourly = 50;
      daily = 15;
      weekly = 3;
      monthly = 1;
    };
  };
};

Now we have basic datasets that will store out stuff, Impermanence can wait now, we need to assign bind mounts for directories and files!

Don't know what bind mounts are? Well simply put. They are mounts that make some directory to appear in normal location, but actually it's in a different dataset all together.

So for example: /home/alice225/Downloads will be deleted on boot. But, not it's content. On the next boot, the content of Downloads that is in /persist dataset will remount itself to /home/alice225/Downloads path.

It will appear seamless to other applications and to yourself. But now you can access the same files in both /home/alice225/Downloads and in /persist/home/alice225/Downloads. And remember, it is not a symlink. Symlinks fool programs, while bind mounts genuinely make files accessible in several locations. But be careful, because they share permissions, and can also be deleted.

Okay, we have locations, let's move some files into them

So, with Impermanence module, there are some options available. Simplest approach would be to use it directly

environment.persistence = {
  # one of out datasets
  "/persist" = {
    # useful option
    hideMounts = true;
    directories = [
      # absolute path to directories in string values
      "/var/log"
      "/var/lib/nixos"
      "/etc/NetworkManager/"
    ];
  };
};

Now you are able to just define your paths as normal and save them. But this is just normal impermanence, This blog post is about My setup specifically, so how about we add some abstractions to the vanilla scheme.

New options

directories and files in impermanence module are just a list of string, so we can make pseudo options to add more strings to the list and ++ them with actual persistence option, or just reference our list.

It will make it easier to define directories under different scopes. For example, setting files in home.nix file, but they will be used in configuration.nix file.

{lib,...}:
{
  options = {
    # options to put directories in, persistence but shortened
    # stolen from @iynaix
    root = {
      directories = lib.mkOption {
        type = lib.types.listOf lib.types.str;
        default = [ ];
        description = "Directories to persist in root filesystem";
      };
      files = lib.mkOption {
        type = lib.types.listOf lib.types.str;
        default = [ ];
        description = "Files to persist in root filesystem";
      };
      cache = {
        directories = lib.mkOption {
          type = lib.types.listOf lib.types.str;
          default = [ ];
          description = "Directories to persist, but not to snapshot";
        };
        files = lib.mkOption {
          type = lib.types.listOf lib.types.str;
          default = [ ];
          description = "Files to persist, but not to snapshot";
        };
      };
    };
    home = {
      directories = lib.mkOption {
        type = lib.types.listOf lib.types.str;
        default = [ ];
        description = "Directories to persist in home directory";
      };
      files = lib.mkOption {
        type = lib.types.listOf lib.types.str;
        default = [ ];
        description = "Files to persist in home directory";
      };
      cache = {
        directories = lib.mkOption {
          type = lib.types.listOf lib.types.str;
          default = [ ];
          description = "Directories to persist, but not to snapshot";
        };
        files = lib.mkOption {
          type = lib.types.listOf lib.types.str;
          default = [ ];
          description = "Files to persist, but not to snapshot";
        };
      };
    };
  };
}
# Holy cow nix is indented to all suns

You can add more options if need be. In this we only define lists for /persist and /cache, but they are separated into root and home. So add root and home options to nixocConfiguration and add only home to home-manager for example.

home is just a simple way to separate directories in /home/ladas552 from just /.

But, you know, I use flake-parts and I don't need to add these options to different files and scope. I can just inherit them all in one file!

{ lib, ... }:
# link to snippet in my config
# https://github.com/Ladas552/Flake-Ocean/blob/85ee207aa2e5e0d2e44aad0a0818a533ceca72cf/modules/nixosModules/Impermanence/imp-options.nix
{
  flake.modules =
    let
      # options to put directories in, persistence but shortened
      # stolen from @iynaix

      root = {};
      home = {};
      # Same thing as above
    in
      {
      nixos.options.options.custom.imp = { inherit root home; };
      hjem.options.options.custom.imp = { inherit home; };
      homeManager.options.options.custom.imp = { inherit home; };
    };
}

This code block is from my Dendrithic config with several module classes. So each nixos, hjem and homeManager classes have their own options module that inherit the same type of options in each module scope.

You probably didn't get any of that, but you don't need to tbh. My setup is My setup, do whatever you want.

Persist in action

Now we can define some important directories and files to persist

custom.imp = {
  root = {
    directories = [
      "/etc/NetworkManager/"
      "/var/lib/NetworkManager"
      "/var/lib/iwd"
    ];
  };
  home = {
    directories = [
      ".librewolf"
    ];
    cache = {
      files = [ ".local/share/com.jeffser.Alpaca/alpaca.db" ];
      directories = [
        ".local/share/nvim"
        ".local/state/nvim"
        ".config/libreoffice"
        ".cache/librewolf"
        ".cache/keepassxc"
        ".config/keepassxc"
        ".cache/nix"
        ".cache/nix-index"
      ];
    };
  };
};

But if you set this thing up, and rebuild it wouldn't do a thing. Remember, these options and list are just place holders. Meant to be easy to write and read. Now we gotta add them to an actual Impermanence module options.

{ lib, config, ... }:
let
  cfg = config.custom.imp;
  # config.custom.meta.user is just my placeholder for `username`
  # just hard code the value, or replace it with your own solution to select a user
  cfghm = config.home-manager.users."${config.custom.meta.user}".custom.imp;
  cfghj = config.hjem.users."${config.custom.meta.user}".custom.imp;
in
  {
  environment.persistence = {
    "/persist" = {
      hideMounts = true;
      # referencing files via our abstraction option
      files = lib.unique cfg.root.files;
      directories = lib.unique (
        # here you can define directories normally 
        [
          "/var/log"
          "/var/lib/nixos"
        ]
        # and concatenate too!
        ++ cfg.root.directories
      );
      # add persists to `/home/user` path
      users."${config.custom.meta.user}" = {
        files = lib.unique ([ ] ++ cfghm.home.files ++ cfghj.home.files);
        directories = lib.unique (
          [ ] ++ cfg.home.directories ++ cfghm.home.directories ++ cfghj.home.directories
        );
      };
    };
    # same as above
    "/cache" = {
      hideMounts = true;
      files = lib.unique cfg.root.cache.files;
      directories = lib.unique cfg.root.cache.directories;
      users."${config.custom.meta.user}" = {
        files = lib.unique (cfg.home.cache.files ++ cfghm.home.cache.files ++ cfghj.home.cache.files);
        directories = lib.unique (
          cfg.home.cache.directories ++ cfghm.home.cache.directories ++ cfghj.home.cache.directories
        );
      };
    };
  };
}

You will need to adjust this code snippets to your own config. For example, if you don't use hjem. And replace config.custom.meta.user with your own username.

I am not stating this approach is the best, but it just how I ended up using Impermanence module, and it might be useful for you to know.

Now upon rebuild Impermanence module should add all the interesting directories to datasets and establish bind mounts.

Wait, did we forget something? Uhhh...

DELETE ERASE REDUCTED

# replace the root mount with tmpfs
# wipes everything if you don't have proper persists, be warned
fileSystems."/" = lib.mkForce {
  device = "tmpfs";
  fsType = "tmpfs";
  neededForBoot = true;
  options = [
    "defaults"
    # whatever size feels comfortable, smaller is better
    "size=1G"
    "mode=755"
  ];
};

This is all you need. So simple in comparison to the whole page above, right?

The main reasons for that are:

  • It's harder to keep what you have gained, but trivial to loose everything you ever had;
  • Also the size=1G wouldn't make it possible to use tmpfs as a main without persists and bind mounts. Bind mount only makes files accessible in 2 locations, but only stored in ZFS dataset.

This should be all you need to start using tmpfs for impermanence on ZFS.

There are some niceties I want to share tho, to make your life easier.

Some sprinkles to your epitome of agony

Snippets

Set this so you aren't lectured by you know what you are doing lecture from sudo every boot

security.sudo.extraConfig = "Defaults lecture=never";

If you use sops-nix, set ssh paths to /persist because otherwise nixos-install won't find the keys.

sops.age.sshKeyPaths = [
  "/persist/home/vimjoyer/.ssh/ssh-key"
];
sops.age.keyFile = lib.mkDefault "/persist/home/vimoyer/.config/sops/age/keys.txt";

Persist everything

Persist every bit that might be useful to you, tokens, cookies and all that if they matter to you. Depending on the application, you might wanna persist only one file. But for something like steam, it compiles shader cache, which is persistable with this snippet

# persist steam
custom.imp.home = {
  cache.directories = [
    ".local/share/Steam"
    ".cache/mesa_shader_cache"
    ".cache/mesa_shader_cache_db"
    ".cache/radv_builtin_shaders"
  ];
};

But how would you know what to persist? Well, first you might want to look at others people config files. Because the best way to avoid pit falls is following the walked road.

Some pretty extended persist list can be found in following configurations:

If they don't use the same modules as you, figure it out on your own. Most of the time programs follow xdg conventions and store files in .config .cache .local/state .local/share. Or instead of persisting the settings, symlink raw files.

Credits and suggestions

You can suggest anything you'd like to add to Some sprinkles to your epitome of agony. Cool configs using impermanence, tips, snippets and so on. Just ping me on Discord or write an issue on github. Your username will be added below as a privilege for being so awesome!

List of awesome people:

  • Iynaix's impermanence abstraction structure and ZFS setup
  • Vimjoyer's Impermanence video introducing basic concept to me
  • talyz for making Impermanence module and extensive readme for the project
  • Flipus and snohater for providing feedback on readability of the thing.
  • You for reading this all, good luck with your NixOS config. Hope this article helped you the same way all the people mentioned above helped me. Without such a vast community, I wouldn't be able to figure all these things out. And I didn't reinvent a wheel, it's all the work of Open Source contributors. So, hopefully you also share your knowledge in the future. Improve on what's old or reduce it to atoms, it's up to you. It's your setup, and only for you to decide, whether you really want to setup it up.