Borg Backup for NixOS!

After a scary incident, I got reminded of how important backups are! My computer runs on NixOS with Hyprland window manager. In an attempt to reclaim some space that all my downloaded games and others had eaten up, after i saw that only 56gb was free for my new games from HumbleBundle.

I did not have a tool to easily get an overview of my system, and decided to run one of the multiple alternatives through a nix shell. Using nix shells lets me use programs without adding them to the system configurations. The tool selected for the job was "Baobab", a disk usage analyzer made for gnome. This was easily started with nix-shell -p baobab --run baobab and the GUI tool pops up.

This tools have a great visualization on what, and where the gigabytes gets eaten up, by forgotten games, downloaded VMs for school and a bunch of ISOs to install VMs.

Where the trouble began

When I found unnecessary files located with baobab, i could right click and "move to wastebasket". This seems simple and fine! But when I tried that method it seemed to me that nothing happened. And after attempting two-three more times i gave up on that method, since nothing happened.. Nothing happened right?... right? Oh boy something happened, at midnight of course when I was tired, and not up for the task. But i did not notice at this point in time.

After I had cleaned up space with other methods, uninstalled games from Steam and Epic games, removed large VM files downloaded for school and other random stuff, I decided to play some Palworld and just relax the rest of the evening. But after experiencing lagging in both the game, and on my music playing in the browser I decided to reboot the computer.

And OH BOY was I in for a surprise when I logged back in! Where was my bookmarks?, where was my cookies and session data? Have we logged in to firefox to sync data? Of course not! During my investigation I opened neovim to look at configuration files, and I was presented with an error saying that I need at least one workspace for my obsidian plugin. Baffled, as I have that, it is located in my Jottacloud sync folder. Sadly I did not have my watch on to monitor my heart rate as I found that the ENTIRE JOTTACLOUD folder was gone, vanished, extinct. Oh atleast it lives in the famous cloud, as I logged in and panicked as the sync folder was WIPED OUT!

Luckily, Jottacloud stores deleted files in the trash for 30 days before they are actually deleted! So to reclaim those files it was a simple right-click and restore! Good, as this is where ALL my school notes and entire Obsidian vault lives! The next issue was all my Firefox bookmarks, and session history.

After some investigation and help of the find command i was able to locate the files, in what baobab decided was the wastebasket. I have a bad habit of using rm to delete files, and those do not go through a recycling process! Inside .local/share/Trash/files I found the missing Jottacloud folder, but that was already restored with jottacloud, and my .mozilla folder with browser data, such as profiles, bookmarks and session data. But I also noticed my .chache folder here - and that explains why my rofi was suggesting weird stuff instead of my most used ones.

Future Proofing!

As this encounter gave me a minor heart attack, it was time to Borg backup, as i have good experience from it when i was using Pop!_OS. Borg backup is a deduplicating archive solution that can compress and encrypt your data. With deduplication new backups will just reference already backed up data if it has not changed. So with each new backupp only changed files are written, the rest is referenced.

But with NixOS and its nature of having everything in configuration files the setup method was quite a bit different from my previous experience with it. In Pop!_OS I had a custom bash script that was ran by a cronjob. But with NixOS we can very easily set up systemd jobs and timers with the nixos configuration.

By looking at the NixOS wiki I saw that this was looking to be easier than my prior experience! I created a separate module that I imported to my main config for my desktop. And I wrote the config with my laptop in mind also, I used the hostname as a part of the repo path. I have copied most of the common-excludes from the wiki, and added some of my own. I have also set up both a local and remote job, as I want to backup to two location. As of the time of writing only remote is configured and working. My BORG_RSH is much shorter than the example from the wiki, this is because I Have added an entry in my .ssh/config file named BORG that handles address and key.

I have also added a prune policy, to control the size of the backup directory. My policy let me keep the 7 latest daily backups, the 4 latest weekly backup. And just for safety the last 2 monthly backups.

With this simple configuration file, NixOS creates a systemd job and timer that backs up my home folder, except the exclusions to my NAS. A MUCH simpler way to back up than my previous bash script. The Wiki also showed how to wait for network before attempting backup, and a notifier if backup fails. This will be implemented, but it was much more important to get the backups rolling.

As with all backups it is important to test that they work! And with the borgfs command a read-only directory can be mounted of a specific backup from the repository! And this was a success!

{
  config,
  lib,
  pkgs,
  ...
}: let
  common-excludes = [
    ".cache"
    "*/cache2"
    "*/Cache"
    ".config/Code/CachedData"
    ".container-diff"
    ".npm/_cacache"
    "*/node_modules"
    "*/bower_components"
    "*/_build"
    "*/.tox"
    "*/venv"
    "*/.venv"
    "Downloads"
    "virtualMachines"
    "vmware"
    "AI"
    ".local/share/Steam"
    "Games"
  ];

  backupPath = "/home/fredrik";

  localBorgJob = name: {
    encryption.mode = "none";
    extraCreateArgs = "--verbose --stats --checkpoint-interval 600";
    repo = "/mnt/backup-drive/${name}";
    compression = "zstd,1";
    startAt = "daily";
    user = "fredrik";
  };

  remoteBorgJob = name: {
    encryption.mode = "none";
    environment.BORG_RSH = "ssh";
    environment.BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK = "yes";
    environment.BORG_RELOCATED_REPO_ACCESS_IS_OK = "yes";
    extraCreateArgs = "--verbose --stats --checkpoint-interval 600";
    repo = "ssh://BORG//mnt/borg/${config.networking.hostName}/${name}";
    compression = "zstd,1";
    startAt = "daily";
    user = "fredrik";
  };

  prunePolicy = {
    within = "1d";
    daily = 7;
    weekly = 4;
    monthly = 2;
  };
in {
  config = {
    services.borgbackup.jobs = {
      # home-local =
      #   localBorgJob "backups/station/home-local"
      #   // rec {
      #     paths = backupPath;
      #     exclude = map (x: paths + "/" + x) common-excludes;
      #     prune.keep = prunePolicy;
      #   };

      home-remote =
        remoteBorgJob "backups/station/home-remote"
        // rec {
          paths = backupPath;
          exclude = map (x: paths + "/" + x) common-excludes;
          prune.keep = prunePolicy;
        };
    };
  };
}

The lesson have been learned, backup important files! When i set up the local job, i will have have backup to a separate SSD in the desktop, one backup to my NAS, and one backup to Jottacloud. Better safe than sorry!