Files
hermenix/docs/RECOVERY.md
T
2026-10-10 22:06:12 -04:00

6.5 KiB

Snapshots

Default frequency: hourly (one snapshot per hour).
Retention: the last 48 snapshots are kept (~2 days).
What is snapshotted: the state disk only. The system disk is never snapshotted (it is rebuilt with hermenix setup in minutes).

Snapshots are created by a systemd timer (hermes-vm-snapshot.timer) that runs a oneshot service calling btrfs subvolume snapshot -r. The timer respects services.hermenix.snapshots.interval (a normal systemd calendar expression) and has Persistent = true + a small randomized delay.

While live snapshots exist on the nodatacow state image, copy-on-write is re-enabled for any blocks the snapshots still reference. This increases fragmentation and the amount of data that appears changed for backups. That is why the default retention (48) is deliberately conservative.

You can change the cadence and how many to keep:

services.hermenix.snapshots = {
  interval = "hourly";   # e.g. "*:0/15" for every 15 min, "daily", etc.
  keep = 48;
};

Inspecting a broken guest

Use these to diagnose before doing anything destructive:

hermenix vm console                    # watch boot / journal (read-only)
hermenix vm ssh                         # interactive shell (vsock)
hermenix vm config                      # current generation, NixOS generations, git log + status of /etc/nixos
hermenix vm config-pull ./guest-live    # copy live /etc/nixos for diffing
hermenix ctl soul                       # what the agent currently thinks it is
hermenix ctl status                     # gateway unit health

Inside the guest (via hermenix vm ssh):

lsblk -f
sudo nix-env --list-generations -p /nix/var/nix/profiles/system
git -C /etc/nixos log --oneline -10
git -C /etc/nixos status --short
journalctl -b -1 -p err   # previous boot errors

Common recovery paths

1. Bad NixOS generation (most common)

The previous generation is still in the boot menu.

hermenix vm ssh "sudo nixos-rebuild switch --rollback"
# or pick an older generation from the bootloader menu on the serial console

Commit before risky changes (inside guest):

git -C /etc/nixos add -A
git -C /etc/nixos commit -m "before risky change"
sudo nixos-rebuild switch --flake /etc/nixos

2. Guest won't boot at all (system disk problem)

The system disk is deliberately disposable. It holds only the NixOS install and the Nix store. Everything that matters (identity, /etc/nixos, workspace, Hermes home) lives on the state disk.

CLI procedure to recover a bricked guest (re-image only the system disk):

  1. Start the recovery:

    hermenix setup
    
    • It detects existing disks and skips creation.
    • It still builds the image and writes it to the system disk only.
    • State disk (and everything under /var/lib/hermes) is left untouched.
  2. Watch it come back:

    hermenix vm console          # follow early boot / journal
    
  3. Get inside and finish setup:

    hermenix vm ssh
    
    # inside the guest
    lsblk -f                     # confirm labels match guest/hardware.nix
    determinate-nixd login
    sudo nix flake lock /etc/nixos
    

Because the state disk was untouched, the agent should come back with its previous SOUL, memories, config, and /etc/nixos.

3. State disk is also suspect / want a truly clean slate

Use --clean:

hermenix setup --clean

This deletes the system image, state image, and UEFI vars, then walks through a full fresh setup. It also removes the stale known_hosts entry for the guest (a new state disk gets a new SSH host key).

Warning: this destroys the current state disk contents. Use backups or btrfs snapshots first if you want any chance of recovering data.

4. Use a recent btrfs snapshot

hermenix vm snapshots
hermenix vm rollback state-2026...
# type 'rollback' to confirm

This replaces the live state subvolume with the snapshot. The previous state is kept as state.old-<timestamp> so you can still recover from a mistaken rollback.

Btrfs + nodatacow note: While snapshots exist, CoW is re-enabled on referenced blocks. This can increase fragmentation on the nodatacow image. Prune them when you're happy.

5. Restore from restic backup

Backups are file-level tars of /var/lib/hermes, not raw disk images.

After you have a working guest (either via re-image or rollback):

hermenix vm ssh
# then on the guest or via host tools, restore the restic snapshot you want
# into /var/lib/hermes (or a fresh directory) and adjust services.hermenix.hermesHome if needed

See the backup configuration in your host flake for the repository and credentials.

Starting completely fresh (nuclear option)

hermenix setup --clean

Deletes system + state images + UEFI vars, drops the stale host key entry, then does a full guided setup again. Subvolumes and old snapshots are left on the btrfs volume.

Quick reference of relevant CLI commands

Command Use for recovery
hermenix setup Re-image system disk only (recommended for bricked guest)
hermenix setup --clean Full wipe of both disks + fresh start
hermenix vm console Watch boot / early errors
hermenix vm ssh Get inside to run generations, git, journalctl
hermenix vm config See running generation + uncommitted /etc/nixos changes
hermenix vm config-pull Pull live config for offline diffing
hermenix vm snapshot Trigger a manual state snapshot before risky work
hermenix vm rollback <name> Restore state disk from snapshot (destructive)
hermenix vm backup Trigger a restic backup
hermenix ctl soul Check current agent identity without entering the guest

For the full list and options, see CLI.md.

When nothing else works

  1. Re-image the system disk with plain hermenix setup.
  2. If that still fails, use a recent btrfs snapshot.
  3. If the state disk is also lost or corrupted, restore from your restic backups into a fresh deployment (or use hermenix setup --clean and re-migrate data).
  4. The original flat HERMES_HOME you migrated from (if any) is still the ultimate fallback, because hermenix migrate only ever copies.