6.5 KiB
Snapshots
Default frequency: hourly (one snapshot per hour).
Retention: the last 48 snapshots are kept (~2 days).
What is snapshotted: the state disk only. The system disk is never snapshotted (it is rebuilt with hermenix setup in minutes).
Snapshots are created by a systemd timer (hermes-vm-snapshot.timer) that runs a oneshot service calling btrfs subvolume snapshot -r. The timer respects services.hermenix.snapshots.interval (a normal systemd calendar expression) and has Persistent = true + a small randomized delay.
While live snapshots exist on the nodatacow state image, copy-on-write is re-enabled for any blocks the snapshots still reference. This increases fragmentation and the amount of data that appears changed for backups. That is why the default retention (48) is deliberately conservative.
You can change the cadence and how many to keep:
services.hermenix.snapshots = {
interval = "hourly"; # e.g. "*:0/15" for every 15 min, "daily", etc.
keep = 48;
};
Inspecting a broken guest
Use these to diagnose before doing anything destructive:
hermenix vm console # watch boot / journal (read-only)
hermenix vm ssh # interactive shell (vsock)
hermenix vm config # current generation, NixOS generations, git log + status of /etc/nixos
hermenix vm config-pull ./guest-live # copy live /etc/nixos for diffing
hermenix ctl soul # what the agent currently thinks it is
hermenix ctl status # gateway unit health
Inside the guest (via hermenix vm ssh):
lsblk -f
sudo nix-env --list-generations -p /nix/var/nix/profiles/system
git -C /etc/nixos log --oneline -10
git -C /etc/nixos status --short
journalctl -b -1 -p err # previous boot errors
Common recovery paths
1. Bad NixOS generation (most common)
The previous generation is still in the boot menu.
hermenix vm ssh "sudo nixos-rebuild switch --rollback"
# or pick an older generation from the bootloader menu on the serial console
Commit before risky changes (inside guest):
git -C /etc/nixos add -A
git -C /etc/nixos commit -m "before risky change"
sudo nixos-rebuild switch --flake /etc/nixos
2. Guest won't boot at all (system disk problem)
The system disk is deliberately disposable. It holds only the NixOS install
and the Nix store. Everything that matters (identity, /etc/nixos, workspace,
Hermes home) lives on the state disk.
CLI procedure to recover a bricked guest (re-image only the system disk):
-
Start the recovery:
hermenix setup- It detects existing disks and skips creation.
- It still builds the image and writes it to the system disk only.
- State disk (and everything under /var/lib/hermes) is left untouched.
-
Watch it come back:
hermenix vm console # follow early boot / journal -
Get inside and finish setup:
hermenix vm ssh # inside the guest lsblk -f # confirm labels match guest/hardware.nix determinate-nixd login sudo nix flake lock /etc/nixos
Because the state disk was untouched, the agent should come back with its
previous SOUL, memories, config, and /etc/nixos.
3. State disk is also suspect / want a truly clean slate
Use --clean:
hermenix setup --clean
This deletes the system image, state image, and UEFI vars, then walks through
a full fresh setup. It also removes the stale known_hosts entry for the
guest (a new state disk gets a new SSH host key).
Warning: this destroys the current state disk contents. Use backups or btrfs snapshots first if you want any chance of recovering data.
4. Use a recent btrfs snapshot
hermenix vm snapshots
hermenix vm rollback state-2026...
# type 'rollback' to confirm
This replaces the live state subvolume with the snapshot. The previous state
is kept as state.old-<timestamp> so you can still recover from a mistaken
rollback.
Btrfs + nodatacow note: While snapshots exist, CoW is re-enabled on referenced blocks. This can increase fragmentation on the nodatacow image. Prune them when you're happy.
5. Restore from restic backup
Backups are file-level tars of /var/lib/hermes, not raw disk images.
After you have a working guest (either via re-image or rollback):
hermenix vm ssh
# then on the guest or via host tools, restore the restic snapshot you want
# into /var/lib/hermes (or a fresh directory) and adjust services.hermenix.hermesHome if needed
See the backup configuration in your host flake for the repository and credentials.
Starting completely fresh (nuclear option)
hermenix setup --clean
Deletes system + state images + UEFI vars, drops the stale host key entry, then does a full guided setup again. Subvolumes and old snapshots are left on the btrfs volume.
Quick reference of relevant CLI commands
| Command | Use for recovery |
|---|---|
hermenix setup |
Re-image system disk only (recommended for bricked guest) |
hermenix setup --clean |
Full wipe of both disks + fresh start |
hermenix vm console |
Watch boot / early errors |
hermenix vm ssh |
Get inside to run generations, git, journalctl |
hermenix vm config |
See running generation + uncommitted /etc/nixos changes |
hermenix vm config-pull |
Pull live config for offline diffing |
hermenix vm snapshot |
Trigger a manual state snapshot before risky work |
hermenix vm rollback <name> |
Restore state disk from snapshot (destructive) |
hermenix vm backup |
Trigger a restic backup |
hermenix ctl soul |
Check current agent identity without entering the guest |
For the full list and options, see CLI.md.
When nothing else works
- Re-image the system disk with plain
hermenix setup. - If that still fails, use a recent btrfs snapshot.
- If the state disk is also lost or corrupted, restore from your restic
backups into a fresh deployment (or use
hermenix setup --cleanand re-migrate data). - The original flat
HERMES_HOMEyou migrated from (if any) is still the ultimate fallback, becausehermenix migrateonly ever copies.