## Snapshots **Default frequency**: hourly (one snapshot per hour). **Retention**: the last 48 snapshots are kept (~2 days). **What is snapshotted**: the state disk only. The system disk is never snapshotted (it is rebuilt with `hermenix setup` in minutes). Snapshots are created by a systemd timer (`hermes-vm-snapshot.timer`) that runs a oneshot service calling `btrfs subvolume snapshot -r`. The timer respects `services.hermenix.snapshots.interval` (a normal systemd calendar expression) and has `Persistent = true` + a small randomized delay. While live snapshots exist on the nodatacow state image, copy-on-write is re-enabled for any blocks the snapshots still reference. This increases fragmentation and the amount of data that appears changed for backups. That is why the default retention (48) is deliberately conservative. You can change the cadence and how many to keep: ```nix services.hermenix.snapshots = { interval = "hourly"; # e.g. "*:0/15" for every 15 min, "daily", etc. keep = 48; }; ``` ## Inspecting a broken guest Use these to diagnose before doing anything destructive: ```bash hermenix vm console # watch boot / journal (read-only) hermenix vm ssh # interactive shell (vsock) hermenix vm config # current generation, NixOS generations, git log + status of /etc/nixos hermenix vm config-pull ./guest-live # copy live /etc/nixos for diffing hermenix ctl soul # what the agent currently thinks it is hermenix ctl status # gateway unit health ``` Inside the guest (via `hermenix vm ssh`): ```bash lsblk -f sudo nix-env --list-generations -p /nix/var/nix/profiles/system git -C /etc/nixos log --oneline -10 git -C /etc/nixos status --short journalctl -b -1 -p err # previous boot errors ``` ## Common recovery paths ### 1. Bad NixOS generation (most common) The previous generation is still in the boot menu. ```bash hermenix vm ssh "sudo nixos-rebuild switch --rollback" # or pick an older generation from the bootloader menu on the serial console ``` Commit before risky changes (inside guest): ```bash git -C /etc/nixos add -A git -C /etc/nixos commit -m "before risky change" sudo nixos-rebuild switch --flake /etc/nixos ``` ### 2. Guest won't boot at all (system disk problem) The **system disk** is deliberately disposable. It holds only the NixOS install and the Nix store. Everything that matters (identity, `/etc/nixos`, workspace, Hermes home) lives on the **state disk**. **CLI procedure to recover a bricked guest (re-image only the system disk):** 1. Start the recovery: ```bash hermenix setup ``` - It detects existing disks and skips creation. - It still builds the image and writes it to the _system disk only_. - State disk (and everything under /var/lib/hermes) is left untouched. 2. Watch it come back: ```bash hermenix vm console # follow early boot / journal ``` 3. Get inside and finish setup: ```bash hermenix vm ssh # inside the guest lsblk -f # confirm labels match guest/hardware.nix determinate-nixd login sudo nix flake lock /etc/nixos ``` Because the state disk was untouched, the agent should come back with its previous SOUL, memories, config, and `/etc/nixos`. ### 3. State disk is also suspect / want a truly clean slate Use `--clean`: ```bash hermenix setup --clean ``` This deletes the system image, state image, and UEFI vars, then walks through a full fresh setup. It also removes the stale `known_hosts` entry for the guest (a new state disk gets a new SSH host key). **Warning**: this destroys the current state disk contents. Use backups or btrfs snapshots first if you want any chance of recovering data. ### 4. Use a recent btrfs snapshot ```bash hermenix vm snapshots hermenix vm rollback state-2026... # type 'rollback' to confirm ``` This replaces the live state subvolume with the snapshot. The previous state is kept as `state.old-` so you can still recover from a mistaken rollback. **Btrfs + nodatacow note**: While snapshots exist, CoW is re-enabled on referenced blocks. This can increase fragmentation on the nodatacow image. Prune them when you're happy. ### 5. Restore from restic backup Backups are file-level tars of `/var/lib/hermes`, not raw disk images. After you have a working guest (either via re-image or rollback): ```bash hermenix vm ssh # then on the guest or via host tools, restore the restic snapshot you want # into /var/lib/hermes (or a fresh directory) and adjust services.hermenix.hermesHome if needed ``` See the backup configuration in your host flake for the repository and credentials. ## Starting completely fresh (nuclear option) ```bash hermenix setup --clean ``` Deletes system + state images + UEFI vars, drops the stale host key entry, then does a full guided setup again. Subvolumes and old snapshots are left on the btrfs volume. ## Quick reference of relevant CLI commands | Command | Use for recovery | | ----------------------------- | --------------------------------------------------------- | | `hermenix setup` | Re-image system disk only (recommended for bricked guest) | | `hermenix setup --clean` | Full wipe of both disks + fresh start | | `hermenix vm console` | Watch boot / early errors | | `hermenix vm ssh` | Get inside to run generations, git, journalctl | | `hermenix vm config` | See running generation + uncommitted /etc/nixos changes | | `hermenix vm config-pull` | Pull live config for offline diffing | | `hermenix vm snapshot` | Trigger a manual state snapshot before risky work | | `hermenix vm rollback ` | Restore state disk from snapshot (destructive) | | `hermenix vm backup` | Trigger a restic backup | | `hermenix ctl soul` | Check current agent identity without entering the guest | For the full list and options, see [CLI.md](CLI.md). ## When nothing else works 1. Re-image the system disk with plain `hermenix setup`. 2. If that still fails, use a recent btrfs snapshot. 3. If the state disk is also lost or corrupted, restore from your restic backups into a fresh deployment (or use `hermenix setup --clean` and re-migrate data). 4. The original flat `HERMES_HOME` you migrated from (if any) is still the ultimate fallback, because `hermenix migrate` only ever copies.