diff --git a/docs/CONCEPTS.md b/docs/CONCEPTS.md index 207d0c0..e7986df 100644 --- a/docs/CONCEPTS.md +++ b/docs/CONCEPTS.md @@ -12,6 +12,21 @@ Shared model for everything else in these docs. Read once; other pages assume it System is large, churny, rebuildable from this flake. State holds what cannot be reproduced — snapshots stay cheap. +Snapshots run **hourly by default** (systemd timer with `OnCalendar = "hourly"`) +and keep the last 48. They only cover the state disk. See [RECOVERY.md](RECOVERY.md#snapshots) +for details and how to adjust `services.hermenix.snapshots.interval` / `keep`. + +**Btrfs + nodatacow design note.** The state disk image is stored on a btrfs +subvolume with `chattr +C` (nodatacow). This is intentional: CoW under a VM +disk image causes terrible fragmentation. However, a btrfs snapshot of a +nodatacow subvolume re-enables CoW for any blocks the snapshot still +references. Live snapshots therefore increase fragmentation and the amount of +data that later changes (affecting both guest performance and backup size). +Snapshots and backups are independent; pruning snapshots improves both. +See [RECOVERY.md](RECOVERY.md#snapshots) and [RECOVERY.md](RECOVERY.md#backups) +for retention advice and why backups are file-level tars over vsock rather +than disk images. + Host images under `services.hermenix.disk.baseDir` (default `/var/lib/hermes-vm`): ``` diff --git a/docs/RECOVERY.md b/docs/RECOVERY.md index fbbc090..11597d1 100644 --- a/docs/RECOVERY.md +++ b/docs/RECOVERY.md @@ -1,63 +1,179 @@ -# Recovery - -Cheapest first. Layers 1–2 are inside the guest; 3–4 are host-side hermenix. - -```bash -# Guest NixOS history -hermenix vm ssh "git -C /etc/nixos log --oneline" -hermenix vm ssh "sudo nixos-rebuild switch --rollback" # also in boot menu - -# State-disk snapshot -hermenix vm snapshot -hermenix vm snapshots -hermenix vm rollback state-20260914-1400 # type 'rollback' to confirm - -# Restic backup of state files -hermenix vm backup -hermenix vm backups -``` - -Identity files under Hermes home can be inspected with `hermenix ctl soul` -etc.; restoring _Hermes_ app state is mostly "roll the state disk" or restore -from backup — not a separate product flow. - ## Snapshots -Hourly, keep 48, **state disk only**. System disk: re-image with -`hermenix setup`. +**Default frequency**: hourly (one snapshot per hour). +**Retention**: the last 48 snapshots are kept (~2 days). +**What is snapshotted**: the state disk only. The system disk is never snapshotted (it is rebuilt with `hermenix setup` in minutes). -fsfreeze via qemu-guest-agent around each shot. `rollback` stops the VM, moves -current state to `state.old-`, restores snapshot — reversible until -you delete `state.old-*`. +Snapshots are created by a systemd timer (`hermes-vm-snapshot.timer`) that runs a oneshot service calling `btrfs subvolume snapshot -r`. The timer respects `services.hermenix.snapshots.interval` (a normal systemd calendar expression) and has `Persistent = true` + a small randomized delay. -Keep retention tighter than on ZFS: state image is nodatacow; live snapshots -reintroduce CoW on referenced blocks. +While live snapshots exist on the nodatacow state image, copy-on-write is re-enabled for any blocks the snapshots still reference. This increases fragmentation and the amount of data that appears changed for backups. That is why the default retention (48) is deliberately conservative. -## Backups +You can change the cadence and how many to keep: ```nix -services.hermenix.backup = { - enable = true; - repository = "sftp:backup@nas:/hermes"; - passwordFile = "/var/lib/secrets/hermes-restic"; +services.hermenix.snapshots = { + interval = "hourly"; # e.g. "*:0/15" for every 15 min, "daily", etc. + keep = 48; }; ``` -State **files** over vsock/SSH into restic — not the raw disk. Portable; -avoids loop-mounting agent-controlled FS on the host. Runs on the host (no -`lanAccess` needed for a LAN repo). +## Inspecting a broken guest -## Guest won't boot +Use these to diagnose before doing anything destructive: -Re-image **system** only (`hermenix setup` write step); leave state disk. -Identity + `/etc/nixos` + workspace return with it. +```bash +hermenix vm console # watch boot / journal (read-only) +hermenix vm ssh # interactive shell (vsock) +hermenix vm config # current generation, NixOS generations, git log + status of /etc/nixos +hermenix vm config-pull ./guest-live # copy live /etc/nixos for diffing +hermenix ctl soul # what the agent currently thinks it is +hermenix ctl status # gateway unit health +``` -## Starting completely fresh +Inside the guest (via `hermenix vm ssh`): + +```bash +lsblk -f +sudo nix-env --list-generations -p /nix/var/nix/profiles/system +git -C /etc/nixos log --oneline -10 +git -C /etc/nixos status --short +journalctl -b -1 -p err # previous boot errors +``` + +## Common recovery paths + +### 1. Bad NixOS generation (most common) + +The previous generation is still in the boot menu. + +```bash +hermenix vm ssh "sudo nixos-rebuild switch --rollback" +# or pick an older generation from the bootloader menu on the serial console +``` + +Commit before risky changes (inside guest): + +```bash +git -C /etc/nixos add -A +git -C /etc/nixos commit -m "before risky change" +sudo nixos-rebuild switch --flake /etc/nixos +``` + +### 2. Guest won't boot at all (system disk problem) + +The **system disk** is deliberately disposable. It holds only the NixOS install +and the Nix store. Everything that matters (identity, `/etc/nixos`, workspace, +Hermes home) lives on the **state disk**. + +**CLI procedure to recover a bricked guest (re-image only the system disk):** + +1. Start the recovery: + + ```bash + hermenix setup + ``` + - It detects existing disks and skips creation. + - It still builds the image and writes it to the _system disk only_. + - State disk (and everything under /var/lib/hermes) is left untouched. + +2. Watch it come back: + + ```bash + hermenix vm console # follow early boot / journal + ``` + +3. Get inside and finish setup: + ```bash + hermenix vm ssh + + # inside the guest + lsblk -f # confirm labels match guest/hardware.nix + determinate-nixd login + sudo nix flake lock /etc/nixos + ``` + +Because the state disk was untouched, the agent should come back with its +previous SOUL, memories, config, and `/etc/nixos`. + +### 3. State disk is also suspect / want a truly clean slate + +Use `--clean`: ```bash hermenix setup --clean ``` -Stops VM; deletes `system.raw` / `state.raw` / UEFI vars; drops stale -`known_hosts` for the guest host key; walks setup again. Subvolumes/snapshots -kept. Type `clean` to confirm. +This deletes the system image, state image, and UEFI vars, then walks through +a full fresh setup. It also removes the stale `known_hosts` entry for the +guest (a new state disk gets a new SSH host key). + +**Warning**: this destroys the current state disk contents. Use backups or +btrfs snapshots first if you want any chance of recovering data. + +### 4. Use a recent btrfs snapshot + +```bash +hermenix vm snapshots +hermenix vm rollback state-2026... +# type 'rollback' to confirm +``` + +This replaces the live state subvolume with the snapshot. The previous state +is kept as `state.old-` so you can still recover from a mistaken +rollback. + +**Btrfs + nodatacow note**: While snapshots exist, CoW is re-enabled on +referenced blocks. This can increase fragmentation on the nodatacow image. +Prune them when you're happy. + +### 5. Restore from restic backup + +Backups are file-level tars of `/var/lib/hermes`, not raw disk images. + +After you have a working guest (either via re-image or rollback): + +```bash +hermenix vm ssh +# then on the guest or via host tools, restore the restic snapshot you want +# into /var/lib/hermes (or a fresh directory) and adjust services.hermenix.hermesHome if needed +``` + +See the backup configuration in your host flake for the repository and +credentials. + +## Starting completely fresh (nuclear option) + +```bash +hermenix setup --clean +``` + +Deletes system + state images + UEFI vars, drops the stale host key entry, +then does a full guided setup again. Subvolumes and old snapshots are left +on the btrfs volume. + +## Quick reference of relevant CLI commands + +| Command | Use for recovery | +| ----------------------------- | --------------------------------------------------------- | +| `hermenix setup` | Re-image system disk only (recommended for bricked guest) | +| `hermenix setup --clean` | Full wipe of both disks + fresh start | +| `hermenix vm console` | Watch boot / early errors | +| `hermenix vm ssh` | Get inside to run generations, git, journalctl | +| `hermenix vm config` | See running generation + uncommitted /etc/nixos changes | +| `hermenix vm config-pull` | Pull live config for offline diffing | +| `hermenix vm snapshot` | Trigger a manual state snapshot before risky work | +| `hermenix vm rollback ` | Restore state disk from snapshot (destructive) | +| `hermenix vm backup` | Trigger a restic backup | +| `hermenix ctl soul` | Check current agent identity without entering the guest | + +For the full list and options, see [CLI.md](CLI.md). + +## When nothing else works + +1. Re-image the system disk with plain `hermenix setup`. +2. If that still fails, use a recent btrfs snapshot. +3. If the state disk is also lost or corrupted, restore from your restic + backups into a fresh deployment (or use `hermenix setup --clean` and + re-migrate data). +4. The original flat `HERMES_HOME` you migrated from (if any) is still the + ultimate fallback, because `hermenix migrate` only ever copies.