more docs

This commit is contained in:
ari committed 2026-10-10 22:06:12 -04:00
1 parent 30427e3975
commit 053b9c4c95
2 files changed
+176 -45

No files matched your search

+15
View File
@@ -12,6 +12,21 @@ Shared model for everything else in these docs. Read once; other pages assume it
System is large, churny, rebuildable from this flake. State holds what cannot
be reproduced — snapshots stay cheap.
Snapshots run **hourly by default** (systemd timer with `OnCalendar = "hourly"`)
and keep the last 48. They only cover the state disk. See [RECOVERY.md](RECOVERY.md#snapshots)
for details and how to adjust `services.hermenix.snapshots.interval` / `keep`.
**Btrfs + nodatacow design note.** The state disk image is stored on a btrfs
subvolume with `chattr +C` (nodatacow). This is intentional: CoW under a VM
disk image causes terrible fragmentation. However, a btrfs snapshot of a
nodatacow subvolume re-enables CoW for any blocks the snapshot still
references. Live snapshots therefore increase fragmentation and the amount of
data that later changes (affecting both guest performance and backup size).
Snapshots and backups are independent; pruning snapshots improves both.
See [RECOVERY.md](RECOVERY.md#snapshots) and [RECOVERY.md](RECOVERY.md#backups)
for retention advice and why backups are file-level tars over vsock rather
than disk images.
Host images under `services.hermenix.disk.baseDir` (default `/var/lib/hermes-vm`):
```
+161 -45
View File
@@ -1,63 +1,179 @@
# Recovery
Cheapest first. Layers 1–2 are inside the guest; 3–4 are host-side hermenix.
```bash
# Guest NixOS history
hermenix vm ssh "git -C /etc/nixos log --oneline"
hermenix vm ssh "sudo nixos-rebuild switch --rollback" # also in boot menu
# State-disk snapshot
hermenix vm snapshot
hermenix vm snapshots
hermenix vm rollback state-20260914-1400 # type 'rollback' to confirm
# Restic backup of state files
hermenix vm backup
hermenix vm backups
```
Identity files under Hermes home can be inspected with `hermenix ctl soul`
etc.; restoring _Hermes_ app state is mostly "roll the state disk" or restore
from backup — not a separate product flow.
## Snapshots
Hourly, keep 48, **state disk only**. System disk: re-image with
`hermenix setup`.
**Default frequency**: hourly (one snapshot per hour).
**Retention**: the last 48 snapshots are kept (~2 days).
**What is snapshotted**: the state disk only. The system disk is never snapshotted (it is rebuilt with `hermenix setup` in minutes).
fsfreeze via qemu-guest-agent around each shot. `rollback` stops the VM, moves
current state to `state.old-<timestamp>`, restores snapshot — reversible until
you delete `state.old-*`.
Snapshots are created by a systemd timer (`hermes-vm-snapshot.timer`) that runs a oneshot service calling `btrfs subvolume snapshot -r`. The timer respects `services.hermenix.snapshots.interval` (a normal systemd calendar expression) and has `Persistent = true` + a small randomized delay.
Keep retention tighter than on ZFS: state image is nodatacow; live snapshots
reintroduce CoW on referenced blocks.
While live snapshots exist on the nodatacow state image, copy-on-write is re-enabled for any blocks the snapshots still reference. This increases fragmentation and the amount of data that appears changed for backups. That is why the default retention (48) is deliberately conservative.
## Backups
You can change the cadence and how many to keep:
```nix
services.hermenix.backup = {
enable = true;
repository = "sftp:backup@nas:/hermes";
passwordFile = "/var/lib/secrets/hermes-restic";
services.hermenix.snapshots = {
interval = "hourly"; # e.g. "*:0/15" for every 15 min, "daily", etc.
keep = 48;
};
```
State **files** over vsock/SSH into restic — not the raw disk. Portable;
avoids loop-mounting agent-controlled FS on the host. Runs on the host (no
`lanAccess` needed for a LAN repo).
## Inspecting a broken guest
## Guest won't boot
Use these to diagnose before doing anything destructive:
Re-image **system** only (`hermenix setup` write step); leave state disk.
Identity + `/etc/nixos` + workspace return with it.
```bash
hermenix vm console # watch boot / journal (read-only)
hermenix vm ssh # interactive shell (vsock)
hermenix vm config # current generation, NixOS generations, git log + status of /etc/nixos
hermenix vm config-pull ./guest-live # copy live /etc/nixos for diffing
hermenix ctl soul # what the agent currently thinks it is
hermenix ctl status # gateway unit health
```
## Starting completely fresh
Inside the guest (via `hermenix vm ssh`):
```bash
lsblk -f
sudo nix-env --list-generations -p /nix/var/nix/profiles/system
git -C /etc/nixos log --oneline -10
git -C /etc/nixos status --short
journalctl -b -1 -p err # previous boot errors
```
## Common recovery paths
### 1. Bad NixOS generation (most common)
The previous generation is still in the boot menu.
```bash
hermenix vm ssh "sudo nixos-rebuild switch --rollback"
# or pick an older generation from the bootloader menu on the serial console
```
Commit before risky changes (inside guest):
```bash
git -C /etc/nixos add -A
git -C /etc/nixos commit -m "before risky change"
sudo nixos-rebuild switch --flake /etc/nixos
```
### 2. Guest won't boot at all (system disk problem)
The **system disk** is deliberately disposable. It holds only the NixOS install
and the Nix store. Everything that matters (identity, `/etc/nixos`, workspace,
Hermes home) lives on the **state disk**.
**CLI procedure to recover a bricked guest (re-image only the system disk):**
1. Start the recovery:
```bash
hermenix setup
```
- It detects existing disks and skips creation.
- It still builds the image and writes it to the _system disk only_.
- State disk (and everything under /var/lib/hermes) is left untouched.
2. Watch it come back:
```bash
hermenix vm console # follow early boot / journal
```
3. Get inside and finish setup:
```bash
hermenix vm ssh
# inside the guest
lsblk -f # confirm labels match guest/hardware.nix
determinate-nixd login
sudo nix flake lock /etc/nixos
```
Because the state disk was untouched, the agent should come back with its
previous SOUL, memories, config, and `/etc/nixos`.
### 3. State disk is also suspect / want a truly clean slate
Use `--clean`:
```bash
hermenix setup --clean
```
Stops VM; deletes `system.raw` / `state.raw` / UEFI vars; drops stale
`known_hosts` for the guest host key; walks setup again. Subvolumes/snapshots
kept. Type `clean` to confirm.
This deletes the system image, state image, and UEFI vars, then walks through
a full fresh setup. It also removes the stale `known_hosts` entry for the
guest (a new state disk gets a new SSH host key).
**Warning**: this destroys the current state disk contents. Use backups or
btrfs snapshots first if you want any chance of recovering data.
### 4. Use a recent btrfs snapshot
```bash
hermenix vm snapshots
hermenix vm rollback state-2026...
# type 'rollback' to confirm
```
This replaces the live state subvolume with the snapshot. The previous state
is kept as `state.old-<timestamp>` so you can still recover from a mistaken
rollback.
**Btrfs + nodatacow note**: While snapshots exist, CoW is re-enabled on
referenced blocks. This can increase fragmentation on the nodatacow image.
Prune them when you're happy.
### 5. Restore from restic backup
Backups are file-level tars of `/var/lib/hermes`, not raw disk images.
After you have a working guest (either via re-image or rollback):
```bash
hermenix vm ssh
# then on the guest or via host tools, restore the restic snapshot you want
# into /var/lib/hermes (or a fresh directory) and adjust services.hermenix.hermesHome if needed
```
See the backup configuration in your host flake for the repository and
credentials.
## Starting completely fresh (nuclear option)
```bash
hermenix setup --clean
```
Deletes system + state images + UEFI vars, drops the stale host key entry,
then does a full guided setup again. Subvolumes and old snapshots are left
on the btrfs volume.
## Quick reference of relevant CLI commands
| Command | Use for recovery |
| ----------------------------- | --------------------------------------------------------- |
| `hermenix setup` | Re-image system disk only (recommended for bricked guest) |
| `hermenix setup --clean` | Full wipe of both disks + fresh start |
| `hermenix vm console` | Watch boot / early errors |
| `hermenix vm ssh` | Get inside to run generations, git, journalctl |
| `hermenix vm config` | See running generation + uncommitted /etc/nixos changes |
| `hermenix vm config-pull` | Pull live config for offline diffing |
| `hermenix vm snapshot` | Trigger a manual state snapshot before risky work |
| `hermenix vm rollback <name>` | Restore state disk from snapshot (destructive) |
| `hermenix vm backup` | Trigger a restic backup |
| `hermenix ctl soul` | Check current agent identity without entering the guest |
For the full list and options, see [CLI.md](CLI.md).
## When nothing else works
1. Re-image the system disk with plain `hermenix setup`.
2. If that still fails, use a recent btrfs snapshot.
3. If the state disk is also lost or corrupted, restore from your restic
backups into a fresh deployment (or use `hermenix setup --clean` and
re-migrate data).
4. The original flat `HERMES_HOME` you migrated from (if any) is still the
ultimate fallback, because `hermenix migrate` only ever copies.