A fresh state disk means openssh generates a brand new host key on
next boot, but the old ~/.ssh/known_hosts entry for "hermes" (from
vsock connections) sticks around - the next hermes-vm ssh/hermes-ctl/
migrate connection would otherwise fail on a host-key mismatch, since
StrictHostKeyChecking=accept-new only accepts genuinely new hosts, not
a changed key for an already-known one.
ssh-keygen -R hermes runs unprivileged (the invoking user's own
known_hosts, not a root-owned path) and is a safe no-op if there's
nothing to remove - verified directly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
nu's own CLI dispatch treats any bare --flag-shaped token as an
attempt to match a flag on the CURRENT command, even when it's meant
for a rest parameter, and even with a `--` separator (confirmed
neither `...args: string` nor `-- ` sidesteps it - both still error
before ever reaching the function body). This broke `hermenix remote
<host> setup --clean` outright, both locally and thus for anyone
running it - confirmed live on hl-fwdesk right after landing --clean.
Changed `main remote`'s second parameter from a rest param to a single
`cmd: string` - the caller now quotes the subcommand and its flags as
one string ("setup --clean", "migrate /var/lib/hermes"). Since it's a
single value bound to a plain string parameter, nu never inspects its
contents for flags. Inside main remote, cmd is split back into words
and each one re-quoted for the actual remote invocation, so the
semantics on the wire are unchanged - just the calling convention.
Verified: the exact previously-failing invocation now parses and
constructs the correct `'setup' '--clean'` remote command (checked
directly, not just parse-checked).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Encodes the "start over" runbook from tonight's incident as a flag:
stops hermes-vm.service, deletes system.raw/state.raw/OVMF_VARS.fd,
then continues through the normal setup flow. Leaves the btrfs
subvolumes and any snapshots untouched - only the files inside them.
Gated behind a confirm-word ("clean") since it's destructive.
Verified in a sandbox: confirms it stops the VM and removes exactly
the three targeted files (nothing else), that declining the
confirm-word aborts without touching anything, and that it correctly
falls through into the normal disk-creation flow afterward.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1. truncate -s treats a bare number as bytes, not gigabytes - typing
"50" at the size prompt (instead of "50G") silently made a 50-byte
file, which then failed mkfs.ext4 with "Device size reported to be
zero." Added normalize-size: a bare integer answer now gets a "G"
suffix appended (matching what the prompts' own defaults imply),
anything with an existing suffix passes through unchanged.
2. Every disk-creation step runs through `priv` (sudo/root), so
subvolumes and disk images always ended up root-owned - but
hermes-vm.service runs as the unprivileged hermes-vm account, which
only had read access via the "other" bits. QEMU failed to open
system.raw for read-write. Added explicit `chown hermes-vm:hermes-vm`
after every creation point (both make-subvol branches, and after
each truncate/mkfs) - also self-heals already-existing
wrongly-owned disks on a re-run, not just fresh ones.
Both confirmed live on hl-fwdesk during tonight's incident. Verified
with a unit test of normalize-size and a full sandboxed interactive
dry run of `setup` (fake sudo/btrfs/truncate/chown/mkfs.ext4) showing
the correct truncate -> chown ordering for both disks.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
guest/common.nix was missing gcc/make/pkg-config/python3/xz, so the
installer's node-pty compile step failed with "No C++ compiler found" -
confirmed live during the hl-fwdesk incident.
Also added systemd.services.hermes-install: runs the installer
unattended (--non-interactive --skip-setup, confirmed supported via
upstream's install.sh --help) as part of first boot, gated on its own
.hermes-installed marker (same pattern as seedHermesState's .seeded). A
real service rather than an activation script since it can take a
while (compiling native modules) and shouldn't block boot or a
nixos-rebuild. `hermes setup`/`hermes gateway install` stay manual -
upstream docs don't show a non-interactive path for those, and they
need real credentials anyway.
Updated the README and the setup/migrate CLI's printed next-steps to
match - two manual commands left instead of four.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
hl-fwdesk's nix resolved git+ssh://.../hermenix.git?ref=main against a
cached ref->rev mapping from an earlier fetch this session, running a
stale commit's nu/hermenix.nu (from before flake.lock existed) and
hitting the exact same "cannot write modified lock file" error again
despite both fixes already being pushed. --refresh forces Nix to check
Gitea for the current tip on every invocation instead of trusting the
cached resolution (respects tarball-ttl otherwise) - required given the
whole design point is "always run whatever's current, no manual pull."
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fixes "cannot write modified lock file" when running hermenix straight
from its git+ssh URL (no local checkout) - Nix had nothing committed to
resolve its own sub-inputs (nixos-generators, determinate, nixpkgs)
against, and can't write one back to a fetched, read-only copy. A
committed, current flake.lock means there's nothing left to resolve on
a normal run.
Also finished threading --no-write-lock-file through the two remaining
CLI-constructed invocations (setup's printed manual-fallback text, and
remote's constructed command) that were missed in the prior commit -
harmless once the lock is current, and keeps things working even if the
lock ever drifts stale before it's refreshed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
hermenix setup/migrate/remote no longer need a local checkout anywhere:
- Host-specific settings (cache signing key, SSH keys) now live at
/etc/hermenix/local.nix on the host itself, read via an absolute-path
import in common.nix (--impure, deliberately - reading outside the
flake's own source tree). No more guest/common.nix patching, no more
guest/common.nix.bak, no more git-pull conflicts on a locally-edited
tracked file (see the hermes-vm.service incident on hl-fwdesk).
- `setup` builds directly from the flake's own git+ssh URL by default
(--flake to override), so it works from any directory on any host -
dropped the "run this from the hermenix repository root" requirement
entirely.
- `remote` now just runs `nix run <flake>#hermenix -- ...` on the target
host directly - dropped the clone-if-missing/--dir/--repo machinery,
since there's no checkout to manage anymore.
guest/local.nix.example documents the expected shape. Verified the
write-then-read-back round trip (nix-instantiate --eval --json) and the
simplified remote command construction against sandboxed stubs before
pushing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
str downcase/str lowercase differ across nu versions (renamed in
0.114.0); avoid depending on either by checking both cases directly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
hermenix remote <host> <setup|migrate> [args...] SSHes to the target
host with a pty, clones this repo there only if it isn't already
checked out (never touches an existing checkout beyond running the
command), then runs `nix run .#hermenix -- ...` in it. Lets setup/
migrate be triggered from wherever hermenix is installed, without
manually SSHing into the target host first.
Verified with a sandboxed dry run (local ssh/git/nix stubs): fresh
clone, existing-checkout skip-clone, custom --dir/--repo, and the
no-args usage guard.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
services.hermesVm -> services.hermenix throughout. setup.sh and
migrate-docker-hermes.sh are replaced by nu/hermenix.nu (setup + migrate
subcommands), packaged as a flake output (packages.${system}.hermenix,
pkgs.writers.writeNuBin) so it's installable without cloning conventions
beyond the repo itself. Both subcommands verified end-to-end against a
sandboxed dry run (stubbed disk/cache tools, a local ssh stand-in for the
migrate path) before landing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>