This commit is contained in:
ari committed 2026-11-01 13:13:04 -05:00
1 parent 0ac61b4ece
commit 7a8fa22e70
7 files changed
+386 -237

No files matched your search

+108
View File
@@ -0,0 +1,108 @@
# Architecture
## System
```mermaid
flowchart LR
slack([Slack]) <-->|Socket Mode| bot
subgraph master["Master host"]
bot[bot] --> mgr[manager]
bot <-->|ACP over WebSocket| acp[acp client]
mgr --> local1[local node]
mgr --> state[(state.json)]
local1 -->|systemd-run + machined D-Bus| c1["nspawn: c-…<br/>goose serve :1984"]
local1 -->|btrfs subvolume + quota| d1[(drives)]
end
subgraph slave["Slave host (0..n)"]
local2[local node] --> c2["nspawn: u-…<br/>goose serve :1984"]
local2 --> d2[(drives)]
end
acp -.->|direct| c1
mgr <==>|"TLS WebSocket (slave dials out)<br/>calls · reports every 5s · agent relay · btrfs send/receive"| local2
acp -.->|relayed over the link| c2
```
**One binary, two modes.** `-mode master` runs the Slack bot, the manager and a local node. `-mode slave` runs only a local node, and keeps one outgoing TLS WebSocket to the master, so slaves can sit behind NAT.
**A message's path:** Slack event → `bot` → `manager.Ensure` (find the container, or place and start it) → `manager.DialAgent` (direct on the master, relayed for slaves) → `acp` session → updates stream back through `bot`'s streamer, which edits the Slack message.
**Containers** are `systemd-nspawn` machines started through `systemd-run`, each in its own `slackerbot-<machine>.service` unit, with CPU and memory limits. They boot the NixOS system built by `nix build .#agent` (`nix/container.nix`), which runs `goose serve` on port 1984. `/nix/store` is shared read-only. `/root` is bind-mounted from a btrfs subvolume on one of the node's drives, and is the only thing that persists. The agent's env file (API keys, the per-container goose secret) is written by the host and bind-mounted in read-only.
**State** lives only on the master, in `state_dir/state.json`: owners (container, size, node, drive, storage limits, idle limit), sessions (Slack thread → ACP session id) and permissions. Nodes report what's running and how much disk each home uses. Nothing else is stored.
## Placement and capacity
- Each node's usable compute = total RAM/CPU − `host_reserve`. Each running container counts its full limit.
- Each drive's usable space = free + what homes use − `reserve`. Each home counts its guarantee or its real usage, whichever is bigger. The caps added up may exceed the space (overbooking is allowed, and the bot flags it).
- New containers go where `max(cpu load, ram load, storage load)` after placement stays lowest (`balance.ChoosePlacement`), on the drive with the most headroom.
- Containers stay on their node until moved. When a node is too full to start a container, the bot moves it to another node automatically.
## Cluster link
- The slave connects to `wss://master/cluster` with `X-Cluster-Secret` and `X-Node-Name`, and pins the master's self-signed certificate by fingerprint, either from the config or trusted on first use.
- Frames are JSON: `call`/`reply` for node operations, `report` for node info every 5s, and `data`/`close` for byte streams. Agent connections are tunnelled as streams.
- A slave that hasn't reported for 20s, or whose link has dropped, counts as offline. The master doesn't trust reports for a short while after it asks a slave to start or stop something, so stale reports don't undo the change.
## Migration
`manager.Migrate`: pick a target with room, `btrfs send` a read-only snapshot in 4 MB chunks through the master into `btrfs receive` on the target, and keep a running container up during this pass. Then the bot waits for any reply in progress to finish, stops the container, sends an incremental snapshot, makes a writable snapshot the new home, switches the owner record, deletes the old home, and starts the container on the target. If anything fails before the switch, the bot rolls back: the half-copied home is removed and the container restarts where it was.
## Code
All Go lives in `slackerbot/` (module `slackerbot`). Packages under `internal/` each do one job, and only import downward:
```mermaid
flowchart TD
main[main] --> bot & manager & local & state & config
bot --> manager & acp & commands & balance
manager --> balance & state & config & storage
local --> config & storage
acp --> node
balance --> node & storage
commands --> units
config --> units
manager --> node
local --> node
node --> model
state --> model
commands --> model
config --> model
balance --> model
units
storage
util
```
`util` (`KeyedMutex`, `RandomHex`) is used by `bot`, `manager` and `local`, and left out of the diagram to keep it readable.
| Package | Role |
|---|---|
| `main.go` | Flags, wiring, master vs slave |
| `model` | Shared types: `Size`, `Owner`, `Session`, permission rules, machine names |
| `units` | Parses and prints sizes (`5G`, `1.5 TB`) |
| `commands` | Parses Slack command text: `scale`, `keep`, `storage`, `grant`, and which permission a command needs |
| `config` | Loads and validates `config.toml` |
| `storage` | Drive math (`DriveReport`, headroom, overbooking) and btrfs subvolume/quota commands |
| `state` | `state.json` persistence |
| `node` | `Node` interface, wire types, offline/full errors, the WebSocket connection used to reach goose |
| `local` | The `Node` on this machine: nspawn via `systemd-run`, machined and systemd over D-Bus, capacity detection, btrfs send/receive |
| `balance` | Builds a view of each node and picks where a container goes |
| `manager` | `Manager` (owners, placement, start/stop/resize/storage), the cluster link (`Remote`, `ServeCluster`, `RunSlave`) and migration. These three share a package because they're tightly coupled |
| `acp` | JSON-RPC client for goose's ACP endpoint. Tool permission requests are auto-approved |
| `bot` | Slack events and buttons, replies, permission checks, `grant`/`revoke`, move announcements, idle reaper, streaming replies |
| `util` | `KeyedMutex`, `RandomHex` |
| `testutil`, `testutil/clustertest` | Fake nodes, and a real master + slave harness for tests |
| `nix/container.nix` | NixOS system for each container |
| `nix/host.nix` | NixOS module: the service, the `ve-*` network, firewall |
## Known limits
- The agent's tool calls are auto-approved, since the container is the sandbox (`acp.handleRequest`).
- Anyone in a channel controls that channel's container and everything the agent can reach from it.
- Containers run as root without user namespaces, and container root can get around btrfs quotas. That's fine for fair sharing among trusted users, but not safe for hostile ones.
- Restarting the master makes slaves look offline for a few seconds. Restarting the bot doesn't stop containers.
- `state.json` is fine for hundreds of users. Swap the `state` package for SQLite if you need more.
- Home folders are never deleted automatically (see INSTALLATION.md).