109 lines
7.0 KiB
Markdown
109 lines
7.0 KiB
Markdown
# Architecture
|
||
|
||
## System
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
slack([Slack]) <-->|Socket Mode| bot
|
||
|
||
subgraph master["Master host"]
|
||
bot[bot] --> mgr[manager]
|
||
bot <-->|ACP over WebSocket| acp[acp client]
|
||
mgr --> local1[local node]
|
||
mgr --> state[(state.json)]
|
||
local1 -->|systemd-run + machined D-Bus| c1["nspawn: c-…<br/>goose serve :1984"]
|
||
local1 -->|btrfs subvolume + quota| d1[(drives)]
|
||
end
|
||
|
||
subgraph slave["Slave host (0..n)"]
|
||
local2[local node] --> c2["nspawn: u-…<br/>goose serve :1984"]
|
||
local2 --> d2[(drives)]
|
||
end
|
||
|
||
acp -.->|direct| c1
|
||
mgr <==>|"TLS WebSocket (slave dials out)<br/>calls · reports every 5s · agent relay · btrfs send/receive"| local2
|
||
acp -.->|relayed over the link| c2
|
||
```
|
||
|
||
**One binary, two modes.** `-mode master` runs the Slack bot, the manager and a local node. `-mode slave` runs only a local node, and keeps one outgoing TLS WebSocket to the master, so slaves can sit behind NAT.
|
||
|
||
**A message's path:** Slack event (`app_mention`, a DM, or a plain reply in a channel thread set to `all-messages`) → `bot` → `manager.Ensure` (find the container, or place and start it) → `manager.DialAgent` (direct on the master, relayed for slaves) → `acp` session → updates stream back through `bot`'s streamer, which edits the Slack message.
|
||
|
||
**Containers** are `systemd-nspawn` machines started through `systemd-run`, each in its own `slackerbot-<machine>.service` unit, with CPU and memory limits. They boot the NixOS system built by `nix build .#agent` (`nix/container.nix`), which runs `goose serve` on port 1984. `/nix/store` is shared read-only. `/root` is bind-mounted from a btrfs subvolume on one of the node's drives, and is the only thing that persists. The agent's env file (API keys, the per-container goose secret) is written by the host and bind-mounted in read-only.
|
||
|
||
**State** lives only on the master, in `state_dir/state.json`: owners (container, size, node, drive, storage limits, idle limit), sessions (Slack thread → ACP session id, plus the thread's listen mode), each channel's default listen mode, and permissions. Nodes report what's running and how much disk each home uses. Nothing else is stored.
|
||
|
||
## Placement and capacity
|
||
|
||
- Each node's usable compute = total RAM/CPU − `host_reserve`. Each running container counts its full limit.
|
||
- Each drive's usable space = free + what homes use − `reserve`. Each home counts its guarantee or its real usage, whichever is bigger. The caps added up may exceed the space (overbooking is allowed, and the bot flags it).
|
||
- New containers go where `max(cpu load, ram load, storage load)` after placement stays lowest (`balance.ChoosePlacement`), on the drive with the most headroom.
|
||
- Containers stay on their node until moved. When a node is too full to start a container, the bot moves it to another node automatically.
|
||
|
||
## Cluster link
|
||
|
||
- The slave connects to `wss://master/cluster` with `X-Cluster-Secret` and `X-Node-Name`, and pins the master's self-signed certificate by fingerprint, either from the config or trusted on first use.
|
||
- Frames are JSON: `call`/`reply` for node operations, `report` for node info every 5s, and `data`/`close` for byte streams. Agent connections are tunnelled as streams.
|
||
- A slave that hasn't reported for 20s, or whose link has dropped, counts as offline. The master doesn't trust reports for a short while after it asks a slave to start or stop something, so stale reports don't undo the change.
|
||
|
||
## Migration
|
||
|
||
`manager.Migrate`: pick a target with room, `btrfs send` a read-only snapshot in 4 MB chunks through the master into `btrfs receive` on the target, and keep a running container up during this pass. Then the bot waits for any reply in progress to finish, stops the container, sends an incremental snapshot, makes a writable snapshot the new home, switches the owner record, deletes the old home, and starts the container on the target. If anything fails before the switch, the bot rolls back: the half-copied home is removed and the container restarts where it was.
|
||
|
||
## Code
|
||
|
||
All Go lives in `slackerbot/` (module `slackerbot`). Packages under `internal/` each do one job, and only import downward:
|
||
|
||
```mermaid
|
||
flowchart TD
|
||
main[main] --> bot & manager & local & state & config
|
||
bot --> manager & acp & commands & balance
|
||
manager --> balance & state & config & storage
|
||
local --> config & storage
|
||
acp --> node
|
||
balance --> node & storage
|
||
commands --> units
|
||
config --> units
|
||
manager --> node
|
||
local --> node
|
||
node --> model
|
||
state --> model
|
||
commands --> model
|
||
config --> model
|
||
balance --> model
|
||
units
|
||
storage
|
||
util
|
||
```
|
||
|
||
`util` (`KeyedMutex`, `RandomHex`) is used by `bot`, `manager` and `local`, and left out of the diagram to keep it readable.
|
||
|
||
| Package | Role |
|
||
|---|---|
|
||
| `main.go` | Flags, wiring, master vs slave |
|
||
| `model` | Shared types: `Size`, `Owner`, `Session`, permission rules, machine names |
|
||
| `units` | Parses and prints sizes (`5G`, `1.5 TB`) |
|
||
| `commands` | Parses Slack command text: `scale`, `keep`, `storage`, `grant`, and which permission a command needs |
|
||
| `config` | Loads and validates `config.toml` |
|
||
| `storage` | Drive math (`DriveReport`, headroom, overbooking) and btrfs subvolume/quota commands |
|
||
| `state` | `state.json` persistence |
|
||
| `node` | `Node` interface, wire types, offline/full errors, the WebSocket connection used to reach goose |
|
||
| `local` | The `Node` on this machine: nspawn via `systemd-run`, machined and systemd over D-Bus, capacity detection, btrfs send/receive |
|
||
| `balance` | Builds a view of each node and picks where a container goes |
|
||
| `manager` | `Manager` (owners, placement, start/stop/resize/storage), the cluster link (`Remote`, `ServeCluster`, `RunSlave`) and migration. These three share a package because they're tightly coupled |
|
||
| `acp` | JSON-RPC client for goose's ACP endpoint. Tool permission requests are auto-approved |
|
||
| `bot` | Slack events and buttons, replies, permission checks, `grant`/`revoke`, move announcements, idle reaper, streaming replies, thread listen modes, stop reactions, the assistant pane |
|
||
| `util` | `KeyedMutex`, `RandomHex` |
|
||
| `testutil`, `testutil/clustertest` | Fake nodes, and a real master + slave harness for tests |
|
||
| `nix/container.nix` | NixOS system for each container |
|
||
| `nix/host.nix` | NixOS module: the service, the `ve-*` network, firewall |
|
||
|
||
## Known limits
|
||
|
||
- The agent's tool calls are auto-approved, since the container is the sandbox (`acp.handleRequest`).
|
||
- Anyone in a channel controls that channel's container and everything the agent can reach from it.
|
||
- Containers run as root without user namespaces, and container root can get around btrfs quotas. That's fine for fair sharing among trusted users, but not safe for hostile ones.
|
||
- Restarting the master makes slaves look offline for a few seconds. Restarting the bot doesn't stop containers.
|
||
- `state.json` is fine for hundreds of users. Swap the `state` package for SQLite if you need more.
|
||
- Home folders are never deleted automatically (see INSTALLATION.md).
|