6.8 KiB
Architecture
System
flowchart LR
slack([Slack]) <-->|Socket Mode| bot
subgraph master["Master host"]
bot[bot] --> mgr[manager]
bot <-->|ACP over WebSocket| acp[acp client]
mgr --> local1[local node]
mgr --> state[(state.json)]
local1 -->|systemd-run + machined D-Bus| c1["nspawn: c-…<br/>goose serve :1984"]
local1 -->|btrfs subvolume + quota| d1[(drives)]
end
subgraph slave["Slave host (0..n)"]
local2[local node] --> c2["nspawn: u-…<br/>goose serve :1984"]
local2 --> d2[(drives)]
end
acp -.->|direct| c1
mgr <==>|"TLS WebSocket (slave dials out)<br/>calls · reports every 5s · agent relay · btrfs send/receive"| local2
acp -.->|relayed over the link| c2
One binary, two modes. -mode master runs the Slack bot, the manager and a local node. -mode slave runs only a local node, and keeps one outgoing TLS WebSocket to the master, so slaves can sit behind NAT.
A message's path: Slack event → bot → manager.Ensure (find the container, or place and start it) → manager.DialAgent (direct on the master, relayed for slaves) → acp session → updates stream back through bot's streamer, which edits the Slack message.
Containers are systemd-nspawn machines started through systemd-run, each in its own slackerbot-<machine>.service unit, with CPU and memory limits. They boot the NixOS system built by nix build .#agent (nix/container.nix), which runs goose serve on port 1984. /nix/store is shared read-only. /root is bind-mounted from a btrfs subvolume on one of the node's drives, and is the only thing that persists. The agent's env file (API keys, the per-container goose secret) is written by the host and bind-mounted in read-only.
State lives only on the master, in state_dir/state.json: owners (container, size, node, drive, storage limits, idle limit), sessions (Slack thread → ACP session id) and permissions. Nodes report what's running and how much disk each home uses. Nothing else is stored.
Placement and capacity
- Each node's usable compute = total RAM/CPU −
host_reserve. Each running container counts its full limit. - Each drive's usable space = free + what homes use −
reserve. Each home counts its guarantee or its real usage, whichever is bigger. The caps added up may exceed the space (overbooking is allowed, and the bot flags it). - New containers go where
max(cpu load, ram load, storage load)after placement stays lowest (balance.ChoosePlacement), on the drive with the most headroom. - Containers stay on their node until moved. When a node is too full to start a container, the bot moves it to another node automatically.
Cluster link
- The slave connects to
wss://master/clusterwithX-Cluster-SecretandX-Node-Name, and pins the master's self-signed certificate by fingerprint, either from the config or trusted on first use. - Frames are JSON:
call/replyfor node operations,reportfor node info every 5s, anddata/closefor byte streams. Agent connections are tunnelled as streams. - A slave that hasn't reported for 20s, or whose link has dropped, counts as offline. The master doesn't trust reports for a short while after it asks a slave to start or stop something, so stale reports don't undo the change.
Migration
manager.Migrate: pick a target with room, btrfs send a read-only snapshot in 4 MB chunks through the master into btrfs receive on the target, and keep a running container up during this pass. Then the bot waits for any reply in progress to finish, stops the container, sends an incremental snapshot, makes a writable snapshot the new home, switches the owner record, deletes the old home, and starts the container on the target. If anything fails before the switch, the bot rolls back: the half-copied home is removed and the container restarts where it was.
Code
All Go lives in slackerbot/ (module slackerbot). Packages under internal/ each do one job, and only import downward:
flowchart TD
main[main] --> bot & manager & local & state & config
bot --> manager & acp & commands & balance
manager --> balance & state & config & storage
local --> config & storage
acp --> node
balance --> node & storage
commands --> units
config --> units
manager --> node
local --> node
node --> model
state --> model
commands --> model
config --> model
balance --> model
units
storage
util
util (KeyedMutex, RandomHex) is used by bot, manager and local, and left out of the diagram to keep it readable.
| Package | Role |
|---|---|
main.go |
Flags, wiring, master vs slave |
model |
Shared types: Size, Owner, Session, permission rules, machine names |
units |
Parses and prints sizes (5G, 1.5 TB) |
commands |
Parses Slack command text: scale, keep, storage, grant, and which permission a command needs |
config |
Loads and validates config.toml |
storage |
Drive math (DriveReport, headroom, overbooking) and btrfs subvolume/quota commands |
state |
state.json persistence |
node |
Node interface, wire types, offline/full errors, the WebSocket connection used to reach goose |
local |
The Node on this machine: nspawn via systemd-run, machined and systemd over D-Bus, capacity detection, btrfs send/receive |
balance |
Builds a view of each node and picks where a container goes |
manager |
Manager (owners, placement, start/stop/resize/storage), the cluster link (Remote, ServeCluster, RunSlave) and migration. These three share a package because they're tightly coupled |
acp |
JSON-RPC client for goose's ACP endpoint. Tool permission requests are auto-approved |
bot |
Slack events and buttons, replies, permission checks, grant/revoke, move announcements, idle reaper, streaming replies |
util |
KeyedMutex, RandomHex |
testutil, testutil/clustertest |
Fake nodes, and a real master + slave harness for tests |
nix/container.nix |
NixOS system for each container |
nix/host.nix |
NixOS module: the service, the ve-* network, firewall |
Known limits
- The agent's tool calls are auto-approved, since the container is the sandbox (
acp.handleRequest). - Anyone in a channel controls that channel's container and everything the agent can reach from it.
- Containers run as root without user namespaces, and container root can get around btrfs quotas. That's fine for fair sharing among trusted users, but not safe for hostile ones.
- Restarting the master makes slaves look offline for a few seconds. Restarting the bot doesn't stop containers.
state.jsonis fine for hundreds of users. Swap thestatepackage for SQLite if you need more.- Home folders are never deleted automatically (see INSTALLATION.md).