Files
slacker-bot/ARCHITECTURE.md
T
2026-11-01 13:13:04 -05:00

6.8 KiB
Raw Blame History

Architecture

System

flowchart LR
    slack([Slack]) <-->|Socket Mode| bot

    subgraph master["Master host"]
        bot[bot] --> mgr[manager]
        bot <-->|ACP over WebSocket| acp[acp client]
        mgr --> local1[local node]
        mgr --> state[(state.json)]
        local1 -->|systemd-run + machined D-Bus| c1["nspawn: c-…<br/>goose serve :1984"]
        local1 -->|btrfs subvolume + quota| d1[(drives)]
    end

    subgraph slave["Slave host (0..n)"]
        local2[local node] --> c2["nspawn: u-…<br/>goose serve :1984"]
        local2 --> d2[(drives)]
    end

    acp -.->|direct| c1
    mgr <==>|"TLS WebSocket (slave dials out)<br/>calls · reports every 5s · agent relay · btrfs send/receive"| local2
    acp -.->|relayed over the link| c2

One binary, two modes. -mode master runs the Slack bot, the manager and a local node. -mode slave runs only a local node, and keeps one outgoing TLS WebSocket to the master, so slaves can sit behind NAT.

A message's path: Slack event → bot → manager.Ensure (find the container, or place and start it) → manager.DialAgent (direct on the master, relayed for slaves) → acp session → updates stream back through bot's streamer, which edits the Slack message.

Containers are systemd-nspawn machines started through systemd-run, each in its own slackerbot-<machine>.service unit, with CPU and memory limits. They boot the NixOS system built by nix build .#agent (nix/container.nix), which runs goose serve on port 1984. /nix/store is shared read-only. /root is bind-mounted from a btrfs subvolume on one of the node's drives, and is the only thing that persists. The agent's env file (API keys, the per-container goose secret) is written by the host and bind-mounted in read-only.

State lives only on the master, in state_dir/state.json: owners (container, size, node, drive, storage limits, idle limit), sessions (Slack thread → ACP session id) and permissions. Nodes report what's running and how much disk each home uses. Nothing else is stored.

Placement and capacity

  • Each node's usable compute = total RAM/CPU − host_reserve. Each running container counts its full limit.
  • Each drive's usable space = free + what homes use − reserve. Each home counts its guarantee or its real usage, whichever is bigger. The caps added up may exceed the space (overbooking is allowed, and the bot flags it).
  • New containers go where max(cpu load, ram load, storage load) after placement stays lowest (balance.ChoosePlacement), on the drive with the most headroom.
  • Containers stay on their node until moved. When a node is too full to start a container, the bot moves it to another node automatically.
  • The slave connects to wss://master/cluster with X-Cluster-Secret and X-Node-Name, and pins the master's self-signed certificate by fingerprint, either from the config or trusted on first use.
  • Frames are JSON: call/reply for node operations, report for node info every 5s, and data/close for byte streams. Agent connections are tunnelled as streams.
  • A slave that hasn't reported for 20s, or whose link has dropped, counts as offline. The master doesn't trust reports for a short while after it asks a slave to start or stop something, so stale reports don't undo the change.

Migration

manager.Migrate: pick a target with room, btrfs send a read-only snapshot in 4 MB chunks through the master into btrfs receive on the target, and keep a running container up during this pass. Then the bot waits for any reply in progress to finish, stops the container, sends an incremental snapshot, makes a writable snapshot the new home, switches the owner record, deletes the old home, and starts the container on the target. If anything fails before the switch, the bot rolls back: the half-copied home is removed and the container restarts where it was.

Code

All Go lives in slackerbot/ (module slackerbot). Packages under internal/ each do one job, and only import downward:

flowchart TD
    main[main] --> bot & manager & local & state & config
    bot --> manager & acp & commands & balance
    manager --> balance & state & config & storage
    local --> config & storage
    acp --> node
    balance --> node & storage
    commands --> units
    config --> units
    manager --> node
    local --> node
    node --> model
    state --> model
    commands --> model
    config --> model
    balance --> model
    units
    storage
    util

util (KeyedMutex, RandomHex) is used by bot, manager and local, and left out of the diagram to keep it readable.

Package Role
main.go Flags, wiring, master vs slave
model Shared types: Size, Owner, Session, permission rules, machine names
units Parses and prints sizes (5G, 1.5 TB)
commands Parses Slack command text: scale, keep, storage, grant, and which permission a command needs
config Loads and validates config.toml
storage Drive math (DriveReport, headroom, overbooking) and btrfs subvolume/quota commands
state state.json persistence
node Node interface, wire types, offline/full errors, the WebSocket connection used to reach goose
local The Node on this machine: nspawn via systemd-run, machined and systemd over D-Bus, capacity detection, btrfs send/receive
balance Builds a view of each node and picks where a container goes
manager Manager (owners, placement, start/stop/resize/storage), the cluster link (Remote, ServeCluster, RunSlave) and migration. These three share a package because they're tightly coupled
acp JSON-RPC client for goose's ACP endpoint. Tool permission requests are auto-approved
bot Slack events and buttons, replies, permission checks, grant/revoke, move announcements, idle reaper, streaming replies
util KeyedMutex, RandomHex
testutil, testutil/clustertest Fake nodes, and a real master + slave harness for tests
nix/container.nix NixOS system for each container
nix/host.nix NixOS module: the service, the ve-* network, firewall

Known limits

  • The agent's tool calls are auto-approved, since the container is the sandbox (acp.handleRequest).
  • Anyone in a channel controls that channel's container and everything the agent can reach from it.
  • Containers run as root without user namespaces, and container root can get around btrfs quotas. That's fine for fair sharing among trusted users, but not safe for hostile ones.
  • Restarting the master makes slaves look offline for a few seconds. Restarting the bot doesn't stop containers.
  • state.json is fine for hundreds of users. Swap the state package for SQLite if you need more.
  • Home folders are never deleted automatically (see INSTALLATION.md).