From 7a8fa22e708d74f29d2526c6a0dcffa8c058d2a3 Mon Sep 17 00:00:00 2001 From: Ari Sosnovsky Date: Sun, 1 Nov 2026 13:13:04 -0500 Subject: [PATCH] docs --- .envrc | 1 + .gitignore | 1 + AGENTS.md | 68 +++++++++++++ ARCHITECTURE.md | 108 ++++++++++++++++++++ INSTALLATION.md | 94 ++++++++++++++++++ README.md | 257 ++++-------------------------------------------- USERGUIDE.md | 94 ++++++++++++++++++ 7 files changed, 386 insertions(+), 237 deletions(-) create mode 100644 .envrc create mode 100644 AGENTS.md create mode 100644 ARCHITECTURE.md create mode 100644 INSTALLATION.md create mode 100644 USERGUIDE.md diff --git a/.envrc b/.envrc new file mode 100644 index 0000000..3550a30 --- /dev/null +++ b/.envrc @@ -0,0 +1 @@ +use flake diff --git a/.gitignore b/.gitignore index ce0f77d..27ef3c5 100644 --- a/.gitignore +++ b/.gitignore @@ -4,3 +4,4 @@ result result-* .commandcode/ +.direnv/ diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..dcd5138 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,68 @@ +# Agent guidelines + +Rules for anyone, human or AI, changing this repo. Read [ARCHITECTURE.md](ARCHITECTURE.md) first. + +## Environment + +- `nix develop` (or direnv with `.envrc`) gives Go, gopls, staticcheck, delve and btrfs-progs. Use it rather than system tools. +- Go code lives in `slackerbot/`, and that's where `go` commands run. Nix files and docs live at the repo root. + +## Checks before you finish + +Run from `slackerbot/`, and all must pass: + +``` +gofmt -l . # must print nothing +go vet ./... +go test ./... +``` + +If you changed `flake.nix`, `nix/`, `go.mod` or `go.sum`, also run from the repo root: + +``` +nix build .#slackerbot +nix build .#agent +nix flake check --no-build +``` + +- Flakes only see files tracked by git. `git add` new files before a nix build. +- After changing Go dependencies, run `go mod tidy`, then set `vendorHash` in `flake.nix` to the `got:` hash the next `nix build .#slackerbot` prints. +- The Nix build skips tests (`doCheck = false`), because they open local ports and read `../config.example.toml`. `go test` is the real test run. +- Tests need no root, btrfs or Slack. The manager and bot tests start a real master and slave over TLS on `127.0.0.1` using fake nodes (`internal/testutil`). Code that shells out to `btrfs`, `systemd-run` or D-Bus (`local`, parts of `storage`) has no automated tests, so test it by hand on a btrfs host. + +## Package rules + +- Each package does one job (see the table in ARCHITECTURE.md). Put new code where its job is, and don't create a package for a single function. +- Imports go downward only: `model`, `units`, `storage` and `util` import nothing internal, and `bot` and `main` sit at the top. Never introduce a cycle. If two packages need a type, it belongs in `model` (or `node` for node wire types). +- `manager`, its cluster link and migration stay in one package on purpose. +- `commands` only parses text. It must never call Slack or the manager. +- The bot talks to nodes only through `manager`, never `local` directly. +- Shared test fakes go in `internal/testutil`. Tests that need a running cluster use `testutil/clustertest`. + +## Code style + +- **Few comments. Code should document itself.** Only add a comment when the code really is unclear, such as a non-obvious reason, a protocol quirk or a race. Don't add comments that repeat the code, section banners or doc comments on obvious functions. +- Follow the existing style: short functions, early returns, errors wrapped with `%w`, and user-facing text written plainly. +- Key struct literals for types from other packages (`model.Size{MemoryMB: 1024, CPUs: 1}`), or `go vet` will complain. +- Keep a function's arguments, results and locals from using a package's name (`node`, `state`, …). +- Text that Slack users see lives in `bot` and goes in plain sentences. Keep it consistent with USERGUIDE.md. + +## Behaviour you must not break + +- The `state.json` format: existing owners, sessions and permissions must still load. JSON field names are effectively a public format. +- The cluster wire protocol: frame types and call names in `manager/cluster.go`. A master and slaves may run different versions for a while. +- Machine names (`u-`, `c-`), unit names (`slackerbot-.service`), and paths (`/var/lib/slackerbot`, the agent env file at `/var/lib/slackerbot/agent.env`, which must match `nix/container.nix`). +- Anything that decides whether a message is a command or goes to the agent: `commands.ErrNotCommand`. Ordinary sentences must still reach the agent. + +## Docs + +Update the matching doc in the same change: +- Slack commands, replies or permissions → USERGUIDE.md +- Packages, data flow, protocol, placement → ARCHITECTURE.md +- Config, Nix module options, setup steps → INSTALLATION.md and `config.example.toml` +- Workflow or rules → this file + +## Git + +- Don't commit, push or rewrite history unless asked. +- Never commit build output (`result`, the `slackerbot` binary, `.direnv/`). diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 0000000..37170bd --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,108 @@ +# Architecture + +## System + +```mermaid +flowchart LR + slack([Slack]) <-->|Socket Mode| bot + + subgraph master["Master host"] + bot[bot] --> mgr[manager] + bot <-->|ACP over WebSocket| acp[acp client] + mgr --> local1[local node] + mgr --> state[(state.json)] + local1 -->|systemd-run + machined D-Bus| c1["nspawn: c-…
goose serve :1984"] + local1 -->|btrfs subvolume + quota| d1[(drives)] + end + + subgraph slave["Slave host (0..n)"] + local2[local node] --> c2["nspawn: u-…
goose serve :1984"] + local2 --> d2[(drives)] + end + + acp -.->|direct| c1 + mgr <==>|"TLS WebSocket (slave dials out)
calls · reports every 5s · agent relay · btrfs send/receive"| local2 + acp -.->|relayed over the link| c2 +``` + +**One binary, two modes.** `-mode master` runs the Slack bot, the manager and a local node. `-mode slave` runs only a local node, and keeps one outgoing TLS WebSocket to the master, so slaves can sit behind NAT. + +**A message's path:** Slack event → `bot` → `manager.Ensure` (find the container, or place and start it) → `manager.DialAgent` (direct on the master, relayed for slaves) → `acp` session → updates stream back through `bot`'s streamer, which edits the Slack message. + +**Containers** are `systemd-nspawn` machines started through `systemd-run`, each in its own `slackerbot-.service` unit, with CPU and memory limits. They boot the NixOS system built by `nix build .#agent` (`nix/container.nix`), which runs `goose serve` on port 1984. `/nix/store` is shared read-only. `/root` is bind-mounted from a btrfs subvolume on one of the node's drives, and is the only thing that persists. The agent's env file (API keys, the per-container goose secret) is written by the host and bind-mounted in read-only. + +**State** lives only on the master, in `state_dir/state.json`: owners (container, size, node, drive, storage limits, idle limit), sessions (Slack thread → ACP session id) and permissions. Nodes report what's running and how much disk each home uses. Nothing else is stored. + +## Placement and capacity + +- Each node's usable compute = total RAM/CPU − `host_reserve`. Each running container counts its full limit. +- Each drive's usable space = free + what homes use − `reserve`. Each home counts its guarantee or its real usage, whichever is bigger. The caps added up may exceed the space (overbooking is allowed, and the bot flags it). +- New containers go where `max(cpu load, ram load, storage load)` after placement stays lowest (`balance.ChoosePlacement`), on the drive with the most headroom. +- Containers stay on their node until moved. When a node is too full to start a container, the bot moves it to another node automatically. + +## Cluster link + +- The slave connects to `wss://master/cluster` with `X-Cluster-Secret` and `X-Node-Name`, and pins the master's self-signed certificate by fingerprint, either from the config or trusted on first use. +- Frames are JSON: `call`/`reply` for node operations, `report` for node info every 5s, and `data`/`close` for byte streams. Agent connections are tunnelled as streams. +- A slave that hasn't reported for 20s, or whose link has dropped, counts as offline. The master doesn't trust reports for a short while after it asks a slave to start or stop something, so stale reports don't undo the change. + +## Migration + +`manager.Migrate`: pick a target with room, `btrfs send` a read-only snapshot in 4 MB chunks through the master into `btrfs receive` on the target, and keep a running container up during this pass. Then the bot waits for any reply in progress to finish, stops the container, sends an incremental snapshot, makes a writable snapshot the new home, switches the owner record, deletes the old home, and starts the container on the target. If anything fails before the switch, the bot rolls back: the half-copied home is removed and the container restarts where it was. + +## Code + +All Go lives in `slackerbot/` (module `slackerbot`). Packages under `internal/` each do one job, and only import downward: + +```mermaid +flowchart TD + main[main] --> bot & manager & local & state & config + bot --> manager & acp & commands & balance + manager --> balance & state & config & storage + local --> config & storage + acp --> node + balance --> node & storage + commands --> units + config --> units + manager --> node + local --> node + node --> model + state --> model + commands --> model + config --> model + balance --> model + units + storage + util +``` + +`util` (`KeyedMutex`, `RandomHex`) is used by `bot`, `manager` and `local`, and left out of the diagram to keep it readable. + +| Package | Role | +|---|---| +| `main.go` | Flags, wiring, master vs slave | +| `model` | Shared types: `Size`, `Owner`, `Session`, permission rules, machine names | +| `units` | Parses and prints sizes (`5G`, `1.5 TB`) | +| `commands` | Parses Slack command text: `scale`, `keep`, `storage`, `grant`, and which permission a command needs | +| `config` | Loads and validates `config.toml` | +| `storage` | Drive math (`DriveReport`, headroom, overbooking) and btrfs subvolume/quota commands | +| `state` | `state.json` persistence | +| `node` | `Node` interface, wire types, offline/full errors, the WebSocket connection used to reach goose | +| `local` | The `Node` on this machine: nspawn via `systemd-run`, machined and systemd over D-Bus, capacity detection, btrfs send/receive | +| `balance` | Builds a view of each node and picks where a container goes | +| `manager` | `Manager` (owners, placement, start/stop/resize/storage), the cluster link (`Remote`, `ServeCluster`, `RunSlave`) and migration. These three share a package because they're tightly coupled | +| `acp` | JSON-RPC client for goose's ACP endpoint. Tool permission requests are auto-approved | +| `bot` | Slack events and buttons, replies, permission checks, `grant`/`revoke`, move announcements, idle reaper, streaming replies | +| `util` | `KeyedMutex`, `RandomHex` | +| `testutil`, `testutil/clustertest` | Fake nodes, and a real master + slave harness for tests | +| `nix/container.nix` | NixOS system for each container | +| `nix/host.nix` | NixOS module: the service, the `ve-*` network, firewall | + +## Known limits + +- The agent's tool calls are auto-approved, since the container is the sandbox (`acp.handleRequest`). +- Anyone in a channel controls that channel's container and everything the agent can reach from it. +- Containers run as root without user namespaces, and container root can get around btrfs quotas. That's fine for fair sharing among trusted users, but not safe for hostile ones. +- Restarting the master makes slaves look offline for a few seconds. Restarting the bot doesn't stop containers. +- `state.json` is fine for hundreds of users. Swap the `state` package for SQLite if you need more. +- Home folders are never deleted automatically (see INSTALLATION.md). diff --git a/INSTALLATION.md b/INSTALLATION.md new file mode 100644 index 0000000..bc837d8 --- /dev/null +++ b/INSTALLATION.md @@ -0,0 +1,94 @@ +# Installation + +## Requirements + +- Linux with systemd (`systemd-nspawn`, `systemd-machined`, `systemd-networkd`), running as root. +- One or more **btrfs** folders for home folders. Quotas are switched on automatically. +- Nix with flakes, to build the bot and the container system. NixOS is easiest, but not required. +- An API key for the model goose should use (e.g. Anthropic). + +## 1. Slack app + +1. Go to → **Create New App** → **From a manifest**, and paste `slack-manifest.yaml`. +2. Install it to your workspace and copy the **bot token** (`xoxb-…`). +3. Under **Basic Information → App-Level Tokens**, create a token with the `connections:write` scope (`xapp-…`). + +The app uses Socket Mode, so the bot needs no public URL. + +## 2. Container system + +Build the NixOS system each container boots: + +``` +nix build git+https://gitea.skyg.ca/ari/slacker-bot.git#agent -o /var/lib/slackerbot/agent-system +# or, from a checkout: +nix build .#agent -o /var/lib/slackerbot/agent-system +``` + +Rebuild the same way after changing `nix/container.nix`. Each container picks up the new build on its next start. `nix/container.nix` runs `goose serve`, which needs a recent goose. If nixpkgs' `goose-cli` is too old, override it there. + +## 3. Config + +Copy `config.example.toml` and edit it. The important parts: + +| Setting | What | +|---|---| +| `system` | The build from step 2, e.g. `/var/lib/slackerbot/agent-system` | +| `state_dir` | Where `state.json`, env files and cluster certificates live | +| `[agent_env]` | Environment for goose (provider, model, API key). `$VARS` are expanded from the bot's environment | +| `[container]`, `[max_container]` | Default and largest container size | +| `[host_reserve]` | RAM/CPU kept free for the host | +| `idle_timeout`, `max_idle_timeout`, `allow_indefinite` | Idle shutdown rules | +| `[storage]` | Default guaranteed (`min`) and cap (`max`) home size, and the largest cap anyone may set (`limit`) | +| `[[drives]]` | btrfs folders for home folders, each with a `reserve` kept free | +| `[permissions]` | Starting permissions, copied in on first start only (see USERGUIDE.md) | +| `[cluster]` | Only for more than one node (see step 5) | + +Secrets go in the environment rather than the file: `SLACK_APP_TOKEN`, `SLACK_BOT_TOKEN`, `CLUSTER_SECRET`, and any API keys used in `[agent_env]`. + +## 4. Run the bot + +### NixOS + +Add the flake to your system flake and import the host module: + +```nix +inputs.slackerbot.url = "git+https://gitea.skyg.ca/ari/slacker-bot.git"; + +# in your host's modules: +imports = [ inputs.slackerbot.nixosModules.host ]; +services.slackerbot = { + enable = true; + configFile = ./slackerbot.toml; + environmentFile = "/etc/slackerbot/secrets"; # SLACK_APP_TOKEN=…, SLACK_BOT_TOKEN=…, ANTHROPIC_API_KEY=… +}; +``` + +The module runs the bot as a systemd service with `/var/lib/slackerbot` as its state directory. It also sets up `systemd-networkd` to give each container's `ve-*` link an address over DHCP with NAT, and trusts those links in the firewall. + +### Other Linux + +``` +nix build git+https://gitea.skyg.ca/ari/slacker-bot.git#slackerbot # or: cd slackerbot && go build +sudo slackerbot -config /etc/slackerbot/config.toml +``` + +Make sure `systemd-networkd` manages `ve-*` links. systemd ships `80-container-ve.network` for this. `btrfs-progs` must be on the bot's `PATH`. + +Check the log for the node's capacity line and "connected to Slack", then DM the bot `help`. + +## 5. More nodes (optional) + +1. On every node, set the same `[cluster] secret` (16+ characters), plus its own `system`, `[[drives]]` and `host_reserve`, and build the container system there. Slaves don't need Slack tokens or `agent_env`, because the master sends those. +2. **Master:** set `cluster.listen = ":7946"` and open the port. On NixOS: `services.slackerbot.clusterPort = 7946`. +3. **Slaves:** set `cluster.master = "wss://:7946"` and a unique `cluster.name` (default: the hostname). Run with `-mode slave`. On NixOS: `services.slackerbot.mode = "slave"`. +4. The master logs its certificate fingerprint at startup. Put it in each slave's `cluster.master_fingerprint`. If you leave it empty, a slave trusts the first certificate it sees, saves it in `state_dir/master.fingerprint`, and refuses any other from then on. + +`nodes` in Slack should now list every node. + +## Maintenance + +- **Permissions** live in the master's `state.json` after the first start, and editing the config doesn't change them. To start over from the config, stop the master and delete the `permissions` entry from `state.json`. +- **Deleting a home folder for good:** stop its container, run `btrfs subvolume delete /`, then, with the bot stopped, remove its owner from `state.json`. Nothing deletes homes automatically, so their guaranteed space stays taken. This includes homes left on a dead slave after **Create new container**. +- **Upgrading** the bot doesn't stop running containers, because each runs in its own `slackerbot-.service` unit. +- Find user and channel IDs in Slack with **Copy member ID** on a profile, or at the bottom of a channel's **About** tab. diff --git a/README.md b/README.md index 863c459..4fc2fcc 100644 --- a/README.md +++ b/README.md @@ -1,246 +1,29 @@ -# slackerbot +# Slackerbot -A Slack bot that runs goose agents in NixOS containers and connects them to Slack. +A Slack bot that gives every DM and every channel its own AI coding agent, running in its own NixOS container. -## Who owns a container +Talk to the bot in Slack, and your message goes to a [goose](https://github.com/block/goose) agent inside a `systemd-nspawn` container. The agent's reply streams back into a Slack thread. Replying in that thread continues the same session. -| Where you talk to the bot | Container | Who can use it | -|---|---|---| -| DM with the bot | `u-` | Only you | -| Channel, private channel or group DM (@mention the bot) | `c-` | Anyone in that channel | +- **One container per DM or channel.** DMs get a private container, and channels share one. +- **Persistent home folder.** `/root` lives on a btrfs subvolume with a guaranteed size and a cap. Everything else is rebuilt on each start. +- **Managed from Slack.** Users can `scale`, `stop`, set idle timeouts with `keep`, resize `storage`, and `move` containers between nodes. Changes are limited by per-channel permissions. +- **One or more machines.** A master runs the Slack bot, and slaves add capacity. New containers go wherever CPU, RAM and storage stay most even, and live migration moves a running container with only a short pause. -In channels, every message reaches the agent prefixed with the sender's name (e.g. `Alice: try the other branch`), so it knows who's asking. +## Docs -## What happens when someone messages the bot +| File | For | +| ---------------------------------- | -------------------------------------------------- | +| [INSTALLATION.md](INSTALLATION.md) | Setting up the Slack app, the host and extra nodes | +| [USERGUIDE.md](USERGUIDE.md) | Slack commands and how permissions work | +| [ARCHITECTURE.md](ARCHITECTURE.md) | How the system and the code are put together | +| [AGENTS.md](AGENTS.md) | Rules for changing the code: tests, layout, style | -1. The bot finds the container for that DM or channel. - - **Running:** uses it. - - **Stopped:** starts it again. Its home folder (`/root`) is still there, and everything else is rebuilt fresh. - - **Doesn't exist:** creates it on the node where CPU, RAM and storage stay most even (see "Running more than one node"), with its home folder on that node's roomiest drive. - - **Everything is full:** replies "maxed out" instead. - - **Its node is offline:** replies "Channel bot is on a dead slave ``, create new container or wait for the slave to be back up?" with **Create new container** and **Wait** buttons. -2. It waits until goose answers on port **1984** inside the container. -3. A new message starts a new agent session, and the reply goes in a thread. - Replying in that thread continues the same session, even days later. -4. The agent's answer streams into the thread as it works, with tool calls shown as `🔧` lines. -5. Every 10 minutes, containers that have gone unused past their limit are shut down. The limit is 6 hours by default, or whatever `keep` set. The owning DM or channel gets a message saying so. -6. Messages in the same thread run one at a time. If two people post at once, the second waits its turn. +## Quick start -## Commands (send as a new message, not in a thread) - -| Command | What it does | -|---|---| -| `sessions` | Lists this DM's or channel's past sessions with links. Open one and reply in its thread to continue. | -| `status` | Shows the container's state, size, time unused and idle limit, with **Stop** and **Scale up** buttons. | -| `stop` | Shuts the container down. Files are kept. | -| `scale up` / `scale down` | Doubles or halves RAM and CPU. | -| `scale 8G 4cpu` | Sets an exact size. No sign means "set to". | -| `scale +2G` / `scale -1cpu` / `scale 512M +0.5cpu` | A `+` or `-` sign means "change by". Memory and CPU can be mixed or left out. | -| `keep 24h` (or `3d`, `1d12h`, `2w`, `5 hours`) | Stops the container after that long unused, instead of the default. | -| `keep indefinite` (or `forever`) | Never stops it for being unused. Use `stop` when done. | -| `keep default` | Goes back to the config's `idle_timeout`. | -| `keep` | Shows the current idle limit. | -| `storage` | Shows home folder usage, guaranteed space and max. | -| `storage 20G` / `storage +5G` / `storage -2G` | Sets or changes the max (the cap). | -| `storage min 8G` | Sets the guaranteed space. | -| `storage min 8G max 20G` | Sets both. | -| `nodes` | Lists every node with free CPU, RAM and storage, how many containers it has, and which one this container is on. Offline nodes are listed too. | -| `move ` (or `migrate `, `move to `) | Moves this container, running or not, to that node. Works inside a thread too. | -| `permissions` | Shows who can scale, change storage, move or set timeouts in this channel or DM. | -| `move auto` (or just `move`) | Moves it to whichever other node keeps resources most even. | -| `help` | Shows this list. | - -**Scale rules:** memory takes `G`/`GB` or `M`/`MB`, and CPU takes `cpu`/`cpus`/`core`/`cores` (fractions are fine). Sizes stay between 256 MB / 0.25 CPU and `max_container`. Growing a running container needs room on the node. Changes apply live, or on next start if it's stopped. Shrinking memory below what's in use may stop programs inside. - -**Keep rules:** the minimum is 10 minutes. `max_idle_timeout` in the config caps how long `keep` can ask for, and `allow_indefinite = false` turns off `keep indefinite`. Setting `keep` restarts the countdown. - -**Storage rules:** sizes take `M`/`MB`, `G`/`GB` or `T`/`TB`. Both numbers are at least 512 MB, the guarantee can't exceed the max, and the max can't exceed `storage.limit` or go below what's already in use. Raising the guarantee is **rejected** if the drive can't promise it. Raising the max past what the drive can hold for everyone is **allowed but flagged** as overbooked. - -If the words after `scale`, `keep`, `storage` or `move` don't look like a size or a time (e.g. "keep the tests passing"), the message goes to the agent instead. - -Commands act on the container that belongs to where you send them. In a channel, anyone can stop or scale the channel's container. In channels, @mention the bot every time, including in threads. Plain replies aren't seen. - -## Super users and permissions - -Changing a container is limited to super users, one permission at a time: - -| Permission | Covers | -|---|---| -| `scale` | `scale …` and the **Scale up** button | -| `storage` | `storage `, `storage min …` | -| `move` | `move …`, `move auto`, and **Create new container** on a dead slave | -| `keep` | `keep