This commit is contained in:
ari committed 2026-11-01 13:13:04 -05:00
1 parent 0ac61b4ece
commit 7a8fa22e70
7 files changed
+386 -237

No files matched your search

+1
View File
@@ -0,0 +1 @@
use flake
+1
View File
@@ -4,3 +4,4 @@
result
result-*
.commandcode/
.direnv/
+68
View File
@@ -0,0 +1,68 @@
# Agent guidelines
Rules for anyone, human or AI, changing this repo. Read [ARCHITECTURE.md](ARCHITECTURE.md) first.
## Environment
- `nix develop` (or direnv with `.envrc`) gives Go, gopls, staticcheck, delve and btrfs-progs. Use it rather than system tools.
- Go code lives in `slackerbot/`, and that's where `go` commands run. Nix files and docs live at the repo root.
## Checks before you finish
Run from `slackerbot/`, and all must pass:
```
gofmt -l . # must print nothing
go vet ./...
go test ./...
```
If you changed `flake.nix`, `nix/`, `go.mod` or `go.sum`, also run from the repo root:
```
nix build .#slackerbot
nix build .#agent
nix flake check --no-build
```
- Flakes only see files tracked by git. `git add` new files before a nix build.
- After changing Go dependencies, run `go mod tidy`, then set `vendorHash` in `flake.nix` to the `got:` hash the next `nix build .#slackerbot` prints.
- The Nix build skips tests (`doCheck = false`), because they open local ports and read `../config.example.toml`. `go test` is the real test run.
- Tests need no root, btrfs or Slack. The manager and bot tests start a real master and slave over TLS on `127.0.0.1` using fake nodes (`internal/testutil`). Code that shells out to `btrfs`, `systemd-run` or D-Bus (`local`, parts of `storage`) has no automated tests, so test it by hand on a btrfs host.
## Package rules
- Each package does one job (see the table in ARCHITECTURE.md). Put new code where its job is, and don't create a package for a single function.
- Imports go downward only: `model`, `units`, `storage` and `util` import nothing internal, and `bot` and `main` sit at the top. Never introduce a cycle. If two packages need a type, it belongs in `model` (or `node` for node wire types).
- `manager`, its cluster link and migration stay in one package on purpose.
- `commands` only parses text. It must never call Slack or the manager.
- The bot talks to nodes only through `manager`, never `local` directly.
- Shared test fakes go in `internal/testutil`. Tests that need a running cluster use `testutil/clustertest`.
## Code style
- **Few comments. Code should document itself.** Only add a comment when the code really is unclear, such as a non-obvious reason, a protocol quirk or a race. Don't add comments that repeat the code, section banners or doc comments on obvious functions.
- Follow the existing style: short functions, early returns, errors wrapped with `%w`, and user-facing text written plainly.
- Key struct literals for types from other packages (`model.Size{MemoryMB: 1024, CPUs: 1}`), or `go vet` will complain.
- Keep a function's arguments, results and locals from using a package's name (`node`, `state`, …).
- Text that Slack users see lives in `bot` and goes in plain sentences. Keep it consistent with USERGUIDE.md.
## Behaviour you must not break
- The `state.json` format: existing owners, sessions and permissions must still load. JSON field names are effectively a public format.
- The cluster wire protocol: frame types and call names in `manager/cluster.go`. A master and slaves may run different versions for a while.
- Machine names (`u-<id>`, `c-<id>`), unit names (`slackerbot-<machine>.service`), and paths (`/var/lib/slackerbot`, the agent env file at `/var/lib/slackerbot/agent.env`, which must match `nix/container.nix`).
- Anything that decides whether a message is a command or goes to the agent: `commands.ErrNotCommand`. Ordinary sentences must still reach the agent.
## Docs
Update the matching doc in the same change:
- Slack commands, replies or permissions → USERGUIDE.md
- Packages, data flow, protocol, placement → ARCHITECTURE.md
- Config, Nix module options, setup steps → INSTALLATION.md and `config.example.toml`
- Workflow or rules → this file
## Git
- Don't commit, push or rewrite history unless asked.
- Never commit build output (`result`, the `slackerbot` binary, `.direnv/`).
+108
View File
@@ -0,0 +1,108 @@
# Architecture
## System
```mermaid
flowchart LR
slack([Slack]) <-->|Socket Mode| bot
subgraph master["Master host"]
bot[bot] --> mgr[manager]
bot <-->|ACP over WebSocket| acp[acp client]
mgr --> local1[local node]
mgr --> state[(state.json)]
local1 -->|systemd-run + machined D-Bus| c1["nspawn: c-…<br/>goose serve :1984"]
local1 -->|btrfs subvolume + quota| d1[(drives)]
end
subgraph slave["Slave host (0..n)"]
local2[local node] --> c2["nspawn: u-…<br/>goose serve :1984"]
local2 --> d2[(drives)]
end
acp -.->|direct| c1
mgr <==>|"TLS WebSocket (slave dials out)<br/>calls · reports every 5s · agent relay · btrfs send/receive"| local2
acp -.->|relayed over the link| c2
```
**One binary, two modes.** `-mode master` runs the Slack bot, the manager and a local node. `-mode slave` runs only a local node, and keeps one outgoing TLS WebSocket to the master, so slaves can sit behind NAT.
**A message's path:** Slack event → `bot` → `manager.Ensure` (find the container, or place and start it) → `manager.DialAgent` (direct on the master, relayed for slaves) → `acp` session → updates stream back through `bot`'s streamer, which edits the Slack message.
**Containers** are `systemd-nspawn` machines started through `systemd-run`, each in its own `slackerbot-<machine>.service` unit, with CPU and memory limits. They boot the NixOS system built by `nix build .#agent` (`nix/container.nix`), which runs `goose serve` on port 1984. `/nix/store` is shared read-only. `/root` is bind-mounted from a btrfs subvolume on one of the node's drives, and is the only thing that persists. The agent's env file (API keys, the per-container goose secret) is written by the host and bind-mounted in read-only.
**State** lives only on the master, in `state_dir/state.json`: owners (container, size, node, drive, storage limits, idle limit), sessions (Slack thread → ACP session id) and permissions. Nodes report what's running and how much disk each home uses. Nothing else is stored.
## Placement and capacity
- Each node's usable compute = total RAM/CPU − `host_reserve`. Each running container counts its full limit.
- Each drive's usable space = free + what homes use − `reserve`. Each home counts its guarantee or its real usage, whichever is bigger. The caps added up may exceed the space (overbooking is allowed, and the bot flags it).
- New containers go where `max(cpu load, ram load, storage load)` after placement stays lowest (`balance.ChoosePlacement`), on the drive with the most headroom.
- Containers stay on their node until moved. When a node is too full to start a container, the bot moves it to another node automatically.
## Cluster link
- The slave connects to `wss://master/cluster` with `X-Cluster-Secret` and `X-Node-Name`, and pins the master's self-signed certificate by fingerprint, either from the config or trusted on first use.
- Frames are JSON: `call`/`reply` for node operations, `report` for node info every 5s, and `data`/`close` for byte streams. Agent connections are tunnelled as streams.
- A slave that hasn't reported for 20s, or whose link has dropped, counts as offline. The master doesn't trust reports for a short while after it asks a slave to start or stop something, so stale reports don't undo the change.
## Migration
`manager.Migrate`: pick a target with room, `btrfs send` a read-only snapshot in 4 MB chunks through the master into `btrfs receive` on the target, and keep a running container up during this pass. Then the bot waits for any reply in progress to finish, stops the container, sends an incremental snapshot, makes a writable snapshot the new home, switches the owner record, deletes the old home, and starts the container on the target. If anything fails before the switch, the bot rolls back: the half-copied home is removed and the container restarts where it was.
## Code
All Go lives in `slackerbot/` (module `slackerbot`). Packages under `internal/` each do one job, and only import downward:
```mermaid
flowchart TD
main[main] --> bot & manager & local & state & config
bot --> manager & acp & commands & balance
manager --> balance & state & config & storage
local --> config & storage
acp --> node
balance --> node & storage
commands --> units
config --> units
manager --> node
local --> node
node --> model
state --> model
commands --> model
config --> model
balance --> model
units
storage
util
```
`util` (`KeyedMutex`, `RandomHex`) is used by `bot`, `manager` and `local`, and left out of the diagram to keep it readable.
| Package | Role |
|---|---|
| `main.go` | Flags, wiring, master vs slave |
| `model` | Shared types: `Size`, `Owner`, `Session`, permission rules, machine names |
| `units` | Parses and prints sizes (`5G`, `1.5 TB`) |
| `commands` | Parses Slack command text: `scale`, `keep`, `storage`, `grant`, and which permission a command needs |
| `config` | Loads and validates `config.toml` |
| `storage` | Drive math (`DriveReport`, headroom, overbooking) and btrfs subvolume/quota commands |
| `state` | `state.json` persistence |
| `node` | `Node` interface, wire types, offline/full errors, the WebSocket connection used to reach goose |
| `local` | The `Node` on this machine: nspawn via `systemd-run`, machined and systemd over D-Bus, capacity detection, btrfs send/receive |
| `balance` | Builds a view of each node and picks where a container goes |
| `manager` | `Manager` (owners, placement, start/stop/resize/storage), the cluster link (`Remote`, `ServeCluster`, `RunSlave`) and migration. These three share a package because they're tightly coupled |
| `acp` | JSON-RPC client for goose's ACP endpoint. Tool permission requests are auto-approved |
| `bot` | Slack events and buttons, replies, permission checks, `grant`/`revoke`, move announcements, idle reaper, streaming replies |
| `util` | `KeyedMutex`, `RandomHex` |
| `testutil`, `testutil/clustertest` | Fake nodes, and a real master + slave harness for tests |
| `nix/container.nix` | NixOS system for each container |
| `nix/host.nix` | NixOS module: the service, the `ve-*` network, firewall |
## Known limits
- The agent's tool calls are auto-approved, since the container is the sandbox (`acp.handleRequest`).
- Anyone in a channel controls that channel's container and everything the agent can reach from it.
- Containers run as root without user namespaces, and container root can get around btrfs quotas. That's fine for fair sharing among trusted users, but not safe for hostile ones.
- Restarting the master makes slaves look offline for a few seconds. Restarting the bot doesn't stop containers.
- `state.json` is fine for hundreds of users. Swap the `state` package for SQLite if you need more.
- Home folders are never deleted automatically (see INSTALLATION.md).
+94
View File
@@ -0,0 +1,94 @@
# Installation
## Requirements
- Linux with systemd (`systemd-nspawn`, `systemd-machined`, `systemd-networkd`), running as root.
- One or more **btrfs** folders for home folders. Quotas are switched on automatically.
- Nix with flakes, to build the bot and the container system. NixOS is easiest, but not required.
- An API key for the model goose should use (e.g. Anthropic).
## 1. Slack app
1. Go to <https://api.slack.com/apps> → **Create New App** → **From a manifest**, and paste `slack-manifest.yaml`.
2. Install it to your workspace and copy the **bot token** (`xoxb-…`).
3. Under **Basic Information → App-Level Tokens**, create a token with the `connections:write` scope (`xapp-…`).
The app uses Socket Mode, so the bot needs no public URL.
## 2. Container system
Build the NixOS system each container boots:
```
nix build git+https://gitea.skyg.ca/ari/slacker-bot.git#agent -o /var/lib/slackerbot/agent-system
# or, from a checkout:
nix build .#agent -o /var/lib/slackerbot/agent-system
```
Rebuild the same way after changing `nix/container.nix`. Each container picks up the new build on its next start. `nix/container.nix` runs `goose serve`, which needs a recent goose. If nixpkgs' `goose-cli` is too old, override it there.
## 3. Config
Copy `config.example.toml` and edit it. The important parts:
| Setting | What |
|---|---|
| `system` | The build from step 2, e.g. `/var/lib/slackerbot/agent-system` |
| `state_dir` | Where `state.json`, env files and cluster certificates live |
| `[agent_env]` | Environment for goose (provider, model, API key). `$VARS` are expanded from the bot's environment |
| `[container]`, `[max_container]` | Default and largest container size |
| `[host_reserve]` | RAM/CPU kept free for the host |
| `idle_timeout`, `max_idle_timeout`, `allow_indefinite` | Idle shutdown rules |
| `[storage]` | Default guaranteed (`min`) and cap (`max`) home size, and the largest cap anyone may set (`limit`) |
| `[[drives]]` | btrfs folders for home folders, each with a `reserve` kept free |
| `[permissions]` | Starting permissions, copied in on first start only (see USERGUIDE.md) |
| `[cluster]` | Only for more than one node (see step 5) |
Secrets go in the environment rather than the file: `SLACK_APP_TOKEN`, `SLACK_BOT_TOKEN`, `CLUSTER_SECRET`, and any API keys used in `[agent_env]`.
## 4. Run the bot
### NixOS
Add the flake to your system flake and import the host module:
```nix
inputs.slackerbot.url = "git+https://gitea.skyg.ca/ari/slacker-bot.git";
# in your host's modules:
imports = [ inputs.slackerbot.nixosModules.host ];
services.slackerbot = {
enable = true;
configFile = ./slackerbot.toml;
environmentFile = "/etc/slackerbot/secrets"; # SLACK_APP_TOKEN=…, SLACK_BOT_TOKEN=…, ANTHROPIC_API_KEY=…
};
```
The module runs the bot as a systemd service with `/var/lib/slackerbot` as its state directory. It also sets up `systemd-networkd` to give each container's `ve-*` link an address over DHCP with NAT, and trusts those links in the firewall.
### Other Linux
```
nix build git+https://gitea.skyg.ca/ari/slacker-bot.git#slackerbot # or: cd slackerbot && go build
sudo slackerbot -config /etc/slackerbot/config.toml
```
Make sure `systemd-networkd` manages `ve-*` links. systemd ships `80-container-ve.network` for this. `btrfs-progs` must be on the bot's `PATH`.
Check the log for the node's capacity line and "connected to Slack", then DM the bot `help`.
## 5. More nodes (optional)
1. On every node, set the same `[cluster] secret` (16+ characters), plus its own `system`, `[[drives]]` and `host_reserve`, and build the container system there. Slaves don't need Slack tokens or `agent_env`, because the master sends those.
2. **Master:** set `cluster.listen = ":7946"` and open the port. On NixOS: `services.slackerbot.clusterPort = 7946`.
3. **Slaves:** set `cluster.master = "wss://<master>:7946"` and a unique `cluster.name` (default: the hostname). Run with `-mode slave`. On NixOS: `services.slackerbot.mode = "slave"`.
4. The master logs its certificate fingerprint at startup. Put it in each slave's `cluster.master_fingerprint`. If you leave it empty, a slave trusts the first certificate it sees, saves it in `state_dir/master.fingerprint`, and refuses any other from then on.
`nodes` in Slack should now list every node.
## Maintenance
- **Permissions** live in the master's `state.json` after the first start, and editing the config doesn't change them. To start over from the config, stop the master and delete the `permissions` entry from `state.json`.
- **Deleting a home folder for good:** stop its container, run `btrfs subvolume delete <drive path>/<u-… or c-…>`, then, with the bot stopped, remove its owner from `state.json`. Nothing deletes homes automatically, so their guaranteed space stays taken. This includes homes left on a dead slave after **Create new container**.
- **Upgrading** the bot doesn't stop running containers, because each runs in its own `slackerbot-<machine>.service` unit.
- Find user and channel IDs in Slack with **Copy member ID** on a profile, or at the bottom of a channel's **About** tab.
+20 -237
View File
@@ -1,246 +1,29 @@
# slackerbot
# Slackerbot
A Slack bot that runs goose agents in NixOS containers and connects them to Slack.
A Slack bot that gives every DM and every channel its own AI coding agent, running in its own NixOS container.
## Who owns a container
Talk to the bot in Slack, and your message goes to a [goose](https://github.com/block/goose) agent inside a `systemd-nspawn` container. The agent's reply streams back into a Slack thread. Replying in that thread continues the same session.
| Where you talk to the bot | Container | Who can use it |
|---|---|---|
| DM with the bot | `u-<user id>` | Only you |
| Channel, private channel or group DM (@mention the bot) | `c-<channel id>` | Anyone in that channel |
- **One container per DM or channel.** DMs get a private container, and channels share one.
- **Persistent home folder.** `/root` lives on a btrfs subvolume with a guaranteed size and a cap. Everything else is rebuilt on each start.
- **Managed from Slack.** Users can `scale`, `stop`, set idle timeouts with `keep`, resize `storage`, and `move` containers between nodes. Changes are limited by per-channel permissions.
- **One or more machines.** A master runs the Slack bot, and slaves add capacity. New containers go wherever CPU, RAM and storage stay most even, and live migration moves a running container with only a short pause.
In channels, every message reaches the agent prefixed with the sender's name (e.g. `Alice: try the other branch`), so it knows who's asking.
## Docs
## What happens when someone messages the bot
| File | For |
| ---------------------------------- | -------------------------------------------------- |
| [INSTALLATION.md](INSTALLATION.md) | Setting up the Slack app, the host and extra nodes |
| [USERGUIDE.md](USERGUIDE.md) | Slack commands and how permissions work |
| [ARCHITECTURE.md](ARCHITECTURE.md) | How the system and the code are put together |
| [AGENTS.md](AGENTS.md) | Rules for changing the code: tests, layout, style |
1. The bot finds the container for that DM or channel.
- **Running:** uses it.
- **Stopped:** starts it again. Its home folder (`/root`) is still there, and everything else is rebuilt fresh.
- **Doesn't exist:** creates it on the node where CPU, RAM and storage stay most even (see "Running more than one node"), with its home folder on that node's roomiest drive.
- **Everything is full:** replies "maxed out" instead.
- **Its node is offline:** replies "Channel bot is on a dead slave `<name>`, create new container or wait for the slave to be back up?" with **Create new container** and **Wait** buttons.
2. It waits until goose answers on port **1984** inside the container.
3. A new message starts a new agent session, and the reply goes in a thread.
Replying in that thread continues the same session, even days later.
4. The agent's answer streams into the thread as it works, with tool calls shown as `🔧` lines.
5. Every 10 minutes, containers that have gone unused past their limit are shut down. The limit is 6 hours by default, or whatever `keep` set. The owning DM or channel gets a message saying so.
6. Messages in the same thread run one at a time. If two people post at once, the second waits its turn.
## Quick start
## Commands (send as a new message, not in a thread)
| Command | What it does |
|---|---|
| `sessions` | Lists this DM's or channel's past sessions with links. Open one and reply in its thread to continue. |
| `status` | Shows the container's state, size, time unused and idle limit, with **Stop** and **Scale up** buttons. |
| `stop` | Shuts the container down. Files are kept. |
| `scale up` / `scale down` | Doubles or halves RAM and CPU. |
| `scale 8G 4cpu` | Sets an exact size. No sign means "set to". |
| `scale +2G` / `scale -1cpu` / `scale 512M +0.5cpu` | A `+` or `-` sign means "change by". Memory and CPU can be mixed or left out. |
| `keep 24h` (or `3d`, `1d12h`, `2w`, `5 hours`) | Stops the container after that long unused, instead of the default. |
| `keep indefinite` (or `forever`) | Never stops it for being unused. Use `stop` when done. |
| `keep default` | Goes back to the config's `idle_timeout`. |
| `keep` | Shows the current idle limit. |
| `storage` | Shows home folder usage, guaranteed space and max. |
| `storage 20G` / `storage +5G` / `storage -2G` | Sets or changes the max (the cap). |
| `storage min 8G` | Sets the guaranteed space. |
| `storage min 8G max 20G` | Sets both. |
| `nodes` | Lists every node with free CPU, RAM and storage, how many containers it has, and which one this container is on. Offline nodes are listed too. |
| `move <node>` (or `migrate <node>`, `move to <node>`) | Moves this container, running or not, to that node. Works inside a thread too. |
| `permissions` | Shows who can scale, change storage, move or set timeouts in this channel or DM. |
| `move auto` (or just `move`) | Moves it to whichever other node keeps resources most even. |
| `help` | Shows this list. |
**Scale rules:** memory takes `G`/`GB` or `M`/`MB`, and CPU takes `cpu`/`cpus`/`core`/`cores` (fractions are fine). Sizes stay between 256 MB / 0.25 CPU and `max_container`. Growing a running container needs room on the node. Changes apply live, or on next start if it's stopped. Shrinking memory below what's in use may stop programs inside.
**Keep rules:** the minimum is 10 minutes. `max_idle_timeout` in the config caps how long `keep` can ask for, and `allow_indefinite = false` turns off `keep indefinite`. Setting `keep` restarts the countdown.
**Storage rules:** sizes take `M`/`MB`, `G`/`GB` or `T`/`TB`. Both numbers are at least 512 MB, the guarantee can't exceed the max, and the max can't exceed `storage.limit` or go below what's already in use. Raising the guarantee is **rejected** if the drive can't promise it. Raising the max past what the drive can hold for everyone is **allowed but flagged** as overbooked.
If the words after `scale`, `keep`, `storage` or `move` don't look like a size or a time (e.g. "keep the tests passing"), the message goes to the agent instead.
Commands act on the container that belongs to where you send them. In a channel, anyone can stop or scale the channel's container. In channels, @mention the bot every time, including in threads. Plain replies aren't seen.
## Super users and permissions
Changing a container is limited to super users, one permission at a time:
| Permission | Covers |
|---|---|
| `scale` | `scale …` and the **Scale up** button |
| `storage` | `storage <size>`, `storage min …` |
| `move` | `move …`, `move auto`, and **Create new container** on a dead slave |
| `keep` | `keep <time>`, `keep indefinite`, `keep default` |
| `all` | The four above (not `manage`) |
| `manage` | Changing this channel's super-user list from Slack |
**Where they're kept:** in the master's `state.json`. The config's `[permissions]` are copied in on the very first start and ignored after that, so changes made in Slack stick. To start over from the config, stop the master and delete the `permissions` entry from `state.json`.
```toml
[permissions]
managers = ["U0ADMIN"] # user managers: can change everything below from Slack
[permissions.default] # used by every channel without its own list
U0ADMIN = ["all"]
[permissions.channels.C07ABC123] # replaces the default list for this channel
U0111 = ["move"]
U0222 = ["all", "manage"] # U0222 also runs this channel's list
[permissions.dm] # what anyone may change on their own DM container
scale = true
storage = false
move = true
keep = true
```
**User managers** (`managers`) can change any channel's list, the default list, the DM flags, and the manager list. **Channel managers** (`manage` in a channel's list) can change only that channel's list. At least one user manager is always kept.
**Managing from Slack:**
| Command | Who | What it does |
|---|---|---|
| `permissions` | anyone | Shows this channel's (or DM's) super users and the user managers |
| `grant @alice @bob move scale` | managers of this channel | Grants permissions here. The first grant in a channel copies the default list, so nobody loses access |
| `revoke @alice move` / `revoke @alice` | managers of this channel | Takes away some, or all, of someone's permissions here |
| `permissions reset` | managers of this channel | This channel goes back to the default list |
| `grant default @alice all` / `revoke default @alice` | user managers | Changes the default list |
| `dm storage on` / `dm storage off` | user managers | Changes what people may do on their own DM container |
| `managers` / `managers add @alice` / `managers remove @alice` | user managers (to change) | Shows or changes the user managers |
People named in a `grant`, `revoke` or `managers` change are tagged in the confirmation, so they're notified.
**Other details:**
- Showing settings (`storage` or `keep` with nothing after them) is open to everyone, as are `status`, `nodes`, `sessions`, `permissions`, `stop` and talking to the agent.
- Automatic moves (when a node is full) don't need a permission.
- Someone without permission gets a short :lock: reply naming who has it, without tagging them.
- DM flags that are left out count as allowed.
- User and channel IDs: in Slack, use "Copy member ID" on a profile, or the ID at the bottom of a channel's About tab.
## Files
All Go code lives in `slackerbot/` (the Go module root). Each package under `slackerbot/internal/` does one job:
| Package | What's in it |
|---|---|
| `main.go` | Startup: config, capacity check, Slack connection |
| `config` | Reads `config.toml` |
| `model` | Shared types: sizes, owners, sessions, super users and per-action permissions |
| `units` | Reads and prints sizes like `5G` |
| `commands` | Understands `scale`, `keep`, `storage` and `grant` messages |
| `storage` | Drive math, btrfs subvolumes and quotas |
| `state` | Remembers owners (DMs and channels), sessions, last activity and permissions in `state.json` |
| `node` | What every node (this machine or a slave) can do |
| `local` | Runs containers on this machine (`systemd-run` + machined over D-Bus, btrfs homes, send/receive), and reads its RAM/CPU |
| `balance` | Picks the node that keeps CPU, RAM and storage most even |
| `manager` | Owners, placement, resizing and storage across all nodes; the master/slave link (TLS WebSocket, agent relay, certificate pinning); moving containers |
| `acp` | Talks to goose (ACP over WebSocket at `/acp`) |
| `bot` | Slack events, commands, buttons, idle cleanup, streaming replies, `grant`/`revoke`/`dm`/`managers` |
| `util`, `testutil` | Small helpers, and fakes for tests |
| `nix/container.nix` | The NixOS system each container boots, with goose on 1984 |
| `nix/host.nix` | Host module: runs the bot and gives containers a network |
## Development
`nix develop` gives a shell with Go, gopls and the other tools. Then, inside `slackerbot/`:
git clone https://gitea.skyg.ca/ari/slacker-bot.git && cd slacker-bot
nix develop # or let direnv load it from .envrc
cd slackerbot && go test ./...
nix build .#slackerbot # the bot
nix build .#agent # the NixOS system each container boots
```
go build ./... && go vet ./... && go test ./...
```
## Setup
**1. Slack app.** Create an app from `slack-manifest.yaml`. Then:
- Install it to your workspace and copy the **bot token** (`xoxb-…`).
- Under Basic Information, make an **app-level token** with the `connections:write` scope (`xapp-…`).
**2. Build the container system.**
```
nix build .#agent -o /var/lib/slackerbot/agent-system
```
Rebuild the same way after changing `nix/container.nix`. Each container picks up the new build on its next start.
**3. Host config.** Add the flake's `nixosModules.host` to your host:
```nix
imports = [ slackerbot.nixosModules.host ];
services.slackerbot = {
enable = true;
configFile = ./slackerbot.toml; # based on config.example.toml
environmentFile = "/etc/slackerbot/secrets"; # SLACK_APP_TOKEN=…, SLACK_BOT_TOKEN=…, ANTHROPIC_API_KEY=…
};
```
After changing Go dependencies, update `vendorHash` in `flake.nix` (Nix prints the new one on the next build).
**Not using NixOS for the host?** Run `nix build .#slackerbot` (or `go build` inside `slackerbot/`), then start `slackerbot -config config.toml` as root. Make sure systemd-networkd manages `ve-*` links (systemd ships `80-container-ve.network` for this).
## How capacity works
At startup the bot reads total RAM and CPU and subtracts `host_reserve`. What's left is shared out.
Each running container counts its full limit (`memory_mb`, `cpus`) against that.
A container only starts, and scale-up only goes through, if it fits. The startup log prints roughly how many default-size containers fit.
## How storage works
Each container's home folder (`/root`, where goose keeps its settings and session history) is a btrfs subvolume on one of the `[[drives]]` in the config. That folder is the only thing that persists. Everything else in the container is rebuilt on each start.
Each home has two numbers:
- **min (guaranteed):** set aside on the drive. A new container is only placed, and a guarantee only raised, if it fits.
- **max (the cap):** a btrfs quota. Writes past it fail inside the container.
Per drive, the bot works out:
- **usable** = free space + what homes already use − the drive's `reserve`
- **guaranteed** = each home's min, or its real usage if that's bigger
- **booked** = all the maxes added up
New guarantees must fit in usable − guaranteed. When booked > usable, the drive is **overbooked**. That's allowed, and `status` and `storage` replies flag it, because if every home fills up to its max, writes start failing.
Every drive must be btrfs. The bot turns on quotas for each drive at startup.
## Running more than one node
One binary, two modes:
- `slackerbot -mode master`: runs the Slack bot and also runs containers itself.
- `slackerbot -mode slave`: runs containers only, for the master it points at.
**Setup**
1. Every node needs its own config with `system`, `[[drives]]` and `host_reserve`, plus the same `cluster.secret` (16+ characters). Each also needs the NixOS container build. Slaves don't need Slack tokens or `agent_env`, since the master sends those.
2. On the master, set `cluster.listen` (e.g. `":7946"`) and open that port. On NixOS, set `services.slackerbot.clusterPort = 7946`.
3. On each slave, set `cluster.master = "wss://<master>:7946"` and give it a unique `cluster.name` (default: hostname). On NixOS, set `services.slackerbot.mode = "slave"`.
4. The master logs its certificate fingerprint at startup. Put it in each slave's `cluster.master_fingerprint`. If you leave it empty, a slave trusts the first certificate it sees, saves it to `state_dir/master.fingerprint`, and refuses any other from then on.
**How it works**
- Slaves open one TLS WebSocket to the master and keep it open, so they can sit behind NAT. They report their CPU, RAM, drives, home folder usage and running containers every 5 seconds.
- The master starts, stops, resizes and sets quotas on a slave's containers over that link. Agent traffic for a slave's container is relayed over it too.
- A slave that hasn't reported for 20 seconds, or whose link dropped, counts as offline. Slaves reconnect on their own, backing off up to 30 seconds.
**Even distribution**
For each node, the master works out how full it would be after adding the new container: CPU (running containers' CPUs ÷ usable CPUs), RAM (the same), and storage (guaranteed space ÷ usable space on its drives). It takes the **worst of the three** and places the container where that number stays lowest. Nodes that can't fit the container's RAM, CPU or guaranteed storage are skipped. This keeps all three resources level across nodes, and it naturally avoids a node that's busy on any one of them.
**Containers stay on their node** until they're moved, because their home folder lives there. `scale`, `storage` and restarts all happen on that node. `status` and `nodes` show where each container runs and what's free on every node.
**Moving a container** (`move <node>`, `move auto`, or automatically):
1. The target must have room for the home folder's guarantee, or its current size if that's bigger. If the container is running, the target also needs room for its RAM and CPU.
2. The home folder is snapshotted on the old node and copied with `btrfs send` → master → `btrfs receive`, in 4 MB pieces. Slaves only talk to the master, so the master relays.
3. **Running containers stay up during the bulk copy.** Then the bot waits for any reply in progress to finish (up to 2 minutes), stops the container, copies only what changed since the first pass, and starts it on the new node. Downtime is just that final sync plus a restart.
4. The container's record switches to the new node, and the old copy and snapshots are deleted. Sessions carry over, since goose keeps them in the home folder.
5. Every move, manual or automatic, starts a **new top-level message** in the owning DM or channel, that @mentions whoever started it, so they're notified ("started by @Alice"). An automatic move says "started by auto process (master was full)" and tags the person whose message triggered it. Progress goes in that message's thread, and the top message gets the outcome (✅ or ❌) when it's done. The thread where the move was started (under the `move` command, or the user's own conversation for automatic moves) gets a reply linking to the move thread.
6. If anything fails before the switch, the half-made copy is removed and the container stays on, and restarts on, its old node. Messages and commands that arrive during a move get "being moved, try again in a moment."
**Automatic moves:** when a stopped container's node is too full to start it again, the bot moves it to the node with the most room and starts it there, instead of replying "maxed out." Progress shows in the same message.
You can't move off an offline node (the files are there), and a move fails if the target already has an old home folder for that container (e.g. left from "Create new container"). Delete that one first.
**When a slave dies**, its users get the dead-slave reply. These also get it: `stop`, `scale` and `storage`, plus the **Stop** and **Scale up** buttons.
- **Create new container:** moves the owner to a fresh container on a healthy node. Old threads start new sessions when replied to. The old home folder stays on the dead slave, untouched, and still counts against that slave's drive until you delete it.
- **Wait:** the bot posts in that DM or channel when the slave reconnects.
## Things to know
- **The agent's actions are auto-approved.** When goose asks permission to run a tool, the bot says yes, since the container is the sandbox. To ask the user instead, change `handleRequest` in `slackerbot/internal/acp` to post Slack buttons.
- **Anyone in a channel controls its container,** including everything the agent can reach from it. Only add the bot to channels whose members you'd trust with that.
- **Containers run as root without user namespaces.** They're isolated, but not strongly enough for people you don't trust. For untrusted users, look at nspawn's `--private-users=pick` (not turned on here, since it needs testing with a shared `/nix/store`).
- **goose version:** `nix/container.nix` uses `goose serve`, which needs a recent goose. If nixpkgs' `goose-cli` is too old, override it with a newer build.
- **Transport:** the bot uses the WebSocket side of goose's `/acp` endpoint. If your goose version only offers the HTTP+SSE side, `DialACP` in `slackerbot/internal/acp` is the one place to change.
- **btrfs quotas aren't a hard security boundary here.** Container root keeps enough privileges that a determined user could raise their own quota. They're good for fair sharing, not for hostile users.
- **Restarting the master** briefly makes every slave look offline until they reconnect, which takes a few seconds. Messages in that window get the dead-slave reply.
- **Restarting the bot doesn't stop containers.** They run as their own systemd units (`slackerbot-u-….service`), and the bot reconnects on the next message.
- **State** is a small JSON file (`state_dir/state.json`). It's fine for hundreds of users. Swap the `state` package for SQLite if you need more.
- Home folders live at `<drive path>/u-<id>` or `<drive path>/c-<id>`. To wipe one for good, stop the container, run `btrfs subvolume delete` on its home, and remove its entry from `state.json` with the bot stopped. Nothing deletes homes automatically, so their guaranteed space stays taken until you do.
+94
View File
@@ -0,0 +1,94 @@
# User guide
How to use slackerbot from Slack.
## Your container
| Where you talk to the bot | Container | Who can use it |
|---|---|---|
| DM with the bot | `u-<user id>` | Only you |
| Channel, private channel or group DM (@mention the bot) | `c-<channel id>` | Anyone in that channel |
In channels, @mention the bot every time, including in threads. The bot doesn't see plain replies. Each message reaches the agent with the sender's name in front (e.g. `Alice: try the other branch`), so the agent knows who's asking.
## Talking to the agent
1. Send a message. If the container is stopped, the bot starts it, and if it doesn't exist yet, the bot creates it. Your home folder (`/root`) is kept between starts.
2. A new top-level message starts a **new session**, and the reply goes in a thread.
3. **Reply in that thread to continue the same session**, even days later. `sessions` lists past ones.
4. The answer streams in as the agent works. Tool calls show up as 🔧 lines.
5. Messages in one thread run one at a time. If two people post at once, the second waits its turn.
6. Containers unused for longer than their idle limit are stopped (default 6 hours, see `keep`). Files are kept, and the bot posts to say it stopped the container.
Other replies you may get:
- **"maxed out":** no node has room right now. Try again later.
- **"being moved, try again in a moment":** a move is in progress.
- **"dead slave":** the node your container lives on is offline. Pick **Create new container** to get a fresh one on a healthy node (old sessions won't carry over), or **Wait** to get a message when the node is back.
## Commands
Send these as a **new message**, not in a thread. `move` is the one exception, and also works in a thread.
| Command | What it does |
|---|---|
| `help` | Shows the command list |
| `status` | State, size, time unused and idle limit, with **Stop** and **Scale up** buttons |
| `sessions` | Past sessions with links. Open one and reply in its thread to continue |
| `stop` | Shuts the container down. Files are kept |
| `scale up` / `scale down` | Doubles or halves RAM and CPU |
| `scale 8G 4cpu` | Sets an exact size |
| `scale +2G` / `scale -1cpu` / `scale 512M +0.5cpu` | `+` or `-` changes by that much. Memory and CPU can be mixed or left out |
| `keep 24h` (or `3d`, `1d12h`, `2w`, `5 hours`) | Stops the container after that long unused |
| `keep indefinite` (or `forever`) | Never stops it for being unused. Use `stop` when done |
| `keep default` | Back to the server's default |
| `keep` | Shows the current idle limit |
| `storage` | Shows home folder usage, guaranteed space and max |
| `storage 20G` / `storage +5G` / `storage -2G` | Sets or changes the max (the cap) |
| `storage min 8G` / `storage min 8G max 20G` | Sets the guaranteed space, or both |
| `nodes` | Every node with what's free on it, and where this container runs |
| `move <node>` / `move to <node>` / `migrate <node>` | Moves this container to that node |
| `move` / `move auto` | Moves it to whichever other node keeps resources most even |
| `permissions` | Who may change what here |
If what follows `scale`, `keep`, `storage` or `move` doesn't look like a size, a time or a node (e.g. "keep the tests passing"), the message goes to the agent instead.
**Scale:** memory takes `G`/`GB` or `M`/`MB`, and CPU takes `cpu`/`cores` (fractions are fine). The minimum is 256 MB and 0.25 CPU, and the server sets a maximum. A running container changes size right away, and a stopped one gets the new size when it next starts. Shrinking memory below what's in use may stop programs inside.
**Keep:** the minimum is 10 minutes. The server can cap the maximum or turn off `keep indefinite`. Setting `keep` restarts the countdown.
**Storage:** sizes take `M`, `G` or `T`. Both numbers are at least 512 MB, the guarantee can't be more than the max, and the max can't go below what's already used. If the drive can't promise a bigger guarantee, the bot refuses it. You can raise the max even when the caps on that drive already add up to more than it holds. The bot flags the drive as **overbooked** but still makes the change.
**Moves:** each move gets its own top-level message saying who started it, with progress in its thread and ✅ or ❌ when done. A running container keeps running during the bulk copy, and is only stopped briefly for a final sync. Sessions come along. When a container's node is too full to restart it, the bot moves it automatically.
## Permissions
Anyone can talk to the agent and use `status`, `sessions`, `nodes`, `permissions` and `stop`, and can view settings with a bare `keep` or `storage`.
**Changing** a container needs a permission:
| Permission | Covers |
|---|---|
| `scale` | `scale …` and the **Scale up** button |
| `storage` | `storage <size>`, `storage min …` |
| `move` | `move …` and **Create new container** |
| `keep` | `keep <time>`, `keep indefinite`, `keep default` |
| `all` | The four above |
| `manage` | Editing this channel's permission list |
Roughly how it fits together:
- **Channels** use a server-wide **default list** of super users, unless the channel has its **own list**. Your first `grant` in a channel copies the default list into it, so nobody loses access.
- **DMs** have no list. Instead, a server-wide switch per permission says whether people may change their own DM container (on unless turned off).
- **User managers** can change everything: any channel's list, the default list, DM switches and the manager list. **Channel managers** (`manage`) can only edit their own channel's list. There's always at least one user manager.
- If you lack a permission, the bot answers with :lock: and names who has it, without tagging them.
- Automatic moves don't need a permission.
| Command | Who | What it does |
|---|---|---|
| `grant @alice @bob move scale` | channel managers | Grants permissions in this channel |
| `revoke @alice move` / `revoke @alice` | channel managers | Removes some or all of someone's permissions here |
| `permissions reset` | channel managers | This channel goes back to the default list |
| `grant default @alice all` / `revoke default @alice` | user managers | Changes the default list |
| `dm storage on` / `dm storage off` | user managers | Changes what people may do on their own DM container |
| `managers` / `managers add @alice` / `managers remove @alice` | anyone / user managers | Shows or changes the user managers |
People named in a `grant`, `revoke` or `managers` change are tagged in the bot's confirmation, so Slack notifies them.