From 650bcff83aed8138974b242584d2244767048506 Mon Sep 17 00:00:00 2001 From: Ari Sosnovsky Date: Sun, 1 Nov 2026 14:20:13 -0500 Subject: [PATCH] better startup flow --- ARCHITECTURE.md | 42 +++--- USERGUIDE.md | 93 ++++++------- slackerbot/internal/bot/bot.go | 21 ++- slackerbot/internal/bot/lifecycle.go | 154 ++++++++++++++++++++++ slackerbot/internal/bot/lifecycle_test.go | 55 ++++++++ slackerbot/internal/manager/manager.go | 34 +++++ slackerbot/internal/model/model.go | 4 + slackerbot/internal/state/store.go | 10 ++ 8 files changed, 345 insertions(+), 68 deletions(-) create mode 100644 slackerbot/internal/bot/lifecycle.go create mode 100644 slackerbot/internal/bot/lifecycle_test.go diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 79422f2..66d4ce8 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -29,9 +29,11 @@ flowchart LR **A message's path:** Slack event (`app_mention`, a DM, or a plain reply in a channel thread set to `all-messages`) → `bot` → `manager.Ensure` (find the container, or place and start it) → `manager.DialAgent` (direct on the master, relayed for slaves) → `acp` session → updates stream back through `bot`'s streamer, which edits the Slack message. +**Lifecycle threads:** `manager.EnsureProgress` reports `StartEvent`s (placing, booting, registered) only when it really starts a container. The bot (`bot/lifecycle.go`) turns them into replies in a dedicated top-level message for that container, pins it, and unpins the previous run's message. The message's channel and timestamp are kept on the owner (`lifecycle_channel`, `lifecycle_ts`). The user's thread only gets a link. Stops (`stop`, the button, the idle reaper) post to the same thread, and `manager.WaitStopped` confirms when the machine is gone. + **Containers** are `systemd-nspawn` machines started through `systemd-run`, each in its own `slackerbot-.service` unit, with CPU and memory limits. They boot the NixOS system built by `nix build .#agent` (`nix/container.nix`), which runs `goose serve` on port 1984. `/nix/store` is shared read-only. `/root` is bind-mounted from a btrfs subvolume on one of the node's drives, and is the only thing that persists. The agent's env file (API keys, the per-container goose secret) is written by the host and bind-mounted in read-only. -**State** lives only on the master, in `state_dir/state.json`: owners (container, size, node, drive, storage limits, idle limit), sessions (Slack thread → ACP session id, plus the thread's listen mode), each channel's default listen mode, and permissions. Nodes report what's running and how much disk each home uses. Nothing else is stored. +**State** lives only on the master, in `state_dir/state.json`: owners (container, size, node, drive, storage limits, idle limit, lifecycle message), sessions (Slack thread → ACP session id, plus the thread's listen mode), each channel's default listen mode, and permissions. Nodes report what's running and how much disk each home uses. Nothing else is stored. ## Placement and capacity @@ -78,25 +80,25 @@ flowchart TD `util` (`KeyedMutex`, `RandomHex`) is used by `bot`, `manager` and `local`, and left out of the diagram to keep it readable. -| Package | Role | -|---|---| -| `main.go` | Flags, wiring, master vs slave | -| `model` | Shared types: `Size`, `Owner`, `Session`, permission rules, machine names | -| `units` | Parses and prints sizes (`5G`, `1.5 TB`) | -| `commands` | Parses Slack command text: `scale`, `keep`, `storage`, `grant`, and which permission a command needs | -| `config` | Loads and validates `config.toml` | -| `storage` | Drive math (`DriveReport`, headroom, overbooking) and btrfs subvolume/quota commands | -| `state` | `state.json` persistence | -| `node` | `Node` interface, wire types, offline/full errors, the WebSocket connection used to reach goose | -| `local` | The `Node` on this machine: nspawn via `systemd-run`, machined and systemd over D-Bus, capacity detection, btrfs send/receive | -| `balance` | Builds a view of each node and picks where a container goes | -| `manager` | `Manager` (owners, placement, start/stop/resize/storage), the cluster link (`Remote`, `ServeCluster`, `RunSlave`) and migration. These three share a package because they're tightly coupled | -| `acp` | JSON-RPC client for goose's ACP endpoint. Tool permission requests are auto-approved | -| `bot` | Slack events and buttons, replies, permission checks, `grant`/`revoke`, move announcements, idle reaper, streaming replies, thread listen modes, stop reactions, the assistant pane | -| `util` | `KeyedMutex`, `RandomHex` | -| `testutil`, `testutil/clustertest` | Fake nodes, and a real master + slave harness for tests | -| `nix/container.nix` | NixOS system for each container | -| `nix/host.nix` | NixOS module: the service, the `ve-*` network, firewall | +| Package | Role | +| ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `main.go` | Flags, wiring, master vs slave | +| `model` | Shared types: `Size`, `Owner`, `Session`, permission rules, machine names | +| `units` | Parses and prints sizes (`5G`, `1.5 TB`) | +| `commands` | Parses Slack command text: `scale`, `keep`, `storage`, `grant`, and which permission a command needs | +| `config` | Loads and validates `config.toml` | +| `storage` | Drive math (`DriveReport`, headroom, overbooking) and btrfs subvolume/quota commands | +| `state` | `state.json` persistence | +| `node` | `Node` interface, wire types, offline/full errors, the WebSocket connection used to reach goose | +| `local` | The `Node` on this machine: nspawn via `systemd-run`, machined and systemd over D-Bus, capacity detection, btrfs send/receive | +| `balance` | Builds a view of each node and picks where a container goes | +| `manager` | `Manager` (owners, placement, start/stop/resize/storage), the cluster link (`Remote`, `ServeCluster`, `RunSlave`) and migration. These three share a package because they're tightly coupled | +| `acp` | JSON-RPC client for goose's ACP endpoint. Tool permission requests are auto-approved | +| `bot` | Slack events and buttons, replies, permission checks, `grant`/`revoke`, move announcements, idle reaper, streaming replies, thread listen modes, stop reactions, the assistant pane | +| `util` | `KeyedMutex`, `RandomHex` | +| `testutil`, `testutil/clustertest` | Fake nodes, and a real master + slave harness for tests | +| `nix/container.nix` | NixOS system for each container | +| `nix/host.nix` | NixOS module: the service, the `ve-*` network, firewall | ## Known limits diff --git a/USERGUIDE.md b/USERGUIDE.md index cb91c92..2404841 100644 --- a/USERGUIDE.md +++ b/USERGUIDE.md @@ -4,9 +4,9 @@ How to use slackerbot from Slack. ## Your container -| Where you talk to the bot | Container | Who can use it | -|---|---|---| -| DM with the bot | `u-` | Only you | +| Where you talk to the bot | Container | Who can use it | +| ------------------------------------------------------- | ---------------- | ---------------------- | +| DM with the bot | `u-` | Only you | | Channel, private channel or group DM (@mention the bot) | `c-` | Anyone in that channel | In channels, @mention the bot to start a session. In a session's thread, the bot by default only reads replies that @mention it, but a thread can be switched to read every reply (see **Listening in threads** below). Each message reaches the agent with the sender's name in front (e.g. `Alice: try the other branch`), so the agent knows who's asking, and the first message of a session tells the agent which channel it's in. @@ -22,8 +22,10 @@ In DMs the bot reads every message. You can also chat with it in Slack's assista 5. To stop a reply part-way, react to your message or the bot's reply with ❌ (`:x:`) or 🛑 (`:octagonal_sign:`). The reply ends with _(cancelled)_, and the session carries on with your next message. 6. Messages in one thread run one at a time. If two people post at once, the second waits its turn. 7. Containers unused for longer than their idle limit are stopped (default 6 hours, see `keep`). Files are kept, and the bot posts to say it stopped the container. +8. **Container updates have their own thread.** Each time a container starts, the bot posts a top-level "Starting container" message in the channel (or your DM) and pins it. Every step goes in that message's thread as its own reply, saying what the bot is waiting for: placing it on a node, creating the home folder and launching it, and waiting for the agent. A last reply says the container is up. Your own thread only gets one message with a link to it. Stopping a container (`stop`, the **Stop** button or the idle limit) is reported in the same thread, ending with a reply that it's down. The pin is replaced the next time the container starts. Other replies you may get: + - **"maxed out":** no node has room right now. Try again later. - **"being moved, try again in a moment":** a move is in progress. - **"dead slave":** the node your container lives on is offline. Pick **Create new container** to get a fresh one on a healthy node (old sessions won't carry over), or **Wait** to get a message when the node is back. @@ -32,13 +34,13 @@ Other replies you may get: Channel threads start out in `only-mentions`: the bot only reads replies that @mention it. Switch with `@slacker /listen …` (the mention matters, because Slack treats a message starting with `/` as a slash command). `listen` without the slash works too. -| Command | Where | What it does | -|---|---|---| -| `@slacker /listen all-messages` | in a session's thread | The bot reads every reply in this thread | -| `@slacker /listen only-mentions` | in a session's thread | Back to replies that @mention the bot | -| `@slacker /listen` | in a thread | Shows what this thread does | +| Command | Where | What it does | +| ---------------------------------------------------------------------------------------- | ----------------------- | ----------------------------------------------------------------- | +| `@slacker /listen all-messages` | in a session's thread | The bot reads every reply in this thread | +| `@slacker /listen only-mentions` | in a session's thread | Back to replies that @mention the bot | +| `@slacker /listen` | in a thread | Shows what this thread does | | `@slacker /listen channel all-messages` (or a top-level `@slacker /listen all-messages`) | anywhere in the channel | Sets the channel's default for threads that haven't set their own | -| `@slacker /listen channel` | anywhere in the channel | Shows the channel's default | +| `@slacker /listen channel` | anywhere in the channel | Shows the channel's default | Anyone in the channel can change these. In an `all-messages` thread, `move` still needs an @mention, so a chatty "move on" doesn't move the container. DMs always read every message. @@ -46,26 +48,26 @@ Anyone in the channel can change these. In an `all-messages` thread, `move` stil Send these as a **new message**, not in a thread. `move` and `listen` are the exceptions, and also work in a thread. In the assistant pane, the first message of a chat counts as new. -| Command | What it does | -|---|---| -| `help` | Shows the command list | -| `status` | State, size, time unused and idle limit, with **Stop** and **Scale up** buttons | -| `sessions` | Past sessions with links. Open one and reply in its thread to continue | -| `stop` | Shuts the container down. Files are kept | -| `scale up` / `scale down` | Doubles or halves RAM and CPU | -| `scale 8G 4cpu` | Sets an exact size | -| `scale +2G` / `scale -1cpu` / `scale 512M +0.5cpu` | `+` or `-` changes by that much. Memory and CPU can be mixed or left out | -| `keep 24h` (or `3d`, `1d12h`, `2w`, `5 hours`) | Stops the container after that long unused | -| `keep indefinite` (or `forever`) | Never stops it for being unused. Use `stop` when done | -| `keep default` | Back to the server's default | -| `keep` | Shows the current idle limit | -| `storage` | Shows home folder usage, guaranteed space and max | -| `storage 20G` / `storage +5G` / `storage -2G` | Sets or changes the max (the cap) | -| `storage min 8G` / `storage min 8G max 20G` | Sets the guaranteed space, or both | -| `nodes` | Every node with what's free on it, and where this container runs | -| `move ` / `move to ` / `migrate ` | Moves this container to that node | -| `move` / `move auto` | Moves it to whichever other node keeps resources most even | -| `permissions` | Who may change what here | +| Command | What it does | +| --------------------------------------------------- | ------------------------------------------------------------------------------- | +| `help` | Shows the command list | +| `status` | State, size, time unused and idle limit, with **Stop** and **Scale up** buttons | +| `sessions` | Past sessions with links. Open one and reply in its thread to continue | +| `stop` | Shuts the container down. Files are kept | +| `scale up` / `scale down` | Doubles or halves RAM and CPU | +| `scale 8G 4cpu` | Sets an exact size | +| `scale +2G` / `scale -1cpu` / `scale 512M +0.5cpu` | `+` or `-` changes by that much. Memory and CPU can be mixed or left out | +| `keep 24h` (or `3d`, `1d12h`, `2w`, `5 hours`) | Stops the container after that long unused | +| `keep indefinite` (or `forever`) | Never stops it for being unused. Use `stop` when done | +| `keep default` | Back to the server's default | +| `keep` | Shows the current idle limit | +| `storage` | Shows home folder usage, guaranteed space and max | +| `storage 20G` / `storage +5G` / `storage -2G` | Sets or changes the max (the cap) | +| `storage min 8G` / `storage min 8G max 20G` | Sets the guaranteed space, or both | +| `nodes` | Every node with what's free on it, and where this container runs | +| `move ` / `move to ` / `migrate ` | Moves this container to that node | +| `move` / `move auto` | Moves it to whichever other node keeps resources most even | +| `permissions` | Who may change what here | If what follows `scale`, `keep`, `storage` or `move` doesn't look like a size, a time or a node (e.g. "keep the tests passing"), the message goes to the agent instead. @@ -83,29 +85,30 @@ Anyone can talk to the agent and use `status`, `sessions`, `nodes`, `permissions **Changing** a container needs a permission: -| Permission | Covers | -|---|---| -| `scale` | `scale …` and the **Scale up** button | -| `storage` | `storage `, `storage min …` | -| `move` | `move …` and **Create new container** | -| `keep` | `keep