Files
nixos-setup/modules/nixos/common/containers/user-guide.md
T
2026-09-29 11:23:25 -04:00

263 lines
8.8 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Container Services — User Guide
Declares Docker Compose stacks in Nix. Each group is rendered to a `compose.yml`,
staged on disk, and managed by a systemd oneshot unit.
## Where things live
| What | Path |
| -------------------------------------------------------------------------------- | ----------------------------------------------------------- |
| Staged compose file | `/var/lib/container-services/<group>/compose.yml` |
| Staged overrides file (composeFile groups with networks/volumes/extraConfig set) | `/var/lib/container-services/<group>/compose.overrides.yml` |
| Nix store source | `/nix/store/…-compose.yml` (copied on each activation) |
| Systemd unit | `container-services-<group>.service` |
| Container logs | `docker compose -p <group> logs -f <service>` |
| Unit journal | `journalctl -u container-services-<group> -f` |
## Quick reference
```bash
# See what's running
docker compose -p <group> ps
# Follow container logs
docker compose -p <group> logs -f <service>
# Inspect the generated compose file
cat /var/lib/container-services/<group>/compose.yml
# Restart the stack (e.g. after nixos-rebuild switch)
systemctl restart container-services-<group>
# Stop / start without a rebuild
systemctl stop container-services-<group>
systemctl start container-services-<group>
# Shell into a running container
docker compose -p <group> exec <service> bash
# Pull a new image and recreate
docker compose -p <group> pull
systemctl restart container-services-<group>
```
## Debugging
### Unit fails to start
```bash
systemctl status container-services-<group>
journalctl -u container-services-<group> -n 50
```
Common causes:
- **Docker not running** — `systemctl status docker`
- **Secret path missing** — agenix may not have decrypted yet; check `systemctl status agenix`
- **Port already in use** — another service holds the host port; `ss -tlnp | grep <port>`
- **Image not pullable** — network issue or wrong image name; try `docker pull <image>` manually
### Container starts but misbehaves
```bash
# Last 100 lines from the container
docker compose -p <group> logs --tail=100 <service>
# Inspect the compose file Nix generated
cat /var/lib/container-services/<group>/compose.yml
# Verify env file was read correctly (check what the container sees)
docker compose -p <group> exec <service> env | grep <KEY>
```
### Permission errors on the data volume
The container's init runs as root and chowns `/opt/data` to `PUID:PGID`. If it still
fails, check ownership on the host:
```bash
ls -la /var/lib/<service-data-dir>
# Should match PUID:PGID set in environment {}
```
If the directory was created by a previous run as root and never chowned, remove it and
let Docker recreate it (named volumes are unaffected; only bind mounts need this):
```bash
sudo rm -rf /var/lib/<service-data-dir>
systemctl restart container-services-<group>
```
### compose.yml looks wrong
The file at `/var/lib/container-services/<group>/compose.yml` is overwritten on every
`ExecStartPre`. If it looks wrong, check the Nix-rendered source in the store directly:
```bash
# Find the store path
systemctl cat container-services-<group> | grep ExecStartPre
# Then cat it
cat /nix/store/…-compose.yml
```
If the store file is wrong, the bug is in your Nix options — rebuild with
`nixos-rebuild build` and inspect the new store path before switching.
## Custom Files
Mount custom files into containers via the `files` option:
```nix
services.app = {
image = "my/app";
files = {
"/etc/app/config.yaml" = ''
server:
port: 8080
'';
"/usr/local/bin/startup.sh" = ''
#!/bin/bash
echo "Starting..."
'';
};
};
```
Files are:
- Written to `/var/lib/container-services/<group>/files/` on the host
- Mounted read-only into the container at the specified paths
- Updated automatically when you rebuild
- Changes trigger an automatic restart (via the `container-services-<group>-files` unit)
---
## Secret Compose Definitions (External `composeFile`)
Keep a whole stack's definition out of version control by storing it in an
agenix secret and pointing the group at it:
```nix
# In your host file
age.secrets.my-stack = {
file = ../secrets/my-stack.age;
};
skyg.nixos.common.container-services.my-stack = {
enable = true;
composeFile = config.age.secrets.my-stack.path;
# Optional: merged in as a base layer, overridden by composeFile itself
# on any overlapping keys.
networks = {
lan = {
driver = "macvlan";
driver_opts.parent = "enp2s0";
};
};
};
```
When `composeFile` is set, the module:
- Copies that file to `/var/lib/container-services/<group>/compose.yml` at
startup, just like a rendered one. It must define `services` itself —
ignores this group's `services`/`files` blocks.
- If this group also sets `networks`/`volumes`/`extraConfig`, those are
rendered to `/var/lib/container-services/<group>/compose.overrides.yml` and
passed to compose **before** the composeFile (`-f compose.overrides.yml -f
compose.yml`), so compose's native multi-file merge applies —
composeFile's own definitions win on any overlapping keys.
- Requires exactly one of `composeFile` or `services` to be set (assertion).
Notes:
- `env_file:` entries in the external file resolve **relative to**
`/var/lib/container-services/<group>/`. Use absolute paths (e.g. another
agenix secret path) or inline `environment:` instead.
- Create the secret with `skyg encrypt <secret-name>`.
---
## Shared Networks (created on the host)
Compose networks declared inside a group are scoped to that group's compose project, so two
groups would each get their own copy. To share **one** network across several stacks (e.g. a
single macvlan network), declare it once with `skyg.nixos.common.containers.networks` and
reference it as `external` via its computed `compose` attribute:
```nix
# once, e.g. in the host file
skyg.nixos.common.containers.networks.lab = {
driver = "macvlan";
driverOpts.parent = "eno1";
subnet = "10.0.0.0/16";
gateway = "10.0.0.1";
ipRange = "10.0.101.16/28";
};
skyg.nixos.common.container-services.my-stack = {
enable = true;
networks = {
internal = { driver = "bridge"; };
lan = config.skyg.nixos.common.containers.networks.lab.compose;
};
services.app = {
image = "example/app:latest";
networks = {
internal = { };
lan = { ipv4_address = "10.0.101.3"; };
};
};
};
```
The network is created by a host systemd unit (`container-network-<name>.service`) before any
group starts; groups only reference it (`external: true`) and never create it.
---
## Internal DNS names
Internal DNS names are declared centrally in `skyg.internalNetworkingMap` (see
`modules/main.nix`), not per service. Each app gets an internal name (`<name>` + `rootDns`,
e.g. `drawdb.app.internal`) and an optional public alias, and the flake renders them into
the OpenWrt router's `dnsmasq.conf` via `skyg openwrt`. **Nothing is configured on the host.**
```nix
# modules/main.nix
skyg.internalNetworkingMap.apps.drawdb = { ip = "10.0.101.2"; };
```
Service files read the IP/names from the map instead of hardcoding them:
```nix
skyg.nixos.common.container-services.drawdb.services.drawdb = {
image = "ghcr.io/drawdb-io/drawdb:latest";
networks.lan.ipv4_address = config.skyg.internalNetworkingMap.apps.drawdb.ip;
};
```
| Field | Notes |
| -------------- | --------------------------------------------------------------------- |
| `ip` | LAN IPv4 address the app's names resolve to. |
| `aliases` | Optional extra names (e.g. public domains) that also resolve to `ip`. |
| `effectiveDns` | Read-only; computed as `<app-name>` + `rootDns`. |
Verify and deploy:
```sh
nix eval .#dnsRecords --json | jq # every record
skyg openwrt --dry-run # render dnsmasq.conf locally
skyg openwrt # diff + confirm + apply
```
---
## Limitations
- **One unit per group.** No per-container systemd units. Use `docker compose logs` for
container output, `journalctl` only for the start/stop lifecycle.
- **`compose down` on stop.** Containers are removed when the unit stops. Data in
bind mounts and named volumes is preserved.
- **Compose project name == group name.** Keep group names DNS-label–safe (letters,
digits, hyphens).