Cowboy

Cowboy is an AI agent harness built on Zellij and WebAssembly. The agent runtime is a Zellij plugin compiled to wasm32-wasip1; around it sit a credential proxy, a seccomp sandbox, and a set of NixOS modules that wire everything together. It runs in two modes:

• Lite — a portable pip install, no NixOS required. The agent runs in your terminal with the core file, shell, search, and web tools.
• Full (NixOS) — agents run as managed services with network-namespace isolation, credential injection, seccomp confinement, and message bridges (Discord, email).

What it does

An agent reads your request, calls tools to act on the system, and reports back. Cowboy provides the runtime, the tools, and the boundaries those tools run inside:

• Tools that run in your real environment — read/write files, run shell commands, search, and search the web. No simulated sandbox; commands execute on the actual system.
• Persistent memory — agents save and retrieve notes across sessions via a Zettelkasten backend. See Memory System.
• Sub-agents — the main agent can spawn read-only research, code, or review sub-agents as separate Zellij panes. See Sub-Agents.
• Hashline editing — line-addressed edits with per-line hash validation, so an edit fails loudly if the file changed underneath it. See Hashline Edit.
• Multiple providers — Anthropic, OpenAI, OpenRouter, Ollama, and Codex (ChatGPT subscriptions) behind one provider interface. Vision-capable models can receive images inline or delegate image description to a separate configured model.
• Browser automation — the optional Camoufox service gives agents a real browser for navigation, snapshots, interaction, JavaScript, and screenshots. See Browser Automation.

How it stays bounded

In full mode, the agent never holds your API keys and cannot reach the network freely:

• A mitmproxy addon injects credentials into outbound requests per-domain, so keys live outside the agent. Write methods (POST/PUT/PATCH/DELETE) are blocked to any domain not on the egress allowlist.
• The agent runs in a network namespace (cowboy-ns) whose traffic is forced through the proxy.
• Shell commands are wrapped by sheepdog, a seccomp sandbox that mediates syscalls (Linux x86_64).
• Outbound messages from bridges can require human approval before they are sent.

See Security Model for the full picture.

Getting started

Lite mode:

pip install get-cowboy
cowboy --model anthropic:claude-sonnet-4-20250514

NixOS, as a managed agent service:

{
  imports = [ inputs.cowboy.nixosModules.default ];

  services.cowboy.agents.dev = {
    enable = true;
    user = "alice";
    model = "claude-sonnet-4-20250514";
  };
}

Next steps

• Installation — install lite or full mode
• Quickstart — your first session
• Configuration — config files, flags, and module options
• Design Overview — how the pieces fit together
• Security Model — the isolation boundaries

Source: github.com/dmadisetti/cowboy. Docs: cowboy.rs.

Installation

Cowboy runs in two modes:

• Lite — pip install, portable, no NixOS. The agent runs in your terminal with the core tools.
• Full (NixOS) — agents run as managed services with network isolation, credential injection, and seccomp confinement.

Both modes require Zellij on PATH — the agent runtime is a Zellij plugin. The Nix packages pull it in for you; for the pip install you provide it yourself.

Lite (pip)

pip install get-cowboy        # PyPI package is "get-cowboy", binary is "cowboy"
cowboy --version

If you don’t already have Zellij:

nix-env -iA nixpkgs.zellij    # or: cargo install zellij / brew install zellij

You can also run it without installing, via uv:

uvx --from get-cowboy cowboy --help

Nix (CLI only)

The default flake package is the WASM plugin, so install the CLI package explicitly:

# Install the cowboy CLI into your profile
nix profile install github:dmadisetti/cowboy#get-cowboy

# Or run it without installing
nix run github:dmadisetti/cowboy#get-cowboy -- --help

Full (NixOS module)

Add the flake input and import the module, then enable one or more agents under services.cowboy.agents.<name>:

# flake.nix
{
  inputs.cowboy.url = "github:dmadisetti/cowboy";

  outputs = { self, nixpkgs, cowboy, ... }: {
    nixosConfigurations.your-host = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules = [
        ./configuration.nix
        cowboy.nixosModules.default
      ];
    };
  };
}

# configuration.nix
{
  services.cowboy.agents.dev = {
    enable = true;
    user = "alice";
    model = "claude-sonnet-4-20250514";
  };
}

This gives each agent a systemd-managed session, a network namespace forced through the credential proxy, optional seccomp confinement (services.cowboy.sheepdog), and optional message bridges (services.cowboy.bridges). The full set of per-agent options lives in Configuration.

macOS is supported via cowboy.darwinModules.default (launchd + pf instead of systemd + netns). Seccomp confinement is Linux-only.

Container

nix build .#docker-image
docker load < result          # loads cowboy:latest (or: podman load < result)
docker run -it cowboy:latest

The image bundles Zellij, the Python CLI, the harness plugin, and the bridge services.

Configuration after install

Set provider credentials as environment variables (lite mode) or via the NixOS module (full mode). Config files are JSON, read in order of increasing precedence:

/etc/cowboy/config.json      # optional hand-written system defaults (not emitted by the NixOS module)
~/.config/cowboy/config.json # user overrides
.cowboy/config.json          # project overrides (lite mode)

export ANTHROPIC_API_KEY="sk-ant-..."
export EXA_API_KEY="..."      # optional, for web search

See Configuration for the full key reference.

Next steps

• Quickstart — your first session
• Configuration — config files, flags, and module options

Quickstart

This assumes Cowboy is installed (see Installation) and you have an API key for at least one provider.

Set a key

export ANTHROPIC_API_KEY="sk-ant-..."
export EXA_API_KEY="..."          # optional, enables web search

Start a session

cowboy --model anthropic:claude-sonnet-4-20250514

This opens a Zellij session running the agent. Type your request in plain language; the agent decides which tools to call and shows each call and its result inline.

To run a single task non-interactively and exit when done:

cowboy -q "Summarize the README and list any broken links"

To stop a running session:

cowboy --stop

Choosing a model

Pass provider:model. The default provider is anthropic.

cowboy --model anthropic:claude-sonnet-4-20250514   # default
cowboy --model anthropic:claude-opus-4-5-20251101
cowboy --model openai:gpt-4.1
cowboy --model openrouter:moonshotai/kimi-k2.5
cowboy --model ollama:llama3.1:8b                   # local
cowboy --model codex:gpt-5.6-luna                  # full NixOS mode

Codex uses a ChatGPT subscription. It is available in the full NixOS deployment, where the credential proxy supplies the subscription token; portable Lite mode requires a provider with a directly configured API key.

A cheaper model can be used for background summarization and context compaction:

cowboy --model anthropic:claude-opus-4-5-20251101 \
       --summary-model anthropic:claude-sonnet-4-20250514 \
       --compact-model anthropic:claude-sonnet-4-20250514

See Configuration for all flags and config keys.

Tools the agent can use

The agent has these tools by default:

It also has built-in tools for hashline editing, memory, and spawning sub-agents.

Example requests

Read README.md and tell me what this project does.
Search for TODO comments under src/ and group them by file.
Find every Python file that imports requests, then add a comment at the top of each.
What changed in the last three commits?

Next steps

• Configuration — flags, config files, and NixOS options
• Sub-Agents — delegate work to parallel agents
• Memory System — persistent notes across sessions
• Security Model — how tools are bounded in full mode

Configuration

Cowboy is configured at three layers, in increasing order of precedence:

1. CLI flags passed to cowboy or harness.
2. Config files (JSON).
3. Environment variables and keyring entries for API keys.

In NixOS deployments the NixOS module does not write /etc/cowboy/config.json (that file is an optional, hand-written system default). Instead it generates per-agent runtime config: /etc/cowboy/agents/<name>.json, /etc/cowboy/plugins.json, and /etc/cowboy/cowboy/sources.json; see NixOS module below.

Config files

Config files are plain JSON. They are merged in this order, with later files overriding earlier ones:

1. /etc/cowboy/config.json — optional hand-written system defaults (not emitted by the NixOS module).
2. ~/.config/cowboy/config.json — user overrides.
3. .cowboy/config.json — project overrides (lite mode).

A config file is a flat JSON object whose keys are forwarded to the WASM plugin. Only the keys listed in Config keys are recognized.

{
  "provider": "anthropic",
  "model": "claude-sonnet-4-20250514",
  "summary_model": "openai:gpt-4.1",
  "compact_model": "openai:gpt-4.1",
  "memory_backend": "qmd",
  "debug": "info"
}

There is no command to generate or validate this file; it is read directly if present and ignored if missing or malformed.

Config keys

These keys are read from the merged config file by the launcher (and the matching KDL keys are forwarded to the WASM plugin). Unless noted, a key is optional.

The launcher also forwards runtime values the plugin needs that are not normally set by hand: data_dir, config_dir, workspace_dir, home_dir, agent_name, wasm_path, initial_prompt, exit_on_idle, and ollama_base_url (from the OLLAMA_BASE_URL environment variable when provider is ollama). For Codex, the full NixOS deployment supplies subscription credentials through the credential proxy; no Codex API key is placed in the agent configuration.

Environment variables

API keys are discovered from the environment (and, on NixOS, from agenix files). The recognized variables are:

export ANTHROPIC_API_KEY="..."
export OPENAI_API_KEY="..."
export OPENROUTER_API_KEY="..."
export EXA_API_KEY="..."        # web search tool
export OLLAMA_BASE_URL="..."    # optional, for a remote ollama server

If no key is found and the provider is not ollama, the launcher prints a warning and continues.

CLI flags

Both entry points (cowboy, the portable launcher, and harness, the full NixOS launcher) share a common set of flags. CLI flags override config-file values.

cowboy adds:

harness adds:

Examples

# Interactive session with an explicit model
cowboy --model anthropic:claude-sonnet-4-20250514

# Distinct models for the main agent and the background tasks
cowboy \
  --model anthropic:claude-opus-4-5-20251101 \
  --summary-model openai:gpt-4.1 \
  --compact-model openai:gpt-4.1

# One-shot, non-interactive run
cowboy -q "Summarize the changes on this branch"

# Stop a named session
cowboy --stop --session my-session

Subcommands

cowboy exposes a number of verbs beyond the bare session launch (run cowboy --help for the full list); this page covers only the config-relevant one. For example, cowboy plugins lists Cowboy plugins registered in /etc/cowboy/plugins.json (NixOS deployments); it takes no flags.

Models

A model is selected as provider:model (via --model or the config provider and model keys). The models defined in the harness are:

A few short aliases are mapped to their canonical Anthropic IDs:

• claude-4-6, claude-4.6 → claude-sonnet-4-20250514
• claude-4-5, claude-4.5 → claude-opus-4-5-20251101

OpenRouter names containing a / are passed through unchanged.

NixOS module

The NixOS module is exposed as inputs.cowboy.nixosModules.default (and inputs.cowboy.darwinModules.default for nix-darwin). There is no separate home-manager module; the NixOS module manages the agent’s home-manager configuration internally.

Agents are defined under services.cowboy.agents.<name> and enabled individually with enable = true. There is no top-level services.cowboy.enable. The single-agent alias services.cowboy.agent is shorthand for services.cowboy.agents.agent.

Per-agent options

The most commonly set per-agent options:

Additional option groups exist for finer control: judge.* (idle-judge model, criteria, escalation levels), workspace.* and mounts (bind mounts into the agent’s workspace), sheepdog.* (per-agent allow/deny rules and blocked syscalls), daemon.* (persistent session resource limits), schedules (systemd-calendar prompts), and skills.homeAssistant.*. Consult modules/options/user.nix for the full schema and defaults.

The top-level services.cowboy.mode (single or multi) controls agent topology; it defaults to single when at most one agent is enabled.

Example

{ config, inputs, ... }:

{
  imports = [ inputs.cowboy.nixosModules.default ];

  services.cowboy.agents.agent = {
    enable = true;
    user = "cowboy";
    uid = 1338;

    model = "anthropic:claude-sonnet-4-20250514";
    summaryModel = "openai:gpt-4.1";
    compactModel = "openai:gpt-4.1";
    memoryBackend = "qmd";

    prompts.system = ''
      You are a coding agent. Prefer using tools over describing manual steps.
    '';
  };
}

Enforcement (on by default)

Cowboy’s three enforcement substrates are on by default — the credential proxy everywhere, the Redis ACL and the syscall sandbox on Linux (neither exists on darwin):

services.cowboy.secretsProxy.enable    # default: true
services.cowboy.pubsub.redisAcl.enable # default: true on Linux, false on darwin
services.cowboy.sheepdog.enable        # default: true on Linux, false on darwin

The credential proxy needs provider secrets at the agenix paths it expects, so a fresh host that does not have them yet has to turn it off deliberately. Whatever you turn off is named in an eval-time warning that says what stopped holding — the warnings follow what is enabled, so nothing goes quiet. See the security model.

Other sub-trees

The module exposes several optional subsystems, each with its own enable. The first two are on by default:

• services.cowboy.secretsProxy.enable — credential-injecting proxy for agent network isolation.
• services.cowboy.sheepdog.enable — syscall-level enforcement of tool-execution permissions (Linux only).
• services.cowboy.camoufox.enable — browser service for agents.
• services.cowboy.pubsub and services.cowboy.bridges — message bus and bridge services (Discord, email).

Related guides

• Memory system — the zk and qmd backends.
• Sub-agents — sub-agent models and prompts.
• Security model — sheepdog, the secrets proxy, and network isolation.

Project site: https://cowboy.rs. Source: https://github.com/dmadisetti/cowboy.

Design Overview

title: Design Overview tags: [overview, architecture]

Design Overview

Cowboy is an AI agent harness built on Zellij and WebAssembly. The agent runtime is a Zellij plugin compiled to wasm32-wasip1; everything around it (network isolation, credential injection, message bridges) is configured declaratively through NixOS modules.

Repository layout

The project is a single monorepo:

cowboy/
  crates/core/      # host-agnostic agent runtime — state machine, tools, memory,
                    #   dispatch, provider clients (Rust; has a `lite` feature)
  crates/harness/   # Zellij WASM plugin front end over core (wasm32-wasip1)
  crates/component/ # the agent built as a wasip2 component (cowboy:agent WIT world)
  crates/montana/   # headless wasmtime embedder + ndjson socket host (Linux-only)
  crates/runtime/   # the `cowboy` OCI runtime (docker --runtime=cowboy; Linux-only)
  crates/sheepdog/  # seccomp syscall sandbox (Rust, native binary)
  crates/a2a/       # protocol-neutral A2A mapping over the WIT describe() export
  crates/contracts/ # native wire contracts (Intent/Approval/Addressed, Descriptor)
  crates/ui/        # shared TUI widgets/atoms (cowboy-ui)
  crates/bootstrap/ # interactive first-boot setup wizard (a Zellij plugin)
  specs/            # spec root — WIT world (specs/wit/), CDDL records, prose
  cowboy/           # Python CLI launcher (PyPI: get-cowboy, binary: cowboy)
  proxy/            # mitmproxy addon for credential injection
  pkgs/bridge/      # Python framework for message bridges
  pkgs/discord/     # Discord bridge service
  pkgs/email/       # Email bridge service
  pkgs/matrix/      # Matrix bridge service
  modules/          # NixOS / home-manager modules (entry: modules/default.nix)

The harness has a lite feature for portable builds without NixOS dependencies. The sheepdog sandbox is compiled to a native binary and baked into the harness at build time via the SENTRY_BINARY environment variable.

The imageless resolver (external)

Rootfs materialization is not in this repo. Imageless — the flake-native resolver that turns a digest-addressed release reference into a realised rootfs — lives in its own repository and is consumed here as a first-class dependency: a flake input supplying its NixOS module and runtime package, plus a Cargo git dependency (pinned to the same rev) supplying the materializer library that crates/runtime links.

The seam is the OCI bundle boundary, not a Cowboy-internal API. cowboy-runtime receives a bundle whose config.json names a release, and asks imageless to materialize it — over the resolver daemon’s socket when the node runs one (IMAGELESS_RESOLVER_SOCKET, exported onto containerd by the module), or in-process for montana/dev where there is no socket. modules/imageless.nix carries only Cowboy’s glue over the external module: it registers the packaged, digest-addressed agent release as a node issuer and enables the resolver by default. The module’s fail-closed cache_only default stands — the agent is a release from cache.nixos.org and the node never evaluates workload Nix.

How the pieces fit

+-----------------------------------------------------------------+
|                         Zellij session                          |
|  +-----------------------------------------------------------+  |
|  |              Cowboy harness (WASM plugin)                  |  |
|  |   agent loop  ·  provider clients  ·  tool execution      |  |
|  |   context/compaction  ·  pubsub source polling            |  |
|  +-----------------------------------------------------------+  |
+-----------------------------------------------------------------+
        | web_request() (HTTP)        | shell tools (fork/exec)
        v                             v
  +------------+              +-----------------+
  | proxy      |              | sheepdog        |
  | (mitmproxy |              | (seccomp        |
  |  addon)    |              |  sandbox)       |
  +------------+              +-----------------+
        |
        v
     LLM / web APIs

The harness issues all HTTP through Zellij’s web_request() host function — no curl, no sidecar daemon. On Linux that traffic leaves through a network namespace and is DNAT’d to the proxy, which injects API credentials (see Security). Shell tool calls are mediated by the sheepdog seccomp sandbox.

Providers

The harness defines an LlmProvider trait (crates/core/src/provider/traits.rs) with implementations for Anthropic, OpenAI, OpenRouter, Ollama, and Codex. A provider produces the request URL, headers, and body, then parses the response; requests are dispatched asynchronously and results arrive as Zellij WebRequestResult events.

The model catalog lives in crates/core/data/models.json. Provider and model are selected by config key (provider, model); the default is anthropic:claude-sonnet-4-20250514. The catalog also includes Codex models (gpt-5.6-luna and gpt-5.6-sol) for ChatGPT subscriptions. In full NixOS deployments, the credential proxy injects the rotating Codex subscription token outside the agent.

Pub/sub bridges

External platforms (Discord, email, Matrix) connect to the agent through a Redis-backed pub/sub layer. Bridge services run as the broker (a human-operated trust domain), publishing inbound messages to Redis streams and consuming outbound replies. Inside the harness, SourceManager (crates/core/src/pubsub/manager.rs) polls those streams via generated shell commands (MessageSource trait) and routes replies back.

External platform        Bridge (broker)            Harness (agent)
  Discord  ----------->  ingest  --> Redis stream --> SourceManager polls
  reply    <-----------  outbox  <-- Redis stream <-- SourceManager writes

Outbound messages can require manual approval before the broker sends them; see Approvals & Outbox.

Configuration

Runtime configuration is JSON. The Python launcher (cowboy/cli.py) reads it and generates a Zellij KDL layout that passes config keys to the WASM plugin as a BTreeMap<String, String>, which the plugin parses in Config::from_configuration() (crates/core/src/config/mod.rs).

Precedence (later overrides earlier):

1. /etc/cowboy/config.json — optional hand-written system defaults (not emitted by the NixOS module)
2. ~/.config/cowboy/config.json — user overrides
3. .cowboy/config.json — project-level overrides (read by the lite init)

In a full NixOS deployment the services.cowboy module tree does not write config.json; it emits per-agent runtime config instead (/etc/cowboy/agents/<name>.json, /etc/cowboy/plugins.json, /etc/cowboy/cowboy/sources.json).

NixOS module tree

The module entry point is inputs.cowboy.nixosModules.default. All options live under services.cowboy:

• services.cowboy.agents.<name> — per-agent configuration
• services.cowboy.secretsProxy — the credential-injecting proxy
• services.cowboy.pubsub — Redis pub/sub backend
• services.cowboy.bridges.<name> — message bridge declarations
• services.cowboy.sheepdog — seccomp sandbox
• services.cowboy.camoufox — headless browser

There is no flat services.cowboy.enable / provider / model; agents are enabled individually under services.cowboy.agents.<name>.

References

• Zellij plugin API
• Claude API tool use
• OCI Runtime Spec v1.2

Security Model

title: Security Model tags: [security, network, proxy, namespace, sandbox]

Security Model

Cowboy isolates the agent with several independent mechanisms. None of them relies on the agent behaving correctly: the agent never holds API credentials, its egress is constrained at the network layer, and its shell tools run under a syscall sandbox.

This page is descriptive, not normative. It describes how the mechanisms work. What Cowboy actually guarantees — and, just as importantly, what it does not — is specs/THREAT-MODEL.md, and that document wins wherever the two disagree. The mechanisms below are on by default (next section), but each can be turned off individually; read this page against what your configuration leaves enabled, not against the prose. Each guarantee in the threat model names a flake check that fails when it stops holding.

What is enforced by default

Three of the mechanisms on this page — the credential proxy, the Redis ACL, and the syscall sandbox — used to be independently opt-in and off by default, while every description of them read as though they were active. A deployer could write a page of sheepdog deny rules and get zero enforcement and zero signal.

They are now on by default, which is what this page and the threat model’s guarantees describe:

The two darwin exceptions are not a weaker default; they are the absence of an implementation. There is no seccomp on darwin and no Redis ACL support in the darwin module tree, so switching them on there would assert a guarantee nothing delivers. Darwin must not be read as equivalent to the Linux deployment — see N-11.

The credential proxy needs provider secrets at the agenix paths it expects. A host that does not have them yet must turn it off deliberately:

services.cowboy.secretsProxy.enable = false;

Every substrate you turn off emits an eval-time warning naming what stopped holding as a result. The warnings track what is enabled — there is no way to disable one quietly.

Network isolation

On Linux (modules/network.nix) the agent runs in a dedicated network namespace cowboy-ns, connected to the host by a veth pair:

• host side: 10.200.0.1
• agent side: 10.200.0.2

Outbound TCP from the namespace is DNAT’d by iptables to the proxy. Non-TCP egress is not simply left to leak past that TCP redirect: a default-drop egress filter allows only loopback, the veth subnet, established replies, and DNS (UDP 53) to the resolver, then drops everything else — so UDP/QUIC (including HTTP/3 on UDP 443, which falls back to proxied TCP) and ICMP cannot bypass the proxy. SSH is not a blanket exception: port 22 leaves the namespace directly only for the hosts listed in secretsProxy.sshDestinations (empty by default); extra direct-egress TCP ports go through secretsProxy.passthroughPorts.

One authority remains outside this boundary and is called out honestly: operator-listed passthrough/SSH destinations (intentional holes).

The host Nix daemon is a second such authority — a fixed-output derivation’s builder gets network access on the host, outside the namespace, so a malicious builder could POST workspace data out unmediated. Because Nix build users (nixbld*) are shared system-wide, this path cannot be firewalled per agent without degrading the operator’s own builds. So agents are denied direct daemon access by default — but note precisely what enforces that, because it is narrower than it looks.

nix.settings.allowed-users cannot express the restriction. It is a list option, so definitions merge by concatenation: cowboy can only ever add users to it, never remove one. (Cowboy used to define it and no longer does — as sole definer its empty list replaced nixpkgs’ ["*"] and denied the daemon to every non-root account on the host.) The only mechanism that denies one agent is the agent-scoped seccomp rule Connect(unix:/nix/var/nix/daemon-socket/socket) in its sheepdog policy.

So allowNixDaemon = false is enforced only when sheepdog is enabled — that is, by default on Linux. With sheepdog.enable = false the setting denies nothing, and the module emits an eval warning saying so. On Darwin there is no sheepdog at all, so it is never enforced there.

An agent that is genuinely denied the daemon cannot manage its own environment implicitly; it still triggers rebuilds through the approval-gated broker (see below). Set services.cowboy.agents.<name>.allowNixDaemon = true to grant direct access to an agent you are content to let reach the shared daemon.

On Darwin (modules/darwin/network.nix) the equivalent is a UID-scoped pf redirect: agent-UID TCP is routed to the loopback proxy, and agent-UID UDP is dropped except DNS (there is no sheepdog on Darwin, so this PF rule is the only non-TCP mediation).

Credential-injecting proxy

The proxy is a mitmproxy addon (proxy/addon.py, class AgentProxy, configured by services.cowboy.secretsProxy). The agent’s requests carry no credentials; the proxy injects them per destination domain.

For each configured route the proxy reads a secret from a file (secret_file), then writes it into the request using the route’s inject_header and template. Default routes inject x-api-key for api.anthropic.com and api.exa.ai, and Authorization: Bearer … for api.openai.com and openrouter.ai. Because the secret lives only on the host side of the proxy, the agent process never sees a real API key. On an HTTP 401 the cached secret is invalidated and re-read from disk on the next request.

When more than one route matches a host, the most specific wins: an exact host route beats a wildcard, and among wildcards the longest matching suffix wins — so a broad *.example.com route can never shadow an exact api.example.com route and inject the wrong key.

Upstream TLS is always verified. The proxy injects real API keys into upstream requests, so it verifies the upstream certificate against the system root store plus any secretsProxy.trustedCa entries (identical on Linux and Darwin). There is no implicit “insecure when no custom CA is set” fallback; secretsProxy.sslInsecure is an explicit, loud, dev-only opt-out that defaults to false.

Egress control (method gating)

When egress_control is enabled, the proxy gates write methods to the allowed_write_domains allowlist; a write to a domain not on the list gets a 403. The gate is fail-closed: only GET, HEAD, OPTIONS, and TRACE are treated as reads and always permitted. Every other method — POST/PUT/ PATCH/DELETE, WebDAV verbs (MKCOL, MOVE, COPY, LOCK, …), and any unknown/custom verb — is treated as a write and must clear the allowlist. So the agent can read freely but cannot mutate external state, via any method, outside the allowlist.

Syscall sandbox (sheepdog)

Shell tool execution is mediated by sheepdog (crates/sheepdog/), a seccomp-notify sandbox. Tool processes run under a seccomp BPF filter that routes security-relevant syscalls (openat, execve, connect, …) to a userspace supervisor (“the gofer”), which checks the resolved path / argv / address against a policy and either performs the operation or returns EACCES. The policy is generated by Nix (modules/lib/policy.nix) and baked into the binary at build time.

Sheepdog is Linux x86_64 only and is not active on Darwin. See Sheepdog policy below and the authoritative contract in crates/sheepdog/POLICY.md.

Approvals

Outbound bridge messages can require human approval before the broker sends them (services.cowboy.bridges.<name>.approval). Approval state is held in Redis with a timeout; an unapproved message is auto-rejected after timeout seconds. See Approvals & Outbox.

Bridges run as the broker

Bridge services (modules/bridges.nix) run as the broker user, a separate trust domain from the agent. Each declaration generates a pub/sub source plus cowboy-<name>-{ping,ingest,outbox} systemd units under cowboy-bridges.target. The agent reaches a bridge only through pub/sub and the approval gate, not by holding the bridge’s credentials.

Bridge units are hardened by default (NoNewPrivileges, ProtectSystem = "strict", ProtectHome, and the kernel/SUIDSGID protections) — an ordinary message bridge only talks to Redis and its provider, so it never needs host privilege. A bridge that genuinely does (the rebuild bridge shells out to sudo nixos-rebuild and writes under the broker’s home) sets services.cowboy.bridges.<name>.privileged = true to opt back out; the default stays locked down so a compromised message bridge cannot escalate or write outside its own state directory.

Per-agent message isolation

With services.cowboy.pubsub.redisAcl enabled, agents are isolated from each other at the message layer (the persona model). Each agent has its own inbox keyspace {agent}:{source}:inbox and an agent_<name> Redis ACL identity scoped to ~<name>:* (its own keyspace, read/write) plus %W~*:outbox (write-only to the shared outboxes — it can reply but cannot read another agent’s pending outbound, nor any approval:* state).

Crucially the agent holds no Redis credential. It reaches Redis only over its own unix socket /run/cowboy/<agent>.sock, owned by that agent’s uid at mode 0600. One redis-auth-proxy is socket-activated off every such socket and injects the right identity based on which socket accepted the connection (its FileDescriptorName) — never SO_PEERCRED or source IP. The filesystem (socket ownership) is what enforces that an agent can reach only its own identity, and the password lives only on the proxy side.

Bridges are on the same ACL, not exempt from it

Bridges route inbound messages to the target agent’s inbox by a channel/guild → agent table. They used to do so holding a single shared identity — user bridge on >PASS ~* &* +@all — in one group-readable env file, which meant every bridge held the entire bus. That is a privilege inversion rather than a convenience: the consult bridge runs an attacker-authored prompt, and with that credential it could read every agent’s routed inbox and HSET the rebuild bridge’s pending approval to approved.

Each bridge now gets bridge_<name>, and its credential lives in /run/cowboy/bridge-<name>-redis.env at mode 0400 owned by that bridge’s own uid — every bridge shares one primary group, so a group-readable file would have undone the uid split. The grant covers:

• ~<name>:outbox, ~<name>:outbox:*, ~<name>:dead — the stream it consumes plus its own bookkeeping (deliveries, inflight, op_latest, approval_sent). The agents’ %W~*:outbox grant deliberately does not match the :* bookkeeping keys, so an agent cannot forge delivery state.
• %W~<agent>:<name>:inbox for each agent, and %W~<name>:inbox as the un-routed fallback — write-only, so a bridge cannot read back another agent’s conversation through its own inbox stream.
• %W~<notify>:outbox if it requires approval — just enough to send the notification.
• ~approval:<name>_map, its own external-id → approval-id map.
• ~approval:* only if it creates approvals (approval.required) or resolves them (it is somebody’s approval.notify target). Approval records are uuid-keyed and a uuid is not a pattern, so this is the tightest expressible grant — and consult, holding neither role, gets none of it.

Commands are +@all -@admin -@dangerous +info: every ordinary data command, so a missing verb can’t surface as an unbounded restart loop, minus the server-control surface (ACL, CONFIG, DEBUG, FLUSHALL, KEYS, SHUTDOWN, MIGRATE, REPLICAOF, MONITOR, …) that would let a bridge edit the boundary from inside. security-bridge-acl fails the build if any of this regresses.

This isolation is at the message layer only: agents share one network namespace and one secrets proxy, so provider credentials are common to all agents on a host. That is by design — all agents on a host are the operator’s own, one provider-credential trust domain — not a gap; per-agent provider-key isolation is deliberately not planned (see Per-Agent Isolation).

The journal is a different case, and this page previously got it wrong. Agent users are no longer placed in the systemd-journal group by default. The journal is the union of everything every unit on the host has ever logged — other agents’ sessions, the proxy’s request lines, and whatever a service leaked into its own output — which is a far wider read capability than anything else in an agent’s allow-list, and it is not covered by the “agents are mutually semi-trusted” argument, because it also crosses out of the agent trust domain entirely. Set services.cowboy.agents.<name>.allowJournal = true to grant it deliberately; that also adds Read(/var/log/**) to the agent’s sheepdog policy so the two layers agree. An agent without it loses nothing operational: build and activation output still reaches it through the rebuild bridge, scrubbed.

Credentials must not reach shared logs in the first place. The proxy strips query strings (which carry signed URLs, tokens, and OAuth codes) from its log lines via safe_path (proxy/addon.py); only host and path without the query are logged. Note the scope of that control: it covers lines the proxy writes. It does not cover what other units log, which is why the sudo path was changed to stop carrying secrets across the privilege boundary at all rather than relying on scrubbing.

Sheepdog policy

The policy is rooted on the OCI Runtime Spec v1.2 shape. Syscalls fall into three tiers:

• Blocked — SCMP_ACT_ERRNO(ENOSYS) at the BPF level (the configurable block-list, e.g. bpf, ptrace, mount).
• Mediated — SECCOMP_RET_USER_NOTIF; the gofer inspects arguments and decides. This set is fixed in seccomp.rs.
• Allowed — everything else, unmediated.

Path-, argv-, and address-aware rules (Read/Edit/Create/Delete/ Bash/Connect) are carried in the document’s rs.cowboy.policy.{deny,allow} annotations, because OCI seccomp can only match scalar syscall arguments, not string paths. Decision precedence is lazy_allow > deny > allow > default-deny.

The OCI-shaped document is currently enforced by sheepdog’s own notify supervisor; making it directly enforceable by a stock runtime (runc/crun) is a deferred upgrade path, not current behavior. See crates/sheepdog/POLICY.md for the full authority matrix and accepted limitations.

Example configuration

services.cowboy.agents.agent = {
  enable = true;
  homeDirectory = "/home/agent";

  # Per-agent policy (sheepdog / seccomp). allow/deny use Claude Code rule
  # syntax; blockedSyscalls names seccomp-blocked syscalls. See
  # modules/options/user.nix for the full schema and defaults.
  sheepdog.deny = [ "Read(/home/agent/.ssh/**)" ];
  sheepdog.allow = [ "Edit(/home/agent/workspace/**)" ];
  sheepdog.blockedSyscalls = [ "ptrace" "bpf" ];

  # Bind extra host paths into the agent (emitted into the OCI config mounts).
  mounts = [ { source = "/srv/data"; destination = "/srv/data"; readOnly = true; } ];
};

services.cowboy.bridges.discord = {
  pkg = pkgs.discord-service;
  env = "/run/agenix/discord-env";
  approval.required = true;
};

File reference

System Patterns

title: System Patterns tags: [patterns, architecture, design]

System Patterns

Recurring design patterns in the cowboy infrastructure. These are conventions the codebase follows, not separate subsystems.

Credentials live outside the agent

The agent never holds an API key. Instead of giving the agent a secret and trusting it not to leak it, secrets are held on the host side of a boundary the agent’s traffic must cross, and injected there.

Cowboy realizes this with the credential-injecting proxy (proxy/addon.py, services.cowboy.secretsProxy): the agent makes an unauthenticated request, and the proxy reads the secret from a file and writes the credential header on the way out. The agent process can read neither the secret file nor the proxy’s configuration. See Security.

The general rule: if a capability needs a secret, put the secret behind a service the agent talks to, not in the agent’s environment.

Declarative over imperative

If state can be declared in Nix, declare it — don’t expose it as an agent tool. Prefer a reconciliation service that converges declared state over a tool that lets the agent drive an API directly.

1. Declare desired state as structured Nix options.
2. Generate a config artifact at build time.
3. A systemd service reconciles declared state with the target.

Tools are reserved for inherently imperative operations: sending a message, running code, interactive queries. This is why bridges, filters, and the proxy are Nix modules that emit config and units rather than agent-callable tools.

Permission-separated, additive config

Split configuration by authority level so the agent can manage some of its own state without escalating privileges.

• A system layer is owned by the administrator and applied through the NixOS configuration.
• An agent layer is owned by the agent and applied through its home-manager configuration.

A reconciliation step merges the two additively, with the system layer winning on conflict: the agent can add, never subtract or shadow. A path unit watching the agent’s config file can trigger the system reconciliation service, bridging the unprivileged agent switch to a privileged service without granting the agent elevated access.

This is the same split that lets a plugin register skills/tools into an agent’s home while infrastructure stays under system control — see Plugin Architecture.

Capability through home, not core

A new capability is added by registering into the agent’s configuration — skills, tools, packages on the agent’s PATH, pub/sub bridges — rather than by modifying the harness. The harness reads per-agent config files (~/.config/cowboy/{skills,tools}.json, the skills directory) at startup, so most extensions are pure Nix with no Rust changes. The plugin contract is the worked-out form of this pattern.

Plugin Architecture

title: Plugin Architecture tags: [architecture, plugins, extensibility] created: 2026-04-24 updated: 2026-06-12

Plugin Architecture

Cowboy’s NixOS module system is the extension surface. An external module (a “plugin”) adds capability to agents by registering into cowboy’s option tree rather than forking the core. The canonical example is mun.nix — a Kerbal Space Program

• Twitch streaming stack that teaches an agent to fly rockets — and the in-tree Home Assistant skill.

Everything below is grounded in the actual implementation. Where the code and the ideal diverge, that’s called out and tracked in Plugin Improvements.

The three layers

A plugin separates into layers so that only the top one couples to cowboy:

In mun.nix this maps to: services.nix + ckan.nix (infrastructure), krpc.nix / marimo-pair.nix via callPackage (packages), and the config block in module.nix guarded by hasCowboy (integration). Disable cowboy and you still have a working KSP streaming rig.

The plugin contract

1. Own your namespace

Declare options under your own top-level namespace, never inside services.cowboy.*:

# Good
options.services.mun = { enable = ...; gameDir = ...; };

# Bad — destabilizes cowboy's option tree and breaks standalone use
options.services.cowboy.ksp = { ... };

2. Read cowboy state via cowboyLib

Cowboy injects a cowboyLib module argument (modules/lib/default.nix). This is the stable plugin ABI — everything under services.cowboy.* that isn’t surfaced here is internal and may change. Members:

cowboyLib.enabledAgents          # attrset of enabled agents (name -> agentConfig)
cowboyLib.agentNames             # [ "agent" ... ] — for building `agents = [ ... ]` selectors
cowboyLib.forAgents (acfg: {…})  # map an HM config fn over every enabled agent, keyed by user
cowboyLib.forAgentsWhere pred f  # …only agents matching `pred name acfg` (generic haAgents)
cowboyLib.singleAgent "context"  # the one enabled agent, or a lazy throw under multi
cowboyLib.broker                 # human operator user (may be null)
cowboyLib.userFor / homeFor name # agent user / home dir by name
cowboyLib.inNamespace            # is agent traffic proxied? (was: read secretsProxy.enable)

Accept it with a fallback so the plugin still evaluates when cowboy is absent:

{ config, lib, pkgs,
  cowboyLib ? { enabledAgents = {}; forAgents = _: {}; },
  ... }:

let hasCowboy = cowboyLib.enabledAgents != {}; in

Per-agent targeting: forAgents and the skills/tools registries fan out to all enabled agents by default. To scope a skill/tool to specific agents, set its agents = [ "name" … ] (empty = all). For per-agent content (prompts that differ by agent), use forAgentsWhere directly — see the Home Assistant pattern.

3. Guard harness integration

Keep infrastructure unconditional; gate cowboy registration on hasCowboy:

config = lib.mkIf cfg.enable {
  # Infrastructure — always
  systemd.services."mun-setup" = { ... };

  # Integration — only with cowboy
  services.cowboy.skills = lib.mkIf hasCowboy { ... };

  home-manager.users = lib.mkIf hasCowboy (
    cowboyLib.forAgents (acfg: { home.packages = [ krpcPython ]; })
  );
};

4. Don’t block cowboy.target

Plugin services join the agent lifecycle with a soft pull-in:

• Use wantedBy = [ "cowboy.target" ]. Do not use partOf against the target — a failing plugin service would tear the whole target down.
• On any service with Restart = "on-failure", set StartLimitBurst and StartLimitIntervalSec. A restart loop without these blocks switch-to-configuration during a rebuild.
• Use requires/after/partOf among the plugin’s own services for internal ordering.

5. Keep packages self-contained

callPackage your own dependencies. Don’t assume cowboy provides any particular package on PATH (the base set is just bash coreutils ripgrep fd jq gh nix git curl, plus camoufox’s browser — see modules/tools.nix).

Extension points

Skills — knowledge + prompt + packages

A skill is a markdown prompt the agent can load, optionally with packages and extra tools. Registered into the global services.cowboy.skills attrset (modules/skills/default.nix):

services.cowboy.skills.ksp-pilot = {
  description = "Control KSP vessels via kRPC";
  prompt = ./skills/ksp-pilot.md;     # or promptText = "...inline...";
  requires = [ krpcPython ];           # see the gotcha below
  additionalTools = [ "bash" "read" "write" ];
  tags = [ "ksp" "control" "krpc" ];
  autoLoad = false;                    # true = loaded at agent startup
  agents = [ ];                        # [] = all enabled agents; or [ "pilot" ]
};

For each agent the skill targets, the module writes ~/.config/cowboy/skills/<name>.md (the harness discovers skills by reading this directory) plus a per-agent skills.json, and installs the skill’s requires packages onto that agent’s PATH — regardless of autoLoad, so an on-demand skill’s binaries are present when the agent loads it. The agents selector filters which agents receive the skill.

Tools — schema’d executable capabilities

A tool is a command template the harness can call with structured args (modules/tools.nix). The default set (bash, web-search, spawn_subagent) merges with anything a plugin adds:

services.cowboy.tools.krpc-eval = {
  package = krpcPython;
  command = "python3 {{script}}";   # placeholders are {{double-brace}}, NOT {single}
  description = "Run a kRPC control script";
  sandbox = "standard";              # "none" | "standard" | "strict"
  timeout = 60;
  agents = [ ];                      # [] = all enabled agents (defaults always reach every agent)
  schema = {
    type = "object";
    properties.script = { type = "string"; description = "Path to script"; };
    required = [ "script" ];
  };
};

Tools are baked per-agent into the harness WASM and written to a per-agent ~/.config/cowboy/tools.json; the agents selector scopes a tool to specific agents (the default tools — bash, web-search, spawn_subagent — leave it empty, so they reach everyone). No first-party plugin currently registers a custom tool — mun exposes KSP control through a skill (prompt + krpcPython on PATH) instead, so the tool path is the less-trodden one.

Plugin registry — be discoverable

Register the plugin in the informational registry so the agent (via its system prompt) and the operator (cowboy plugins / /etc/cowboy/plugins.json) can enumerate what’s installed:

services.cowboy.plugins.mun = {
  description = "KSP control + Twitch streaming";
  version = "1";
  extensionPoints = [ "skills" "units" ];
  units = [ "mun-setup.service" "mun-mods.service" ];  # the real units it owns
  agents = [ ];                                        # [] = surfaced to all agents
};

This is purely descriptive — it wires up no capability, it just makes the plugin enumerable. The per-agent tools.json system_prompt_suffix carries the list to the agent with no harness changes.

Bridges — bidirectional message channels

A bridge connects an external platform to the agent’s pubsub (modules/bridges.nix, options in modules/options/bridges.nix). Each declaration auto-generates a pubsub source plus three systemd units (cowboy-<name>-{ping,ingest,outbox}) under cowboy-bridges.target:

services.cowboy.bridges.discord = {
  pkg = myBridgePkg;                # provides bin/discord-{ping,ingest,outbox}
  env = "/run/agenix/discord-env";  # EnvironmentFile
  user = config.services.cowboy.broker;  # defaults to secretsProxy.broker
  icon = "💬";
  approval = {
    required = true;
    notify = "discord";             # which outbox sends the approval prompt
    notify_channel = "1489...";     # channel id within that outbox
    timeout = 3600;                 # auto-reject after N seconds
  };
};

Bridges run as the broker (a human-operated trust domain), not as the agent — gating is via approval + per-agent tool allowlists, not per-agent bridge config.

The systemd target

Any unit can join the agent’s start/stop lifecycle:

systemd.services.my-capability.wantedBy = [ "cowboy.target" ];

Per-agent skills: the Home Assistant pattern

The global skills registry distributes to all agents. When a capability should reach only some agents, the in-tree HA skill (modules/skills/home-assistant.nix) shows the escape hatch: it bypasses the registry and writes the prompt directly into the homes of agents that opted in via a per-agent option (agentOpts.skills.homeAssistant):

let haAgents = lib.filterAttrs (_: a: a.skills.homeAssistant.enable) enabledAgents; in
home-manager.users = lib.mapAttrs' (_: acfg:
  lib.nameValuePair acfg.user {
    home.file.".config/cowboy/skills/home-assistant.md".source = mkHaSkillPrompt acfg;
    home.packages = [ pkgs.curl pkgs.jq ];
  }
) haAgents;

The agents selector now handles per-agent selection directly in the registry, so a plain skill that only needs targeting no longer needs this bypass. HA still uses it because its prompt is per-agent content — the endpoint, token, and managed units differ per agent — which a selection-only list can’t express. Use cowboyLib.forAgentsWhere for that case; HA is the worked example.

Worked example: mun.nix end to end

mun.nix/
├── module.nix      # options + integration layer (skills, HM packages, OBS scene)
├── services.nix    # infrastructure: mun-setup bind-mount + mun-stream/mun-screenshot scripts
├── ckan.nix        # infrastructure: declarative mod install
├── krpc.nix        # package: kRPC python client (callPackage)
├── marimo-pair.nix # package: marimo helpers (callPackage)
└── skills/*.md     # skill prompts

Consumer wiring (machines/lambda.nix):

imports = [
  inputs.cowboy.nixosModules.default
  ../modules/mun.nix/module.nix
];

services.mun = {
  enable = true;
  gameDir = "/home/dylan/.local/share/Steam/steamapps/common/Kerbal Space Program";
  user = "dylan";
  stream.enable = true;
  marimo.enable = true;
};

What lights up because cowboy is present:

1. Four skills (ksp-pilot, mission-plan, ksp-brief, ksp-marimo) written into the agent’s ~/.config/cowboy/skills/.
2. krpcPython, xdotool, uv (+ marimo helpers) on the agent’s PATH.
3. The agent’s polkit-managed unit list (managedServices.units) so it can cycle the stack itself.

Launch note: the running stack is started by the mun-stream script (a writeShellScriptBin in services.nix), which spawns Xvfb, KSP, the harness, marimo and OBS together — not by per-unit systemd services. The mun-ksp.service / mun-stream.service names that appear in lambda.nix’s managedServices.units do not correspond to real units on the host. See Improvements §3.

Checklist for a new plugin

• Options under your own namespace, with an enable flag.
• cowboyLib accepted with a { enabledAgents = {}; forAgents = _: {}; } fallback.
• All cowboy registration guarded by hasCowboy.
• Packages via callPackage; not assumed present.
• Plugin services use wantedBy = [ "cowboy.target" ] + StartLimit*, never partOf the target.
• A skill’s binaries go in its requires (installed for every targeted agent, on-demand or not).
• Scope to specific agents with agents = [ … ]; for per-agent content, use forAgentsWhere.
• Register in services.cowboy.plugins.<name> so the plugin is discoverable.
• Every unit in managedServices.units / plugins.<name>.units actually exists (the validator warns otherwise).

Plugin Improvements

title: Plugin System — Improvements tags: [architecture, plugins, roadmap] created: 2026-06-12

Plugin System — Improvements

The plugin architecture works and has a real consumer (mun.nix), but the contract had sharp edges. Each item below is a concrete gap observed in the code, with the evidence, the impact, and the fix.

Status (2026-06-12): all six landed. The contract changes are documented in plugins.md. One deliberate deviation from the original plan: #3’s check ships as a build-time warning, not a hard assertion — see that section for why. Implementation notes are inline below each item.

1. Per-agent targeting for plugin-supplied skills and tools

The gap. cowboyLib.forAgents, services.cowboy.skills, and services.cowboy.tools all fan out to every enabled agent. A plugin cannot say “this skill is for the research agent only.” The system already supports multiple agents (services.cowboy.agents.<name>, mode = "multi"), so this is a real limitation, not a hypothetical.

Evidence. modules/skills/home-assistant.nix exists solely to work around this. Its header comment says it all:

# Bypasses global cfg.skills registry to install only into HA-enabled agents

It reimplements the registry’s home-manager fan-out by hand, filtering on a per-agent option (agentOpts.skills.homeAssistant.enable). Every plugin that wants per-agent scoping must copy this boilerplate.

Impact. In single-agent setups (lambda today) it’s harmless. The moment a second agent appears, every globally-registered skill/tool leaks to it — including ones that carry capability (extra additionalTools, packages on PATH). This is the same single-agent assumption flagged throughout the module set.

Fix (landed). cowboyLib gained forAgentsWhere pred f and agentNames (modules/lib/default.nix), and both skills and tools gained an agents = [ … ] selector (empty = all enabled agents, back-compat). The consumer fan-outs build a per-agent skill/tool set (skillsFor / toolsFor) and a per-agent skills.json / tools.json; the per-agent tools manifest also feeds the per-agent WASM override. All pure Nix — the harness already reads per-agent files.

Correction to the original claim: home-assistant.nix does not fully collapse into a registry entry. The agents selector handles per-agent selection, but HA’s prompt is per-agent content (endpoint / token / managed units differ per agent), which a selection-only list can’t express. HA stays as the worked example of forAgentsWhere for content-templated skills.

2. On-demand skill requires are never installed

The gap. modules/skills/default.nix installs a skill’s requires packages into home.packages only when autoLoad = true:

home.packages = lib.concatLists (
  lib.mapAttrsToList (name: skill:
    if skill.autoLoad then skill.requires else [ ]   # <-- on-demand skills get nothing
  ) cfg.skills
);

The plugins doc previously claimed “the requires packages are added to each agent’s home.packages” — true only for the auto-loaded subset.

Impact. An on-demand skill’s prompt tells the agent to run a tool that isn’t on PATH. The agent discovers the gap at runtime, mid-task. mun.nix sidesteps it by pushing krpcPython via forAgents regardless of skill state — meaning the requires field is effectively decorative for its on-demand skills.

Fix (landed). Option (a): modules/skills/default.nix now installs the requires of every skill targeting an agent, regardless of autoLoad (home.packages dedups by store path, so this is cheap). mun.nix no longer needs to push krpcPython by hand — though it still does so for tools outside the skill set (xdotool, uv), which is fine.

3. Phantom systemd units in managedServices

The gap. machines/lambda.nix grants the agent polkit rights over

managedServices.units = [
  ... "mun-ksp.service" "mun-xvfb.service" "mun-setup.service"
  "mun-stream.service" "mun-marimo.service"
];

but mun.nix/services.nix only ever defines one of these (mun-setup). KSP/stream/marimo are launched by the mun-stream shell script, not by units. The other four .service names refer to nothing.

Impact. Polkit rules are generated for units that don’t exist. The agent’s mental model (and the HA-style skill prompt that lists managed units) advertises controls that silently do nothing. It also means the doc’s own example — which shows mun-ksp/mun-stream as proper services with StartLimit* — never matched reality.

Proposal. Pick one model and make it true:

• Service model (recommended): convert the mun-stream script into real mun-ksp / mun-stream / mun-marimo units with the wantedBy = [ "cowboy.target" ] + StartLimit* discipline the contract already requires. Then managedServices.units is honest and the agent can systemctl restart mun-ksp as advertised.
• Script model: drop the phantom units from managedServices and give the agent a documented mun-stream invocation instead.

Either way, validate that every managedServices.units entry resolves to a declared unit.

Fix (landed) — prune + warn (not assert). The phantom mun-* units (plus always-phantom steam.service / obs.service) were pruned from machines/lambda.nix; mun.nix keeps only its real units (mun-setup / mun-mods) and the launch path (mun-stream script) is documented. modules/services.nix gained a suffix-aware unit-existence check across services/targets/timers/sockets/mounts/paths, gated by managedServices.validateUnits (default true).

Why a warning, not the planned hard assertion: lambda’s media units are declared conditionally — plex.nix is imported only under sensitive.lib.sellout, readarr further depends on a books flag, and transmission is currently disabled. A hard assertion would break the spoof/template build (where those units legitimately don’t exist) and would force every consumer to mirror that conditional logic into its unit list. A warning surfaces the bug without that fragility. (The validator promptly earned its keep: it flags transmission.service on lambda, which is listed in managedServices but not currently a declared unit — left in place pending a decision to enable transmission or drop the entry.)

4. cowboyLib is too thin to discourage reaching into internals

The gap. cowboyLib exposes only enabledAgents and forAgents. Anything else a plugin needs — the broker user, the netns/proxy state, whether ACLs are on, per-agent home dirs by name — it reads straight out of config.services.cowboy.*. That recouples plugins to cowboy’s internal layout, the exact thing the namespace rule (contract §1) tries to prevent.

Impact. Plugins break when cowboy refactors internals. The [managedServices/daemon/sheepdog scoping refactor already on the books] would move several options; every plugin reading them directly is collateral.

Proposal. Treat cowboyLib as the stable plugin ABI and widen it deliberately:

cowboyLib = {
  inherit enabledAgents forAgents;
  broker        = cfg.broker;
  homeFor       = name: enabledAgents.${name}.homeDirectory;
  userFor       = name: enabledAgents.${name}.user;
  inNamespace   = cfg.secretsProxy.enable;   # is traffic proxied?
  # ... a curated, documented surface
};

Document it as the contract; everything else under services.cowboy.* is internal and may change without notice.

Fix (landed). modules/lib/default.nix now exposes forAgentsWhere, agentNames, broker, userFor/homeFor, and inNamespace alongside the existing enabledAgents/forAgents/singleAgent, documented in plugins.md as the stable ABI.

5. No plugin-side validation or discovery

The gap. Plugins are wired purely by manual imports in the machine config. There’s no assertion surface that a plugin can hook to validate its assumptions (e.g. “stream.enable requires an NVIDIA GPU and a twitch key path that exists”), and no registry of “installed plugins” the agent or operator can introspect.

Impact. Misconfiguration fails late (at activation or, worse, at runtime in the mun-stream script’s preflight command -v checks rather than at eval time).

Proposal. Lightweight, additive:

• Encourage assertions in plugin modules (none of the current option files assert their cross-field invariants).
• Optionally, a services.cowboy.plugins.<name> informational registry a plugin sets ({ description, version, extensionPoints }) so the agent’s context and a cowboy plugins CLI subcommand can enumerate what’s installed. Cheap, and turns “what can this agent do here” into a query.

Fix (landed). services.cowboy.plugins.<name> (modules/options/plugins.nix) is the registry; modules/plugins.nix emits /etc/cowboy/plugins.json; the per-agent tools.json system_prompt_suffix surfaces the list to the agent with zero Rust changes; and cowboy plugins (cowboy/cli.py) prints it for operators. mun.nix self-registers. The “encourage assertions” half is seeded with a representative warnings entry in mun.nix (streaming without marimo leaves a blank dashboard panel).

6. Duplicated package construction

The gap. mun.nix calls pkgs.callPackage ./krpc.nix {} and builds krpcPython in both module.nix and services.nix. Same closure, defined twice, kept in sync by hand.

Impact. Minor — Nix dedups the store path — but it’s a maintenance trap (edit one, forget the other) and a bad example for plugin authors who copy mun as the template.

Fix (landed). nix/modules/mun.nix/packages.nix defines krpcPython, krpcPkg, and marimoPairPkg once; module.nix and services.nix both inherit from import ./packages.nix { inherit pkgs; }. This also makes the “Packages layer” from the contract a real file.

Status

Follow-ups left open: decide transmission.service’s fate on lambda (enable or drop); consider promoting the unit validator to a hard assertion on hosts whose unit lists are fully static; and the lazy/ephemeral requires install (#2 option b) if PATH closure size ever matters.

Component ABI

The agent core (cowboy-core) is host-agnostic. Its entire outside world is a narrow seam: effects out (a Host trait) and inputs in — split into semantic Intents (the shared vocabulary) and raw HarnessEvents (the Zellij transport plus effect completions). This chapter is its ABI: the WIT world cowboy:agent@0.2.0 at specs/wit/cowboy-agent.wit — the single WIT source every bindgen consumer reads — which lets a runtime embed core without linking Rust in-process.

The world is small enough to state in a sentence. It exports one interface, agent, with five functions — initialize, handle-command, handle-host-event, snapshot, describe — and imports one, host, for the effects a component requests: exec, http, set-timer, close-self, spawn-peer, and own-agent-id.

Two host paths

Core has two adapters, and only one goes through WIT:

The Zellij adapter deliberately bypasses the component ABI. It needs keystroke-level fidelity to drive the terminal UI (prompt-line editing, navigation, expand/collapse), and it can link Rust, so paying the component-boundary tax buys it nothing. The WIT world exists for hosts that cannot link Rust — and those hosts don’t remote a TUI, they render the JSON display model and build their own input affordances.

The TUI lives in the adapter. cowboy-core is a headless agent model: it owns the DisplayItem data and AgentHarness::frame_json, while the renderers, syntax highlighting (syntect), modal input, and view state (ViewState) live in crates/harness/src/tui — wrapped around the core as Tui { agent, view }, which derefs to AgentHarness. The generic ANSI atoms (colors, symbols, text, spinner, key types) live in cowboy-ui. Core depends on neither cowboy-ui nor syntect, so the WASI component sheds both. Every view↔core coupling crosses the Host seam as a defaulted hook (below), never a field poke.

The named-intent seam

Raw keystrokes never cross the ABI. Core’s input surface splits in two:

• Navigation / view — scroll, expand/collapse, search, prompt-line editing. Client-local: the Zellij adapter owns it (it holds the TUI projection); a JSON client navigates its own copy of the frame. Never crosses WIT.
• Semantic intents — the handful of actions that change agent/session state: submit / interrupt / approval / debug / quit. The type is Intent, owned by cowboy-contracts and re-exported by cowboy_core::intent so the native adapters and the component adapter cannot drift.

Both adapters converge on one vocabulary via AgentHarness::handle_intent — no bifurcation. The Zellij adapter’s semantic key arms (in crates/harness/src/tui/input: prompt.rs Enter → Submit, navigate.rs Esc → Interrupt and d → Debug, mod.rs sentry y/o/A/n → Approval) call the same method the WIT component maps command’s cases onto. Intent derives serde, so it is also the montana driver’s inbound wire shape ({"submit":"hi"}, "interrupt", {"approval":"once"}, "debug", "quit", specified by specs/control.cddl) — one vocabulary end to end, not a second protocol.

Commands vs. host events

The world’s inbound surface is two types, delivered through two exports, because they have different producers and different trust semantics — a transport command must never be confusable with an effect completion:

• command — semantic operations an operator or protocol adapter applies to the instance: submit(string), interrupt, approval(approval), shutdown. Delivered by handle-command.
• host-event — completions and notifications the host produces: timer, command-result, web-result, permission-result. Delivered by handle-host-event.

Both return bool: true when observable state changed and a fresh snapshot is worth pulling.

debug is deliberately not a WIT command — it opens a local log viewer, so it is a host diagnostic rather than an agent operation, and it stays in the native Intent vocabulary only. Intent::Quit is what command.shutdown maps onto guest-side.

Neither type carries key, mouse, or paste. Core keeps the full Key/KeyEvent/Pasted + HarnessEvent types for the Zellij transport. If a TUI-remoting host ever appears, a key interface plus an ANSI-render export can be added to the world — additive, non-breaking.

One component instance has one foreground turn. The world exposes no task handle, no detach/background operation, and no reattachment; submitting while work is active queues the input, and interrupt asks the guest to abandon its turn without promising cancellation of already-issued host effects. Peers spawned via host.spawn-peer are separate instances, not background tasks of the spawner.

Rust-only host verbs

Some seams stay in the Rust Host trait and never enter the world at all, because they are meaningless off a terminal.

The pane verb is one: pane visibility is client-local view state a windowed host acts on, so montana/remote never needs it. pane_op lives in the Rust trait (the native Zellij adapter implements it); the WASI component no-ops it.

Defaulted view hooks

Four more Rust-only Host methods carry the view couplings — all defaulted, so headless hosts inherit them for free (they carry no meaning off a terminal), and only ZellijHost overrides them:

• reveal_latest() — “a new item was appended; snap the view to it.” Core calls it wherever the view must follow appended output; ZellijHost records a bit its Tui drains after the event (honoring the user’s mode).
• open_debug(path) — ZellijHost opens the debug log in a floating pane; headless no-ops. Keeps the zellij action new-pane … less +F exec out of core.
• has_draft_input() -> bool — core’s idle judge/autoreprompt guard; the draft lives in the adapter’s view state, so the default is false.
• editor_active() -> bool — core’s heartbeat cadence; the $EDITOR round-trip is a Zellij-only affordance, so the default is false.

ZellijHost and its Tui share an Rc<HostSignals> cell block: core→view for reveal_latest, view→core for has_draft_input/editor_active. Like pane_op, none of these are in the WIT world — they are Rust-trait seams a windowed host implements and everyone else ignores.

The opaque-correlation rule

Every effect that expects a reply carries a correlation — an opaque list<tuple<string, string>> created by the guest (core holds it as Ctx, an ordered map, and converts at the boundary). The host echoes it back verbatim on the matching host-event and never reads or writes it. The request “kind” lives inside it as a string, meaningful only to core (see dispatch.rs).

A correlation identifies an effect callback, not a foreground or background agent task — it is not a handle a host can hold, poll, or reattach to.

The payoff: new command/web kinds are new correlation string values, not new ABI surface. The world never changes when core grows a new internal effect kind. A host that treats the correlation as bytes-in/bytes-out is forward-compatible by construction.

Effects are requests, not grants

An exec or http call across the host import is a request. Nothing about crossing the boundary authorizes it: the host evaluates it under the active backend and capability policy (sheepdog, the credential proxy, the OCI sandbox — see Security) and may refuse. The same is true of initialize’s config map: its entries are configuration, not capability grants. own-agent-id is the one purely informational import — the instance’s own id, which core’s heartbeat uses to address itself.

Additive timer semantics

host.set-timer(secs) is additive and one-shot: each call arms an independent timer, and there is no cancellation. This mirrors Zellij’s set_timeout, which has no cancel API. Core’s heartbeat (heartbeat.rs) compensates with an armed_count that culls the resulting fan-out down to a single live chain.

A host must not “improve” this to one cancelable timer. Core’s re-arm logic assumes additive delivery; a cancel-on-rearm host would stall or double-fire the heartbeat. A headless embedder implements this as a plain deadline min-heap: push on set-timer, pop-and-deliver when the earliest deadline passes.

The JSON frame contract

snapshot() returns one line of literal JSON:

{"status": "WaitingForInput", "items": [ /* DisplayItem… */ ]}

status is AgentStatus; items is the DisplayItem list core also renders to ANSI. ANSI highlight caches (highlighted_lines, highlighted_input) are #[serde(skip)]’d — they are render artifacts, not state.

Deliberately unstabilized. WIT specifies only the carrier — a string holding one JSON object. The shape inside tracks core’s display model and will change as that model settles; a typed, versioned presentation contract is deferred work, so no host may treat this serialization as a stable interface by accident. montana’s ndjson stream wraps these frames per agent as {"seq", "agent", "frame"} — that envelope is specified, by the control-frame rule in specs/control.cddl.

The describe contract

describe() returns the protocol-neutral agent descriptor, also as one JSON object in a string: name, description, version, provider, model, capabilities, and the skill list (one entry per tool, carrying its JSON Schema input schema verbatim). Unlike the snapshot, this record is specified — by specs/descriptor.cddl, with Descriptor in cowboy-contracts as the Rust shape.

It exists so a host can advertise the agent without knowing anything about Cowboy’s internals: crates/a2a maps this one export onto an A2A agent card and its skill list, and montana serves that mapping. Protocol adapters are bindings onto describe + handle-command, not alternative agent APIs.

/connect — a harness as a client of another harness

/connect <host:port>#<secret-path> (or /connect <montana-uds-path>) turns a Zellij harness into a thin client of a remote agent, built entirely on the two contracts above: inbound, montana’s ndjson envelope rehydrates into Vec<DisplayItem> (pinned by display_item_round_trips_through_json); outbound, every user action is one Intent wire line (pinned by intent_wire_shapes). The fragment names a bearer-token file — the token is the first line of the connection, exactly the cowboy-runtime __forward handshake; the token itself never enters plugin config or argv.

Mechanically it is drop-and-connect: the plugin writes a request JSON to connect_request_path and quits; the Python launcher’s relaunch loop (cli._run) starts a fresh Zellij whose plugin sees connect_target at load() and comes up in client mode (crates/core/src/client.rs) — no providers, no session, no API keys. Because a WASM plugin cannot hold a socket, a cowboy _connect bridge process owns the connection and streams each frame line in via zellij pipe --name connect-frame; sends go out as one-shot cowboy _connect --send connections. /disconnect is the same sentinel dance back to a local session. Standalone entry: cowboy connect TARGET#SECRET-PATH.

Caveats: in a full NixOS (netns) deployment, outbound TCP is DNAT’d to the secrets proxy — a /connect target port must be in secretsProxy.passthroughPorts (UDS targets are unaffected; sheepdog always allows AF_UNIX), and the in-session handoff loop exists only in lite launches (full mode: run cowboy connect from a host shell — a client session needs no agent config anyway). A read-only __forward --ro port works as an observer: frames flow, intents are dropped server-side.

Known Zellij leaks in core

A few Zellij-isms survive in cowboy-core. None break a headless host; they degrade gracefully and are tracked here for eventual promotion to proper host verbs. (The two obvious candidates are already out: the $EDITOR round-trip is Tui-local, and the debug pane is the open_debug host hook.)

• Subagent wasm path (spawn.rs) — the fallback peer path assumes a Zellij plugin layout (data_dir()/zellij/plugins/cowboy-harness.wasm). A headless host overrides it via config / AGENT_HARNESS_WASM.
• Model-facing prose (config/loaders.rs) — the spawn_subagent tool description says “in a separate Zellij pane”. Cosmetic; visible to the model, not load-bearing.

Filesystem: dissolution, not ABI

Roughly two-thirds of core’s command dispatches are file I/O emulated by shelling out (cat, sh -c) because a Zellij WASI plugin cannot do direct file I/O. Those are not ABI surface: under a wasi:filesystem-capable host they dissolve into ordinary in-core code against preopened directories. The three production sites that already use std::fs directly (context/plan.rs, context/tool_result_meta.rs, config/loaders.rs) work today under montana’s identity preopens (host path X preopened at guest path X, so exec and in-process fs agree). Converting the shelled-out I/O to direct fs is a future, capability-gated simplification — it shrinks core without touching this world.

Sub-Agents

Sub-agents are child agent instances spawned by the harness to run a scoped task with a restricted toolset. Each runs in its own Zellij pane and communicates with the parent through files on disk.

Source: crates/core/src/subagent.rs, crates/core/src/spawn.rs, crates/core/src/subagent_config.rs.

How spawning works

The parent calls the built-in spawn_subagent tool. The harness then:

1. Writes the task to prompt.md and the filtered tool manifest to tools.json in a new directory under <cowboy_dir>/subagents/.
2. Spawns a new plugin instance via Zellij’s load_new_plugin() (a visible pane, load_in_background: false, skip_plugin_cache: true), passing a JSON-encoded SubagentConfig under the subagent_config key.
3. The child, on its first heartbeat tick, writes a lock file and its pane_id, reads prompt.md + tools.json, and processes the prompt as a user message.
4. When the child finishes, it writes response.md and removes lock.
5. The parent detects completion on its heartbeat poll and injects the response back into its own conversation.

WASM cannot write files directly, so all disk I/O is funneled through shell commands the harness emits and Zellij executes.

Directory layout

Each sub-agent gets a directory named <type>-<short_uuid>:

<cowboy_dir>/subagents/<type>-<id>/
  prompt.md     # task + metadata, written by parent
  tools.json    # filtered tool manifest, written by parent
  lock          # present while running, written by child
  pane_id       # Zellij pane id, written by child
  response.md   # final output, written by child on completion

<cowboy_dir> is $HOME on a ranch install and $XDG_DATA_HOME/cowboy (default ~/.local/share/cowboy) in lite mode.

Sub-agent types

Three types are defined in the SubAgentType enum. Each restricts the tools the child may call:

Research and Review are read-only with respect to the filesystem; only Code gets write and the hashline edit tool.

Models

SubagentConfig::cheap_defaults() picks an inexpensive model so sub-agents are cheap to run. The default per provider:

The selection falls back to the main harness provider/model, then to the first available keyed provider.

Status

The parent tracks each child with the SubAgentStatus enum:

• Starting — directory created, pane spawning
• Running — lock present
• Completed — response.md present, lock removed
• Failed(String) — lock removed without a response.md

Polling happens on the heartbeat timer.

Memory System

Persistent, cross-session memory for the agent. Notes are stored as Markdown files and surfaced back into the model’s context via a search step before the agent acts.

Source: crates/core/src/memory.rs. Tools are registered in crates/core/src/tools/builtin.rs.

Backends

Memory is pluggable behind the MemoryBackend trait. Two backends exist:

• ZkBackend — zettelkasten built on the zk CLI. Keyword-based full-text search. Always available, including lite builds. This is the default for lite.
• QmdBackend — hybrid BM25 + vector search via the qmd CLI, using zk for note authoring. Available only in ranch (non-lite) builds, where it is the default (memory_backend = "qmd").

The backend is selected by the memory_backend config key ("qmd" or "zk"). qmd_min_score (default 0.4) sets the relevance cutoff for the qmd backend. If a lite build is asked for qmd, it falls back to zk.

Because the harness runs as a WASM plugin and cannot spawn processes directly, each backend method returns a shell command string. The harness executes it, receives the result, and parses stdout into notes.

Tools

The agent interacts with memory through two built-in tools:

• __MEMORY_SAVE__ — write a note.
• __MEMORY_SEARCH__ — search notes.

Notes

A parsed note (MemoryNote) has:

Directory layout

Notes live under <cowboy_dir>/memory/, a zettelkasten managed by zk:

<cowboy_dir>/memory/
  .zk/        # zk configuration and index
  daily/      # YYYY-MM-DD.md journals
  facts/      # atomic knowledge notes
  decisions/  # decision records
  templates/  # zk note templates

<cowboy_dir> is $HOME on a ranch install and $XDG_DATA_HOME/cowboy (default ~/.local/share/cowboy) in lite mode.

Pre-tool retrieval

Memory search is wired in as a retrieval step before the agent runs tools, gated by the pre_tool_retrieval config flag (default on). When enabled, the harness searches memory based on the conversation and injects matching notes into context so the model can use prior knowledge without an explicit search. Retrieval does not fire inside sub-agents.

The MemoryBackend trait also exposes init_commands(), recent(limit), search_deep(query), remember(title, content, template), journal(entry), and an optional search_skills_deep(query) for searching skill files alongside memory.

API Integration

The harness talks to LLM providers over Zellij’s web_request() API (the WebAccess permission). There is no synchronous HTTP call: the provider layer only formats requests and parses responses, and Zellij delivers the result asynchronously as an event.

Source: crates/core/src/provider/.

Providers

Five providers implement the LlmProvider trait, selected by ProviderType:

• ClaudeProvider (Anthropic Messages API)
• OpenAIProvider (Chat Completions / Responses API)
• CodexProvider (ChatGPT subscription Responses API with SSE)
• OpenRouterProvider
• OllamaProvider (local, keyless)

The LlmProvider trait

The trait is deliberately not async — Zellij handles async via events. It only serializes requests and deserializes responses:

#![allow(unused)]
fn main() {
pub trait LlmProvider {
    fn name(&self) -> &str;

    fn format_request(
        &self,
        messages: &[Message],
        tools: &[Tool],
        system: &str,
    ) -> (String, BTreeMap<String, String>, Vec<u8>);

    fn parse_response(&self, body: &[u8]) -> Result<LlmResponse, ProviderError>;
    fn parse_error(&self, status: u16, body: &[u8]) -> ProviderError;

    fn set_model(&mut self, model: &str);
    fn api_key(&self) -> &str;
}
}

The request flow is:

1. format_request() produces (url, headers, body).
2. The harness issues web_request() with that data.
3. Zellij delivers Event::WebRequestResult.
4. parse_response() (on HTTP 200) or parse_error() (otherwise) interprets it.

set_model() supports switching the model at runtime (the /model command).

Credentials

The agent does not hold real API keys. When the secrets proxy is enabled (services.cowboy.secretsProxy.enable), the harness sends placeholder keys and the proxy injects the real credentials on the wire based on per-domain mappings. Codex is the same pattern: its rotating ChatGPT subscription token is injected outside the agent. See Security Model. For keyless providers (Ollama) api_key() is empty.

Provider defaults

ClaudeProvider targets /v1/messages. Its defaults: model claude-sonnet-4-20250514, max_tokens 8192, extended thinking enabled with a token budget. OpenAIProvider defaults to gpt-4o and auto-selects the Responses API for reasoning models (o-series, gpt-5, codex) to capture reasoning summaries, which are mapped onto thinking content blocks.

Message model

Messages carry a string role and a vector of ContentBlocks, matching the Claude API’s native content-block model. A block’s block_type is one of text, tool_use, tool_result, or thinking.

Errors and retries

ProviderError includes ParseError, ApiError, NetworkError, RateLimited, Timeout, InvalidRequest, AuthenticationError, and Overloaded. is_retryable() marks rate limits (429), network errors, server errors (5xx), and timeouts as retryable; auth and other 4xx errors are not. Automatic retry is wired in: a RetryState with exponential backoff drives a bounded retry of failed LLM calls from the ranch handlers (crates/core/src/handlers.rs, schedule_llm_retry) — a retryable error schedules a backed-off retry until max_attempts is exhausted, after which the failure surfaces as an error message.

Bridges

Bridge services (Discord, email) do not use the provider layer. They reach the agent through the pub/sub message system: inbound messages are polled and processed as user input, and replies flow back out the same way. See Approvals & Outbox.

Approvals & Outbox

title: Outbox Approval Protocol tags: [pubsub, approval, discord, email]

Outbox Approval Protocol

Outbox services can hold an outbound message for human approval before sending it. The agent never sends directly; it writes to an outbox stream, and a broker service decides whether to forward the message, optionally gating it behind a human reaction.

Source: pkgs/bridge/base.py (OutboxService), pkgs/bridge/ingest.py (IngestService), with per-platform implementations in pkgs/discord/ and pkgs/email/.

See also Security Model.

Transport

Messages move over Redis Streams. Each bridge {name} has an inbox stream ({name}:inbox) and an outbox stream ({name}:outbox). OutboxService reads its outbox via a consumer group ({name}-outbox / worker {name}-worker) and acknowledges each entry after handling it. Approval state lives in plain Redis hashes so any service can poll it.

Flow

When approval_required is set, an outbound message is held and routed through a notification channel:

1. The agent writes a message to {source}:outbox.
2. OutboxService.process_one() sees approval_required and the message has no approval_id, so it calls request_approval():
   • creates approval:{uuid} with status=pending, source, channel_id, a content_preview (first 200 chars), and created_at
   • writes a notification to the configured notify outbox ({approval_notify}:outbox) carrying the approval_id
3. The notify service (e.g. Discord) sends the notification. Because the message carries an approval_id and the send returns an external_id, the base loop records the mapping in approval:{source}_map (external id → approval id).
4. A human reacts to the notification.
5. The notify service’s ingest handler calls OutboxService.resolve_approval(), which looks up the approval id from the map and sets status to approved or rejected plus the approver.
6. The original outbox service is polling in wait_for_approval(); it sees the resolved status and either sends the message or drops it, then acks.

State machine

approval:{uuid} = {
  status:          pending | approved | rejected | expired
  source:          email | discord | ...
  channel_id:      destination (email address, channel id, ...)
  content_preview: first 200 chars of the message body
  created_at:      unix timestamp
  approver:        who resolved it (set on resolution)
}

pending --+-- approve --> approved --> send
          +-- reject  --> rejected --> drop
          +-- timeout --> expired  --> drop

On timeout wait_for_approval() returns false and deletes the approval hash.

Redis keys

approval:{uuid}         # HASH — approval state
approval:{source}_map   # HASH — external message id -> approval id

The requesting service writes and polls approval:{uuid}. The notify service owns approval:{source}_map. Bridges share only the key format — Discord and email do not know about each other.

Implementation

request_approval(), wait_for_approval(), the static resolve_approval(), and the auto-tracking in process_one() all live in the shared OutboxService. A per-platform bridge only needs to:

1. Return SendResult(ok=True, external_id=...) from its send().
2. Call OutboxService.resolve_approval(pubsub, source, external_id, status, approver) from its ingest handler when a reaction or reply arrives (IngestService.try_resolve_approval() wraps this).

systemd services

Each bridge {name} is run as three systemd services: cowboy-{name}-ingest, cowboy-{name}-outbox, and cowboy-{name}-ping.

Configuration

Approval is configured per bridge under services.cowboy.bridges.<name>.approval (and equivalently on services.cowboy.pubsub.sources.<name>.approval):

services.cowboy.bridges.discord.approval = {
  required = false;       # hold outbound messages for manual approval
  notify = "discord";     # which outbox to send approval notifications to
  notify_channel = "";    # channel id within that outbox
  timeout = 3600;         # auto-reject after N seconds (0 = no timeout)
};

If required is true, notify_channel must be set. The Redis ACL enforcement option keeps the agent restricted to stream commands so it cannot touch the approval hashes directly.

Adding a new approval channel

To approve via something other than Discord:

1. Have the new service’s outbox return external_id from send().
2. Have its ingest call OutboxService.resolve_approval(...).
3. Set approval.notify = "<service>" on the bridge that needs approval.

No changes to OutboxService or the requesting service are required.

Hashline Edit Format

Hashline is the harness’s read/edit format. Each line is tagged with a short hash of its content, so an edit can be rejected when the file has changed since the agent last read it.

Source: crates/core/src/hashline.rs. Exposed as the built-in tools __HASHLINE_READ__ and __HASHLINE_EDIT__.

Format

Lines are formatted as {line_num}:{hash}|{content}:

1:a3|fn main() {
2:7f|    println!("hello");
3:b2|}

__HASHLINE_READ__ returns a file in this form. The agent then references lines by line:hash when editing.

Hash function

The hash is FNV-1a (offset basis 2166136261, prime 16777619) truncated to the low byte (hash & 0xff) and formatted as two lowercase hex characters. With 256 possible values it is meant for change detection, not collision resistance.

Edit operations

__HASHLINE_EDIT__ takes an array of operations. Each references a target line by line:hash:

• replace — replace lines from start to optional end (inclusive) with new content
• insert_before — insert content before the referenced line
• insert_after — insert content after the referenced line
• delete — delete lines from start to optional end (inclusive)

Every referenced line’s hash is validated before anything is applied. If any hash does not match the current file, the whole edit is rejected and the agent is told to re-read. Operations are applied in reverse line order (so earlier edits don’t shift later line numbers) and the file is written atomically via a temp file and rename. The parser accepts either operations or edits for the array key, and either content or new_text for the text field.

Browser Automation

The harness can drive a real browser through Camoufox, an anti-detection Firefox fork. Agents navigate pages, read accessibility snapshots, interact with elements, run JavaScript, and take screenshots.

Source: pkgs/camoufox/, modules/camoufox.nix, modules/options/camoufox.nix.

Components

• camoufox — the Camoufox browser binary (anti-detection Firefox, daijro/camoufox).
• camofox-server — a Node.js REST server wrapping Camoufox (jo-inc/camofox-browser).
• cowboy-browser — a shell CLI (pkgs/camoufox/cowboy-browser.sh) that the agent calls; it talks to the server’s REST API.

All three are exposed as flake packages: nix build .#camoufox, .#camofox-server, .#cowboy-browser.

Deployment

The server runs inside a shared NixOS container (cowboy-camoufox) with one camoufox-<user> systemd service per agent. Each agent’s server listens on basePort + (uid - 1338), where basePort defaults to 9377. Browsing happens under a headless Xvfb display.

Enable it with:

services.cowboy.camoufox.enable = true;

This requires services.cowboy.secretsProxy.enable = true — the camoufox container uses the proxy’s veth networking, and the module asserts this.

Options under services.cowboy.camoufox:

CLI

cowboy-browser keeps a session id in a file under $XDG_RUNTIME_DIR and targets the server at $CAMOFOX_URL (default http://10.200.0.1:9377, the veth host address — localhost does not reach the container). Subcommands:

cowboy-browser navigate <url>      Open URL (creates a session if needed), return a snapshot
cowboy-browser snapshot [--full]   Accessibility-tree snapshot of the current page
cowboy-browser click <ref>         Click an element by ref
cowboy-browser type <ref> <text>   Type text into an element
cowboy-browser scroll <direction>  Scroll (up/down/left/right)
cowboy-browser press <key>         Press a key (Enter, Escape, Tab, ...)
cowboy-browser back                Navigate back
cowboy-browser eval <js> | eval -  Run JS in the page ('-' reads from stdin)
cowboy-browser screenshot [file]   Save a screenshot (default /tmp/cowboy-browser-screenshot.png)
cowboy-browser close               Close the browser session

Elements are referenced as @e1, @e2, … from the snapshot output (the @ is optional).

Example

cowboy-browser navigate https://example.com
# snapshot lists interactive elements with @eN refs
cowboy-browser type @e3 "search query"
cowboy-browser click @e7
cowboy-browser screenshot ./result.png

The agent invokes these through its bash tool. cowboy-browser unsets the HTTP proxy variables for its own requests, since the camofox server is local to the veth and does not need credential injection.

Site Documentation

This site is an mdBook built from the docs/ directory and published at cowboy.rs.

Layout

docs/
  book.toml      # mdBook configuration
  src/           # Markdown sources (SUMMARY.md is the table of contents)
  theme/         # custom theme assets (cowboy-hat)
  book/          # build output — git-ignored, regenerated by mdbook

docs/book/ is generated and excluded from version control. Edit files in docs/src/ and add new pages to docs/src/SUMMARY.md.

book.toml

Key settings (docs/book.toml):

• src = "src", build output build-dir = "book"
• default-theme = "navy", preferred-dark-theme = "navy"
• site-url = "https://cowboy.rs/", cname = "cowboy.rs"
• git-repository-url = "https://github.com/dmadisetti/cowboy" and an edit-url-template pointing at docs/src/{path}
• search and section folding enabled
• a custom “cowboy-hat” mark styled via additional-css (theme/css/cowboy-hat-fixed.css)

Building

# Nix (preferred — pins mdbook)
nix build .#docs        # output in ./result, the rendered book

# Or directly with mdbook
cd docs
mdbook build            # writes docs/book/
mdbook serve            # live-reload preview on http://localhost:3000

The flake’s docs package runs mdbook build and installs the book/ output.

Adding a page

1. Create the Markdown file under docs/src/ in the appropriate section.
2. Add an entry to docs/src/SUMMARY.md (pages not listed there are orphans and will not appear in the navigation).
3. Run mdbook build (or mdbook serve) and check the page renders.

Verifying rendering with the browser

Because the harness can drive a browser (see Browser Automation), an agent can serve the built book and confirm pages render before publishing:

mdbook serve          # or serve docs/book/ with any static server
cowboy-browser navigate http://localhost:3000
cowboy-browser screenshot ./home.png

This is optional — mdbook build succeeding plus correct SUMMARY.md entries and resolving internal links is the baseline check.

AI Governance

Cowboy constrains what an agent can do through several independent mechanisms. They overlap deliberately: a command rejected by one layer is not relied upon to be caught by another.

This page summarizes how they are configured. There is no runtime rule engine or FilterAction-style API in the harness — governance is the sum of the mechanisms below. Command, path, and syscall enforcement is done by sheepdog at the kernel boundary (see below), not by a separate string-matching filter layer.

Approvals (human in the loop)

For outbound messages that should not be sent autonomously, the bridge approval protocol holds a message until a human reacts to a notification. Configure it per bridge:

services.cowboy.bridges.discord.approval = {
  required = true;
  notify = "discord";
  notify_channel = "<channel-id>";
  timeout = 3600;
};

Approval state is tracked in Redis hashes and resolved from human reactions. See Approvals & Outbox for the full protocol.

Egress allowlist (secrets proxy)

When the secrets proxy is enabled, agent HTTP traffic is forced through it. The proxy injects real API credentials (the agent only ever holds placeholders) and enforces an egress policy: read methods (GET, HEAD, OPTIONS, TRACE) are always allowed, but state-mutating methods (POST, PUT, PATCH, DELETE) are allowed only to domains on an allowlist. Everything else returns 403.

services.cowboy.secretsProxy = {
  enable = true;
  domainMappings = {
    "api.anthropic.com" = {
      secretPath = "/run/agenix/anthropic-key";
      headerName = "x-api-key";
    };
  };
  # Extra write-allowed domains. Domains in domainMappings are implicitly
  # write-allowed.
  allowedWriteDomains = [ "github.com" "api.github.com" "*.githubusercontent.com" ];
};

The allowlist enforcement lives in the mitmproxy addon (proxy/addon.py); write methods, the allowed-domain check, and wildcard matching are implemented there. See Security Model.

Sheepdog (seccomp sandbox)

Sheepdog enforces file, network, and exec rules at the syscall level rather than by string matching. Rules are verb-granular — Bash, Read, Edit, Create, Delete, and Connect — and resolve to allow or deny, with optional runtime-granted exceptions (lazy permissions) taking precedence over baked-in denies.

services.cowboy.sheepdog = {
  enable = true;
  lazyPerms = true;   # allow runtime permission grants
};

services.cowboy.agents.<name>.sheepdog = {
  deny  = [ "Connect(0.0.0.0/0)" ];
  allow = [ "Read(/home/*/workspace/**)" "Edit(/home/*/workspace/**)" ];
  blockedSyscalls = [ /* ... */ ];
  readonlyPaths = [ /* ... */ ];
  maskedPaths   = [ /* ... */ ];
};

Sheepdog is Linux-only. See crates/sheepdog/src/policy.rs and modules/options/sheepdog.nix.

See also

• Approvals & Outbox
• Security Model
• Sub-Agents

Concurrency Locking

The harness WASM plugin runs a single agent session as an event loop. To keep LLM requests, tool execution, and summarization from interleaving, the session holds a SessionLock (crates/core/src/lock.rs) that treats those three activities as mutually exclusive states and queues user input that arrives while any of them is in flight.

SessionLock

SessionLock tracks:

• llm_active: bool — an LLM request is in flight.
• summarizing: bool — a tool-output summarization call is in flight.
• active_tools: HashMap<String, ActiveToolState> — tool executions keyed by their unique call_id.
• input_queue: VecDeque<String> — user inputs received while the session was busy.

The session is locked when any of those is active:

#![allow(unused)]
fn main() {
pub fn is_locked(&self) -> bool {
    self.llm_active || self.summarizing || !self.active_tools.is_empty()
}
}

A tool boundary is the inverse — no LLM request, no summarization, and no active tools. Queued input is only replayed at a boundary.

LLM lock

try_acquire_llm() sets llm_active and returns false if a request is already in flight, so concurrent LLM requests cannot be dispatched. force_release_llm() clears it when the response is handled.

Tool tracking

register_tool(call_id, tool_name, timeout) records an ActiveToolState with a start time and timeout. complete_tool(call_id) removes it. Results are matched by the unique call_id, not by tool name, so multiple concurrent calls to the same tool are tracked independently.

check_timeouts() returns and removes any tool whose elapsed time exceeds its timeout; the harness calls this on heartbeat ticks to clean up stuck executions.

Summarization gating

set_summarizing(true) is set while a large tool output is being summarized by the cheap summary model, and cleared when the response arrives. While set, the session is locked and user input is queued, exactly like an active LLM request.

Input queueing and coalescing

When is_locked() is true, incoming user input is pushed onto input_queue rather than processed immediately. At the next tool boundary, boundary_reached() drains the queue via drain_ready_inputs():

• A single queued input is returned verbatim.
• Multiple inputs are coalesced into one message under a [Multiple inputs received while processing:] header, joined by ---.

boundary_reached() is called after a tool result is recorded and when a batch of pending tool calls drains to empty. It returns None unless the session is actually at a boundary, so input is never injected mid-turn.

Status

Implemented in crates/core/src/lock.rs with unit tests covering LLM mutual exclusion, boundary detection, input coalescing, timeout removal, and the combined locked-state predicate.

Result Summarization

Large tool outputs are stored on disk as recallable artifacts and replaced in the context window with a short stub. When summarization is enabled, the stub’s body is produced by a cheap “summary” model instead of blunt truncation.

Implementation: crates/core/src/tools/summarize.rs (classification, prompts, request formatting, stubs, fallback truncation) integrated through harness.rs (handle_command_result) and handlers.rs (handle_summarization_response).

Threshold

A tool output is summarized when its length exceeds summarize_threshold, configurable via the summarize_threshold KDL key and defaulting to DEFAULT_SUMMARIZE_THRESHOLD = 12000 characters. Input sent to the summary model is capped at MAX_SUMMARIZABLE_LENGTH = 50000 characters. Summarization can be disabled with summarize_enabled = false.

Summary models

The summary model’s provider is taken from the summary_model config (or a summary_provider override) and resolved through the shared key chain. Default model per provider:

Requests use temperature: 0.3 and max_tokens: 2048 (or the provider-appropriate completion-tokens parameter for OpenAI-compatible APIs).

Output classification

The tool name determines an output type, which selects a type-specific prompt and fallback head/tail ratio (ToolOutputType::from_tool_name):

Each type has its own prompt: command output preserves errors and exit codes verbatim, search results are grouped by file with line numbers, directory listings are grouped by kind, structured data extracts names and versions, and web content extracts main facts and quotes.

Code-file handling

For FileContent, SummarizationRequest::code_file splits the file into three parts: a before-section and after-section (summarized) around a relevant range that is reproduced exactly with hashlines. When no range is supplied the relevant range defaults to the middle third of the file. The prompt instructs the model to keep the surrounding summaries to one or two sentences and preserve the hashline section verbatim.

Pipeline

1. A tool result arrives in handle_command_result. The full output is logged to the session’s messages.jsonl exactly once, regardless of what enters context.
2. If the output exceeds the threshold and the session is ready, it is stored as a disk artifact and kept in memory for UI expansion.
3. If summarization is enabled, a PendingSummary is queued, the request is dispatched to the summary model, and set_summarizing(true) gates user input until the response returns.
4. In handle_summarization_response, the lock is cleared, the provider-specific response is parsed, and the summary is wrapped in an artifact stub. On a parse failure or non-200 status, fallback_truncate is used instead.

If summarization is disabled, step 3 is skipped and the artifact stub is built directly from fallback_truncate.

Artifact stubs and recall

format_artifact_stub wraps the summary with a header and a footer noting that the full output is stored:

[Artifact <call_id> | <tool> | <bytes> bytes, <lines> lines]
<summary>
[Full output stored — use recall_artifact("<call_id>") to search or read more]

The agent retrieves the full output on demand through the recall_artifact builtin tool (wire name __RECALL_ARTIFACT__), which reads or searches the stored artifact by call_id.

Fallback truncation

fallback_truncate splits at line boundaries using the type’s head/tail ratio, operates on character counts to stay UTF-8 safe, and inserts a [...N chars omitted...] marker between the kept head and tail. It is used whenever the summary model is disabled, unreachable, or returns an unparseable response.

Message Delivery Reliability Fixes

Problem

Discord messages are dropped — both messages that arrive during harness restarts AND new messages that arrive after the harness is back online.

Root Causes

The harness polls Redis Streams via XREADGROUP using > (new-only) with a random consumer name per restart (harness-{uuid}, see pubsub/redis.rs:56). Four compounding issues cause message loss:

1. No PEL recovery (unacked messages stuck forever)

When the harness reads a message via >, it is assigned to consumer harness-{uuid1}. If the harness restarts before XACK, the message remains in the Pending Entries List (PEL) under the old consumer name. On restart, a new consumer harness-{uuid2} polls with > — which only returns messages never delivered to any consumer in the group. PEL messages are skipped. There is no XAUTOCLAIM in the Rust harness to reclaim them.

The Python outbox services already implement XAUTOCLAIM (see pkgs/bridge/redis_provider.py:127, DEFAULT_RECOVER_IDLE_MS = 30_000), but the Rust harness does not.

2. No startup backlog drain

XGROUP CREATE uses id 0-0, meaning the group’s last-delivered-id starts at the beginning of the stream. But once any consumer in the group has read a message (advancing the last-delivered-id), messages that arrived between the last successful poll and the restart may have already been “delivered” to the old consumer — they’re in the PEL, not “new”. The harness never reads with 0 (which would return PEL messages for the current consumer), so it never drains the backlog on startup.

3. Fire-and-forget XACK

handlers.rs:25:

#![allow(unused)]
fn main() {
"pubsub_ack" | "pubsub_reply" | "pubsub_reaction" => Some(true),
}

XACK command results are silently consumed. If XACK fails (transient Redis error, connection blip), the message stays in the PEL with no retry, no alert, no logging. The next harness restart won’t reclaim it (see issue 1).

4. Only one message processed per idle tick

process_pubsub_messages() (handlers.rs:143) pops one message from the inbox via next_message(), processes it, then returns. If a poll returned 10 messages, 9 remain queued until subsequent idle ticks. With a 5s poll interval and a 30s heartbeat, processing 10 queued messages takes up to 5 minutes — during which the user sees no response and assumes messages were dropped.

Proposed Fixes

Fix 1: Add XAUTOCLAIM to reclaim PEL messages

Files: pubsub/redis.rs, pubsub/types.rs, pubsub/manager.rs

Add a claim_command() method to the MessageSource trait and RedisStreamsSource. This generates an XAUTOCLAIM command:

redis-cli XAUTOCLAIM {stream} {group} {consumer} {min_idle_ms} 0-0 COUNT 10

XAUTOCLAIM claims messages from the PEL that have been idle for at least min_idle_ms, transferring ownership to the current consumer. This recovers messages stranded by crashed/old consumers.

The SourceManager should issue claim_command() periodically (every 60s, matching the Python outbox’s DEFAULT_RECOVER_INTERVAL_S) alongside poll_command().

Changes:

• pubsub/types.rs: Add claim_command() to MessageSource trait with a default impl returning None.
• pubsub/redis.rs: Implement claim_command() on RedisStreamsSource — returns the XAUTOCLAIM shell command. Output is parsed by the same parse_poll_result() since the format is similar (XAUTOCLAIM returns [cursor, [(id, {fields})...], [deleted_ids]]).
• pubsub/manager.rs: Add a claim_commands() method that generates claim commands for all enabled sources. Track last_claim timestamps per source. Add a claim_interval field (default 60s).
• ranch/handlers.rs: Add claim_message_sources() method. Call it from on_idle_tick() alongside poll_message_sources(). Add "pubsub_claim" to handle_platform_command() — reuse the same handler as pubsub_poll since the output format is compatible.

New constant:

#![allow(unused)]
fn main() {
/// Minimum idle time (ms) before a PEL message can be claimed back.
const RECOVER_IDLE_MS: u64 = 30_000;
/// How often to run PEL recovery sweeps (seconds).
const RECOVER_INTERVAL_S: f64 = 60.0;
}

Fix 2: Drain PEL on startup with 0

Files: pubsub/redis.rs, pubsub/manager.rs

Add a first_poll boolean to RedisStreamsSource (or to SourceManager per-source). On the very first poll after source registration, use 0 instead of > as the read ID:

XREADGROUP GROUP {group} {consumer} COUNT 10 STREAMS {stream} '0'

This reads messages in the PEL for this consumer. Since the consumer name is new (random UUID), the PEL for this consumer will be empty on the very first poll — but combined with Fix 1 (XAUTOCLAIM), the first claim sweep will pick up orphaned PEL messages from dead consumers and assign them to the new consumer, and subsequent 0 polls will read them.

Alternative approach: Instead of tracking first_poll, just always issue both a 0 poll (for PEL) and an > poll (for new) on each tick. The 0 poll is cheap — it returns immediately if the PEL is empty. This is simpler and more robust than tracking first-poll state.

Changes:

• pubsub/redis.rs: Add a pel_poll_command() method that uses 0 as the stream ID. The main poll_command() continues using >.
• pubsub/manager.rs: poll_commands() issues both poll_command() and pel_poll_command() per source. Use distinct context types ("pubsub_poll" vs "pubsub_pel_poll") so results are handled the same way.
• Track inflight state separately for PEL and new polls to avoid double-issuing.

Fix 3: Verify XACK succeeded

Files: ranch/handlers.rs, pubsub/manager.rs

Change handle_platform_command to actually check XACK results instead of silently returning true:

#![allow(unused)]
fn main() {
"pubsub_ack" => {
    let source = context.get("source").cloned().unwrap_or_default();
    let message_id = context.get("message_id").cloned().unwrap_or_default();
    if exit_code == Some(0) {
        // XACK returns the number of acked entries (0 = nothing acked)
        let count = String::from_utf8_lossy(stdout).trim();
        if count == "0" || count.is_empty() {
            log_warn!(
                self.logger,
                "pubsub: XACK returned 0 for message {} on source '{}' \
                 (already acked or wrong group?)",
                message_id, source
            );
        } else {
            log_debug!(
                self.logger,
                "pubsub: XACK confirmed {} entry/entries on source '{}'",
                count, source
            );
        }
    } else {
        log_warn!(
            self.logger,
            "pubsub: XACK FAILED for message {} on source '{}': exit={:?} stderr={}",
            message_id, source, exit_code,
            String::from_utf8_lossy(stderr).trim()
        );
        // Re-queue the ack for the next tick
        self.source_manager.requeue_ack(&source, &message_id);
    }
    Some(true)
}
}

Changes to manager.rs:

Add a requeue_ack() method that puts a PendingAck back into pending_acks if the XACK command failed, so it will be retried on the next send_pubsub_acks() call.

#![allow(unused)]
fn main() {
pub fn requeue_ack(&mut self, source: &str, message_id: &str) {
    self.pending_acks.push(PendingAck {
        source: source.to_string(),
        message_id: message_id.to_string(),
        processed_at: Instant::now(),
    });
}
}

Also add a failed_acks counter to PendingAck or a wrapper struct to avoid infinite retry loops — after N failed attempts, log an error and drop the ack (the message will be reclaimed by XAUTOCLAIM eventually, providing a second chance at processing).

Fix 4: Process multiple messages per idle tick

Files: ranch/handlers.rs

Change process_pubsub_messages() to loop through all queued messages instead of processing just one:

#![allow(unused)]
fn main() {
pub fn process_pubsub_messages(&mut self) {
    if !matches!(
        self.state.status,
        AgentStatus::WaitingForInput | AgentStatus::Idle | AgentStatus::WaitingForExternal
    ) {
        return;
    }

    while self.source_manager.has_pending() {
        let msg = match self.source_manager.next_message() {
            Some(m) => m,
            None => break,
        };

        log_info!(
            self.logger,
            "Processing message {}/{} from {}: {}",
            // ... index, total, source, preview
            msg.source,
            if msg.content.chars().count() > 50 {
                let preview: String = msg.content.chars().take(50).collect();
                format!("{preview}...")
            } else {
                msg.content.clone()
            }
        );

        // ... existing processing logic ...
    }
}
}

Caveat: Processing multiple messages means multiple LLM calls in sequence. Each process_user_input() sends a prompt to the LLM and the response comes back asynchronously. The harness can’t process the next message until the current LLM response completes.

Alternative (recommended): Instead of a tight loop, process one message per tick but reduce the idle delay when the inbox has queued messages. In heartbeat.rs, plan_next_tick() can check if source_manager.has_pending() and use a fast delay (e.g., 0.5s) to drain the queue quickly:

#![allow(unused)]
fn main() {
} else if has_pending_messages {
    TickPlan {
        delay: delay::ACTIVE,  // 0.3s — fast drain
        rerender: false,
    }
} else {
    // Genuinely idle: back off
    TickPlan {
        delay: idle_delay,
        rerender: false,
    }
}
}

This way messages are processed one per tick but at 0.3s intervals instead of 5-30s, so 10 queued messages drain in ~3 seconds instead of up to 5 minutes.

Summary of Changes

Testing

• Add unit tests for claim_command() in redis.rs (verify command format)
• Add unit test for pel_poll_command() (verify 0 is used instead of >)
• Add unit test for requeue_ack() in manager.rs
• Add integration test: poll with >, simulate crash (no ack), restart, XAUTOCLAIM recovers the message
• Run existing test suite: cargo test -p cowboy-harness

Per-Agent Isolation

Status: substrate A implemented (per-agent message-layer isolation). Substrate B (per-agent network namespaces / provider keys) is deliberately not planned — see “Why substrate B is not needed” below. This turned multi-agent from “one shared trust domain” into per-agent isolation at the message layer, which is the layer that matters here.

Implemented behind services.cowboy.pubsub.redisAcl.enable: each agent has its own inbox keyspace, its own agent_<name> ACL identity, and reaches Redis only through its own 0600 unix socket (holding no credential); the host-side bridge routes inbound mail per agent. Eval/reasoning-verified — the socket-activation and ACL behaviour still wants runtime confirmation on a real NixOS host.

Problem

Today multi-agent is honestly documented as one trust domain (services.cowboy.mode): all enabled agents share one network namespace, one secrets proxy, and one Redis with a single agent ACL identity. The pubsub keyspace is per-source, not per-agent — bridges write discord:inbox, and every agent reads it under the same consumer group (cowboy). Redis hands each stream entry to exactly one consumer, so two agents on discord:inbox load-balance the channel: a message destined for alice can be consumed by bob. That is correct for an interchangeable worker pool, but wrong for distinct agents (personas).

Two audit findings live here:

• N user-level Redis instances on port 6379 in the no-proxy/no-bridges fallback (modules/pubsub.nix) → port contention in multi mode.
• The ACL cannot be finer-grained than the data model: with all sources mapped to all agents, there is nothing per-agent to scope a key to.

Target model

Agents become distinct personas, each owning its own keyspace:

The trust direction holds because the bridge runs host-side as the broker/infra user, outside any agent boundary, with the full-access bridge ACL identity. It fans messages in (routes inbound to the right agent’s inbox) and out (reads every agent’s outbox), while each agent’s scoped identity sees only its own lane.

Inbound: shared bridge, route by channel

One external identity (one discord bot). The bridge consults a channel/guild → agent routing table and writes {agent}:{source}:inbox instead of {source}:inbox. The routing chokepoint is IngestService.publish (pkgs/bridge/ingest.py), which today writes the single self.stream.

Outbound: fan-in

For scoped ACLs to hold, alice writes alice:…:outbox (she cannot write a shared discord:outbox under ~alice:*). So the outbox service (pkgs/bridge/base.py) must read across all per-agent outbox streams and merge — the asymmetry to inbound’s one-to-one routing.

Identity: per-agent proxy on per-agent unix sockets (substrate A)

The agent must hold zero Redis credentials — consistent with how the HTTP mitmproxy already injects provider keys by topology. A per-agent proxy only enforces this if an agent cannot reach another agent’s proxy. On the current single shared cowboy-ns, TCP ports do not enforce that (alice can dial bob’s port). The enforceable substrate on today’s shared-host/multi-uid model is the filesystem:

• One redis-auth-proxy process listens on per-agent unix sockets /run/cowboy/<agent>.sock, owned by that agent’s uid. (N processes only if each should hold just its own password, for blast-radius compartmentalization.)
• Identity is determined by which socket accepted the connection → inject agent_<name>. No SO_PEERCRED (unreliable across user namespaces), no source-IP map.
• redis-cli speaks -s <sock>; the harness cli_prefix() (crates/core/src/pubsub/redis.rs) swaps -h/-p for -s.

The unix socket is a non-exfiltrable capability — there is no credential the agent can copy out. The same mechanism spans shared-host and container deployments; only the substrate that scopes the socket to one agent changes:

In containers the bind-mount is the real guard, not the uid bits: user-namespace remapping means the in-container uid need not equal the host uid that owns the socket. A further benefit there — Redis stays entirely off the container network (unix socket, no TCP), so there is nothing to firewall. This ties into the container-proxy direction (proxy on by default, credential injection).

This was chosen over two alternatives:

• Scoped key held by the agent — simplest, works on shared netns with no proxy, but makes Redis the one credential the agent holds (exfiltratable, asymmetric with the HTTP model). Rejected for inconsistency.
• Per-agent network namespace (substrate B) — the container-grade endgame. It additionally gives per-agent provider keys (today one mitmproxy injects the same provider creds for all agents — the real shared trust domain). Out of scope here; substrate A delivers per-agent Redis identity on the current topology without the netns project, and B remains the upgrade path.

Implementation slices (all landed)

0. Fix the fallback (audit Finding 4). ✅ Always run the system cowboy-redis when backend == redis; delete the per-agent user-level fallback; re-point Redis’s wantedBy/partOf to the always-present cowboy.target. Guarded by checks.nixos-multiagent-eval.

1. Per-agent keyspace + ACL + sockets. ✅ ACL gen emits agent_<name> (~<name>:* + %W~*:outbox) with per-agent password files. One redis-auth-proxy socket-activated off per-agent 0600 sockets injects the identity by FileDescriptorName. Per-agent sources.json uses {agent}:{source}:inbox over redis+unix:// the socket; harness cli_prefix() speaks -s.

2. Bridge routing. ✅ routes/defaultAgent options; route_inbox sends inbound to {agent}:{source}:inbox; discord + email ingests and the IngestService base route per message. The outbox stays shared — agents get write-only (%W) access and the bridge drains it — rather than the originally-sketched per-agent-outbox fan-in (simpler, same isolation: an agent can send but cannot read others’ pending outbound).

   Agent-initiated responders (consult, rebuild) follow the same rule: the request carries the originating agent (agent/user field, from $USER on its own socket) and the service writes its reply to {agent}:{source}:inbox, which ~<name>:* already covers. This replaced a briefly-shipped %R~*:inbox grant that let every agent read every other agent’s routed inbox (SLOP audit Finding 2); the wildcard is gone and no agent can XRANGE/ XREADGROUP another persona’s inbox.

3. Cleanup + contract. ✅ The socket-activated per-agent proxy replaced the single shared-identity TCP proxy. mode = "multi" (modules/options/user.nix), this doc, and the SLOP audit’s Finding 4 are updated to the layered-isolation reality.

Why substrate B is not needed

Substrate B would give each agent its own network namespace and therefore its own mitmproxy with per-agent provider credentials. We are not doing this.

All agents on a host are the operator’s own agents, billed to the operator’s own provider accounts — they are one provider-credential trust domain by design. Isolating provider keys per agent would add real complexity (N namespaces, N proxies, IP allocation, the systemd fan-out) to defend against a threat that does not exist in this deployment: there is no adversarial tenant whose access to a shared key would matter. The isolation that does matter — one agent reading or spoofing another’s mail — is exactly what substrate A delivers at the message layer.

If the model ever changes to running mutually-untrusted agents on one host (true multi-tenant), revisit this: per-agent netns is the mechanism, and the unix-socket Redis identity from substrate A already composes with it. Until then, shared provider credentials across an operator’s own agents is the intended design, not a gap.

Plan Mode

title: Plan Mode tags: [plan, roadmap] library: cowboy status: partial

Plan Mode

Plan mode would have the agent maintain an explicit plan and gate its actions against that plan. Today the plan state exists and is tracked, but nothing drives or restricts the agent’s behavior from it. This page describes what is implemented and what is not.

What exists

A PlanState struct (crates/core/src/context/plan.rs) holds:

• objective: String
• open_tasks: Vec<PlanTask> — each task has content and a status of pending, in_progress, or completed
• blockers: Vec<String>
• next_step: String
• updated_at: String

The agent updates this state through the update_plan_state builtin tool (wire name __UPDATE_PLAN_STATE__). The handler (handle_update_plan_state in handlers.rs) replaces the supplied fields, normalizes the task list so at most one task is in_progress, stamps updated_at, and persists the result.

State is persisted per session to plan_state.json in the session directory (load_plan_state / save_plan_state) and reloaded on session resume.

When the plan is non-empty, it is rendered into a transient context packet that is injected into the LLM request (view/context_packet.rs), showing the objective, up to eight open tasks with their statuses, blockers, and the next step. This keeps a compact plan in front of the model without storing it in the conversation transcript.

What does not exist

There is no plan-mode loop and no execution gating. Specifically:

• The agent’s available tools are not restricted based on plan state.
• There is no read-only / planning phase that must complete before a write/execute phase.
• There is no approval workflow, no plan-vs-action diffing, and no automatic enforcement that actions match open_tasks or next_step.

In other words, the plan is recorded and surfaced to the model, but it is advisory: the model may ignore it, and nothing in the harness blocks or sequences actions according to it. update_plan_state is the only mechanism, and it is an ordinary tool the model chooses to call.

Possible direction

A full plan mode would add, on top of the existing state:

• A planning phase restricted to read-only tools, ending in an explicit plan.
• Gating that ties subsequent tool calls to approved tasks.
• An approval step before transitioning from planning to execution.

These require harness changes that have not been started. The persisted PlanState and the update_plan_state tool are the groundwork; the loop and enforcement are not yet built.

Self-Modification

This is a design sketch, not a feature. No supporting code exists in the repository: there is no agent-managed flake, no generation snapshotting, no self-switch workflow, and no PR-gated approval path. Nothing here is implemented. The heartbeat mechanism in the WASM plugin is unrelated — it handles Zellij timer events for polling.

Idea

The agent would own a Nix configuration (a home-manager flake) and be able to edit it, with changes split by risk:

• Low risk, applied directly — tool and skill definitions, prompts, and the agent’s own working state.
• High risk, requires human approval — anything affecting the security boundary: network/filesystem permissions, egress and filter rules, model and endpoint selection, and system service definitions.

The split exists so that the agent can iterate on its own prompts and tools without a human in the loop, while changes that could widen its access are forced through review.

Sketch of a safe-switch flow

A candidate apply flow, if this were built, might look like:

1. Snapshot the current generation.
2. Apply the new configuration with home-manager switch.
3. Run a health check (tools resolve, skills load, filters active).
4. Roll back to the snapshot if the health check fails or a confirmation is not received within a timeout.

The high-risk category would instead open a pull request rather than apply locally, so a human approves before the change lands.

What would be needed

Realizing any of this requires new work that does not exist today:

• A module that provisions the agent’s own flake and constrains which settings it may change.
• Generation snapshotting, rollback, and retention.
• A health-check protocol the harness can run after a switch.
• An audit trail of switches and approvals.
• Integration with a bridge to open and track approval PRs.

Until those are built, treat this document as a design direction only.

Cowboy Runtime — original design (historical)

Status: historical design document, kept for the rationale. The runtime this doc motivated is implemented in crates/runtime and packaged in the flake as cowboy-runtime (Linux-only); montana and the OCI bundle producer landed before it. The implemented contract is specs/OCI.md (verbs, flags, annotations, lifecycle), with specs/README.md mapping each contract to its implementation and crates/runtime/README.md tracking status. The prose below describes the tree as it stood before the runtime existed — it is the problem statement and design rationale, not a description of today. Where the shipped runtime diverged, an inline (Shipped: …) note says how.

What shipped, and where it diverged from this design:

• The gap is closed. cowboy-runtime executes the cowboy bundle and enforces its policy on the container path today; “nothing executes the bundle” below is the pre-runtime problem statement.
• Narrower verb surface. The public ABI is features / create / start / state / kill / delete (+ --version). The planned exec, update, events, and ps were not built and are explicitly excluded (specs/OCI.md).
• Seccomp Deny shipped (2026-07-09) as an opt-in — COWBOY_RUNTIME_SECCOMP=deny or the rs.cowboy.policy.seccomp annotation, fail-closed — with the default still Unsafe pending enforcement soak.
• Montana is linked in, not launched. __init is PID 1 inside the cage and forks montana in-process as a library; the image ships only the wasm component, no montana binary.
• containerd goes through Imageless. Instead of pointing the stock runc-shim at our binary directly, containerd invokes the external imageless-runc interposer (github:dmadisetti/imageless), which materializes the bundle and execs cowboy-runtime as its delegate — still under the stock runc-v2 shim.
• Egress is AF_UNIX, not a netns sidecar. The cage runs NetworkMode::None; egress rides shared-volume UNIX sockets to a companion container (redis + mitmproxy).
• The gofer/NOTIFY listener stayed deferred. Verb mediation ships as native per-tool sentry_binary wrapping plus an ERRNO syscall boundary installed in __init; seccomp.listenerPath remains unwired.

Thesis

At the time of writing, we already emitted a conformant OCI bundle for a cowboy agent, but nothing executed it on the container path. The bundle was consumed two ways (modules/lib/policy.nix): cowboy’s own launcher lowered it to systemd-run + native sheepdog + a joined netns (deliberately dropping file isolation), and a stock runtime (runc/crun) could run it as a bundle. But when an operator did the obvious thing — docker run cowboy:latest, or a k8s Pod — Docker/runc read only the image config (Cmd/Env/WorkingDir/Volumes) and generated its own runtime spec from CLI flags. Our carefully-produced runtime-spec policy (namespaces, linux.resources, maskedPaths/readonlyPaths, seccomp, and the rs.cowboy.policy.* verb rules) was never consulted. It was real and conformant, but inert on the path most people would actually use.

The proposed runtime closes that gap. It is a small runtime binary — --runtime=cowboy — that executes our shape of OCI bundle, enforcing the policy we already describe, with montana as the workload and nucleus used as a library to supply the isolation primitives cowboy did not then apply on the container path. The goal is that docker run --runtime=cowboy and a k8s RuntimeClass: cowboy give you the isolation the bundle has always described, instead of the runtime defaults.

Nucleus (nucleus-container on crates.io) is a lightweight, Nix-native, security-hardened container runtime in Rust — namespaces, cgroups v2, pivot_root, capabilities, seccomp, and landlock, with SHA-256 rootfs attestation and fail-closed production semantics. It exposes a [lib] nucleus target, so we consume those primitives directly rather than shelling out to its CLI.

Background: the pre-runtime tree (the state this was written against)

• The shape producer already existed. modules/lib/policy.nix::mkOciPolicy is the single producer of an OCI Runtime Spec v1.2 document; modules/lib/oci.nix::mkOciConfig assembles it into a full bundle (standard proc/sys/dev mounts, the full namespace set, a real closure-as-rootfs, noNewPrivileges, resources); modules/container.nix emits both the runnable bundle (services.cowboy.ociBundle — “runc run -b <out> cowboy”) and a loadable Docker image (flake.nix docker-image).
• Cowboy-specific policy already rode in-band. Verb-granular filesystem rules have no native OCI slot (seccomp argument matching is scalar — the whole reason the sheepdog gofer exists), so they travel as annotations under rs.cowboy.policy.deny / .allow. The gofer-mediated syscall set (goferSyscalls) and a seccomp.listenerPath were emitted only for the container consumer; listenerPath was null by default and unwired — and the gofer/NOTIFY file boundary remains deferred today.
• Montana had no isolation. The headless embedder (crates/montana) ran the agent component and executed leaf tools via a plain std::process::Command on a scratch thread — no landlock, seccomp, namespaces, chroot, or rlimits — and its native exec escaped even the WASI capability model (it ran directly on the host). Subagents were additional component instances in-process, so they shared whatever boundary montana had (then: none). (Shipped: montana stays isolation-agnostic as a crate by design, but under cowboy-runtime it runs inside the nucleus cage — landlock, namespaces, pivot_root, resource limits, the stacked seccomp layers — with sheepdog wrapping every tool argv.)
• Sheepdog is a core concern, not a montana one. The sentry/argv-wrapping lives in crates/core (ranch/sentry.rs) and is inert unless SENTRY_BINARY / a sentry_binary config key is set. Montana just runs the argv it is handed; if sentry_binary is configured, core wraps tool argv before calling the host exec effect.

The net gap, then: on the container path we got the image’s process config and Docker’s default cage, plus (only if configured) sheepdog’s per-tool mediation inside the guest. The outer OCI runtime policy — the part mkOciConfig describes — was not applied by anyone. Closing this is what crates/runtime now does.

Goals

1. Execute the cowboy OCI shape natively under Docker and k8s, enforcing the runtime-spec policy the bundle already carries: namespaces, linux.resources, maskedPaths/readonlyPaths, the seccomp block-list, and the rs.cowboy.policy.* verb layer.
2. Use nucleus as a library to supply the isolation primitives cowboy lacked on the container path (the “stop-gaps”): mount-ns + pivot_root (file isolation cowboy opted out of), pid-ns, user-ns (the rootless enabler), cgroups, landlock, and boundary seccomp. Nucleus is a dependency of the runtime, not the runtime and not a separate container.
3. Keep montana the workload, unchanged. The runtime provides the cage montana lacks; montana stays isolation-agnostic. Its then-unsandboxed exec becomes structurally bounded by the cage and (via sheepdog) semantically mediated.
4. Make the runtime a closed execution domain. It runs attested/marked cowboy payloads and refuses everything else, so it cannot be repurposed as a general-purpose hardened sandbox.
5. k8s-first. Deploy as a RuntimeClass that coexists with runc/gvisor, with per-Pod opt-in and node steering.

Non-goals

• Not a general-purpose runtime. We do not compete with runc for arbitrary workloads; non-cowboy payloads are intentionally rejected (see Anti-abuse).
• Not a replacement for sheepdog. The cage is structural (reachability); sheepdog remains the semantic layer (read/edit/create/delete verbs, egress, exec trusted-prefix). They compose.
• Not the WASM-native shim (runwasi) model. Montana is the workload process; the wasm component lives inside it. Making the wasm component itself the first-class OCI artifact is a possible later pivot, noted under Alternatives, not this design.
• Not per-tool re-caging. Isolation is applied once at the container boundary (entry seam) and inherited; we do not build a fresh cage per leaf command.

The cowboy OCI shape

“A specific shape of OCI made intentionally for cowboy payloads” means: a standard OCI Runtime Spec v1.2 bundle that a stock runtime can still run (in degraded form), plus a set of cowboy annotations our runtime additionally honors. Concretely the shape is the existing mkOciConfig output extended with:

• Workload descriptor. The process is montana; the shape must carry (or the image must embed) the wasm component path and the flat agent config (provider/model/keys/dirs) montana passes to load(). Montana took these as a config map; the shape formalizes where they live (image env + an annotation or a mounted config file).
• The verb layer (rs.cowboy.policy.deny / .allow) — already produced; consumed by sheepdog.
• The mediation wiring (goferSyscalls NOTIFY set + seccomp.listenerPath) — already produced but unwired; the runtime is what would finally bind the listener. (Shipped: still unwired — the gofer/NOTIFY file boundary is deferred; verb mediation is native sentry_binary wrapping.)
• A cowboy marker. A required annotation (and, when the threat model warrants, an attested rootfs digest) that identifies the bundle as a legitimate cowboy payload. This is the gate that makes the runtime a closed domain. (Shipped as rs.cowboy.payload — a workload-type gate, not an authentication boundary; specs/OCI.md.)

Design property: the shape stays a valid OCI bundle. A stock runtime ignores the rs.cowboy.* annotations and runs a degraded-but-safe container (ERRNO-only seccomp, structural isolation, no verb mediation). Our runtime reads the annotations and enforces the full policy. Forward- and backward-compatible by construction — the same discipline gVisor/Kata use for their own annotations.

The runtime

Form. A binary implementing the runc CLI ABI, planned as create / start / state / kill / delete / exec, plus update / events / ps as needed. Speaking this ABI means it slots into Docker --runtime= and into k8s via the stock containerd runc-shim pointed at our binary — we write no containerd shim. Invocations are stateless processes that share per-container state under a --root directory, as every OCI runtime does. (Shipped: the public surface is features / create / start / state / kill / delete / --version only; exec, update, events, and ps are excluded, and unknown commands fail with usage rather than falling through — specs/OCI.md. The k8s path runs through the imageless-runc interposer; see Deployment.)

Lifecycle mapping. The runtime’s verbs bracket montana’s existing startup:

• create — read the bundle; validate the cowboy marker (fail closed otherwise); build the cage with nucleus-lib from the consumed config.json (namespaces, cgroups, mounts, pivot, seccomp, landlock); prepare montana as the init process, blocked before it runs.
• start — release montana into its event loop.
• state / kill / delete — the standard reporting and teardown; kill maps signals to montana (graceful quit vs. hard stop); delete reaps and frees the cage.
• exec — kubectl exec / liveness-probe support (an additional process in the cage), or a stub initially. (Shipped: not built; specs/OCI.md excludes it.)

Nucleus as the isolation library. The runtime translates the consumed config.json into calls on nucleus’s primitives — the same document mkOciConfig produces. Nucleus supplies exactly the stop-gaps cowboy’s own launcher drops: pivot_root + mount-ns (real file isolation into the closure rootfs), pid-ns, user-ns (so the pivot/pid work rootless), cgroups (from linux.resources), boundary seccomp (from linux.seccomp), and landlock (new — no equivalent existed anywhere in cowboy). Nucleus is linked as a dependency; the runtime owns lifecycle and policy translation, nucleus owns the syscalls.

Sheepdog integration — two paths, same policy. The verb layer is enforced by sheepdog, which the runtime can wire two ways:

• Native. Set sentry_binary in montana’s config so core wraps tool argv with sheepdog and it installs its own filter per tool — mediation inside the guest. Simplest first step; works without touching the seccomp listener.
• Agent mode. The runtime installs the NOTIFY seccomp filter from the bundle and binds the seccomp.listenerPath socket to sheepdog’s already-implemented runc-agent receiver, so verb rules enforce at the container boundary. The runtime is the natural (and then-missing) driver for this socket.

(Shipped: the native path, plus sheepdog’s default-ALLOW + ERRNO(ENOSYS) escape boundary installed in __init over montana; agent mode — the NOTIFY listener — remains deferred, see crates/runtime/README.md for why.)

Montana as workload. Unchanged. It runs inside the cage; its in-process subagents inherit the cage automatically (one cage per montana = one trust domain, consistent with per-agent-isolation.md). The runtime provides what montana structurally lacks; montana provides the embedder + comms it already has. (Shipped: montana is linked into the runtime as a library and forked in-process by __init — there is no montana binary in the image, only the wasm component.)

Isolation model

• Entry seam, single cage. Built once at create, inherited by montana, every tool it execs, and every in-process subagent. We do not re-cage per operation.
• Cage derived from the bundle’s mounts. The landlock allowlist is the union of the declared OCI mounts (rw/ro as declared) + the rootfs + /nix/store (ro, exec). This makes PVC / NFS / FUSE / emptyDir volumes auto-covered: the same mount list that defines the container filesystem defines the cage, so they can’t drift.
• Toggle model with k8s deferral. On bare Docker the runtime applies the full set from the bundle. On k8s the Pod sandbox already provides netns/pid-ns/cgroups (CRI shaped the config.json accordingly); the runtime honors what’s there and adds the layers k8s doesn’t give per-workload — landlock, fine seccomp, and the cowboy verb layer — rather than double-building namespaces.
• Relationship to sheepdog. Nucleus/the cage is the structural wall (what is reachable at all); sheepdog is the semantic policy within it (which verb on which path, egress, exec trusted-prefix). Complementary — and landlock meaningfully shrinks sheepdog’s trusted computing base (a path blocked structurally can’t be reached even if a mediation race slips).

Seccomp posture: Unsafe now, Deny later

(This split shipped as designed: both modes are built. Deny landed 2026-07-09 as an opt-in — COWBOY_RUNTIME_SECCOMP=deny or the annotation, fail-closed — and the default stays Unsafe until the harvested allowlist survives enforcement soak on varied real workloads; the flip is a one-line change in crates/runtime/src/spec.rs.)

The syscall layer is an explicit, named mode (rs.cowboy.policy.seccomp / COWBOY_RUNTIME_SECCOMP, resolved env > annotation > default) rather than an accident of whatever profile the bundle carried.

• Unsafe (the default, then and now). nucleus at Trace (allow-all + a per-syscall NDJSON firehose) with cowboy’s own escape-denylist floor stacked underneath via sheepdog in __init (the kernel takes the most-restrictive per syscall: escape syscalls → ERRNO, everything else → allowed + logged). This is deliberately permissive — it is the data-collection posture: the firehose records the real syscall closure of montana + wasmtime JIT + tokio so the allowlist can be derived from ground truth rather than guessed. It is emphatically not “seccomp off”: the escape floor (mount family, ptrace, bpf, kexec, module loading, unshare/setns, …) always applies. Every create announces the mode; it is recorded in state.json. The floor is cowboy-authored (rs.cowboy.policy.seccomp-block, or a canonical DEFAULT_SECCOMP_BLOCK) — the runtime deliberately ignores the bundle’s linux.seccomp, because under docker that field is docker’s own default profile, and harvesting it forwarded a clone3 ERRNO entry that sheepdog’s fail-closed map couldn’t resolve, bricking __init before montana ever ran.

• Deny (planned here; built 2026-07-09, opt-in). A default-DENY allowlist: nucleus Enforce + a cowboy-supplied profile derived from the Unsafe firehose, with nucleus’s separate “would-have-denied” audit sink on and the sheepdog floor reduced to a backstop. Enforce + a verbatim profile file (not nucleus’s built-in allowlist, which is too narrow and drags the allow_network/resolv.conf bug) lets cowboy own the allowlist while reusing nucleus’s mature profile loader + sha256 pinning + deny logger. Fail-closed at every seam: an unresolvable, empty, or poisoned allowlist aborts create — never a silent downgrade to Unsafe (a requested tightening that isn’t available must fail-closed). The value is kernel-attack-surface reduction (perf_event_open/io_uring/userfaultfd/keyctl-class), driven by measured data, not curation. (Shipped as designed: the harvested DEFAULT_SECCOMP_ALLOW seed plus a rs.cowboy.policy.seccomp-allow override, opt-in while the default soaks at Unsafe.) Precursor noted at the time: replace sheepdog’s hand-maintained syscall_number map with libseccomp per-arch resolution — the clone3 incident showed even a ~50-name denylist is fragile to a mis-typed or arch-shifted name.

Anti-abuse: a closed execution domain

The concern is hygiene, not (initially) adversaries: --runtime=cowboy should not become a general hardened sandbox that unrelated workloads quietly adopt.

• Runtime create-gate (the real control). At create, assert the cowboy marker (and optionally an attested rootfs digest). Missing → exit non-zero with a clear, helpful message pointing operators at runc. Because the concern is accidental reuse, the marker can be a plain annotation (forgeable is fine); it escalates to a signature/attestation only if the threat model shifts to multi-tenant. Hard-fail, not pass-through — the point is to make the runtime useless for anything but cowboy.
• k8s admission (UX). A ValidatingAdmissionPolicy that scopes who may select runtimeClassName: cowboy (a namespace/label) rejects misuse at apply time with a clean error, before scheduling. Backstopped by the runtime gate, which also covers Docker/Podman where there is no admission layer.
• Reverse direction is safe. A cowboy payload accidentally run under stock runc just runs montana without the extra hardening — degraded, not dangerous. So we only gate the forward direction.

Deployment

• Docker. Register runtimes.cowboy in daemon.json pointing at the binary; docker run --runtime=cowboy.
• k8s. Register a containerd runtime handler (stock runc-shim with our binary as BinaryName) + a RuntimeClass (handler: cowboy); Pods set runtimeClassName: cowboy. Node steering via the RuntimeClass scheduling block so cowboy Pods land only on nodes where the runtime and a suitable kernel are present. Coexists with runc/gvisor/kata on the same cluster. (Shipped: the handler’s binary is the external imageless-runc interposer (github:dmadisetti/imageless) — it materializes the bundle from its release annotation and execs cowboy-runtime as its delegate, still under the stock runc-v2 shim; see crates/runtime/README.md.)
• Node requirements. Landlock (kernel ≥ 5.13; some managed node images ship it off — use best-effort application and log when it no-ops so we never believe we’re caged when we aren’t); cgroup v2; seccomp user-notification + ADDFD (≥ 5.9) for sheepdog agent mode.

Storage

All backends reduce to entries in the bundle’s mounts, honored by the runtime and auto-covered by the derived cage; the isolation model is indifferent to which is chosen. Guidance:

• Durable per-agent state (memory, sessions, data dir) → a block PVC (RWO).
• Shared cross-pod / multi-agent state → NFS (kernel client, RWX, gVisor-safe) — but mind NFS locking (keep lock-sensitive state off RWX), uid squash vs. the agent uid, and that NFS I/O bypasses the agent’s egress proxy (the mount lives above the agent netns).
• Object-store-backed workspaces → FUSE, mounted outside the sandbox by a privileged CSI/sidecar and bind-mounted in (agents can’t mount — sheepdog blocks it); mind mount propagation, and note FUSE is incompatible with a gVisor escalation.
• Scratch + the runtime sockets → emptyDir/tmpfs (unix sockets want node-local ephemeral storage).

Ownership must line up: the agent uid needs to own its PVC (fsGroup/runAsUser) and its per-agent socket, and map to NFS exports.

Comms & credentials

The runtime owns only the lifecycle plane. The cowboy control plane is unchanged and rides beside it, wired via the mounts/netns the runtime honors:

• Driver (intents in ↔ frames out): montana’s ndjson socket (unix 0600 / TCP). On k8s expose it as a containerPort + Service; mirror frames to stdout as ndjson for free kubectl logs.
• Inter-agent bus: the per-agent redis socket, bind-mounted in; identity keyed by which socket accepted the connection (topology, not credential), which survives containers exactly as per-agent-isolation.md anticipated. Wired as a shared volume + redis sidecar or a cluster Service.
• Egress + credentials: the mitmproxy sidecar + a NetworkPolicy; provider keys are stubbed with the proxy placeholder and injected at the network layer, so the agent process never holds them. (Shipped: the cage runs NetworkMode::None — no netns to police with a NetworkPolicy; egress rides a shared-volume AF_UNIX socket to a companion container carrying the redis bus and the credential-injecting mitmproxy. Same policy, different transport.)

Key risks & open questions

1. Seccomp stacking (highest). Three seccomp layers can coexist: the k8s RuntimeDefault profile, the bundle’s boundary filter, and sheepdog’s notify filter. They compose (most-restrictive wins), so we must verify the Pod default does not block what sheepdog itself needs (seccomp, /proc/<pid>/mem reads, the NOTIF ioctls / ADDFD). This is the most likely silent breakage.
2. Landlock availability on managed nodes — best-effort + explicit logging.
3. PID-1 duties. When montana is the container init (pid-ns on, or the Pod’s pid namespace), it must reap zombies and forward signals — a small but non-optional obligation.
4. Wiring seccomp.listenerPath and choosing the default sheepdog path under the runtime. (Resolved: native sentry_binary shipped first; the listener remains deferred.)
5. Identity/uid reconciliation across PVC fsGroup, the redis socket owner, and NFS squash.
6. Marker strength. Plain annotation is right for hygiene; revisit toward attestation if we ever run untrusted third-party agents.

Alternatives considered

• Nucleus as the runtime binary (rather than a library the cowboy runtime calls). Rejected for now: we want the runtime to own the cowboy-specific concerns (marker gate, montana workload, verb wiring) and treat nucleus as an isolation toolkit. Revisit if the two collapse cleanly.
• A containerd shim (montana as the runwasi-style workload). More work and containerd-specific; its unique payoff is making the wasm component the first-class OCI artifact, which we don’t need while montana is always the host for comms/subagents. Deferred, not foreclosed.
• Do nothing / rely on docker run + baked policy. This was the state at the time of writing: degraded and confusing (the bundle looked enforced but wasn’t on the container path). Rejected — closing that gap was the point, and the shipped runtime closed it.

Relationship to existing code (reuse, don’t rebuild)

• Reuse policy.nix / oci.nix / container.nix as the shape producer; add only the cowboy marker annotation and the workload descriptor, and wire the existing seccompListenerPath option.
• New: the runtime binary, and a nucleus library dependency (nucleus-container) for its isolation primitives.
• Unchanged: sheepdog (native or its already-implemented runc agent mode); montana (the workload — optionally gains a sentry_binary config value to turn on native mediation).

Phasing (rough)

1. Shape. Extend mkOciConfig with the cowboy marker + workload descriptor; wire seccompListenerPath. (nix only)
2. Runtime skeleton. runc CLI ABI over --root; lifecycle verbs; launch montana as workload; the marker gate. No isolation yet — parity with “real lifecycle, Docker’s cage.”
3. Nucleus stop-gaps. Apply cgroups + pivot/mount-ns + landlock (derived from mounts) + boundary seccomp from the consumed config.json.
4. Sheepdog wiring. Native (sentry_binary) first; then agent mode (bind listenerPath).
5. Docker smoke. docker run --runtime=cowboy; verify enforcement (masked paths denied, limits applied, landlock scoping).
6. k8s. RuntimeClass, toggle deferral, storage, comms sidecars.
7. Hardening. Seccomp-stacking verification, PID-1 reaper, admission policy, optional attestation.

Implementation status against these phases lives in crates/runtime/README.md; the shipped contract is specs/OCI.md.