Cowboy Runtime — original design (historical)

Status: historical design document, kept for the rationale. The runtime this doc motivated is implemented in crates/runtime and packaged in the flake as cowboy-runtime (Linux-only); montana and the OCI bundle producer landed before it. The implemented contract is specs/OCI.md (verbs, flags, annotations, lifecycle), with specs/README.md mapping each contract to its implementation and crates/runtime/README.md tracking status. The prose below describes the tree as it stood before the runtime existed — it is the problem statement and design rationale, not a description of today. Where the shipped runtime diverged, an inline (Shipped: …) note says how.

What shipped, and where it diverged from this design:

• The gap is closed. cowboy-runtime executes the cowboy bundle and enforces its policy on the container path today; “nothing executes the bundle” below is the pre-runtime problem statement.
• Narrower verb surface. The public ABI is features / create / start / state / kill / delete (+ --version). The planned exec, update, events, and ps were not built and are explicitly excluded (specs/OCI.md).
• Seccomp Deny shipped (2026-07-09) as an opt-in — COWBOY_RUNTIME_SECCOMP=deny or the rs.cowboy.policy.seccomp annotation, fail-closed — with the default still Unsafe pending enforcement soak.
• Montana is linked in, not launched. __init is PID 1 inside the cage and forks montana in-process as a library; the image ships only the wasm component, no montana binary.
• containerd goes through Imageless. Instead of pointing the stock runc-shim at our binary directly, containerd invokes the external imageless-runc interposer (github:dmadisetti/imageless), which materializes the bundle and execs cowboy-runtime as its delegate — still under the stock runc-v2 shim.
• Egress is AF_UNIX, not a netns sidecar. The cage runs NetworkMode::None; egress rides shared-volume UNIX sockets to a companion container (redis + mitmproxy).
• The gofer/NOTIFY listener stayed deferred. Verb mediation ships as native per-tool sentry_binary wrapping plus an ERRNO syscall boundary installed in __init; seccomp.listenerPath remains unwired.

Thesis

At the time of writing, we already emitted a conformant OCI bundle for a cowboy agent, but nothing executed it on the container path. The bundle was consumed two ways (modules/lib/policy.nix): cowboy’s own launcher lowered it to systemd-run + native sheepdog + a joined netns (deliberately dropping file isolation), and a stock runtime (runc/crun) could run it as a bundle. But when an operator did the obvious thing — docker run cowboy:latest, or a k8s Pod — Docker/runc read only the image config (Cmd/Env/WorkingDir/Volumes) and generated its own runtime spec from CLI flags. Our carefully-produced runtime-spec policy (namespaces, linux.resources, maskedPaths/readonlyPaths, seccomp, and the rs.cowboy.policy.* verb rules) was never consulted. It was real and conformant, but inert on the path most people would actually use.

The proposed runtime closes that gap. It is a small runtime binary — --runtime=cowboy — that executes our shape of OCI bundle, enforcing the policy we already describe, with montana as the workload and nucleus used as a library to supply the isolation primitives cowboy did not then apply on the container path. The goal is that docker run --runtime=cowboy and a k8s RuntimeClass: cowboy give you the isolation the bundle has always described, instead of the runtime defaults.

Nucleus (nucleus-container on crates.io) is a lightweight, Nix-native, security-hardened container runtime in Rust — namespaces, cgroups v2, pivot_root, capabilities, seccomp, and landlock, with SHA-256 rootfs attestation and fail-closed production semantics. It exposes a [lib] nucleus target, so we consume those primitives directly rather than shelling out to its CLI.

Background: the pre-runtime tree (the state this was written against)

• The shape producer already existed. modules/lib/policy.nix::mkOciPolicy is the single producer of an OCI Runtime Spec v1.2 document; modules/lib/oci.nix::mkOciConfig assembles it into a full bundle (standard proc/sys/dev mounts, the full namespace set, a real closure-as-rootfs, noNewPrivileges, resources); modules/container.nix emits both the runnable bundle (services.cowboy.ociBundle — “runc run -b <out> cowboy”) and a loadable Docker image (flake.nix docker-image).
• Cowboy-specific policy already rode in-band. Verb-granular filesystem rules have no native OCI slot (seccomp argument matching is scalar — the whole reason the sheepdog gofer exists), so they travel as annotations under rs.cowboy.policy.deny / .allow. The gofer-mediated syscall set (goferSyscalls) and a seccomp.listenerPath were emitted only for the container consumer; listenerPath was null by default and unwired — and the gofer/NOTIFY file boundary remains deferred today.
• Montana had no isolation. The headless embedder (crates/montana) ran the agent component and executed leaf tools via a plain std::process::Command on a scratch thread — no landlock, seccomp, namespaces, chroot, or rlimits — and its native exec escaped even the WASI capability model (it ran directly on the host). Subagents were additional component instances in-process, so they shared whatever boundary montana had (then: none). (Shipped: montana stays isolation-agnostic as a crate by design, but under cowboy-runtime it runs inside the nucleus cage — landlock, namespaces, pivot_root, resource limits, the stacked seccomp layers — with sheepdog wrapping every tool argv.)
• Sheepdog is a core concern, not a montana one. The sentry/argv-wrapping lives in crates/core (ranch/sentry.rs) and is inert unless SENTRY_BINARY / a sentry_binary config key is set. Montana just runs the argv it is handed; if sentry_binary is configured, core wraps tool argv before calling the host exec effect.

The net gap, then: on the container path we got the image’s process config and Docker’s default cage, plus (only if configured) sheepdog’s per-tool mediation inside the guest. The outer OCI runtime policy — the part mkOciConfig describes — was not applied by anyone. Closing this is what crates/runtime now does.

Goals

1. Execute the cowboy OCI shape natively under Docker and k8s, enforcing the runtime-spec policy the bundle already carries: namespaces, linux.resources, maskedPaths/readonlyPaths, the seccomp block-list, and the rs.cowboy.policy.* verb layer.
2. Use nucleus as a library to supply the isolation primitives cowboy lacked on the container path (the “stop-gaps”): mount-ns + pivot_root (file isolation cowboy opted out of), pid-ns, user-ns (the rootless enabler), cgroups, landlock, and boundary seccomp. Nucleus is a dependency of the runtime, not the runtime and not a separate container.
3. Keep montana the workload, unchanged. The runtime provides the cage montana lacks; montana stays isolation-agnostic. Its then-unsandboxed exec becomes structurally bounded by the cage and (via sheepdog) semantically mediated.
4. Make the runtime a closed execution domain. It runs attested/marked cowboy payloads and refuses everything else, so it cannot be repurposed as a general-purpose hardened sandbox.
5. k8s-first. Deploy as a RuntimeClass that coexists with runc/gvisor, with per-Pod opt-in and node steering.

Non-goals

• Not a general-purpose runtime. We do not compete with runc for arbitrary workloads; non-cowboy payloads are intentionally rejected (see Anti-abuse).
• Not a replacement for sheepdog. The cage is structural (reachability); sheepdog remains the semantic layer (read/edit/create/delete verbs, egress, exec trusted-prefix). They compose.
• Not the WASM-native shim (runwasi) model. Montana is the workload process; the wasm component lives inside it. Making the wasm component itself the first-class OCI artifact is a possible later pivot, noted under Alternatives, not this design.
• Not per-tool re-caging. Isolation is applied once at the container boundary (entry seam) and inherited; we do not build a fresh cage per leaf command.

The cowboy OCI shape

“A specific shape of OCI made intentionally for cowboy payloads” means: a standard OCI Runtime Spec v1.2 bundle that a stock runtime can still run (in degraded form), plus a set of cowboy annotations our runtime additionally honors. Concretely the shape is the existing mkOciConfig output extended with:

• Workload descriptor. The process is montana; the shape must carry (or the image must embed) the wasm component path and the flat agent config (provider/model/keys/dirs) montana passes to load(). Montana took these as a config map; the shape formalizes where they live (image env + an annotation or a mounted config file).
• The verb layer (rs.cowboy.policy.deny / .allow) — already produced; consumed by sheepdog.
• The mediation wiring (goferSyscalls NOTIFY set + seccomp.listenerPath) — already produced but unwired; the runtime is what would finally bind the listener. (Shipped: still unwired — the gofer/NOTIFY file boundary is deferred; verb mediation is native sentry_binary wrapping.)
• A cowboy marker. A required annotation (and, when the threat model warrants, an attested rootfs digest) that identifies the bundle as a legitimate cowboy payload. This is the gate that makes the runtime a closed domain. (Shipped as rs.cowboy.payload — a workload-type gate, not an authentication boundary; specs/OCI.md.)

Design property: the shape stays a valid OCI bundle. A stock runtime ignores the rs.cowboy.* annotations and runs a degraded-but-safe container (ERRNO-only seccomp, structural isolation, no verb mediation). Our runtime reads the annotations and enforces the full policy. Forward- and backward-compatible by construction — the same discipline gVisor/Kata use for their own annotations.

The runtime

Form. A binary implementing the runc CLI ABI, planned as create / start / state / kill / delete / exec, plus update / events / ps as needed. Speaking this ABI means it slots into Docker --runtime= and into k8s via the stock containerd runc-shim pointed at our binary — we write no containerd shim. Invocations are stateless processes that share per-container state under a --root directory, as every OCI runtime does. (Shipped: the public surface is features / create / start / state / kill / delete / --version only; exec, update, events, and ps are excluded, and unknown commands fail with usage rather than falling through — specs/OCI.md. The k8s path runs through the imageless-runc interposer; see Deployment.)

Lifecycle mapping. The runtime’s verbs bracket montana’s existing startup:

• create — read the bundle; validate the cowboy marker (fail closed otherwise); build the cage with nucleus-lib from the consumed config.json (namespaces, cgroups, mounts, pivot, seccomp, landlock); prepare montana as the init process, blocked before it runs.
• start — release montana into its event loop.
• state / kill / delete — the standard reporting and teardown; kill maps signals to montana (graceful quit vs. hard stop); delete reaps and frees the cage.
• exec — kubectl exec / liveness-probe support (an additional process in the cage), or a stub initially. (Shipped: not built; specs/OCI.md excludes it.)

Nucleus as the isolation library. The runtime translates the consumed config.json into calls on nucleus’s primitives — the same document mkOciConfig produces. Nucleus supplies exactly the stop-gaps cowboy’s own launcher drops: pivot_root + mount-ns (real file isolation into the closure rootfs), pid-ns, user-ns (so the pivot/pid work rootless), cgroups (from linux.resources), boundary seccomp (from linux.seccomp), and landlock (new — no equivalent existed anywhere in cowboy). Nucleus is linked as a dependency; the runtime owns lifecycle and policy translation, nucleus owns the syscalls.

Sheepdog integration — two paths, same policy. The verb layer is enforced by sheepdog, which the runtime can wire two ways:

• Native. Set sentry_binary in montana’s config so core wraps tool argv with sheepdog and it installs its own filter per tool — mediation inside the guest. Simplest first step; works without touching the seccomp listener.
• Agent mode. The runtime installs the NOTIFY seccomp filter from the bundle and binds the seccomp.listenerPath socket to sheepdog’s already-implemented runc-agent receiver, so verb rules enforce at the container boundary. The runtime is the natural (and then-missing) driver for this socket.

(Shipped: the native path, plus sheepdog’s default-ALLOW + ERRNO(ENOSYS) escape boundary installed in __init over montana; agent mode — the NOTIFY listener — remains deferred, see crates/runtime/README.md for why.)

Montana as workload. Unchanged. It runs inside the cage; its in-process subagents inherit the cage automatically (one cage per montana = one trust domain, consistent with per-agent-isolation.md). The runtime provides what montana structurally lacks; montana provides the embedder + comms it already has. (Shipped: montana is linked into the runtime as a library and forked in-process by __init — there is no montana binary in the image, only the wasm component.)

Isolation model

• Entry seam, single cage. Built once at create, inherited by montana, every tool it execs, and every in-process subagent. We do not re-cage per operation.
• Cage derived from the bundle’s mounts. The landlock allowlist is the union of the declared OCI mounts (rw/ro as declared) + the rootfs + /nix/store (ro, exec). This makes PVC / NFS / FUSE / emptyDir volumes auto-covered: the same mount list that defines the container filesystem defines the cage, so they can’t drift.
• Toggle model with k8s deferral. On bare Docker the runtime applies the full set from the bundle. On k8s the Pod sandbox already provides netns/pid-ns/cgroups (CRI shaped the config.json accordingly); the runtime honors what’s there and adds the layers k8s doesn’t give per-workload — landlock, fine seccomp, and the cowboy verb layer — rather than double-building namespaces.
• Relationship to sheepdog. Nucleus/the cage is the structural wall (what is reachable at all); sheepdog is the semantic policy within it (which verb on which path, egress, exec trusted-prefix). Complementary — and landlock meaningfully shrinks sheepdog’s trusted computing base (a path blocked structurally can’t be reached even if a mediation race slips).

Seccomp posture: Unsafe now, Deny later

(This split shipped as designed: both modes are built. Deny landed 2026-07-09 as an opt-in — COWBOY_RUNTIME_SECCOMP=deny or the annotation, fail-closed — and the default stays Unsafe until the harvested allowlist survives enforcement soak on varied real workloads; the flip is a one-line change in crates/runtime/src/spec.rs.)

The syscall layer is an explicit, named mode (rs.cowboy.policy.seccomp / COWBOY_RUNTIME_SECCOMP, resolved env > annotation > default) rather than an accident of whatever profile the bundle carried.

• Unsafe (the default, then and now). nucleus at Trace (allow-all + a per-syscall NDJSON firehose) with cowboy’s own escape-denylist floor stacked underneath via sheepdog in __init (the kernel takes the most-restrictive per syscall: escape syscalls → ERRNO, everything else → allowed + logged). This is deliberately permissive — it is the data-collection posture: the firehose records the real syscall closure of montana + wasmtime JIT + tokio so the allowlist can be derived from ground truth rather than guessed. It is emphatically not “seccomp off”: the escape floor (mount family, ptrace, bpf, kexec, module loading, unshare/setns, …) always applies. Every create announces the mode; it is recorded in state.json. The floor is cowboy-authored (rs.cowboy.policy.seccomp-block, or a canonical DEFAULT_SECCOMP_BLOCK) — the runtime deliberately ignores the bundle’s linux.seccomp, because under docker that field is docker’s own default profile, and harvesting it forwarded a clone3 ERRNO entry that sheepdog’s fail-closed map couldn’t resolve, bricking __init before montana ever ran.

• Deny (planned here; built 2026-07-09, opt-in). A default-DENY allowlist: nucleus Enforce + a cowboy-supplied profile derived from the Unsafe firehose, with nucleus’s separate “would-have-denied” audit sink on and the sheepdog floor reduced to a backstop. Enforce + a verbatim profile file (not nucleus’s built-in allowlist, which is too narrow and drags the allow_network/resolv.conf bug) lets cowboy own the allowlist while reusing nucleus’s mature profile loader + sha256 pinning + deny logger. Fail-closed at every seam: an unresolvable, empty, or poisoned allowlist aborts create — never a silent downgrade to Unsafe (a requested tightening that isn’t available must fail-closed). The value is kernel-attack-surface reduction (perf_event_open/io_uring/userfaultfd/keyctl-class), driven by measured data, not curation. (Shipped as designed: the harvested DEFAULT_SECCOMP_ALLOW seed plus a rs.cowboy.policy.seccomp-allow override, opt-in while the default soaks at Unsafe.) Precursor noted at the time: replace sheepdog’s hand-maintained syscall_number map with libseccomp per-arch resolution — the clone3 incident showed even a ~50-name denylist is fragile to a mis-typed or arch-shifted name.

Anti-abuse: a closed execution domain

The concern is hygiene, not (initially) adversaries: --runtime=cowboy should not become a general hardened sandbox that unrelated workloads quietly adopt.

• Runtime create-gate (the real control). At create, assert the cowboy marker (and optionally an attested rootfs digest). Missing → exit non-zero with a clear, helpful message pointing operators at runc. Because the concern is accidental reuse, the marker can be a plain annotation (forgeable is fine); it escalates to a signature/attestation only if the threat model shifts to multi-tenant. Hard-fail, not pass-through — the point is to make the runtime useless for anything but cowboy.
• k8s admission (UX). A ValidatingAdmissionPolicy that scopes who may select runtimeClassName: cowboy (a namespace/label) rejects misuse at apply time with a clean error, before scheduling. Backstopped by the runtime gate, which also covers Docker/Podman where there is no admission layer.
• Reverse direction is safe. A cowboy payload accidentally run under stock runc just runs montana without the extra hardening — degraded, not dangerous. So we only gate the forward direction.

Deployment

• Docker. Register runtimes.cowboy in daemon.json pointing at the binary; docker run --runtime=cowboy.
• k8s. Register a containerd runtime handler (stock runc-shim with our binary as BinaryName) + a RuntimeClass (handler: cowboy); Pods set runtimeClassName: cowboy. Node steering via the RuntimeClass scheduling block so cowboy Pods land only on nodes where the runtime and a suitable kernel are present. Coexists with runc/gvisor/kata on the same cluster. (Shipped: the handler’s binary is the external imageless-runc interposer (github:dmadisetti/imageless) — it materializes the bundle from its release annotation and execs cowboy-runtime as its delegate, still under the stock runc-v2 shim; see crates/runtime/README.md.)
• Node requirements. Landlock (kernel ≥ 5.13; some managed node images ship it off — use best-effort application and log when it no-ops so we never believe we’re caged when we aren’t); cgroup v2; seccomp user-notification + ADDFD (≥ 5.9) for sheepdog agent mode.

Storage

All backends reduce to entries in the bundle’s mounts, honored by the runtime and auto-covered by the derived cage; the isolation model is indifferent to which is chosen. Guidance:

• Durable per-agent state (memory, sessions, data dir) → a block PVC (RWO).
• Shared cross-pod / multi-agent state → NFS (kernel client, RWX, gVisor-safe) — but mind NFS locking (keep lock-sensitive state off RWX), uid squash vs. the agent uid, and that NFS I/O bypasses the agent’s egress proxy (the mount lives above the agent netns).
• Object-store-backed workspaces → FUSE, mounted outside the sandbox by a privileged CSI/sidecar and bind-mounted in (agents can’t mount — sheepdog blocks it); mind mount propagation, and note FUSE is incompatible with a gVisor escalation.
• Scratch + the runtime sockets → emptyDir/tmpfs (unix sockets want node-local ephemeral storage).

Ownership must line up: the agent uid needs to own its PVC (fsGroup/runAsUser) and its per-agent socket, and map to NFS exports.

Comms & credentials

The runtime owns only the lifecycle plane. The cowboy control plane is unchanged and rides beside it, wired via the mounts/netns the runtime honors:

• Driver (intents in ↔ frames out): montana’s ndjson socket (unix 0600 / TCP). On k8s expose it as a containerPort + Service; mirror frames to stdout as ndjson for free kubectl logs.
• Inter-agent bus: the per-agent redis socket, bind-mounted in; identity keyed by which socket accepted the connection (topology, not credential), which survives containers exactly as per-agent-isolation.md anticipated. Wired as a shared volume + redis sidecar or a cluster Service.
• Egress + credentials: the mitmproxy sidecar + a NetworkPolicy; provider keys are stubbed with the proxy placeholder and injected at the network layer, so the agent process never holds them. (Shipped: the cage runs NetworkMode::None — no netns to police with a NetworkPolicy; egress rides a shared-volume AF_UNIX socket to a companion container carrying the redis bus and the credential-injecting mitmproxy. Same policy, different transport.)

Key risks & open questions

1. Seccomp stacking (highest). Three seccomp layers can coexist: the k8s RuntimeDefault profile, the bundle’s boundary filter, and sheepdog’s notify filter. They compose (most-restrictive wins), so we must verify the Pod default does not block what sheepdog itself needs (seccomp, /proc/<pid>/mem reads, the NOTIF ioctls / ADDFD). This is the most likely silent breakage.
2. Landlock availability on managed nodes — best-effort + explicit logging.
3. PID-1 duties. When montana is the container init (pid-ns on, or the Pod’s pid namespace), it must reap zombies and forward signals — a small but non-optional obligation.
4. Wiring seccomp.listenerPath and choosing the default sheepdog path under the runtime. (Resolved: native sentry_binary shipped first; the listener remains deferred.)
5. Identity/uid reconciliation across PVC fsGroup, the redis socket owner, and NFS squash.
6. Marker strength. Plain annotation is right for hygiene; revisit toward attestation if we ever run untrusted third-party agents.

Alternatives considered

• Nucleus as the runtime binary (rather than a library the cowboy runtime calls). Rejected for now: we want the runtime to own the cowboy-specific concerns (marker gate, montana workload, verb wiring) and treat nucleus as an isolation toolkit. Revisit if the two collapse cleanly.
• A containerd shim (montana as the runwasi-style workload). More work and containerd-specific; its unique payoff is making the wasm component the first-class OCI artifact, which we don’t need while montana is always the host for comms/subagents. Deferred, not foreclosed.
• Do nothing / rely on docker run + baked policy. This was the state at the time of writing: degraded and confusing (the bundle looked enforced but wasn’t on the container path). Rejected — closing that gap was the point, and the shipped runtime closed it.

Relationship to existing code (reuse, don’t rebuild)

• Reuse policy.nix / oci.nix / container.nix as the shape producer; add only the cowboy marker annotation and the workload descriptor, and wire the existing seccompListenerPath option.
• New: the runtime binary, and a nucleus library dependency (nucleus-container) for its isolation primitives.
• Unchanged: sheepdog (native or its already-implemented runc agent mode); montana (the workload — optionally gains a sentry_binary config value to turn on native mediation).

Phasing (rough)

1. Shape. Extend mkOciConfig with the cowboy marker + workload descriptor; wire seccompListenerPath. (nix only)
2. Runtime skeleton. runc CLI ABI over --root; lifecycle verbs; launch montana as workload; the marker gate. No isolation yet — parity with “real lifecycle, Docker’s cage.”
3. Nucleus stop-gaps. Apply cgroups + pivot/mount-ns + landlock (derived from mounts) + boundary seccomp from the consumed config.json.
4. Sheepdog wiring. Native (sentry_binary) first; then agent mode (bind listenerPath).
5. Docker smoke. docker run --runtime=cowboy; verify enforcement (masked paths denied, limits applied, landlock scoping).
6. k8s. RuntimeClass, toggle deferral, storage, comms sidecars.
7. Hardening. Seccomp-stacking verification, PID-1 reaper, admission policy, optional attestation.

Implementation status against these phases lives in crates/runtime/README.md; the shipped contract is specs/OCI.md.