Sandboxing¶

mini-ork executes agent work in a workspace it controls, and the isolation policy decides how much of the host that workspace can reach. There are two independent axes — on-host isolation and remote placement — and one per-node switch that multiplies whichever axis is active.

The default is the cheapest thing that works: a local workspace with no sandbox. Everything below is opt-in, and each layer states what it confines and what it does not.

Two axes¶

flowchart TB
    subgraph LOCAL["Axis 1 — workspace backend (this host)"]
        L[local<br/>no isolation] --> D[docker<br/>container]
        D --> M[microvm<br/>hardware VM]
    end
    subgraph REMOTE["Axis 2 — remote placement (another machine)"]
        R[remote<br/>RemoteWorkspace] --> ST[static node<br/>nodes.yaml / MO_NODE_URL]
        R --> DT[daytona<br/>sandbox per run]
    end
    NODE["MO_NODE_ISOLATION=1<br/>one sandbox per node"] -. multiplies .-> REMOTE
    NODE -.-> LOCAL

Axis	Selector	Backends	Where work runs
Workspace backend	`MO_SANDBOX_BACKEND` + `MO_SANDBOX_SCOPE=tool\\|agent`	`local`, `docker`, `microvm`	This host, bind-mounting a shared drive at `/workspace`
Remote placement	`--placement remote --env <profile>` (`MO_PLACEMENT`, `MO_NODE_ENV`)	`remote` → `RemoteWorkspace` over the node-agent protocol	Any machine running `mini-ork node-agent`

The axes are orthogonal. You can run a Docker-isolated workspace locally, or place a run on a remote VM without any on-host container, or combine them. What you cannot do is treat one as a substitute for the other: a Docker container on your laptop does not protect you from a hostile provider, and a remote VM is not isolated from its own siblings unless per-node isolation is on.

Axis 1: on-host workspace backends¶

`local`¶

The workspace is a directory on the host: a git worktree when the recipe mutates a repository, or an in-place directory when it does not. There is no boundary beyond filesystem permissions. This is the default because it is the fastest and because most runs are trusted.

Use local when:

you wrote the kickoff and control the repository;
the recipe's nodes are trusted prompts, not third-party code;
you want minimal boot cost.

`docker`¶

The workspace is a container with the run directory bind-mounted at /workspace. The container gets its own filesystem and process tree; it shares the host kernel.

MO_SANDBOX_BACKEND=docker MO_SANDBOX_SCOPE=agent mini-ork run code-fix ./kickoff.md

MO_SANDBOX_SCOPE decides when the boundary is entered:

Scope	Meaning
`tool`	Isolate tool execution
`agent`	Isolate the whole agent turn, including its workspace writes

The trade-off is straightforward: Docker stops a node from writing outside its mount and from seeing the host's process table. It does not stop a kernel escape. On macOS, Colima or another Docker host must be able to bind-mount the run directory — a path the VM cannot see produces an empty workspace, not an error. Prefer paths under a directory the Docker host shares.

`microvm`¶

The workspace runs in a hardware-virtualised VM. This is the strongest on-host boundary and the most expensive to boot. Reach for it when the code being run is untrusted and the host is shared.

Backends are selected through a lazy catalog in mini_ork/runtime/sandbox.py, so adding one is a module, not an edit to the resolver.

Axis 2: remote placement¶

--placement remote --env <profile> runs agent nodes on another machine through RemoteWorkspace. The remote side runs mini-ork node-agent, which speaks one HTTP protocol regardless of where it lives.

mini-ork run code-fix ./kickoff.md --placement remote --env daytona

The node-agent handles the hard parts of remote execution:

git-snapshot tree sync up and down (mini_ork/remote/tree_sync.py);
the run-dir mirror (run_mirror.py);
detached processes that survive a dropped connection and reattach after a restart;
per-lane secret scoping (secrets_scope.py);
engine pinning by git sha (node_agent/engines.py);
setup steps that prepare the image.

Static nodes¶

static is the default provider. It resolves a node from nodes.yaml or MO_NODE_URL; nothing is created or destroyed. You boot the VM, install the node-agent, and keep it running. This is the simple tier: one long-lived machine you control.

Ephemeral cloud sandboxes¶

A provider creates a machine on demand and releases it at run end. The first implementation is Daytona; the profile chooses it:

# config/environments/daytona.yaml
provider: daytona            # static (default) | daytona | <plugin>

MO_NODE_PROVIDER overrides the profile per invocation. With neither set, the provider is static, which is byte-for-byte the pre-provider behaviour.

The contract is four verbs:

class NodeProvider(Protocol):
    name: str
    def create(self, *, run_id, profile, env, engine_root) -> NodeEndpoint: ...
    def attach(self, handle, *, env) -> NodeEndpoint | None: ...   # never creates
    def release(self, handle, *, env) -> None: ...                 # idempotent
    def reap(self, *, max_age_s, dry_run, env) -> list[str]: ...   # TTL sweep, fail-open

A NodeEndpoint carries the node URL plus two secrets, both excluded from repr and from the on-disk locator: the node-agent bearer token, and provider-level auth headers (for example Daytona's preview token). RemoteWorkspace._request is the single HTTP call site and adds them to every request, including streams.

Two properties matter more than the API shape:

attach never creates. Teardown paths (kill_run, run-end release) use attach, so tearing down a run can never provision a new machine by accident.
Release is idempotent and fail-open. A double release is harmless; a release that cannot reach the provider is logged and retried by the TTL reaper rather than failing the run.

Per-node isolation¶

By default a remote run gets one sandbox for the run, keyed on the run id. Every node — implementer, reviewer, verifier — re-attaches to it. That is the right shape for cost: a boot costs seconds and a round trip, and paying it once per run is what makes a multi-node recipe affordable.

It is the wrong shape for isolation. A reviewer shares a machine, a process tree and a filesystem with the implementer it reviews. Per-lane secrets are scoped in time, but a node that can read the sandbox disk after its sibling ran has seen its sibling's lane.

MO_NODE_ISOLATION=1 mini-ork run code-fix ./kickoff.md --placement remote --env daytona

BEFORE (one sandbox per run)          AFTER MO_NODE_ISOLATION=1

  control plane (local)                 control plane (local)
  ├── plan, state.db, UI, git           ├── plan, state.db, UI, git
  └── sandbox "mo-<run>"                ├── sandbox "mo-<run>--implementer-1-3f2a1b"
      ├── implementer node              │   └── implementer node
      └── reviewer node                 └── sandbox "mo-<run>--reviewer-a91c0d22"
                                            └── reviewer node

The session key is the lever¶

Every dispatch gets its workspace from get_run_session, which caches one Workspace per (run_id, backend) and calls up() once. That cache is deliberate — a 2-lane fallback chain would otherwise boot two containers.

The consequence is subtle: creating a per-node provider entry is not enough. The first node resolves its sandbox, and every sibling reuses it through the cached session without touching the provider again. A unit test of the provider would pass while a real run shared one machine.

So the registry key gains the node:

key = (run_id, backend, node_id)   # node_id is "" unless isolation is on

Isolation-off keeps "" and therefore the exact historical key and behaviour. Both the layer that caches the workspace and the layer that resolves the node read the same helper (run_nodes.isolated_node(env)), so they cannot disagree about whether two nodes share a machine.

What stays local¶

The control plane: the plan, the run directory, state.db, the UI, git and the publisher. The run-level record <run_dir>/.node-provider.json keeps its name and meaning.

The local worktree is the handoff between nodes. A node's replica syncs down at teardown and the next node syncs up from it, so a sequential recipe needs no cloud filesystem to pass work along. The handoff crosses sandboxes through the control plane, which is already the only writer to the worktree.

Fan-out is still a limitation

Two nodes writing the same path at the same time have nowhere to put the conflict. Per-node isolation does not solve that; content-addressed artifact transport is the intended answer. Turn isolation on for sequential recipes, and treat parallel fan-out as a separate problem.

Records, naming and teardown¶

<run_dir>/
  .node-provider.json                      # the run's sandbox (unchanged)
  .node-provider.lock
  .nodes/
    implementer_1/
      .node-provider.json                  # this node's sandbox
      .node-provider.lock
    reviewer_1/
      .node-provider.json

The node id is untrusted input to a path join. run_nodes.safe_node_id maps it to one path segment ([^A-Za-z0-9._-] -> "_") and appends a digest when the mapping changed something, because the mapping is lossy: implementer:0 and implementer_0 would otherwise share a directory — and sharing a directory is sharing a sandbox.

The cloud idempotency key is the sandbox name: mo-<run-slug>--<node>, hashed on the per-node path. A lost run-dir record can still re-find the sandbox by name, and two nodes cannot converge on each other's.

Teardown has three paths, and all of them must free every node's sandbox or the run leaks one machine per node until the TTL:

Path	How it frees them
Run end (in-process)	`release_remote_session` → `release_all_nodes(run_dir, env)`
`kill_run` (out of process)	`sandbox_reaper.reap_by_marker` → backend hook → `release_all_nodes`
SIGKILL / crash	Nothing — the TTL reaper is the net

release_all_nodes walks <run_dir>/.nodes/*/, so it frees what the filesystem says exists rather than what the current process's environment says. A kill_run issued from the UI — a process that never saw MO_NODE_ISOLATION — therefore still frees everything.

Cost¶

Isolation multiplies boot cost by node count: a 6-node sequential recipe boots 6 node sandboxes, plus the run-level sandbox that stays for control-plane work between nodes. The count is N+1, not N. Sandboxes are freed as the run proceeds where the path allows, and the TTL (default 360 minutes) bounds a run that dies, but the floor is N boots.

Turn it on when the security property is worth the boots:

untrusted or third-party node code;
nodes that handle different secrets;
a shared deployment where one node's compromise must not reach its siblings.

Leave it off for a trusted single-tenant run, where one sandbox per run is cheaper and produces the same result.

Choosing a policy¶

Situation	Policy
You wrote the kickoff, trusted repo, laptop	Default: `local`, no remote placement
Untrusted code, single host	`MO_SANDBOX_BACKEND=docker` with `MO_SANDBOX_SCOPE=agent`
Strongest single-host boundary	`MO_SANDBOX_BACKEND=microvm`
Keep the laptop free, one machine you manage	`--placement remote` with a static node
No pre-provisioned machine, per-run cloud box	`--placement remote --env <profile>` with a provider
Nodes must not share a machine or secrets	Add `MO_NODE_ISOLATION=1`

Trust boundaries, stated plainly¶

A workspace backend confines filesystem and process access. It does not vet the model's intent, the prompts, or the content of the diff.
Remote placement moves execution; it does not add isolation between nodes unless per-node isolation is on.
Per-node isolation separates nodes from each other. It does not separate a node from the control plane, which remains the trusted party and the only writer to the local worktree.
Secrets are scoped per lane, not per node prompt. A node holding a lane's credential can use it for anything its sandbox can reach. Rotate credentials after a run you did not trust.
The TTL reaper is a net, not a guarantee. A crash plus a provider outage can leave a sandbox billing until the provider's own reaper catches it.

Cloud sandbox providers — the provider contract in full.
Run lifecycle — where workspace isolation sits in a run.
Artifact graph — the handoff between nodes.
Remote nodes — operator setup for a static node.
Cloud sandboxes — profiles and providers.