Skip to content

Observability

Every run leaves a durable, queryable record: node results in SQLite, per-node prompts and output on disk, gate transcripts, and a live event stream. This page maps those surfaces, shows how to read them from the CLI, and covers exporting traces to an OpenTelemetry backend such as Langfuse.

mini-ork does not require a server or an external monitoring account. The local state database is the source of truth; remote tracing is opt-in.

Where the evidence lives

A run writes to .mini-ork/runs/<run-id>/ and mirrors its structured state into .mini-ork/state.db.

Surface Path Holds
Run dir .mini-ork/runs/<run-id>/ Plan, node prompts, outputs, artifacts, gate logs
Verdict <run-dir>/run-verdict.json Final verdict, failing gates, reasons
Gate logs <run-dir>/verifier-*.log Raw stdout/stderr from each gate
LLM calls state.db.llm_calls One row per API turn: lane, model, tokens, cost, session id
Run events state.db.run_events Node started/finished, heartbeats, finish reasons
Runs state.db.task_runs Run status, verdict, recipe, timestamps

The database is a normal SQLite file. Point any tool at it:

sqlite3 .mini-ork/state.db \
  'SELECT id, recipe, status, verdict, created_at
     FROM task_runs ORDER BY created_at DESC LIMIT 5;'

Use task_runs.id

The primary key column is id, not run_id. Queries copied from older notes that select run_id from task_runs will fail.

Reading the event stream

run_events powers the node DAG in the web app and IDE. One row per state change, ordered by time:

SELECT event_type, payload_json, datetime(created_at, 'unixepoch') AS at
  FROM run_events
 WHERE run_id = '<run-id>'
 ORDER BY created_at;

event_type values include node start and end, heartbeat, and terminal states such as succeeded, failed and blocked; payload_json carries the node id and event detail. A node with a start row and no end row is not "slow" — it is still running or its process died; check the run lease and the node log.

The board surface

mini-ork board is the read API the web app and IDE consume. It is also the most convenient way to inspect state from a terminal.

mini-ork board                 # project state: runs, counts, learnings, errors
mini-ork board --json          # the same payload, machine-readable
mini-ork board --shell         # shell-friendly rendering
mini-ork board show <run-id>   # one run, all tabs

The payload carries version, header, project, home, generated_at, runs, total, counts, learnings, automations, scheduler, recipes, workspaces and errors. errors is not decorative: a board that renders while errors is non-empty is showing partial truth.

Node views

mini-ork board node <run-id> <node-id> --view overview
mini-ork board node <run-id> <node-id> --view output
mini-ork board node <run-id> <node-id> --view prompt
mini-ork board node <run-id> <node-id> --view stream --offset 0
View Contents
overview Node type, lane, status, timing, finish reason
output The node's final response, as recorded
prompt The exact prompt dispatched to the lane
stream Incremental output, read by byte offset
telemetry Calls, tokens, cost for this node
learning Lessons or directives attributed to the node
changes Files the node touched
artifacts Declared artifacts and their presence
harness The harness/transport used for the lane

--offset makes stream tail-able: call it again with the last offset to read only new bytes. There is no websocket contract to implement; polling this view is the supported live view.

One node, not the whole run

When debugging, start with --view telemetry and --view output. Most "the model failed" incidents are visible as a truncated output or a zero token count, not as a missing gate.

Run-level inspection

mini-ork board checkpoints <run-id>       # recorded checkpoints
mini-ork verifier list                    # recent gate evaluations
mini-ork run --resume <run-id>            # continue a stopped run

The resume contract prints one [mini-ork-resume] line stating which nodes are skipped and which will be re-dispatched, before any work starts. Trust that line over the human summary — it is generated from the same state the executor reads. See Recovery.

OpenTelemetry and Langfuse

mini-ork can emit a span tree per run:

task_run "code-fix-1781…"
├── agent "implementer"
│   ├── llm_call turn=0
│   └── llm_call turn=1
├── agent "verifier"
└── verifier "tests-pass"        (non-LLM span)

Spans use llm.*, langfuse.* and mini-ork.* attribute namespaces, so a Langfuse or OpenInference-compatible backend renders them without a mapping layer.

Configuration

export MO_OTEL=1
export LANGFUSE_PUBLIC_KEY="pk-lf-..."
export LANGFUSE_SECRET_KEY="sk-lf-..."
export LANGFUSE_OTLP_ENDPOINT="https://cloud.langfuse.com/api/public/otel/v1/traces"

LANGFUSE_OTLP_ENDPOINT defaults to Langfuse Cloud. Point it at a self-hosted instance or any OTLP/HTTP collector that accepts JSON.

Variable Effect
MO_OTEL 1 enables span buffering for a run; unset means no telemetry
MO_OTEL_DRY_RUN 1 prints the payload instead of sending it
LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY Basic auth for the OTLP request
LANGFUSE_OTLP_ENDPOINT OTLP/HTTP traces URL

How a run emits spans

With MO_OTEL=1, the executor writes a root span when the run starts and agent spans as nodes finish. Spans buffer as JSONL at $MINI_ORK_RUN_DIR/.otel-spans.jsonl. At run end the executor flushes them and joins the llm_call layer from state.db.

If the endpoint is unreachable, the buffer survives. Nothing is lost and the run does not fail because telemetry failed:

python3 -m mini_ork.otel_export --from-jsonl .mini-ork/runs/<run-id>/.otel-spans.jsonl

A successful flush renames the buffer to *.sent. A buffer that is still present after a run is the signal that export did not complete — re-run the command above.

Offline export

A finished run can be exported from the database alone, without a live buffer:

python3 -m mini_ork.otel_export <task_run_id> --db .mini-ork/state.db

Trace and span ids are derived deterministically from task_run_id (a sha256 prefix), so exporting the same run twice produces the same ids and the backend deduplicates rather than double-counting. This also means live and offline exports of one run agree.

Add --dry-run to inspect the OTLP JSON without sending it.

Telemetry can contain source code

Prompts and outputs are sent to the configured endpoint. Treat the OTLP destination as a data processor with the same trust level as your model provider. Leave MO_OTEL unset when that is not acceptable.

Logs and where to look first

Question Look at
Why did the run fail? <run-dir>/run-verdict.json, then the failing verifier-*.log
What did the model actually see? board node … --view prompt
What did it answer? board node … --view output
Was the call expensive? board node … --view telemetry, llm_calls
Did the node even start? run_events for that node
Did a gate run at all? verifier list; absence is a finding, not a pass

A gate that never ran must never be read as a pass. mini-ork reports UNVERIFIED or VACUOUS rather than green for missing evidence; see Verification and gates.

Retention

The run dir and state.db grow with every run. mini-ork garden reports orphaned runs, stale locks and drift; mini-ork sandbox-gc reclaims leaked sandboxes. Neither deletes run evidence. Archiving or pruning runs is an operator decision — copy .mini-ork/runs/<run-id>/ (or the whole home) before removing anything.

Next