Skip to content

Troubleshooting

Most mini-ork failures are diagnosed in the same order: check state, check the lane, check the evidence, then act. This page is organised as symptom → likely cause → fix, with the diagnostic command that distinguishes them.

First three commands

mini-ork doctor                  # environment, paths, DB, lanes
mini-ork board                   # runs, active queue, recent failures
mini-ork validate                # static checks over recipes and contracts

doctor is the cheapest explanation of "nothing works". board shows what the engine believes about runs. validate catches recipe and contract errors before a run can hit them.

Environment and installation

Symptom Likely cause Fix
mini-ork: command not found ~/.local/bin not on PATH Add it, or call bin/mini-ork directly
ModuleNotFoundError: mini_ork Editable install points at a moved checkout Re-run make install from the engine root
Python version error Interpreter below 3.11 Re-run the installer with PYTHON=python3.11
doctor reports git missing Git not installed or not on PATH Install git; worktrees and checkpoints need it
doctor reports sqlite missing The sqlite3 CLI is absent Install it; the Python runtime embeds SQLite, but operator commands use the CLI

Paths and state

Symptom Likely cause Fix
Runs write into the wrong directory MINI_ORK_HOME or MINI_ORK_DB is set in the shell echo $MINI_ORK_HOME, then unset or point it at the intended project
home DB last migrated by vX Engine and project state disagree Run mini-ork update from the intended engine
no such table from board DB never migrated, or MINI_ORK_DB points at a fresh path mini-ork init in the project, then mini-ork update
Project registry looks empty A different MINI_ORK_HOME is in effect Use an absolute --db path in the command, or fix the env

The path contract is defined in Engine map. Persistent confusion almost always traces to one exported variable.

Providers and lanes

Symptom Likely cause Fix
no provider lane for role No lane in agents.yaml serves that role mini-ork providers status, then add or reconfigure a lane
401 / auth error Missing or wrong key for that provider mini-ork providers configure; verify with the provider, not by printing the key
Node hangs until timeout Wrong endpoint or a key the provider rejects late Check the lane's base URL; rc=124 after MO_NODE_TIMEOUT_S is the symptom
All work lands on one expensive lane Routing policy or fallback chain resolves there Inspect MO_ROUTING_POLICY; check the lane order in agents.yaml
Lane marked unhealthy Health check failed, or a provider is rate-limited mini-ork providers status shows last check and reason

Never paste a key into a kickoff, a prompt, or a log. If a key may have leaked, rotate it before investigating anything else.

Planning and classification

Symptom Likely cause Fix
Wrong recipe chosen Kickoff lacks scope or acceptance detail Force it: mini-ork run <recipe> kickoff.md
Planner fails repeatedly on JSON Kickoff too large or self-contradictory Split it; one outcome per kickoff
Plan has one giant step No concrete file or verification hint Name the files and the success check in the kickoff
Plan references files that do not exist Stale kickoff or moved base Refresh the kickoff; see Recovery for a moved base

Execution

Symptom Likely cause Fix
artifact_contract failure Producer did not write a declared output Read the node's manifest under workspace/manifests/; check the path
Artifact completion rejected Output stale, empty, or failing its kind contract Inspect node.artifact_completion in the run events
Consumer never starts A parent failed, so its descendants are blocked Fix the parent; do not tick the consumer by hand
Worktree creation refused File-surface claim overlaps a live worktree Use a different slug, or wait for the owning run
Edits landed in the wrong tree Run executed in place instead of a worktree Check the recipe's workspace policy; see Worktrees
Cost pause A budget or daily cap tripped Raise the cap, or wait for the daily reset

The artifact rules are in Artifact graph.

Verification

Symptom Likely cause Fix
REFUTED with no failing test A relation or differential check vetoed the patch Read the relations block in verdict.json; the veto names its evidence
UNVERIFIED even though tests pass Suite adequacy downgraded a green, or replay did not reach the change Read suite_adequacy.json and the levels_reasons field
VACUOUS The checks that ran could not discriminate Add a check that can fail; this is not a pass
Gate returns defer The evaluator could not see its evidence Fix the evidence input; defer is never a green
Verdict differs after a feature toggle One technique was disabled or enabled Compare the knobs listed in Verification stack

Every verdict cites files under the run directory. If you cannot find the evidence, treat the verdict as unproven and re-run.

Scheduler and automation

Symptom Likely cause Fix
Epic stuck as escalated Attempt cap reached mini-ork epics show <id>, then mini-ork epics retry <id>
Epic held as blocked Post-verdict hook exited 10 Read held_reason; only epics retry releases a held epic
Scheduler exits 4 Pre-dispatch hook deferred under --once Expected; it re-probes after the idle interval
Scheduler exits 2 Cost pause or budget cap Raise the cap or wait for the daily reset
Hooks time out Hook exceeded MO_SCHED_HOOK_TIMEOUT_S Shorten the hook; rc=124 counts as a failed attempt

The full retry contract is in Scheduler and retries.

Reading the database

sqlite3 .mini-ork/state.db '.tables'
sqlite3 .mini-ork/state.db \
  'SELECT id, recipe, status, verdict, created_at FROM task_runs ORDER BY created_at DESC LIMIT 5;'
sqlite3 .mini-ork/state.db \
  'SELECT run_id, node_id, status, rc FROM node_runs ORDER BY started_at DESC LIMIT 10;'

Treat the schema as a debugging surface, not a public API: columns grow between releases. Prefer board and the run directory for anything you expect to keep working.

Reading a node failure

Every failure has an evidence trail. Read it in this order:

mini-ork board show <run-id>                 # run state and node outcomes
mini-ork board node <run-id> <node-id>       # this node's output and logs
ls .mini-ork/runs/<run-id>/nodes/<node-id>/  # prompt, output, artifacts
rc Conventional meaning
0 Success
124 Timed out (MO_NODE_TIMEOUT_S)
126 Found but not executable
127 Command not found
other The tool ran and failed — read its output

Do not infer from the exit code alone whether files landed. Check the declared outputs in workspace/manifests/<node>.outputs.json and their freshness.

When to stop debugging and escalate

Escalate instead of retrying when any of these is true:

  • the same failure survives two retries with unchanged evidence;
  • the failure is in the engine rather than the target (trace it to a module in Engine map);
  • the fix would require editing shared infrastructure, secrets or migrations;
  • a run shows a REFUTED verdict on a change someone already merged.

For those, use Failure triage to attribute the failure, then Recovery to choose between retry, resume and repair.