Skip to content

Scheduler and retries

Retries exist at two levels in mini-ork, and they solve different problems. At the node level, the dispatcher retries transient transport failures so a single 429 does not kill a run. At the epic level, the scheduler decides whether a failed attempt gets another try with what the last attempt measured. Neither level retries a failure unchanged and expects a different result.

Node-level retries and timeouts

Node dispatch is bounded by two knobs:

Knob Default Effect
MO_NODE_TIMEOUT_S 1500 Wall-clock limit for one node dispatch
MO_DISPATCH_RETRY_BASE_S 5 Base sleep for transient-failure backoff
MO_DISPATCH_RETRY_MAX_SLEEP_S 45 Cap on the backoff sleep

A node that exhausts its budget fails with rc=124 (timeout). That is a real failure, not a crash: artifact-based completion may still complete the node if every declared output is fresh, non-empty and well-formed. Otherwise the run records the failure and stops at that node.

Distinguish the shapes:

  • rc=124 — timeout; the work may or may not have landed;
  • an auth error — misconfiguration; a retry cannot fix it;
  • a throttle response — transient; backoff handles it;
  • a non-zero tool exit — the work ran and failed; read the output.

The epic queue

mini-ork scheduler drains a dependency-ordered queue of epics. By default each epic gets one attempt: a failed run becomes escalated and stays there. Retry is opt-in, which matters for recursive-improvement campaigns where a failed attempt is information.

mini-ork epics ingest roadmap.md      # create epics and dependencies
mini-ork scheduler                    # run under tmux/launchd, not inside an agent turn

Retry contract

Feature Knob Behaviour
Retry with a cap MO_SCHED_MAX_ATTEMPTS=N or epics.max_attempts A failed attempt returns the epic to not started until N attempts are used, then escalated
Fair retry order — Within one priority tier, the least-attempted ready epic runs first
Carry-over MO_SCHED_CARRY_OVER (default 1) A re-dispatch receives the original kickoff plus ## Previous attempts for the last two
Required verifiers MO_SCHED_REQUIRED_VERIFIERS=a,b done also requires each named verifier JSON to exist and pass
Pre-dispatch hook MO_SCHED_PRE_DISPATCH_HOOK=/path Decide whether to spend the attempt at all
Post-verdict hook MO_SCHED_POST_VERDICT_HOOK=/path Accept, or hold the computed outcome
Hook timeout MO_SCHED_HOOK_TIMEOUT_S (default 600) A hook still running is killed with its children
Per-epic recipe epics.recipe Overrides MO_SCHED_RECIPE for that epic

Carry-over

A re-dispatch is handed the original kickoff plus a ## Previous attempts section describing the last two attempts: verdict, reason, every failing verifier_*.json with its reason (reason, else error_summary), and the run's own notes — reflection.md / cycle-report.md when the recipe writes one, else the reviewer's review-reviewer.json.

Carry-over kickoffs are written to runs/scheduler/kickoffs/<epic>-attempt-<N>.md, where N is the history number. Set MO_SCHED_CARRY_OVER=0 to hand the runner the original kickoff instead.

Required verifiers

With MO_SCHED_REQUIRED_VERIFIERS=cycle-gate,live-smoke:

  • runs/<run>/verifier_cycle-gate.json and verifier_live-smoke.json must both exist and pass;
  • a payload with a pass key is judged by that key alone (true passes);
  • without one, status / verdict must be one of pass, passed, success, proven, ok;
  • the executor copies the verifier log into the file, so lines before the JSON are expected — the last top-level JSON object is read;
  • a verifier that never ran counts as failing.

Hooks

The hook value is a path to one executable, not a command line.

Pre-dispatch:

Hook exit Outcome
0 Dispatch
75 Defer — epic returns to the queue, no attempt used; the pool stops admitting; --once exits 4; the daemon idles --idle-secs and re-probes
anything else A failed attempt

Post-verdict:

Hook exit Outcome
0 Accept the computed outcome
10 Hold — status blocked, held_reason is the hook's last stdout line
anything else A failed attempt

A held epic is never released by a dependency finishing. Only mini-ork epics retry <id> releases it.

A hook still running at MO_SCHED_HOOK_TIMEOUT_S is killed (rc=124); a hook that cannot start — missing, not executable (rc=126) — is a failed attempt.

Hook environment:

Variable Hooks
MO_EPIC_ID, MO_EPIC_ATTEMPT, MO_EPIC_KICKOFF, MO_EPIC_RECIPE Both
MO_RUN_ID, MO_RUN_DIR, MO_VERDICT, MO_OUTCOME, MO_OUTCOME_REASON Post-verdict only

Hook output goes to runs/scheduler/dispatch-<ts>-<epic>-a<N>.hooks.log, next to the run's log.

Gates decide status; they do not undo the run

Required verifiers and the post-verdict hook decide the epic's status after a run. They do not revert what the recipe already did inside it — in-place recipes commit their edits to the target tree before the scheduler sees the verdict. If a rejected attempt must not leave its commit behind, reset the target to a clean base in the pre-dispatch hook, using a worktree dedicated to the campaign.

History

epic_attempts holds one row per attempt: run id, recipe, kickoff, verdict, outcome, reason, start and finish time, numbered 1, 2, 3… over the epic's whole life. epics.attempts counts attempts in the current cap cycle; epics retry --reset-attempts restarts that count, never the history.

mini-ork epics show <id>                     # now lists every attempt
mini-ork epics set <id> --recipe my-rsi --max-attempts 3
mini-ork epics retry <id> [--reset-attempts]

epics retry releases an escalated or held epic: one attempt past its cap, or with --reset-attempts a fresh cycle of max_attempts.

Roadmaps can carry these settings under each epic's heading; epics ingest applies them:

## Data import (id: import-1)
- recipe: my-rsi
- max attempts: 3

## Reporting (id: report-1)
- depends on: import-1

Example: RSI campaign

export MO_SCHED_MAX_ATTEMPTS=3
export MO_SCHED_REQUIRED_VERIFIERS=cycle-gate,live-smoke,no-scope-creep
export MO_SCHED_PRE_DISPATCH_HOOK=$PWD/hooks/pre.sh
export MO_SCHED_POST_VERDICT_HOOK=$PWD/hooks/post.sh
mini-ork epics ingest roadmap.md
mini-ork scheduler

hooks/pre.sh refuses to spend a run whose outcome is already known, and starts each attempt from origin/main so a rejected attempt's commit is not inherited. $TARGET must be a worktree dedicated to the campaign — this discards its local commits.

#!/usr/bin/env bash
curl -sf --max-time 10 "$APP_HEALTH_URL" >/dev/null || { echo "app down"; exit 75; }
git -C "$TARGET" fetch -q origin main && git -C "$TARGET" reset -q --hard origin/main \
  || { echo "cannot reset $TARGET to origin/main"; exit 1; }

hooks/post.sh ships what passed and holds what needs a human:

#!/usr/bin/env bash
if grep -q destructive "$MO_RUN_DIR/verifier_cycle-gate.json" 2>/dev/null; then
  echo "destructive migration held for review"; exit 10
fi
[[ "$MO_OUTCOME" == done ]] || exit 0          # failures retry on their own
git -C "$TARGET" push -q origin HEAD:main || { echo "push failed"; exit 1; }

Scheduler exit codes

rc Meaning
0 Queue drained
1 Fatal error
2 Cost pause / budget cap
3 Max iterations reached
4 A pre-dispatch hook deferred under --once

Scheduled automations

Automations fire recipes on a cadence without an epic queue:

mini-ork automations list
mini-ork automations add          # register one
mini-ork automations pause <id>
mini-ork automations resume <id>
mini-ork automations run <id>     # fire now, regardless of schedule
mini-ork automations tick         # fire every due, enabled automation
mini-ork automations scheduler install     # install the OS scheduler
mini-ork automations scheduler status
mini-ork automations scheduler uninstall

tick is what the OS scheduler calls. Run it from launchd or systemd, not from inside an agent turn: a tick that blocks on a long run starves the next tick.

Observing a drain

mini-ork board                        # run states and recent failures
mini-ork epics show <id>              # attempt history for one epic
tail -f .mini-ork/runs/scheduler/*.log

If an epic is stuck, check in this order: its status (escalated vs blocked), the last attempt's held_reason or failure reason, then the hook log. An escalated epic needs epics retry; a held one needs a human decision.