Skip to main content
Status:✏️ DRAFT
Date:📅 2026-07-18
Domain:📦 Harness Lifecycle State Machine

SPEC-0003: Harness Lifecycle State Machine

Overview​

The explicit state machine every harness moves through under daemon supervision. The predecessor, zsh-harnessd, collapsed everything to green/yellow/red by ANDing "supervisor active" with "tmux session up"; this spec makes the machine explicit, adds crash-loop safety, and gives the TUI real states to render. See 📝 ADR-0005 (supervision layers) and the design notes for the transition diagram.

Requirements​

Requirement: State Model​

The daemon SHALL track each harness in exactly one of seven states: stopped (○), starting (◌), running (●), degraded (◐), restarting (◌), stopping (◌), and failed (✖). Separately from state, the daemon SHALL persist an orthogonal enabled boolean (does the daemon want this running?) in its state file (📝 ADR-0007). enabled is intent; state is reality.

Scenario: Intent vs. reality​

  • WHEN a harness exits while enabled is true during a restart cycle
  • THEN the harness MAY briefly report enabled && stopped without either field being considered inconsistent

Requirement: Autostart​

On daemon start, every harness that is enabled — directly or via an autostart profile (📝 ADR-0006) — SHALL transition to starting, except a harness with operating_hours that is out of hours and holds no valid lease, which SHALL begin held (SPEC-0012 REQ "Gate Enforcement").

Scenario: Daemon boot​

  • WHEN the daemon starts and a harness is marked enabled
  • THEN the daemon spawns it (state starting) without operator action

Requirement: Restart On Exit​

While a harness is enabled and its restart policy permits it (REQ "Restart Policy"; the always-restart default), any exit — including a clean exit code 0 — SHALL be followed by a restart (restarting, wait restart_delay, then starting), incrementing the restart count (↻). If the harness is not enabled, an exit SHALL transition it to stopped. An exit the daemon caused to enforce operating hours is not a restart-eligible exit: it SHALL transition the harness to stopped with enabled unchanged (SPEC-0012 REQ "Gate Enforcement").

Scenario: Clean exit while enabled​

  • WHEN an enabled harness under the default restart policy exits with code 0
  • THEN the daemon restarts it after restart_delay (agents and watchers are meant to be long-lived)

Scenario: Exit while disabled​

  • WHEN a harness exits and enabled is false
  • THEN the harness transitions to stopped and is not respawned

Requirement: Restart Policy​

Each harness MAY carry a restart policy mirroring Docker Compose's restart directive: "always" (the default; an omitted key normalizes to it), "unless-stopped" (equivalent to "always" here — a manual stop already persists enabled=false, so both behave like Docker's unless-stopped), "on-failure" (restart only on a non-zero exit; a spawn failure counts as a failure), and "no" (never restart automatically).

When the policy suppresses the respawn, the exit is final: the daemon SHALL clear enabled (the policy outcome persists like an explicit stop, so the harness stays down across daemon restarts, as in Docker), reset crash-loop bookkeeping, and transition the harness to stopped — except a spawn failure, which SHALL land in failed so a command that never came up surfaces loudly rather than as a clean stop. Because the policy is consulted only at exit time, a config change that alters only restart SHALL apply immediately to the running harness (it is not run-affecting; see REQ "Config Change Application").

Scenario: Clean exit under on-failure​

  • WHEN an enabled harness with restart = "on-failure" exits with code 0
  • THEN it transitions to stopped, enabled becomes false, and it is not respawned

Scenario: Failing exit under on-failure​

  • WHEN an enabled harness with restart = "on-failure" exits non-zero
  • THEN the daemon restarts it per REQ "Restart On Exit"

Scenario: Spawn failure under restart=no​

  • WHEN a harness with restart = "no" fails to spawn at all
  • THEN it transitions to failed (not stopped) and is not retried

Requirement: Crash-Loop Detection​

The daemon SHALL detect crash loops: N exits within a window T transitions the harness to degraded, and restart_delay SHALL escalate as a capped exponential backoff (📝 ADR-0005). A single run surviving longer than T SHALL reset the counter and return the harness to running.

Scenario: Flapping harness​

  • WHEN a harness exits N times within window T
  • THEN its state becomes degraded and subsequent restart delays escalate exponentially up to the cap

Scenario: Recovery resets backoff​

  • WHEN a degraded harness's current run survives longer than window T
  • THEN the exit counter resets and the state returns to running

Requirement: Backoff Give-Up​

After the backoff cap / a maximum-attempts threshold, the daemon SHALL stop auto-restarting the harness and transition it to failed. The TUI surfaces failed loudly (needs a human). A manual restart SHALL clear the failed state and begin a fresh start cycle.

Scenario: Giving up​

  • WHEN a degraded harness exhausts its restart attempts
  • THEN the daemon moves it to failed and stops respawning it

Scenario: Manual recovery​

  • WHEN an operator issues restart on a failed harness
  • THEN the failure latch clears and the harness transitions to starting

Requirement: Graceful Stop​

A stop request on a running, degraded, or restarting harness SHALL transition it to stopping: send SIGTERM, wait a grace period, SIGKILL if needed, tear down the PTY session, then transition to stopped and set enabled=false. An operator stop clears intent; a stop the daemon applies to enforce operating hours runs the same sequence but leaves enabled unchanged (SPEC-0012 REQ "Gate Enforcement").

Scenario: Process ignores SIGTERM​

  • WHEN a stop-requested process is still alive after the grace period
  • THEN the daemon sends SIGKILL, reaps the PTY, and records stopped

Scenario: Hours stop leaves intent alone​

  • WHEN the daemon stops a harness to enforce its operating hours
  • THEN the harness transitions through stopping to stopped and enabled is unchanged, so the next window starts it again

Requirement: Config Change Application​

Configuration changes to a running harness (📝 ADR-0006 hot reload) SHALL apply on the next (re)start, never by silently bouncing the process. The daemon SHALL expose enough information for the TUI to flag "config changed — restart to apply."

Scenario: Edit while running​

  • WHEN harness.toml changes a running harness's definition and is reloaded
  • THEN the running process is untouched and the change takes effect on the next restart

Requirement: Lifecycle Events​

The daemon SHALL emit protocol events (SPEC-0002) on lifecycle activity: harness_state_changed { name, from, to }, harness_exited { name, code }, and harness_flapping { name, restarts, next_retry_in }.

Scenario: State change notification​

  • WHEN any harness transitions between states
  • THEN subscribed clients receive harness_state_changed without polling

Requirement: Durable Log Rotation And Compression​

The daemon SHALL write each harness's durable log to $XDG_STATE_HOME/harness/logs/<name>.log. A project harness's log goes one directory down, at logs/<project>/<name>.log. The daemon SHALL rotate the active file once it would pass 8 MiB or is older than 24 hours. Rotation renames it to <name>-<stamp>.log and keeps the five newest backups (📝 ADR-0007). Two rotations SHALL NOT share a stamp.

A rotated backup is sealed. Unless [daemon] compress_logs is false (SPEC-0010), the daemon SHALL compress each backup to <name>-<stamp>.log.zst (zstd) in the background, off the PTY reader. It SHALL NOT compress the active file. A compression SHALL NOT be able to lose a backup. The compressed file appears only by an atomic rename after its contents are synced, and the plain file is removed only after that. When a backup is left plain, the next rotation and the next daemon boot SHALL compress it.

Every reader of the durable log SHALL read a backup in either form, and SHALL read the plain file when both exist. That includes the lifecycle lines behind run correlation. The backup limit SHALL count a backup once, whatever form it is in, and pruning SHALL delete every form. Neither pruning nor project teardown SHALL touch a harness whose name merely starts with this harness's name.

Scenario: A rotated backup is compressed​

  • WHEN web.log rotates to web-20260928T090000.000.log
  • THEN that backup becomes web-20260928T090000.000.log.zst, which decodes to the same bytes, and web.log itself stays plain

Scenario: Pruning compressed backups leaves a sibling harness alone​

  • WHEN harnesses web and web-api both have compressed backups and web rotates past its limit
  • THEN web keeps its five newest backups in whatever form, and every web-api file is untouched

Scenario: Lifecycle lines survive compression​

  • WHEN a harness's older backups are compressed and one is present in both forms after a crash
  • THEN its lifecycle lines read back once each, oldest first, as they did before compression

Requirement: Operator Notification​

When a harness needs a person, the daemon SHALL tell one: if the global [notify] table names a program, the daemon SHALL run it on a transition into failed (give-up), on crash-loop escalation (flapping), when the runaway tool-loop guard stops a harness (loop_stopped), when the session guard rotates a wedged session or fails to (session_rotated), on a failed or timed-out scheduled or triggered run (run_failed, opt-in), once per park when a harness is parked on an exhausted provider quota (parked, SPEC-0021 REQ-13: its message SHALL name the reset and the classifier rule, never the error text), once per stall episode when the stall watchdog finds a persistent harness alive but making no progress (stalled, 📝 ADR-0045: its message SHALL name the idle span, the action taken and the worker's process tree, and an alert SHALL be opened only once the watchdog has stopped restarting it), and when a harness it reported failed or loop_stopped is running again (recovered). A triggered harness never gives up — a failed run lands it in failed and its next firing is the retry (SPEC-0008 REQ "Firing And Overlap") — so that transition SHALL NOT send failed, and the next firing's start SHALL NOT send recovered; the run itself is reported as run_failed, whose message SHALL name the run ledger's streak of consecutive failed runs (SPEC-0022 REQ-5) when it is more than one. The program is a daemon-side hook, not a supervised harness (📝 ADR-0033 governs what the daemon supervises, and is untouched): it SHALL be exec'd without a shell from an absolute argv[0], SHALL receive the event as JSON on stdin and as HARNESS_NOTIFY_EVENT, HARNESS_NOTIFY_HARNESS, HARNESS_NOTIFY_HOST, HARNESS_NOTIFY_STATE and HARNESS_NOTIFY_MESSAGE in its environment, and every string SHALL pass the daemon's credential redaction (📝 ADR-0008) before it leaves. The message SHALL carry the most useful cause the daemon has — the agent's last output line before exit, or the looping tool and count — and the command that shows more.

Delivery SHALL never block supervision (📝 ADR-0007): it is asynchronous and bounded, a full queue drops the notification and counts it, and a hook that outlives the configured timeout SHALL be killed with its whole process group. A repeat of the same event for the same harness inside cooldown SHALL be suppressed; a recovered clears that harness's failed and loop_stopped cooldowns. Outcomes SHALL be logged and exported as harness_notify_deliveries_total{event,result} (SPEC-0013). The table SHALL be validated at load, refused in project files and harness_d drop-ins, and applied on reload. harness doctor SHALL report whether a hook is configured and its last delivery, and harness doctor --notify-test SHALL have the daemon run it once with a test event.

Scenario: Give-up reaches a person​

  • WHEN a harness exits 1 on every start after printing Error: You must be logged in to use Remote Control. and gives up into failed
  • THEN the daemon runs the notify hook with HARNESS_NOTIFY_EVENT=failed, and the message and cause quote that line

Scenario: A failing one-shot is not a give-up​

  • GIVEN a scheduled harness whose every run exits 1, and events naming failed, recovered and run_failed
  • WHEN two firings run and fail
  • THEN the hook receives run_failed for each, the second saying 2 in a row, and never failed or recovered

Scenario: A park is heard once​

  • GIVEN a scheduled one-shot whose provider refuses it with a 402
  • WHEN its run parks it until 00:20 and the 00:15 firing is skipped by the park
  • THEN the hook receives one parked naming 00:20 and the rule, and no failed or run_failed

Scenario: A stall is heard once​

  • GIVEN a persistent harness with stall_timeout = "45m" and stall_action = "restart"
  • WHEN its worker makes no model call for 45 minutes, is restarted, and the replacement then makes a call
  • THEN the hook receives one stalled naming the idle span, the restart and the worker's process tree, and neither failed nor recovered

Scenario: A loop-guard stop is not silent​

  • WHEN the loop guard stops a harness and clears its enabled intent
  • THEN the hook receives loop_stopped naming the tool and the count, and a later transition of that harness to running sends recovered

Scenario: A flapping harness does not spam​

  • WHEN a harness flaps repeatedly inside the cooldown window
  • THEN the hook runs once for it and later occurrences are counted as suppressed

Scenario: A hung hook cannot stall the daemon​

  • WHEN the hook does not exit within timeout
  • THEN its process group is killed, the delivery is recorded as timeout, and supervision was never blocked on it

Direct relationships declared in YAML frontmatter (per the SDD plugin's artifact-graph convention). Run /sdd:graph chain SPEC-0003 for the transitive view.