Metrics

The daemon serves Prometheus metrics at GET /metrics, on 127.0.0.1:10229 by default. They lead with the thing harness list cannot tell you: whether a running agent can actually get a model to answer.

On 2026-09-14 a provider's weekly quota ran out. Four agents failed every model call for about twenty hours, and harness list showed all four as running the whole time. That was accurate, because the processes were up. The supervisor watches processes, and a process can stay up while the provider refuses every request it sends. The only record of the refusals was in each agent's own transcript. The daemon now reads those transcripts and publishes what it finds next to the process state, so one alert can check both.

Scraping it

Loopback needs no auth:

# prometheus.yml (vmagent accepts the same block)
scrape_configs:
  - job_name: harness
    static_configs:
      - targets: ["127.0.0.1:10229"]

To scrape from another machine, bind a non-loopback address and give the daemon a bearer token. If a non-loopback bind has no token, the daemon refuses to start, before it launches any harness:

[server]
metrics_listen = "0.0.0.0:10229"
metrics_token_file = "~/.config/harness/metrics.token"   # chmod 600

scrape_configs:
  - job_name: harness
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/harness-metrics.token
    static_configs:
      - targets: ["agentbox.lan:10229"]

The token has to come from a file, for three reasons:

• harness.toml holds no secrets (ADR-0008).
• HARNESS_* variables may not carry credentials (SPEC-0010).
• Every harness inherits the daemon's environment when it is spawned, so a token set as a daemon environment variable would reach every agent the daemon runs.

A file is also what Vault Agent or OpenBao templates write out. If the file is readable by group or others, the daemon logs a warning.

metrics_listen = "off" turns the listener off. You may want that on a shared machine, because any local account can connect to 127.0.0.1.

Changing either key requires a daemon restart. harness reload re-reads harness definitions but does not rebind listeners. The token file is also read once, at startup, so rotating the token means rewriting the file and then restarting the daemon.

If the port is already in use, the daemon logs an error and keeps supervising without the listener. Your scraper then sees the target as down (up == 0), and it already alerts on that. Exiting would stop every harness on the host just because one port was taken.

Alerts

"Running, but no model call has succeeded in fifteen minutes." This is the shape of the 2026-09-14 outage:

time() - harness_last_successful_call_timestamp > 900
  and on(instance, harness) harness_harness_state{state="running"} == 1

The on(instance, harness) clause is required. The state series has an extra state label, and a bare and matches only series whose label sets are identical, so without the clause this expression would never fire.

# A quota wall. It usually hits every harness that shares a provider at the same time.
sum by (instance, harness) (increase(harness_model_call_errors_total{class="quota"}[10m])) > 0

# Terminal failed state: the supervisor gave up, and only a person can restart the harness.
harness_harness_state{state="failed"} == 1

# Approaching failed: consecutive failures climb before the harness gets there.
harness_consecutive_failures >= 3

# Restart churn: a crash loop, or launches that burn a metered budget.
increase(harness_restarts_total[1h]) > 10

# The error classifier no longer recognises a provider's errors (the wording probably changed).
increase(harness_model_call_errors_unclassified_total[1h]) > 0

# The [notify] hook stopped reaching anyone: it errors, times out, or its queue overflows.
increase(harness_notify_deliveries_total{result=~"error|timeout|dropped"}[1h]) > 0

# The daemon's own collection is broken or losing events.
increase(harness_metrics_collection_errors_total[15m]) > 0

# The endpoint is unreachable: the daemon is down, or the port could not bind.
up{job="harness"} == 0

# A trigger source has been deaf for ten minutes: a channel out of `connected`,
# or a webhook source that is not being served.
max_over_time(harness_trigger_source_up[10m]) == 0

# Someone is knocking with the wrong credential, or a sender's secret rotated.
increase(harness_trigger_events_total{outcome="unauthorized"}[15m]) > 0

# A channel link that keeps dropping.
increase(harness_trigger_reconnects_total[1h]) > 5

harness_trigger_source_up is 0 for a source that is disabled or unbound as well, so scope the first rule to the sources you expect to be live, for example with {source=~"channel.sb|webhook.gitea-pr"}.

Daemon memory

The daemon's own memory is on the same scrape. In September 2026 a daemon reached 12 GB because every spawn leaked a terminal emulator of about 98 MiB (https://github.com/stump-wtf/harness/issues/18). The leak signature is something that should return to a baseline climbing in step with the run count instead:

• go_goroutines rising by a fixed step per spawn and never falling back;
• go_memory_classes_heap_objects_bytes or go_memstats_heap_objects rising in step with harness_state_transitions_total{to="starting"}, which counts spawns.

A high footprint with a flat live heap is GC headroom, not a leak: go_memory_classes_heap_free_bytes and go_gc_heap_goal_bytes are high, and [daemon] memory_limit is the lever. The two need different fixes, so graph them together:

# Live heap against the GC's target and the total mapped.
go_memory_classes_heap_objects_bytes{job="harness"}
go_gc_heap_goal_bytes{job="harness"}
go_memory_classes_total_bytes{job="harness"}

# Goroutines left behind per spawn over six hours. A healthy daemon hovers
# near 0; a leak holds near a constant (1 or more per spawn).
delta(go_goroutines{job="harness"}[6h])
  / on(instance) clamp_min(sum by (instance) (increase(harness_state_transitions_total{job="harness",to="starting"}[6h])), 1)

An alert on the second expression at > 0.5 fires on a per-spawn leak within a few dozen runs. Tune the window to how often your harnesses spawn. To find what is leaking, see Diagnosing daemon memory.

What is exported

Series	Type	Notes
harness_harness_state{harness,state}	gauge	running, stopped, failed, flapping. Every declared harness reports all four, with 1 for its current state and 0 for the others.
harness_restarts_total{harness}	counter	The RESTARTS column of harness list.
harness_consecutive_failures{harness}	gauge	The supervisor's give-up count. Once it exceeds the restart budget, the harness moves to failed.
harness_state_transitions_total{harness,to}	counter	One series per supervisor state (to has 7 values), starting at zero.
harness_model_calls_total{harness,outcome}	counter	success counts tool calls. error counts agent error marks.
harness_agent_program_calls_total{harness,program,outcome}	counter	Shell tool calls by the program they run. outcome is ok or error, taken from the tool result. A program is a lowercase executable name; anything else, and every name past 64 for one harness, is counted as program="other". A program never run is absent, not zero.
harness_model_call_errors_total{harness,class}	counter	quota, auth, timeout, transport, other. Crush and Claude Code harnesses only, for now; see below.
harness_model_call_errors_unclassified_total{harness}	counter	Errors that matched no known pattern. Each one is also counted as class="other".
harness_last_successful_call_timestamp{harness}	gauge	Unix seconds. Absent until the harness's first success in this daemon's lifetime. It is never reported as zero, because zero would read as 1970.
harness_sessions_started_total{harness}	counter	Agent sessions the daemon has seen become active.
harness_session_active{harness}	gauge	1 when the process is up and the agent wrote to a session in the last 10 minutes.
harness_scheduled_runs_total{harness,outcome}	counter	Scheduled harnesses only, counted from the run ledger. success, or failure (a run that failed, timed_out, budget_exceeded, model_mismatch or model_unattested). Skipped, missed, cancelled, interrupted and quota-parked runs are not counted.
harness_runs_total{harness,kind,outcome}	counter	Every closed run in the run ledger, one-shot (kind="oneshot") and resident process lifetimes (kind="resident"). Every harness reports every outcome its kind can produce, starting at zero. harness runs --since … --outcome … --json returns the same records.
harness_run_duration_seconds{harness,kind}	histogram	Closed runs with a known end. Buckets from 1s to 24h.
harness_run_tokens_total{harness,type}	counter	input, output, cache_read, cache_write. Absent until a run records token usage, which needs agent-trace usage items.
harness_run_cost_usd_total{harness,source}	counter	recorded (the agent recorded it) or priced (from a price table). An unknown cost is not counted. Absent until a run carries a cost.
harness_ledger_append_errors_total	counter	Failed run ledger writes and syncs, retries included. Non-zero means the ledger's disk is failing.
harness_ledger_bytes	gauge	Total size of the run ledger's day files.
harness_run_feed_dropped_total{subscriber}	counter	Run records a feed subscriber missed because it fell behind. subscriber="metrics" above zero means the run counters above read low.
harness_scheduled_next_run_timestamp{harness}	gauge	Scheduled harnesses only. Absent when there is no next window.
harness_template_render_failures_total{harness,reason}	counter	command harnesses only, both reasons starting at zero. unresolved: a required argv template value was absent, so the run was recorded skipped (template_unresolved) or the start failed. grammar: a template did not parse at spawn. Nothing was exec'd either way.
harness_trigger_source_up{source,kind}	gauge	Every declared [channel.*] and [webhook.*] source. 1 while a channel is connected or a webhook source is listening; 0 in every other state (connecting, backoff, error, no_listener, disabled, unbound).
harness_trigger_events_total{source,outcome}	counter	fired, ignored, duplicate, unauthorized, too_large, rate_limited, invalid. Every declared source reports all seven, starting at zero. See below.
harness_trigger_last_event_timestamp{source}	gauge	Unix seconds of the source's latest firing. Absent until it first fires in this daemon's lifetime.
harness_trigger_reconnects_total{source}	counter	Channel sources only. Times the stream came back to connected after the first connection.
harness_metrics_collection_errors_total{collector}	counter	supervisor, schedule, observer, lifecycle, triggers, runs. observer and lifecycle also count events the collector lost because it fell behind, so the matching counters read low; runs counts a run feed that closed while the daemon ran.
harness_metrics_harnesses_overflowed	gauge	How many harnesses were folded into __other__.
harness_notify_deliveries_total{event,result}	counter	Runs of the [notify] hook: ok, error, timeout, and notifications that never ran it, dropped (queue full) and suppressed (inside the cooldown).
harness_observer_*	mixed	Health of the transcript reader: delivered and dropped events, ambiguous and unattributed items, parse errors, scan errors, sessions tracked.
go_goroutines	gauge	The daemon's goroutines. It returns to a baseline between runs; a count that stays up by a fixed step per spawn is a leak. See Daemon memory.
go_memory_classes_heap_objects_bytes	gauge	Live heap: memory held by objects the program can still reach, plus garbage not yet swept. This is what a leak grows.
go_memory_classes_heap_free_bytes, go_memory_classes_heap_released_bytes	gauge	Heap the GC has freed. free is still mapped; released has been returned to the OS. Growth here is GC headroom, not a leak.
go_memory_classes_*_bytes	gauge	The rest of the runtime's breakdown: stacks, metadata, profiling buckets, other. go_memory_classes_total_bytes is all of it, which is close to what the OS sees as the daemon's Go memory.
go_gc_heap_goal_bytes	gauge	The heap size the GC is steering for. With the default GOGC=100 it sits near twice the live heap. Under [daemon] memory_limit it flattens below the limit.
go_gc_gomemlimit_bytes	gauge	The soft memory limit in effect, from memory_limit or GOMEMLIMIT. 9.223372036854776e+18 (MaxInt64) means none.
go_memstats_heap_objects	gauge	Allocated heap objects. It rises with a leak, in step with spawns.
other go_*, process_*		The rest of the Go collector's defaults (GC pauses, go_memstats_*, threads) and the process collector (process_resident_memory_bytes, CPU, file descriptors).

What "running" means

The supervisor has seven states and the metric has four:

Supervisor state	state label
failed	failed
held by its operating_hours (any state other than failed)	stopped
degraded, or any state while the crash-loop flag is set	flapping
running, starting	running
stopped, stopping, restarting (not in a crash loop)	stopped

A harness held outside its operating hours is down on purpose, so it reads stopped, never failed or flapping. The metric does not tell a held harness apart from one an operator stopped.

Where the model-call numbers come from

The daemon reads the transcript each agent writes: Claude Code's JSONL, Crush's SQLite store, Codex's session files, or Pi's session files. It counts a tool call as a successful model call, because the model answered with work. It counts an error mark as a failed one. A turn that ends in plain text with no tool call is not counted.

Only a harness whose adapter writes a readable transcript (claude-code, crush, codex, pi, omp, opencode, or a command harness bound to one of those with transcripts) and that has a workdir gets model-call series. A generic harness, an unbound command harness, or an agent harness with no workdir, has none. The daemon omits values it cannot compute rather than reporting a zero, which would look like a healthy, idle agent.

Crush records provider errors in its transcript as a failed turn. Claude Code records them as flagged API-error records (rate_limit (429): …, server_error: …), which the daemon reads and classifies. Codex successes are counted, but its provider errors do not appear yet.

Until an adapter's errors do appear, its harnesses have no error-side series: harness_model_calls_total{outcome="error"}, harness_model_call_errors_total and harness_model_call_errors_unclassified_total are absent for codex, pi, omp and opencode harnesses today, not zero. A zero would say "no quota errors" through the very outage the quota alert exists to catch. For those harnesses, the staleness alert on harness_last_successful_call_timestamp is the one that fires.

Error classes

The daemon classifies each error when it reads it, so alert rules never need to match provider wording:

• quota: rate limits, spent weekly or monthly allowances, credit balances (429, 402, rate_limit_error, insufficient_quota, "usage limit"). Restarting does not fix these.
• auth: missing, invalid or expired credentials (401, 403, "invalid api key", authentication_error).
• timeout: the provider accepted the request but took too long (408, 504, "deadline exceeded").
• transport: the provider could not be reached or could not serve the request: refused or reset connections, DNS and TLS errors, timeouts while connecting (dial tcp …: i/o timeout, a TLS handshake timeout), 500, 502, 503, and provider overload (529 overloaded_error). Overload counts as transport, not quota. It reflects the provider's capacity, has no reset time, and clears on its own.
• other: everything else. This includes context-window rejections. The daemon recognises those (its session guard rotates a Crush session that is wedged on them), so they are counted as other but never as unclassified.

If the unclassified counter starts rising, a provider has probably changed its error wording.

Trigger outcomes

Each webhook delivery to a served route, and each channel doorbell, is counted once, under how it ended:

outcome	Meaning
fired	Reached the bound harnesses. Each harness then starts, queues or skips the run, and that decision is in its run history, not here.
ignored	A webhook event not in the source's events list, or an event for a source a reload had just unbound. Answered 202, nothing runs.
duplicate	A delivery ID that already fired on the route in the last 24 hours.
unauthorized	Failed verification: 401.
too_large	Body over max_body: 413.
rate_limited	Over the route's rate_limit: 429.
invalid	A channel message that is not a well-formed doorbell, or a webhook body the sender stopped sending partway.

A request to a route that is not served (unknown, disabled or unbound) is a 404 and is not counted anywhere, so a stranger probing /hooks/ cannot create series. A 405 and a 503 (the listener at its concurrency limit) are not counted either: they say something about the request or the listener, not about the source.

Cardinality

The harness label is capped at 50 distinct values. Harnesses beyond the cap share harness="__other__", where counters add up and harness_harness_state counts how many overflow harnesses are in each state. When a harness is removed, its slot is freed. Session IDs, prompts, model names, credentials and environment values never appear as labels.

harness_agent_program_calls_total's program label is capped at 64 distinct names per harness, and only names shaped like a lowercase executable are kept, so an env assignment, an argument or a path is counted as program="other" rather than becoming a label. Overflow folds the same way. A query for one program is still exact; sum by (program) just shows other as the long tail.

The source label takes only the names of sources your config declares, so it has no cap of its own. A source that a reload removes or renames disappears from the next scrape; the new name starts from zero. Delivery IDs, event names, payloads and doorbell text never appear as labels.