Troubleshooting

Most Lorenz failures trace to one of a few causes: the workflow file won't parse, the config fails validation, an eligible issue refuses to dispatch, a reload keeps stale settings, an agent bridge isn't on PATH, or a tracker/worker is unreachable. This page maps each symptom to its cause and fix. It is for operators running the lorenz daemon.

Two tools do most of the work. Run lorenz doctor before you start to catch config and environment problems up front. Watch the dashboards and the event stream while running. The poll.lastError field and the last 20 recentEvents are the only in-memory error surface, so the exact event name in the snapshot is your primary diagnostic.

Many errors below are emitted as runtime events. The full catalog lives in reference/events.md.

The workflow won't load or parse

Lorenz loads WORKFLOW.md before every poll. A load failure at startup aborts the daemon; a load failure on reload keeps the last-good settings (see Reload kept stale settings). The error strings are stable prefixes, so match on them exactly.

Front matter is delimited by a literal --- on the first line and a closing ---. If the first line is not exactly ---, the whole file is treated as the prompt body with empty config, which usually surfaces later as a config validation failure (no tracker.kind), not a parse error.

An empty or whitespace-only Markdown body is not an error: Lorenz falls back to a built-in default prompt template that renders issue.identifier, issue.title, and issue.description.

Prompt variables are snake_case even though config keys are camelCase internally. Use state_type, branch_name, assignee_id, blocked_by, assigned_to_worker, created_at, updated_at. A camelCase variable name throws template_parse_error under strict mode.

Config validation fails

Config validation runs at startup and again as the per-reload dispatch hook. Run lorenz doctor to see the same dispatch_config check without starting the daemon. Error labels are emitted in snake_case (for example tracker.active_states must be a list of strings), even though parsing happens on camelCase keys.

When the same workflow validates here but fails in CI or a different shell, the difference is almost always an unset environment variable used by $VAR or an env fallback. See Secrets won't resolve.

Nothing dispatches

The daemon is polling, candidates come back, but no run starts. An issue is dispatched only when it passes every eligibility gate. Walk the gates in order.

• Wrong active states. Candidates are polled by tracker.active_states (default [Todo, In Progress]). An issue sitting in any other state is never fetched. Move it to an active state or add that state to active_states.
• Terminal state. States in tracker.terminal_states (default [Closed, Cancelled, Canceled, Duplicate, Done]) are treated as done and trigger workspace cleanup, not dispatch.
• Not routed. With tracker.dispatch.only_routes set, an issue must carry a matching route label (prefix tracker.dispatch.route_label_prefix, default Lorenz:). With accept_unrouted: false (default true), an unrouted issue is skipped. Reconciliation also stops an in-flight run whose issue becomes unrouted.
• Blocked. An issue with open blockers is not dispatched, and an in-flight run is stopped during reconciliation with reason blocked. Blockers come from a blockers array or from relations entries whose type is blocks.
• Concurrency caps hit. Dispatch cannot exceed agent.max_concurrent_agents (default 10), or the per-state cap from status_overrides.<state>.agent.max_concurrent_agents. Reserved (in-acquire) slots count toward every cap, so a worker pool mid-acquire can hold capacity. The snapshot shows these as dispatch_skipped events.
• Retry not yet due. A failed run schedules a retry with backoff. The issue is filtered out of eligibility until the retry deadline. Failure backoff is 10000 * 2^(attempt-1) capped at agent.max_retry_backoff_ms (default 300000); a clean exit schedules a continuation retry at roughly 1000ms.
• Worker host has no capacity (pool path). A pool-governed claim that finds no capacity emits dispatch_skipped with reason worker_host_capacity and reschedules without backoff. An acquire error emits dispatch_skipped with worker_pool_acquire_error <msg>.

Blocked dispatches surface in the snapshot blocked lane and in the dashboard. If --dry-run is set the daemon evaluates candidates and emits dry_run events without dispatching; confirm you did not leave it on.

If the poll itself is failing, look for poll_error in the events. A failing validateDispatch throws and aborts the whole poll tick, so no reconciliation or dispatch happens that tick.

Reload kept stale settings

You edited WORKFLOW.md, but behavior did not change.

Hot-reload runs before each poll and is transactional. It only acts when the content stamp (mtime, size, sha256) changed. All throwing side effects run first: the slots-per-machine co-residence gate, then coordinator.reconcile. Only after they all succeed does the runtime swap in the new workflow, settings, and tracker client.

On any reload failure the runtime keeps the last-good settings (no partial apply) and emits workflow_reload_failed with the error message. A successful reload emits workflow_reloaded with the path. So if your edit had a parse or validation error, the daemon keeps running on the previous good config and tells you only through workflow_reload_failed in the event stream.

Agent bridge missing

The ACP executor drives an external bridge subprocess. The built-in bridges are codex-acp (for Codex) and claude-agent-acp (for Claude), and each in turn needs the underlying agent CLI on PATH: codex for codex bridges, claude for claude bridges.

lorenz doctor checks this directly in its agent_bridge and agent_cli checks. The checks run only for executor: acp, and inspect the active config plus every status_overrides state. Remote workers (worker.ssh_hosts non-empty) skip the bridge probe with a warning instead of testing over SSH.

The bridge command is a single shell string split on whitespace. parseAcpAgentOptions rejects unknown option keys, so a typo'd or non-existent key fails config validation rather than being ignored.

Linear rate limits (429)

The Linear client and the linear_graphql tool both retry HTTP 429 honoring Retry-After, with up to 4 retries (base delay 1000ms, max 30000ms). Retry-After is parsed as integer seconds or an HTTP-date; a blank value falls back to exponential backoff. After retries are exhausted the client throws linear api status 429.

Tracker pagination guards

Linear and Jira REST polling use bounded cursor pagination. A normal multi-page poll follows every page until the backend says it is complete. Malformed cursors, repeated cursors, page-count overflows, and item-count overflows fail loudly because continuing would risk an infinite loop or a silent under-read.

SSH worker unreachable

Remote runs execute on hosts in worker.ssh_hosts over SSH. Use LORENZ_SSH_CONFIG to point ssh -F at a specific config file.

Poison classification matches on a fixed set of error prefixes, so ssh_timeout: poisons a worker but a string like invalid_ssh_timeout stays healthy. A run aborted by stall reconciliation is forced to poison regardless of how it resolved.

Workspace bootstrap hook failed

Each issue workspace runs up to four lifecycle hooks via bash -lc: after_create, before_run, after_run, before_remove. Their failure policy differs, which decides whether a hook problem stops the run.

• after_create failure or timeout aborts workspace creation (throws).
• before_run failure aborts the attempt.
• after_run failure is logged and ignored (a stderr update, not a run failure).
• before_remove failure is caught and ignored; cleanup continues.

So only after_create and before_run fail-fast. If a teardown hook is broken, the run still completes and you only see a logged warning.

Finding logs and traces

The live error surface is the in-memory snapshot: poll.lastError plus the last 20 recentEvents. Beyond that there are two on-disk sources.

• Event log file. When logging.log_file is set (default ~/.lorenz/log/lorenz.log), runtime events are appended as JSON. The daemon flag --logs-root <path> redirects this to <path>/log/lorenz.log.
• Per-issue trace. Every AgentUpdate is written as one JSON line to <server.trace_dir>/<urlencoded issueId>/trace.jsonl. The default trace_dir is ~/.lorenz/issues (note: issues, not traces). View a single trace file standalone with pnpm traceviz <file.jsonl>, which serves the dashboard SPA at http://localhost:4040/#/trace/<issueId>.

The live dashboard renders the same traces under the #/trace/<issueId> hash route, streaming over /ws. Several runtime event types (rate_limit, workspace_prepared, session_started, process_exit, stderr, fs_write, approval_required, approval_auto_approved) are parsed then dropped by the trace viewer, so a run can look quiet in the UI while still recording those lines on disk. See observability.md for the full picture.

Querying run history

The lorenz runs command queries the observability HTTP API for completed-run history.

Using lorenz doctor and lorenz-debug

lorenz doctor [workflowPath] validates a workflow and the local environment without starting the daemon. It runs a short-circuiting pipeline of checks with stable ids: workflow_file, workflow_load, dispatch_config, dashboard_assets, log_path, then agent_bridge / agent_cli. If workflow_file or workflow_load errors, doctor returns immediately. Exit status is 1 only when a check reports error; warnings keep exit 0. Missing dashboard assets and an unwritable log path are warnings, not errors.

For a stuck or repeatedly-retrying run, use the lorenz-debug skill, which traces Lorenz and agent logs by issue and session id to explain why a run stalled, retried, or failed. Reach for it when the event stream alone does not explain the behavior. Session ids are composed as <thread_id>-<turn_id>, which is the key for correlating trace lines to a specific turn.

See also

• cli.md - the lorenz, lorenz runs, and lorenz doctor commands and flags.
• dispatch.md - the eligibility and routing rules behind "nothing dispatches".
• observability.md - the dashboards, the trace viewer, and the event stream.
• reference/events.md - the full catalog of runtime event names.
• reference/configuration.md - every config key, default, and alias.