Skip to main content
Identify whether you lost observation, a request response, the execution process, or an external operation result. A timeout is not proof of failure; reconnecting is not resubmitting work.

Retry an identical submission

Persist the request before sending. With an initialized client:
Different content under the same Session/key conflicts. Do not generate a new key to bypass idempotency_conflict. An already accepted identical request returns its original record even if a connection was subsequently disabled; this does not permit new work. Bound retry counts and use backoff for transient transport faults. Distinguish authentication, input, and state conflicts. The SDK does not blindly replay non-idempotent operations.

Keep a recovery record before the network call

A recoverable Run submission needs more than a last-seen error. Persist the business request ID, authorized Session ID, exact input and origin, idempotency key, attempt time, returned Run ID when known, and last committed event cursor. Keep these in private application storage: original task content may be sensitive. Record response loss as an unknown submission outcome, not a failed task. Use the SQLite intent example for the persistence boundary and the event checkpoint example for observation recovery. These examples address different records; an event cursor cannot recover a missing submission key, and a submission key does not acknowledge UI delivery. Bounded retries should end in a recoverable application state with preserved identifiers. A retry limit being reached does not establish that remote work failed. Let an operator or later recovery job resume investigation from the same record.

Cancel and close explicitly

Cancellation prevents new Tool dispatch, but dispatched requests can complete or become unknown. Late approvals cannot revive execution. Cancelling a Run and cleaning up a Session are separate; close unused Sessions and confirm deletion. An AbortSignal or stopped Python stream stops observation, not execution. Do not map leaving a page to cancellation unless that is an explicit, communicated product behavior. Cancellation can immediately settle a queued Run as cancelled, while executing work enters cancelling first. Poll the durable Run state until terminal; a successful cancel response alone is not proof that in-flight work has stopped. Explicit Session close can return session_busy while work remains active. Settle or cancel that work, then close and verify cloud cleanup as a separate outcome.

Reconcile external writes

Inspect tool_calls / toolCalls for the invocation and request hash. Verify the actual remote outcome before using resolve_tool_call / resolveToolCall with a result and evidence. Resolution delivers a verified result without reissuing the remote request. Leave unknown outcomes unknown when evidence is insufficient. Management permissions apply; see the reconciliation protocol.

Verify recovery

Exercise stream loss, lost submission responses, worker restart, lost write responses, and lost delete responses separately. Confirm original Run/resource identity and correct side-effect counts. Preserve unresolved identifiers. Simulation proves only its modeled branches; record cloud/model journeys independently. Sources: recovery architecture, durable events. Next: Backend integration.

Define when each recovery attempt is finished

An observation deadline means the application stopped waiting, not that Hexagon cancelled the task. Return a resumable status to your user and keep the recovery record. A cleanup exception must remain visible in that record; do not catch it merely to print “completed” after the business result succeeded. For a concrete sequence, suppose the second turn timed out before returning its Run ID. Reload its saved input/origin/key and resend exactly once per bounded retry policy. Once R2 is returned, store R2 and resume its stream from the committed cursor. If its Tool call later becomes unknown, stop submission retries: recovery now belongs to that call’s external outcome, not to the original Run request. Closing the Session afterward is another operation whose result must be confirmed separately. Sources: submission and cancellation, Runtime supervision, Tool reconciliation, error reference. Next: resource reconciliation, troubleshooting, events and recovery concepts.