Retry an identical submission
Persist the request before sending. With an initialized client:idempotency_conflict. An already accepted identical request returns its original record even if a connection was subsequently disabled; this does not permit new work.
Bound retry counts and use backoff for transient transport faults. Distinguish authentication, input, and state conflicts. The SDK does not blindly replay non-idempotent operations.
Keep a recovery record before the network call
A recoverable Run submission needs more than a last-seen error. Persist the business request ID, authorized Session ID, exact input and origin, idempotency key, attempt time, returned Run ID when known, and last committed event cursor. Keep these in private application storage: original task content may be sensitive. Record response loss as an unknown submission outcome, not a failed task. Use the SQLite intent example for the persistence boundary and the event checkpoint example for observation recovery. These examples address different records; an event cursor cannot recover a missing submission key, and a submission key does not acknowledge UI delivery.
Bounded retries should end in a recoverable application state with preserved identifiers. A retry limit being reached does not establish that remote work failed. Let an operator or later recovery job resume investigation from the same record.
Cancel and close explicitly
Reconcile external writes
Inspecttool_calls / toolCalls for the invocation and request hash. Verify the actual remote outcome before using resolve_tool_call / resolveToolCall with a result and evidence. Resolution delivers a verified result without reissuing the remote request. Leave unknown outcomes unknown when evidence is insufficient. Management permissions apply; see the reconciliation protocol.
Verify recovery
Exercise stream loss, lost submission responses, worker restart, lost write responses, and lost delete responses separately. Confirm original Run/resource identity and correct side-effect counts. Preserve unresolved identifiers. Simulation proves only its modeled branches; record cloud/model journeys independently. Sources: recovery architecture, durable events. Next: Backend integration.Define when each recovery attempt is finished
An observation deadline means the application stopped waiting, not that Hexagon cancelled the task. Return a resumable status to your user and keep the recovery record. A cleanup exception must remain visible in that record; do not catch it merely to print “completed” after the business result succeeded.
For a concrete sequence, suppose the second turn timed out before returning its Run ID. Reload its saved input/origin/key and resend exactly once per bounded retry policy. Once R2 is returned, store R2 and resume its stream from the committed cursor. If its Tool call later becomes unknown, stop submission retries: recovery now belongs to that call’s external outcome, not to the original Run request. Closing the Session afterward is another operation whose result must be confirmed separately.
Sources: submission and cancellation, Runtime supervision, Tool reconciliation, error reference. Next: resource reconciliation, troubleshooting, events and recovery concepts.