Inside the run engine: how agents survive retries, timeouts, and restarts

An agent that works on your laptop is a demo. An agent that keeps working through network errors, rate limits, and deploys is a product. This is how we built the second kind.

Every step is a checkpoint

Each step of a run is recorded before and after it executes. If anything fails, the run resumes from the last completed step instead of starting again.

const result = await step('score-lead', async () => {
	const company = await crm.getCompany(lead.companyId);
	return scoreAgainstProfile(company, profile);
});

Idempotent by design

Retries are only safe if repeating a step has no extra effect. Every tool call carries an idempotency key, so sending an email twice never sends two emails.

await gmail.send(draft, {
	// Same key on a retry: the provider returns the first result instead of sending again.
	idempotencyKey: `${run.id}:${step.id}:send-intro`,
});

Timeouts with a plan

Each step has a timeout and a fallback. When an app is slow, the step waits and retries with backoff; when it’s down, the run pauses and tells someone.

Failure	What the engine does	Who is told
Slow response	Retries with backoff, up to 5 times	Nobody
Rate limited	Waits for the limit to reset	Nobody
App down	Pauses the run	The run’s owner
Bad data	Stops and explains why	The run’s owner