External Calls Fail Sometimes: Exponential Backoff and a Retry Budget

External and service calls fail intermittently: build retries in from the start. But don’t retry just anything. First classify the operation: idempotent (safe) / safe-with-key / non-idempotent (risky).

Why it matters

Without retries, a momentary 429 fails the whole job. Retry a non-idempotent write blindly and the same payment or the same email goes out twice. Both are expensive.

Transient failures only, with backoff

Retry on: 408, 429, 500, 502, 503, 504, and network timeouts. Everything else (400, 401, 403) will fail the same way, so stop immediately.

function withBackoff(fn, { max = 5 } = {}) {
  let wait = 400;
  for (let attempt = 1; ; attempt++) {
    try {
      return fn();
    } catch (err) {
      if (attempt >= max || !isTransient(err)) throw err;
      Utilities.sleep(wait + Math.floor(Math.random() * 200)); // jitter
      wait = Math.min(wait * 2, 8000); // exponential, capped
    }
  }
}

• Add jitter so multiple runs don’t retry at the same instant.
• Keep max elapsed time under the 6-minute limit. If it might exceed, save a cursor and continue on the next run.

A retry budget: per run, not per call

max above only caps a single call. The real failure is several calls each retrying five times and burning the whole 6 minutes — leaving the rows you actually needed to process untouched. So budget retries per run, not per call: spend down a shared pool of remaining time and remaining attempts.

// one budget shared by the whole run, so no single endpoint eats all 6 minutes
const budget = { until: Date.now() + 5 * 60 * 1000, left: 30 };
const canRetry = () => Date.now() < budget.until && budget.left-- > 0;
// replace the `attempt >= max` check inside withBackoff with `!canRetry()`.

When the budget runs out, stop, save a cursor, and continue on the next run under the 6-minute ceiling. A single run not finishing is normal — pretending it finished while only doing half is the accident.

Non-idempotent writes need an idempotency key

// process once even if the same request arrives twice
if (props.getProperty("done:" + idemKey)) return cached;
const result = doWrite();
props.setProperty("done:" + idemKey, "1");

The idempotency marker lives in PropertiesService, so keep keys short and sweep them periodically to stay under the Properties and Cache limits. And since two runs can pass the check at the same instant, wrap money-moving writes in LockService as well.

Deeper: normalize partial failure

Batch calls with fetchAll and some may fail. Normalize results as { ok, status, body, error }, and instead of failing everything, proceed with the successes and retry only the failures. Log a correlation id, not the raw payload.

One line to keep: retry only transient failures with backoff + jitter, and guard non-idempotent writes with an idempotency key.