Grant Automation NIH · NSF · DoD

Retry and rate-limit strategies for overnight RFP batches

An overnight batch of 300 Requests for Proposals (RFPs) fails in a way that a single-document run never reveals: one transient timeout is noise, but 300 documents hammering the same National Institutes of Health (NIH) or Grants.gov endpoint at once trip rate limits, and a naive retry loop answers each 429 by immediately retrying — amplifying the stampede until the whole run collapses and a research administrator arrives to an empty results table and no idea which documents processed. This page builds the failure-handling layer that makes a 300-document run finish deterministically, as the resilience companion to async batch processing for large RFPs. Where the asyncio patterns reference shapes the concurrency, this one hardens it: exponential backoff with jitter, a bounded semaphore, a per-run retry budget, a circuit breaker, and a dead-letter queue so a single poison document can never sink the batch.

Phase 1 — Classify failures before you retry them

The first discipline is refusing to retry blindly. Retrying a permanent failure — a malformed PDF, a 400 from a rejected payload, a schema violation — wastes the retry budget and delays the documents that could have succeeded. Retrying a transient failure — a 429 rate-limit response, a 503, a connection timeout — is exactly right, but only with back-pressure. Every failure must be classified into one of those two buckets before any retry decision, mirroring the routed-review discipline the compliance validation rule engines apply to bad data.

Implementation steps:

  1. Enumerate transient signals. HTTP 429, 500, 502, 503, 504, and connection or read timeouts are retryable. They describe the server’s momentary state, not the request’s validity.
  2. Enumerate permanent signals. HTTP 400, 401, 403, 404, 422, and any local parse or validation exception are not retryable. Retrying cannot change the outcome.
  3. Default to permanent. An unrecognized error is treated as non-retryable and sent to the dead-letter queue for triage. Silently retrying the unknown is how a batch loops forever.
  4. Honor Retry-After. When a 429 or 503 carries a Retry-After header, that value overrides the computed backoff — the server has told you exactly how long to wait.
python
from dataclasses import dataclass

TRANSIENT_STATUS = frozenset({429, 500, 502, 503, 504})

@dataclass(slots=True)
class FetchError(Exception):
    status: int | None       # HTTP status, or None for a client-side timeout
    retry_after: float | None = None

    @property
    def is_transient(self) -> bool:
        if self.status is None:          # timeout / connection reset
            return True
        return self.status in TRANSIENT_STATUS

Pin the environment so an overnight run reproduces and the concurrency ceiling is explicit:

bash
python -m venv .venv
source .venv/bin/activate
pip install "aiohttp>=3.9" "tenacity>=8.2"   # tenacity optional; the loop below is self-contained

Phase 2 — Backoff with jitter, bounded concurrency, and a retry budget

The core of a deterministic batch is three mechanisms working together. A Semaphore caps how many documents are in flight so the run never exceeds the portal’s rate ceiling. Exponential backoff with jitter spreads retries across time so 300 documents that all fail at second zero do not all retry at second one — the classic thundering-herd collapse. And a per-run retry budget bounds total wasted work, so the batch cannot spend the whole night retrying a handful of doomed documents. Idempotency ties it together: each retry must be safe to repeat, keyed on a stable document identifier so a duplicate never double-submits to Grants.gov.

python
import asyncio
import random

async def backoff_sleep(attempt: int, base: float = 0.5, cap: float = 30.0,
                        retry_after: float | None = None) -> None:
    """Sleep with full-jitter exponential backoff, capped, honoring Retry-After."""
    if retry_after is not None:
        await asyncio.sleep(retry_after)             # server's explicit instruction wins
        return
    ceiling = min(cap, base * (2 ** attempt))        # 0.5, 1, 2, 4, ... capped at 30s
    await asyncio.sleep(random.uniform(0, ceiling))  # full jitter: spread the herd

async def process_one(doc_id: str, *, sem: asyncio.Semaphore, budget: "RetryBudget",
                      max_attempts: int = 5) -> "Result":
    """Fetch and parse one RFP with bounded concurrency and a shared retry budget."""
    async with sem:                                  # never exceed max concurrency
        for attempt in range(max_attempts):
            try:
                return await fetch_and_parse(doc_id)  # idempotent: keyed on doc_id
            except FetchError as err:
                if not err.is_transient:
                    raise                             # permanent -> straight to dead-letter
                if attempt == max_attempts - 1 or not budget.spend():
                    raise RetryExhausted(doc_id)      # out of attempts or out of budget
                await backoff_sleep(attempt, retry_after=err.retry_after)
    raise RetryExhausted(doc_id)                      # unreachable, satisfies the type checker

The retry budget is a single shared counter so no one document can monopolize the run’s tolerance for failure:

python
class RetryBudget:
    """A run-wide ceiling on retries, shared across all in-flight documents."""
    def __init__(self, total: int) -> None:
        self._remaining = total          # e.g. 3 * document_count

    def spend(self) -> bool:
        if self._remaining <= 0:
            return False                 # budget exhausted: stop retrying, dead-letter
        self._remaining -= 1
        return True

The diagram below traces one document through the loop: bounded admission, the attempt, the transient-versus-permanent classification, the budget check, and the two terminal sinks — a completed result, or the dead-letter queue that isolates a poison document from the rest of the batch.

Retry, backoff, and dead-letter flow for one batched document A semaphore admits at most N concurrent documents. Each admitted document is attempted. A decision asks whether it succeeded; if yes, the result is emitted and the document is marked done. If no, a decision asks whether the failure is transient, such as a 429, 503, or timeout; a permanent failure is sent left to the dead-letter queue. A transient failure reaches a retry-budget decision; if the budget is exhausted the document is also sent to the dead-letter queue, and if budget remains the document sleeps for an exponential backoff with jitter and then loops back up to be attempted again. Semaphore: admit ≤ N tasks Attempt call fetch · parse · submit Success? Emit result mark done Transient? 429 · 503 · timeout Retry budget left? Dead-letter queue held for triage Sleep base·2ⁿ + jitter yes no permanent transient exhausted retry retry after backoff
Only transient failures with remaining budget loop back; everything else terminates deterministically at a result or the dead-letter queue.

Phase 3 — Circuit breakers, side effects, and the deadline cutoff

Backoff and budget handle the common case. Three harder edge cases decide whether the batch is truly deterministic.

A circuit breaker stops flogging a dead endpoint. When a portal is genuinely down, every document will exhaust its retries in sequence, wasting the whole night. A breaker tracks the recent failure rate against one endpoint and, past a threshold, opens — failing fast and dead-lettering new work for that endpoint without attempting it — then half-opens after a cooldown to test recovery with a single probe. This is the same status-aware polling discipline used when polling eRA Commons for submission status updates.

Non-idempotent side effects must never be retried blindly. Fetching and parsing is safe to repeat; submitting to Grants.gov is not. A retry that re-submits creates a duplicate application. Guard any state-changing call with an idempotency key derived from the document identifier and submission cycle, and check for a prior success before re-issuing.

The deadline cutoff is non-negotiable. An overnight run has a hard wall — the batch must finish before staff arrive. Enforce a wall-clock cutoff: past it, in-flight retries stop, remaining work is dead-lettered as deadline-cut, and the run reports partial completion rather than bleeding into the morning.

Edge case Symptom in a 300-doc run Mitigation
Thundering herd All retries fire at the same second, re-tripping 429s Full-jitter backoff spreads retries across the window
Endpoint outage Every document exhausts retries in sequence Circuit breaker opens, fails fast, half-opens on cooldown
Duplicate submission Retry re-submits to Grants.gov Idempotency key per doc + cycle; check prior success
Portal rate limit Sustained 429s under high concurrency Lower Semaphore ceiling; honor Retry-After
Deadline overrun Run bleeds past the morning cutoff Wall-clock cutoff dead-letters remaining work

Phase 4 — Verifying determinism with a flaky mock

A resilience layer is only trustworthy if its failure paths are tested, and you cannot test failure paths against a live agency portal. Drive the loop with a deterministic flaky mock that fails a fixed number of times before succeeding, plus one that never succeeds, and assert on the terminal state and the retry accounting — not on timing, which jitter deliberately randomizes.

python
import pytest

class FlakyEndpoint:
    """Fails `fail_times` with a transient error, then succeeds. Deterministic."""
    def __init__(self, fail_times: int, status: int = 503) -> None:
        self.fail_times, self.status, self.calls = fail_times, status, 0

    async def __call__(self, doc_id: str) -> str:
        self.calls += 1
        if self.calls <= self.fail_times:
            raise FetchError(status=self.status)
        return f"parsed:{doc_id}"

@pytest.mark.asyncio
async def test_transient_failures_recover_within_budget(monkeypatch) -> None:
    monkeypatch.setattr("mod.asyncio.sleep", _no_sleep)   # collapse backoff for speed
    endpoint = FlakyEndpoint(fail_times=2)
    monkeypatch.setattr("mod.fetch_and_parse", endpoint)
    sem, budget = asyncio.Semaphore(4), RetryBudget(total=10)
    result = await process_one("NIH-R01-001", sem=sem, budget=budget)
    assert result == "parsed:NIH-R01-001"
    assert endpoint.calls == 3                             # 2 failures + 1 success

@pytest.mark.asyncio
async def test_poison_document_is_dead_lettered(monkeypatch) -> None:
    monkeypatch.setattr("mod.asyncio.sleep", _no_sleep)
    monkeypatch.setattr("mod.fetch_and_parse", FlakyEndpoint(fail_times=99))
    with pytest.raises(RetryExhausted):                   # never sinks the batch
        await process_one("DoD-BAA-777", sem=asyncio.Semaphore(4),
                          budget=RetryBudget(total=3))

@pytest.mark.asyncio
async def test_permanent_error_is_not_retried(monkeypatch) -> None:
    async def hard_400(doc_id: str) -> str:
        raise FetchError(status=400)
    monkeypatch.setattr("mod.fetch_and_parse", hard_400)
    with pytest.raises(FetchError):
        await process_one("NSF-CAREER-42", sem=asyncio.Semaphore(4),
                          budget=RetryBudget(total=10))

Round out the tests with a manual checklist: confirm the semaphore ceiling is never exceeded under load, that Retry-After overrides the computed backoff, that the dead-letter queue captures every terminal failure with its reason (permanent, exhausted, deadline-cut), that no document is submitted twice, and that a completed run emits a reconciliation summary — succeeded, dead-lettered, and cut — so the morning’s first task is triage, not forensics.

Frequently asked questions

Why add jitter instead of plain exponential backoff?

Plain exponential backoff makes 300 documents that failed at the same instant all retry at the same later instant — second 1, then second 2, then second 4 — which re-trips the very rate limit that caused the failure. Full jitter picks a random sleep between zero and the exponential ceiling, spreading the retries across the whole window so the load smooths out instead of arriving in synchronized waves. It is the single most important line in a batch that shares one endpoint.

What belongs in the dead-letter queue versus a retry?

Retry only transient failures — 429, 5xx, and timeouts — because they describe the server’s momentary state. Everything else goes to the dead-letter queue: permanent failures like a 400 or a parse error that retrying cannot fix, documents that exhausted their retry budget, and work stopped by the deadline cutoff. Each entry carries its reason so triage the next morning knows whether to re-run it, fix the source document, or escalate.

How does a retry budget differ from per-document max attempts?

Per-document max_attempts bounds how hard the loop tries one document; the retry budget is a single counter shared across the whole run. Without the shared budget, a dozen doomed documents can each burn their full attempt allowance, and their backoff sleeps push the batch past its deadline. The budget caps total wasted work so a wave of poison documents cannot consume the night that the healthy 290 needed.

Why is idempotency critical for the submission step?

Fetching and parsing are safe to repeat, but submitting to Grants.gov changes state, and a blind retry after a timeout can create a duplicate application even when the first submission actually succeeded. Derive an idempotency key from the document identifier and submission cycle, record it on success, and check it before re-issuing, so a retry that follows an ambiguous timeout confirms the prior outcome instead of duplicating it.

When should the circuit breaker open?

When the recent failure rate against one endpoint crosses a threshold — enough consecutive or windowed failures to conclude the portal itself is down rather than one document being unlucky. Once open, the breaker fails fast and dead-letters new work for that endpoint without attempting it, sparing the retry budget, then half-opens after a cooldown to test recovery with a single probe before resuming normal flow.

Up one level: Async Batch Processing for Large RFPs

Related pages