# Submission Portal Sync

The last mile of grant automation is also the least forgiving: an assembled proposal package that passed every internal check is worthless until a federal portal has accepted it, timestamped it, and returned a tracking identifier that survives a deadline dispute. Submission portal sync is the discipline of driving that hand-off programmatically — pushing a package into Grants.gov, the National Science Foundation (NSF) Research.gov service, or a Department of Defense (DoD) system, then watching the submission move through the agency’s validation states and pulling back receipts and errors before a correction window closes. It is the stage where the automation stops producing documents and starts producing legally consequential events, and it sits at the outer edge of the RFP ingestion and parsing workflow as the mirror image of acquisition: where ingestion pulls solicitations in, sync pushes applications out under the same audit and idempotency discipline.

“Sync” here means three distinct operations that beginners often collapse into one. First, submit: transmit the application package and receive an agency-issued tracking number that proves the moment of receipt. Second, track: poll the portal for status transitions as the package advances from received, through validation, to agency retrieval. Third, reconcile: retrieve the receipts, validation errors, and warnings the agency emits, and route them back into the pipeline so a rejected package can be corrected and resubmitted before the deadline. Treating submission as fire-and-forget is the single most expensive mistake in the whole lifecycle, because a package that silently failed validation at 4:55 p.m. is indistinguishable from a successful one until the tracking status is read — and by then the window has closed.

# Prerequisites and environment setup

The client patterns on this page assume Python 3.10 or newer, because they use modern type-hint syntax (list[str], str | None) and Pydantic v2 models throughout. Two libraries carry the work: requests for a plain REST transport and zeep for the SOAP web services that Grants.gov System-to-System (S2S) still exposes. Pin them so a submission attempted today reproduces byte-for-byte when it is retried after a portal outage:

python -m venv .venv
source .venv/bin/activate
pip install "requests>=2.31" "zeep>=4.2" "pydantic>=2.6" "tenacity>=8.2"

Beyond libraries, portal sync has hard institutional prerequisites that no amount of code can substitute for, and every one of them is an out-of-band registration that must be completed weeks ahead:

• SAM.gov registration. The submitting organization must hold an active registration in the System for Award Management (SAM.gov) with a current Unique Entity Identifier (UEI). An expired SAM.gov registration silently blocks submission across every federal portal, so the pipeline should treat registration expiry as a first-class pre-flight check, not a runtime surprise.
• Authorized Organization Representative (AOR). Only a person holding the AOR role, delegated through the organization’s E-Business Point of Contact, may submit on the institution’s behalf. The credentials the client uses are institutional, not personal, and they carry the legal authority to bind the organization.
• eRA Commons and agency accounts. National Institutes of Health (NIH) submissions route through Grants.gov but are tracked in eRA Commons, which requires linked Commons IDs for the principal investigator and the signing official. NSF submissions require a registered Research.gov account tied to the organization.
• A validated package. The bytes you submit must already be an assembled, agency-conformant package — the PDF package generation stage produces the attachments, and the institutional approval routing stage certifies that the signing official approved the submission. Sync never assembles; it only transmits what assembly and approval have already blessed.

Store all credentials in a secrets manager and inject them at runtime. A tracking identifier and an AOR credential are the two things an auditor will ask for first, so they must never live in source control.

# Core mechanism — the submit-and-status web-service model

Every federal portal, whatever its transport, exposes the same conceptual pair of operations: a submit call that accepts a packaged application and returns a receipt, and a status call that accepts a tracking identifier and returns the current state. Grants.gov S2S historically models these as SOAP operations — a submission operation that consumes a Grants.gov application XML document with its attachments encoded as MTOM (Message Transmission Optimization Mechanism) parts, and a status operation keyed by the returned Grants.gov tracking number. NSF’s Research.gov and the DoD systems wrap the same idea in their own request and response shapes. The engineering goal is to hide those differences behind a single typed client interface so the rest of the pipeline never learns which portal it is talking to.

The Grants.gov application XML is not something to hand-build here. It is the same structured form data produced by the agency form population stage — the SF-424 (R&R) cover components, budget forms, and their attachment slots — serialized to the agency’s submission schema. Portal sync consumes that XML, wraps it in the transport envelope the portal expects, attaches the PDFs, and transmits. Keeping the XML construction upstream means the sync layer stays a thin, well-tested transport rather than a second place where form logic can drift.

The typed client below sketches the shared interface. Endpoint locations are supplied entirely through injected configuration — the code names no host — so the same client drives a live portal, a staging portal, or a mock without a code change:

from typing import Protocol
from pydantic import BaseModel

class PortalClient(Protocol):
    """The single interface every portal transport implements.

    Endpoint URLs are never literals here; they arrive via configuration so
    the same code path serves production, staging, and a mocked portal.
    """
    def submit(self, envelope: "SubmissionEnvelope") -> "SubmissionReceipt": ...
    def status(self, tracking_id: str) -> "StatusResponse": ...

class PortalConfig(BaseModel):
    """Per-agency transport settings resolved from a secrets/config store."""
    agency: str                 # "NIH" | "NSF" | "DoD"
    submit_endpoint: str        # injected, never hard-coded in prose or code
    status_endpoint: str
    transport: str              # "soap" | "rest"
    verify_tls: bool = True

With this shape in place, submission becomes a matter of building an envelope, transmitting it, capturing the tracking identifier, and then polling status until the package reaches a terminal state. The two deepest guides in this section walk the two halves end to end: submitting to Grants.gov with the System-to-System API covers the submit call and its edge cases, while polling eRA Commons for submission status updates covers the status lifecycle.

# Idempotency-aware implementation

The defining hazard of portal sync is the duplicate submission. A network timeout on the submit call leaves the client uncertain whether the package landed; a naive retry then transmits the same application twice, and two tracking numbers for one proposal is a compliance incident that a program officer must untangle by hand. The production pattern therefore attaches a client-generated idempotency key derived from the package content hash, records the tracking identifier the instant it is returned, and never re-submits a package whose key is already associated with a tracking number.

The typed models below define the submission envelope and the status response as Pydantic v2 models, with a validator that guarantees the idempotency key and the package hash are present before a submit is ever attempted:

from datetime import datetime
from enum import Enum
from pydantic import BaseModel, Field, field_validator

class PortalStatus(str, Enum):
    """Normalized status vocabulary across portals."""
    RECEIVED = "received"
    VALIDATED = "validated"
    AGENCY_RETRIEVED = "agency_retrieved"
    REJECTED = "rejected"
    ERROR = "error"

class SubmissionEnvelope(BaseModel):
    """Everything needed to transmit one application exactly once."""
    opportunity_id: str = Field(pattern=r"^[A-Z0-9-]+$")
    agency: str
    package_sha256: str = Field(min_length=64, max_length=64)  # content hash
    idempotency_key: str                                       # derived from the hash
    application_xml: str                                       # from agency form population
    attachment_paths: list[str] = Field(default_factory=list)  # the PDF package
    aor_credential_ref: str                                    # secrets-manager handle

    @field_validator("idempotency_key")
    @classmethod
    def _key_binds_to_hash(cls, v: str, info) -> str:
        # The key must be reproducible from the package, so a retry of the
        # SAME bytes computes the SAME key and is recognized as a duplicate.
        pkg_hash = info.data.get("package_sha256", "")
        if pkg_hash and not v.startswith(pkg_hash[:16]):
            raise ValueError("idempotency_key must be derived from package_sha256")
        return v

class SubmissionReceipt(BaseModel):
    tracking_id: str            # agency-issued proof of receipt
    received_at: datetime
    status: PortalStatus = PortalStatus.RECEIVED

class StatusResponse(BaseModel):
    tracking_id: str
    status: PortalStatus
    observed_at: datetime
    errors: list[str] = Field(default_factory=list)
    warnings: list[str] = Field(default_factory=list)

The submit routine wraps the transport in bounded retries with exponential backoff, but — critically — it consults a persistent submission ledger before every attempt so a retry of an already-tracked package returns the stored receipt instead of transmitting again:

from tenacity import retry, stop_after_attempt, wait_exponential

class SyncEngine:
    def __init__(self, client: PortalClient, ledger: "SubmissionLedger") -> None:
        self._client = client
        self._ledger = ledger  # durable map: idempotency_key -> SubmissionReceipt

    def submit_once(self, env: SubmissionEnvelope) -> SubmissionReceipt:
        prior = self._ledger.get(env.idempotency_key)
        if prior is not None:
            return prior              # duplicate guard: never transmit twice
        receipt = self._transmit(env)
        self._ledger.put(env.idempotency_key, receipt)  # record BEFORE returning
        return receipt

    @retry(stop=stop_after_attempt(4),
           wait=wait_exponential(multiplier=2, min=2, max=30))
    def _transmit(self, env: SubmissionEnvelope) -> SubmissionReceipt:
        # Transport-level retries are safe only because the ledger check above
        # guarantees at-most-once *logical* submission per package.
        return self._client.submit(env)

Recording the receipt before returning is what makes the pattern safe across a process crash: if the ledger write succeeds, a restarted worker finds the tracking number and never resubmits; if it fails, the same idempotency key produces the same logical submission on the next run.

# Agency-specific configuration

The three funding bodies expose the same submit-and-status concept through different transports, authentication models, and — most treacherously — different status vocabularies. A poller written against Grants.gov’s states will misread Research.gov’s, so the status mapping is versioned configuration keyed by agency, never a shared constant. The matrix below captures the differences that most often break a portal client.

Two rows deserve emphasis. NIH submissions cross two systems: Grants.gov issues the first tracking number, then eRA Commons issues its own accession and runs a second layer of validation whose errors and warnings are what actually determine whether the application reaches peer review. And the DoD path is gated entirely on SAM.gov — an entity whose registration lapsed cannot submit to eBRAP at all, regardless of the package’s quality. Encoding these as a typed profile keeps the branching explicit rather than scattered through the transport code:

from pydantic import BaseModel

class AgencyPortalProfile(BaseModel):
    agency: str
    transport: str                       # "soap" | "rest"
    status_map: dict[str, PortalStatus]  # portal wording -> normalized status
    terminal_states: tuple[PortalStatus, ...]
    dual_system: bool = False            # NIH: Grants.gov then eRA Commons

PROFILES: dict[str, AgencyPortalProfile] = {
    "NIH": AgencyPortalProfile(
        agency="NIH", transport="soap", dual_system=True,
        status_map={"Received": PortalStatus.RECEIVED,
                    "Validated": PortalStatus.VALIDATED,
                    "AgencyTracking": PortalStatus.AGENCY_RETRIEVED},
        terminal_states=(PortalStatus.AGENCY_RETRIEVED, PortalStatus.REJECTED)),
    "NSF": AgencyPortalProfile(
        agency="NSF", transport="rest",
        status_map={"Submitted": PortalStatus.RECEIVED,
                    "Forwarded": PortalStatus.AGENCY_RETRIEVED},
        terminal_states=(PortalStatus.AGENCY_RETRIEVED, PortalStatus.REJECTED)),
    "DoD": AgencyPortalProfile(
        agency="DoD", transport="rest",
        status_map={"Uploaded": PortalStatus.RECEIVED,
                    "Accepted": PortalStatus.AGENCY_RETRIEVED},
        terminal_states=(PortalStatus.AGENCY_RETRIEVED, PortalStatus.REJECTED)),
}

# Error handling and edge cases

Portal sync fails in ways that are specific to the fact that a deadline is approaching and a third party controls the outcome. Each of the following is a routed outcome, not an unhandled exception, and each must leave an audit trail:

• Rejected package. The portal accepts the transmission but validation reports the package as non-conformant — a missing required attachment, an XML schema mismatch, or a form field the agency rejects. The tracking number still exists; the correct response is to surface the specific error, correct the package upstream, and resubmit under a new idempotency key derived from the corrected bytes. A rejected package is recoverable only if someone reads the status.
• Credential or registration expiry. An expired SAM.gov registration or a lapsed AOR delegation produces an authentication failure that no retry will fix. Detect it distinctly from a transient network error and escalate to a human immediately, because the remediation is an out-of-band renewal that can take days.
• Duplicate submission. A retry after an ambiguous timeout risks a second tracking number. The idempotency ledger above is the primary guard; the secondary guard is a status query by the package hash before any manual resubmission.
• Portal downtime near the deadline. Federal portals experience load-driven slowdowns in the final hour before a common deadline. The client must distinguish a 503-class transient (retry with backoff, keep trying until the deadline) from a 4xx rejection (stop and escalate), and it must record every attempt so that, if the portal was demonstrably unavailable, the institution has the evidence needed to request a late-submission accommodation.

The unifying rule is the same one that governs ingestion: fail loudly, attach the tracking identifier and package hash, and let the orchestrator choose between retry, correction, and escalation. A silent submission failure is the one outcome that must never occur, because it is indistinguishable from success until the window has closed.

# Integration with the downstream pipeline

Portal sync is the terminal transmit stage, and its inputs and outputs bind it tightly to the assembly and audit systems on either side. Its input is the finished PDF package plus the sign-off produced by institutional approval routing — sync must refuse to transmit a package that approval has not certified. Its output is a stream of status events — the initial receipt, each transition, and any errors or warnings — every one of which is written to audit logging and provenance so the institution can later prove exactly when a package was submitted and in what state the agency acknowledged it. At portfolio scale, submissions concentrated around a shared deadline reuse the concurrency and rate-limit discipline documented in async batch processing for large RFPs.

Because the confirm and error paths both terminate in the audit store, a later reviewer can reconstruct the full submission history of any proposal from tracking identifier to final state, which is exactly the evidentiary record a pre- or post-award review demands.

# Testing and verification

Portal clients earn trust against a mocked portal, never against a live agency endpoint — submitting a real package to test the code is both wasteful and, near a deadline, dangerous. Build a fake PortalClient that returns scripted receipts and a scripted sequence of status responses, then assert the two properties that matter most: submission is idempotent, and the tracking identifier is captured and persisted.

import pytest
from datetime import datetime, timezone

class FakePortalClient:
    """Deterministic stand-in for a real portal transport."""
    def __init__(self) -> None:
        self.submit_calls = 0
    def submit(self, env: SubmissionEnvelope) -> SubmissionReceipt:
        self.submit_calls += 1
        return SubmissionReceipt(tracking_id="GRANT-TEST-0001",
                                 received_at=datetime.now(timezone.utc))
    def status(self, tracking_id: str) -> StatusResponse:
        return StatusResponse(tracking_id=tracking_id,
                              status=PortalStatus.AGENCY_RETRIEVED,
                              observed_at=datetime.now(timezone.utc))

def _envelope() -> SubmissionEnvelope:
    pkg = "a" * 64
    return SubmissionEnvelope(opportunity_id="PA-24-001", agency="NIH",
                              package_sha256=pkg, idempotency_key=pkg[:16] + "-1",
                              application_xml="<xml/>", aor_credential_ref="vault://aor")

def test_submit_is_idempotent() -> None:
    client, ledger = FakePortalClient(), InMemoryLedger()
    engine = SyncEngine(client, ledger)
    first = engine.submit_once(_envelope())
    second = engine.submit_once(_envelope())     # same bytes, same key
    assert first.tracking_id == second.tracking_id
    assert client.submit_calls == 1              # transmitted exactly once

def test_tracking_id_is_persisted() -> None:
    ledger = InMemoryLedger()
    SyncEngine(FakePortalClient(), ledger).submit_once(_envelope())
    stored = ledger.get(_envelope().idempotency_key)
    assert stored is not None and stored.tracking_id

Beyond the automated suite, a release checklist catches what mocks cannot: confirm SAM.gov and AOR pre-flight checks run before any transmit, that a 4xx rejection escalates while a 503 retries, that every status event reaches the audit log, and that a deliberately duplicated submission returns the stored receipt rather than a second tracking number. Only when those hold should the client be pointed at a live portal in the final submission window.

Submission portal sync is where the entire automation lifecycle is finally proven or disproven: an assembled, compliant, approved package becomes an accepted federal submission with a defensible timestamp, or it does not. Building the submit-poll-confirm loop as an idempotent, audited, agency-aware transport turns the most stressful hour of a funding cycle into a deterministic, observable process — and gives the institution the tracking evidence it needs when a deadline is ever contested.

# Related

• Submitting to Grants.gov with the System-to-System API — the submit call, package construction, and resubmission edge cases in depth.
• Polling eRA Commons for submission status updates — tracking NIH transitions and surfacing errors before the correction window closes.
• PDF package generation — the assembled attachments that portal sync transmits.
• Institutional approval routing — the sign-off that gates any transmission.
• Audit logging and provenance — where every submission status event is recorded for later proof.

Up one level: RFP Ingestion & Parsing Workflows