# Agency Form Population

Every federal proposal ships with a stack of government fillable forms that no amount of well-written narrative can substitute for: the SF424 (R&R) cover form, the R&R Budget, the R&R Senior/Key Person Profile, and — for the National Institutes of Health (NIH) — the PHS 398 research plan and cover-page-supplement forms. These forms are not decoration; they are the machine-readable spine of a submission, and a single blank required field, a checkbox encoded as the wrong export value, or a principal investigator name that fails to match the eRA Commons profile will trigger a hard rejection at the receiving system before a reviewer ever opens the science. Populating them by hand across a portfolio of proposals is slow, error-prone, and impossible to audit, which is why form population is a first-class stage of the Document Assembly & Audit Logging workflow rather than a clerical afterthought. This page shows how to drive those forms programmatically from one canonical, typed data model — filling AcroForm PDFs, emitting the structured application package the Grants.gov system consumes, and validating that every mandatory field is present before the package leaves your control.

The core difficulty is that the same piece of data — a project title, a budget period’s fringe amount, a person’s eRA Commons ID — must land in a different named slot for every agency and every form technology. The National Science Foundation (NSF) collects most of what SF424 covers as structured web entry inside Research.gov; the Department of Defense (DoD) attaches its own cover sheets and volume forms to a Broad Agency Announcement (BAA); and Grants.gov itself accepts a single XML application package whose element names track a published form family and version. A form-population layer that hard-codes one agency’s field names is guaranteed to misfile the others. The pattern below separates what the data means from where each agency wants it, so the same validated record populates an AcroForm PDF, a Grants.gov XML package, and a Research.gov structured payload without duplicating a single value.

# Prerequisites and environment setup

The code on this page targets Python 3.10 or newer, because it uses modern type-hint syntax (str | None, list[dict]) and Pydantic v2. Two libraries do the heavy lifting: a PDF field-manipulation library for the AcroForm forms, and an XML library for the Grants.gov package. Pin them so a package assembled today reproduces byte-for-byte when it is regenerated after a form-family revision:

python -m venv .venv
source .venv/bin/activate
pip install "pypdf>=4.2" "pikepdf>=8.11" "lxml>=5.1" "pydantic>=2.6"

pypdf reads and writes AcroForm field dictionaries in pure Python and is the workhorse for filling text and checkbox fields; pikepdf (a binding over the QPDF library) handles the lower-level operations that pypdf cannot, most importantly flattening a filled form so its values are baked into the page content and can no longer be edited. lxml builds and schema-validates the Grants.gov XML package. Two assumptions matter for everything that follows. First, the blank government forms are versioned — a form is identified by its family (for example, the SF424 (R&R) family or PHS 398) and an approved revision, and a package must be built against the exact form version an opportunity requires; treat the blank template and its version as an input recorded in your audit log, not as a constant baked into code. Second, the forms are true fillable PDFs with an interactive AcroForm dictionary, not scanned images — if a “form” arrives flattened or as a scan, there are no fields to populate and it must be routed to review.

# Core mechanism — the AcroForm field model and the XML mapping

A fillable PDF stores its interactive fields in an /AcroForm dictionary at the document root. Each field is an object with a fully qualified name (/T, often dotted for hierarchy, e.g. RR_Budget.Period1.FringeBenefits), a field type (/FT — /Tx for text, /Btn for checkboxes and radio groups, /Ch for choice lists), a value (/V), and, for buttons, a set of appearance states whose export values are what actually get written. The single most common form-population bug is treating a checkbox as a boolean: a checkbox is “on” only when its /V equals one of its defined export states (frequently /Yes, /1, or a form-specific token like /On_1), and writing True or "X" leaves it visually unchecked and logically empty. Reading the field tree first — never guessing field names — is therefore mandatory.

from pypdf import PdfReader

def inspect_fields(template_path: str) -> None:
    """Print every AcroForm field: qualified name, type, and (for buttons) states.

    Run this against a blank agency template BEFORE writing any mapping — field
    names are agency- and version-specific and must never be guessed.
    """
    reader = PdfReader(template_path)
    fields = reader.get_fields() or {}
    for name, meta in fields.items():
        ftype = meta.get("/FT")            # /Tx text, /Btn button, /Ch choice
        states = meta.get("/_States_")     # export values for checkboxes/radios
        print(f"{name!r:<48} {ftype}  states={states}")

The Grants.gov path is structurally the same problem in a different serialization. Instead of a field dictionary, the application package is an XML document: a wrapping submission envelope references one XML fragment per form, and each form’s elements are namespaced to that form family and version. Populating it means emitting elements with the right namespace and local name rather than setting /V on a field object — but the mapping concept is identical, because both technologies reduce to “put this canonical value at this agency-specific address.” That shared shape is what lets one mapping layer serve both, and it is the reason the sections below model the address as data rather than as code.

# Field-mapping implementation

The production pattern defines a canonical, typed model of the proposal once, then attaches a per-agency, per-form map from canonical attributes to concrete field addresses. The canonical model is where validation lives — a Pydantic v2 model rejects a missing project title or a negative budget total at construction, long before any form is touched — mirroring the schema-validation-with-Pydantic discipline the ingestion pipeline uses upstream.

from pydantic import BaseModel, Field, field_validator

class KeyPerson(BaseModel):
    era_commons_id: str
    first_name: str
    last_name: str
    role: str                              # "PD/PI", "Co-Investigator", ...

class BudgetPeriod(BaseModel):
    period: int = Field(ge=1)
    direct_costs: float = Field(ge=0)
    fringe: float = Field(ge=0)
    indirect_costs: float = Field(ge=0)

    @property
    def total(self) -> float:
        return round(self.direct_costs + self.fringe + self.indirect_costs, 2)

class Proposal(BaseModel):
    """Canonical, agency-neutral proposal record — the single source of truth."""
    project_title: str = Field(min_length=1, max_length=200)
    applicant_org: str
    is_resubmission: bool = False
    pd_pi: KeyPerson
    senior_key: list[KeyPerson] = Field(default_factory=list)
    budget: list[BudgetPeriod]

    @field_validator("project_title")
    @classmethod
    def _no_control_chars(cls, v: str) -> str:
        # AcroForm text fields silently drop control characters; reject early.
        if any(ord(c) < 32 for c in v):
            raise ValueError("project_title contains control characters")
        return v

The map itself is plain data — a dictionary keyed by the canonical path, valued by the concrete AcroForm field name (or, for XML, the namespaced element). Keeping it declarative means a form-version bump is a data edit, not a code change, and the same canonical record drives every target:

from typing import Callable
from pypdf import PdfReader, PdfWriter

# Canonical attribute path -> AcroForm field name for one agency/form/version.
SF424_CORE_MAP: dict[str, str] = {
    "project_title":  "RR_SF424.ProjectTitle",
    "applicant_org":  "RR_SF424.Applicant.OrganizationName",
    "pd_pi.last_name":"RR_SF424.PDPI.LastName",
    "pd_pi.first_name":"RR_SF424.PDPI.FirstName",
}

def _resolve(obj: object, dotted: str) -> object:
    for part in dotted.split("."):
        obj = getattr(obj, part)
    return obj

def fill_acroform(template: str, out_path: str,
                  proposal: Proposal, field_map: dict[str, str]) -> None:
    """Fill a blank AcroForm template from the canonical model via a field map."""
    reader = PdfReader(template)
    writer = PdfWriter()
    writer.append(reader)
    values = {field_map[path]: str(_resolve(proposal, path)) for path in field_map}
    for page in writer.pages:
        writer.update_page_form_field_values(page, values)
    # NeedAppearances forces viewers to regenerate field appearances.
    writer.set_need_appearances_writer(True)
    with open(out_path, "wb") as fh:
        writer.write(fh)

Checkbox and radio fields need the export state, not a string, so a small typed coercion sits between the model and update_page_form_field_values. A boolean like is_resubmission must resolve to the field’s declared on-state (/Yes) when true and to /Off when false, and the coercion reads the live states off the field rather than assuming a convention, because different forms encode “on” differently:

def checkbox_value(fields: dict, field_name: str, flag: bool) -> str:
    """Map a boolean onto a checkbox's real export state, never a literal bool."""
    states = fields[field_name].get("/_States_") or ["/Off", "/Yes"]
    off = "/Off" if "/Off" in states else states[0]
    on = next(s for s in states if s != off)   # the non-off state is "checked"
    return on if flag else off

Because the mapping and the state resolution are both data-driven, a second agency is a second dictionary, and a new form version is a diff against the old one — reviewable, versionable, and testable in isolation. Nothing about adding NSF’s Research.gov target or a DoD BAA cover form touches the canonical Proposal model; the model is written once and every agency adapter reads from it.

# Agency-specific configuration

The four form technologies in play are not interchangeable, and the population layer must branch on them explicitly. The table below is the branch table for this stage: it maps how each agency wants the same proposal delivered, which library path applies, and where the field names come from. Encode it as configuration keyed by agency, not as scattered conditionals.

Two rows drive the most rework. The delivery-technology row means NSF is not a PDF-filling problem at all for most components — Research.gov collects the SF424-equivalent data as structured fields, so the NSF adapter maps the canonical model onto an API payload rather than an /AcroForm dictionary. And the person-identity row is where silent rejections concentrate: NIH matches the PD/PI against the eRA Commons profile, so an ID that is merely well-formed but does not correspond to a registered account passes local validation and fails at the agency. Those numbering and identity conventions are decoded upstream in the NIH FOA schema mapping and NSF proposal-guide taxonomy work; form population consumes that decoded metadata rather than re-deriving it.

# Error handling and edge cases

Form population fails in a small set of recurring, diagnosable ways, and a production filler treats each as a routed outcome — a structured error bound to a field name — rather than a stack trace that aborts a batch.

• Missing or misnamed target field. A field name in the map that is absent from the template means the form version changed underneath the map. update_page_form_field_values will silently ignore an unknown key, so verify the map against the live field tree at load time and raise if any mapped name is not present, rather than shipping a form with a value quietly dropped.
• Checkbox and radio encoding. Writing "true", "X", or 1 where the field expects /Yes leaves the box empty. Resolve every button value to one of the field’s declared export states, and treat an unrepresentable value as an error.
• Field overflow and truncation. AcroForm text fields have a /MaxLen and a fixed rendered box; a project title longer than the field silently truncates in some viewers and overflows in others. Enforce the model’s max_length to match the form’s /MaxLen, and reject overflow rather than emitting a form that reads differently than the source record.
• Repeating groups that outrun the form. The R&R Budget has a fixed number of period columns per page; a proposal with more budget periods than the template provides needs a continuation page, not a value written to a non-existent field.
• Appearance not regenerated. A filled value with no appearance stream renders blank in strict viewers. Set NeedAppearances, and for a final submission flatten the form so the values are part of the page content and cannot be altered.

class FieldMappingError(Exception):
    """A mapped canonical path has no corresponding field on the template."""

def validate_map(template: str, field_map: dict[str, str]) -> None:
    """Fail loudly if any mapped field name is absent from the live template."""
    present = set((PdfReader(template).get_fields() or {}).keys())
    missing = [name for name in field_map.values() if name not in present]
    if missing:
        raise FieldMappingError(
            f"{len(missing)} mapped field(s) absent from {template}: {missing[:5]}"
        )

The rule is uniform across every agency adapter: bind every failure to the offending field name, refuse to emit a partially populated form, and let the orchestrator decide between re-mapping, manual review, or a retry against a corrected template. A form that is 99% filled but silently missing one required field is worse than a hard failure, because it clears a superficial “is the file there?” check while guaranteeing an agency rejection.

# Integration with the downstream pipeline

Populated forms are not a deliverable on their own — they are inputs to assembly. A filled, flattened SF424 (R&R) and R&R Budget are exactly the kind of component that the PDF package generation stage merges with the rendered narrative and biosketches into a single PDF/A-compliant submission package. From there, the assembled package (or, on the Grants.gov path, the XML application) is handed to submission portal sync, which transmits it and tracks its status. Every value this stage writes carries a provenance record into the audit logging and provenance stage, so any field on a submitted form can be traced back to the canonical record and the form version it was mapped against.

The contract with the next stage is narrow: form population never decides whether a proposal is compliant — that is the job of the compliance validation rule engines — it only guarantees that every value the canonical model holds has landed in the correct field, in the correct encoding, on the correct form version. Keeping that boundary sharp is what makes the whole assembly auditable: a reviewer can trace any character on a submitted SF424 back to the exact model attribute and mapping row that produced it.

# Testing and verification

Form population earns trust through round-trip tests against committed blank templates, not assertions about live agency systems that revise their forms on their own schedule. The essential guarantee is a round trip: fill a field from the model, read it back off the written PDF, and assert the value survived — this catches encoding bugs, appearance problems, and silent drops in one check. Pair it with a completeness assertion that every field the opportunity marks required is non-empty, and a reconciliation that budget-period totals on the R&R Budget equal the sum of their line items.

import pytest
from pypdf import PdfReader

TEMPLATE = "tests/fixtures/sf424_rr_blank.pdf"

def _sample_proposal() -> Proposal:
    return Proposal(
        project_title="Automated Compliance Validation for Federal Proposals",
        applicant_org="Example University",
        pd_pi=KeyPerson(era_commons_id="JDOE", first_name="Jane",
                        last_name="Doe", role="PD/PI"),
        budget=[BudgetPeriod(period=1, direct_costs=200000,
                             fringe=48000, indirect_costs=124000)],
    )

def test_mapped_fields_exist_on_template() -> None:
    validate_map(TEMPLATE, SF424_CORE_MAP)  # raises if the form version drifted

def test_values_round_trip(tmp_path) -> None:
    out = tmp_path / "filled.pdf"
    prop = _sample_proposal()
    fill_acroform(TEMPLATE, str(out), prop, SF424_CORE_MAP)
    read_back = PdfReader(str(out)).get_fields() or {}
    assert read_back["RR_SF424.ProjectTitle"]["/V"] == prop.project_title

def test_every_required_field_is_populated(tmp_path) -> None:
    out = tmp_path / "filled.pdf"
    fill_acroform(TEMPLATE, str(out), _sample_proposal(), SF424_CORE_MAP)
    fields = PdfReader(str(out)).get_fields() or {}
    required = {"RR_SF424.ProjectTitle", "RR_SF424.Applicant.OrganizationName"}
    assert all(fields[name].get("/V") for name in required)

Beyond the automated suite, a manual acceptance pass catches what fixtures miss: open the filled PDF in a strict viewer and confirm every value renders (not just stores), confirm checkboxes read as checked, confirm the flattened submission copy is no longer editable, and confirm a deliberately drifted field map fails the validate_map gate rather than shipping a form with a dropped value. Only once those pass should the populated forms be released to PDF package generation. The deepest walkthrough of the SF424 (R&R) core form and the R&R Budget — repeating budget periods, senior/key persons, and the PDF-versus-XML fork — is covered in populating SF424 (R&R) forms programmatically.

# Related

• Populating SF424 (R&R) forms programmatically — the deep how-to for the SF424 core form and R&R Budget.
• PDF package generation — where filled forms merge into a PDF/A submission package.
• Submission portal sync — transmitting the assembled package to Grants.gov and tracking status.
• Audit logging and provenance — recording which value landed in which field on which form version.
• Standardizing budget justification templates across agencies — the source of the budget numbers these forms carry.

Up one level: Document Assembly & Audit Logging