Grant Automation NIH · NSF · DoD

Populating SF424 (R&R) forms programmatically

The SF424 (R&R) core form and its companion R&R Budget are the two forms every research-grant application to a Grants.gov opportunity must carry, and they are where automated assembly most often fails silently: a budget period written to a column the template does not have, a Senior/Key Person profile whose eRA Commons ID passes format checks but does not match a registered account, or a checkbox left logically empty because a boolean was written where an export state was required. This guide walks the full mechanics of filling those two forms from a typed model — first inspecting the interactive field tree, then filling it with pypdf and pikepdf, then handling the repeating and conditional structures that trip up naive fillers, then validating the result — as the deep dive under agency form population. The National Institutes of Health (NIH) submits these forms as part of a Grants.gov XML application package, and the same canonical data drives both the fillable-PDF path and the XML path.

Phase 1 — Inspect the field tree and map the names

You cannot fill a form whose field names you have guessed. The SF424 (R&R) is a fillable PDF with an /AcroForm dictionary, and every field carries a fully qualified, often dotted name (/T), a type (/FT), and — for checkboxes and radio groups — a set of export states. The first job is to enumerate that tree from the blank, versioned template and write down the exact names, because they differ by form family and revision and will drift when the form is reissued.

  1. Open the blank template and list every field. Capture the qualified name, the type, and the export states of every button field; these three facts are the entire vocabulary you are allowed to target.
  2. Record the form version. The field names are only valid for one revision of the SF424 (R&R) family; store the version alongside the map so a reissue is a detectable diff, not a mystery rejection.
  3. Draft the canonical-to-field map. Attach each meaningful canonical attribute to a concrete field name, leaving unmapped every field the opportunity does not require.
  4. Note the repeating regions. The R&R Budget lays out budget periods as fixed, numbered field groups; identify how many periods one form page provides before you fill anything.
python
from pypdf import PdfReader

def dump_field_tree(template_path: str) -> dict[str, dict]:
    """Return {qualified_name: {type, states}} for a blank SF424 (R&R) template.

    Guessing field names is the primary source of silent misfiles; this is the
    only sanctioned way to learn them, and it must be re-run when the form
    version changes.
    """
    reader = PdfReader(template_path)
    tree: dict[str, dict] = {}
    for name, meta in (reader.get_fields() or {}).items():
        tree[name] = {
            "type": meta.get("/FT"),          # /Tx text, /Btn button, /Ch choice
            "states": meta.get("/_States_"),  # e.g. ['/Off', '/Yes'] for a checkbox
        }
    return tree

These field names are only valid for one revision of the form family, so re-run this dump whenever the template is reissued. Also set up the environment; pin the libraries so a filled form regenerates identically after a template revision:

bash
python -m venv .venv
source .venv/bin/activate
pip install "pypdf>=4.2" "pikepdf>=8.11" "pydantic>=2.6"

Phase 2 — Build a typed model and fill the fields

With the names known, model the proposal as a typed record and fill the form through the map. The typed model is the source of truth and the validation boundary: it uses Pydantic v2, so a missing title or a negative cost is rejected at construction rather than surfacing as a blank field on a submitted form. The flow is deliberately three-staged — canonical model, field map, filled form — so each piece is testable alone.

Filling the SF424 (R&R) from a canonical model through a field map A left-to-right flow of four stages. A typed canonical Proposal model feeds a per-agency field map that pairs canonical attribute paths with AcroForm field names. The field map feeds a fill-and-flatten stage using pypdf and pikepdf, which produces the filled SF424 core form and R&R Budget. That output feeds a validation stage that reads values back and reconciles budget totals. Each stage is independently testable. Canonical model typed Proposal Pydantic v2 Field map canonical path → field name Fill + flatten pypdf · pikepdf SF424 + R&R Budget Validate read-back + totals reconcile
The canonical model is validated once, the map is pure data, and only the fill-and-flatten stage touches the PDF — keeping each concern testable in isolation.

The fill routine writes text values, coerces booleans to the field’s export state, and sets NeedAppearances so strict viewers regenerate the visible appearance of each field:

python
from pypdf import PdfReader, PdfWriter
from pydantic import BaseModel, Field

class BudgetPeriod(BaseModel):
    period: int = Field(ge=1)
    direct_costs: float = Field(ge=0)
    indirect_costs: float = Field(ge=0)

    @property
    def total(self) -> float:
        return round(self.direct_costs + self.indirect_costs, 2)

def fill_sf424(template: str, out_path: str, text_values: dict[str, str],
               checkbox_states: dict[str, str]) -> None:
    """Fill the SF424 (R&R) core form, then set NeedAppearances.

    text_values / checkbox_states are already keyed by concrete AcroForm field
    names produced from the Phase-1 map; checkbox values must be export states
    (e.g. '/Yes'), never Python booleans.
    """
    reader = PdfReader(template)
    writer = PdfWriter()
    writer.append(reader)
    merged = {**text_values, **checkbox_states}
    for page in writer.pages:
        writer.update_page_form_field_values(page, merged)
    writer.set_need_appearances_writer(True)   # force appearance regeneration
    with open(out_path, "wb") as fh:
        writer.write(fh)

Flattening is a separate, deliberate step run only for the final submission copy: it bakes the filled values into the page content so they can no longer be edited, which is what you want in an archived, transmitted package. pikepdf performs it via the QPDF layer:

python
import pikepdf

def flatten_form(in_path: str, out_path: str) -> None:
    """Flatten AcroForm fields into static page content for submission."""
    with pikepdf.open(in_path) as pdf:
        pdf.flatten_annotations()   # merge field appearances into page streams
        pdf.save(out_path)

Phase 3 — Edge cases and agency-specific overrides

Four structures on these two forms break a filler that assumes flat, one-to-one fields.

  • Repeating budget periods. The R&R Budget provides a fixed number of period columns per form page. A four-year project on a template that holds three periods needs a continuation page — writing period 4 to a Period3-style name overwrites period 3. Detect overflow against the template’s period count and emit a continuation form rather than silently colliding.
  • Senior/Key Person profiles. The Senior/Key Person Profile repeats a person block (name, eRA Commons ID, role) N times; the first block is the PD/PI and the rest are keyed by index. Map the list positionally and validate that the PD/PI block is present and its identity key matches a registered account, since a well-formed-but-unregistered ID passes locally and fails at the agency.
  • Checkbox coding. “Is this a resubmission?” is a button whose on-state is an export token like /Yes, not True. Resolve every boolean to the field’s declared state and treat an unrepresentable value as an error, not a default.
  • PDF versus Grants.gov XML. The same canonical record must also serialize to the Grants.gov XML application package, where each value is a namespaced element rather than an /AcroForm field. Keep the canonical model and the mapping declarative so one record drives both targets.
Structure Naive approach that fails Correct handling
5th budget period, 3-period form Write to Period3 again Overflow to a continuation page
Senior/Key person list Assume a single PI field Positional blocks; PI first, index the rest
Resubmission flag Write True Write the export state /Yes
Grants.gov delivery Fill PDF and upload it Emit the namespaced XML package
Budget totals Trust the entered total field Recompute from line items and reconcile

The delivery split is the one most likely to surprise a first implementation: a filled PDF and a Grants.gov XML package are two renderings of one validated record, and which one an opportunity wants is metadata, not a code branch you should be rewriting per proposal. The XML route is covered end to end in submitting to Grants.gov with the System-to-System API, and the numbers the R&R Budget carries come from standardizing budget justification templates across agencies.

Phase 4 — Validate the filled forms

Validation is a round trip plus a reconciliation. Read every required field back off the written PDF and assert it is non-empty and equal to the source value; then confirm each budget period’s total equals the sum of its line items and that the periods sum to the requested total. Test against a committed blank fixture, never a live agency form.

python
import pytest
from pypdf import PdfReader

TEMPLATE = "tests/fixtures/sf424_rr_blank.pdf"
REQUIRED = {"RR_SF424.ProjectTitle", "RR_SF424.PDPI.LastName"}

def test_required_fields_non_empty(tmp_path) -> None:
    out = tmp_path / "filled.pdf"
    fill_sf424(TEMPLATE, str(out),
               {"RR_SF424.ProjectTitle": "Reproducible Grant Assembly",
                "RR_SF424.PDPI.LastName": "Doe"},
               {"RR_SF424.IsResubmission": "/Off"})
    fields = PdfReader(str(out)).get_fields() or {}
    assert all(fields[name].get("/V") for name in REQUIRED)

def test_value_round_trips(tmp_path) -> None:
    out = tmp_path / "filled.pdf"
    title = "Reproducible Grant Assembly"
    fill_sf424(TEMPLATE, str(out), {"RR_SF424.ProjectTitle": title}, {})
    fields = PdfReader(str(out)).get_fields() or {}
    assert fields["RR_SF424.ProjectTitle"]["/V"] == title  # survived the write

def test_budget_totals_reconcile() -> None:
    periods = [BudgetPeriod(period=1, direct_costs=200000, indirect_costs=124000),
               BudgetPeriod(period=2, direct_costs=210000, indirect_costs=130200)]
    grand_total = sum(p.total for p in periods)
    assert grand_total == 664200.0  # line items reconcile to the requested total

A manual acceptance pass then catches what fixtures cannot: open the filled form in a strict viewer to confirm values render and checkboxes read as checked, confirm the flattened copy is no longer editable, confirm every Senior/Key Person block resolved to the right person, and confirm a drifted field name fails the map check instead of dropping a value. Only then release the forms to agency form population’s downstream packaging.

Frequently asked questions

Why do my filled values show up blank when I open the PDF?

The values were written to the field dictionaries but the viewer did not regenerate the visible appearance streams. Set NeedAppearances on the writer (as fill_sf424 does) so a strict viewer redraws each field, or flatten the form for the final copy so the values become part of the page content and always render.

How do I check a checkbox on the SF424 (R&R)?

Write the field’s export state, not a boolean. Inspect the button in Phase 1 to learn its on-state — commonly /Yes or /1 — and write that token. Writing True, "X", or 1 leaves the box logically empty even if something appears to change, which is a frequent cause of a silently incomplete submission.

What happens when a project has more budget periods than the form provides?

The R&R Budget has a fixed number of period columns per page. Detect that your period count exceeds the template’s and emit a continuation page rather than reusing the last period’s field names, which would overwrite an earlier period. Recompute totals across all pages during validation.

Should I fill the PDF or build the Grants.gov XML package?

Whichever the opportunity requires — and drive both from the same validated canonical record. The PDF path fills /AcroForm fields with pypdf; the XML path emits namespaced elements per form family and version. Because both are just addresses for the same values, keep the mapping declarative so switching targets is configuration, not a rewrite.

Why validate by reading the value back instead of trusting the write?

A round trip catches encoding drops, truncation against a field’s /MaxLen, and appearance problems in one check. Reading the value off the written file and asserting it equals the source value proves the data survived the fill, which a write-only assertion cannot.

Up one level: Agency Form Population