# PDF Package Generation

The last thing a grant pipeline does before a deadline is also the thing most likely to void a submission: fusing the rendered narrative, the filled agency forms, and every attachment into one correctly ordered PDF that a federal portal will accept. A package that assembles in the wrong order, drops a bookmark the reviewer needs, ships a subset font that will not embed, or arrives without the archival conformance an agency mandates is rejected at upload — after every upstream stage has already succeeded. This topic sits at the end of the document assembly and audit section because it is where earlier correctness is either preserved into a submittable artifact or silently lost. Here the concern is deterministic assembly: given a typed manifest of components, produce the same byte-stable merged document every time, with a bookmark tree, stamped page numbers, embedded fonts, and — where the agency requires it — PDF/A archival output.

Package generation is deceptively mechanical. The failure modes are not “the merge crashed” but “the merge succeeded and produced a document that is subtly non-conformant”: a form field that was never flattened, an attachment whose page size differs from the narrative, a font referenced but not embedded, an outline that points at the wrong physical page after a component shifted. Each of those passes a shallow “does it open?” check and fails the agency validator. This page builds the assembly engine that closes those gaps, using pikepdf and pypdf for merge, outline, and metadata work, reportlab for programmatic page generation and stamping, and weasyprint where a component originates as HTML rather than a pre-rendered PDF.

# Prerequisites and environment setup

The code here assumes Python 3.10 or newer, because it uses modern type-hint syntax (list[Component], str | None) and dataclass slots. Pin the assembly toolchain so a package produced today reproduces exactly when it is regenerated after a narrative edit or a form revision — reproducibility is not a nicety here, it is what lets an audit prove which bytes were submitted.

python -m venv .venv
source .venv/bin/activate
pip install "pikepdf==9.4.0" "pypdf==5.1.0" "reportlab==4.2.5" "weasyprint==63.0"

Two assumptions shape everything below. First, the inputs are heterogeneous: the narrative arrives as a PDF from the narrative template rendering stage, the forms arrive as filled PDFs from agency form population, and attachments arrive as whatever a principal investigator uploaded — Word-exported PDFs, scanned letters of support, figure sheets. The assembler must normalize that variety without corrupting any single component. Second, the output contract is not “a PDF” but “a PDF this agency’s portal accepts,” which means the merge, the outline structure, the metadata, and the conformance profile are all agency-parameterized rather than universal constants. pikepdf wraps the QPDF C++ library and gives low-level, loss-preserving access to the PDF object model; pypdf is pure-Python and convenient for outline and page-tree manipulation. Using both, chosen per task, is deliberate — neither alone covers every operation cleanly.

# Core mechanism — merge, outline, and metadata as one object model

A PDF is a tree of indirect objects: a catalog points at a page tree, the page tree holds page objects in physical order, and an optional /Outlines dictionary holds a bookmark hierarchy whose entries carry destinations (/Dest) that resolve to specific pages. Merging is therefore not string concatenation — it is copying page objects from several source documents into one page tree while remapping every cross-reference so nothing dangles. The three things a grant package needs are exactly these three sub-trees kept consistent: the ordered page tree (physical layout), the outline tree (navigable structure reviewers rely on), and the document metadata (title, producer, and — for archival output — an XMP packet).

The mechanism below merges an ordered list of source PDFs and records, as it goes, the starting physical page index of each component. Those indices are what later let the assembler attach a bookmark to the first page of every component, so the outline stays truthful even though each source contributed a different page count.

from pypdf import PdfWriter, PdfReader

def merge_with_component_offsets(
    ordered_pdf_paths: list[str],
) -> tuple[PdfWriter, list[int]]:
    """Merge PDFs in order, returning the writer and each component's start page.

    The returned offsets are 0-based physical page indices: offsets[i] is the
    page in the merged document where source ordered_pdf_paths[i] begins. These
    drive the outline pass, so a bookmark always lands on the real first page of
    its component even after upstream page counts change.
    """
    writer = PdfWriter()
    offsets: list[int] = []
    for path in ordered_pdf_paths:
        offsets.append(len(writer.pages))  # start index BEFORE appending
        reader = PdfReader(path)
        for page in reader.pages:
            writer.add_page(page)          # copies page object into one tree
    return writer, offsets

Recording offsets before appending — not after — is the subtle correctness point: the first component starts at index 0, and each subsequent component starts wherever the running page total then stands. With that map in hand, the outline pass adds one top-level bookmark per component, and metadata is written once over the assembled whole:

from pypdf import PdfWriter

def add_component_outline(
    writer: PdfWriter,
    component_titles: list[str],
    offsets: list[int],
) -> None:
    """Attach one top-level bookmark per component at its start page.

    pypdf resolves the page index to an indirect page reference, so the
    destination survives later saves. Reviewers navigate the assembled PDF by
    this tree; NIH assembly reproduces the same structure via eRA Commons.
    """
    for title, page_index in zip(component_titles, offsets):
        writer.add_outline_item(title, page_index)  # /Dest -> that page

def set_package_metadata(writer: PdfWriter, title: str, producer: str) -> None:
    """Write document information so the portal and archive can identify it."""
    writer.add_metadata({
        "/Title": title,
        "/Producer": producer,
        "/Creator": "grant-automation assembly engine",
    })

This is the whole spine of package generation: an ordered page tree, an outline keyed to component offsets, and metadata over the result. Everything that follows hardens this spine against the ways real components deviate from the ideal.

# Manifest-aware deterministic assembly

Assembling from a loose list of file paths works until the order is wrong, a component is missing, or two teams disagree on whether the biosketches precede or follow the budget. Production assembly is driven by a typed manifest — a declared, validated description of which components exist, in what order, and how each maps to a bookmark. The manifest is the single source of truth; the merge is a pure function of it. That is what makes assembly deterministic and auditable: given the same manifest and the same input bytes, you get the same package, and any change is a diff against a structured object rather than a re-run of an opaque script.

The manifest uses Pydantic v2 so an out-of-order or malformed package fails loudly at construction rather than producing a quietly wrong PDF. This mirrors the validation discipline the ingestion side applies through its schema validation with Pydantic layer.

from pydantic import BaseModel, Field, field_validator

class Component(BaseModel):
    """One ordered piece of the package and its bookmark label."""
    order: int = Field(ge=0)
    bookmark_title: str
    source_path: str
    required: bool = True

class PackageManifest(BaseModel):
    """The declared, validated composition of a submittable package."""
    opportunity_id: str
    agency: str
    components: list[Component]

    @field_validator("components")
    @classmethod
    def _ordered_and_unique(cls, v: list[Component]) -> list[Component]:
        orders = [c.order for c in v]
        if orders != sorted(orders):
            raise ValueError("components must be listed in ascending order")
        if len(set(orders)) != len(orders):
            raise ValueError("duplicate component order values")
        return v

The assembler consumes a validated manifest, verifies every required component is present on disk before touching the merge, then produces the merged writer, the outline, and the metadata in one deterministic pass:

from pathlib import Path
from pypdf import PdfWriter

def assemble_package(manifest: PackageManifest, out_path: str) -> None:
    """Produce one ordered, bookmarked, metadata-stamped package PDF.

    Fails before merging if any required component is missing, so a partial
    package is never written. The sort makes physical order a pure function of
    the manifest's declared order rather than list insertion order.
    """
    ordered = sorted(manifest.components, key=lambda c: c.order)
    for c in ordered:
        if c.required and not Path(c.source_path).is_file():
            raise FileNotFoundError(f"required component missing: {c.source_path}")

    writer, offsets = merge_with_component_offsets([c.source_path for c in ordered])
    add_component_outline(writer, [c.bookmark_title for c in ordered], offsets)
    set_package_metadata(
        writer,
        title=f"{manifest.agency} application — {manifest.opportunity_id}",
        producer="grant-automation assembly engine",
    )
    with open(out_path, "wb") as fh:
        writer.write(fh)

Page numbering and running headers are stamped as a separate overlay pass so the manifest stays about composition, not decoration. The pattern is to generate a transparent stamp page with reportlab for each physical page — carrying “Page N of M” and an optional opportunity identifier — and merge it onto the corresponding assembled page. Stamping after assembly, not before, is what keeps the numbers continuous across component boundaries; a component that was numbered internally would restart at 1 and mislead a reviewer.

# Agency-specific configuration

The same assembly engine must emit materially different packages per agency, because the National Institutes of Health (NIH), the National Science Foundation (NSF), and the Department of Defense (DoD) disagree about how a submission should be structured, bookmarked, and archived. Hard-coding one agency’s layout guarantees a rejected upload for the others. Treat the rows below as versioned configuration selected by the manifest’s agency field, not as defaults.

Two rows carry most of the risk. First, who assembles: for NIH you often deliver components and let eRA Commons stitch them, so your job is producing correctly bookmarked, correctly named components rather than the final PDF — but you still assemble a local preview package to validate before upload, and that preview must match what the portal will build. For a DoD BAA you assemble the volumes yourself, so the deterministic merge above is the deliverable. Second, PDF/A expectation: where a BAA cites federal record-retention requirements, the package must be archival-conformant, which is a distinct normalization pass covered end to end in generating PDF/A-compliant grant packages. Encoding these differences as a typed profile keeps the branching explicit rather than scattered through the assembler:

from dataclasses import dataclass

@dataclass(frozen=True, slots=True)
class AgencyPackageProfile:
    agency: str
    self_assembled: bool      # True: you emit the final PDF; False: portal merges
    require_pdfa: bool
    page_size: str = "LETTER"
    flatten_forms: bool = True

PROFILES: dict[str, AgencyPackageProfile] = {
    "NIH": AgencyPackageProfile("NIH", self_assembled=False, require_pdfa=False),
    "NSF": AgencyPackageProfile("NSF", self_assembled=False, require_pdfa=False),
    "DoD": AgencyPackageProfile("DoD", self_assembled=True, require_pdfa=True),
}

# Error handling and edge cases

Package generation fails in a small, recurring set of ways, and a production assembler treats each as a routed outcome rather than a crash that loses the run:

• Un-embedded or subset fonts. A component references a font that is not embedded, or embeds a subset missing a glyph the reviewer’s viewer cannot substitute. The portal rejects it. Detect it before upload by walking each page’s /Resources /Font dictionaries and asserting every font has a /FontFile, /FontFile2, or /FontFile3 stream; route missing embeds to re-rendering rather than shipping.
• Corrupt or truncated attachments. A PI’s uploaded PDF has a broken cross-reference table. pikepdf.open() raises PdfError. Catch it, name the offending component, and fail the whole assembly — a package missing one attachment is worse than no package, because it looks complete.
• Encrypted or password-protected PDFs. A letter of support exported with owner-level encryption blocks merge and violates every agency’s “no encryption” rule. Detect reader.is_encrypted, attempt an empty-password decrypt, and if that fails route the component back to the submitter — never ship encrypted content into a federal portal.
• Page-size mismatch. An attachment is A4 or oversized-figure size while the narrative is US Letter, producing a package that a reviewer sees as inconsistent and some portals reject. Compare each page’s /MediaBox against the profile’s expected size and flag or normalize before merge.
• Form fields not flattened. A filled /AcroForm that is not flattened can render differently across viewers or be silently altered. Flatten fields to static content before assembly so the submitted appearance is fixed.

import logging
import pikepdf

logger = logging.getLogger(__name__)

class PackageError(Exception):
    """Raised when a component cannot be safely merged into a package."""

def assert_fonts_embedded(pdf_path: str) -> None:
    """Raise if any font on any page lacks an embedded font-file stream."""
    with pikepdf.open(pdf_path) as pdf:
        for n, page in enumerate(pdf.pages, start=1):
            fonts = page.get("/Resources", {}).get("/Font", {})
            for name, font in dict(fonts).items():
                descriptor = font.get("/FontDescriptor", {})
                has_file = any(k in descriptor for k in
                               ("/FontFile", "/FontFile2", "/FontFile3"))
                if not has_file:
                    raise PackageError(
                        f"{pdf_path} page {n}: font {name} is not embedded")

def open_or_route(pdf_path: str) -> pikepdf.Pdf:
    """Open a component, routing encrypted/corrupt files to review."""
    try:
        pdf = pikepdf.open(pdf_path)
    except pikepdf.PasswordError as exc:
        logger.error("Encrypted component %s — routing to submitter", pdf_path)
        raise PackageError(f"encrypted component: {pdf_path}") from exc
    except pikepdf.PdfError as exc:
        logger.error("Corrupt component %s: %s", pdf_path, exc)
        raise PackageError(f"corrupt component: {pdf_path}") from exc
    return pdf

The rule is uniform with the rest of the platform: fail loudly, name the component, and let the orchestrator decide between re-render, re-request, or abort. A package that assembles despite a defective component is the one outcome that must never reach a portal, because it passes the “it opens” check while being non-submittable.

# Integration with the downstream pipeline

Package generation is the hinge between assembly and submission. Its output — one deterministic, bookmarked, font-embedded PDF (or, for a self-assembled DoD volume set, several) — feeds two consumers. Downstream, it is handed to the submission portal sync stage that transmits it to Grants.gov or Research.gov and tracks status. But before it may be transmitted, it must pass back through the page-limit and font enforcement checks — the assembled page count, the embedded-font guarantee, and the typography floors are all re-validated on the final bytes, not the pre-merge components, because assembly can itself introduce a violation (a stamp overlay that nudges a glyph into a margin, a component that pushed the narrative over its ceiling). That re-check closes the loop with the same NIH 12-page enforcement logic the compliance engine already owns.

The contract with the submission stage is narrow and auditable: package generation emits bytes plus a manifest hash, and never itself contacts a portal. That separation is what lets a later reviewer reconstruct exactly which components, in which order, produced the exact PDF that was submitted — the provenance the audit logging and provenance topic then records.

# Testing and verification

Assembly earns trust through structural assertions on the produced bytes, not on the inputs. Commit small fixture components — a two-page narrative, a filled form, a one-page attachment — and assert that the assembled package has the right page count, a bookmark per component landing on the right page, and embedded fonts throughout.

import pytest
from pypdf import PdfReader

def test_page_count_is_sum_of_components(tmp_path) -> None:
    manifest = build_fixture_manifest()      # 2 + 1 + 1 = 4 pages
    out = tmp_path / "package.pdf"
    assemble_package(manifest, str(out))
    assert len(PdfReader(str(out)).pages) == 4

def test_outline_has_one_bookmark_per_component(tmp_path) -> None:
    manifest = build_fixture_manifest()
    out = tmp_path / "package.pdf"
    assemble_package(manifest, str(out))
    outline = PdfReader(str(out)).outline
    assert len(outline) == len(manifest.components)

def test_first_bookmark_targets_first_page(tmp_path) -> None:
    manifest = build_fixture_manifest()
    out = tmp_path / "package.pdf"
    assemble_package(manifest, str(out))
    reader = PdfReader(str(out))
    first = reader.outline[0]
    assert reader.get_destination_page_number(first) == 0

def test_all_fonts_embedded(tmp_path) -> None:
    manifest = build_fixture_manifest()
    out = tmp_path / "package.pdf"
    assemble_package(manifest, str(out))
    assert_fonts_embedded(str(out))          # raises PackageError if not

def test_encrypted_component_is_rejected() -> None:
    with pytest.raises(PackageError):
        open_or_route("tests/fixtures/encrypted_letter.pdf")

Beyond the automated suite, a manual acceptance checklist catches what fixtures miss: open the package and confirm the outline tree matches the manifest order, that “Page N of M” is continuous across component boundaries, that no form field remains interactive, that every page is US Letter, and — for a DoD volume set — that each volume is a self-contained, bookmarked PDF. Only a package that clears both the structural tests and this checklist should be released to the compliance re-check and then to submission.

Deterministic, manifest-driven assembly turns the riskiest minute of a grant cycle into a reproducible, testable step: the same manifest and inputs always yield the same submittable bytes, every bookmark and font is verifiable, and the whole package is re-checked against compliance before it leaves the building. That is what makes the final PDF an audit-ready artifact rather than a last-minute gamble.

# Related

• Narrative template rendering — produces the narrative PDF this stage merges first.
• Agency form population — produces the filled, flattened forms that become package components.
• Generating PDF/A-compliant grant packages — the archival normalization pass for agencies that require it.
• Submission portal sync — the downstream stage that transmits the finished package.
• Page-limit & font enforcement — the compliance gate the final merged bytes must pass.

Up one level: Document Assembly & Audit Logging