Generating a pre-submission deficiency checklist in Python

A compliance engine that returns a list of raw verdicts helps no one at 4:00 PM on a deadline; a principal investigator needs to know which problems block submission, which can wait, who has to fix each one, and exactly where in the document it lives. This guide builds the stage that turns structured compliance verdicts into a routed, human-readable deficiency checklist — grouped by severity, addressed to the person who can act on it, and annotated with a fix instruction and a pointer to the offending location. It is the rendering half of automated checklist generation: where the generator enumerates what must be checked, this stage takes the verdicts those checks produced and shapes them into a report a mixed audience of investigators, grants administrators, and institutional signatories can each read and act on without wading through the others’ items.

The failure this prevents is a report nobody can triage. A flat dump of forty findings, undifferentiated by severity or owner, gets the same treatment as no report at all — the investigator scans for the word “error,” misses the one blocking defect buried among advisory nits, and submits anyway. A routed checklist makes the blocking items impossible to miss and puts every finding in front of the one person who can resolve it.

Phase 1 — Model a deficiency from a compliance verdict

A verdict from the compliance engine is machine-shaped: a check identifier, a pass or fail, a measurement, sometimes a coordinate. A deficiency is human-shaped: a severity, an owner, a plain-language message, a fix instruction, and a location a reader can jump to. The first phase defines that transformation as a typed model so every rendered line carries the same fields and nothing routes to the wrong desk.

Three enumerations carry the routing logic. Severity decides grouping and whether the item blocks; the owner decides the desk; the location gives the reader somewhere to look.

python

from __future__ import annotations

from enum import Enum
from pydantic import BaseModel, Field, field_validator


class Severity(str, Enum):
    BLOCKER = "blocker"    # submission is administratively non-compliant
    WARNING = "warning"    # allowed but risky; may be waivable
    INFO = "info"          # advisory; no action strictly required


class Owner(str, Enum):
    PI = "principal_investigator"      # narrative content, science sections
    GRANTS_ADMIN = "grants_admin"      # formatting, budget, structure
    AOR = "authorized_rep"             # eligibility, institutional certs, sign-off


class Location(BaseModel):
    """Where the deficiency lives, precise enough to jump to."""

    document: str
    section: str | None = None
    page: int | None = None
    # Optional glyph box (x0, top, x1, bottom) for an annotator to circle.
    bbox: tuple[float, float, float, float] | None = None


class Deficiency(BaseModel):
    code: str                          # stable check id, e.g. "FMT-PAGES"
    severity: Severity
    owner: Owner
    message: str                       # what is wrong, in plain language
    fix_hint: str                      # what to do about it
    location: Location
    waivable: bool = False             # a WARNING an AOR may accept on the record

    @field_validator("fix_hint")
    @classmethod
    def _fix_hint_is_actionable(cls, v: str) -> str:
        if not v.strip():
            raise ValueError("every deficiency must carry an actionable fix_hint")
        return v

Mapping verdicts to deficiencies is where the engine’s structural output — the same typed violations produced by required section mapping and the page-limit routines — becomes routable. The mapping is a pure function of the verdict and the agency profile, so the same verdict always yields the same routed line:

python

def deficiencies_from_verdicts(
    verdicts: list[dict[str, object]],
    document: str,
) -> list[Deficiency]:
    """Translate raw compliance verdicts into routed, human-readable items."""
    routing = {
        "FMT-PAGES":  (Severity.BLOCKER, Owner.GRANTS_ADMIN,
                       "Narrative exceeds the page limit",
                       "Cut the narrative to the ceiling and re-export the PDF."),
        "SEC-MISSING": (Severity.BLOCKER, Owner.PI,
                        "A required section is absent",
                        "Add the missing section and re-run the structural check."),
        "FMT-MARGIN": (Severity.WARNING, Owner.GRANTS_ADMIN,
                       "Text drifts into the reserved margin",
                       "Reset the page margin and confirm no figure bleeds past it."),
        "ELIG-SAM":   (Severity.BLOCKER, Owner.AOR,
                       "Institutional registration is not current",
                       "Renew the registration before the AOR signs the package."),
    }
    items: list[Deficiency] = []
    for v in verdicts:
        if v.get("passed"):
            continue                          # only failures become deficiencies
        code = str(v["code"])
        severity, owner, message, fix = routing[code]
        items.append(Deficiency(
            code=code, severity=severity, owner=owner,
            message=message, fix_hint=fix,
            location=Location(
                document=document,
                section=v.get("section"),        # type: ignore[arg-type]
                page=v.get("page"),              # type: ignore[arg-type]
                bbox=v.get("bbox"),              # type: ignore[arg-type]
            ),
            waivable=(severity is Severity.WARNING),
        ))
    return items

Phase 2 — Group by severity and route to an owner, then render

A useful checklist is read two ways at once: a program office reads it by severity — show me everything that blocks — while each owner reads only their lane. The report model supports both projections from one canonical list, so the Markdown a PI receives and the console summary an administrator scans never diverge.

One verdict stream, three owner lanes: each deficiency is colored by severity and routed to the desk that can resolve it, so no reader has to sift the others' items.

The report model holds the canonical list and offers the two projections plus renderers. Grouping by severity puts blockers first; routing by owner slices the same list per desk:

python

from collections import defaultdict


class DeficiencyReport(BaseModel):
    solicitation_id: str
    document: str
    items: list[Deficiency]

    @property
    def is_clean(self) -> bool:
        return not any(i.severity is Severity.BLOCKER for i in self.items)

    def by_severity(self) -> dict[Severity, list[Deficiency]]:
        order = [Severity.BLOCKER, Severity.WARNING, Severity.INFO]
        grouped: dict[Severity, list[Deficiency]] = {s: [] for s in order}
        for item in self.items:
            grouped[item.severity].append(item)
        return grouped

    def for_owner(self, owner: Owner) -> list[Deficiency]:
        return [i for i in self.items if i.owner is owner]

    def to_markdown(self) -> str:
        lines = [f"# Pre-submission deficiencies — {self.solicitation_id}", ""]
        for severity, items in self.by_severity().items():
            if not items:
                continue
            lines.append(f"## {severity.value.title()} ({len(items)})")
            for i in items:
                where = i.location.section or f"page {i.location.page}" or "document"
                waiv = " _(waivable)_" if i.waivable else ""
                lines.append(f"- **[{i.owner.value}]** {i.message} — _{where}_{waiv}")
                lines.append(f"  - Fix: {i.fix_hint}")
            lines.append("")
        if self.is_clean:
            lines.append("_No blocking deficiencies. Package is clear to assemble._")
        return "\n".join(lines)

A console renderer for a terminal-driven pipeline is a thin variant of the same projection — it reads by_severity(), prints blockers in red and warnings in yellow, and exits non-zero when is_clean is false so the report drops cleanly into a pre-push gate. Because both renderers read one canonical list, a fix that clears a blocker updates every view at once. Which severity is a blocker versus a waivable warning is not decided here: those weights come from the threshold tuning for compliance layer, so a near-miss on a low-weight dimension surfaces as a warning rather than a hard block.

Phase 3 — Edge cases: waivable warnings, agency-required items, and empty reports

Three situations break a naive renderer that treats every finding identically and an empty list as success.

Waivable warnings. Some warnings are genuinely acceptable on the record — a formatting deviation an institution has cleared before. These carry waivable=True and route to the Authorized Organizational Representative (AOR), whose acceptance is logged rather than silently dropped. The waiver decision is an approval event, so it belongs with the institutional approval routing workflow, not with the renderer, which only marks the item eligible for one.
Agency-specific required items. A deficiency that is a blocker for one agency is absent for another: an eRA Commons identifier is a National Institutes of Health (NIH) blocker, Research.gov registration is a National Science Foundation (NSF) one, and a System for Award Management registration is a Department of Defense (DoD) one. The routing map is therefore parameterized by agency profile, so swapping the profile swaps which codes exist and how they are weighted, never the rendering logic.
Empty means two different things. An empty deficiency list is only good news if every check actually ran. A report with zero items because the document passed is clean; a report with zero items because the checks never executed is unverified, and conflating them is how a broken pipeline ships a non-compliant package. The report must carry a coverage record and distinguish the two states explicitly:

python

class ReportStatus(str, Enum):
    CLEAN = "clean"            # all checks ran, no blockers
    DEFICIENT = "deficient"    # at least one blocker
    UNVERIFIED = "unverified"  # checks did not all run — NOT clean


def resolve_status(
    report: DeficiencyReport,
    checks_expected: set[str],
    checks_run: set[str],
) -> ReportStatus:
    """An empty report is only CLEAN when every expected check actually ran."""
    if checks_run < checks_expected:
        return ReportStatus.UNVERIFIED       # coverage gap: never call this clean
    if not report.is_clean:
        return ReportStatus.DEFICIENT
    return ReportStatus.CLEAN

Situation	Deficiency list	Coverage	Reported status	Downstream effect
Document passed every check	empty	complete	CLEAN	assembler may package
One page-limit blocker	one blocker	complete	DEFICIENT	packaging blocked
Waivable margin warning only	one waivable warning	complete	CLEAN	package with logged waiver
A required check never executed	empty	incomplete	UNVERIFIED	routed to review, not packaged

Phase 4 — Validation and pytest

Because deficiency mapping and rendering are pure functions, they test deterministically against constructed verdicts — no PDFs, no clock dependence, no mocked I/O.

python

import pytest
from pydantic import ValidationError


def test_only_failures_become_deficiencies() -> None:
    verdicts = [
        {"code": "FMT-PAGES", "passed": True},                 # clean check
        {"code": "SEC-MISSING", "passed": False, "section": "Broader Impacts"},
    ]
    items = deficiencies_from_verdicts(verdicts, document="proposal.pdf")
    assert len(items) == 1
    assert items[0].owner is Owner.PI          # missing section routes to the PI


def test_blocker_makes_report_not_clean() -> None:
    item = Deficiency(
        code="FMT-PAGES", severity=Severity.BLOCKER, owner=Owner.GRANTS_ADMIN,
        message="over limit", fix_hint="cut the narrative",
        location=Location(document="proposal.pdf", page=16),
    )
    report = DeficiencyReport(solicitation_id="BAA-1", document="proposal.pdf", items=[item])
    assert not report.is_clean


def test_empty_report_with_coverage_gap_is_unverified() -> None:
    report = DeficiencyReport(solicitation_id="PA-25-1", document="proposal.pdf", items=[])
    status = resolve_status(
        report, checks_expected={"FMT-PAGES", "SEC-MISSING"}, checks_run={"FMT-PAGES"},
    )
    assert status is ReportStatus.UNVERIFIED   # empty but not clean


def test_fix_hint_is_mandatory() -> None:
    with pytest.raises(ValidationError):
        Deficiency(
            code="X", severity=Severity.INFO, owner=Owner.PI,
            message="note", fix_hint="   ",     # blank hint is rejected
            location=Location(document="proposal.pdf"),
        )

A manual acceptance pass covers what unit tests cannot: read the rendered Markdown as each owner and confirm every item is actionable and correctly addressed, confirm a waivable warning shows its waiver path, and confirm that a deliberately disabled check produces an UNVERIFIED status rather than a false CLEAN. Only when a blocking verdict reliably produces a blocking, correctly routed line should the report gate a submission. A routed deficiency checklist is what turns a wall of verdicts into the short, owned to-do list a proposal team can actually clear before the portal closes.

Frequently asked questions

How is this different from the automated checklist generator?

The generator enumerates the requirements a solicitation imposes and declares what must be checked; this stage consumes the verdicts those checks produced and renders them as a routed, severity-grouped report for humans. One is the requirement side, the other is the reporting side. Keeping them separate lets one generator drive every agency while each rendered report is tailored to its readers.

Why route deficiencies to different owners instead of one list?

Because the people who fix a proposal have different jobs. A missing science section is the principal investigator’s to write, a margin or budget-format defect is the grants administrator’s to correct, and an expired institutional registration is the authorized representative’s to renew. A single undifferentiated list forces everyone to read everything and lets the one blocking item hide among advisory nits.

Why does an empty deficiency list need a status field?

Because empty can mean two opposite things. If every expected check ran and found nothing, the package is clean. If the checks never executed, the empty list is a coverage gap, and treating it as clean ships a non-compliant package. The resolve_status helper compares the checks that ran against the checks expected and returns UNVERIFIED whenever coverage is incomplete.

Who decides whether a warning is waivable?

The severity and waivability come from the threshold-tuning layer’s weights, and the act of accepting a waiver is an approval event recorded by the institutional approval routing workflow. The renderer only marks an item eligible for a waiver and addresses it to the authorized representative; it never silently drops a warning.

Threshold tuning for compliance — sets the weights that decide a finding’s severity and waivability.
Required section mapping — a source of the structural verdicts this report renders.
Institutional approval routing — records the waiver and sign-off events a deficiency report routes to the AOR.
Automated checklist generation — the requirement-enumeration stage whose verdicts feed this renderer.

Up one level: Automated Checklist Generation

# Generating a pre-submission deficiency checklist in Python

# Phase 1 — Model a deficiency from a compliance verdict

# Phase 2 — Group by severity and route to an owner, then render

# Phase 3 — Edge cases: waivable warnings, agency-required items, and empty reports

# Phase 4 — Validation and pytest

# Frequently asked questions

# Related

Generating a pre-submission deficiency checklist in Python

Phase 1 — Model a deficiency from a compliance verdict

Phase 2 — Group by severity and route to an owner, then render

Phase 3 — Edge cases: waivable warnings, agency-required items, and empty reports

Phase 4 — Validation and pytest

Frequently asked questions

Related