# Agency Schema Registry & Versioning

A grant automation platform breaks the moment two stages disagree about which rule is in force: the parser encodes a 12-page ceiling, the compliance engine still checks against 15, and the assembly renderer pins a font floor nobody ratified. The cure is a single, authoritative store — an agency schema registry — that holds every requirement profile the National Institutes of Health (NIH), the National Science Foundation (NSF), and the Department of Defense (DoD) impose, versions each one, and serves exactly the profile that governed a proposal on the day its deadline fell. This section defines that registry as the backbone of the core architecture and Request for Proposal (RFP) taxonomy: the place where a Funding Opportunity Announcement (FOA) schema, a Proposal & Award Policies & Procedures Guide (PAPPG) rule set, and a Broad Agency Announcement (BAA) constraint matrix stop being scattered constants and become versioned, addressable, effective-dated records that every other stage loads rather than re-derives.

Without a registry, agency rules live wherever a developer last hard-coded them, and the same page limit gets copied — subtly wrong — into the extractor, the validator, and three renderers. With one, there is a single source of truth: parsing, the compliance validation rule engines, and assembly all resolve the identical profile object, so a policy change is one registry commit instead of a scavenger hunt across the codebase. The sections below build that store from the ground up — how a profile is identified and versioned, why versions must be immutable and effective-dated, how a resolver picks the correct one from (agency, mechanism, deadline), and how the whole thing migrates forward when an agency reissues its rules.

# Prerequisites and environment setup

The patterns here assume Python 3.10 or newer, because the registry leans on modern typing (str | None, list[SchemaProfile]) and on Pydantic v2 for its validated models. Two libraries carry the weight: pydantic models and validates each profile, and packaging parses and compares semantic version strings using the same algorithm the Python ecosystem uses for release ordering — you never hand-roll version comparison.

python -m venv .venv
source .venv/bin/activate
pip install "pydantic>=2.6" "packaging>=24.0"

Two assumptions shape everything downstream. First, the registry is the authority, not a cache: no stage may carry its own copy of a page limit or font rule, because a second copy is a second source of truth and therefore a future disagreement. Second, agency policy is time-scoped. A proposal is judged against the rules in force on its submission deadline, not the rules in force when it is validated, so the registry must be able to answer “which version governed 2026-02-05?” and not merely “which version is current?”. That single requirement — resolution by effective date — is what separates a real registry from a dictionary of latest values, and it drives the whole design.

# Core mechanism — the registry and resolver model

The registry is two cooperating parts. The store holds every version of every agency profile as an immutable record, keyed so that no two versions ever collide and no existing version is ever mutated in place. The resolver is a pure function: given an agency, a funding mechanism, and a deadline, it returns the one profile version that was effective on that date — or raises, loudly, when no version applies. Keeping these separate matters, because the store is boring, append-only infrastructure while the resolver encodes the one piece of policy logic every consumer depends on.

A profile’s identity has three coordinates. The agency and mechanism (an NIH R01, an NSF CAREER solicitation, a DoD SBIR topic) name which rule set. The semantic version (MAJOR.MINOR.PATCH) names which revision of that rule set. And an effective date range names when that revision governed. The resolver never guesses; it filters the store to the matching (agency, mechanism), keeps only the versions whose effective window contains the deadline, and among those returns the highest semantic version. The deadline — not “now”, not the proposal’s creation date — is the pivot, which is why back-dated reprocessing of a proposal submitted last cycle still yields the historically correct verdict.

from datetime import date

# The resolver contract in miniature: identity in, one immutable profile out.
# (agency, mechanism) selects the rule *family*; the deadline selects the
# *version* that was legally in force; ties break to the highest SemVer.
def resolve(agency: str, mechanism: str, deadline: date) -> "SchemaProfile":
    candidates = [
        p for p in STORE
        if p.agency == agency
        and p.mechanism == mechanism
        and p.effective_from <= deadline
        and (p.effective_until is None or deadline <= p.effective_until)
    ]
    if not candidates:
        raise VersionNotFound(agency, mechanism, deadline)
    # Highest SemVer among the date-eligible versions wins.
    return max(candidates, key=lambda p: p.parsed_version)

The important property is that resolve() is deterministic and side-effect free: the same three inputs always return the same profile, forever, because the store is immutable. That determinism is what makes a compliance decision reproducible a year later during an audit, and it is why the audit and version control discipline described in the parent architecture can pin every verdict to a concrete profile version rather than to “whatever the rules were at the time.”

# Version-aware registry implementation

The production store is a typed model, not a bare dictionary, so that a malformed profile fails at construction rather than at resolution time. The SchemaProfile below carries its semantic version as a real parsed object, validates its own effective-date window, and computes a content hash so the same version can be addressed and verified by digest. The Registry wraps a list of profiles and exposes the resolver as a method, refusing to admit two profiles that share an identity.

from __future__ import annotations
from datetime import date
from functools import cached_property
import hashlib
import json
from packaging.version import Version, InvalidVersion
from pydantic import BaseModel, Field, field_validator, model_validator

class SchemaProfile(BaseModel):
    """One immutable, effective-dated version of an agency requirement profile."""
    model_config = {"frozen": True}  # profiles are never mutated after creation

    agency: str                       # "NIH" | "NSF" | "DoD"
    mechanism: str                    # "R01", "CAREER", "SBIR-Phase-I", ...
    version: str                      # MAJOR.MINOR.PATCH semantic version
    effective_from: date              # first day this version governs
    effective_until: date | None = None  # None means "still in force"
    governing_document: str           # human-readable source, e.g. "PAPPG 24-1"
    max_narrative_pages: int
    min_font_pt: float
    min_margin_pt: float
    requires_data_management_plan: bool = False
    rules: dict[str, object] = Field(default_factory=dict)  # extension bag

    @field_validator("version")
    @classmethod
    def _valid_semver(cls, v: str) -> str:
        try:
            Version(v)  # rejects "24-1", "latest", "" — must be MAJOR.MINOR.PATCH
        except InvalidVersion as exc:
            raise ValueError(f"{v!r} is not a valid semantic version") from exc
        return v

    @model_validator(mode="after")
    def _sane_window(self) -> "SchemaProfile":
        if self.effective_until is not None and self.effective_until < self.effective_from:
            raise ValueError("effective_until precedes effective_from")
        return self

    @cached_property
    def parsed_version(self) -> Version:
        """SemVer object for ordering; parsing is validated above."""
        return Version(self.version)

    @cached_property
    def content_hash(self) -> str:
        """Stable digest of the profile's rule payload — the addressable id."""
        payload = self.model_dump(mode="json", exclude={"effective_until"})
        canonical = json.dumps(payload, sort_keys=True, separators=(",", ":"))
        return hashlib.sha256(canonical.encode()).hexdigest()[:16]

class VersionNotFound(LookupError):
    def __init__(self, agency: str, mechanism: str, deadline: date) -> None:
        super().__init__(f"No {agency}/{mechanism} profile effective on {deadline}")

class AmbiguousEffectiveWindow(ValueError):
    """Two versions of one identity claim overlapping effective dates."""

class Registry:
    def __init__(self, profiles: list[SchemaProfile]) -> None:
        self._profiles = list(profiles)
        self._guard_overlaps()

    def _guard_overlaps(self) -> None:
        # Reject a store where one deadline could resolve to two versions.
        by_id: dict[tuple[str, str], list[SchemaProfile]] = {}
        for p in self._profiles:
            by_id.setdefault((p.agency, p.mechanism), []).append(p)
        for (agency, mech), group in by_id.items():
            ordered = sorted(group, key=lambda p: p.effective_from)
            for earlier, later in zip(ordered, ordered[1:]):
                end = earlier.effective_until
                if end is None or end >= later.effective_from:
                    raise AmbiguousEffectiveWindow(
                        f"{agency}/{mech}: {earlier.version} and {later.version} overlap")

    def resolve(self, agency: str, mechanism: str, deadline: date) -> SchemaProfile:
        candidates = [
            p for p in self._profiles
            if p.agency == agency and p.mechanism == mechanism
            and p.effective_from <= deadline
            and (p.effective_until is None or deadline <= p.effective_until)
        ]
        if not candidates:
            raise VersionNotFound(agency, mechanism, deadline)
        return max(candidates, key=lambda p: p.parsed_version)

    def by_hash(self, content_hash: str) -> SchemaProfile:
        """Addressability: fetch the exact profile a past verdict cited."""
        for p in self._profiles:
            if p.content_hash == content_hash:
                return p
        raise KeyError(content_hash)

Three design choices earn their keep. Freezing the model (frozen=True) makes a profile immutable, so a resolved object cannot be edited by a consumer and quietly change the rule for the next caller. The content_hash makes each version addressable: a stored compliance verdict records the hash, and by_hash() retrieves the byte-identical rule set that produced it — proof, not paraphrase. And the overlap guard runs at construction, converting an ambiguous store into an immediate error instead of a resolver that silently returns whichever version happened to sort first. The rules extension bag lets agency-specific constraints — a DoD export-control trigger, an NSF mentoring-plan requirement — ride along without a schema migration for every new field.

# Agency-specific configuration

Each agency versions its rules on a different cadence and anchors them to a different governing document, so the registry cannot treat “a version” as one uniform concept. NSF publishes a discretely numbered PAPPG on a roughly annual schedule, which maps cleanly onto a bumped version with a known effective date; the NSF proposal guide taxonomy work tracks exactly those editions. NIH reissues FOAs individually and continuously, so a profile’s identity often keys on the FOA number as much as the mechanism. DoD BAAs carry no central schedule at all — each announcement is effectively its own rule set with its own dates. The table maps how the three fit one registry model.

The practical consequence is that the same SchemaProfile shape holds all three, but the ceremony around minting a new version differs. An NSF edition is a planned event you can pre-load with a future effective_from; a DoD BAA is an ad-hoc event you register when the announcement drops. Either way the resolver logic is identical — it only ever asks “which version’s window contains this deadline?” — which is the payoff of pushing agency idiosyncrasy into data rather than into branching code.

# Error handling and edge cases

A registry fails safely only if every ambiguous or missing case raises rather than guesses. Four failure modes recur, and each maps to an explicit outcome.

• Version not found. No profile’s effective window contains the deadline — commonly a deadline that predates the earliest registered version, or a mechanism nobody has profiled yet. resolve() raises VersionNotFound; the orchestrator routes the proposal to human review instead of validating against a default, because “no known rule” is never the same as “compliant.”
• Ambiguous effective date. Two versions of one identity claim overlapping windows, so a single deadline could resolve to either. The Registry constructor rejects this at load time via AmbiguousEffectiveWindow, turning a subtle non-determinism into a startup failure that a person must fix before anything runs.
• Open-ended windows. A None on effective_until means “still in force,” which is correct for the current version but dangerous if a new version is added without closing the prior one — the overlap guard catches exactly that, forcing the migration step below to set the outgoing version’s end date.
• Breaking schema migration. A new agency edition adds a required field or removes one an old profile relied on. Because profiles are immutable, you never edit the old version; you add a new one and, when the shape of SchemaProfile itself must change, migrate stored profiles through an explicit, tested upgrade function rather than trusting Pydantic to coerce silently.

from datetime import date, timedelta

def supersede(registry_profiles: list[SchemaProfile],
              old: SchemaProfile,
              new: SchemaProfile) -> list[SchemaProfile]:
    """Register `new` as the successor to `old` without ever mutating history.

    Closes the outgoing version's window the day before the new one starts, so
    the store stays overlap-free and every past deadline still resolves to the
    version that actually governed it.
    """
    if new.effective_from <= old.effective_from:
        raise ValueError("successor must start after its predecessor")
    closed_old = old.model_copy(
        update={"effective_until": new.effective_from - timedelta(days=1)}
    )
    # `old` itself is untouched; we append a closed copy + the new version.
    return [p for p in registry_profiles if p is not old] + [closed_old, new]

Deciding whether a new edition is a breaking (MAJOR) change or an additive one is its own discipline — the versioning agency schema profiles with semantic versions guide walks through the exact rules for classifying a rule-set change and resolving the right version by deadline.

# Integration with the downstream pipeline

The registry is not a stage the document passes through; it is a service every stage calls. Parsing loads a profile to know which sections and limits a solicitation must express, the compliance engine loads the same profile to render verdicts, and assembly loads it to shape agency-specific output — all three resolving the identical version by the same (agency, mechanism, deadline) key. The diagram shows the registry serving one resolved, versioned profile to each consumer.

Concretely, the resolved profile feeds the NIH FOA schema mapping stage the page limits and section order an FOA declares, hands the NSF proposal guide taxonomy the edition-specific PAPPG rules, and supplies the DoD BAA requirement extraction stage the constraint matrix for a given announcement. The compliance validation rule engines then evaluate a draft against that exact profile, and because the profile carries a content_hash, the verdict records which rule set was applied — so a reviewer can reconstruct the decision without ambiguity. This is the load-bearing contract of the whole architecture: consumers never interpret policy independently; they resolve one profile and obey it.

# Testing and verification

The registry earns trust through tests that assert the two properties everything depends on: the resolver returns the deadline-correct version, and stored profiles are genuinely immutable and addressable. Build the store from fixtures with known effective windows, then pin behavior at the boundaries.

import pytest
from datetime import date

def build_registry() -> Registry:
    return Registry([
        SchemaProfile(agency="NSF", mechanism="CAREER", version="1.0.0",
                      effective_from=date(2023, 1, 1), effective_until=date(2024, 5, 19),
                      governing_document="PAPPG 23-1", max_narrative_pages=15,
                      min_font_pt=10.0, min_margin_pt=72.0),
        SchemaProfile(agency="NSF", mechanism="CAREER", version="2.0.0",
                      effective_from=date(2024, 5, 20), effective_until=None,
                      governing_document="PAPPG 24-1", max_narrative_pages=15,
                      min_font_pt=11.0, min_margin_pt=72.0),
    ])

def test_resolver_picks_version_in_force_on_deadline() -> None:
    reg = build_registry()
    # A deadline before the 24-1 cutover must resolve to the 23-1 edition.
    early = reg.resolve("NSF", "CAREER", date(2024, 3, 1))
    assert early.version == "1.0.0" and early.min_font_pt == 10.0
    # A deadline after the cutover must resolve to 24-1.
    late = reg.resolve("NSF", "CAREER", date(2024, 6, 1))
    assert late.version == "2.0.0" and late.min_font_pt == 11.0

def test_missing_version_raises_not_found() -> None:
    reg = build_registry()
    with pytest.raises(VersionNotFound):
        reg.resolve("NSF", "CAREER", date(2020, 1, 1))  # predates all versions

def test_profiles_are_immutable() -> None:
    reg = build_registry()
    profile = reg.resolve("NSF", "CAREER", date(2024, 6, 1))
    with pytest.raises(Exception):  # frozen model rejects mutation
        profile.max_narrative_pages = 20

def test_overlapping_windows_are_rejected_at_load() -> None:
    with pytest.raises(AmbiguousEffectiveWindow):
        Registry([
            SchemaProfile(agency="NIH", mechanism="R01", version="1.0.0",
                          effective_from=date(2024, 1, 1), effective_until=None,
                          governing_document="FOA PA-24-A", max_narrative_pages=12,
                          min_font_pt=11.0, min_margin_pt=36.0),
            SchemaProfile(agency="NIH", mechanism="R01", version="1.1.0",
                          effective_from=date(2024, 6, 1), effective_until=None,
                          governing_document="FOA PA-24-B", max_narrative_pages=12,
                          min_font_pt=11.0, min_margin_pt=36.0),
        ])

Beyond the automated suite, a review checklist catches what fixtures cannot: confirm that every mechanism the pipeline processes has at least one registered version, that the current version of each identity has an open effective_until, that no stored verdict references a content_hash the store cannot retrieve, and that a superseded version’s window closes exactly one day before its successor begins. Only when those hold is the registry safe to treat as the pipeline’s single source of truth.

A schema registry turns agency policy from scattered constants into governed, versioned, effective-dated data — one place to change a rule, one function to resolve it, and one hash to prove which rule was applied. That is what lets parsing, compliance, and assembly agree, cycle after cycle, even as NIH, NSF, and DoD revise their requirements on their own independent clocks.

# Related

• Versioning agency schema profiles with semantic versions — deciding what bumps MAJOR/MINOR/PATCH and resolving by effective date.
• NIH FOA schema mapping — a downstream consumer that loads the resolved FOA profile.
• NSF proposal guide taxonomy — models the versioned PAPPG editions the registry stores.
• DoD BAA requirement extraction — consumes per-announcement constraint matrices from the registry.
• Compliance validation rule engines — evaluates drafts against the exact profile version the resolver returns.

Up one level: Core Architecture & RFP Taxonomy