# DoD BAA compliance matrix generation in Python

A Broad Agency Announcement (BAA) from the Department of Defense (DoD) fails a proposal differently than a standard solicitation does. Because BAAs are open, rolling instruments that fund basic and applied research, they scatter binding obligations across an evolving base announcement, topic-specific calls, and layered references to the Federal Acquisition Regulation (FAR), the Defense Federal Acquisition Regulation Supplement (DFARS), and component security supplements. Miss a single shall clause — an export-control attestation, a data-rights assertion, a phased milestone deliverable — and the proposal is ruled non-responsive before technical review, or worse, the obligation surfaces as a finding during a post-award Contracting Officer (CO) audit. A compliance matrix is the artifact that prevents that: one row per obligation, traceable back to the source clause. This page shows how to generate that matrix deterministically in Python, as the automation endpoint of the DoD BAA requirement extraction workflow, so that every conditional obligation, reporting cadence, and security mandate is captured, normalized, and mapped to an institutional response template rather than tracked by hand in a spreadsheet.

The pipeline has four phases: deterministic ingestion, obligation extraction and serialization, agency-specific edge-case handling, and validation. Each phase hands a typed artifact to the next, and each artifact is auditable on its own.

# Phase 1 — Deterministic ingestion and structural anchoring

DoD BAAs arrive as unstructured PDFs, HTML portal exports, or hybrid XML packages. Flattening them into a raw text stream destroys the section numbering and table boundaries that give a shall clause its scope, so ingestion has to be coordinate-aware. This step reuses the same geometry-preserving reader documented in PDF text extraction with pdfplumber; the difference here is that the output is anchored back to structural coordinates so a later audit can point to the exact page and position of every extracted obligation. Keep raw text recovery strictly separate from semantic classification — that separation is the Core Architecture & RFP Taxonomy discipline that keeps the pipeline debuggable.

Implementation steps:

1. Open once, stream pages. Use pdfplumber.open() as a context manager so a 200-page BAA stays memory-bounded.
2. Extract words with tolerances. Pass x_tolerance and y_tolerance to stop multi-column topic tables from fragmenting into unreadable word soup.
3. Retain coordinates. Store page, x0, and top alongside each token so the matrix can cite a position, not just a page.
4. Fail fast on unreadable input. A scanned or corrupt BAA that yields zero characters must halt the run, not silently produce an empty matrix.

import logging

import pdfplumber

logger = logging.getLogger(__name__)


def extract_structured_text(pdf_path: str) -> list[dict[str, object]]:
    """Extract coordinate-anchored word tokens from a DoD BAA PDF.

    Note: pdfplumber.extract_words() returns dicts keyed
    text, x0, x1, top, bottom, doctop, upright. Font size is NOT
    included here; read page.chars if a rule depends on typography.
    """
    extracted_blocks: list[dict[str, object]] = []
    try:
        with pdfplumber.open(pdf_path) as pdf:
            for page_num, page in enumerate(pdf.pages, start=1):
                for block in page.extract_words(x_tolerance=3, y_tolerance=3):
                    extracted_blocks.append(
                        {
                            "page": page_num,
                            "x0": block["x0"],
                            "y0": block["top"],
                            "text": block["text"],
                        }
                    )
    except Exception as exc:
        logger.error("PDF ingestion failed for %s: %s", pdf_path, exc)
        raise RuntimeError("Ingestion halted: unreadable document structure.") from exc

    if not extracted_blocks:
        raise RuntimeError("No text recovered — route to OCR pre-step, do not pass empty.")
    return extracted_blocks

Validate bounding-box continuity to tell a ruled topic table apart from body prose before the tokens ever reach the classifier. If a rule depends on font size — for example, distinguishing a section heading from the clause beneath it — read page.chars instead, because extract_words() exposes no font metadata.

# Phase 2 — Obligation extraction and matrix serialization

The core transformation turns anchored tokens into typed obligation records. DoD compliance hinges on precise modal-verb detection: the engine must isolate mandatory indicators (shall, must, will, are required to) while ignoring permissive language (may, should, encouraged). Conditional obligations — the ones gated by if, when, unless, or subject to — need an activation flag so the response side knows a requirement is contingent, not absolute.

Model the obligation as a Pydantic v2 record rather than a loose dict, so an unrecognized modal verb fails loudly at construction instead of poisoning the matrix downstream. This is the same Pydantic validation layer pattern the ingestion pipeline uses to harden every parsed structure.

import re

from pydantic import BaseModel, field_validator

MANDATORY = re.compile(r"\b(shall|must|will|are required to|is required to)\b", re.I)
CONDITIONAL = re.compile(r"\b(if|when|unless|provided that|subject to)\b", re.I)
REG_REF = re.compile(r"(?:FAR|DFARS|DoDI|NIST SP)\s*[\d.\-]+(?:\(\w+\))?", re.I)
EXCEPTION = re.compile(r"(?:unless|except|unless otherwise directed by)\s[^.]+", re.I)

_ALLOWED_MODALS = {"shall", "must", "will", "are required to", "is required to"}


class ComplianceObligation(BaseModel):
    requirement_id: str
    source_text: str
    modal_verb: str
    is_conditional: bool
    activation_condition: str | None = None
    regulatory_ref: str | None = None
    exception_clause: str | None = None

    @field_validator("modal_verb")
    @classmethod
    def _known_modal(cls, v: str) -> str:
        if v.lower() not in _ALLOWED_MODALS:
            raise ValueError(f"unrecognized mandatory modal verb: {v!r}")
        return v.lower()


def parse_obligations(segments: list[str]) -> list[ComplianceObligation]:
    obligations: list[ComplianceObligation] = []
    for segment in segments:
        modal_match = MANDATORY.search(segment)
        if not modal_match:
            continue  # permissive-only sentence: not a binding obligation
        cond_match = CONDITIONAL.search(segment)
        exc_match = EXCEPTION.search(segment)
        ref_match = REG_REF.search(segment)
        obligations.append(
            ComplianceObligation(
                requirement_id=f"REQ-{len(obligations) + 1:04d}",
                source_text=segment.strip(),
                modal_verb=modal_match.group(1),
                is_conditional=cond_match is not None,
                activation_condition=cond_match.group(0) if cond_match else None,
                regulatory_ref=ref_match.group(0) if ref_match else None,
                exception_clause=exc_match.group(0) if exc_match else None,
            )
        )
    return obligations

Run the regexes on sentence-level segments, not whole pages, so a shall in one clause never binds to a condition in the next. Preserve inline exceptions verbatim: a unless otherwise directed by the Contracting Officer phrase is exactly what a CO override review looks for post-award.

Serialize the records into a typed pandas DataFrame, then stamp it with an immutable audit hash so any later tampering with the matrix is detectable:

import hashlib
from datetime import datetime, timezone

import pandas as pd


def serialize_to_matrix(obligations: list[ComplianceObligation]) -> pd.DataFrame:
    if not obligations:
        raise ValueError("No obligations extracted — verify source and modal patterns.")

    df = pd.DataFrame([o.model_dump() for o in obligations])
    # Nullable columns must be filled before a non-nullable string cast.
    for col in ("activation_condition", "regulatory_ref", "exception_clause"):
        df[col] = df[col].fillna("")
    df = df.astype(
        {
            "requirement_id": "string",
            "source_text": "string",
            "modal_verb": "category",
            "is_conditional": "boolean",
        }
    )
    df.attrs["audit_hash"] = hashlib.sha256(df.to_json().encode()).hexdigest()
    df.attrs["generated_utc"] = datetime.now(timezone.utc).isoformat()
    return df

# Phase 3 — Edge cases and agency-specific overrides

The base extractor is agency-agnostic, but a DoD BAA carries obligations no NIH or NSF solicitation does, and the funding components themselves diverge. A matrix that treats every BAA identically will silently drop the highest-risk clauses. The table below captures the overrides the pipeline must layer on top of the generic modal-verb pass.

Because these overrides are policy that changes between announcement editions, keep the thresholds and clause maps in the compliance threshold tuning layer rather than hard-coded here, and resolve cross-references against the compliance validation rule engines before export. Encoding differences and missing tables are the other common failure surface, so the pipeline degrades gracefully instead of crashing an overnight batch:

import logging
from datetime import datetime, timezone
from pathlib import Path

logger = logging.getLogger(__name__)


class CompliancePipelineError(Exception):
    """Pipeline-level failure that should trigger automated alerting."""


def run_pipeline(pdf_path: str, output_dir: Path) -> Path:
    try:
        segments = [s["text"] for s in extract_structured_text(pdf_path)]
        matrix = serialize_to_matrix(parse_obligations(segments))
        report = validate_matrix(matrix)
        if not all(report.values()):
            logger.warning("Matrix validation failed — review flags before submission.")
        stamp = datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S")
        out = output_dir / f"compliance_matrix_{stamp}.csv"
        matrix.to_csv(out, index=False)
        return out
    except FileNotFoundError as exc:
        logger.critical("Source document missing: %s", exc)
        raise CompliancePipelineError("BAA not found — verify distribution path.") from exc
    except UnicodeDecodeError:
        logger.warning("Encoding mismatch — attempting fallback parser.")
        return _fallback_extract(pdf_path, output_dir)
    except Exception as exc:
        logger.error("Pipeline failure: %s", exc)
        raise CompliancePipelineError("Unrecoverable parse — manual review required.") from exc

The diagram below maps the run_pipeline control flow, including the fallback and halt branches:

Log every fallback activation. A silent switch to a degraded parser is itself an audit finding, because it means the matrix was built from a lower-fidelity read than the reviewer assumes.

# Phase 4 — Validation and verification

Before the matrix is handed to a proposal assembler or a CO, confirm it is internally consistent and untampered. Validation runs immediately after serialization and blocks export on any structural failure, giving bidirectional traceability between source text, extracted obligation, and institutional response field.

import logging

import pandas as pd

logger = logging.getLogger(__name__)

_VALID_MODALS = ["shall", "must", "will", "are required to", "is required to"]


def validate_matrix(df: pd.DataFrame) -> dict[str, bool]:
    report = {
        "no_null_requirements": bool(df["requirement_id"].notna().all()),
        "modal_verbs_valid": bool(df["modal_verb"].isin(_VALID_MODALS).all()),
        "conditional_flag_typed": bool(df["is_conditional"].notna().all()),
        "audit_hash_intact": "audit_hash" in df.attrs
        and len(df.attrs["audit_hash"]) == 64,
    }
    if not all(report.values()):
        logger.warning("Matrix validation failed — review flagged fields.")
    return report

Beyond the automated report, a manual acceptance checklist catches what the assertions miss:

• Every row cites a source_text that traces back to a real clause, not a merged cross-clause fragment.
• Conditional obligations preserve their if/unless trigger verbatim in activation_condition.
• Export-control and NIST SP 800-171 clauses are flagged for restricted handling, not left in the general pool.
• The audit_hash recomputed from the released CSV matches the value stored at generation time.
• A deliberately corrupt fixture routes to review rather than emitting an empty matrix.

Only once those pass should the matrix feed the downstream response-template mapping. Pair this with the Pydantic schema validation stage so the serialized JSON is checked against the institutional schema before it reaches a submission portal.

# Frequently asked questions

# Related

• DoD BAA requirement extraction — the parent workflow this matrix generator completes.
• PDF text extraction with pdfplumber — the coordinate-aware ingestion this pipeline reads from.
• Schema validation with Pydantic — hardening the obligation records before export.
• Compliance validation rule engines — where DFARS and export-control clauses are finally enforced.
• Compliance threshold tuning — owning the per-component thresholds this matrix flags against.

Up one level: DoD BAA requirement extraction