# DoD BAA Requirement Extraction

A Department of Defense (DoD) Broad Agency Announcement (BAA) rejects proposals for reasons a National Institutes of Health (NIH) or National Science Foundation (NSF) submission never encounters: a missing export-control attestation, an unmarked data-rights assertion, a security-clearance obligation buried three paragraphs deep in a topic description. Requirement extraction is the workflow stage that reads a BAA’s dense, narrative-driven text and emits a machine-readable list of the obligations a proposal must satisfy — before a contracting officer’s review turns an oversight into a non-selectable submission. This page sits inside the Core Architecture & RFP Taxonomy that governs how every solicitation is parsed and classified, and it focuses on the hardest input in that taxonomy: defense solicitations, whose requirements are conditional, cross-referenced, and rarely tabulated.

Unlike the highly templated Funding Opportunity Announcements that NIH issues, a BAA is written under Federal Acquisition Regulation (FAR) 35.016 to solicit innovative research across broad topic areas, so its structure varies by issuing office — the Defense Advanced Research Projects Agency, the Office of Naval Research, the Air Force Office of Scientific Research, and the Congressionally Directed Medical Research Programs all publish BAAs with different section conventions. Extraction therefore cannot assume fixed section numbers or a stable schema. It must locate obligations by pattern and context, score its own confidence, and hand ambiguous cases to a human reviewer. The sections below cover the environment you need, the extraction mechanism, a production-grade typed implementation, how DoD parameters differ from NIH and NSF, the failure modes that corrupt real pipelines, how extracted requirements feed the downstream compliance stack, and how to test that the output is trustworthy.

# Prerequisites and Environment Setup

Requirement extraction runs on Python 3.10 or newer — the implementation below uses structural pattern matching and the modern typing syntax that older interpreters do not support. The core dependencies are a coordinate-aware PDF reader, a schema/validation layer, and an optional linguistic model for named entity recognition (NER):

python -m venv .venv && source .venv/bin/activate
pip install "pdfplumber>=0.11" "pydantic>=2.6" "spacy>=3.7"
python -m spacy download en_core_web_sm   # optional NER model for entity spans

The input assumptions matter as much as the packages. A BAA arrives as a PDF that is almost always born-digital (text-layer present) but occasionally scanned, in which case an optical character recognition (OCR) fallback is required before any text is available. BAAs also reference external documents — a separate Proposal Preparation Instructions attachment, an agency addendum, or the Defense Federal Acquisition Regulation Supplement (DFARS) clauses that govern data rights — so the extractor must treat the announcement as the root of a small document set, not a single file. Text acquisition itself is handled upstream by PDF text extraction with pdfplumber, which preserves the bounding-box coordinates and font metadata this stage depends on to distinguish a mandatory clause from a running header. Everything on this page assumes that stage has already produced clean, coordinate-tagged text.

# Core Mechanism: From Narrative Text to Typed Requirements

Extraction proceeds in three internal passes. First, segmentation splits the announcement into candidate blocks — the NLP section boundary detection workflow identifies where a topic area, an eligibility clause, or a submission-instruction block begins and ends, because a BAA’s headings are inconsistent and cannot be trusted as delimiters. Second, classification runs each block through a set of category detectors: rule-based regular expressions catch the high-precision markers (a dollar cap, a clearance level, a DFARS clause number), while NER catches the fuzzier spans (an organization named as a required teaming partner, a country named in an export restriction). Third, scoring assigns each candidate a confidence value and attaches provenance — the source page and character offset — so that every downstream decision can be traced back to the exact sentence that produced it.

The unit of output is a typed requirement object. Modeling it explicitly, rather than emitting loose dictionaries, is what lets the rest of the taxonomy treat DoD, NIH, and NSF obligations through one interface. Each requirement carries the category it belongs to, the raw text it was lifted from, the normalized value the detector extracted, a confidence score, and a provenance record:

from enum import Enum
from pydantic import BaseModel, Field, field_validator


class RequirementCategory(str, Enum):
    """Obligation classes a DoD BAA can impose on a proposal."""
    TECHNICAL_SCOPE = "technical_scope"
    SECURITY_CLEARANCE = "security_clearance"
    EXPORT_CONTROL = "export_control"
    DATA_RIGHTS = "data_rights"
    BUDGET_CONSTRAINT = "budget_constraint"
    ADMINISTRATIVE = "administrative"
    REPORTING = "reporting"


class Provenance(BaseModel):
    """Where in the source document a requirement was found."""
    page: int = Field(..., ge=1)
    char_start: int = Field(..., ge=0)
    char_end: int = Field(..., ge=0)


class BAARequirement(BaseModel):
    clause_id: str
    category: RequirementCategory
    raw_text: str
    extracted_value: str | None = None
    confidence: float = Field(..., ge=0.0, le=1.0)
    provenance: Provenance
    needs_review: bool = False

    @field_validator("raw_text")
    @classmethod
    def trim_and_bound(cls, v: str) -> str:
        # Keep enough context for a reviewer without storing whole pages.
        cleaned = " ".join(v.split())
        return cleaned[:600]

Because the model uses Pydantic v2 with a field_validator, malformed spans are rejected at construction time rather than surfacing as corrupt rows in a compliance matrix later. The needs_review flag is the hinge between automation and human oversight: it is set whenever confidence falls below a per-category threshold, which is exactly the knob that threshold tuning for compliance exists to calibrate.

# Rule-Aware Extraction Implementation

The production extractor pairs a category-specific pattern table with a segmentation step, then constructs validated BAARequirement objects. Security-clearance and DFARS-clause patterns are high precision and earn high base confidence; a budget cap is precise but frequently restated across sections, so its matches are deduplicated on the normalized value. Export-control detection deliberately errs toward flagging, because a false negative there is far more expensive than a false positive:

import re
from collections.abc import Iterator

# Per-category detectors. High-precision markers get high base confidence.
PATTERNS: dict[RequirementCategory, re.Pattern[str]] = {
    RequirementCategory.SECURITY_CLEARANCE: re.compile(
        r"(?:requires?|must (?:possess|hold)|eligible for)\s+(?:an?\s+)?"
        r"(TOP\s+SECRET|SECRET|CONFIDENTIAL|PUBLIC\s+TRUST|"
        r"facility\s+(?:security\s+)?clearance)",
        re.IGNORECASE,
    ),
    RequirementCategory.EXPORT_CONTROL: re.compile(
        r"\b(ITAR|International\s+Traffic\s+in\s+Arms\s+Regulations|"
        r"EAR|Export\s+Administration\s+Regulations|export[- ]controlled)\b",
        re.IGNORECASE,
    ),
    RequirementCategory.DATA_RIGHTS: re.compile(
        r"\b(unlimited\s+rights|government\s+purpose\s+rights|"
        r"limited\s+rights|DFARS\s+252\.227-\d{4})\b",
        re.IGNORECASE,
    ),
    RequirementCategory.BUDGET_CONSTRAINT: re.compile(
        r"(?:not\s+to\s+exceed|maximum|cap(?:ped)?\s+at|limit(?:ed)?\s+to)\s+"
        r"\$?\s*([\d,]+(?:\.\d{2})?)",
        re.IGNORECASE,
    ),
}

# Categories where a missed obligation is costlier than a false alarm.
FLAG_ON_ANY_MATCH: frozenset[RequirementCategory] = frozenset(
    {RequirementCategory.EXPORT_CONTROL, RequirementCategory.DATA_RIGHTS}
)

BASE_CONFIDENCE: dict[RequirementCategory, float] = {
    RequirementCategory.SECURITY_CLEARANCE: 0.93,
    RequirementCategory.EXPORT_CONTROL: 0.88,
    RequirementCategory.DATA_RIGHTS: 0.86,
    RequirementCategory.BUDGET_CONSTRAINT: 0.90,
}

REVIEW_THRESHOLD = 0.85


def iter_sections(text: str) -> Iterator[tuple[int, str]]:
    """Yield (offset, block) pairs from a BAA's narrative text.

    A real pipeline replaces this with NLP section-boundary detection;
    the offset lets each requirement record its provenance.
    """
    offset = 0
    for block in re.split(r"\n\s*\n", text):
        if block.strip():
            yield offset, block
        offset += len(block) + 2


def extract_requirements(text: str, page: int = 1) -> list[BAARequirement]:
    found: list[BAARequirement] = []
    seen_values: set[tuple[RequirementCategory, str]] = set()

    for offset, block in iter_sections(text):
        for category, pattern in PATTERNS.items():
            match = pattern.search(block)
            if not match:
                continue
            value = (match.group(1) if match.groups() else match.group(0)).strip()
            dedup_key = (category, value.lower())
            if dedup_key in seen_values:
                continue
            seen_values.add(dedup_key)

            confidence = BASE_CONFIDENCE.get(category, 0.75)
            flag = (
                category in FLAG_ON_ANY_MATCH
                or confidence < REVIEW_THRESHOLD
            )
            found.append(
                BAARequirement(
                    clause_id=f"{category.value.upper()[:3]}-{len(found) + 1:03d}",
                    category=category,
                    raw_text=block,
                    extracted_value=value,
                    confidence=confidence,
                    provenance=Provenance(
                        page=page,
                        char_start=offset + match.start(),
                        char_end=offset + match.end(),
                    ),
                    needs_review=flag,
                )
            )
    return found

The extracted list is the raw material for a compliance matrix, and turning it into a reviewer-ready, agency-formatted grid — with responsible-party assignment, deadline mapping, and fulfillment status — is covered in depth on the child page, DoD BAA compliance matrix generation in Python. Whichever schema the requirements ultimately validate against, they pass through the same schema validation with Pydantic discipline used across the ingestion stack, which keeps the DoD path type-compatible with the NIH and NSF paths.

# Agency-Specific Configuration

The same extraction engine serves all three funders, but the detectors and thresholds it loads differ sharply. NIH and NSF solicitations rarely trigger security or export logic and expose their limits in predictable places; a DoD BAA inverts that, making the conditional categories primary. The table below captures the parameters an extractor must switch on by agency — the difference between the DoD column and the others is precisely why defense solicitations need this dedicated workflow rather than the schema-mapping approach that suffices for the NIH FOA Schema Mapping and NSF Proposal Guide Taxonomy reference sections.

Two DoD-only columns drive most of the configuration divergence. The two-step phasing means a single BAA carries two distinct requirement sets — white-paper obligations and full-proposal obligations — and the extractor must tag each requirement with the phase it applies to, or a team will validate a white paper against full-proposal rules. And because a BAA’s dollar threshold is frequently expressed per phase or per award instrument rather than as one project cap, the budget detector must retain every distinct threshold rather than collapsing them, so that the conditional cost-reasonableness logic downstream fires on the right number.

# Error Handling and Edge Cases

Real BAAs break naive extractors in predictable ways, and each failure has a specific remedy:

• Scanned or image-only PDFs. When the text layer is empty, pdfplumber returns nothing and every detector silently finds zero requirements — a dangerous false pass. Guard against it by asserting a minimum extracted-character count per page and routing pages below the floor through OCR before classification.
• Cross-referenced obligations. A topic description that reads “subject to the data-rights requirements in Attachment 2” contains a real requirement whose text lives in another file. Detect the reference, record an unresolved requirement with needs_review=True, and resolve it only after the attachment is ingested into the same document set.
• Amendments and modifications. BAAs are amended after release, and an amendment can change a deadline or a page limit without restating the surrounding section. Version every requirement against the announcement revision it came from, and re-run extraction on each amendment rather than diffing prose by hand.
• The fundamental-research exclusion. An export-control marker is not always a binding obligation — much university research qualifies for the fundamental-research exclusion, which lifts ITAR/EAR controls. The detector should flag the trigger but never auto-assert the obligation; that determination is a conditional-rule conflict resolved by policy, not pattern matching.
• Ambiguous or conflicting caps. When two blocks state different dollar limits, do not silently pick one. Emit both, mark the pair for review, and let a human or a downstream rule adjudicate — the same threshold-mismatch discipline enforced across the compliance validation rule engines.

The unifying principle is that the extractor’s job is to surface obligations with honest confidence, not to resolve them. Anything it cannot classify with high confidence should be flagged rather than dropped, because a dropped requirement is invisible while a flagged one is merely one review away from correct.

# Integration with the Downstream Pipeline

Extracted requirements do not stand alone — they are the input to requirement normalization, where conditional gates decide which specialized compliance matrices get injected into the proposal package. A budget-constraint requirement that exceeds the indirect-cost cap pulls in a budget-constraint matrix that ultimately routes to the budget justification format standards engine; an export-control flag pulls in an export matrix. The following diagram traces those conditional gates:

From the normalization stage, the requirement set flows to two consumers. The required section mapping workflow uses it to confirm that every mandated volume and attachment is present, and the automated checklist generation workflow renders it as a human-readable tracker that program staff work against through the proposal lifecycle. Because each requirement carries provenance, every checklist line links back to the exact BAA sentence that created it — the single source of truth that keeps a proposal team from arguing over what the solicitation “really” said.

# Testing and Verification

Extraction quality is only trustworthy when it is regression-tested against known BAA fixtures. The most durable approach is golden-file testing: store a handful of representative announcement excerpts alongside the requirement set each should produce, and assert on categories, counts, confidence, and the review flag. A focused pytest suite catches the two failures that matter most — a detector that stops firing after a pattern edit, and a threshold change that silently stops flagging conditional obligations:

import pytest

BAA_EXCERPT = """
1.0 Technical Scope. Offerors shall address dual-use autonomy applications.

2.0 Security. All key personnel must possess a SECRET clearance prior to award.

3.0 Data Rights. Deliverables are subject to Government Purpose Rights under
DFARS 252.227-7013.

4.0 Budget. The total cost of any single award shall not to exceed $1,500,000.

5.0 Export. Work may be export-controlled under ITAR; offerors must certify status.
"""


def test_all_dod_categories_detected() -> None:
    reqs = extract_requirements(BAA_EXCERPT)
    found = {r.category for r in reqs}
    assert RequirementCategory.SECURITY_CLEARANCE in found
    assert RequirementCategory.EXPORT_CONTROL in found
    assert RequirementCategory.DATA_RIGHTS in found
    assert RequirementCategory.BUDGET_CONSTRAINT in found


def test_budget_value_is_normalized() -> None:
    reqs = extract_requirements(BAA_EXCERPT)
    budget = next(
        r for r in reqs if r.category == RequirementCategory.BUDGET_CONSTRAINT
    )
    assert budget.extracted_value == "1,500,000"


def test_export_control_is_always_flagged() -> None:
    reqs = extract_requirements(BAA_EXCERPT)
    export = next(
        r for r in reqs if r.category == RequirementCategory.EXPORT_CONTROL
    )
    assert export.needs_review is True


def test_provenance_offsets_are_within_bounds() -> None:
    reqs = extract_requirements(BAA_EXCERPT)
    for r in reqs:
        assert 0 <= r.provenance.char_start < r.provenance.char_end

Beyond the automated suite, a manual verification checklist before any extracted set is trusted for a live submission should confirm: every page produced non-empty text (no silent OCR gaps); every conditional category — security, export, data rights — was evaluated even if it produced no match; each requirement’s phase tag (white paper vs full proposal) is set; every low-confidence and cross-referenced item is flagged for review rather than dropped; and the amendment revision is recorded so the set can be re-run if the BAA changes. Passing both the golden-file suite and this checklist is what lets an institution treat the extracted requirements as an audit-ready record rather than a first draft.

# Related

• NIH FOA Schema Mapping — translate templated NIH announcements into typed validation rules
• NSF Proposal Guide Taxonomy — model versioned PAPPG requirements programmatically
• Budget Justification Format Standards — normalize agency financial schemas the extractor feeds
• DoD BAA compliance matrix generation in Python — render extracted requirements into a tracked matrix
• Threshold tuning for compliance — calibrate the confidence thresholds that gate human review
• NLP Section Boundary Detection — segment BAA narrative text before classification

Up: Core Architecture & RFP Taxonomy