# Extracting DoD SBIR/STTR topic requirements in Python

A Department of Defense (DoD) Small Business Innovation Research (SBIR) or Small Business Technology Transfer (STTR) solicitation is not one requirement set — it is a bundle of dozens of numbered topics, each carrying its own objective, phase structure, eligibility rules, and award ceilings. A small business that proposes against topic AF251-0042 must satisfy that topic’s Phase I cap and Technical Point of Contact (TPOC) window, not the solicitation’s generic front matter, and a pipeline that flattens the whole document into one blob loses the very boundaries that decide responsiveness. This page shows how to parse the topic index, isolate a single topic’s requirements, and structure them into a typed model your compliance layer can check — the topic-level companion to the parent DoD BAA requirement extraction workflow. Getting the topic boundary right is the difference between checking a proposal against real obligations and checking it against noise.

# Phase 1 — Parse the topic index and isolate one topic

DoD SBIR/STTR solicitations open with a topic index — a list mapping each topic number to its title, technology area, and the page where its full description begins. That index is the map; isolating a single topic is the first real transformation, because everything downstream operates on one topic’s text, never the whole 300-page release.

Implementation steps:

1. Recover text with coordinates. SBIR releases arrive as unstructured PDFs, so text must be recovered geometry-first before the index is readable; this reuses the coordinate-aware reader documented in the parent workflow rather than a flat text dump.
2. Match topic numbers, not titles. Topic numbers follow a stable component-coded pattern — a component prefix, a two-digit year, and a sequence number (AF251-0042, N251-018, A251-033). Anchor on that pattern; titles vary in casing and wrapping.
3. Build an index of spans. Map each topic number to the character or page span where its description starts and ends — a topic runs until the next topic number appears. This span map is what makes isolation a slice, not a re-parse.
4. Isolate on demand. Given a topic number, return only that topic’s text. Detecting where one topic ends and the next begins is the same problem solved by NLP section boundary detection, applied to topic boundaries instead of proposal sections.

Pin the environment so a topic extracted today reproduces when the solicitation is re-checked after an amendment:

python -m venv .venv
source .venv/bin/activate
pip install "pdfplumber==0.11.4" "pydantic>=2.6"

Build the topic span index first. The regex anchors on the component-coded number so a title that wraps across lines never fragments the match:

import re
from dataclasses import dataclass

# Component prefix (letters), two-digit year, dash, sequence: AF251-0042, N251-018.
TOPIC_NUMBER = re.compile(r"\b([A-Z]{1,3}\d{2}[A-Z]?-\d{3,4})\b")


@dataclass(slots=True)
class TopicSpan:
    number: str
    start: int          # char offset where this topic's text begins
    end: int            # char offset where the next topic begins (or EOF)


def index_topics(full_text: str) -> list[TopicSpan]:
    """Map every topic number to the text span it owns."""
    hits = [(m.group(1), m.start()) for m in TOPIC_NUMBER.finditer(full_text)]
    # Collapse repeat mentions: keep the first (index) occurrence per number.
    seen: dict[str, int] = {}
    for number, pos in hits:
        seen.setdefault(number, pos)
    ordered = sorted(seen.items(), key=lambda kv: kv[1])
    spans: list[TopicSpan] = []
    for i, (number, start) in enumerate(ordered):
        end = ordered[i + 1][1] if i + 1 < len(ordered) else len(full_text)
        spans.append(TopicSpan(number, start, end))
    return spans


def isolate_topic(full_text: str, spans: list[TopicSpan], number: str) -> str:
    """Return only the text belonging to one topic number."""
    for span in spans:
        if span.number == number:
            return full_text[span.start:span.end]
    raise KeyError(f"topic {number!r} not found in solicitation")

# Phase 2 — Structure a topic into a typed requirement model

Isolated topic text is still prose. The value is a typed record: the phase structure with its award caps, the TPOC, the key dates, and the eligibility flags — each a field a downstream check can assert against. Model it with Pydantic v2 so a malformed extraction fails loudly at construction instead of leaking a None into the compliance layer. SBIR and STTR share this shape; STTR differs mainly in requiring a research-institution partner, which is a flag on the same model.

Core transformation steps:

1. Segment the topic body. Split the isolated text into its labeled subsections — Objective, Description, Phase I, Phase II, Phase III, References — using the topic’s own headers as anchors.
2. Extract phase caps. Pull the Phase I and Phase II dollar ceilings and periods of performance from their subsections; these drive the budget checks downstream.
3. Capture the TPOC and dates. Record the Technical Point of Contact and the question-and-answer and close dates, since a late TPOC question or a missed close is a hard disqualifier.
4. Flag eligibility conditions. Mark ITAR/export-control sensitivity and the STTR research-institution requirement so the eligibility gate can branch on them.

from datetime import date
from pydantic import BaseModel, Field, field_validator


class PhaseCap(BaseModel):
    """Award ceiling and period of performance for one SBIR/STTR phase."""
    phase: str                          # "I" | "II"
    max_award_usd: int
    months: int

    @field_validator("max_award_usd", "months")
    @classmethod
    def _positive(cls, v: int) -> int:
        if v <= 0:
            raise ValueError("award ceiling and period of performance must be positive")
        return v


class TopicRequirement(BaseModel):
    """A single DoD SBIR/STTR topic, structured for compliance checking."""
    number: str
    title: str
    component: str                      # "AF" | "N" | "A" | ...
    program: str                        # "SBIR" | "STTR"
    phase_caps: list[PhaseCap] = Field(default_factory=list)
    tpoc_name: str | None = None
    close_date: date | None = None
    itar_controlled: bool = False
    requires_research_institution: bool = False   # STTR partner requirement

    @field_validator("number")
    @classmethod
    def _valid_topic_number(cls, v: str) -> str:
        if not TOPIC_NUMBER.fullmatch(v):
            raise ValueError(f"malformed topic number: {v!r}")
        return v

    @field_validator("requires_research_institution")
    @classmethod
    def _sttr_needs_partner(cls, v: bool, info) -> bool:
        # STTR always requires a research-institution partner; SBIR never does.
        if info.data.get("program") == "STTR" and not v:
            raise ValueError("STTR topics must require a research-institution partner")
        return v

The diagram traces the full path: from the solicitation PDF, through the topic index, to one isolated topic, fanning into its extracted fields, and converging on the typed model.

# Phase 3 — Component formats, amendments, and export flags

A topic parser that works on one component’s release breaks on the next. DoD SBIR/STTR is administered by components — Air Force (AF), Navy (N), Army (A), and others — and each formats its topics differently, so the extraction rules must branch by component prefix.

Component-specific formats. The Air Force publishes topics through its own portal with distinct subsection headers and open-topic mechanisms; the Navy and Army use the numbered-topic layout more literally. Detect the component from the topic-number prefix and select the matching subsection anchors rather than assuming one header vocabulary. Where a component allows “open” topics with no fixed Phase I cap, model the ceiling as absent, not zero.

Amendments and modifications. Solicitations are amended after release — a close date slips, a topic is withdrawn, a TPOC changes. Never merge an amendment into the base text destructively. Re-run extraction against the amended release and diff the resulting TopicRequirement records, so a moved close date or a pulled topic surfaces as an explicit change your team reviews.

ITAR/export flags and phase transitions. A topic marked as involving International Traffic in Arms Regulations (ITAR) or Export Administration Regulations (EAR) controlled technology gates who may participate, so that flag must be extracted, not inferred later. Phase transitions carry their own rules: a Phase II proposal is generally invited only from a Phase I awardee, and Phase III is not competed under the solicitation at all. Model the phase a proposal targets so the eligibility gate can reject an uninvited Phase II. The per-component thresholds these checks compare against — award ceilings, response windows — belong in how to tune compliance thresholds for DoD BAA, not hard-coded here, and the extracted matrix of obligations feeds DoD BAA compliance matrix generation in Python.

# Phase 4 — Validation and pytest

Topic extraction fails quietly — a shifted header drops a field to None and the proposal sails through a check that never ran. Test that isolation returns the right span, that the phase caps parse, and that the STTR partner rule fires. Use a committed fixture, never a live solicitation, since releases change without notice.

FIXTURE = """
AF251-0042  Autonomous Sensor Fusion
Objective: develop ...
Phase I: maximum $150,000 over 6 months ...
Phase II: maximum $1,250,000 over 24 months ...
N251-018  Undersea Comms
Objective: ...
"""


def test_isolate_returns_only_target_topic() -> None:
    spans = index_topics(FIXTURE)
    body = isolate_topic(FIXTURE, spans, "AF251-0042")
    assert "Autonomous Sensor Fusion" in body
    assert "Undersea Comms" not in body          # next topic must be excluded


def test_topic_span_boundaries_are_contiguous() -> None:
    spans = index_topics(FIXTURE)
    numbers = [s.number for s in spans]
    assert numbers == ["AF251-0042", "N251-018"]


def test_sttr_without_partner_is_rejected() -> None:
    try:
        TopicRequirement(number="A251-033", title="X", component="A",
                         program="STTR", requires_research_institution=False)
    except ValueError:
        pass
    else:
        raise AssertionError("STTR topic without a research-institution partner should fail")


def test_phase_cap_rejects_nonpositive_award() -> None:
    try:
        PhaseCap(phase="I", max_award_usd=0, months=6)
    except ValueError:
        pass
    else:
        raise AssertionError("a zero award ceiling must be rejected")

Round out unit tests with an acceptance checklist: confirm isolation excludes the neighboring topic, that a component-specific header set was selected from the topic-number prefix, that ITAR/EAR flags were captured at parse time rather than inferred, that an amendment produced a reviewable record diff instead of an in-place merge, and that STTR topics carry the research-institution requirement. A topic model that survives every component’s format and every amendment is the only reliable input to the compliance matrix.

# Frequently asked questions

# Related

• DoD BAA compliance matrix generation in Python — turning extracted topic obligations into a traceable matrix.
• How to tune compliance thresholds for DoD BAA — where per-component award caps and response windows are owned.
• NLP section boundary detection — the boundary-detection technique this parser applies to topic edges.
• DoD BAA Requirement Extraction — the parent workflow this topic-level extraction feeds.

Up one level: DoD BAA Requirement Extraction