# How to tune compliance thresholds for a DoD Broad Agency Announcement

A Department of Defense (DoD) Broad Agency Announcement (BAA) restates its own formatting, cost, and export-control rules in every issuance, so a compliance threshold hard-coded for one call is wrong for the next. A margin-drift tolerance that is safe against one component’s PDF toolchain buries the reviewer in false alarms against another; an export-control trigger tuned loosely enough to stay quiet on a domestic proposal will miss a genuine International Traffic in Arms Regulations (ITAR) exposure on the following one. This guide shows how to make those thresholds data-driven — modeled as validated per-BAA data and calibrated against a labeled sample of past submissions — as a specialization of the broader threshold tuning for compliance discipline. The goal is a threshold set for each announcement that catches the defects the DoD screen actually punishes without drowning a proposal team in noise.

Unlike a National Institutes of Health notice, whose formatting envelope is stable across cycles, a BAA is a moving target: its dollar thresholds for cost-reasonableness scrutiny, its classification-marking rules, and its foreign-collaborator disclosure triggers are set per call and per component. That volatility is exactly why the numbers cannot be constants in code — they are hydrated from the extracted solicitation and then calibrated to local ground truth.

# Phase 1 — Model the per-BAA threshold set as validated data

The first move is to give each BAA its own threshold profile rather than sharing one global tolerance. A profile bundles the five dimensions a DoD screen most often turns on: how much margin drift is treated as renderer noise, how much font-size deviation is tolerated, the optical character recognition (OCR) confidence floor below which extracted text is not trusted, the dollar figure above which a cost line draws reasonableness scrutiny, and the sensitivity of the ITAR and Export Administration Regulations (EAR) trigger. Each dimension is hydrated from the requirements the DoD BAA requirement extraction workflow parses out of the announcement, then refined by calibration.

Model the profile as a Pydantic v2 record so a malformed configuration fails at construction instead of silently mis-screening a whole submission window:

from __future__ import annotations

from pydantic import BaseModel, Field, field_validator


class BAAThresholdProfile(BaseModel):
    """Calibrated compliance thresholds for one Broad Agency Announcement."""

    baa_id: str
    margin_drift_pt: float = Field(default=0.75, ge=0.0, le=6.0)      # points of slack
    font_deviation_pt: float = Field(default=0.05, ge=0.0, le=1.0)    # sub-point rounding
    ocr_confidence_floor: float = Field(default=0.90, ge=0.0, le=1.0)
    cost_review_usd: float = Field(default=250_000.0, gt=0.0)         # reasonableness trigger
    export_sensitivity: float = Field(default=0.60, ge=0.0, le=1.0)   # ITAR/EAR trigger

    @field_validator("export_sensitivity")
    @classmethod
    def _sensitivity_leaves_headroom(cls, v: float) -> float:
        # A sensitivity of exactly 1.0 flags every keyword and is never a
        # calibration result; force it below the ceiling so tuning is real.
        if v >= 1.0:
            raise ValueError("export_sensitivity must stay below 1.0 to be tunable")
        return v

    def flags_cost_line(self, amount_usd: float) -> bool:
        return amount_usd >= self.cost_review_usd

    def trusts_ocr(self, page_confidence: float) -> bool:
        return page_confidence >= self.ocr_confidence_floor

Bundling the thresholds in one typed object rather than passing five loose floats has an audit payoff: every screening run records exactly which profile version produced it, so a verdict can always be replayed against the thresholds that were in force. That profile is also what the DoD BAA compliance matrix generation in Python step reads when it decides which requirement rows to mark satisfied, so the two stages never disagree about where a line falls.

# Phase 2 — Calibrate against a labeled sample: the precision/recall trade-off

A threshold chosen by intuition is a guess; a threshold derived from labeled history is a decision with a known error rate. Frame each tunable dimension as a detector and score it on a corpus of past submissions a human already judged — for the export-control trigger, submissions labeled as genuinely needing an ITAR or EAR statement versus those that did not. Two error rates move in opposition as the trigger sensitivity rises: recall (the share of genuine triggers the screen catches) climbs, while precision (the share of flags that were real) falls, because a more sensitive trigger fires on more benign keyword hits. Tuning picks the operating point that satisfies the mandated recall floor at the best precision the data allows.

The calibrator sweeps candidate sensitivities, scores each against the labeled corpus, and returns the loosest setting that still clears the recall floor — loosest because, for a fixed recall, a lower sensitivity means fewer false alarms:

from __future__ import annotations

from dataclasses import dataclass


@dataclass(frozen=True)
class LabeledSubmission:
    """One past submission with the trigger score it received and the truth."""

    trigger_score: float          # detector output in [0, 1]
    truly_flagged: bool           # human ground truth: did it need a statement?


def _precision_recall(
    cases: list[LabeledSubmission], sensitivity: float
) -> tuple[float, float]:
    """A case fires when its score meets the sensitivity threshold."""
    tp = sum(c.truly_flagged and c.trigger_score >= sensitivity for c in cases)
    fp = sum(not c.truly_flagged and c.trigger_score >= sensitivity for c in cases)
    fn = sum(c.truly_flagged and c.trigger_score < sensitivity for c in cases)
    precision = tp / (tp + fp) if (tp + fp) else 1.0
    recall = tp / (tp + fn) if (tp + fn) else 1.0
    return precision, recall


def calibrate_export_sensitivity(
    cases: list[LabeledSubmission],
    recall_floor: float = 0.95,
    step: float = 0.01,
) -> float:
    """Highest sensitivity threshold still meeting the recall floor.

    A higher threshold is stricter about what counts as a hit, so it yields
    the best precision among settings that still catch enough real triggers.
    """
    best = 0.0
    sensitivity = 0.0
    while sensitivity < 1.0:
        _, recall = _precision_recall(cases, round(sensitivity, 4))
        if recall >= recall_floor:
            best = sensitivity          # still catching enough; keep tightening
        else:
            break                       # tighter would drop below the floor
        sensitivity += step
    return best

The recall_floor=0.95 default encodes a policy, not a preference: for export control a missed exposure is a categorical failure, so the calibrator will spend precision freely to keep recall high and let the surplus false positives flow to a human queue. For a benign dimension like margin drift, the same machinery runs with the objective inverted — there, precision matters more and the floor moves. The point is that every threshold becomes an explicit, reviewable argument fitted to data rather than a literal buried in screening code.

# Phase 3 — DoD-specific overrides: classification markings and foreign-collaborator triggers

Two DoD dimensions do not tune like the others because a miss on either is unrecoverable, so their thresholds are set by rule rather than by the precision/recall sweep.

• Classification markings. A BAA that permits or requires controlled unclassified information imposes portion-marking rules whose detector runs at maximum sensitivity by design — a single unmarked controlled paragraph is a hard stop, so there is no false-positive budget to optimize against. The threshold here is effectively fixed, and the calibrator is bypassed with a recorded justification.
• Foreign-collaborator triggers. Disclosure of foreign affiliations and support is set per BAA and per component, and the trigger keys on named-entity and country signals rather than a formatting delta. Its sensitivity floor is raised whenever the announcement cites foreign-influence or research-security language, and that adjustment is hydrated from the extracted requirements, not chosen locally.
• Per-component dollar thresholds. The cost-reasonableness figure that flags a budget line varies by component and by contract vehicle, so cost_review_usd is populated from the solicitation, never defaulted across BAAs.

The table contrasts how each dimension is set, and against a stable NIH baseline to make the DoD volatility explicit.

Because these overrides originate in the announcement text, the resolved profile records the provenance of every threshold, so a reviewer reading the DoD BAA compliance matrix generation in Python output can see why a given number is what it is rather than trusting it blind.

# Phase 4 — Validation and regression tests

Because the profile is validated data and the calibrator is pure numeric logic, both test deterministically against a fixed labeled corpus — no live BAA exports, which change per call.

import pytest
from pydantic import ValidationError


def test_saturated_export_sensitivity_is_rejected() -> None:
    with pytest.raises(ValidationError):
        BAAThresholdProfile(baa_id="HR001125S0001", export_sensitivity=1.0)


def test_calibrator_meets_the_recall_floor() -> None:
    cases = [
        LabeledSubmission(trigger_score=0.30, truly_flagged=False),
        LabeledSubmission(trigger_score=0.72, truly_flagged=True),
        LabeledSubmission(trigger_score=0.88, truly_flagged=True),
        LabeledSubmission(trigger_score=0.55, truly_flagged=False),
    ]
    sensitivity = calibrate_export_sensitivity(cases, recall_floor=0.95)
    _, recall = _precision_recall(cases, sensitivity)
    assert recall >= 0.95            # never drops a real trigger below the floor


def test_cost_trigger_uses_the_per_baa_figure() -> None:
    profile = BAAThresholdProfile(baa_id="N0001425S0002", cost_review_usd=500_000.0)
    assert profile.flags_cost_line(600_000.0)
    assert not profile.flags_cost_line(400_000.0)


def test_low_confidence_ocr_page_is_distrusted() -> None:
    profile = BAAThresholdProfile(baa_id="FA865025S0003", ocr_confidence_floor=0.92)
    assert not profile.trusts_ocr(0.85)   # routes to manual review, not a silent pass

Beyond the unit suite, a pre-submission calibration checklist keeps the thresholds honest against each new announcement:

• Every threshold in the resolved profile traces to a BAA clause or a calibrated value, never a bare literal in screening code.
• Re-scoring the labeled corpus under a pinned profile version reproduces the same precision and recall — no silent drift between runs.
• The export-control trigger clears its mandated recall floor on a holdout sample, and the classification-marking detector runs at fixed maximum sensitivity.
• A brand-new BAA with no labeled history falls back to the extracted-requirement defaults and is marked provisional until real submissions accumulate.

The calibrated profile does not act alone: its verdicts flow into automated checklist generation, which turns each flagged dimension into a routed deficiency line, and its provenance is what lets a program office defend a threshold under audit. Tuning thresholds to labeled DoD history rather than guessing keeps the screen both strict where a miss is unrecoverable and quiet where a flag would only erode trust.

# Frequently asked questions

# Related

• DoD BAA requirement extraction — parses the announcement clauses that hydrate each threshold.
• DoD BAA compliance matrix generation in Python — reads the resolved profile to mark requirement rows satisfied.
• Automated checklist generation — turns each flagged dimension into a routed, human-readable deficiency line.
• Threshold tuning for compliance — the general calibration discipline this DoD build specializes.

Up one level: Threshold Tuning for Compliance