# NLP Section Boundary Detection

Federal funding announcements, program solicitations, and agency-specific guidance documents are structurally heterogeneous by design. Unlike standardized commercial contracts, solicitations from the National Institutes of Health (NIH), the National Science Foundation (NSF), and the Department of Defense (DoD) frequently interleave administrative requirements, scientific evaluation criteria, budgetary constraints, and compliance mandates within dense, multi-layered prose. For research administrators, grant writers, and university technology teams, manually mapping these documents to internal proposal templates introduces unacceptable latency and compliance risk. Section boundary detection solves this structural ambiguity by algorithmically identifying where one regulatory segment terminates and another begins, transforming unstructured text into discrete, machine-actionable units. Within the broader RFP Ingestion & Parsing Workflows pipeline, this stage is the structural backbone that makes automated requirement routing, compliance gap analysis, and dynamic proposal assembly possible — and getting it wrong means every downstream validation rule fires against the wrong span of text.

# Prerequisites and environment setup

This workflow assumes Python 3.10 or newer, since the type annotations and the datetime.timezone usage in the pipeline below rely on modern typing behavior. The three load-bearing dependencies are the natural language toolkit, the data-validation layer, and the upstream text extractor:

python -m pip install "spacy>=3.7" "pydantic>=2.6" "pdfplumber>=0.11"
python -m spacy download en_core_web_sm

The en_core_web_sm pipeline supplies tokenization and sentence segmentation; the section classifier itself is a custom spancat (span categorizer) component you train on annotated solicitations, covered in Training spaCy for grant proposal section detection.

Document assumptions matter as much as package versions. This stage expects already-extracted plain text with preserved reading order, not raw PDF bytes. NIH Funding Opportunity Announcements (FOAs) and NSF program solicitations arrive as multi-column PDFs with running headers; DoD Broad Agency Announcements (BAAs) frequently ship as hybrid PDF/HTML packages. Feed all three through PDF Text Extraction with pdfplumber first so that headers, footers, and marginalia are stripped and logical reading order is reconstructed before any token reaches the classifier.

# Core mechanism

Section boundary detection is a document segmentation and sequence labeling problem. Traditional implementations rely on deterministic pattern matching, scanning for capitalized headings, numeric enumerations, or agency boilerplate phrases. While computationally inexpensive, heuristic pipelines degrade rapidly when confronted with stylistic variability across funding mechanisms, or when an agency deviates from its historical formatting conventions between one solicitation cycle and the next. Production-grade systems increasingly frame the task as token-level classification with explicit start and continuation markers, so that a boundary is a learned property of the surrounding language rather than a brittle regex on line shape.

The reliability of any boundary detector is fundamentally constrained by the fidelity of its upstream text extraction layer. By carrying bounding-box metadata alongside the raw character stream, engineers can feed spatially contextualized tokens into the classifier, which sharply improves precision on documents with irregular pagination or nested subsection numbering. The diagram below traces how a federal PDF moves from raw bytes to labeled section boundaries through this dependency.

Internally, the classifier operates over a labeling scheme. A BILOU tagging convention (Begin, Inside, Last, Outside, Unit) marks each token’s role within a candidate section span, which lets the model distinguish a genuine Begin for a “Budget Justification” heading from an inline cross-reference to that same phrase elsewhere in the prose. The training data therefore associates character offsets with a section label rather than free text:

# Annotation shape consumed by the spancat training loop.
# Each span is (start_char, end_char, label) over the extracted text.
training_example: dict[str, object] = {
    "text": "Budget Justification. Applicants must itemize personnel ...",
    "spans": {
        "sc": [  # "sc" is the default spancat spans key
            (0, 20, "BUDGET_JUSTIFICATION"),   # heading + boundary
            (22, 58, "BUDGET_NARRATIVE"),
        ]
    },
}

Two lexically identical strings — a “Budget Justification” heading versus a sentence that merely references the budget justification — share the same tokens but occupy distinct structural roles. The span categorizer resolves that ambiguity from context, which is precisely what deterministic pattern matching cannot do.

# Coordinate-aware implementation

The production pattern ingests spatially extracted text, applies the trained span categorizer, and normalizes the model output into typed records before anything downstream touches it. Spans emitted by spancat live in doc.spans["sc"], and each span exposes .label_, .text, .start_char, and .end_char:

import spacy
from spacy.language import Language

def detect_boundaries(extracted_text: str, model: Language) -> list[dict[str, object]]:
    """
    Detect section boundaries using a spancat-trained spaCy model.
    Spans are stored in doc.spans["sc"] by the spancat component.
    See: Training spaCy for grant proposal section detection.
    """
    doc = model(extracted_text)
    boundaries: list[dict[str, object]] = []
    for span in doc.spans.get("sc", []):
        boundaries.append({
            "title": span.label_,
            "content": span.text,
            "start_char": span.start_char,
            "end_char": span.end_char,
            "type": "start",
        })
    # Sort by document position so downstream routing sees sections in reading order.
    boundaries.sort(key=lambda b: b["start_char"])
    return boundaries

Identifying boundaries is only half the job; each extracted segment must then be coerced into a strict, typed structure before it can be routed to a compliance framework. This is where the boundary detector hands off to the Schema Validation with Pydantic layer. Modeling every section as a Pydantic v2 model turns the boundary output into an automated compliance checkpoint — malformed titles, empty content, and disallowed boundary types are rejected at the type boundary rather than surfacing as silent corruption three stages later:

from datetime import datetime, timezone
from pydantic import BaseModel, Field, field_validator

_ALLOWED_BOUNDARY_TYPES = {"start", "end", "inline"}

class GrantSection(BaseModel):
    section_id: str
    title: str
    boundary_type: str  # "start", "end", or "inline"
    content: str
    compliance_flags: list[str] = Field(default_factory=list)
    extracted_at: datetime = Field(
        default_factory=lambda: datetime.now(timezone.utc)
    )

    @field_validator("boundary_type")
    @classmethod
    def _known_boundary(cls, v: str) -> str:
        if v not in _ALLOWED_BOUNDARY_TYPES:
            raise ValueError(f"unknown boundary_type: {v!r}")
        return v

    @field_validator("content")
    @classmethod
    def _non_empty(cls, v: str) -> str:
        if not v.strip():
            raise ValueError("section content must not be empty")
        return v

class ComplianceReport(BaseModel):
    rfp_id: str
    sections: list[GrantSection]
    validation_status: str
    missing_requirements: list[str]


def validate_and_route(
    rfp_id: str, boundaries: list[dict[str, object]]
) -> ComplianceReport:
    validated_sections: list[GrantSection] = []
    missing: list[str] = []

    for idx, b in enumerate(boundaries):
        try:
            section = GrantSection(
                section_id=f"SEC-{idx + 1:03d}",
                title=str(b["title"]),
                boundary_type=str(b["type"]),
                content=str(b["content"]),
            )
            # Example rule: flag budget sections missing a cost justification.
            if "budget" in section.title.lower() and "cost" not in section.content.lower():
                section.compliance_flags.append("MISSING_COST_JUSTIFICATION")
            validated_sections.append(section)
        except ValueError as exc:
            missing.append(f"Failed validation near boundary {idx}: {exc}")

    return ComplianceReport(
        rfp_id=rfp_id,
        sections=validated_sections,
        validation_status="PASS" if not missing else "PARTIAL",
        missing_requirements=missing,
    )

# Execution flow
# nlp = spacy.load("./output/model-best")   # trained spancat pipeline
# raw_text = extract_reading_order(pdf_path) # from the pdfplumber stage
# bounds = detect_boundaries(raw_text, nlp)
# report = validate_and_route("NIH-RFA-2024-001", bounds)
# print(report.model_dump_json(indent=2))

The label vocabulary the model emits should match the section taxonomy your institution already tracks. For NIH mechanisms that taxonomy is derived from the NIH FOA Schema Mapping process; for NSF it follows the NSF Proposal Guide taxonomy. Aligning the classifier’s labels to those canonical section names means a detected boundary can be routed directly to the correct validation rule without an intermediate translation table.

# Agency-specific configuration

The section vocabulary and the textual cues that signal a boundary differ substantially across the three funding bodies. A single classifier can serve all three, but its label set and post-processing thresholds should be configured per agency, because a “Project Description” boundary in an NSF solicitation is not interchangeable with a “Research Strategy” boundary in an NIH FOA or a “Technical Volume” boundary in a DoD BAA.

DoD BAAs warrant the tightest span-confidence threshold because their layouts are the least standardized; the extraction and normalization patterns that make those documents tractable are detailed under DoD BAA Requirement Extraction. Loading these parameters from a per-agency config keeps the classifier code identical across mechanisms:

from pydantic import BaseModel, Field

class AgencyBoundaryConfig(BaseModel):
    agency: str
    span_confidence: float = Field(ge=0.0, le=1.0)
    canonical_sections: list[str]
    max_nesting_depth: int

AGENCY_CONFIG: dict[str, AgencyBoundaryConfig] = {
    "NIH": AgencyBoundaryConfig(
        agency="NIH", span_confidence=0.60, max_nesting_depth=2,
        canonical_sections=["Specific Aims", "Research Strategy", "Budget Justification"],
    ),
    "NSF": AgencyBoundaryConfig(
        agency="NSF", span_confidence=0.55, max_nesting_depth=3,
        canonical_sections=["Project Summary", "Project Description", "Budget Justification"],
    ),
    "DoD": AgencyBoundaryConfig(
        agency="DoD", span_confidence=0.70, max_nesting_depth=4,
        canonical_sections=["Technical Volume", "Cost Volume", "Statement of Work"],
    ),
}

# Error handling and edge cases

Boundary detection fails in characteristic ways, and each failure mode has a concrete remediation:

• Column bleed from extraction. When the upstream extractor interleaves text from two columns, the classifier sees interrupted sentences and emits fragmented spans. Resolution: reconstruct reading order at the pdfplumber stage using bounding-box x0 coordinates before the text reaches the model — never patch it here.
• Overlapping spans. spancat, unlike a strict named-entity recognizer, can emit overlapping spans for the same tokens. When two candidate sections claim the same offsets, keep the higher-scoring span and demote the other to boundary_type="inline" so it is recorded but not routed as a section start.
• Sub-threshold headings. A genuine heading scoring just below the agency confidence threshold is silently dropped, producing a section that swallows its successor. Resolution: log every span within 0.05 of the threshold and surface it for human review rather than discarding it outright.
• Nested subsection collisions. NSF documents nest three levels deep; a flat label set collapses a subsection into its parent. Resolution: encode depth in the label (for example PROJECT_DESCRIPTION_L2) and cap it at the agency’s max_nesting_depth.
• Repeated boilerplate. Agency certifications and standard assurances recur verbatim across sections and trigger duplicate boundaries. Resolution: deduplicate by normalized content hash before routing, retaining the first occurrence in reading order.

Because validate_and_route raises on empty or mistyped spans rather than swallowing them, these edge cases become explicit missing_requirements entries in the ComplianceReport instead of undetectable data loss — the report’s PARTIAL status is the signal that a document needs a second pass.

# Integration with downstream pipeline

Validated GrantSection records are the interface contract between this stage and everything that follows. Routing them correctly is what lets the rest of the system stay deterministic. Each typed section feeds three consumers: the compliance validation rule engines that check page limits, fonts, and mandatory content; the required section mapping layer that confirms no mandated section is absent; and the document assembler that stitches approved content into the submission package. At high solicitation volume, that fan-out is handled off the request path by the Async Batch Processing for Large RFPs workers, which decouple extraction, detection, and validation into non-blocking pools while preserving a deterministic audit trail per document.

The routing key is the section’s title label, mapped through the same agency taxonomy used to train the classifier, so a detected “Budget Justification” boundary lands on the budget rule set without a lookup table in between.

# Testing and verification

Boundary detection regressions are silent — the pipeline still runs, it just labels the wrong spans — so verification has to assert on span positions and types, not merely that the code executes. A focused pytest suite pins the contract:

import pytest
from pydantic import ValidationError

def test_boundaries_sorted_by_position() -> None:
    raw = [
        {"title": "B", "content": "second", "type": "start", "start_char": 40},
        {"title": "A", "content": "first", "type": "start", "start_char": 0},
    ]
    raw.sort(key=lambda b: b["start_char"])
    assert [b["title"] for b in raw] == ["A", "B"]

def test_empty_content_is_rejected() -> None:
    with pytest.raises(ValidationError):
        GrantSection(section_id="SEC-001", title="Budget",
                     boundary_type="start", content="   ")

def test_unknown_boundary_type_is_rejected() -> None:
    with pytest.raises(ValidationError):
        GrantSection(section_id="SEC-001", title="Budget",
                     boundary_type="middle", content="text")

def test_budget_without_cost_is_flagged() -> None:
    report = validate_and_route("NIH-TEST-001", [
        {"title": "Budget Justification", "content": "personnel and travel",
         "type": "start"},
    ])
    assert "MISSING_COST_JUSTIFICATION" in report.sections[0].compliance_flags
    assert report.validation_status == "PASS"

Before promoting a retrained model to production, confirm the following on a held-out set of real solicitations:

• Every mandated section for the agency’s mechanism produces exactly one start boundary.
• No two start boundaries share overlapping character offsets after post-processing.
• Span-confidence scores sit above the agency threshold, with borderline spans logged for review.
• ComplianceReport.validation_status is PASS on documents known to be complete and PARTIAL on documents with a deliberately removed section.
• Round-tripping report.model_dump_json() back through ComplianceReport.model_validate_json() reproduces the record exactly.

# Related

• PDF Text Extraction with pdfplumber — the upstream reading-order extraction this stage depends on
• Schema Validation with Pydantic — the typed validation layer that hardens boundary output
• Async Batch Processing for Large RFPs — scaling detection across high-volume grant cycles
• Training spaCy for grant proposal section detection — annotating and training the span categorizer this page uses
• Compliance validation rule engines — the downstream consumer of routed sections

Up one level: RFP Ingestion & Parsing Workflows