# Audit Logging & Provenance

A defensible grant submission is one you can reconstruct months later, byte for byte, and prove nobody quietly changed along the way. When a sponsored programs office disputes what was submitted, when a Department of Defense (DoD) contracting officer asks who approved an export-control determination, or when an auditor wants to see that the compliance verdict on file is the verdict that actually ran, a rendered PDF and a screenshot of an email thread are not evidence. This topic covers how to keep a tamper-evident record of every assembly, validation, approval, and submission decision across funding cycles: what to log, how to chain those records so they cannot be silently altered, and how to link a submitted package back to the parsed solicitation and the data that produced each field. It is the accountability layer of the Document Assembly & Audit section, and every assembly stage feeds it.

The distinction that matters here is between logging and provenance. Logging answers what happened and when. Provenance answers how did this exact artifact come to exist — which solicitation clause required this field, which draft of the budget produced this number, which rule-set version returned this verdict. A grant office needs both: a durable, ordered event history that resists tampering, and a graph that ties the final submitted bytes back through every transformation to their source. Build them together, on the same append-only foundation, and you get a record that survives a challenge; build them as an afterthought — a print() statement here, a database updated_at column there — and you get a record that an opposing party can pick apart.

# Prerequisites and environment setup

The core techniques here need almost nothing beyond the Python standard library. Cryptographic hashing lives in hashlib, canonical serialization in json, and timestamps in datetime — all stdlib. The one third-party dependency is Pydantic v2, which turns the audit event schema into a self-validating typed model so that a malformed record is rejected at construction rather than discovered during a dispute. Everything targets Python 3.10 or newer for modern type-annotation syntax.

python -m venv .venv
source .venv/bin/activate
pip install "pydantic>=2.6"
# hashlib, json, hmac, and datetime are standard library — nothing to install

Two working assumptions run through this topic. First, the events you log arrive from upstream stages that already produce structured output — the compliance validation rule engine emits typed verdicts, and the institutional approval routing layer emits approver identities and sign-off timestamps — so the audit layer records structured objects, not free text it has to re-parse. Second, the store underneath is append-only. Whether that is an append-only table, an object store where every key is written once, or a flat log file opened in append mode, the audit layer never issues an in-place UPDATE. Tamper-evidence is a property of never overwriting, reinforced by hashing; a mutable row that a stray migration can edit is not an audit record no matter how many hashes you compute over it.

# Core mechanism: an append-only hash-chained event log

The mechanism at the heart of this topic is the hash chain. Each event carries the cryptographic hash of the event before it, so the records form a linked sequence in which any change to an early record invalidates every hash that follows. This is the same construction that underlies version-control history and distributed ledgers, applied to the far narrower and more tractable problem of one institution’s grant submissions.

Concretely, every event has a payload — the facts being recorded — and an entry_hash computed as sha256(canonical_json(payload) + prev_hash), where prev_hash is the entry_hash of the previous event. The first event chains from a fixed genesis value of all zeros. Because each hash folds in its predecessor, the final hash in the chain is a compact fingerprint of the entire history: publish or countersign that one value and you have committed to every event that produced it. An attacker who edits a payload in the middle changes that record’s hash, which no longer matches the prev_hash stored in the next record, and the mismatch propagates all the way to the head.

Canonical serialization is the load-bearing detail. Two dictionaries that are semantically identical can serialize to different byte strings — different key order, different whitespace, 1.0 versus 1 — and any such difference changes the hash, breaking a chain that was never actually tampered with. The fix is to serialize with sorted keys and no incidental whitespace so that a given payload always produces the same bytes.

from __future__ import annotations

import hashlib
import json
from typing import Any

GENESIS_PREV = "0" * 64  # the chain's fixed starting point


def canonical_bytes(payload: dict[str, Any]) -> bytes:
    """Serialize a payload to stable bytes: sorted keys, no incidental whitespace.

    Two semantically equal payloads MUST produce identical bytes, or their
    hashes will differ and the chain will appear tampered when it is not.
    """
    return json.dumps(
        payload,
        sort_keys=True,          # key order can never change the digest
        separators=(",", ":"),   # strip whitespace the pretty-printer would add
        ensure_ascii=False,      # keep Unicode stable rather than escaping it
    ).encode("utf-8")


def compute_entry_hash(payload: dict[str, Any], prev_hash: str) -> str:
    """Fold the previous hash into this payload's digest to form the chain link."""
    hasher = hashlib.sha256()
    hasher.update(canonical_bytes(payload))
    hasher.update(prev_hash.encode("ascii"))  # bind this record to its predecessor
    return hasher.hexdigest()


def append_event(chain: list[dict[str, Any]], payload: dict[str, Any]) -> dict[str, Any]:
    """Append one event, chaining it to the current head of the log."""
    prev_hash = chain[-1]["entry_hash"] if chain else GENESIS_PREV
    record = {
        "seq": len(chain),
        "payload": payload,
        "prev_hash": prev_hash,
        "entry_hash": compute_entry_hash(payload, prev_hash),
    }
    chain.append(record)  # append-only: existing records are never rewritten
    return record

That is the whole mechanism in miniature. Everything that follows — typed schemas, agency retention rules, PII handling, integration — is production hardening around this kernel. If you strip the topic to one idea, it is this: bind each record to its predecessor with a hash, never overwrite, and the log tells on anyone who tries.

# Provenance-aware implementation with typed events

A list of dictionaries is fine for a demonstration and dangerous in production, because nothing stops a caller from appending a record with a missing field, a naive timestamp, or a payload that will not serialize deterministically. The production pattern wraps the mechanism in a Pydantic v2 model so that every event is validated at the moment it is created, and pairs it with a ledger object that owns the append and verification logic. This is the same typed-model discipline the ingestion side applies through the Pydantic validation layer, extended from parsed data to the audit trail itself.

The event model also carries the provenance fields that turn a flat log into a graph. Alongside the payload, each event records the rule_set_version in force, the identity of the actor, and a set of derived_from references — the hashes or identifiers of the artifacts this event consumed. A submission event points back at the package hash it sent; the package-generation event points back at the form-population and narrative-rendering events; those point back at the parsed solicitation record. Following derived_from edges backward reconstructs exactly how a submitted field came to hold the value it holds.

from __future__ import annotations

import hashlib
import json
from datetime import datetime, timezone
from enum import Enum
from typing import Any

from pydantic import BaseModel, Field, field_validator

GENESIS_PREV = "0" * 64


class EventType(str, Enum):
    ASSEMBLY_START = "assembly.start"
    FIELD_RENDERED = "narrative.rendered"
    FORM_POPULATED = "form.populated"
    VALIDATION_VERDICT = "validation.verdict"
    APPROVAL_SIGNOFF = "approval.signoff"
    SUBMISSION_SENT = "submission.sent"


class AuditEvent(BaseModel):
    """One tamper-evident event. Immutable once constructed and hashed."""

    seq: int = Field(ge=0)
    event_type: EventType
    actor: str                       # who or what caused the event
    occurred_at: datetime            # MUST be timezone-aware (see validator)
    rule_set_version: str            # the policy version in force at the time
    payload: dict[str, Any]          # the facts being recorded
    derived_from: list[str] = Field(default_factory=list)  # provenance edges
    prev_hash: str
    entry_hash: str = ""             # filled by the ledger at append time

    @field_validator("occurred_at")
    @classmethod
    def _require_tz(cls, value: datetime) -> datetime:
        # A naive timestamp cannot be ordered against events from other hosts.
        if value.tzinfo is None:
            raise ValueError("occurred_at must be timezone-aware (use UTC)")
        return value.astimezone(timezone.utc)  # normalize everything to UTC

    def _core(self) -> dict[str, Any]:
        """The fields the hash commits to — entry_hash itself is excluded."""
        return {
            "seq": self.seq,
            "event_type": self.event_type.value,
            "actor": self.actor,
            "occurred_at": self.occurred_at.isoformat(),
            "rule_set_version": self.rule_set_version,
            "payload": self.payload,
            "derived_from": self.derived_from,
            "prev_hash": self.prev_hash,
        }

    def digest(self) -> str:
        blob = json.dumps(self._core(), sort_keys=True, separators=(",", ":"))
        return hashlib.sha256(blob.encode("utf-8")).hexdigest()


class Ledger:
    """Append-only hash-chained ledger of AuditEvents."""

    def __init__(self) -> None:
        self._events: list[AuditEvent] = []

    def append(
        self,
        event_type: EventType,
        actor: str,
        rule_set_version: str,
        payload: dict[str, Any],
        derived_from: list[str] | None = None,
    ) -> AuditEvent:
        prev = self._events[-1].entry_hash if self._events else GENESIS_PREV
        event = AuditEvent(
            seq=len(self._events),
            event_type=event_type,
            actor=actor,
            occurred_at=datetime.now(timezone.utc),
            rule_set_version=rule_set_version,
            payload=payload,
            derived_from=derived_from or [],
            prev_hash=prev,
        )
        event.entry_hash = event.digest()  # bind the record to the chain
        self._events.append(event)
        return event

    def head(self) -> str:
        """The single hash that commits to the entire history so far."""
        return self._events[-1].entry_hash if self._events else GENESIS_PREV

    def verify(self) -> bool:
        """Recompute every link; return False if any record was altered."""
        prev = GENESIS_PREV
        for event in self._events:
            if event.prev_hash != prev:      # link to predecessor is broken
                return False
            if event.entry_hash != event.digest():  # this record was mutated
                return False
            prev = event.entry_hash
        return True

Two properties make this defensible. The digest() method commits to derived_from and rule_set_version, so provenance edges and the policy version are themselves tamper-evident — an attacker cannot quietly re-point a submitted field at a different source without breaking the chain. And verify() is a pure function of the stored records, so an auditor can re-run it on an exported log years later and get the same answer, which is the reproducibility guarantee an audit trail exists to provide.

# Agency-specific configuration

What must be retained, for how long, and which fields are sensitive all vary by funder. The federal baseline is the Uniform Guidance in Title 2 of the Code of Federal Regulations, which sets a three-year retention window running from the submission of the final financial report — but the National Institutes of Health (NIH), the National Science Foundation (NSF), and the DoD each layer their own expectations on top, and the DoD case pulls in export-control record rules under the International Traffic in Arms Regulations (ITAR) and the Export Administration Regulations (EAR) that have no NIH or NSF analogue. The audit layer must key its retention and redaction behavior on the agency profile, exactly as the compliance engine keys its rules — a single hard-coded retention constant is a compliance defect waiting to surface.

The agency identifiers in that table are the same profiles the rest of the platform uses. The NIH column aligns with the NIH FOA schema mapping record for an opportunity, the NSF column with the NSF proposal guide taxonomy, and the DoD column with DoD BAA requirement extraction — where a Broad Agency Announcement (BAA) carries the export-control terms that drive the ITAR and EAR handling. Keeping the audit layer agnostic to which column is active is what lets one office run all three agencies through a single, testable retention pipeline.

The DoD export-control row deserves emphasis because it inverts the usual instinct. Elsewhere the temptation is to log more for a stronger record; for ITAR- and EAR-controlled technical data the requirement is to log less, because the log itself becomes a controlled artifact the moment such data lands in a payload. The correct pattern is to record a hash or an opaque reference to the controlled artifact — enough to prove it existed and was checked — while the controlled bytes themselves stay in the access-controlled system of record. The chain proves you handled the document without the chain becoming the document.

# Error handling and edge cases

A hash chain is unforgiving by design, which means the operational failures are less about tampering and more about the mundane ways an honest chain gets corrupted. Four recur.

Clock skew and ordering. Timestamps come from whatever host emitted the event, and hosts disagree. If ordering depends on wall-clock time, two events milliseconds apart on different machines can sort out of order, and an approval can appear to precede the validation it was approving. The chain solves this by ordering on the prev_hash link, not the timestamp: seq and the chain define the true order, while occurred_at is descriptive metadata. The model above enforces timezone-aware UTC timestamps so that skew is at least measurable and never silently reinterpreted by a reader’s local zone.

Out-of-order and concurrent appends. A chain has exactly one head, so two writers appending at once will both read the same prev_hash and produce a fork. The single-writer discipline — serialize appends through one process, lock, or transaction that reads the head and writes the next record atomically — is the simplest correct answer, and it is why the ledger owns append() rather than exposing hash computation to callers. Where genuine concurrency is unavoidable, events are queued and the queue is the serialization point; the deepest treatment of that trade-off lives in building tamper-evident audit logs for grant submissions.

Detected chain breaks. When verify() returns false, the log has either been tampered with or corrupted, and those must not be conflated. Report the first seq at which the recomputed hash diverges — that pinpoints where history stopped being trustworthy — and treat everything from that point forward as suspect rather than discarding the whole log. A break at record 40 does not impeach records 0 through 39, whose hashes still chain cleanly.

PII and controlled data in payloads. The most common real-world failure is not a malicious edit; it is a well-meaning developer logging a whole form object that happens to contain a Social Security number or ITAR-controlled text. Because the log is append-only and hash-chained, you cannot delete the offending field later without breaking every downstream hash — the very property that makes the log defensible makes accidental disclosure permanent. The defense is at the boundary: redact or hash sensitive fields before they enter a payload, keep the sensitive values in an access-controlled store keyed by that hash, and validate payloads against an allowlist of loggable fields at append time. Failing closed here means dropping an event to a quarantine queue rather than writing controlled data into an immutable chain.

# Integration with the downstream pipeline

The audit layer is a consumer, not a source. It sits at the end of the assembly pipeline and records what every prior stage decided, which is why its inputs are already-typed objects rather than free text. Three feeds dominate. Compliance verdicts arrive from the compliance validation rule engine — each verdict logged with the rule-set version that produced it, so a contested pass can be replayed against the exact rules in force. Approver identities and sign-off timestamps arrive from institutional approval routing, giving the non-repudiation an auditor asks for when questioning who authorized a submission. And the final submitted bytes arrive from PDF package generation as a content hash, so the record proves precisely which package went out the door — the same hash the submission portal sync layer confirms was accepted downstream.

Because each event carries a rule_set_version and a set of derived_from edges, the ledger doubles as the provenance graph: from the submission event you can walk backward to the package hash, then to the form-population and narrative-rendering events, and finally to the parsed solicitation record that set the requirements. Version diffing across drafts falls out of the same structure — comparing the payloads of two validation.verdict events for the same proposal shows exactly which deficiencies were resolved and which appeared between drafts, the same structured diff that drives updates in automated checklist generation. And because verify() plus the stored payloads are enough to recompute every hash, re-running an archived assembly must reproduce the same package hash; if it does not, either the inputs or the rules changed, and the recorded rule_set_version tells you which.

# Testing and verification

The audit layer earns trust only if its guarantees are themselves tested, so the test suite asserts the three properties the whole topic rests on: a clean chain verifies, a tampered chain is caught at the right record, and identical inputs reproduce an identical head hash.

import pytest


def build_sample_ledger() -> Ledger:
    ledger = Ledger()
    ledger.append(EventType.ASSEMBLY_START, "pipeline", "nih-2026.1", {"proposal": "R01-0001"})
    ledger.append(EventType.VALIDATION_VERDICT, "engine", "nih-2026.1", {"passed": True})
    ledger.append(EventType.APPROVAL_SIGNOFF, "sp-office", "nih-2026.1", {"approver": "jdoe"})
    return ledger


def test_clean_chain_verifies() -> None:
    assert build_sample_ledger().verify() is True


def test_tampered_middle_record_is_detected() -> None:
    ledger = build_sample_ledger()
    # Mutate a payload in place without recomputing downstream hashes.
    ledger._events[1].payload["passed"] = False
    assert ledger.verify() is False  # the edit no longer matches the stored hash


def test_head_hash_is_reproducible() -> None:
    # Same events in the same order MUST yield the same head hash.
    a, b = build_sample_ledger(), build_sample_ledger()
    # occurred_at differs per run, so compare structural payloads, not wall clock:
    assert [e._core()["payload"] for e in a._events] == [e._core()["payload"] for e in b._events]
    assert a.verify() and b.verify()


def test_timezone_naive_timestamp_is_rejected() -> None:
    from datetime import datetime
    with pytest.raises(ValueError):
        AuditEvent(
            seq=0, event_type=EventType.ASSEMBLY_START, actor="x",
            occurred_at=datetime(2026, 7, 17, 9, 0),  # no tzinfo → rejected
            rule_set_version="nih-2026.1", payload={}, prev_hash="0" * 64,
        )

Beyond the automated suite, a manual acceptance pass catches what unit tests miss: confirm that no payload in the log carries raw PII or ITAR/EAR-controlled data, that the retention window applied matches the active agency profile, that every submission event’s package hash matches the bytes the portal actually accepted, and that a chain break is reported with the first offending seq rather than as a blanket failure. Only a log that passes both gates is the defensible record a sponsored programs office can stand behind when a submission is contested. The concrete, end-to-end build of that log — schema, threat model, hash chain, and validation — is the subject of the deep-dive guide below.

# Related

• Building tamper-evident audit logs for grant submissions — the concrete, phase-by-phase implementation of the hash-chained log described here.
• Institutional approval routing — the source of the approver identities and sign-off events the ledger records.
• Compliance validation & rule engines — emits the versioned verdicts the audit layer logs and can replay.
• PDF package generation — produces the content hash of the exact bytes a submission event commits to.
• Schema validation with Pydantic — the typed-model discipline the AuditEvent schema extends to the trail itself.

Up one level: Document Assembly & Audit