CTI Claim Provenance

Question

Do explicit provenance constraints help a model bind an answer to the right frozen source evidence, and do they also improve the exact structured answer? The project keeps those outcomes separate.

Resume highlights

• Built a 64-question, 24-group point-in-time CTI benchmark; constrained output raised evidence binding from 83.3% to 86.5% but lowered exact accuracy from 33.3% to 26.0%; a 144-response follow-up found no material schema benefit.
• Tracked CISA KEV/CSAF, NVD, CVE, MITRE ATT&CK, and vendor source provenance with SHA-256 hashes.
• Enforced point-in-time cutoff and source-authority rules with evidence spans, abstention, and human review.
• Automated typed-schema validation, split isolation, abstention, secret scanning, and privacy-safe aggregate checks.

Engineering highlights

• 64 reviewed questions organized into 24 dependency groups.
• Point-in-time cutoff and source-authority rules with evidence spans and source hashes.
• Typed expected answers, including explicit abstention cases.
• Group-aware split isolation and a documented human-review record.
• A privacy-safe 192-row result projection and offline release verifier.

Method

I designed the benchmark schema, related question families, cutoff and authority rules, evidence packets, typed expected-answer format, abstention cases, review record, split constraints, privacy-safe result projection, aggregate checks, and release verifier. Each question was evaluated with gpt-5.6-luna under a citation-prompted baseline and a constrained pipeline, then scored separately for evidence binding and strict canonical-answer matching.

Observed result

The constrained pipeline had a small gain in selecting and binding the relevant evidence, particularly on temporal questions, but it did not produce a broad accuracy gain: exact canonical-answer matching declined. The result is a tradeoff between evidence selection and answer construction, not proof that either pipeline is generally more accurate.

A targeted 144-response replication compared explicit datatype instructions with strict question-specific schemas. Both conditions produced 72/72 contract-valid responses, while agent-reviewed semantic accuracy differed by one response (67/72 versus 68/72). The follow-up did not show a material schema benefit.

What the public release contains

The public repository contains the reviewed benchmark corpus, evidence packets, human-review record, privacy-safe result projection, and release verifier. It does not include the original provider-run collection and generation pipeline, so the release supports inspection of the benchmark design and verification of the published aggregates rather than a full rerun of the model experiment.

Limits

This is one model, one frozen corpus, and one run per cell. Question families share source dependencies, so the cells are not independent observations. The reported percentages should not be treated as significance-tested or causal estimates.

Project details

Language

Python

Corpus

64 questions in 24 dependency groups

Evaluation

192 completed cells

Release checks

Corpus, splits, documentation, privacy, and aggregates

Measures

Evidence binding and strict answer match