Source Ablation Protocol for Testing AI Citation

Source ablation is a paired-input experiment for testing whether an AI answer changes when a specific evidence passage is removed, replaced, or controlled. The protocol does not prove source use by itself. It separates answer change, citation change, and factual-support change so teams can report uncertainty instead of turning citation presence into causal proof.

This is an unexecuted Para Labs protocol. It is not a finding about any current model, vendor, search engine, or retrieval system. Use it when an AI answer cites a source and the research question is narrower than visibility: did that answer materially depend on the evidence supplied to it?

Source ablation tests evidence dependence, not ordinary AI visibility

A source-ablation test asks whether a cited or supplied passage makes a measurable difference to an answer under controlled conditions. That is different from measuring whether a source was cited. The public Machine Relations Index reports source-domain citation rates across six answer engines; its September 12, 2026 public dataset covers 21,452 cited domains, 120,136 citation events, 15,154 answer runs, and 863 monitored prompts. Those measurements show which domains appear in citations, not whether any single cited passage caused a specific answer.

The distinction matters because generative answer systems can synthesize information from multiple sources. The Princeton-led GEO paper defines generative engines as systems that gather and summarize information to answer user queries, then introduces visibility measurement for source inclusion in generated responses (arXiv:2311.09735). Visibility measurement is useful, but it does not resolve causal dependence.

A clean ablation protocol keeps that boundary intact. It can say, "the output changed when this passage was removed." It should not say, "the model ignored the evidence" just because the recommendation stayed similar. Redundant sources, prior model knowledge, stochastic generation, cached retrieval, prompt sensitivity, and equivalent replacement evidence can all preserve an answer after one passage disappears.

Paired-input preregistration template for source ablation

The preregistration should lock the query, evidence fixture, model setting, scoring rubric, and inference limit before any output is inspected. Source attribution research has noted that assigning influence to retrieved documents in RAG systems is difficult and can be computationally expensive, especially when each utility evaluation requires another model call (arXiv:2507.04480). That is why a lightweight editorial protocol needs preregistration discipline instead of after-the-fact interpretation.

Use this template before running the test:

Field	Locked value to preregister	Why it matters
Research question	The exact claim being tested, such as whether Passage A supports Recommendation X	Prevents post-hoc expansion from citation presence to source causality
Model or engine identity	Provider, product surface, model name if available, date, region, and retrieval mode	Makes repeat runs comparable and records black-box limits
Prompt pair	Control prompt and ablation prompt, identical except for the evidence treatment	Keeps the treatment from becoming a wording experiment
Evidence fixture	Source passages, URLs, publication dates, snippet boundaries, and source IDs	Lets annotators inspect support at claim level
Randomization plan	Order of paired prompts, repetitions, temperature if configurable, and seed if available	Reduces run-order and stochastic-output artifacts
Primary outcome	Answer-change label, citation-change label, and factual-support-change label	Prevents one metric from standing in for the whole result
Inference boundary	What the test can and cannot conclude	Blocks causal overclaiming from a small black-box experiment

The minimum paired input is simple: one prompt receives the full evidence fixture, and the matched prompt receives the same fixture with the target passage removed. For stronger evidence, add a replacement condition and a sham-deletion condition rather than only full versus removed.

Controls should test redundancy, sham deletion, and replacement evidence

A source-ablation result is interpretable only when the control cells rule out obvious alternative explanations. Anthropic's Claude citation documentation is a useful platform example because it separates provided documents, citation pointers, and citation quality rather than treating a free-form citation string as enough; Anthropic says its citations feature parses citations into response formats and points to provided documents (Anthropic documentation). Even when a tool exposes citation pointers, a research protocol still has to test whether the answer content depends on the cited source.

A practical source-ablation matrix should include these cells:

Cell	Evidence treatment	What it tests	Interpretable outcome
Full fixture	All approved passages included	Baseline answer, citations, and support	Establishes the comparison output
Target deletion	Remove the passage whose influence is being tested	Sensitivity to the target source	Change suggests dependence but does not prove exclusivity
Sham deletion	Remove an irrelevant or non-supporting passage with similar length	Prompt-length and editing artifacts	Similar output should remain stable if deletion alone is not driving change
Replacement evidence	Replace target passage with a different source that supports the same fact	Redundant support	Stable answer with changed citation can indicate substitutable evidence
Contradictory sentinel	Add a clearly marked conflicting passage only when ethically safe and internal	Susceptibility to wrong evidence	Requires careful reporting; do not use deceptive public prompts
No-source baseline	Ask the same question with no supplied fixture when allowed	Prior knowledge or retrieval outside the fixture	Stable answer may reflect background knowledge, not proof the target was unused

Run cells in randomized order and repeat them enough times to see whether variation is larger than the treatment effect. If the model surface does not expose temperature, seed, retrieval state, or model version, record that as a limitation instead of pretending the setting is fixed.

Annotation rubric: answer change, citation change, and factual support change

Annotators should score the answer, the citation, and the support relationship as separate objects. RAG attribution work makes the same separation visible: the MIRAGE paper frames verifiability as a challenge and notes that self-citing LLMs can fail to faithfully reflect context usage, even when they generate citations (ACL Anthology, EMNLP 2024). A citation marker and a supported claim are not the same unit.

Use three labels for each paired run:

Dimension	Label	Definition	Evidence required
Answer change	Same recommendation	The main answer or recommendation is materially unchanged	Matched conclusion and no material change to ranked options, caveats, or action
Answer change	Changed recommendation	The answer changes the selected option, ranking, caveat, or decision	Before/after outputs differ in a way a reader would act on
Answer change	Changed framing only	The conclusion stays stable but the explanation, caveat, or confidence changes	Wording change affects reasoning but not the decision
Citation change	Same cited source	The target source is still cited or named after deletion	Must inspect whether retrieval outside the fixture reintroduced it
Citation change	Citation substitution	A different source is cited for the same claim	Replacement may mean redundancy, not absence of dependence
Citation change	Citation loss	The claim remains but the citation disappears	Support must be judged separately before calling it unsupported
Factual support	Supported	The cited or supplied evidence entails the claim at the needed granularity	Annotator can point to the supporting passage
Factual support	Partially supported	The source supports a weaker or adjacent claim	Report the exact unsupported portion
Factual support	Unsupported	The answer asserts a material claim not supported by the cited or supplied source	Do not infer intent; label the support relation only

Disagreements should be adjudicated at the claim level. A second annotator should be able to reproduce the label from the prompt, source fixture, answer text, citation list, and scoring sheet without knowing which cell was expected to win.

Null-result reporting should preserve causal humility

A null result in source ablation means the selected outcome did not move under the tested condition, not that the source had no role. The Nature paper on retrieval-augmented scientific synthesis evaluated organization, factual precision, citation quality, and use of supporting excerpts as separate quality components, and reported human-versus-LLM judge agreement rather than collapsing the assessment into one pass/fail verdict (Nature, 2026). Source-ablation reports should do the same kind of denominator discipline.

Use this null-result template:

In this preregistered paired-input test, removing Passage A did not materially change the primary answer label across the observed runs. Citation behavior and factual-support labels were scored separately. This result does not prove Passage A was unused. Plausible explanations include redundant evidence in the fixture, prior model knowledge, retrieval outside the supplied context, stochastic variation, prompt insensitivity, or replacement evidence. The valid inference is limited to the tested prompt, source fixture, model surface, date, and scoring rubric.

For a positive result, use the same caution:

Removing Passage A changed the preregistered answer label and factual-support label under the tested conditions. This is evidence of sensitivity to the evidence treatment, not proof that Passage A was the only cause of the original answer.

That reporting language is the point of the protocol. The goal is not to produce dramatic gotchas. It is to stop teams from making two common mistakes: treating a citation as proof of causal source use, or treating an unchanged answer as proof that the source was ignored.

Machine Relations implication: citation measurement needs intervention tests

Machine Relations treats AI visibility as a source-condition problem: brands need evidence that machines can retrieve, cite, and represent without ambiguity. Source ablation adds a narrower experimental layer. It helps teams test whether a particular source condition changes an observed answer, while the broader share of citation metric tracks how often sources appear across answer runs.

The operating sequence is therefore: measure citation occurrence, identify the claim that matters, build a source fixture, preregister the paired-input test, annotate answer/citation/support separately, and report the inference boundary. AuthorityTech can use this kind of protocol in AI visibility work because it avoids the false certainty that weak measurement systems create.

Teams that want a faster external map before designing intervention tests can run an AI visibility audit, then use source ablation only for the sources and claims whose causal role is worth testing.

FAQ

What is source ablation for AI citations?

Source ablation is a paired-input test that removes, replaces, or controls one evidence passage to see whether an AI answer, citation, or support label changes. It is a method for testing sensitivity to evidence, not a standalone proof of what a model internally used.

Does an unchanged AI answer prove the removed source was ignored?

No. An unchanged answer can come from redundant sources, prior model knowledge, retrieval outside the supplied fixture, stochastic variation, or a prompt that is insensitive to the evidence treatment. The correct null-result claim is narrower: the preregistered outcome did not move under the tested condition.

Why separate answer change from citation change?

Answer change, citation change, and factual-support change answer different questions. A model can keep the same recommendation but cite a different source, lose a citation while preserving a claim, or change a citation without changing factual support. Treating those as one metric creates false certainty.

How does source ablation fit with Machine Relations?

Machine Relations measures and improves the evidence conditions under which AI systems cite and recommend brands. Source ablation is one experimental tool inside that system: it tests whether a specific evidence condition appears to affect a specific observed answer under a locked protocol.