# Source Ablation Protocol for Testing AI Citation Dependence

> A preregistered paired-input protocol for testing whether an AI answer depends on supplied evidence, with controls, annotation labels, and null-result reporting.

- Published: 2026-09-13
- URL: https://paralabs.ai/blog/source-ablation-ai-citation-dependence-preregistration
- Tags: ai-visibility, measurement, experiments

---

Source ablation is a paired-input experiment for testing whether an AI answer changes when a specific evidence passage is removed, replaced, or controlled. The protocol does not prove source use by itself. It separates answer change, citation change, and factual-support change so teams can report uncertainty instead of turning citation presence into causal proof.

This is an unexecuted Para Labs protocol. It is not a finding about any current model, vendor, search engine, or retrieval system. Use it when an AI answer cites a source and the research question is narrower than visibility: did that answer materially depend on the evidence supplied to it?

## Source ablation tests evidence dependence, not ordinary AI visibility

**A source-ablation test asks whether a cited or supplied passage makes a measurable difference to an answer under controlled conditions.** That is different from measuring whether a source was cited. The public [Machine Relations Index](https://machinerelations.ai/index) reports source-domain citation rates across six answer engines; its September 12, 2026 public dataset covers 21,452 cited domains, 120,136 citation events, 15,154 answer runs, and 863 monitored prompts. Those measurements show which domains appear in citations, not whether any single cited passage caused a specific answer.

The distinction matters because generative answer systems can synthesize information from multiple sources. The Princeton-led GEO paper defines generative engines as systems that gather and summarize information to answer user queries, then introduces visibility measurement for source inclusion in generated responses ([arXiv:2311.09735](https://arxiv.org/abs/2311.09735)). Visibility measurement is useful, but it does not resolve causal dependence.

A clean ablation protocol keeps that boundary intact. It can say, "the output changed when this passage was removed." It should not say, "the model ignored the evidence" just because the recommendation stayed similar. Redundant sources, prior model knowledge, stochastic generation, cached retrieval, prompt sensitivity, and equivalent replacement evidence can all preserve an answer after one passage disappears.

## Paired-input preregistration template for source ablation

**The preregistration should lock the query, evidence fixture, model setting, scoring rubric, and inference limit before any output is inspected.** Source attribution research has noted that assigning influence to retrieved documents in RAG systems is difficult and can be computationally expensive, especially when each utility evaluation requires another model call ([arXiv:2507.04480](https://arxiv.org/abs/2507.04480)). That is why a lightweight editorial protocol needs preregistration discipline instead of after-the-fact interpretation.

Use this template before running the test:

| Field | Locked value to preregister | Why it matters |
|---|---|---|
| Research question | The exact claim being tested, such as whether Passage A supports Recommendation X | Prevents post-hoc expansion from citation presence to source causality |
| Model or engine identity | Provider, product surface, model name if available, date, region, and retrieval mode | Makes repeat runs comparable and records black-box limits |
| Prompt pair | Control prompt and ablation prompt, identical except for the evidence treatment | Keeps the treatment from becoming a wording experiment |
| Evidence fixture | Source passages, URLs, publication dates, snippet boundaries, and source IDs | Lets annotators inspect support at claim level |
| Randomization plan | Order of paired prompts, repetitions, temperature if configurable, and seed if available | Reduces run-order and stochastic-output artifacts |
| Primary outcome | Answer-change label, citation-change label, and factual-support-change label | Prevents one metric from standing in for the whole result |
| Inference boundary | What the test can and cannot conclude | Blocks causal overclaiming from a small black-box experiment |

The minimum paired input is simple: one prompt receives the full evidence fixture, and the matched prompt receives the same fixture with the target passage removed. For stronger evidence, add a replacement condition and a sham-deletion condition rather than only full versus removed.

## Controls should test redundancy, sham deletion, and replacement evidence

**A source-ablation result is interpretable only when the control cells rule out obvious alternative explanations.** Anthropic's Claude citation documentation is a useful platform example because it separates provided documents, citation pointers, and citation quality rather than treating a free-form citation string as enough; Anthropic says its citations feature parses citations into response formats and points to provided documents ([Anthropic documentation](https://platform.claude.com/docs/en/build-with-claude/citations)). Even when a tool exposes citation pointers, a research protocol still has to test whether the answer content depends on the cited source.

A practical source-ablation matrix should include these cells:

| Cell | Evidence treatment | What it tests | Interpretable outcome |
|---|---|---|---|
| Full fixture | All approved passages included | Baseline answer, citations, and support | Establishes the comparison output |
| Target deletion | Remove the passage whose influence is being tested | Sensitivity to the target source | Change suggests dependence but does not prove exclusivity |
| Sham deletion | Remove an irrelevant or non-supporting passage with similar length | Prompt-length and editing artifacts | Similar output should remain stable if deletion alone is not driving change |
| Replacement evidence | Replace target passage with a different source that supports the same fact | Redundant support | Stable answer with changed citation can indicate substitutable evidence |
| Contradictory sentinel | Add a clearly marked conflicting passage only when ethically safe and internal | Susceptibility to wrong evidence | Requires careful reporting; do not use deceptive public prompts |
| No-source baseline | Ask the same question with no supplied fixture when allowed | Prior knowledge or retrieval outside the fixture | Stable answer may reflect background knowledge, not proof the target was unused |

Run cells in randomized order and repeat them enough times to see whether variation is larger than the treatment effect. If the model surface does not expose temperature, seed, retrieval state, or model version, record that as a limitation instead of pretending the setting is fixed.

## Annotation rubric: answer change, citation change, and factual support change

**Annotators should score the answer, the citation, and the support relationship as separate objects.** RAG attribution work makes the same separation visible: the MIRAGE paper frames verifiability as a challenge and notes that self-citing LLMs can fail to faithfully reflect context usage, even when they generate citations ([ACL Anthology, EMNLP 2024](https://aclanthology.org/2024.emnlp-main.347/)). A citation marker and a supported claim are not the same unit.

Use three labels for each paired run:

| Dimension | Label | Definition | Evidence required |
|---|---|---|---|
| Answer change | Same recommendation | The main answer or recommendation is materially unchanged | Matched conclusion and no material change to ranked options, caveats, or action |
| Answer change | Changed recommendation | The answer changes the selected option, ranking, caveat, or decision | Before/after outputs differ in a way a reader would act on |
| Answer change | Changed framing only | The conclusion stays stable but the explanation, caveat, or confidence changes | Wording change affects reasoning but not the decision |
| Citation change | Same cited source | The target source is still cited or named after deletion | Must inspect whether retrieval outside the fixture reintroduced it |
| Citation change | Citation substitution | A different source is cited for the same claim | Replacement may mean redundancy, not absence of dependence |
| Citation change | Citation loss | The claim remains but the citation disappears | Support must be judged separately before calling it unsupported |
| Factual support | Supported | The cited or supplied evidence entails the claim at the needed granularity | Annotator can point to the supporting passage |
| Factual support | Partially supported | The source supports a weaker or adjacent claim | Report the exact unsupported portion |
| Factual support | Unsupported | The answer asserts a material claim not supported by the cited or supplied source | Do not infer intent; label the support relation only |

Disagreements should be adjudicated at the claim level. A second annotator should be able to reproduce the label from the prompt, source fixture, answer text, citation list, and scoring sheet without knowing which cell was expected to win.

## Null-result reporting should preserve causal humility

**A null result in source ablation means the selected outcome did not move under the tested condition, not that the source had no role.** The Nature paper on retrieval-augmented scientific synthesis evaluated organization, factual precision, citation quality, and use of supporting excerpts as separate quality components, and reported human-versus-LLM judge agreement rather than collapsing the assessment into one pass/fail verdict ([Nature, 2026](https://www.nature.com/articles/s41586-025-10072-4)). Source-ablation reports should do the same kind of denominator discipline.

Use this null-result template:

> In this preregistered paired-input test, removing Passage A did not materially change the primary answer label across the observed runs. Citation behavior and factual-support labels were scored separately. This result does not prove Passage A was unused. Plausible explanations include redundant evidence in the fixture, prior model knowledge, retrieval outside the supplied context, stochastic variation, prompt insensitivity, or replacement evidence. The valid inference is limited to the tested prompt, source fixture, model surface, date, and scoring rubric.

For a positive result, use the same caution:

> Removing Passage A changed the preregistered answer label and factual-support label under the tested conditions. This is evidence of sensitivity to the evidence treatment, not proof that Passage A was the only cause of the original answer.

That reporting language is the point of the protocol. The goal is not to produce dramatic gotchas. It is to stop teams from making two common mistakes: treating a citation as proof of causal source use, or treating an unchanged answer as proof that the source was ignored.

## Machine Relations implication: citation measurement needs intervention tests

[Machine Relations](https://machinerelations.ai/glossary/machine-relations) treats AI visibility as a source-condition problem: brands need evidence that machines can retrieve, cite, and represent without ambiguity. Source ablation adds a narrower experimental layer. It helps teams test whether a particular source condition changes an observed answer, while the broader [share of citation](https://machinerelations.ai/glossary/share-of-citation) metric tracks how often sources appear across answer runs.

The operating sequence is therefore: measure citation occurrence, identify the claim that matters, build a source fixture, preregister the paired-input test, annotate answer/citation/support separately, and report the inference boundary. [AuthorityTech](https://authoritytech.io) can use this kind of protocol in AI visibility work because it avoids the false certainty that weak measurement systems create.

Teams that want a faster external map before designing intervention tests can run an [AI visibility audit](https://app.authoritytech.io/visibility-audit), then use source ablation only for the sources and claims whose causal role is worth testing.

## FAQ

### What is source ablation for AI citations?

Source ablation is a paired-input test that removes, replaces, or controls one evidence passage to see whether an AI answer, citation, or support label changes. It is a method for testing sensitivity to evidence, not a standalone proof of what a model internally used.

### Does an unchanged AI answer prove the removed source was ignored?

No. An unchanged answer can come from redundant sources, prior model knowledge, retrieval outside the supplied fixture, stochastic variation, or a prompt that is insensitive to the evidence treatment. The correct null-result claim is narrower: the preregistered outcome did not move under the tested condition.

### Why separate answer change from citation change?

Answer change, citation change, and factual-support change answer different questions. A model can keep the same recommendation but cite a different source, lose a citation while preserving a claim, or change a citation without changing factual support. Treating those as one metric creates false certainty.

### How does source ablation fit with Machine Relations?

Machine Relations measures and improves the evidence conditions under which AI systems cite and recommend brands. Source ablation is one experimental tool inside that system: it tests whether a specific evidence condition appears to affect a specific observed answer under a locked protocol.

## Machine-readable related links

- Primary concept: [Ai Visibility](https://paralabs.ai/blog)
- Related concept: [Measurement](https://paralabs.ai/blog)
- Related concept: [Experiments](https://paralabs.ai/blog)
- Supporting research: [Same product name, wrong variant: a controlled AI product-identity test](https://paralabs.ai/blog/same-product-name-wrong-variant-ai-product-identity-test)
- Supporting research: [OpenAI's Data Agent Turns AI Visibility Measurement Into a Governed Data Problem](https://paralabs.ai/blog/openai-data-agent-ai-visibility-measurement-control-plane-2026)
- Supporting research: [The AI Visibility Index Shows Brand Discovery Is Now a Citation Rate Problem](https://paralabs.ai/blog/ai-visibility-index-brand-discovery-citation-rate-problem-2026)
- Research index: [Para Labs research index](https://paralabs.ai/blog)
- Machine manifest: [Para Labs machine manifest](https://paralabs.ai/machine-manifest.json)