“glass top left of table” RefCOCO
ICML 2026 Workshop · Mechanistic Interpretability
Retrieval Heads Meet Vision
Uncovering How VLMs Locate and Extract Visual Information
1 KAIST · 2 Independent Researcher
Uncovering How VLMs Locate and Extract Visual Information
Vision-language models (VLMs) can locate an image region referred to by a text prompt and route the corresponding visual evidence to the output, yet the internal mechanism behind this behavior is not understood. Inspired by retrieval heads in large language models, we ask whether VLMs contain an analogous mechanism for visual retrieval. We answer affirmatively by introducing Visual Retrieval Heads (VRHs), a small subset of attention heads (about 1.7–2.6%) that are causally responsible for grounding text descriptions to image regions.
To find them, we recast existing head-scoring methods under a unified design space over query tokens, key aggregation, and cross-sample aggregation. We then show that scoring attention from output prediction tokens with a sum over the ground-truth referent region most reliably identifies causal heads. Across four VLMs and five referring-expression benchmarks, masking only the top 20 VRHs reduces grounding accuracy by up to 80 percentage points, while masking the same number of random heads has little effect.
Beyond replicating the causal–sparse–universal triad established for text retrieval heads, VRHs exhibit several properties not previously reported: they generalize across visual reference tasks; they are functionally specific, preserving output format while corrupting localization; and they are architecturally shared, transferring causally across VLMs that share an LLM backbone but differ in vision encoder, projector, and instruction tuning.
In LLMs, a handful of retrieval heads route the right input token to the output during long-context recall. They are causal, sparse, and universal. Do VLMs have an analogue for finding the right image region?
Ablating them collapses retrieval while leaving fluency intact — they are necessary, not merely correlated with attention.
Fewer than a few percent of all heads carry the behavior; the rest can be perturbed with little effect.
They emerge across model families and scales, suggesting a shared internal mechanism rather than an idiosyncratic one.
Prior work stops short: OCR Heads and VERA retrieve text rendered inside images; Localization Heads reach general images but characterize heads only by attention shape, never verifying causal necessity. We close this gap — the first heads shown to be causally responsible for visual retrieval from general images.
| Specialized heads | Visual retrieval source | Causally validated? |
|---|---|---|
| Retrieval Heads (Wu et al.) | Text (long context) | yes |
| QRHead | Text (long context) | yes |
| OCR Heads | Rendered text in images | yes |
| VERA | Visual documents | yes |
| Localization Heads | General images | no |
| Visual Retrieval Heads Ours | General images | yes |
Table 1. VRHs are the first attention heads shown to be causally responsible for visual retrieval from general images.
We use referring-expression visual grounding as a probing task: given an image and an expression, the model must output a bounding box. The ground-truth box makes the retrieval target spatially explicit — it defines exactly which visual tokens the model should consult.
Prior methods all score a head by how strongly it attends from a set of query tokens to the key tokens in the evidence region. For grounding, we let the key tokens be the visual tokens overlapping the ground-truth box. Written in one form:
Existing methods are just different choices along these three axes. So we enumerate every combination, mask each variant's top heads, and keep the one whose removal hurts grounding the most. Try it below — pick a setting and see which method it corresponds to.
Table 2 & the design space. Different aggregation choices barely move results — but the query-token set matters a great deal: the strongest causal signal appears when the model is autoregressively producing the grounded answer, not when it merely encodes the prompt.
Figure 2. Causal validation of head-scoring variants (RefCOCO, four VLMs). We mask each variant's top-20 heads and report grounding accuracy — lower is better (more causal). Our default (Qout, Φsum, Ωavg) drives the largest drop. The Qs, Φ, and Ω bars each flip one axis; Δ is the gap to Ours. Flipping the query set hurts most — output tokens are where the causal signal lives.
To test a head's causal contribution, we zero the attention output of head (l, h) before it is added to the residual stream — leaving every other head, MLP, and layer-norm untouched. This is the same intervention used for text retrieval heads, so the results are directly comparable. All experiments run on a single 48 GB GPU.
Across four VLMs and three benchmark families, masking a few VRHs collapses grounding — far beyond random heads or heads found by prior methods.
Figure 3. Comparison with existing specialized heads (RefCOCO). As k grows, masking the top-k VRHs degrades grounding far more sharply than masking heads from Retrieval Heads, OCR Heads, VERA, Localization Heads, or random selection — VRHs are a distinct, causally critical set.
Figure 4. Universality of VRHs. Grounding accuracy after masking the top-k VRHs across four VLMs and three benchmark families (RefCOCO/+/g, RefSpatialBench, Toloka). Masking only a few VRHs (red) consistently causes large drops, while random heads (gray) barely move — a shared head-level mechanism for visual grounding.
Fewer than 15% of heads receive any non-negligible score, and only 0–4% score highly — yet masking those few is enough to break grounding. Visual retrieval is concentrated, not diffuse: it mirrors the sparse-but-causal organization of retrieval heads in LLMs.
![Stacked horizontal bars for four VLMs showing the fraction of heads with score in (0,1e-3], (1e-3,5e-3], and greater than 5e-3. The vast majority of heads have negligible scores.](static/figures/sparsity_source.png)
Figure 5. Sparsity of VRH scores (RefCOCO). Distribution of head scores across the four VLMs; only a small fraction of heads score above 10⁻³.
The evidence behind Figure 1, made interactive. Switch between all heads, the top-20 VRHs alone, and everything except the VRHs — and watch where the model looks, and whether its box stays on target.
Attention · all heads
click to step
Attention overlays are the paper's own qualitative examples (Qwen2.5-VL-7B). Accuracy figures are dataset-level (RefCOCO, Toloka) from the appendix: masking the top-20 VRHs — 2.6% of heads — collapses grounding, while masking the same number of random heads barely moves it.
Every object is tinted its own color. Move the cursor over one — it lights up, and we force the top-20 Visual Retrieval Heads to attend only to that segment. The precomputed output then shows how the model’s predicted box (red) changes under that intervention.
Each object segment is an automatic mask; steering forces the top-20 VRHs’ attention uniformly onto the pointed segment (decode-only, matching how VRHs were detected). Boxes are precomputed object-locate outputs from Qwen3-VL-8B. The showcase omits incomplete or unusable segment outputs; steering can redirect a prediction, but is not guaranteed to match every selected segment.
Every cell is one attention head, arranged by layer (rows) and head index (columns). The top-20 VRHs light up — clustered in the model's middle layers. Hover to inspect.
For Qwen2.5-VL-7B, just 20 of 784 heads (2.6%) form the VRH set — and they sit in a tight band of middle layers (≈ 14–19), exactly where a reusable retrieval circuit would route evidence.
Top-20 (layer, head) indices are taken verbatim from the paper's appendix.
The 12 highest-ranked VRHs on the map above are individually clickable — knock out any combination and inspect how the predicted box (red) changes relative to the ground truth (green). The buttons apply the full top-20 intervention or a 20-random-head control. Each box is a precomputed Qwen2.5-VL-7B output for that exact set of masked heads; invalid outputs and control failures remain visible.
VRHs do more than replicate text retrieval heads. They reveal a general mechanism for visual reference resolution.
Discovered only from bounding-box grounding, the same heads remain causal on attribute, spatial, counting, vision-centric, and visual-math benchmarks — they implement visual reference resolution in general, not object grounding in particular.
Masking VRHs keeps the output's format valid — the box parses, the sentence reads well — but the localization is wrong. It selectively breaks where the model looks, the visual analogue of the LLM failure mode.
VRHs found in one VLM remain causal when masked at the same indices in another VLM that shares the LLM backbone — even with a different vision encoder, projector, and instruction tuning.
The same top heads emerge from different output-token choices and from as few as 5–10 annotated grounding examples — a strong, repeatable signature rather than a weak average.
Masking the top-20 VRHs detected on RefCOCO degrades performance on seven held-out vision-QA benchmarks across all four VLMs, while random-head masking does almost nothing.
| Model | Condition | VAW | Spatial457 | SpatialRGPT | CV-Bench | MMStar | CountBenchQA | MathVista |
|---|
Table 3. Cross-task generalization. Red = drop under VRH masking (k=20). Random masking (k=20) stays near the baseline.
Figure 7. (a) Under VRH masking, boxes stay syntactically valid but become mislocated; parse failures stay rare. (b) A single random output token or all output tokens both yield ~7% accuracy when masked (random heads: ~86%). (c) 5–200 detection samples recover the same heads.
This is exactly the "fluent-but-not-factual" mode reported for text retrieval heads, now observed in vision: the model keeps answering confidently while no longer looking at the referent.
Attention maps on visual grounding. Left to right: input, all heads, the top-20 VRHs only, and the model with the VRHs masked. Attention concentrates on the referent under all heads and is reproduced by the VRHs alone — but once the VRHs are masked it scatters, and the predicted box (red) flies off the ground truth (green).
All heads: blue · correct VRHs masked: gray · wrong
All heads: yellow · correct VRHs masked: purple · wrong
All heads: yellow · correct VRHs masked: green · wrong
All heads: tan · correct VRHs masked: brown · wrong
Attention maps on visual question answering. Each panel shows the input (green = ground-truth region), the top-20 VRH attention, all-heads attention, and attention after masking the VRHs. Keep the VRHs and the answer is correct; mask them and attention drifts off the evidence, flipping the answer to a fluent but visually unfaithful one.
Figure 6. Detection consistency across same-backbone VLMs. Top: layer-head score maps line up across same-backbone pairs (top-20 overlap up to 18/20; Spearman ρ ≈ 0.9). Bottom: transferring one model's top-k VRHs to its sibling (same indices) still collapses grounding — the circuit lives in the language backbone.
VRHs aren't just an analytic lens — they're an optimization target. Using a VRH-masked model as the negative policy for direct preference optimization (DPO) concentrates exactly the failure we want to suppress, yielding consistent grounding gains over a random-head-masked control.
Modest but consistent: about +0.78 points on average across three grounding benchmarks — an initial demonstration that VRHs are a mechanistic substrate that can be directly optimized.
Figure A3. DPO with VRH-based negatives. Perturbing causally relevant heads gives a more informative preference signal than random-head dropping.
A sparse set of attention heads carries text-referred visual evidence to the output — and removing them, alone, breaks grounding.
The same heads serve attributes, spatial reasoning, counting, and visual math, and transfer across same-backbone VLMs.
Cheap to detect (5–10 examples) and directly optimizable — a lever for diagnosing and improving visual grounding.
Limitations & what's next. Our signal is attention-based: it captures where evidence is routed from, not what is read out. Combining VRH detection with activation patching or causal mediation, and extending beyond single images to multi-image, video, and interleaved inputs, are natural next steps — with applications to visual-context selection and multimodal cache compression.
@inproceedings{park2026retrievalheads,
title = {Retrieval Heads Meet Vision: Uncovering How VLMs
Locate and Extract Visual Information},
author = {Park, Chanho and Choi, Daehyeon and Lee, Jihyun
and Sung, Minhyuk},
booktitle = {ICML 2026 Workshop on Mechanistic Interpretability},
year = {2026}
}