Concept-Based Interpretability for Retrieval-Augmented Generation (RAG) Systems on Large Language Models
Overview
Retrieval-Augmented Generation (RAG) has emerged as an established architectural pattern to ground large language models in verified external knowledge. By conditioning generation on retrieved passages, these systems reduce factual hallucination in knowledge-intensive domains. However, external grounding does not guarantee internal transparency. When an autoregressive model synthesizes information from diverse retrieved passages, the intermediate steps between document retrieval, context integration, and final token production remain hidden.
This research project investigates concept-based interpretability tailored for retrieval-augmented architectures. In my capacity as a Research Mentor at the Trustworthy and Generative Machine Learning (TGML) Lab, I guide undergraduate researchers in formulating methods that identify, track, and verify conceptual representations across the retrieval-generation interface.
Methodological Framework
Standard attribution techniques in language generation typically operate at the level of individual token saliency. In multi-document retrieval settings, token-level scores become noisy and fail to indicate whether a generated assertion inherits a verified conceptual premise from the source or introduces an ungrounded extrapolation.
Our investigation adapts Concept Activation Vectors (CAV) and Automatic Concept-based Explanations (ACE) to trace conceptual consistency within RAG pipelines. Rather than treating internal activations as uninterpretable high-dimensional points, we extract directional vectors corresponding to specific semantic attributes, domain constraints, and factual assertions.
By projecting intermediate dense representations onto these concept directions, the framework audits whether retrieved passages activate the intended concepts prior to decoding. Furthermore, this approach enables tracing how concepts evolve across transformer layers during generation, identifying silent retrieval failures where conflicting or irrelevant passages degrade the final output.
Mentorship and Current Scope
The project involves mentoring a B.Sc. student through experimental design, representation extraction, and evaluation. Current milestones include constructing controlled benchmark sets with varying degrees of document conflict and semantic overlap, formalizing directional concept extraction across transformer residual streams, and evaluating the sensitivity of generation under targeted concept ablation.
Lab Dispatch Uplink Reserved
Technical writeups, experiment traces, and theoretical musings for this work will synchronize here as the lab.maryamrezaee.me ecosystem comes online.