Active Improving Method May 2025 – Present

Concept-Based Interpretability for Retrieval-Augmented Generation (RAG) Systems on Large Language Models

Author / Contributor
MohammadHossein MahdiKhani (Research Mentee), Maryam Rezaee (Research Mentor), Fatemeh Seyyedsalehi (Lab Director)
Affiliation / Advisor Trustworthy and Generative Machine Learning (TGML) Lab, Sharif University of Technology, Iran
Keywords: LLMs RAG XAI Concept Activation Vectors Interpretability

Overview

Retrieval-Augmented Generation (RAG) has emerged as an established architectural pattern to ground large language models in verified external knowledge. By conditioning generation on retrieved passages, these systems reduce factual hallucination in knowledge-intensive domains. However, external grounding does not guarantee internal transparency. When an autoregressive model synthesizes information from diverse retrieved passages, the intermediate steps between document retrieval, context integration, and final token production remain hidden.

This research project investigates concept-based interpretability tailored for retrieval-augmented architectures. In my capacity as a Research Mentor at the Trustworthy and Generative Machine Learning (TGML) Lab, I guide undergraduate researchers in formulating methods that identify, track, and verify conceptual representations across the retrieval-generation interface.

Methodological Framework

Standard attribution techniques in language generation typically operate at the level of individual token saliency. In multi-document retrieval settings, token-level scores become noisy and fail to indicate whether a generated assertion inherits a verified conceptual premise from the source or introduces an ungrounded extrapolation.

Our investigation adapts Concept Activation Vectors (CAV) and Automatic Concept-based Explanations (ACE) to trace conceptual consistency within RAG pipelines. Rather than treating internal activations as uninterpretable high-dimensional points, we extract directional vectors corresponding to specific semantic attributes, domain constraints, and factual assertions.

By projecting intermediate dense representations onto these concept directions, the framework audits whether retrieved passages activate the intended concepts prior to decoding. Furthermore, this approach enables tracing how concepts evolve across transformer layers during generation, identifying silent retrieval failures where conflicting or irrelevant passages degrade the final output.

Mentorship and Current Scope

The project involves mentoring a B.Sc. student through experimental design, representation extraction, and evaluation. Current milestones include constructing controlled benchmark sets with varying degrees of document conflict and semantic overlap, formalizing directional concept extraction across transformer residual streams, and evaluating the sensitivity of generation under targeted concept ablation.

Uplink Ready

Lab Dispatch Uplink Reserved

Technical writeups, experiment traces, and theoretical musings for this work will synchronize here as the lab.maryamrezaee.me ecosystem comes online.