Print Sep 2024 – Aug 2026

Interpretability in Generative Models: Investigating the Mechanisms Behind Output Generation in Large Language Models

Author / Contributor
Maryam Rezaee (Master's Researcher)
Affiliation / Advisor Dr. Fatemeh Seyyedsalehi, Trustworthy and Generative Machine Learning (TGML) Lab, Department of Mathematical Sciences, Sharif University of Technology, Iran
Venue / Context Master's Thesis in Computer Science Graded Excellent (Highest Distinction), Department of Mathematical Sciences, Sharif University of Technology, Iran
Keywords: Generative Models Model Interpretability Black-Box Auditing Energy-Based Models Mechanistic Interpretability

Overview

The deployment of Large Language Models (LLMs) across high-stakes domains has outpaced our understanding of their internal decision-making processes. As these architectures grow in capacity and shift toward proprietary hosting behind closed application programming interfaces (APIs), the gap between empirical capability and algorithmic transparency becomes critical. Modern generative models function as high-dimensional probabilistic mappings, where standard attribution paradigms fail due to the open-ended nature of natural language outputs and the absence of internal parameter access.

This master’s thesis investigates the mechanisms governing output generation in generative language architectures. Conducted at the Trustworthy and Generative Machine Learning (TGML) Lab under the supervision of Dr. S. Fatemeh Seyyedsalehi, the research addresses two principal objectives: providing a unified theoretical taxonomy of existing interpretability paradigms in natural language processing, and formulating a scalable post-hoc framework for instance-based input-output attribution under strictly black-box constraints.

Literature Review and Methodological Taxonomy

The thesis conducts an extensive literature review that systematizes interpretability methods into four distinct categories:

  1. Intrinsic Interpretability: Architectures engineered to be transparent by design, such as linear models, decision trees, and attention-constrained networks, which trade off expressive capacity against explicit readability.
  2. Post-Hoc Attribution: Perturbation- and surrogate-based methods that evaluate feature importance after model execution without modifying model parameters.
  3. Concept-Based Interpretability: Formulations that map internal representations or external evidence onto human-understandable concept bottlenecks and activation vectors.
  4. Mechanistic Interpretability: Fine-grained reverse engineering of internal neural circuits, weight matrices, and sparse autoencoder features in open-weight models.

Additionally, the review examines the resurgence of Energy-Based Models (EBMs) in natural language processing, highlighting their ability to score global text compatibility and structural cohesion without intractable partition functions.

The Post-Hoc Black-Box Attribution Framework

Building on the theoretical analysis, the thesis introduces an amortized attribution framework designed specifically for zeroth-order generative endpoints. Classical black-box attribution algorithms rely on token-level deletions that destroy grammatical syntax, trigger out-of-distribution hallucinations, and demand prohibitive numbers of API queries.

To circumvent these failure modes, the proposed framework elevates the atomic feature unit from subword tokens to syntactically complete sentences. By treating each sentence as an operational propositional concept, the framework preserves grammatical integrity during feature selection. The system trains an Energy-Based Model surrogate to map the compatibility landscape between input prompts and observed outputs, guided by composite contrastive objectives. This surrogate landscape supervises an amortized interpreter network that predicts sentence-level influence scores in constant time, eliminating runtime API costs and providing a principled tool for auditing black-box generative systems.

Uplink Ready

Lab Dispatch Uplink Reserved

Technical writeups, experiment traces, and theoretical musings for this work will synchronize here as the lab.maryamrezaee.me ecosystem comes online.

Citation

@mastersthesis{rezaee2026interpretability,
  author       = {Maryam Rezaee},
  title        = {Interpretability in Generative Models: Investigating the Mechanisms Behind Output Generation in Large Language Models},
  school       = {Department of Mathematical Sciences, Sharif University of Technology},
  year         = {2026},
  month        = {August},
  type         = {Master's Thesis},
  address      = {Tehran, Iran},
  note         = {Supervised by Dr. S. Fatemeh Seyyedsalehi. TGML Lab},
  url          = {https://github.com/TGML-Lab/MaryamRezaee-MScThesis}
}