Arc-CBM: Prototype-Grounded Concept Bottlenecks with Manifold-Grounded Visual Auditing for Medical Image Analysis
Overview
Medical deep learning models achieve strong benchmark accuracy but frequently fail when deployed across different hospital systems, imaging hardware, and patient populations. These domain shifts expose shortcut learning, where classifiers exploit acquisition artifacts or institutional correlations rather than pathology. Concept Bottleneck Models (CBMs) address this vulnerability by routing predictions through human-interpretable clinical concepts.
Despite their transparency, standard CBMs rely on opaque, fully parametric concept predictors. When concept supervision originates from automatic report extraction, label noise becomes absorbed into learned weight matrices and cannot be audited directly. Arc-CBM resolves this limitation by framing concept grounding as an auditable memory problem. Rather than fitting parametric classifiers, we ground clinical concepts directly in representative exemplars using spherical prototype representations.
Methodology
Arc-CBM decouples concept definition from black-box parameter optimization while preserving end-to-end task performance. The framework operates on frozen visual representations from medical foundation models and introduces a complementary manifold-grounded attribution layer.
Prototypical Concept Grounding with Arc-CBM
Given a frozen medical image encoder, each radiograph is mapped to a normalized embedding on the unit hypersphere. For every candidate clinical concept, we collect balanced sets of positive and negative support images derived from report annotations. Because standard arithmetic means drift inside the hypersphere, Arc-CBM computes class centroids and projects them back onto the unit sphere.
Concept activations are evaluated as temperature-scaled cosine similarities against these dual prototypes. This yields an exact concept log-odds margin without requiring gradient steps over the vision backbone. Mislabeled instances or clinical ambiguities can be diagnosed by inspecting the nearest training cases—the prototype archetypes—and rectified simply by updating the support pool. Downstream classification relies on a linear layer constrained by a clinically informed sign prior.
Manifold-Grounded Visual Auditing with ArcMAP
To explain individual concept predictions spatially, we introduce ArcMAP (Archetype-Referenced Concept Manifold Attribution Projection). Traditional gradient saliency maps frequently highlight off-manifold noise or high-frequency pixel artifacts. ArcMAP computes the gradient of the oriented prototype margin with respect to penultimate transformer patch tokens, then projects this vector onto local tangent spaces estimated from nearby real radiographs.
This projection restricts attribution to directions supported by actual clinical anatomy. Because the tangent projection operates at the representation level, ArcMAP produces localized, token-wise visual audits for any specified concept without additional backward passes through the image backbone or synthetic image manipulations.
Findings
Arc-CBM was evaluated across five confounded chest radiograph benchmarks and five unconfounded clinical datasets. On confounded benchmarks, the framework achieves 66.0% out-of-domain accuracy and 73.3% domain-average accuracy, outperforming standard linear probes and competitive concept baselines.
The prototypical formulation exhibits strong data efficiency. When the support memory is restricted to 50 labeled examples per concept—a forty-fold reduction from the standard setting—Arc-CBM achieves 64.7% out-of-domain accuracy, remaining within 1.3 percentage points of the full data regime. This stability confirms that exemplar-grounded concept memories provide both compact auditing footprints and resilient out-of-distribution generalization in safety-critical medical diagnosis.
Lab Dispatch Uplink Reserved
Technical writeups, experiment traces, and theoretical musings for this work will synchronize here as the lab.maryamrezaee.me ecosystem comes online.