Poster at ECVP 2025

ECVP is the European Conference on Visual Perception. In 2025, it took place in Mainz, Germany. This poster presented the application of my interpretability method (IGC) to an fMRI deep encoding model.

Abstract

Integrated Gradient Correlation (IGC) is a unique dataset-wise attribution method recently introduced to improve the interpretability of deep neural networks. Its main feature is to reveal the localization of input information relevant to output predictions at a task-level. Applied to a deep encoding model of brain signals from visual stimuli, we show that IGC provides a direct estimation of population Receptive Fields (pRF) and functional Localizers (fLoc).

Specifically, our model is trained on the Natural Scenes Dataset (NSD) and predicts surface-based fMRI data (visual cortex) from RGB images, as well as content information from the associated Common Objects in COntext dataset (COCO Panoptic Segmentation). For each brain signal unit (vertex), the resulting IGC attribution is a 3-dimensional tensor, covering all spatial dimensions and content type (RGB + COCO categories). Another key aspect of IGC attributions is to be easily summarized by summation, so that aggregations along categories give pRF maps, and additions across spatial dimensions expose functional abilities (e.g., 'bodies' or 'places' processing specificities). To make our approach feasible during a typical fMRI acquisition session, we designed a partially pre-trained procedure that provides accurate results with a sequence of 1k images only.

Compared to traditional pRF estimation methods, our deep encoding model explains more variance. The use of natural images also offers extracted data with a wider validity than artificial stimuli. Concerning fLoc, we found that our approach can avoid artificially sharp boundaries created by conventional univariate or contrastive statistical analysis, such as t-test. Consequently, IGC and deep encoding models constitute a novel method for pRF and fLoc estimation that is fully integrated and easily extensible to more diverse or finer grain localizers of any content present in natural images.

IGC method (arXiv preprint)  —  IGC package (Python/PyTorch)