Task-agnostic Interpretability Evaluator

PyTorch

This repository contains the code and experiments for the paper "Evaluating Interpretable Methods via Geometric Alignment of Functional Distortions" by Hedström et al., 2025 (with Survey Certification!).

Please note that this repository is under active development!

Citation

If you find this work interesting or useful in your research, use the following Bibtex annotation to cite us:

@article{hedstrom2025explanation,
    title={Evaluating Interpretable Methods via Geometric Alignment of Functional Distortions},
    author={
      Hedstr{\"o}m, Anna and
      Bommer, Philine Lou and
      Tom, Burns and
      Lapuschkin, Sebastian and
      Samek, Wojciech and
      H{\"o}hne, Marina M-C
    },
    journal={Transactions on Machine Learning Research},
    year={2025},
    url={https://openreview.net/forum?id=ukLxqA8zXj},
}

Repository overview

The repository is organised as follows:

• The src/ folder contains all necessary functions.
• The nbs/ folder includes notebooks for generating the plots in the paper and for benchmarking experiments.
• The assets/ folder contains all files to reproduce the experiments.
• The tests/ folder contains the tests.

All evaluation metrics used in these experiments are implemented in Quantus, a widely-used toolkit for metric-based XAI evaluation. Benchmarking is performed with tools from MetaQuantus, a specialised framework for meta-evaluating metrics in interpretability.

Installation

Install the necessary packages using the provided requirements.txt:

pip install -r requirements.txt

Package requirements

Required packages are:

python>=3.10.1
torch>=2.0.0
quantus>=0.5.0
metaquantus>=0.0.5
captum>=0.6.0

Thank you

We hope our repository is beneficial to your work and research. If you have any feedback, questions, or ideas, please feel free to raise an issue in this repository. Alternatively, you can reach out to us directly via email for more in-depth discussions or suggestions.

📧 Contact us:

• Anna Hedström: hedstroem.anna@gmail.com

Thank you for your interest and support!