RAT-Bench

Benchmark for evaluating PII removal and text anonymization tools for the paper RAT-Bench, with a focus on re-identification risk.

Our benchmark dataset is openly accessible. We also maintain an extendable leaderboard of text anonymization tools.

⚙️ Installation

This project uses the uv package manager.

1. Set up the virtual environment

Run the following inside the project root:

uv sync

This will create a virtual environment inside a .venv folder. Run all next commands within this virtual environment.

Make sure to then activate the environment, by running source .venv/bin/activate.

To install the correctmatch package from Rocher et al. that is used to compute re-identification risk, run the following:

python install_correctmatch.py

You'll need to install the spacy transformers by running

uv pip install pip
python -m spacy download en_core_web_lg

If you would need anonymizers also for other languages, repeat the same process to download es_core_news_lg(Spanish) and zh_core_web_lg(Chinese).

Copy pii_benchmark/credentials_example.py → pii_benchmark/credentials.py.

Add your API keys inside credentials.py.

(Note: credentials.py is already in .gitignore and won’t be committed.)

To run our pipeline for the Textwash anonymizer, download their models from here. Create a directory named data_textwash and place all the files in there (make sure the files are directly in data_textwash: the path to the models should be data_textwash/en, not data_textwash/models/en).

⚙️ Data

We have included the processed datasets needed to generate the benchmark and run our experiments in this repo:

data/100_profiles.csv contains the profiles, including direct and indirect identifiers, used to generate benchmark entries. data/population_sample.csv contains the 3M sample of US census data, preprocessed to account for weighting of each row.

🚀 Adding and Evaluating a New Anonymizer

All anonymizers are built on the abstract Anonymizer class.

🛠️ Step 1: Add Your Anonymizer

1. Create a new file in pii_benchmark/anonymizers/ (e.g., my_anonymizer.py).
2. Inside, define a class that:
   • Inherits from Anonymizer
   • Implements the anonymize method.
3. Open pii_benchmark/anonymizers/get_anonymizers.py and register your anonymizer by adding it to the cases dictionary.

✅ Example structure:

from pii_benchmark.anonymizers.base import Anonymizer

class MyAnonymizer(Anonymizer):
    def anonymize(self, text: str) -> str:
        # Your anonymization logic here
        return text

Open get_anonymizers.py and register your anonymizer by adding it to the cases dictionary.

✨ Step 2: Run Anonymization

To download the benchmark tests from hugging face and anonymize with your method, run:

sh scripts/anonymize/medical/all_levels_hf.sh
sh scripts/anonymize/chatbot/all_levels_hf.sh

To anonymize with your method on a locally stored version of the benchmark tests, run:

sh scripts/anonymize/medical/all_levels.sh
sh scripts/anonymize/chatbot/all_levels.sh

⚠️ Important: Update the anon_methods parameter in each script to match your anonymizer’s name.

⏱️ Note: If you want to see the runtime performance of your anonymizer per profile, set the timing_flag parameter in each script to 1.

🔍 Step 3: Evaluate Re-identification Risk

After anonymization, compute the re-identification risk with:

sh scripts/attack/medical/all_levels.sh
sh scripts/attack/chatbot/all_levels.sh

📊 (Optional) Step 4: Compute utility scores of anonymized data

Compute the utility scores of each anonymizer with:

sh scripts/utility/medical/all_levels.sh
sh scripts/utility/chatbot/all_levels.sh

Generating the data from scratch

We have included 100 benchmark entries per level in the benchmark folder in this repo. To run a new generation, run:

sh scripts/generate/level_{LEVEL}.sh

for the desired level of difficulty. You can also instead obtain our generated benchmark dataset here.

Anonymization scripts are included in scripts/anonymize, and attacker scripts are included in scripts/attack.

A full run of the pipeline for one difficulty level, from generation to re-identification rate computation, can be achieved by running the following:

1. sh scripts/generate/level_1.sh
2. sh scripts/anonymize/level_1.sh
3. sh scripts/attack/level_1.sh

References

If you found our work useful for your research, kindly cite our paper:

@article{krvco2026rat,
  title={RAT-Bench: A Comprehensive Benchmark for Text Anonymization},
  author={Kr{\v{c}}o, Nata{\v{s}}a and Yao, Zexi and Meeus, Matthieu and de Montjoye, Yves-Alexandre},
  journal={arXiv preprint arXiv:2602.12806},
  year={2026}
}