What training samples were used to train half_models in a benchmark

[Crabtain959]

When loading a benchmark, is the half_models trained with the training samples whose indices are presented in groundtruth in the benchmark? According to the paper, I think the answer is no.

Across all settings, we pre-train 100 models for LDS calculation on a random half-dataset controlled by a fixed random seed generation procedure. The first 50 models are used in our benchmark experiments to assemble and compute the attribution scores. For example, TRAK and TracIn can utilize multiple models to improve their performance on the same task. The last 50 models are used for LDS calculation, which requires several models that are trained on a portion of the original training dataset.

Then is there a way to get the training data used to train the first 50 models (the ones used in the benchmark experiments to assemble and compute the attribution scores), that is the 50 half models contained in a benchmark?

Below is how I found the mismatch, just in case I did something wrong:
For example, if I load the benchmark as follows:

from dattri.benchmark.load import load_benchmark
model_details, groundtruth = load_benchmark(
    model="mlp", dataset="mnist", metric="lds"
)

There are 50 half models in model_details, and the second tensor in groundtruth has 50 rows, where "each row refers to the indices of the training samples used to retrain the model" according to this documentation. Was the ith half model trained with the training samples on the ith row in groundtruth? I tried to compare them, and it seems that they don't match. The following is how I tested this:

First load the test data:

eval_test_loader = DataLoader(
    model_details["test_dataset"],
    batch_size=8,
    sampler=model_details["test_sampler"],
    shuffle=False
)

Then manually compute the target values for the test data using ith half model:

model_details['target_func'](model_details['models_half'][i], eval_test_loader)

Then compare it with the ith row of target values contained in groundtruth

groundtruth[0][i]

And it turned out these two are not the same.

Thank you for your help!

[TheaperDeng]

Thank you for the question @Crabtain959 !

Let's first clarify two concepts first:

• The models used to calculate LDS/LOO ground truth: These models are trained with a portion of data points (e.g., half for LDS, Leave-one-out for LOO).
• The reference models used to calculate attribution scores: These models are used for TDA (training data attribution) methods to calculate the attribution scores.

The models of these two concepts are not necessarily to be the same ones. In a more intuitive setting (as what we did), they should be different. While the behavior of using the same models to calculate LDS ground truths and attribution scores is not clear.

The models provided in model_details["models_half"] and model_details["models_full"] are used as The reference models used to calculate attribution scores. The model_details["models_half"] are used for TRAK algorithm since it explicitly asking for models trained with half dataset and the model_details["models_full"] are used for other algorithms such as IF.

In our benchmark suite, we provide the ground truth (including the indices and the groundtruth value of target_func), while we don't provide the models used to calculate the ground truth. Since evaluating the LDS/LOO does not need those models. Still, you may create the models through training with the indices like following

model_details, groundtruth = load_benchmark(
    model="mlp", dataset="mnist", metric="lds"
)

# first model's indices
first_model_indices = groundtruth[1][0]

# create a dataloader with the first model's indices
dataloader = DataLoader(
    model_details["train_dataset"],
    batch_size=64,
    sampler=SubsetSampler(first_model_indices),
)

# train the model
first_model_for_LDS_gt = model_details["train_func"](dataloader, device=device)

But you may not get exactly the same models as used for the ground truth calculation because there are some randomness in the training process. Could you help us understand why you need such models? It could help us to improve our benchmark suite in potential future updates (e.g., provide the The models used to calculate LDS/LOO ground truth together with the ground truth values)

[Crabtain959]

Thank you so much for the detailed response @TheaperDeng ! This is very clear and helpful.

I wish I had a fancier reason to ask for the following and sorry to disappoint you, but I mainly wanted to explore and compare models trained on different subsets of the data, and see how that affects the final attribution performance.

I don’t necessarily need the models used to calculate the ground truth. As you pointed out, there’s inherent randomness in the training process, so my main interest is just in having access to trained models along with their corresponding training sample indices.

If I understand correctly, the benchmark provides trained half_models but not the training sample indices for them, and separately provides the indices used for ground truth but not the corresponding trained models. Would it be possible to share the training sample indices associated with the half_models?

Alternatively, if it’s easier, having the models for ground truth alongside their training samples would work just as well for my purposes, but I completely understand if those aren’t meant to be shared for fair evaluation.

Thank you again for the great work!

[TheaperDeng]

Thank you for the feedback and it helps my understanding! We will try to provide the corresponding models or the training indices later once we got chance. I apologize this may takes some time but we will put this to the TODO list.