You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Note, that due to the calculation of embeddings the function call might take a while. Embedding 10k samples on a Mac M2 take for example about 40secs. Limit the size of the passed DataFrames, or use the `max_sample_size_embeddings` parameter to speed up the report.
84
-
85
-
## Function signature
86
-
87
-
```python
88
-
defreport(
89
-
*,
90
-
syn_tgt_data: pd.DataFrame,
91
-
trn_tgt_data: pd.DataFrame,
92
-
hol_tgt_data: pd.DataFrame |None=None,
93
-
syn_ctx_data: pd.DataFrame |None=None,
94
-
trn_ctx_data: pd.DataFrame |None=None,
95
-
hol_ctx_data: pd.DataFrame |None=None,
96
-
ctx_primary_key: str|None=None,
97
-
tgt_context_key: str|None=None,
98
-
report_path: str| Path |None="model-report.html",
99
-
report_title: str="Model Report",
100
-
report_subtitle: str="",
101
-
report_credits: str=REPORT_CREDITS,
102
-
report_extra_info: str="",
103
-
max_sample_size_accuracy: int|None=None,
104
-
max_sample_size_embeddings: int|None=None,
105
-
statistics_path: str| Path |None=None,
106
-
on_progress: ProgressCallback |None=None,
107
-
) -> tuple[Path, Metrics |None]:
108
-
"""
109
-
Generate HTML report and metrics for comparing synthetic and original data samples.
110
-
111
-
Args:
112
-
syn_tgt_data: Synthetic samples
113
-
trn_tgt_data: Training samples
114
-
hol_tgt_data: Holdout samples
115
-
syn_ctx_data: Synthetic context samples
116
-
trn_ctx_data: Training context samples
117
-
hol_ctx_data: Holdout context samples
118
-
ctx_primary_key: Column within the context data that contains the primary key
119
-
tgt_context_key: Column within the target data that contains the key to link to the context
120
-
report_path: Path of where to store the HTML report
121
-
report_title: Title of the HTML report
122
-
report_subtitle: Subtitle of the HTML report
123
-
report_credits: Credits of the HTML report
124
-
report_extra_info: Extra details to be included to the HTML report
125
-
max_sample_size_accuracy: Max sample size for accuracy
126
-
max_sample_size_embeddings: Max sample size for embeddings (similarity & distances)
127
-
statistics_path: Path of where to store the statistics to be used by `report_from_statistics`
128
-
on_progress: A custom progress callback
129
-
Returns:
130
-
1. Path to the HTML report
131
-
2. Pydantic Metrics:
132
-
- `accuracy`: # Accuracy is defined as (100% - Total Variation Distance), for each distribution, and then averaged across.
133
-
- `overall`: Overall accuracy of synthetic data, i.e. average across univariate, bivariate and coherence.
134
-
- `univariate`: Average accuracy of discretized univariate distributions.
135
-
- `bivariate`: Average accuracy of discretized bivariate distributions.
136
-
- `coherence`: Average accuracy of discretized coherence distributions. Only applicable for sequential data.
137
-
- `overall_max`: Expected overall accuracy of a same-sized holdout. Serves as reference for `overall`.
138
-
- `univariate_max`: Expected univariate accuracy of a same-sized holdout. Serves as reference for `univariate`.
139
-
- `bivariate_max`: Expected bivariate accuracy of a same-sized holdout. Serves as reference for `bivariate`.
140
-
- `coherence_max`: Expected coherence accuracy of a same-sized holdout. Serves as reference for `coherence`.
141
-
- `similarity`: # All similarity metrics are calculated within an embedding space.
142
-
- `cosine_similarity_training_synthetic`: Cosine similarity between training and synthetic centroids.
143
-
- `cosine_similarity_training_holdout`: Cosine similarity between training and holdout centroids. Serves as reference for `cosine_similarity_training_synthetic`.
144
-
- `discriminator_auc_training_synthetic`: Cross-validated AUC of a discriminative model to distinguish between training and synthetic samples.
145
-
- `discriminator_auc_training_holdout`: Cross-validated AUC of a discriminative model to distinguish between training and holdout samples. Serves as reference for `discriminator_auc_training_synthetic`.
146
-
- `distances`: # All distance metrics are calculated within an embedding space. An equal number of training and holdout samples is considered.
147
-
- `ims_training`: Share of synthetic samples that are identical to a training sample.
148
-
- `ims_holdout`: Share of synthetic samples that are identical to a holdout sample. Serves as reference for `ims_training`.
149
-
- `dcr_training`: Average L2 nearest-neighbor distance between synthetic and training samples.
150
-
- `dcr_holdout`: Average L2 nearest-neighbor distance between synthetic and holdout samples. Serves as reference for `dcr_training`.
151
-
- `dcr_share`: Share of synthetic samples that are closer to a training sample than to a holdout sample. This shall not be significantly larger than 50\%.
152
-
"""
153
-
```
154
-
155
-
## Metrics
156
-
157
-
Three sets of metrics are calculated to compare synthetic data with the original data.
158
-
159
-
### Accuracy
160
-
161
-
The L1 distances between the discretized marginal distributions of the synthetic and the original training data are being calculated for all columns.
162
-
The reported accuracy is expressed as 100% minus the total variational distance (TVD), which is half the L1 distance between the two distributions.
163
-
These accuracies are then averaged to produce a single accuracy score, where higher scores indicate better synthetic data.
164
-
165
-
1.**Univariate Accuracy**: The accuracy of the univariate distributions for all target columns is measured.
166
-
2.**Bivariate Accuracy**: The accuracy of all pair-wise distributions for target columns, as well as for target columns with respect to the context columns, is measured.
167
-
3.**Coherence Accuracy**: The accuracy of the auto-correlation for all target columns is measured. This is applicable only for sequential data.
168
-
169
-
An overall accuracy score is calculated as the average of these aggregate-level scores.
170
-
171
-
### Similarity
172
-
173
-
All records are embedded into an embedding space to calculate two metrics:
174
-
175
-
1.**Cosine Similarity**: The cosine similarity between the centroids of the synthetic and the original training data is calculated and compared to the cosine similarity between the centroids of the original training and holdout data. Higher scores indicate better synthetic data.
176
-
2.**Discriminator AUC**: A binary classifier is trained to determine whether synthetic and original training data can be distinguished based on their embeddings. This score is compared to the same metric for the original training and holdout data. A score close to 50% indicates that synthetic samples are indistinguishable from original samples.
177
-
178
-
### Distances
179
-
180
-
All records are embedded into an embedding space, and individual-level L2 distances between samples are measured. For each synthetic sample, the distance to the nearest original sample (DCR) is calculated. This is done once with respect to original training records and once with respect to holdout records. These DCRs are then compared. For privacy-safe synthetic data, it is expected that synthetic data is as close to original training data as it is to original holdout data.
0 commit comments