Update `ForcedAlignment/evaluate.py`

[keighrim]

The current ForcedAlignment/evaluate.py script is incompatible with the newly proposed gold format (see clamsproject/aapb-annotations#121). It uses brittle text-matching logic that is now obsolete by the new process.py in the linked PR.

I propose refactoring evaluate.py to use character offsets for alignment instead of text. The updated script will no longer perform any text normalization or string comparison. Instead, it will leverage the alignment-start and alignment-end (or possibly a different set of column names when the PR is merged) character offsets from the gold .tsv files as the ground truth.

The core change will be a complete rewrite of the _read_pred method. To create the sparse hypothesis segments, the script will:

1. Read the character range (alignment-start, alignment-end) for a segment from the gold .tsv file.
2. Scan the Token annotations in the prediction MMIF file.
3. Identify the first and last Token annotations whose own character offsets fall within the gold segment's character range.
4. Use the timestamps associated with these first and last tokens to define the boundaries of the new hypothesis segment.

This approach is significantly more robust and elegant. It completely removes the dependency on the proprietary gold transcript during evaluation and eliminates all fragile text-matching code.

---

One more thing to consider during the re-implementing is how we use the pyannote.mectirc library.

Current implementation uses "downsampling" of dense annotation (token-level) in the MMIF to match sparse annotation (every 10 tokens) on the gold data, then uses SegmentationCoverage and SegmentationPurity for the metrics, which seemed correct when I implemented But now I'm requesting @shel-ho for confirmation of this usage or suggestions for alternative metrics from their recent research on the subject matter.

[shel-ho]

SegmentationCoverage and SegmentationPurity work in giving initial metrics regarding the performance of a forced aligner; however, if we can guarantee the correct number of boundaries and wish to follow standard metrics in the literature for more fine-grained analysis on performance, then we can run metrics on the start and end boundaries such as

• accuracy under different tolerances (10, 25, 50, 100ms) or
• mean, median, and/or standard deviation on boundary differences.

[keighrim]

Based on a recent analysis of pyannote metrics (thanks @shel-ho ), here's some of my thought.

tl;dr Let's just use DER.

Data Structure and Constraints

After updating "gold" dataset (clamsproject/aapb-annotations#121 and clamsproject/aapb-annotations#124), now forced alignment evaluation operates under specific structural guarantees:

Gold standard data:

• TSV file with N rows, each representing a text segment
• Columns: start, end (temporal boundaries), alignment-start, alignment-end (character indices)

Prediction data:

• MMIF JSON with token-level predictions
• Each token has character offsets matching the source text
• Each token is linked to exactly one TimeFrame (temporal boundaries)

Evaluation process:

1. For each gold row, find tokens matching the character range [alignment-start, alignment-end]
2. Extract TimeFrame boundaries from matched tokens
3. Compare predicted temporal boundaries to gold temporal boundaries

Key constraint: The evaluation creates exactly N hypothesis segments (one per gold row) with matching labels (character indices). This 1-to-1 correspondence is guaranteed by the data structure.

common segmentation errors are not concerned

• Over-segmentation: Cannot occur. Hypothesis segments are built by grouping tokens that match each gold character range. Even if tokens are granular, they're aggregated into one segment per gold row.

• Under-segmentation: Cannot occur. Each gold row produces exactly one hypothesis segment. Segments cannot be merged because they're explicitly created from distinct character ranges.

• Missing segments: Cannot occur. If tokens exist for a character range (which the prediction guarantees), a hypothesis segment will be created. The only exception would be complete prediction failure, which is logged as an error.

Conclusion: The only source of error is boundary timing misalignment between predicted and reference segment boundaries.

DER as the Sole Metric

Diarization Error Rate (DER)

Formula: DER = (False Alarm + Missed Detection + Confusion) / Total Reference Duration

Why DER is ideal:

• Comprehensive temporal error measurement: DER captures all three error types that can occur from boundary misalignment—confusion (time intervals where hypothesis has wrong label), missed detection (reference time not covered), and false alarm (hypothesis extending beyond reference).

• Direct interpretability: DER = 0.05 means "5% of the reference duration has label misalignment." This single number quantifies overall alignment quality.

• Appropriate task framing: By treating each text segment as a unique "speaker" (labeled by character index), forced alignment becomes analogous to speaker diarization. The confusion component specifically measures boundary accuracy.

Why other metrics were rejected:

• Segmentation Precision/Recall: Measures whether boundaries are within a tolerance threshold (binary per boundary). Provides less granular information than DER and requires arbitrary tolerance selection. With 1-to-1 correspondence, it doesn't add diagnostic value.

• Segmentation Coverage/Purity: Measures segment-level overlap quality. With guaranteed 1-to-1 label correspondence, these metrics are redundant with DER's confusion component. Both effectively measure IoU quality, but DER expresses it as an error rate.

• Detection Error Rate: Designed for binary speech/non-speech detection. Lacks the confusion component needed to assess boundary alignment between multiple segments.

Conclusion

DER provides a single, comprehensive, and interpretable metric for forced alignment evaluation, capturing the only source of error (boundary timing) without redundancy from supplementary metrics.

[keighrim]

closed via #87