Detect common transcript errors

[waldoj]

I'm seeing Whisper do goofy, incorrect stuff sometimes.

One is the insertion of earlier phrases later on in transcripts, sprinkled through the transcript.

Another is repeating a phrase dozens of times:

[00:04:05.000 --> 00:04:08.000]   >> A question whether that is necessary for an objection
[00:04:08.000 --> 00:04:11.000]   to the question.
[00:04:11.000 --> 00:04:14.000]   >> I think that's a good question. I think that's a good
[00:04:14.000 --> 00:04:17.000]   question. I think that's a good question.
[00:04:17.000 --> 00:04:20.000]   >> I think that's a good question. I think that's a good
[00:04:20.000 --> 00:04:23.000]   question. I think that's a good question. I think that's a
[00:04:23.000 --> 00:04:26.000]   good question. I think that's a good question. I think that's
[00:04:26.000 --> 00:04:29.000]   a good question. I think that's a good question. I think that's
[00:04:29.000 --> 00:04:32.000]   a good question.
[00:04:32.000 --> 00:04:35.000]   >> I think that's a good question. I think that's a good
[00:04:35.000 --> 00:04:38.000]   question. I think that's a good question.

Identify a lightweight way to identify these artifacts in transcripts and flag them for review or re-processing.

[waldoj]

Well, it ain't fancy, but this would identify the presence of often-repeating lines:

grep -E ".{31,}" transcript.srt |sort |uniq -c |sort -nr |head -1 |xargs |cut -d " " -f 1

[waldoj]

Now figure out what to do with these, once detected. My best guess is this:

• Store the list in a file
• Use the file as a blacklist for future transcribing
• Remove references to the transcripts from the JSON, if present
• Surface this list via some mechanism, so that transcripts aren't just silently, permanently missing

[waldoj]

OK, they're now stored in a file, and used to prevent adding new references to the JSON. I think I should just manually prune existing JSON references.

It won't be possible to properly close this issue until I understand the source of this problem (why is Whisper doing this?) and how to fix it.