No punctuations for some transcripts?

[ReichYang]

Hi, first of all, thank you for this amazing work!!! I'm about to do some analysis on this corpus but there is one issue: some transcripts do not have any punctuations, for example, 20150722_Wed_Alex.txt. And it seems to happen randomly? It might mess up some of my NLP pipelines. Do you have any insights into this issue, or if any amendments could be done? Thanks in advance!

[milahu]

youtube subtitles, generated by google speech, never have punctuations, and they have typos like #2

if you see punctuations, they come from manual postprocessing

[ReichYang]

youtube subtitles, generated by google speech, never have punctuations, and they have typos like #2

if you see punctuations, they come from manual postprocessing

I don't think they got the transcript from YouTube? I thought they used the Whisper AI for transcription. The issue is more of a tuning and the initialization of the model. See openai/whisper#194

[Fudge]

I'm using Whisper from OpenAI, yes. And it sometimes doesn't get things right. Not sure what to do about it.

[ReichYang]

I'm using Whisper from OpenAI, yes. And it sometimes doesn't get things right. Not sure what to do about it.

Hi, thanks for the reply. It looks like if one passes an initial prompt with punctuation it might fix it? (openai/whisper#194) But I'm not sure if that's doable for you or will cost too much

[Fudge]

If it's just a few episodes, I can re-do them. I've used a prompt with punctuation the last few months, so more recent stuff should be more reliable.

[Fudge]

I've started re-transcribing everything using a more recent and much faster Whisper implementation, letting me use the large-v2 model and an initial prompt with punctuation. It's going to take a bit over a month of uninterrupted work to get it done.

[ReichYang]

I've started re-transcribing everything using a more recent and much faster Whisper implementation, letting me use the large-v2 model and an initial prompt with punctuation. It's going to take a bit over a month of uninterrupted work to get it done.

Thank you so much! I was going to post here about the episodes that have missing punctuations but I somehow forgot. IIRC there are more than just a few and they appear randomly in terms of time so it was also hard to select a cut-off. I look forward to the new dataset!!!

[ReichYang]

@Fudge Hello, just wanna check in back on the issue. Based on the commits, it looks like all transcripts have been updated--is that the case?

[Fudge]

I've fed everything through WhisperX with punctuation prompts, so it's about as good as I can get it without manually editing.