OCR Corpora Early 2025 Publication Thread

[ctschroeder]

OCR publication thread for early 2025

Do not close this issue until all checkboxes below are complete or have been rescheduled:

List of corpora:

In Processed OCR folder (need chapter divisions + sentence splitting+full automatic NLP processing like the bible corpora)

• Giron Legendes (11 docs)
  • chapter divisions added/checked
  • metadata updated

all documents above need to be moved to https://github.com/CopticScriptorium/auto-corpora

In GitDox

• apocalypse.paul (2)
  - [ ] corpus name needed
  - [ ] other metadata updated
  - possibly error in data -- translation on p. 1043 begins with folio 24a but OCR coptic begins in the middle of folio 6a p. 533

• pscyril.alexandria

  • On Mary still in XML mode (auto tagging?)
  • entities and identities

• pscyril.jerusalem

  • on the cross
    • needs corpus name
    • metadata updated
    • chapter & verse need to be updated in spreadsheet based on open tags in XML
  • on Mary
    • needs corpus name
    • metadata updated
    • chapter & verse need to be updated in spreadsheet based on open tags in XML

• psepiphanius on Mary

  • translation span
  • entities and identities
  • metadata

• pschrysostom

  • translation span
  • entities and identities
  • metadata

• pscelestinus

• pstimothy.alex

• psote.psoi

• timothy.discourse

[ctschroeder]

I know we were going to put Giron in the "automatically processed straight from XML like the big bible corpora" bucket, but with the sentence splitting and chapter divisions required, I think it will be easier to do both in GitDox. Chapter divisions are easier to see in the spreadsheet mode, and you can use the entity mode to do the sentence-splitting. So I'm making the executive decision to put this stuff in GitDox. @amir-zeldes @EarlyCodices

[amir-zeldes]

Fine by me - 11 docs should still be manageable, I'd be more worried for really big stuff. Keep in mind that one potential advantage of "XML Bible mode" is that tokenization is done fresh each time as tokenizers improve, so if you're correcting any segmentation manually, GD can make more sense (or you could introduce pipes in the XML repo). If you want fresh re-tokenization each release, you should not put things in GD - that would be the other main reason not to choose it.

[ctschroeder]

I don't think we are checking segmentation manually, but this is a good point. We have lots to do so I'll ponder this. They already have pipes from Lydia's work. I found when I was working on other documents for the full-auto-from-xml mode, validation was a real pain for reasons I don't fully remember and probably repressed.