PyEpiDoc

PyEpiDoc is a Python (>=3.10) library for parsing and interacting with TEI XML EpiDoc files. It has been tested on Python on Linux (Ubuntu) and Windows.

PyEpiDoc has been designed for use, in the first instance, with the I.Sicily corpus. For information on the encoding of I.Sicily texts in TEI EpiDoc, see the I.Sicily GitHub wiki.

NB: PyEpiDoc is currently under active development.

Install (no dev dependencies)

Locally

To install PyEpiDoc along with its dependencies (lxml):

1. Clone or download the repository.

2. Navigate into the cloned / downloaded repository.

3. From within the cloned repository, install at the user level with:

pip install . --user

In a virtual environment

If you are using a venv virtual environment:

1. Make sure the virtual environment has been activated, e.g. on Linux:

source env/bin/activate

2. Install with pip:

pip install .

Uninstall

pip uninstall pyepidoc

Install for development

To install PyEpiDoc along with its dependencies (lxml) and development dependencies (pytest, mypy), e.g. in a virtual environment:

1. Clone or download the repository;

2. Navigate into the cloned / downloaded repository.

3. From within the cloned repository, install with:

   pip install .[dev]
   

Running the Jupyter Notebooks

Jupyter notebooks are included in the repository under notebooks/ to provide example usage:

• getting_started.ipynb
• abbreviations.ipynb
• setting_ids.ipynb

For instructions on installing Jupyter notebook, see https://docs.jupyter.org/en/latest/install/notebook-classic.html. Alternatively, see also https://jupyter.org/install.

Once Jupyter notebook is installed, to run getting_started.ipynb, type:

jupyter notebook getting_started.ipynb

Example usage

Given a tokenized EpiDoc file ISic000001.xml in an examples/ folder in the current working directory.

Load the EpiDoc file

from pyepidoc import EpiDoc

doc = EpiDoc("examples/ISic000001_tokenized.xml")

Print the text of the edition

print(doc.edition_text)

Print all tokens in an edition (e.g. <w>, <name> etc.)

tokens = doc.tokens
print(' '.join([str(token) for token in tokens]))

Produce a tokenized version of a given EpiDoc file

Given an untokenized EpiDoc file ISic000032_untokenized.xml in an examples folder in the current working directory:

from pyepidoc import EpiDoc

# Load the EpiDoc file
doc = EpiDoc("examples/ISic000032_untokenized.xml")

# Tokenize the edition with default settings
doc.tokenize()

# Print list of tokens
print('Tokens: ', doc.tokens_list_str)

# Save the results to a new XML file
doc.to_xml_file("examples/ISic000032_tokenized.xml")

Corpus level analysis

Given a corpus of EpiDoc XML files in a folder corpus/ in the current working directory, the following code filters the corpus and writes a text file containing the ids of all Latin funerary inscriptions from Catania / Catina:

from pyepidoc import EpiDocCorpus
from pyepidoc.epidoc.enums import TextClass
from pyepidoc.file.funcs import str_to_file

# Load the corpus
corpus = EpiDocCorpus('corpus')

# Filter the corpus to find the funerary inscriptions
funerary_corpus = corpus.filter_by_textclass([TextClass.Funerary.value])

# Within the funerary corpus, find all the Latin inscriptions from Catania / Catina:
catina_funerary_corpus = (
    funerary_corpus
        .filter_by_orig_place(['Catina'])
        .filter_by_languages(['la'])
)

# Output the of this set of documents to a file ```catina_funerary_ids_la.txt``` 
# in the current working directory.
catina_funerary_ids = '\n'.join(catina_funerary_corpus.ids)
str_to_file(catina_funerary_ids, 'catina_funerary_ids_la.txt')

Validate EpiDoc XML

There are two ways to validate an EpiDoc XML file:

1. Validate on load, e.g.:

from pyepidoc import EpiDoc

doc = EpiDoc('examples/ISic000001_tokenized.xml', validate_on_load=True)

• This validates according to the RelaxNG schema tei-epidoc.rng in the pyepidoc root directory.
• By default validate_on_load is set to False.

2. Validate against a custom RelaxNG schema:

from pyepidoc import EpiDoc
doc = EpiDoc('examples/ISic000001_tokenized.xml')

doc.validate_by_relaxng(fp='path/to/relaxngschema.rng')

Code organisation

Package structure

The PyEpiDoc package has four subpackages:

• xml containing modules with base classes for XML handling;
• epidoc containing modules for handling EpiDoc specific XML handling, e.g. <ab>, <w> etc.;
• analysis containing modules for analysing EpiDoc files and corpora, e.g. of abbreviations;
• shared containing modules and classes for use generally in the project.

Probably the most useful subpackage in the first instance will be epidoc, and in particular epidoc.py and corpus.py, which, via the classes EpiDoc and EpiDocCorpus, provide APIs to EpiDoc files and corpora respectively.

Modifying tokenizer behaviour

The treatment of a given token by the tokenizer may be affected by one or more of the following:

• Status in pyepidoc/epidoc/epidoctypes.py
• Presence in pyepidoc/constants.py in SubsumableRels

The token will be subsumed into a neighbouring <w> token if it is not separated by whitespace if:

• it is listed in as a dep of e.g. <w> in SubsumableRels

The token will be subsumed into a neighbouring <w> token regardless of the presence of intervening whitespace if:

• it is listed in as a dep of e.g. <w> in SubsumableRels and
• it is a member of AlwaysSubsumableType in epidoctypes.py

Code integrity

Run the tests

with pytest installed (the dev installation will do this for you):

2. To run all the tests, in the project root directory, type:

   pytest
   

If pytest is not available to the currently active version of Python, it may be necessary to specify the Python executable with pytest installed, e.g.:

```
python3.10 -m pytest
```

Check the types

To check the integrity of the type annotations, with mypy installed (the dev installation will do this for you):

mypy src/pyepidoc

If mypy is not available to the currently active version of Python, it may be necessary to specify the Python executable with mypy installed, e.g.:

```
python3.10 -m mypy src/pyepidoc
```

Features to be included in future

XML comments

XML comments should now be handled correctly, and reproduced in new files.

Dependencies

PyEpiDoc depends on lxml (BSD 3). Development dependencies are mypy (MIT), pytest (MIT) and pytest-cov (MIT). Licenses for these dependencies are included in the LICENSES directory.

Licencing

• The software for PyEpiDoc (src/pyepidoc ) was written by Robert Crellin as part of the Crossreads project at the Faculty of Classics, University of Oxford, and is licensed under MIT (see LICENSES/LICENSE-pyepidoc).

• Example and test .xml files, contained in the examples/, example_corpus/ and tests/ subfolders are either directly from, or derived from, the I.Sicily corpus, which are made available under the CC-BY-4.0 licence (see LICENSES/LICENSE-texts and https://github.com/ISicily/ISicily/blob/master/licence.txt).

• The TEI EpiDoc schema is licensed under the GNU General Public license (see the license on the EpiDoc repository) (see LICENSES/LICENSE-EpiDoc-schema and LICENSES/gpl-3.0.txt).

• The repository as a whole is licensed under the GNU GPL v 3 license. My understanding is that this license is one-way compatible with the CC-BY-4.0 licence, MIT and BSD-3 licenses, such that it is possible for the requirements of those licenses to be fulfilled under GPL (see https://creativecommons.org/2015/10/08/cc-by-sa-4-0-now-one-way-compatible-with-gplv3/).

Funding

This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 885040, “Crossreads”).