CU-8699nk284 Update MedCAT service to v2 (#27)

* CU-8699nk284: Update requirements to v2

* CU-8699nk284: Add DeID requirement

* CU-8699nk284: Update code to be in line with v2

* CU-8699nk284: Update README with v2 link

* CU-8699nk284: Update README with v2 link (2nd place)

* CU-8699nk284: Fix examples tutorial link

* CU-8699nk284: Update model card stuff for v2 compatibility

* CU-8699nk284: Avoid running docker hub push on pull requets

* CU-8699nk284: Fix typo in import path

* CU-8699nk284: Fix further typo

* CU-8699nk284: Fix access path for MetaCAT config categery name

* CU-8699nk284: Update to latest medcat v2 release

* CU-8699nk284: Bump supported version to latest (to fix legacy CDB conversion)

* U-8699nk284: Fix some config access

* U-8699nk284: Fix some more config access

* CU-8699nk284: Bump dependency to latest

* CU-8699nk284: Update to latest v2 version

* CU-8699nk284: Bump to latest medcat version again (this time finally, I hope)

* CU-8699nk284: Fix config access

* CU-8699nk284: Mock CDB load during test to use en_core_web_md instead of en_core_sci_lg

* CU-8699nk284: [TEMP] Add debug output

* CU-8699nk284: Add client test within mocked / changed spacy model context manager

* CU-8699nk284: [TEMP] Add debug (more) output

* CU-8699nk284: Mock a different method and more generally during testing

* CU-8699nk284: Fix import during testing

* CU-8699nk284: Remove (most of) the debug output

* CU-8699nk284: Bump requirements to latest

* CU-8699nk284: Update ultiprocessing method to v2

* CU-8699nk284: Fix keyword argument name

* CU-8699nk284: Update multiprocessing

* CU-8699nk284: Update to latest MedCAT version

* CU-8699nk284: Update to latest requirements

* CU-8699nk284: Use newer python in workflow

* CU-8699nk284: Bump requirements to latest

* CU-8699nk284: Bump requirements to latest

* Revert "CU-8699nk284: Avoid running docker hub push on pull requets"

This reverts commit 0f9cfc2.

* CU-8699nk284: Fix Dockerfile for GitHub URL-based installs

* CU-8699nk284: Fix hash attribute path

Author: mart-r

.github/workflows/medcat-service_run-tests.yml

@@ -29,7 +29,7 @@ jobs:
       - name: Install Python 3
         uses: actions/setup-python@v5
         with:
-          python-version: 3.9
+          python-version: 3.11
           cache: 'pip' # caching pip dependencies
 
       - name: Install dependencies

medcat-service/Dockerfile

@@ -6,6 +6,9 @@ ENV CRYPTOGRAPHY_DONT_BUILD_RUST=1
 WORKDIR /cat
 COPY ./requirements.txt /cat
 
+# NOTE: need git for URL based installs
+RUN apt-get update && apt-get install -y git
+
 # Install Python dependencies
 ARG USE_CPU_TORCH=true
 # NOTE: Allow building without GPU so as to lower image size (GPU is disabled by default)

medcat-service/README.md

@@ -1,6 +1,6 @@
 # Introduction
 
-This project implements the [MedCAT](https://github.com/CogStack/MedCAT/) NLP application as a service behind a REST API. The general idea is to be able send the text to MedCAT NLP service and receive back the annotations. The REST API is built using [Flask](https://flask.palletsprojects.com/).
+This project implements the [MedCAT](https://github.com/CogStack/cogstack-nlp/blob/main/medcat-v2/) NLP application as a service behind a REST API. The general idea is to be able send the text to MedCAT NLP service and receive back the annotations. The REST API is built using [Flask](https://flask.palletsprojects.com/).
 
 Git Branches:
   - devel: development branch, latest updates and features, might be unstable.
@@ -327,4 +327,4 @@ The main settings that can be used to improve the performance when querying larg
 ## MedCAT library
 MedCAT parameters are defined in selected `envs/env_medcat*`  file. 
 
-For details on available MedCAT parameters please refer to [the official GitHub repository](https://github.com/CogStack/MedCAT/).
+For details on available MedCAT parameters please refer to [the official GitHub repository](https://github.com/CogStack/cogstack-nlp/blob/main/medcat-v2/).

medcat-service/medcat_service/nlp_processor/medcat_processor.py

@@ -10,8 +10,9 @@
 from medcat.cat import CAT
 from medcat.cdb import CDB
 from medcat.config import Config
-from medcat.meta_cat import MetaCAT
-from medcat.utils.ner.deid import DeIdModel
+from medcat.config.config_meta_cat import ConfigMetaCAT
+from medcat.components.addons.meta_cat import MetaCATAddon
+from medcat.components.ner.trf.deid import DeIdModel
 from medcat.vocab import Vocab
 
 
@@ -188,7 +189,7 @@ def process_content_bulk(self, content):
         # use generators both to provide input documents and to provide resulting annotations
         # to avoid too many mem-copies
         invalid_doc_ids = []
-        ann_res = []
+        ann_res = {}
 
         start_time_ns = time.time_ns()
 
@@ -197,11 +198,14 @@ def process_content_bulk(self, content):
                 ann_res = self.cat.deid_multi_texts(MedCatProcessor._generate_input_doc(content, invalid_doc_ids),
                                                     redact=self.DEID_REDACT)
             else:
-                ann_res = self.cat.multiprocessing_batch_char_size(
-                    MedCatProcessor._generate_input_doc(content, invalid_doc_ids), nproc=self.bulk_nproc)
-
+                text_input = MedCatProcessor._generate_input_doc(content, invalid_doc_ids)
+                ann_res = {
+                    ann_id: res for ann_id, res in
+                    self.cat.get_entities_multi_texts(
+                        text_input, n_process=self.bulk_nproc)
+                }
         except Exception as e:
-            self.log.error(repr(e))
+            self.log.error("Unable to process data", exc_info=e)
 
         additional_info = {"elapsed_time": str((time.time_ns() - start_time_ns) / 10e8)}
 
@@ -239,11 +243,12 @@ def _populate_model_card_info(self, config: Config):
         Args:
             config (Config): MedCAT configuration object.
         """
-        self.model_card_info["ontologies"] = config.version.ontology \
-            if (isinstance(config.version.ontology, list)) else str(config.version.ontology)
-        self.model_card_info["meta_cat_model_names"] = [i["Category Name"] for i in config.version.meta_cats] \
-            if (isinstance(config.version.meta_cats, list)) else str(config.version.meta_cats)
-        self.model_card_info["model_last_modified_on"] = str(config.version.last_modified)
+        self.model_card_info["ontologies"] = config.meta.ontology \
+            if (isinstance(config.meta.ontology, list)) else str(config.meta.ontology)
+        self.model_card_info["meta_cat_model_names"] = [
+            cnf.general.category_name for cnf in config.components.addons
+            if (isinstance(cnf, ConfigMetaCAT))]
+        self.model_card_info["model_last_modified_on"] = str(config.meta.last_saved)
 
     # helper MedCAT methods
     #
@@ -281,7 +286,7 @@ def _create_cat(self):
                 cat.cdb.filter_by_cui(cuis_to_keep)
 
             if self.app_model.lower() in ["", "unknown", "medmen"]:
-                self.app_model = cat.config.version.id
+                self.app_model = cat.config.meta.hash
 
             self._populate_model_card_info(cat.config)
 
@@ -305,13 +310,13 @@ def _create_cat(self):
         spacy_model = os.getenv("SPACY_MODEL", "")
 
         if spacy_model != "":
-            cdb.config.general["spacy_model"] = spacy_model
+            cdb.config.general.nlp.modelname = spacy_model
         else:
             logging.warning("SPACY_MODEL environment var not set" +
                             ", attempting to load the spacy model found within the CDB : "
-                            + cdb.config.general["spacy_model"])
+                            + cdb.config.general.nlp.modelname)
 
-            if cdb.config.general["spacy_model"] == "":
+            if cdb.config.general.nlp.modelname == "":
                 raise ValueError("No SPACY_MODEL env var declared, the CDB loaded does not have a\
                      spacy_model set in the config variable! \
                  To solve this declare the SPACY_MODEL in the env_medcat file.")
@@ -330,18 +335,21 @@ def _create_cat(self):
         if os.getenv("APP_MODEL_META_PATH_LIST", None) is not None:
             self.log.debug("Loading META annotations ...")
             for model_path in os.getenv("APP_MODEL_META_PATH_LIST").split(":"):
-                m = MetaCAT.load(model_path)
+                m = MetaCATAddon.deserialise_from(model_path)
                 meta_models.append(m)
 
-        if cat:
-            meta_models.extend(cat._meta_cats)
+        # if cat:
+        #     meta_models.extend(cat._meta_cats)
 
         if self.app_model.lower() in [None, "unknown"]:
-            self.app_model = cdb.config.version.id
+            self.app_model = cdb.config.meta.hash
 
-        config.general["log_level"] = os.getenv("LOG_LEVEL", logging.INFO)
+        config.general.log_level = os.getenv("LOG_LEVEL", logging.INFO)
 
-        cat = CAT(cdb=cdb, config=config, vocab=vocab, meta_cats=meta_models)
+        cat = CAT(cdb=cdb, config=config, vocab=vocab)
+        # add MetaCATs
+        for mc in meta_models:
+            cat.add_addon(mc)
 
         self._populate_model_card_info(cat.config)
 

medcat-service/models/examples/examples.md

@@ -2,7 +2,7 @@
 
 ## [example-medcat-v1-model-pack][(models/examples/example-medcat-v1-model-pack.zip)
 - This model pack is built by running the MedCAT V1 Tutorial Part 3.1.
-- https://github.com/CogStack/MedCATtutorials/blob/5a07e4d77da404631cc16b47d3f1c6bd028de396/notebooks/introductory/Part_3_1_Building_a_Concept_Database_and_Vocabulary.ipynb
+- https://github.com/CogStack/cogstack-nlp/blob/main/medcat-v1-tutorials/notebooks/introductory/Part_3_1_Building_a_Concept_Database_and_Vocabulary.ipynb
 
 It isn't a trained model, but has the concepts "Kidney Failure" and "Failure of Kidneys" built in
 

medcat-service/requirements.txt

@@ -6,7 +6,7 @@ setuptools==78.1.1
 simplejson==3.19.3
 werkzeug==3.1.3
 setuptools-rust==1.11.0
-medcat==1.16.0
+medcat[meta-cat,spacy,deid] @ git+https://github.com/CogStack/cogstack-nlp.git@refs/tags/medcat/v0.13.5#subdirectory=medcat-v2
 # pinned because of issues with de-id models and past models (it will not do any de-id)
 transformers>=4.34.0,<5.0.0
-requests==2.32.4
+requests==2.32.4