Skip to content

Commit 6c75531

Browse files
tomolopolisTom Searle
tomolopolis
and
Tom Searle
authored
CU-869b5ncen: fix bugs, beginnings of a test for api backend (#224)
* CU-869b5ncen: fix bugs, beginnings of a test for api backend * CU-869b5ncen: make example.json available to docker for running tests --------- Co-authored-by: Tom Searle <tom@cogstack.org>
1 parent 6b0d14b commit 6c75531

File tree

9 files changed

+623
-26
lines changed

9 files changed

+623
-26
lines changed

‎.github/workflows/medcat-trainer_ci.yml‎

Lines changed: 50 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,8 @@ name: medcat-trainer ci-build
33
on:
44
pull_request:
55
paths:
6-
- 'medcat-trainer/**'
7-
- '.github/workflows/medcat-trainer**'
6+
- "medcat-trainer/**"
7+
- ".github/workflows/medcat-trainer**"
88

99
defaults:
1010
run:
@@ -23,7 +23,7 @@ jobs:
2323
- name: Set up Python
2424
uses: actions/setup-python@v6
2525
with:
26-
python-version: '3.10'
26+
python-version: "3.10"
2727

2828
- name: Install dependencies
2929
run: |
@@ -45,7 +45,6 @@ jobs:
4545
cd client
4646
python -m build
4747
48-
4948
test-frontend:
5049
runs-on: ubuntu-latest
5150
steps:
@@ -55,25 +54,67 @@ jobs:
5554
- name: Set up Node.js
5655
uses: actions/setup-node@v6
5756
with:
58-
node-version: '20'
57+
node-version: "20"
5958

6059
- name: Install dependencies
6160
run: |
62-
cd webapp/frontend
63-
npm ci
61+
cd webapp/frontend
62+
npm ci
6463
6564
- name: Run frontend tests
6665
run: |
6766
cd webapp/frontend
6867
npm run test:unit
6968
70-
69+
test-backend:
70+
runs-on: ubuntu-latest
71+
steps:
72+
- name: Checkout main
73+
uses: actions/checkout@v5
74+
with:
75+
ref: ${{ github.ref }}
76+
77+
- name: Set up Python
78+
uses: actions/setup-python@v6
79+
with:
80+
python-version: "3.11"
81+
82+
- name: Install system dependencies
83+
run: |
84+
sudo apt-get update
85+
sudo apt-get install -y build-essential
86+
87+
- name: Install Rust
88+
run: |
89+
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
90+
echo "$HOME/.cargo/bin" >> $GITHUB_PATH
91+
92+
- name: Install Python dependencies
93+
run: |
94+
python -m pip install --upgrade pip
95+
cd webapp
96+
pip install -r requirements.txt
97+
98+
- name: Download spaCy model
99+
run: |
100+
python -m spacy download en_core_web_md
101+
102+
- name: Run Django tests
103+
env:
104+
DB_ENGINE: sqlite3
105+
SECRET_KEY: test-secret-key
106+
DEBUG: 1
107+
run: |
108+
cd webapp/api
109+
python manage.py test
110+
71111
# Build and test webapp container
72112
build-and-push:
73113
runs-on: ubuntu-latest
74-
needs:
114+
needs:
75115
- test-client
76116
- test-frontend
117+
- test-backend
77118
steps:
78119
- name: Checkout main
79120
uses: actions/checkout@v5

‎.github/workflows/medcat-trainer_qa.yml‎

Lines changed: 51 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ name: medcat-trainer qa-build
22

33
on:
44
push:
5-
branches: [ main ]
5+
branches: [main]
66

77
permissions:
88
id-token: write
@@ -19,12 +19,12 @@ jobs:
1919
- name: Checkout main
2020
uses: actions/checkout@v5
2121
with:
22-
ref: 'main'
22+
ref: "main"
2323

2424
- name: Set up Python
2525
uses: actions/setup-python@v6
2626
with:
27-
python-version: '3.10'
27+
python-version: "3.10"
2828

2929
- name: Install dependencies
3030
run: |
@@ -64,29 +64,72 @@ jobs:
6464
- name: Set up Node.js
6565
uses: actions/setup-node@v6
6666
with:
67-
node-version: '20'
67+
node-version: "20"
6868

6969
- name: Install dependencies
7070
run: |
71-
cd webapp/frontend
72-
npm ci
71+
cd webapp/frontend
72+
npm ci
7373
7474
- name: Run frontend tests
7575
run: |
7676
cd webapp/frontend
7777
npm run test:unit
7878
79+
test-backend:
80+
runs-on: ubuntu-latest
81+
steps:
82+
- name: Checkout main
83+
uses: actions/checkout@v5
84+
with:
85+
ref: "main"
86+
87+
- name: Set up Python
88+
uses: actions/setup-python@v6
89+
with:
90+
python-version: "3.11"
91+
92+
- name: Install system dependencies
93+
run: |
94+
sudo apt-get update
95+
sudo apt-get install -y build-essential
96+
97+
- name: Install Rust
98+
run: |
99+
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
100+
echo "$HOME/.cargo/bin" >> $GITHUB_PATH
101+
102+
- name: Install Python dependencies
103+
run: |
104+
python -m pip install --upgrade pip
105+
cd webapp
106+
pip install -r requirements.txt
107+
108+
- name: Download spaCy model
109+
run: |
110+
python -m spacy download en_core_web_md
111+
112+
- name: Run Django tests
113+
env:
114+
DB_ENGINE: sqlite3
115+
SECRET_KEY: test-secret-key
116+
DEBUG: 1
117+
run: |
118+
cd webapp/api
119+
python manage.py test
120+
79121
# Build and test webapp container
80122
build-and-push:
81123
runs-on: ubuntu-latest
82-
needs:
124+
needs:
83125
- test-client
84126
- test-frontend
127+
- test-backend
85128
steps:
86129
- name: Checkout main
87130
uses: actions/checkout@v5
88131
with:
89-
ref: 'main'
132+
ref: "main"
90133

91134
- name: Build
92135
run: |

‎.github/workflows/medcat-trainer_release.yml‎

Lines changed: 71 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ name: medcat-trainer release-build
33
on:
44
push:
55
tags:
6-
- 'medcat-trainer/v*.*.*'
6+
- "medcat-trainer/v*.*.*"
77

88
permissions:
99
id-token: write
@@ -29,7 +29,7 @@ jobs:
2929
- name: Set up Python
3030
uses: actions/setup-python@v6
3131
with:
32-
python-version: '3.10'
32+
python-version: "3.10"
3333

3434
- name: Install dependencies
3535
run: |
@@ -57,10 +57,78 @@ jobs:
5757
with:
5858
packages_dir: medcat-trainer/client/dist
5959

60+
test-frontend:
61+
runs-on: ubuntu-latest
62+
steps:
63+
- name: Checkout main
64+
uses: actions/checkout@v5
65+
with:
66+
ref: "main"
67+
68+
- name: Set up Node.js
69+
uses: actions/setup-node@v6
70+
with:
71+
node-version: "20"
72+
73+
- name: Install dependencies
74+
run: |
75+
cd webapp/frontend
76+
npm ci
77+
78+
- name: Run frontend tests
79+
run: |
80+
cd webapp/frontend
81+
npm run test:unit
82+
83+
test-backend:
84+
runs-on: ubuntu-latest
85+
steps:
86+
- name: Checkout main
87+
uses: actions/checkout@v5
88+
with:
89+
ref: "main"
90+
91+
- name: Set up Python
92+
uses: actions/setup-python@v6
93+
with:
94+
python-version: "3.11"
95+
96+
- name: Install system dependencies
97+
run: |
98+
sudo apt-get update
99+
sudo apt-get install -y build-essential
100+
101+
- name: Install Rust
102+
run: |
103+
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
104+
echo "$HOME/.cargo/bin" >> $GITHUB_PATH
105+
106+
- name: Install Python dependencies
107+
run: |
108+
python -m pip install --upgrade pip
109+
cd webapp
110+
pip install -r requirements.txt
111+
112+
- name: Download spaCy model
113+
run: |
114+
python -m spacy download en_core_web_md
115+
116+
- name: Run Django tests
117+
env:
118+
DB_ENGINE: sqlite3
119+
SECRET_KEY: test-secret-key
120+
DEBUG: 1
121+
run: |
122+
cd webapp/api
123+
python manage.py test
124+
60125
# Build and test webapp container
61126
build-and-push:
62127
runs-on: ubuntu-latest
63-
needs: test-and-publish-client
128+
needs:
129+
- test-and-publish-client
130+
- test-frontend
131+
- test-backend
64132
steps:
65133
- name: Checkout main
66134
uses: actions/checkout@v5

‎medcat-trainer/webapp/api/api/data_utils.py‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,12 @@
11
import json
22
import logging
3+
import os
34
import re
45
from collections import defaultdict
56
from datetime import datetime
67
from typing import Dict, List
78

9+
import pandas as pd
810
from django.contrib.auth.models import User
911
from django.db import transaction
1012
from django.db.models import Q
@@ -19,6 +21,11 @@
1921
logger = logging.getLogger(__name__)
2022

2123

24+
class InvalidParameterError(Exception):
25+
"""Exception raised when invalid parameters are provided"""
26+
pass
27+
28+
2229
def dataset_from_file(dataset: Dataset):
2330
if '.csv' in dataset.original_file.path:
2431
df = pd.read_csv(dataset.original_file.path, on_bad_lines='error')
@@ -210,8 +217,10 @@ def upload_projects_export(
210217
m_a = MetaAnnotation()
211218
m_a.annotated_entity = a
212219
# there will be at least one or more of these available.
213-
m_a = MetaTask.objects.filter(name=task_name).first()
220+
m_a.meta_task = MetaTask.objects.filter(name=task_name).first()
214221
m_a.validated = meta_anno['validated']
222+
m_a.acc = meta_anno['acc']
223+
m_a.meta_task_value = MetaTaskValue.objects.filter(name=meta_anno['value']).first()
215224
m_a.save()
216225
# missing acc on the model
217226
anno_to_doc_ind = {a.start_ind: a for a in annos}

‎medcat-trainer/webapp/api/api/tests.py‎

Lines changed: 0 additions & 3 deletions
This file was deleted.

‎medcat-trainer/webapp/api/api/tests/__init__.py‎

Whitespace-only changes.

0 commit comments

Comments
 (0)