Skip to content

Commit f3919ef

Browse files
treld-w-moore
trel
authored and
d-w-moore
committed
[_697] Github workflows for running PRC tests
Co-authored-by: d-w-moore <dmoore@renci.org> Includes a docker compose configuration in which the PRC test suite can be run in a way representative of typical operational use, and the version of iRODS server and python interpreter that we install are also easily reconfigurable. Additionally, this affords us the opportunity to run the test suite on a client node all its own, the iRODS server being reachable only via the network.
1 parent f700ab5 commit f3919ef

26 files changed

+791
-0
lines changed

‎.github/workflows/main.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@
22
# https://github.com/actions/starter-workflows/blob/master/ci/python-package.yml
33
# (C) Github, MIT License
44

5+
# Static type checking tests using `mypy`.
6+
57
name: "Python type checking"
68

79
on: [push, pull_request]

‎.github/workflows/run-test-suite-multiple-node.yml‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# Create a networked set of containers (via a Docker compose project) on which to run the client test suite.
2+
# (For further information, see the README in `docker-testing`.)
3+
4+
name: run-test-suite-multiple-node
5+
6+
on: [push, pull_request]
7+
8+
jobs:
9+
tests:
10+
name: Python ${{ matrix.python }}, iRODS ${{ matrix.irods_server }}
11+
runs-on: ubuntu-latest
12+
defaults:
13+
run:
14+
working-directory: ./test_harness/multiple_node
15+
strategy:
16+
matrix:
17+
python: ['3.9','3.13']
18+
irods_server: ['4.3.4','5.0.2']
19+
20+
steps:
21+
- name: Checkout
22+
uses: actions/checkout@v4
23+
24+
- name: Start containers
25+
run: ./start_containers.sh "${{ matrix.irods_server }}" "${{ matrix.python }}"
26+
27+
- name: run test
28+
run: |
29+
while :; do
30+
client_container=$(docker ps --format "{{.Names}}"|grep python.client)
31+
[ -n "$client_container" ] && break
32+
sleep 1
33+
done
34+
echo "client_container = [$client_container]"
35+
docker exec "${client_container}" /repo_root/test_harness/multiple_node/run_tests.sh
36+
37+
- name: Stop containers
38+
if: always()
39+
run: ./stop_containers.sh "${{ matrix.irods_server }}"

‎test_harness/multiple_node/README.md‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
# A Topological Setup for Testing the Python Client
2+
3+
The `docker-testing` directory contains the necessary files for building and
4+
running the client test suite from the perspective of a specific client node
5+
(separate from the iRODS server node it targets for the tests).
6+
7+
The client, provider, and database are currently run on distinct nodes within
8+
a network topology set up by `docker compose` for the tests.
9+
10+
We currently allow a choice of Python interpreter and iRODS server to be installed
11+
on the client and provider nodes, respectively.
12+
13+
The choice of versions are dictated when running the test:
14+
15+
| Environment Variable | Min supported version | Max supported version |
16+
| -------------------- | --------------------- | --------------------- |
17+
| IRODS_PACKAGE_VERSION | 4.3.1 | 5.0.2 |
18+
| PYTHON_VERSION | 3.9 | 3.13 |
19+
20+
Currently the database server is fixed as Postgres.
21+
22+
## Details of usage
23+
24+
The file `.github/workflows/run-the-tests.yml` describes a Github action which
25+
will be started to run the client test suite in response to creating a pull
26+
request, or pushing new commits to the GitHub branch, containing it.
27+
28+
The command-line recipe outlined in the file also supports running the
29+
test suite on any workstation with docker compose installed.
30+
31+
A summary of how to run the tests "at the bench" follows:
32+
33+
1. Change the working directory to the root directory of the repository, e.g.:
34+
```
35+
cd /path/to/python-irodsclient
36+
```
37+
38+
2. Run:
39+
```
40+
./docker-testing/start_containers.sh 4.3.4 3.11
41+
```
42+
This builds and runs the docker images for the project, with "4.3.4" being the iRODS
43+
version installed on the provider and "3.11" being the version of python installed on the client side.
44+
45+
3. Run:
46+
```
47+
docker exec <client_container_name> /repo_root/docker-testing/run_tests.sh
48+
```
49+
(Note: `/repo_root` is an actual literal path, internal to the container.)
50+
You'll see the test output displayed on the console. At completion, xmlrunner outputs are in
51+
`/tmp/python-irodsclient/test-reports`.
52+
53+
4. Tail docker logs to see the iRODS server log.
54+
```
55+
docker logs -f <provider_container_name>
56+
```

‎test_harness/multiple_node/harness-docker-compose-irods-4.yml‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
services:
2+
irods-catalog:
3+
build:
4+
context: irods_catalog_${irods_major}
5+
environment:
6+
- POSTGRES_PASSWORD=testpassword
7+
8+
python-client:
9+
build:
10+
context: python_client
11+
args:
12+
python_version: ${python_version}
13+
command:
14+
tail -f /dev/null
15+
volumes:
16+
- ${repo_external}:/repo_root:ro
17+
- /tmp/irods-client-share.py-${python_version}:/irods_shared
18+
depends_on:
19+
irods-catalog-provider:
20+
condition: service_healthy
21+
22+
irods-catalog-provider:
23+
volumes:
24+
- /tmp/irods-client-share.py-${python_version}:/irods_shared
25+
build:
26+
context: irods_catalog_provider_${irods_major}
27+
args:
28+
irods_version: ${irods_version}
29+
shm_size: 500mb
30+
healthcheck:
31+
test: ["CMD", "su", "-", "irods", "-c", "ils || exit 1"]
32+
interval: 10s
33+
timeout: 10s
34+
retries: 3
35+
depends_on:
36+
- irods-catalog
37+

‎test_harness/multiple_node/harness-docker-compose-irods-5.yml‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
services:
2+
irods-catalog:
3+
build:
4+
context: irods_catalog_${irods_major}
5+
environment:
6+
- POSTGRES_PASSWORD=testpassword
7+
8+
python-client:
9+
build:
10+
context: python_client
11+
args:
12+
python_version: ${python_version}
13+
command:
14+
tail -f /dev/null
15+
volumes:
16+
- ${repo_external}:/repo_root:ro
17+
- /tmp/irods-client-share.py-${python_version}:/irods_shared
18+
depends_on:
19+
irods-catalog-provider:
20+
condition: service_healthy
21+
22+
irods-catalog-provider:
23+
volumes:
24+
- /tmp/irods-client-share.py-${python_version}:/irods_shared
25+
build:
26+
context: irods_catalog_provider_${irods_major}
27+
args:
28+
irods_version: ${irods_version}
29+
shm_size: 500mb
30+
healthcheck:
31+
test: ["CMD", "su", "-", "irods", "-c", "ils || exit 1"]
32+
interval: 10s
33+
timeout: 10s
34+
retries: 3
35+
depends_on:
36+
- irods-catalog
37+

‎test_harness/multiple_node/harness-docker-compose.yml‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
services:
2+
irods-catalog:
3+
build:
4+
context: irods_catalog_${irods_major}
5+
environment:
6+
- POSTGRES_PASSWORD=testpassword
7+
8+
python-client:
9+
build:
10+
context: python_client
11+
args:
12+
python_version: ${python_version}
13+
command:
14+
tail -f /dev/null
15+
volumes:
16+
- ${repo_external}:/repo_root:ro
17+
- /tmp/irods-client-share.py-${python_version}:/irods_shared
18+
depends_on:
19+
irods-catalog-provider:
20+
condition: service_healthy
21+
22+
irods-catalog-provider:
23+
volumes:
24+
- /tmp/irods-client-share.py-${python_version}:/irods_shared
25+
build:
26+
context: irods_catalog_provider_${irods_major}
27+
args:
28+
irods_version: ${irods_version}
29+
shm_size: 500mb
30+
healthcheck:
31+
test: ["CMD", "su", "-", "irods", "-c", "ils || exit 1"]
32+
interval: 10s
33+
timeout: 10s
34+
retries: 3
35+
depends_on:
36+
- irods-catalog
37+

‎test_harness/multiple_node/irods_catalog_4/Dockerfile‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
FROM postgres:12
2+
3+
COPY init-user-db.sh /docker-entrypoint-initdb.d/init-user-db.sh

‎test_harness/multiple_node/irods_catalog_4/init-user-db.sh‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
#!/bin/bash
2+
3+
# Adapted from "Initialization script" in documentation for official Postgres dockerhub:
4+
# https://hub.docker.com/_/postgres/
5+
set -e
6+
7+
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
8+
CREATE DATABASE "ICAT";
9+
CREATE USER irods WITH PASSWORD 'testpassword';
10+
GRANT ALL PRIVILEGES ON DATABASE "ICAT" to irods;
11+
EOSQL

‎test_harness/multiple_node/irods_catalog_5/Dockerfile‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
FROM postgres:16
2+
3+
COPY init-user-db.sh /docker-entrypoint-initdb.d/init-user-db.sh

‎test_harness/multiple_node/irods_catalog_5/init-user-db.sh‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
#!/bin/bash
2+
3+
# Adapted from "Initialization script" in documentation for official Postgres dockerhub:
4+
# https://hub.docker.com/_/postgres/
5+
set -e
6+
7+
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
8+
CREATE DATABASE "ICAT";
9+
CREATE USER irods WITH PASSWORD 'testpassword';
10+
GRANT ALL PRIVILEGES ON DATABASE "ICAT" to irods;
11+
ALTER DATABASE "ICAT" OWNER TO irods
12+
EOSQL

0 commit comments

Comments
 (0)