From 87025e10eea6a444b14de4418b8ffd5d166f8afa Mon Sep 17 00:00:00 2001 From: Antoine Phan Date: Tue, 27 May 2025 11:01:19 -0400 Subject: [PATCH 1/7] Add READMEs to each folder To have local documentation and avoid navigating the wiki --- Development.md | 10 +++++ README.md | 56 +++++++++++++++++++++++++++ e2e/README.md | 0 gpu-celery/README.md | 0 hooks/README.md | 5 +++ iipsrv/README.md | 0 nginx/README.md | 1 + postgres/README.md | 5 +++ rodan-client/{readme.md => README.md} | 33 ++++++++++++++++ 9 files changed, 110 insertions(+) create mode 100644 Development.md create mode 100644 README.md create mode 100644 e2e/README.md create mode 100644 gpu-celery/README.md create mode 100644 hooks/README.md create mode 100644 iipsrv/README.md create mode 100644 nginx/README.md create mode 100644 postgres/README.md rename rodan-client/{readme.md => README.md} (62%) diff --git a/Development.md b/Development.md new file mode 100644 index 000000000..6189cf300 --- /dev/null +++ b/Development.md @@ -0,0 +1,10 @@ + +# Architecture +## Rodan Frontend components + +Rodan frontend has 3 main components, all Dockerized: +- `nginx` - a reverse proxy server. It also serves the minified version of the web viewer when not working on the frontend. +- `rodan-client` - web viewer, for developing the Rodan-Client front end +- `iipsrv` - a high-performance feature-rich image server for web-based streamed viewing and zooming of ultra high-resolution images (Source: https://github.com/ruven/iipsrv) + +Developers can read the READMEs in each folder to work on each components diff --git a/README.md b/README.md new file mode 100644 index 000000000..687fb7451 --- /dev/null +++ b/README.md @@ -0,0 +1,56 @@ +# Rodan + +- Rodan ![GitHub release](https://img.shields.io/github/release/ddmal/rodan) ![GitHub pull requests](https://img.shields.io/github/issues-pr/ddmal/rodan) ![GitHub issues](https://img.shields.io/github/issues/ddmal/rodan) + - Master Branch ![GitHub last commit (branch)](https://img.shields.io/github/last-commit/ddmal/rodan/master) + - Develop Branch ![GitHub last commit (branch)](https://img.shields.io/github/last-commit/ddmal/rodan/develop) + +This repository contains Docker images that can be used to set up [Rodan](https://github.com/DDMAL/rodan) locally for development. These images can also be used in the future with slight modifications for deployment to a swarm production environment. Please see the wiki for more information about deploying Rodan. [Rodan Wiki](https://github.com/DDMAL/Rodan/wiki) + +#### Objectives + +- Simplify the installation process of Rodan on all platforms. +- Maintain clear installation documentation. + +## Quick Start + +If you are working on **Rodan** or **Rodan Jobs** + +- Make sure you have Rodan submodule cloned in `${repository_root}/rodan/code` and **it is up to date** with the branch you wish to work with. The branches should be either `develop`, or the **name of the feature** you would like to include into develop. The `master` branch is only for version releases and is supposed to be a guaranteed working version. +- Follow the instructions here: https://github.com/DDMAL/Rodan/wiki/Working-on-Rodan + - Note the `BRANCHES` environment variable in the installation scripts, you can set the environment variable locally by running the following command: `export BRANCHES="develop"`. + +If you are working on **Rodan-Client** + +- Make sure you have Rodan-Client cloned in `${repository_root}/rodan-client/code` and it is up to date with the branch you wish to work with. +- Follow the instructions here: https://github.com/DDMAL/Rodan/wiki/Working-on-Rodan-Client + +## Tips for Interacting with Running Containers + +The following commands may seem familiar to you if you have worked with Posix systems, or bash shells in general. Many of the commands that exist for docker, by just adding the prefix `docker`. + +- If you would like to see a list of all running containers on your machine, execute: `docker ps` +- To copy files between the container and the host, it is the same way you would use scp between different computers, execute: `docker cp`, +- Other commands like `docker top` are also available to monitor resources outside of the containers. + +A similar concept to using `exec` is using SSH to connect to another computer. We use `exec` to connect to a specific container. It is much simpler to use `docker compose exec`, instead of the `docker exec`. Docker compose will search the configuration inside `docker-compose.yml` to know which service is being referenced. The format of the command works this way: + +- `docker compose exec ` +- The command could be anything eg: `/opt/some_directory/my_shell_script.sh` +- A command you will use frequently is: `docker compose exec rodan bash` or `docker compose exec celery bash` for investigating problems. **You should not be using this command to edit files, use `docker volumes` and your IDE outside of the container.** + +Consult the documentation of the [Docker command line](https://docs.docker.com/engine/reference/commandline/cli/) for additional information. + +## Automated Build + +The images are rebuilt and pushed automatically on a nightly basis at 2am. This accomplished with a cron job. You must point the cron job to the nightly script on one of the staging virtual machines. Any account will do and no authentication required, add this line to the crontab. Docker hub will send a Slack notification if the image has built. We should expect 5 new images daily, or more if there was a new tagged release of any of them. + +```shell +0 2 * * 1-5 /srv/webapps/rodan-docker/scripts/nightly +``` + +You may also force Docker Cloud to rebuild new images when new commits are pushed to a Git repository. Unfortunately, we had problems connecting the `rodan-docker` GitHub repository to Docker Cloud due to authentication issues, so we set up a private repository on Bitbucket instead. + +## Additional Information + +For more information about volumes in Docker, see [Use volumes](https://docs.docker.com/engine/admin/volumes/volumes/) in the Docker documentation. See also the docs for the [`volumes` section](https://docs.docker.com/compose/compose-file/#volumes) of the `docker-compose.yml` file. + diff --git a/e2e/README.md b/e2e/README.md new file mode 100644 index 000000000..e69de29bb diff --git a/gpu-celery/README.md b/gpu-celery/README.md new file mode 100644 index 000000000..e69de29bb diff --git a/hooks/README.md b/hooks/README.md new file mode 100644 index 000000000..f57d3e149 --- /dev/null +++ b/hooks/README.md @@ -0,0 +1,5 @@ +# Rodan Docker hooks +> _From https://github.com/DDMAL/Rodan/wiki/Repository-Structure_ +> +> The `hooks` folder is specific to docker, and it holds settings for the automatic docker builds in the cloud. + diff --git a/iipsrv/README.md b/iipsrv/README.md new file mode 100644 index 000000000..e69de29bb diff --git a/nginx/README.md b/nginx/README.md new file mode 100644 index 000000000..a7aeb47b6 --- /dev/null +++ b/nginx/README.md @@ -0,0 +1 @@ +# Rodan's Nginx diff --git a/postgres/README.md b/postgres/README.md new file mode 100644 index 000000000..c5cdd6b31 --- /dev/null +++ b/postgres/README.md @@ -0,0 +1,5 @@ +# Rodan main database +> From https://github.com/DDMAL/Rodan/wiki/Repository-Structure + +- The `maintenance` folder is a collection of database maintenance scripts. These are vital to backups, do not change them unless you know what you are doing. +- Rodan uses Redis to query data from `postgres`, plus some perks such as data caching. \ No newline at end of file diff --git a/rodan-client/readme.md b/rodan-client/README.md similarity index 62% rename from rodan-client/readme.md rename to rodan-client/README.md index 070c017c5..f4beed9f5 100644 --- a/rodan-client/readme.md +++ b/rodan-client/README.md @@ -1,3 +1,36 @@ +# Rodan client-facing UI +## Folder structure +> From https://github.com/DDMAL/Rodan/wiki/Repository-Structure + +- `code`: submodule for the repository +- `config`: configuration files for the container +- `scripts`: point to start scripts and docker entry points for the container. These scripts exist to help you for that specific container. There can also be scripts to help build the service outside of the dockerfile. + + +# __rodan-client__ + +Rodan-Client is a GUI that allows you to interact with Rodan jobs and workflows. + +**UPDATE:** You should not need to install rodan-client this way anymore. `Rodan-Docker` does everything for us. + +## Prerequisites +Install `Homebrew`, `Yarn`, `Gulp` and `git` if you have not already. You should install both `yarn` and `gulp` globally. +``` +/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" +brew install yarn +yarn global add gulp +``` + +## Installation +- Clone project `git clone https://github.com/DDMAL/rodan-client` +- Change to the project directory and run `yarn install` +- Copy the `configuration.example.json` file, and rename it to `configuration.json` +- Edit the `SERVER_HOST` and `SERVER_PORT` accordingly + - If you're using docker in local development, use `localhost` for the server, and port `8000`. You will also need to set `SERVER_HTTPS` to false unless you setup local SSL certificates. +- From the root project directory, travel to `./node_modules/.bin/`. +- Run `gulp` + + ## Step-by-step to run Rodan Client on M1 1. Navigate to rodan. `cp rodan-client/local-dev/arm-compose.yml .` 2. Navigate to rodan. `make run_arm` From d3ce085793020b6ffed62f97d976c375f01f0932 Mon Sep 17 00:00:00 2001 From: Antoine Phan Date: Tue, 27 May 2025 11:19:36 -0400 Subject: [PATCH 2/7] Development notes for Rodan architecture --- Development.md | 26 ++++++++++++++++++++++-- readme.md | 55 -------------------------------------------------- 2 files changed, 24 insertions(+), 57 deletions(-) delete mode 100644 readme.md diff --git a/Development.md b/Development.md index 6189cf300..bccb75ed4 100644 --- a/Development.md +++ b/Development.md @@ -1,6 +1,11 @@ - # Architecture -## Rodan Frontend components +> From [Architecture Revamp #1297]https://github.com/DDMAL/Rodan/issues/1297#issuecomment-2910020124 +> and an old wiki document: [Rodan pre-Docker times](https://github.com/DDMAL/Rodan/wiki/Installation-without-Docker) +> +> Refercing a lot from [How Rodan Docker Works](https://github.com/DDMAL/Rodan/wiki/How-Rodan-Docker-Works) +> + +## Rodan frontend Rodan frontend has 3 main components, all Dockerized: - `nginx` - a reverse proxy server. It also serves the minified version of the web viewer when not working on the frontend. @@ -8,3 +13,20 @@ Rodan frontend has 3 main components, all Dockerized: - `iipsrv` - a high-performance feature-rich image server for web-based streamed viewing and zooming of ultra high-resolution images (Source: https://github.com/ruven/iipsrv) Developers can read the READMEs in each folder to work on each components + +## Rodan middle layer +### `RabbitMQ` job queue +- RabbitMQ is a message queue to track jobs for `celery` - a Python library used by Rodan for job management. +- When a user "run a job" on the client, a job is queued via RabbitMQ to the backend. + +### `Redis` and `PostgreSQL` +- `redis` - a key-value database used by the Rodan server to manage websocket connections. It is used to auto-refresh results on the web interface. +- `postgres-plpython` (inside `postgres` folder) - the PostgreSQL database used by the Rodan server, with the Python extensions installed and custom backup functions. + +## Rodan backend: `celery`-related and RESTful API `rodan-main` +- `rodan-main` - the Rodan server providing a REST API. +- `celery` - the Rodan asynchronous task runner. There are different celery services for each queue: + - Main `celery`: handles basic Rodan tasks, like uploading resources and saving workflows + - `py3-celery` + - `gpu-celery`: Rodan GPU-running jobs, such as PACO, Classifier, etc. + diff --git a/readme.md b/readme.md deleted file mode 100644 index ea2e27b88..000000000 --- a/readme.md +++ /dev/null @@ -1,55 +0,0 @@ -# Rodan - -- Rodan ![GitHub release](https://img.shields.io/github/release/ddmal/rodan) ![GitHub pull requests](https://img.shields.io/github/issues-pr/ddmal/rodan) ![GitHub issues](https://img.shields.io/github/issues/ddmal/rodan) - - Master Branch ![GitHub last commit (branch)](https://img.shields.io/github/last-commit/ddmal/rodan/master) - - Develop Branch ![GitHub last commit (branch)](https://img.shields.io/github/last-commit/ddmal/rodan/develop) - -This repository contains Docker images that can be used to set up [Rodan](https://github.com/DDMAL/rodan) locally for development. These images can also be used in the future with slight modifications for deployment to a swarm production environment. Please see the wiki for more information about deploying Rodan. [Rodan Wiki](https://github.com/DDMAL/Rodan/wiki) - -#### Objectives - -- Simplify the installation process of Rodan on all platforms. -- Maintain clear installation documentation. - -## Quick Start - -If you are working on **Rodan** or **Rodan Jobs** - -- Make sure you have Rodan submodule cloned in `${repository_root}/rodan/code` and **it is up to date** with the branch you wish to work with. The branches should be either `develop`, or the **name of the feature** you would like to include into develop. The `master` branch is only for version releases and is supposed to be a guaranteed working version. -- Follow the instructions here: https://github.com/DDMAL/Rodan/wiki/Working-on-Rodan - - Note the `BRANCHES` environment variable in the installation scripts, you can set the environment variable locally by running the following command: `export BRANCHES="develop"`. - -If you are working on **Rodan-Client** - -- Make sure you have Rodan-Client cloned in `${repository_root}/rodan-client/code` and it is up to date with the branch you wish to work with. -- Follow the instructions here: https://github.com/DDMAL/Rodan/wiki/Working-on-Rodan-Client - -## Tips for Interacting with Running Containers - -The following commands may seem familiar to you if you have worked with Posix systems, or bash shells in general. Many of the commands that exist for docker, by just adding the prefix `docker`. - -- If you would like to see a list of all running containers on your machine, execute: `docker ps` -- To copy files between the container and the host, it is the same way you would use scp between different computers, execute: `docker cp`, -- Other commands like `docker top` are also available to monitor resources outside of the containers. - -A similar concept to using `exec` is using SSH to connect to another computer. We use `exec` to connect to a specific container. It is much simpler to use `docker compose exec`, instead of the `docker exec`. Docker compose will search the configuration inside `docker-compose.yml` to know which service is being referenced. The format of the command works this way: - -- `docker compose exec ` -- The command could be anything eg: `/opt/some_directory/my_shell_script.sh` -- A command you will use frequently is: `docker compose exec rodan bash` or `docker compose exec celery bash` for investigating problems. **You should not be using this command to edit files, use `docker volumes` and your IDE outside of the container.** - -Consult the documentation of the [Docker command line](https://docs.docker.com/engine/reference/commandline/cli/) for additional information. - -## Automated Build - -The images are rebuilt and pushed automatically on a nightly basis at 2am. This accomplished with a cron job. You must point the cron job to the nightly script on one of the staging virtual machines. Any account will do and no authentication required, add this line to the crontab. Docker hub will send a Slack notification if the image has built. We should expect 5 new images daily, or more if there was a new tagged release of any of them. - -```shell -0 2 * * 1-5 /srv/webapps/rodan-docker/scripts/nightly -``` - -You may also force Docker Cloud to rebuild new images when new commits are pushed to a Git repository. Unfortunately, we had problems connecting the `rodan-docker` GitHub repository to Docker Cloud due to authentication issues, so we set up a private repository on Bitbucket instead. - -## Additional Information - -For more information about volumes in Docker, see [Use volumes](https://docs.docker.com/engine/admin/volumes/volumes/) in the Docker documentation. See also the docs for the [`volumes` section](https://docs.docker.com/compose/compose-file/#volumes) of the `docker-compose.yml` file. From 7d5f42b2ba658a209a3813d0420377356a0c80bc Mon Sep 17 00:00:00 2001 From: Antoine Phan Date: Wed, 28 May 2025 09:36:38 -0400 Subject: [PATCH 3/7] Update documentation for Rodan Client --- rodan-client/README.md | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/rodan-client/README.md b/rodan-client/README.md index f4beed9f5..11dc0f066 100644 --- a/rodan-client/README.md +++ b/rodan-client/README.md @@ -1,4 +1,6 @@ -# Rodan client-facing UI +# Rodan client +- A GUI that allows you to interact with Rodan jobs and workflows. + ## Folder structure > From https://github.com/DDMAL/Rodan/wiki/Repository-Structure @@ -7,17 +9,19 @@ - `scripts`: point to start scripts and docker entry points for the container. These scripts exist to help you for that specific container. There can also be scripts to help build the service outside of the dockerfile. -# __rodan-client__ - -Rodan-Client is a GUI that allows you to interact with Rodan jobs and workflows. +--- -**UPDATE:** You should not need to install rodan-client this way anymore. `Rodan-Docker` does everything for us. +**UPDATE:** You should not need to install rodan-client this way anymore. **Docker** does everything for us. ## Prerequisites -Install `Homebrew`, `Yarn`, `Gulp` and `git` if you have not already. You should install both `yarn` and `gulp` globally. -``` -/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" +Using your package manager, e.g., [`homebrew`](https://brew.sh) for MacOS; `apt` for Linux/WSL; `chocolatey` for Windows, install the following: +- Install `yarn` via Homebrew: +```sh brew install yarn +``` + +- Install `gulp` if you have not already. You should install both `yarn` and `gulp` globally. +```sh yarn global add gulp ``` From 49e99c75887d9e91c3098ed6af783c2e6eecea2f Mon Sep 17 00:00:00 2001 From: Antoine Phan Date: Wed, 28 May 2025 10:32:47 -0400 Subject: [PATCH 4/7] WIP: Celery documentation --- python3-celery/README.md | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 python3-celery/README.md diff --git a/python3-celery/README.md b/python3-celery/README.md new file mode 100644 index 000000000..25e96578f --- /dev/null +++ b/python3-celery/README.md @@ -0,0 +1,6 @@ +# Python3 Celery Workers + +This Docker contains the following workers: +- `pixel_wrapper` +- `neon_wrapper` +- `gamera` From dc82381875459f8d243a715b241db0a08ed19170 Mon Sep 17 00:00:00 2001 From: Antoine Phan Date: Thu, 29 May 2025 09:28:35 -0400 Subject: [PATCH 5/7] python3-celery documentation --- python3-celery/README.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/python3-celery/README.md b/python3-celery/README.md index 25e96578f..253cf2d6e 100644 --- a/python3-celery/README.md +++ b/python3-celery/README.md @@ -1,6 +1,10 @@ -# Python3 Celery Workers +# Python3 Celery -This Docker contains the following workers: -- `pixel_wrapper` -- `neon_wrapper` -- `gamera` +This Docker contains the following jobs: +| Job title | Link/Info | +| --------- | ---- | +| `pixel_wrapper` | https://github.com/DDMAL/pixel_wrapper.git | +| `neon_wrapper` | https://github.com/DDMAL/neon_wrapper | +| `gamera` | Pre-built Docker Image [ddmal/gamera4:2.0.0](FROM ddmal/gamera4:2.0.0 AS gamera) | + +The container uses `celery` to spin up multiple **workers** that process **jobs**. Each **job** is executed when a **worker** becomes available and a task is present in the RabbitMQ broker queue. \ No newline at end of file From 9ad42f967f451e9fd3e98440b1134a44be71aa7e Mon Sep 17 00:00:00 2001 From: Antoine Phan Date: Fri, 30 May 2025 12:38:59 -0400 Subject: [PATCH 6/7] Update documentation for iipsrv and the "useless" Dockerfile --- Dockerfile | 2 +- iipsrv/Dockerfile | 2 ++ iipsrv/README.md | 22 ++++++++++++++++++++++ 3 files changed, 25 insertions(+), 1 deletion(-) diff --git a/Dockerfile b/Dockerfile index 31abcc274..cba117e44 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,3 +1,3 @@ FROM debian -# For more information about why this is here, check the build hooks. +# For more information about why this is here, check `hooks/build` file on line 38 RUN printenv \ No newline at end of file diff --git a/iipsrv/Dockerfile b/iipsrv/Dockerfile index a13bdbf7d..cfa6f7b87 100644 --- a/iipsrv/Dockerfile +++ b/iipsrv/Dockerfile @@ -1,3 +1,4 @@ +# ---- Install and Build `iipsrv` ---- FROM alpine:3.18.5 AS builder RUN apk add --no-cache \ @@ -15,6 +16,7 @@ RUN ./autogen.sh RUN ./configure --enable-openjpeg RUN make +# ---- Run the `iipsrv` server ---- FROM alpine:3.18.5 RUN apk add --no-cache \ diff --git a/iipsrv/README.md b/iipsrv/README.md index e69de29bb..21425ebb8 100644 --- a/iipsrv/README.md +++ b/iipsrv/README.md @@ -0,0 +1,22 @@ +# IIPImage - High Resolution Streaming Image Server +> GPLv3 Licensed + +- `iipsrv` is high-performance feature-rich image server for web-based streamed viewing and zooming of ultra high-resolution images +- Source code: https://github.com/ruven/iipsrv + +## Dockerfile +(Courtesy of ChatGPT) Here is a short bullet-point description of what this Dockerfile does: + +### Build Stage +* **Base Image (Build Stage)**: Uses `alpine:3.18.5` for a lightweight build environment. +* **Installs Build Tools**: Adds necessary packages like `autoconf`, `automake`, `build-base`, and image libraries. +* **Downloads iipsrv Source**: Fetches the latest `iipsrv` source code from GitHub. +* **Builds iipsrv**: Configures and compiles `iipsrv` with OpenJPEG support. + +### Runtime Stage / Main Image +* **Base Image (Runtime Stage)**: Uses `alpine:3.18.5` for the final runtime container. +* **Installs Runtime Dependencies**: Includes required libraries like `libpng`, `tiff`, and `openjpeg`. +* **Copies Compiled Binary**: Transfers the built `iipsrv.fcgi` binary from the build stage. +* **Sets Environment Variables**: Configures image quality, file path prefix, verbosity, and server port. +* **Exposes Port**: Opens the specified port (default: 9003) for incoming connections. +* **Entrypoint**: Launches the `iipsrv` server bound to all interfaces on the given port. From a6ed3328f1e92c76da65b9005a59c63bd24b628e Mon Sep 17 00:00:00 2001 From: Antoine Phan Date: Fri, 30 May 2025 12:43:45 -0400 Subject: [PATCH 7/7] Checkpoint: GPU Celery docs --- gpu-celery/README.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/gpu-celery/README.md b/gpu-celery/README.md index e69de29bb..2f8382472 100644 --- a/gpu-celery/README.md +++ b/gpu-celery/README.md @@ -0,0 +1,10 @@ +# GPU Celery +> Summarized by @homework36, from https://github.com/DDMAL/Rodan/issues/1297#issuecomment-2901651690 +> +> As of May 2025. + +For the GPU-celery container, we have the following jobs: +- Text Alignment +- Background Removal +- SAE_binarization +- Paco classifier \ No newline at end of file