diff --git a/Development.md b/Development.md new file mode 100644 index 000000000..bccb75ed4 --- /dev/null +++ b/Development.md @@ -0,0 +1,32 @@ +# Architecture +> From [Architecture Revamp #1297]https://github.com/DDMAL/Rodan/issues/1297#issuecomment-2910020124 +> and an old wiki document: [Rodan pre-Docker times](https://github.com/DDMAL/Rodan/wiki/Installation-without-Docker) +> +> Refercing a lot from [How Rodan Docker Works](https://github.com/DDMAL/Rodan/wiki/How-Rodan-Docker-Works) +> + +## Rodan frontend + +Rodan frontend has 3 main components, all Dockerized: +- `nginx` - a reverse proxy server. It also serves the minified version of the web viewer when not working on the frontend. +- `rodan-client` - web viewer, for developing the Rodan-Client front end +- `iipsrv` - a high-performance feature-rich image server for web-based streamed viewing and zooming of ultra high-resolution images (Source: https://github.com/ruven/iipsrv) + +Developers can read the READMEs in each folder to work on each components + +## Rodan middle layer +### `RabbitMQ` job queue +- RabbitMQ is a message queue to track jobs for `celery` - a Python library used by Rodan for job management. +- When a user "run a job" on the client, a job is queued via RabbitMQ to the backend. + +### `Redis` and `PostgreSQL` +- `redis` - a key-value database used by the Rodan server to manage websocket connections. It is used to auto-refresh results on the web interface. +- `postgres-plpython` (inside `postgres` folder) - the PostgreSQL database used by the Rodan server, with the Python extensions installed and custom backup functions. + +## Rodan backend: `celery`-related and RESTful API `rodan-main` +- `rodan-main` - the Rodan server providing a REST API. +- `celery` - the Rodan asynchronous task runner. There are different celery services for each queue: + - Main `celery`: handles basic Rodan tasks, like uploading resources and saving workflows + - `py3-celery` + - `gpu-celery`: Rodan GPU-running jobs, such as PACO, Classifier, etc. + diff --git a/Dockerfile b/Dockerfile index 31abcc274..cba117e44 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,3 +1,3 @@ FROM debian -# For more information about why this is here, check the build hooks. +# For more information about why this is here, check `hooks/build` file on line 38 RUN printenv \ No newline at end of file diff --git a/readme.md b/README.md similarity index 99% rename from readme.md rename to README.md index ea2e27b88..687fb7451 100644 --- a/readme.md +++ b/README.md @@ -53,3 +53,4 @@ You may also force Docker Cloud to rebuild new images when new commits are pushe ## Additional Information For more information about volumes in Docker, see [Use volumes](https://docs.docker.com/engine/admin/volumes/volumes/) in the Docker documentation. See also the docs for the [`volumes` section](https://docs.docker.com/compose/compose-file/#volumes) of the `docker-compose.yml` file. + diff --git a/e2e/README.md b/e2e/README.md new file mode 100644 index 000000000..e69de29bb diff --git a/gpu-celery/README.md b/gpu-celery/README.md new file mode 100644 index 000000000..2f8382472 --- /dev/null +++ b/gpu-celery/README.md @@ -0,0 +1,10 @@ +# GPU Celery +> Summarized by @homework36, from https://github.com/DDMAL/Rodan/issues/1297#issuecomment-2901651690 +> +> As of May 2025. + +For the GPU-celery container, we have the following jobs: +- Text Alignment +- Background Removal +- SAE_binarization +- Paco classifier \ No newline at end of file diff --git a/hooks/README.md b/hooks/README.md new file mode 100644 index 000000000..f57d3e149 --- /dev/null +++ b/hooks/README.md @@ -0,0 +1,5 @@ +# Rodan Docker hooks +> _From https://github.com/DDMAL/Rodan/wiki/Repository-Structure_ +> +> The `hooks` folder is specific to docker, and it holds settings for the automatic docker builds in the cloud. + diff --git a/iipsrv/Dockerfile b/iipsrv/Dockerfile index a13bdbf7d..cfa6f7b87 100644 --- a/iipsrv/Dockerfile +++ b/iipsrv/Dockerfile @@ -1,3 +1,4 @@ +# ---- Install and Build `iipsrv` ---- FROM alpine:3.18.5 AS builder RUN apk add --no-cache \ @@ -15,6 +16,7 @@ RUN ./autogen.sh RUN ./configure --enable-openjpeg RUN make +# ---- Run the `iipsrv` server ---- FROM alpine:3.18.5 RUN apk add --no-cache \ diff --git a/iipsrv/README.md b/iipsrv/README.md new file mode 100644 index 000000000..21425ebb8 --- /dev/null +++ b/iipsrv/README.md @@ -0,0 +1,22 @@ +# IIPImage - High Resolution Streaming Image Server +> GPLv3 Licensed + +- `iipsrv` is high-performance feature-rich image server for web-based streamed viewing and zooming of ultra high-resolution images +- Source code: https://github.com/ruven/iipsrv + +## Dockerfile +(Courtesy of ChatGPT) Here is a short bullet-point description of what this Dockerfile does: + +### Build Stage +* **Base Image (Build Stage)**: Uses `alpine:3.18.5` for a lightweight build environment. +* **Installs Build Tools**: Adds necessary packages like `autoconf`, `automake`, `build-base`, and image libraries. +* **Downloads iipsrv Source**: Fetches the latest `iipsrv` source code from GitHub. +* **Builds iipsrv**: Configures and compiles `iipsrv` with OpenJPEG support. + +### Runtime Stage / Main Image +* **Base Image (Runtime Stage)**: Uses `alpine:3.18.5` for the final runtime container. +* **Installs Runtime Dependencies**: Includes required libraries like `libpng`, `tiff`, and `openjpeg`. +* **Copies Compiled Binary**: Transfers the built `iipsrv.fcgi` binary from the build stage. +* **Sets Environment Variables**: Configures image quality, file path prefix, verbosity, and server port. +* **Exposes Port**: Opens the specified port (default: 9003) for incoming connections. +* **Entrypoint**: Launches the `iipsrv` server bound to all interfaces on the given port. diff --git a/nginx/README.md b/nginx/README.md new file mode 100644 index 000000000..a7aeb47b6 --- /dev/null +++ b/nginx/README.md @@ -0,0 +1 @@ +# Rodan's Nginx diff --git a/postgres/README.md b/postgres/README.md new file mode 100644 index 000000000..c5cdd6b31 --- /dev/null +++ b/postgres/README.md @@ -0,0 +1,5 @@ +# Rodan main database +> From https://github.com/DDMAL/Rodan/wiki/Repository-Structure + +- The `maintenance` folder is a collection of database maintenance scripts. These are vital to backups, do not change them unless you know what you are doing. +- Rodan uses Redis to query data from `postgres`, plus some perks such as data caching. \ No newline at end of file diff --git a/python3-celery/README.md b/python3-celery/README.md new file mode 100644 index 000000000..253cf2d6e --- /dev/null +++ b/python3-celery/README.md @@ -0,0 +1,10 @@ +# Python3 Celery + +This Docker contains the following jobs: +| Job title | Link/Info | +| --------- | ---- | +| `pixel_wrapper` | https://github.com/DDMAL/pixel_wrapper.git | +| `neon_wrapper` | https://github.com/DDMAL/neon_wrapper | +| `gamera` | Pre-built Docker Image [ddmal/gamera4:2.0.0](FROM ddmal/gamera4:2.0.0 AS gamera) | + +The container uses `celery` to spin up multiple **workers** that process **jobs**. Each **job** is executed when a **worker** becomes available and a task is present in the RabbitMQ broker queue. \ No newline at end of file diff --git a/rodan-client/readme.md b/rodan-client/README.md similarity index 62% rename from rodan-client/readme.md rename to rodan-client/README.md index 070c017c5..11dc0f066 100644 --- a/rodan-client/readme.md +++ b/rodan-client/README.md @@ -1,3 +1,40 @@ +# Rodan client +- A GUI that allows you to interact with Rodan jobs and workflows. + +## Folder structure +> From https://github.com/DDMAL/Rodan/wiki/Repository-Structure + +- `code`: submodule for the repository +- `config`: configuration files for the container +- `scripts`: point to start scripts and docker entry points for the container. These scripts exist to help you for that specific container. There can also be scripts to help build the service outside of the dockerfile. + + +--- + +**UPDATE:** You should not need to install rodan-client this way anymore. **Docker** does everything for us. + +## Prerequisites +Using your package manager, e.g., [`homebrew`](https://brew.sh) for MacOS; `apt` for Linux/WSL; `chocolatey` for Windows, install the following: +- Install `yarn` via Homebrew: +```sh +brew install yarn +``` + +- Install `gulp` if you have not already. You should install both `yarn` and `gulp` globally. +```sh +yarn global add gulp +``` + +## Installation +- Clone project `git clone https://github.com/DDMAL/rodan-client` +- Change to the project directory and run `yarn install` +- Copy the `configuration.example.json` file, and rename it to `configuration.json` +- Edit the `SERVER_HOST` and `SERVER_PORT` accordingly + - If you're using docker in local development, use `localhost` for the server, and port `8000`. You will also need to set `SERVER_HTTPS` to false unless you setup local SSL certificates. +- From the root project directory, travel to `./node_modules/.bin/`. +- Run `gulp` + + ## Step-by-step to run Rodan Client on M1 1. Navigate to rodan. `cp rodan-client/local-dev/arm-compose.yml .` 2. Navigate to rodan. `make run_arm`