Rodan Project Restructure - Architecture revamp

[notkaramel]

This is going to be very ambitious: I want to restructure Rodan's architecture and revamp how the Dockerfile are being configured currently.

After discussion with @homework36, we think that Rodan could benefit vastly from a restructure:

• Each job (GPU and non-GPU) can have its own container.
  • Each job has its own list of dependencies, e.g., tensorflow 2.5.1 with keras 2.5.0 only work with python3.7.
  • This could resolve A LOT of "cross dependencies" between jobs; and between the Dockerfile base image to the jobs themselves.
• Usage of Kubernetes (K8s)
  • Since each job has its own container, there will be a lot of containers. K8s can help with orchestrating the system.
• Limited GPU resources on staging and production servers
  • As of writing this, I'm unsure if K8s (or docker compose) has the ability to orchestrate GPU resources. Quick research gave me Scheduling GPUs - K8s docs
• Update folder structure
  • This would make development less daunting and confusing. Some folders are nested and being called from other folders

---

• I will follow up on this thread on how to restructure (with graphics, maybe)
• I will also create sub-issues since this is a relative big project to be done for a summer

---

List of Rodan issues that could benefit from this change:

• optimize Dockerfile #1289
• docker hub build failure due to package version issue #1260, relating to PRs Rebuilding Dockerfile for GPU-Celery #1294 and Optimize gpu-celery container #751
• cannot run PACO train on prod (vGPU server) #1181
• Delete files that aren't used anywhere in the code base. #1018 (why hasn't anyone merged this PR 😭)
• rewrite Dockerfile for arm64 and have a separate GPU-celery container for local rodan on m-chip machines #1288
• and many more!!

[notkaramel]

@homework36 I found some resources that might be helpful:

• https://kubernetes.io/docs/tutorials/kubernetes-basics/
• https://training.linuxfoundation.org/training/introduction-to-kubernetes/

[notkaramel]

A quick note on philosophy and design choices: Microservice architecture and Separation of concerns

• Rodan follows a half monolith (rodan-main, gpu-celery, and some others) and a half microservices (postgres, rodan-client, etc.) architecture as of writing.
• I believe there will be benefit to break down some existing containers into smaller microservices, both for development and production.
• There will be cost, inevitably: developers coming to the project will have to (maybe) learn K8s on top of Docker, which might take some extra work hours. I will try my best to minimize this and leave production stuff to deployment time only, i.e., development will only run K8s once, and focus only on Docker

---

• Reference: https://aws.amazon.com/microservices/

[homework36]

For the GPU-celery container, we have the following jobs:

1. Text Alignment
2. Background Removal
3. SAE_binarization
4. Paco classifier

I would say we can separate these first and make four Docker images/containers for them, and we should also consider upgrading the TensorFlow version. I believe 2.5.1 is too old. Currently, the only choice for GPU-celery base image is nvidia/cuda:11.2.2-cudnn8-runtime-ubuntu20.04 and it has multiple risks too. Also, Python 3.7 is EOL, and it is probably not a good idea to keep using that in production, if we can upgrade...

I just tested nvidia/cuda:11.2.2-cudnn8-runtime-ubuntu20.04 (see comment here) and it seems to work.

[notkaramel]

I would say we can separate these first and make four Docker images/containers for them

Sounds good! I'm all for it :)

My concern for now is: celery is a tool to manage different asynchronous jobs, hence why it's used in the rodan-main Django app. I'm currently unsure if K8s or Docker can do what celery does, but I have hopes!

[notkaramel]

in another word: I'm thinking of (potentially) replacing celery with K8s/Docker to separate the jobs from its "mother" container

[notkaramel]

I found this forum post that might be interesting, on Celery and K8s: https://discuss.kubernetes.io/t/are-queue-based-background-workers-still-relevant-on-k8s/9965

[notkaramel]

I would say we can separate these first and make four Docker images/containers for them, and we should also consider upgrading the TensorFlow version. I believe 2.5.1 is too old. Currently, the only choice for GPU-celery base image is nvidia/cuda:11.2.2-cudnn8-runtime-ubuntu20.04 and it has multiple risks too. Also, Python 3.7 is EOL, and it is probably not a good idea to keep using that in production, if we can upgrade...

I just tested nvidia/cuda:11.2.2-cudnn8-runtime-ubuntu20.04 (see comment here) and it seems to work.

I just checked and it looks amazing and so clean (sorry it took a while)! Thank you :)

I just figured out the architecture of the current Rodan, so I think we can slowly start the separation process for all of these microservices (will update in the following comment)

[notkaramel]

EDIT: This understanding phase is now partially incorrect on the backend and job queues side. Please scroll down for updates

Excalidraw sketch: https://excalidraw.com/#json=gqZd4RX1ouWp2tsd36yAO,-xWXHim2dTW4j4Qd89SE1g

I might be wrong on some of these (only spent a few days with some reseach).
I will refer to things in Kubernetes terms, but you can imagine each of them is a "Container" or a VM.

Node 1: Frontend

This is the client facing side of Rodan, with rodan-client holding the source code for the interface. nginx is used to serve the build files to the incoming users/clients. iipsrv from what I understand is an image streaming server for frontend, perhaps it's used in showing results and images, I guess?

Node 2: Job queue & database

The client sends a job request (in redis format instead of HTTP, to my understanding) to the job broker/queue, either RabbitMQ or Redis. The job is then sent to the corresponding backend workers (via celery), and the system returns a hashcode as a "Promise" or "a ticket". Once the job is completed in the backend, the result is sent back to the Redis backend, and some of the data is stored in PostgreSQL so users can retrieve it whenever they want, with that "job ticket" from Redis.

I haven't figured out why they both exist, Redis can act as both the broker and backend to hold the results at the same time.

Node 3, 4, 5, 6: Backend (mega-node)

This is a series of nodes that holds a Django app that uses celery to distribute jobs to its workers. These nodes includes:

• rodan-main - the folder has a Django app
• celery
• py3-celery: what's the difference between this and celery??
• gpu-celery: it holds a set of ~4 GPU-needed jobs

My educated guess is to do the following:

• Keep celery in some CPU-running app, like rodan-main, since they are still functional. It poses minor inconvenience for devs who want to fix one small job, but all jobs needs to be online during dev time.
• Make a clearer separation of the source code for each node, i.e., all of them can use files in rodan-main but needs to be documented. Otherwise, we want to separate each node into its own folder.

Miscallaneous

• There's a folder for scripts. I can see the purpose of these scripts, but in terms of "Locality of Behaviour" (Here's a brief intro: https://htmx.org/essays/locality-of-behaviour/), it makes it super confusing for incoming devs to figure out what is going on.
• Is it conflicting with "Separation of Concerns" design pattern? YES! but the cost is far lower for future maintainer to figure out what's wrong with one particular part of this gigantic project IMHO.