TaskSystem - Worker system for processing tasks

This is an exercise in adding a task so that it can be processed asynchronously and stopped at any time by manual action.

🏬 Architecture

flowchart LR
    subgraph Web API
        TA[TaskApi] -- Request --> TM[TaskManager] 
    end
    
    TM -- add_task/1 --> TQ
    TM -- list_tasks/0 --> TST
    TM -- stop_task/1 --> TW

    subgraph Application
        TS[TaskSupervisor] -- Supervises --> TW[TaskWorker]
        TR[TaskWorkerRegistry] --> TW
        TQ[TaskQueue] -- Consume --> TW
        TW -- Log --> LOG@{ shape: lean-r, label: "task result" }
    end

    subgraph Storage
       DETS[DETS] --> TQ
       TST[(TaskStorage)] <-- Fetch / Insert / Delete --> TW
    end

Loading

📋 Technical choices

1. Data Storage

   • For persistence, I used a DETS because the data is saved directly to a file on disk, which can be retrieved later
   • To be able to retrieve the list of tasks currently being processed and to share the state of the various workers, I used an Agent, which has the advantage of being a wrapper for state management and sharing.

2. Performance

   • To manage an excess of tasks that could be sent all at once, I used the :queue module provided in Erlang, which is a FIFO queue system for queuing and redistributing in the same order the different data that need to be processed
   • For the workers, I used a system of consumers who come to look if there's a message to extract it from the queue and process it afterwards, which makes it possible to have a concurrent system for processing data more quickly.
   • To avoid duplicating processes in workers, I used the Registry module, which provides a uniqueness constraint, as each registry entry is directly linked to its affiliated process.

3. Web API

   • To keep the API as simple as possible, I just used a plug and bandit as the HTTP server is the one used by default on the Phoenix framework, when generating a new project, but it would have been the same to use plug_cowboy (or "cowboy" for short).

✅ TODO

• Implement a GenServer-based worker (TaskWorker)
• Implement a supervisor (TaskSupervisor)
• Implement a public API module (TaskManager)
• Implement a persistent task queue
• Provide a Dockerfile and deployment instructions
• Provide tests covering
• Include documentation
• Implement a simple Web API

💻 Local development

1. Clone the repository

git clone https://github.com/quentin-rodriguez/task-system.git
cd task-system

2. Change Version

Use the versions specified in the .tool-versions file with a tool such as mise or asdf.

mise install
# or
asdf install

3. Install dependencies

mix deps.get

4. Start the application

iex -S mix

5. (Optional) Use web API

• Create a new task

curl -X POST http://localhost:4000/tasks -d '{"name": "Jean", "number": "42"}'

• Get a list of running tasks

curl -X GET http://localhost:4000/tasks

• Stop a running task

curl -X DELETE http://localhost:4000/tasks/:id

📥 Production deployment

1. Install Fly CLI

curl -L https://fly.io/install.sh | sh

2. Login with Fly CLI

fly auth login

3. Initialize the application

fly launch

4. Deploy the application

fly deploy