Skip to content

eco2-team/backend

Folders and files

NameName
Last commit message
Last commit date

Latest commit

ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 

Repository files navigation

Ecoยฒ Backend

Version: v1.0.7 | Changelog

515829337-6a4f523a-fa37-49de-b8e1-0a5befe26605
  • GPT Vision + Rule-based-retrieval ๊ธฐ๋ฐ˜ AI ์–ด์‹œ์Šคํ„ดํŠธ๋กœ, ํ๊ธฐ๋ฌผ ์ด๋ฏธ์ง€ ๋ถ„๋ฅ˜ยท๋ถ„๋ฆฌ๋ฐฐ์ถœ ์•ˆ๋‚ดยท์ฑ—๋ด‡ ๊ธฐ๋Šฅ์„ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค.
  • Self-managed Kubernetes 21-Nodes ํด๋Ÿฌ์Šคํ„ฐ์—์„œ Istio Service Mesh(mTLS, Auth Offloading)์™€ ArgoCD GitOps๋กœ ์šด์˜ํ•ฉ๋‹ˆ๋‹ค.
  • Redis Streams + Pub/Sub + State KV ๊ธฐ๋ฐ˜ Event Relay Layer๋กœ ์‹ค์‹œ๊ฐ„ SSE ์ด๋ฒคํŠธ๋ฅผ ์ฒ˜๋ฆฌํ•˜๊ณ , KEDA๋กœ ์ด๋ฒคํŠธ ๋“œ๋ฆฌ๋ธ ์˜คํ† ์Šค์ผ€์ผ๋ง์„ ์ˆ˜ํ–‰ํ•ฉ๋‹ˆ๋‹ค.
  • RabbitMQ + Celery ๋น„๋™๊ธฐ Task Queue๋กœ AI ํŒŒ์ดํ”„๋ผ์ธ์„ ์ฒ˜๋ฆฌํ•˜๊ณ , EFK + Jaeger๋กœ ๋กœ๊น…ยทํŠธ๋ ˆ์ด์‹ฑ์„ ์ˆ˜์ง‘ํ•ฉ๋‹ˆ๋‹ค.
  • 7๊ฐœ ๋„๋ฉ”์ธ ๋งˆ์ดํฌ๋กœ์„œ๋น„์Šค(auth, my, scan, chat, character, location, image)๋ฅผ ๋ชจ๋…ธ๋ ˆํฌ๋กœ ๊ด€๋ฆฌํ•ฉ๋‹ˆ๋‹ค.
  • ์ •์ƒ ๋ฐฐํฌ ์ค‘: https://frontend.dev.growbin.app

Service Architecture

Untitled-2025-12-29-1721
Edge Layer               : Route 53, AWS ALB, Istio Ingress Gateway
Service Layer            : auth, users, my, scan, character, location, chat (w/ Envoy Sidecar)
Integration Layer        :
  - Event Relay          : Redis Streams + Pub/Sub + State KV, Event Router, SSE Gateway
  - Worker (Storage)     : auth-worker, auth-relay, users-worker, character-worker, my-worker, character-match-worker
  - Worker (AI)          : scan-worker (Visionโ†’Ruleโ†’Answerโ†’Reward)
Persistence Layer        : PostgreSQL, Redis (Blacklist/State/Streams/Pub-Sub/Cache)
Platform Layer           : ArgoCD, Istiod, KEDA, Prometheus, Grafana, Kiali, Jaeger, EFK Stack

๋ณธ ์„œ๋น„์Šค๋Š” 5-Layer Architecture๋กœ ๊ตฌ์„ฑ๋˜์—ˆ์Šต๋‹ˆ๋‹ค.

  • Edge Layer: AWS ALB๊ฐ€ SSL Termination์„ ์ฒ˜๋ฆฌํ•˜๊ณ , ํŠธ๋ž˜ํ”ฝ์„ Istio Ingress Gateway๋กœ ์ „๋‹ฌํ•ฉ๋‹ˆ๋‹ค. Gateway๋Š” VirtualService ๊ทœ์น™์— ๋”ฐ๋ผ North-South ํŠธ๋ž˜ํ”ฝ์„ ๋ผ์šฐํŒ…ํ•ฉ๋‹ˆ๋‹ค.
  • Service Layer: ๋ชจ๋“  ๋งˆ์ดํฌ๋กœ์„œ๋น„์Šค๋Š” Istio Service Mesh ๋‚ด์—์„œ ๋™์ž‘ํ•˜๋ฉฐ, Envoy Sidecar๋ฅผ ํ†ตํ•ด mTLS ํ†ต์‹ , ํŠธ๋ž˜ํ”ฝ ์ œ์–ด, ๋ฉ”ํŠธ๋ฆญ ์ˆ˜์ง‘์„ ์ˆ˜ํ–‰ํ•ฉ๋‹ˆ๋‹ค. authโ†’users gRPC ํ†ต์‹ ์œผ๋กœ ๋„๋ฉ”์ธ ๊ฐ„ ๋™๊ธฐ ํ˜ธ์ถœ์„ ์ฒ˜๋ฆฌํ•ฉ๋‹ˆ๋‹ค.
  • Integration Layer - Event Relay: Redis Streams(๋‚ด๊ตฌ์„ฑ) + Pub/Sub(์‹ค์‹œ๊ฐ„) + State KV(๋ณต๊ตฌ) 3-tier ์ด๋ฒคํŠธ ์•„ํ‚คํ…์ฒ˜๋กœ SSE ํŒŒ์ดํ”„๋ผ์ธ์„ ์ฒ˜๋ฆฌํ•ฉ๋‹ˆ๋‹ค. RabbitMQ + Celery ๋น„๋™๊ธฐ Task Queue๋กœ AI ํŒŒ์ดํ”„๋ผ์ธ(Visionโ†’Ruleโ†’Answerโ†’Reward)์„ ์ฒ˜๋ฆฌํ•˜๊ณ , KEDA๊ฐ€ ์ด๋ฒคํŠธ ๋“œ๋ฆฌ๋ธ ์˜คํ† ์Šค์ผ€์ผ๋ง์„ ์ˆ˜ํ–‰ํ•ฉ๋‹ˆ๋‹ค.
  • Integration Layer - Worker: Storage Worker(worker-storage ๋…ธ๋“œ)๋Š” Persistence Layer์— ์ ‘๊ทผํ•˜์—ฌ ๋ฐ์ดํ„ฐ๋ฅผ ๋™๊ธฐํ™”ํ•ฉ๋‹ˆ๋‹ค. auth-worker๋Š” RabbitMQ์—์„œ ๋ธ”๋ž™๋ฆฌ์ŠคํŠธ ์ด๋ฒคํŠธ๋ฅผ ์†Œ๋น„ํ•ด Redis์— ์ €์žฅํ•˜๊ณ , auth-relay๋Š” Redis Outbox ํŒจํ„ด์œผ๋กœ ์‹คํŒจ ์ด๋ฒคํŠธ๋ฅผ ์žฌ๋ฐœํ–‰ํ•ฉ๋‹ˆ๋‹ค. users-worker๋Š” Celery Batch๋กœ ์บ๋ฆญํ„ฐ ์†Œ์œ ๊ถŒ์„ PostgreSQL์— UPSERTํ•ฉ๋‹ˆ๋‹ค. AI Worker(worker-ai ๋…ธ๋“œ)๋Š” OpenAI API์™€ ํ†ต์‹ ํ•˜๋ฉฐ, scan-worker๊ฐ€ Visionโ†’Ruleโ†’Answerโ†’Reward ์ฒด์ธ์„ gevent pool๋กœ ์ฒ˜๋ฆฌํ•ฉ๋‹ˆ๋‹ค.
  • Persistence Layer: ์„œ๋น„์Šค๋Š” ์˜์†์„ฑ์„ ์œ„ํ•ด PostgreSQL, Redis๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. Redis๋Š” ์šฉ๋„๋ณ„๋กœ ๋ถ„๋ฆฌ(Blacklist/OAuth State/Streams/Pub-Sub/Cache)๋˜๋ฉฐ, Helm Chart๋กœ ๊ด€๋ฆฌ๋˜๋Š” ๋…๋ฆฝ์ ์ธ ๋ฐ์ดํ„ฐ ์ธํ”„๋ผ์ž…๋‹ˆ๋‹ค.
  • Platform Layer: Istiod๊ฐ€ Service Mesh๋ฅผ ์ œ์–ดํ•˜๊ณ , ArgoCD๊ฐ€ GitOps ๋™๊ธฐํ™”๋ฅผ ๋‹ด๋‹นํ•ฉ๋‹ˆ๋‹ค. KEDA๊ฐ€ ์ด๋ฒคํŠธ ๋“œ๋ฆฌ๋ธ ์˜คํ† ์Šค์ผ€์ผ๋ง์„ ์ˆ˜ํ–‰ํ•˜๊ณ , Observability ์Šคํƒ(Prometheus/Grafana/Kiali, Jaeger, EFK Stack)์ด ๋ฉ”ํŠธ๋ฆญยทํŠธ๋ ˆ์ด์‹ฑยท๋กœ๊น…์„ ํ†ตํ•ฉ ๊ด€๋ฆฌํ•ฉ๋‹ˆ๋‹ค.

๊ฐ ๊ณ„์ธต์€ ์„œ๋กœ ๋…๋ฆฝ์ ์œผ๋กœ ๊ธฐ๋Šฅํ•˜๋„๋ก ์„ค๊ณ„๋˜์—ˆ์œผ๋ฉฐ, Platform Layer๊ฐ€ ์ „ ๊ณ„์ธต์„ ์ œ์–ด ๋ฐ ๊ด€์ธกํ•ฉ๋‹ˆ๋‹ค. ํ”„๋กœ๋•์…˜ ํ™˜๊ฒฝ์„ ์ „์ œ๋กœ ํ•œ Self-manged Kubernetes ๊ธฐ๋ฐ˜ ํด๋Ÿฌ์Šคํ„ฐ๋กœ ์ปจํ…Œ์ด๋„ˆํ™”๋œ ์–ดํ”Œ๋ฆฌ์ผ€์ด์…˜์˜ ์˜ค์ผ€์ŠคํŠธ๋ ˆ์ด์…˜์„ ์ง€์›ํ•ฉ๋‹ˆ๋‹ค. Istio Service Mesh๋ฅผ ๋„์ž…ํ•˜์—ฌ mTLS ๋ณด์•ˆ ํ†ต์‹ , ํŠธ๋ž˜ํ”ฝ ์ œ์–ด(VirtualService), ์ธ์ฆ ์œ„์ž„(Auth Offloading)์„ ๊ตฌํ˜„ํ–ˆ์Šต๋‹ˆ๋‹ค. ํด๋Ÿฌ์Šคํ„ฐ์˜ ์•ˆ์ •์„ฑ๊ณผ ์„ฑ๋Šฅ์„ ๋ณด์žฅํ•˜๊ธฐ ์œ„ํ•ด ๋ชจ๋‹ˆํ„ฐ๋ง ์‹œ์Šคํ…œ์„ ๋„์ž…, IaC(Infrastructure as Code) ๋ฐ GitOps ํŒŒ์ดํ”„๋ผ์ธ์„ ๊ตฌ์ถ•ํ•ด ๋ชจ๋…ธ๋ ˆํฌ ๊ธฐ๋ฐ˜ ์ฝ”๋“œ๋ฒ ์ด์Šค๊ฐ€ SSOT(Single Source Of Truth)๋กœ ๊ธฐ๋Šฅํ•˜๋„๋ก ์ œ์ž‘๋˜์—ˆ์Šต๋‹ˆ๋‹ค.


Services Snapshot

์„œ๋น„์Šค ์„ค๋ช… ์ด๋ฏธ์ง€/ํƒœ๊ทธ
auth JWT ์ธ์ฆ/์ธ๊ฐ€ (RS256) docker.io/mng990/eco2:auth-{env}-latest
my ์‚ฌ์šฉ์ž ์ •๋ณด docker.io/mng990/eco2:my-{env}-latest
scan Lite RAG + GPT 5.1 Vision ํ๊ธฐ๋ฌผ ๋ถ„๋ฅ˜ docker.io/mng990/eco2:scan-{env}-latest
chat Lite RAG + GPT 5.1 ์ฑ—๋ด‡ docker.io/mng990/eco2:chat-{env}-latest
character ์บ๋ฆญํ„ฐ ์ œ๊ณต docker.io/mng990/eco2:character-{env}-latest
location ์ง€๋„/์ˆ˜๊ฑฐํ•จ ๊ฒ€์ƒ‰ docker.io/mng990/eco2:location-{env}-latest
images ์ด๋ฏธ์ง€ ์—…๋กœ๋“œ docker.io/mng990/eco2:image-{env}-latest

Celery Workers โœ…

Worker ๋…ธ๋“œ ์„ค๋ช… Queue Scaling
scan-worker worker-ai AI ํŒŒ์ดํ”„๋ผ์ธ ์ฒ˜๋ฆฌ (Visionโ†’Ruleโ†’Answerโ†’Reward) scan.vision, scan.rule, scan.answer, scan.reward KEDA (RabbitMQ)
character-match-worker worker-storage ์บ๋ฆญํ„ฐ ๋งค์นญ ์ฒ˜๋ฆฌ character.match KEDA (RabbitMQ)
character-worker worker-storage ์บ๋ฆญํ„ฐ ์†Œ์œ ๊ถŒ ์ €์žฅ (batch) character.reward KEDA (RabbitMQ)
my-worker worker-storage ๋งˆ์ดํŽ˜์ด์ง€ ์บ๋ฆญํ„ฐ ๋™๊ธฐํ™” (batch) my.reward KEDA (RabbitMQ)
users-worker worker-storage ์œ ์ € ์บ๋ฆญํ„ฐ ์†Œ์œ ๊ถŒ PostgreSQL UPSERT (Clean Arch) users.character KEDA (RabbitMQ)
celery-beat worker-storage DLQ ์žฌ์ฒ˜๋ฆฌ ์Šค์ผ€์ค„๋ง - ๋‹จ์ผ ์ธ์Šคํ„ด์Šค

Auth Workers (Clean Architecture) โœ…

Worker ๋…ธ๋“œ ์„ค๋ช… ์ž…๋ ฅ ์ถœ๋ ฅ
auth-worker worker-storage ๋ธ”๋ž™๋ฆฌ์ŠคํŠธ ์ด๋ฒคํŠธ โ†’ Redis ์ €์žฅ RabbitMQ blacklist.events Redis blacklist:{jti}
auth-relay worker-storage Redis Outbox โ†’ RabbitMQ ์žฌ๋ฐœํ–‰ (Outbox Pattern) Redis outbox:blacklist RabbitMQ blacklist.events

Event Relay Components โœ…

Component ์„ค๋ช… Scaling
event-router Redis Streams โ†’ Pub/Sub Fan-out, State KV ๊ด€๋ฆฌ KEDA (Streams Pending)
sse-gateway Pub/Sub ๊ตฌ๋… โ†’ SSE ํด๋ผ์ด์–ธํŠธ ์ „๋‹ฌ KEDA (์—ฐ๊ฒฐ ์ˆ˜)

๊ฐ ๋„๋ฉ”์ธ์€ ๊ณตํ†ต FastAPI ํ…œํ”Œ๋ฆฟยทDockerfileยทํ…Œ์ŠคํŠธ๋ฅผ ๊ณต์œ ํ•˜๊ณ , Kustomize overlay์—์„œ ์ด๋ฏธ์ง€ ํƒœ๊ทธ์™€ ConfigMap/Secret๋งŒ ๋ถ„๊ธฐํ•ฉ๋‹ˆ๋‹ค.


AI Domain Progress

ECA49AD6-EA0C-4957-8891-8C6FA12A2916

ํ•ญ๋ชฉ ์ง„ํ–‰ ๋‚ด์šฉ (2025-11 ๊ธฐ์ค€)
Vision ์ธ์‹ ํŒŒ์ดํ”„๋ผ์ธ domains/chat/app/core/ImageRecognition.py, vision.py์—์„œ Azure Vision โ†’ OpenAI GPT-5.2/gemini-3.0-flash-preview ์กฐํ•ฉ์œผ๋กœ ํ๊ธฐ๋ฌผ ์ด๋ฏธ์ง€๋ฅผ ๋ถ„๋ฅ˜. item_class_list.yaml, situation_tags.yaml์— ์นดํ…Œ๊ณ ๋ฆฌ/์ƒํ™ฉ ํƒœ๊ทธ ์ •์˜ ํ›„ Prompt์— ์ž๋™ ์‚ฝ์ž….
Text/Intent ๋ถ„๋ฅ˜ text_classifier.py, prompts/text_classification_prompt.txt ๊ธฐ๋ฐ˜์œผ๋กœ ์‚ฌ์šฉ์ž ์งˆ์˜๋ฅผ intent/priority๋กœ ์ž๋™ ๋ถ„๋ฅ˜ํ•˜์—ฌ ๋‹ต๋ณ€ ๋ผ์šฐํŒ….
RAG/์ง€์‹ ๋ฒ ์ด์Šค app/core/source/*.json์— ์Œ์‹๋ฌผ/์žฌํ™œ์šฉ ํ’ˆ๋ชฉ๋ณ„ ์ฒ˜๋ฆฌ ์ง€์นจ์„ ๋‹ค์ˆ˜์˜ JSON์œผ๋กœ ์ถ•์ ํ•˜๊ณ , rag.py๊ฐ€ ๊ฒ€์ƒ‰ยท์š”์•ฝํ•ด ๋‹ต๋ณ€์— ์ธ์šฉ.
๋‹ต๋ณ€ ์ƒ์„ฑ Prompt prompts/answer_generation_prompt.txt, vision_classification_prompt.txt๋ฅผ ํ†ตํ•ด ๋‹ค์ค‘ ์†Œ์Šค ๊ฒฐ๊ณผ๋ฅผ ํ•˜๋‚˜์˜ ์นœ์ ˆํ•œ ์‘๋‹ต์œผ๋กœ ๊ตฌ์„ฑ. multi-turn ์ปจํ…์ŠคํŠธ์™€ tone์„ prompt ๋ ˆ๋ฒจ์—์„œ ์ œ์–ด.
API ๊ตฌ์กฐ domains/chat/app โ†’ FastAPI + chat/app/core/* ์„œ๋น„์Šค ๊ณ„์ธต์œผ๋กœ ๋ถ„๋ฆฌ. /api/v1/chat ์—”๋“œํฌ์ธํŠธ๋Š” text/vision ์š”์ฒญ์„ ์ž๋™ ํŒ๋ณ„ํ•˜๊ณ  OpenAI ํ˜ธ์ถœ์„ ์ถ”์ƒํ™”.
ํ…Œ์ŠคํŠธ/์šด์˜ tests/test_app.py๋กœ API ๋ ˆ๋ฒจ smoke test, requirements.txt์— OpenAI/Azure SDK ๊ณ ์ •.

Event Relay Layer โœ…

Status: Redis Streams + Pub/Sub + State KV ๊ธฐ๋ฐ˜ Event Relay ์•„ํ‚คํ…์ฒ˜ ์™„๋ฃŒ

flowchart LR
    subgraph Worker["๐Ÿ”ง Celery Worker"]
        SW["scan-worker"]
    end

    subgraph Streams["๐Ÿ“Š Redis Streams"]
        RS[("scan:events:*<br/>(๋‚ด๊ตฌ์„ฑ)")]
    end

    subgraph Router["๐Ÿ”€ Event Router"]
        ER["Consumer Group<br/>XREADGROUP"]
    end

    subgraph State["๐Ÿ’พ State KV"]
        SK[("scan:state:*<br/>(๋ณต๊ตฌ/์กฐํšŒ)")]
    end

    subgraph PubSub["๐Ÿ“ก Redis Pub/Sub"]
        PS[("sse:events:*<br/>(์‹ค์‹œ๊ฐ„)")]
    end

    subgraph Gateway["๐ŸŒ SSE Gateway"]
        SG["Pub/Sub ๊ตฌ๋…<br/>State ๋ณต๊ตฌ<br/>Streams Catch-up"]
    end

    subgraph Client["๐Ÿ‘ค Client"]
        CL["Browser/App"]
    end

    SW -->|XADD| RS
    RS -->|XREADGROUP| ER
    ER -->|SETEX| SK
    ER -->|PUBLISH| PS
    SK -.->|GET ์žฌ์ ‘์†| SG
    PS -->|SUBSCRIBE| SG
    SG -->|SSE| CL

    classDef worker fill:#fff9c4,stroke:#f9a825,stroke-width:2px,color:#000
    classDef streams fill:#ffccbc,stroke:#e64a19,stroke-width:2px,color:#000
    classDef router fill:#b3e5fc,stroke:#0288d1,stroke-width:2px,color:#000
    classDef state fill:#d1c4e9,stroke:#512da8,stroke-width:2px,color:#000
    classDef pubsub fill:#c8e6c9,stroke:#388e3c,stroke-width:2px,color:#000
    classDef gateway fill:#b2dfdb,stroke:#00796b,stroke-width:2px,color:#000
    classDef client fill:#e1bee7,stroke:#7b1fa2,stroke-width:2px,color:#000

    class SW worker
    class RS streams
    class ER router
    class SK state
    class PS pubsub
    class SG gateway
    class CL client
Loading
์ปดํฌ๋„ŒํŠธ ์—ญํ•  ์Šค์ผ€์ผ๋ง
Event Router Streams โ†’ Pub/Sub Fan-out, State ๊ฐฑ์‹ , ๋ฉฑ๋“ฑ์„ฑ ๋ณด์žฅ KEDA (Pending ๋ฉ”์‹œ์ง€)
SSE Gateway Pub/Sub โ†’ Client, State ๋ณต๊ตฌ, Streams Catch-up KEDA (์—ฐ๊ฒฐ ์ˆ˜)
Redis Streams ์ด๋ฒคํŠธ ๋กœ๊ทธ (๋‚ด๊ตฌ์„ฑ), Consumer Group ์ง€์› ์ƒค๋”ฉ (4 shards)
Redis Pub/Sub ์‹ค์‹œ๊ฐ„ Fan-out (fire-and-forget) ์ „์šฉ ์ธ์Šคํ„ด์Šค
State KV ์ตœ์‹  ์ƒํƒœ ์Šค๋ƒ…์ƒท, ์žฌ์ ‘์† ๋ณต๊ตฌ Streams Redis ๊ณต์œ 

Async Task Pipeline (Celery) โœ…

Status: RabbitMQ + Celery + KEDA ์ด๋ฒคํŠธ ๋“œ๋ฆฌ๋ธ ์˜คํ† ์Šค์ผ€์ผ๋ง ์™„๋ฃŒ

flowchart LR
    subgraph Client["๐Ÿ‘ค Client"]
        CL["Browser/App"]
    end

    subgraph API["๐ŸŒ Scan API"]
        SA["POST /api/v1/scan<br/>Dispatch Chain"]
    end

    subgraph MQ["๐Ÿ“ฌ RabbitMQ"]
        VQ[("scan.vision")]
        RQ[("scan.rule")]
        AQ[("scan.answer")]
        WQ[("scan.reward")]
    end

    subgraph Workers["๐Ÿ”ง Celery Workers (gevent)"]
        VW["Vision Worker<br/>GPT Vision ๋ถ„์„"]
        RW["Rule Worker<br/>RAG ๊ทœ์ • ๊ฒ€์ƒ‰"]
        AW["Answer Worker<br/>GPT ๋‹ต๋ณ€ ์ƒ์„ฑ"]
        WW["Reward Worker<br/>๋ณด์ƒ ํŒ์ •"]
    end

    subgraph External["๐Ÿค– OpenAI API"]
        OAI["GPT-4o Vision<br/>GPT-4o-mini"]
    end

    subgraph Streams["๐Ÿ“Š Redis Streams"]
        RS[("scan:events:*<br/>(Event Relay๋กœ ์ „๋‹ฌ)")]
    end

    subgraph DB["๐Ÿ’พ PostgreSQL"]
        PG[("๊ฒฐ๊ณผ ์ €์žฅ")]
    end

    subgraph Scale["โšก KEDA"]
        KD["ํ ๊ธธ์ด ๊ธฐ๋ฐ˜<br/>์˜คํ† ์Šค์ผ€์ผ๋ง"]
    end

    CL -->|POST| SA
    SA -->|Dispatch| VQ
    SA -.->|202 Accepted| CL

    VQ --> VW
    VW -->|API Call| OAI
    VW -->|XADD| RS
    VW -->|Chain| RQ

    RQ --> RW
    RW -->|XADD| RS
    RW -->|Chain| AQ

    AQ --> AW
    AW -->|API Call| OAI
    AW -->|XADD| RS
    AW -->|Chain| WQ

    WQ --> WW
    WW -->|Batch Insert| PG
    WW -->|XADD stage=done| RS

    KD -.->|Monitor| MQ
    KD -.->|Scale| Workers

    classDef client fill:#e1bee7,stroke:#7b1fa2,stroke-width:2px,color:#000
    classDef api fill:#b2dfdb,stroke:#00796b,stroke-width:2px,color:#000
    classDef mq fill:#bbdefb,stroke:#1976d2,stroke-width:2px,color:#000
    classDef worker fill:#fff9c4,stroke:#f9a825,stroke-width:2px,color:#000
    classDef external fill:#ffcc80,stroke:#e65100,stroke-width:2px,color:#000
    classDef streams fill:#ffccbc,stroke:#e64a19,stroke-width:2px,color:#000
    classDef db fill:#c8e6c9,stroke:#388e3c,stroke-width:2px,color:#000
    classDef scale fill:#b3e5fc,stroke:#0288d1,stroke-width:2px,color:#000

    class CL client
    class SA api
    class VQ,RQ,AQ,WQ mq
    class VW,RW,AW,WW worker
    class OAI external
    class RS streams
    class PG db
    class KD scale
Loading
๐Ÿ“‹ Sequence Diagram (์ƒ์„ธ ํ๋ฆ„)
sequenceDiagram
    participant Client
    participant ScanAPI as Scan API
    participant RabbitMQ
    participant KEDA
    participant VisionWorker as Vision Worker
    participant RuleWorker as Rule Worker
    participant AnswerWorker as Answer Worker
    participant RewardWorker as Reward Worker
    participant RedisStreams as Redis Streams
    participant PostgreSQL

    Client->>ScanAPI: POST /api/v1/scan
    ScanAPI->>RabbitMQ: Dispatch Chain (job_id)
    ScanAPI-->>Client: 202 Accepted {job_id}

    KEDA->>RabbitMQ: ํ ๊ธธ์ด ๋ชจ๋‹ˆํ„ฐ๋ง
    KEDA->>VisionWorker: Scale Up (๋ฉ”์‹œ์ง€ ์ฆ๊ฐ€ ์‹œ)

    RabbitMQ->>VisionWorker: scan.vision queue
    VisionWorker->>VisionWorker: GPT Vision ๋ถ„์„
    VisionWorker->>RedisStreams: XADD stage=vision
    VisionWorker->>RabbitMQ: Chain โ†’ scan.rule

    RabbitMQ->>RuleWorker: scan.rule queue
    RuleWorker->>RuleWorker: RAG ๊ทœ์ • ๊ฒ€์ƒ‰
    RuleWorker->>RedisStreams: XADD stage=rule

    RuleWorker->>RabbitMQ: Chain โ†’ scan.answer
    RabbitMQ->>AnswerWorker: scan.answer queue
    AnswerWorker->>AnswerWorker: GPT ๋‹ต๋ณ€ ์ƒ์„ฑ
    AnswerWorker->>RedisStreams: XADD stage=answer

    AnswerWorker->>RabbitMQ: Chain โ†’ reward.character
    RabbitMQ->>RewardWorker: reward.character queue
    RewardWorker->>PostgreSQL (Batch): ๋ณด์ƒ ์ €์žฅ
    RewardWorker->>RedisStreams: XADD stage=done
Loading
์ปดํฌ๋„ŒํŠธ ์—ญํ•  Queue ์Šค์ผ€์ผ๋ง
scan-worker Vision ๋ถ„์„, RAG ๊ฒ€์ƒ‰, ๋‹ต๋ณ€ ์ƒ์„ฑ, ๋ณด์ƒ ํŒ์ • scan.vision, scan.rule, scan.answer, scan.reward KEDA (ํ ๊ธธ์ด)
character-match-worker ์บ๋ฆญํ„ฐ ๋งค์นญ ์ฒ˜๋ฆฌ character.match KEDA (ํ ๊ธธ์ด)
character-worker ์บ๋ฆญํ„ฐ ์†Œ์œ ๊ถŒ ์ €์žฅ (batch) character.reward KEDA (ํ ๊ธธ์ด)
my-worker ๋งˆ์ดํŽ˜์ด์ง€ ์บ๋ฆญํ„ฐ ๋™๊ธฐํ™” (batch) my.reward KEDA (ํ ๊ธธ์ด)
celery-beat DLQ ์žฌ์ฒ˜๋ฆฌ ์Šค์ผ€์ค„๋ง (5๋ถ„ ์ฃผ๊ธฐ) - ๋‹จ์ผ ์ธ์Šคํ„ด์Šค
RabbitMQ AMQP ๋ฉ”์‹œ์ง€ ๋ธŒ๋กœ์ปค vhost: eco2 Quorum Queue

Logging Pipeline (EFK Stack)

flowchart LR
    subgraph Pods["Kubernetes Pods"]
        API["API Pods<br/>(auth, scan, chat...)"]
        Workers["Celery Workers<br/>(scan, character-match, character, my)"]
        Infra["Infra Pods<br/>(istio, argocd...)"]
    end

    subgraph FluentBit["Fluent Bit (DaemonSet)"]
        Tail["Tail Input<br/>(/var/log/containers/*.log)"]
        Parser["Parser<br/>(JSON, regex)"]
        Filter["Filter<br/>(kubernetes metadata)"]
        Output["Output<br/>(es plugin)"]
    end

    subgraph EFK["EFK Stack"]
        ES[("Elasticsearch<br/>(3 nodes)")]
        Kibana["Kibana<br/>(UI)"]
    end

    API -->|stdout/stderr| Tail
    Workers -->|stdout/stderr| Tail
    Infra -->|stdout/stderr| Tail

    Tail --> Parser
    Parser --> Filter
    Filter --> Output
    Output -->|HTTP/9200| ES

    ES --> Kibana

    classDef pods fill:#326CE5,stroke:#fff,color:white
    classDef fluent fill:#009688,stroke:#fff,color:white
    classDef efk fill:#FF9800,stroke:#fff,color:white

    class API,Workers,Infra pods
    class Tail,Parser,Filter,Output fluent
    class ES,Kibana efk
Loading
์ปดํฌ๋„ŒํŠธ ์—ญํ•  ์„ค์ •
Fluent Bit ๋กœ๊ทธ ์ˆ˜์ง‘ ๋ฐ ํฌ์›Œ๋”ฉ (DaemonSet) /var/log/containers/*.log ์ˆ˜์ง‘, JSON ํŒŒ์‹ฑ
Elasticsearch ๋กœ๊ทธ ์ €์žฅ ๋ฐ ์ธ๋ฑ์‹ฑ 3-node cluster, ์ธ๋ฑ์Šค: logstash-YYYY.MM.DD
Kibana ๋กœ๊ทธ ๊ฒ€์ƒ‰ ๋ฐ ์‹œ๊ฐํ™” Discover, Dashboard, Alerting

๋กœ๊ทธ ํฌ๋งท (JSON ๊ตฌ์กฐํ™”)

{
  "timestamp": "2025-12-22T10:30:00.000Z",
  "level": "INFO",
  "logger": "scan.vision_task",
  "message": "Vision analysis completed",
  "task_id": "550e8400-e29b-41d4-a716-446655440000",
  "user_id": "123e4567-e89b-12d3-a456-426614174000",
  "duration_ms": 2340,
  "kubernetes": {
    "namespace": "scan",
    "pod_name": "scan-worker-5d8f9b7c4-x2k9p",
    "container_name": "scan-worker"
  }
}

Bootstrap Overview

Cluster   : kubeadm Self-Managed (21 Nodes)
GitOps    :
  Layer0 - Terraform (AWS ์ธํ”„๋ผ)
  Layer1 - Ansible (kubeadm, CNI)
  Layer2 - ArgoCD App-of-Apps Sync-wave + Kustomize/Helm
  Layer3 - GitHub Actions + Docker Hub
Architecture :
  Edge Layer        - Route 53, AWS ALB, Istio Ingress Gateway
  Service Layer     - auth, users, my, scan, character, location, chat
  Integration Layer :
    - Event Relay   - Redis Streams + Pub/Sub + State KV, Event Router, SSE Gateway
    - Worker (Storage) - auth-worker, auth-relay, users-worker, character-worker, my-worker
    - Worker (AI)   - scan-worker (Visionโ†’Ruleโ†’Answerโ†’Reward)
    - KEDA (Event-driven Autoscaling)
  Persistence Layer - PostgreSQL, Redis (Blacklist/State/Streams/Pub-Sub/Cache ๋ถ„๋ฆฌ)
  Platform Layer    - ArgoCD, Istiod, KEDA, Observability (Prometheus, Grafana, EFK, Jaeger)
Network   : Calico CNI + Istio Service Mesh (mTLS)
Node Isolation :
  - worker-storage  - Taint: domain=worker-storage:NoSchedule (Persistence ์ ‘๊ทผ Worker ์ „์šฉ)
  - worker-ai       - Taint: domain=worker-ai:NoSchedule (AI/OpenAI API ํ˜ธ์ถœ Worker ์ „์šฉ)
  1. Terraform์œผ๋กœ AWS ์ธํ”„๋ผ๋ฅผ ๊ตฌ์ถ•ํ•ฉ๋‹ˆ๋‹ค.
  2. Ansible๋กœ ๊ตฌ์ถ•๋œ AWS ์ธํ”„๋ผ๋ฅผ ์—ฎ์–ด K8s ํด๋Ÿฌ์Šคํ„ฐ๋ฅผ ๊ตฌ์„ฑํ•˜๊ณ , ArgoCD root-app์„ ์„ค์น˜ํ•ฉ๋‹ˆ๋‹ค.
  3. ๋ชจ๋“  ์ปดํฌ๋„ŒํŠธ๋Š” ArgoCD root-app๊ณผ sync๋œ ์ƒํƒœ์ด๋ฉฐ, root-app์€ develop ๋ธŒ๋žœ์น˜๋ฅผ ๋ฐ”๋ผ๋ด…๋‹ˆ๋‹ค.
  4. develop ๋ธŒ๋žœ์น˜์— push๊ฐ€ ๋ฐœ์ƒํ•˜๋ฉด CI ํŒŒ์ดํ”„๋ผ์ธ์„ ๊ฑฐ์ณ ํ…Œ์ŠคํŠธ, ๋„์ปค ์ด๋ฏธ์ง€ ํŒจํ‚ค์ง•, ํ—ˆ๋ธŒ ์—…๋กœ๋“œ๊นŒ์ง€ ์ˆ˜ํ–‰ํ•ฉ๋‹ˆ๋‹ค.
  5. ArgoCD root-app์€ develop ๋ธŒ๋žœ์น˜์˜ ๋ณ€๊ฒฝ์‚ฌํ•ญ์ด ๊ฐ์ง€๋˜๋ฉด ํ•ด๋‹น ํŒŒํŠธ๋ฅผ ์—…๋ฐ์ดํŠธํ•ด ์ฝ”๋“œ ๋ณ€๊ฒฝ์ด ํด๋Ÿฌ์Šคํ„ฐ๋กœ ๋ฐ˜์˜๋ฉ๋‹ˆ๋‹ค.

GitOps Architecture

9093CE45-C239-4549-B1FA-10D2800BAD58_1_105_c

ArgoCD App-of-Apps ํŒจํ„ด ๊ธฐ๋ฐ˜ GitOps. ๋ชจ๋“  ๋ฆฌ์†Œ์Šค๋Š” sync-wave๋กœ ์˜์กด์„ฑ ์ˆœ์„œ ๋ณด์žฅ.

Wave ๋ ˆ์ด์–ด ๋ฆฌ์†Œ์Šค
0-10 ํ”Œ๋žซํผ CRD, Namespace, RBAC, Istio, NetworkPolicy, Secrets
15-32 ์ธํ”„๋ผ ALB, Monitoring, PostgreSQL, Redis, RabbitMQ
35-50 ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜ KEDA, APIs, Workers, Event Router, Routing
  • App-of-Apps: ๋ฃจํŠธ ์•ฑ โ†’ ApplicationSet ์ƒ์„ฑ โ†’ sync-wave ๊ฐ’์œผ๋กœ ๋ฐฐํฌ ์ˆœ์„œ ๊ฐ•์ œ
  • Sync Hook: PostSync Job์œผ๋กœ DB ๋งˆ์ด๊ทธ๋ ˆ์ด์…˜ ์ž๋™ ์‹คํ–‰
  • CI/CD: ์ฝ”๋“œ ๋ณ€๊ฒฝ โ†’ GitHub Actions โ†’ Docker Hub โ†’ ArgoCD Auto-Sync

Release Summary (v1.0.7)

  • Event Relay Layer + AI ํŒŒ์ดํ”„๋ผ์ธ โœ…
    • Redis Streams(๋‚ด๊ตฌ์„ฑ) + Pub/Sub(์‹ค์‹œ๊ฐ„) + State KV(๋ณต๊ตฌ) 3-tier ์ด๋ฒคํŠธ ์•„ํ‚คํ…์ฒ˜ ๊ตฌํ˜„
    • Event Router: Consumer Group(XREADGROUP)์œผ๋กœ Streams ์†Œ๋น„, Pub/Sub Fan-out, ๋ฉฑ๋“ฑ์„ฑ ๋ณด์žฅ
    • SSE Gateway: Pub/Sub ๊ตฌ๋… ๊ธฐ๋ฐ˜ ์‹ค์‹œ๊ฐ„ ์ „๋‹ฌ, State ๋ณต๊ตฌ, Streams Catch-up
    • Celery Chain(Visionโ†’Ruleโ†’Answerโ†’Reward): GPT 5.1 Vision + GPT 5.1-mini ์กฐํ•ฉ
    • gevent pool (100 greenlets) + httpx connection pooling, ๋‹จ์ผ ์š”์ฒญ โ‰ˆ 12์ดˆ
    • ๋ถ€ํ•˜ ํ…Œ์ŠคํŠธ ๊ฒฐ๊ณผ (๋‹จ์ผ ๋…ธ๋“œ ๊ธฐ์ค€, ์ด์ „ Celery Events ๋Œ€๋น„ 2.8๋ฐฐ ํ–ฅ์ƒ)
VU ์š”์ฒญ ์ˆ˜ ์™„๋ฃŒ์œจ Throughput E2E p95 Scan p95 ์ƒํƒœ
50 685 99.7% 198 req/m 17.7์ดˆ 93ms โœ… ์—ฌ์œ 
200 1,649 99.8% 367 req/m 33.2์ดˆ 83ms โœ… ์•ˆ์ •
250 1,754 99.9% 418 req/m 40.5์ดˆ 78ms โญ SLA ๊ธฐ์ค€
300 1,732 99.9% 402 req/m 48.5์ดˆ 83ms โš ๏ธ ํฌํ™” ์‹œ์ž‘
400 1,901 98.9% 422 req/m 62.2์ดˆ 207ms โš ๏ธ ํ•œ๊ณ„ ๊ทผ์ ‘
500 1,990 94.0% 438 req/m 76.4์ดˆ 154ms โŒ ๋‹จ์ผ ๋…ธ๋“œ ํ•œ๊ณ„
  • KEDA ์ด๋ฒคํŠธ ๋“œ๋ฆฌ๋ธ ์˜คํ† ์Šค์ผ€์ผ๋ง โœ…

    • scan-worker: RabbitMQ ํ ๊ธธ์ด ๊ธฐ๋ฐ˜ ์ž๋™ ์Šค์ผ€์ผ๋ง (1-3 replicas)
    • event-router: Redis Streams pending ๋ฉ”์‹œ์ง€ ๊ธฐ๋ฐ˜ ์Šค์ผ€์ผ๋ง
    • character-match-worker: RabbitMQ character.match ํ ๊ธฐ๋ฐ˜ ์Šค์ผ€์ผ๋ง
    • Prometheus Adapter ์—ฐ๋™์œผ๋กœ ์ปค์Šคํ…€ ๋ฉ”ํŠธ๋ฆญ ๊ธฐ๋ฐ˜ HPA ๊ตฌํ˜„
  • ๋ถ€ํ•˜ ํ…Œ์ŠคํŠธ ๋ฐ ์Šค์ผ€์ผ๋ง ๊ฒ€์ฆ โœ…

    • 21-Node ํด๋Ÿฌ์Šคํ„ฐ: Event Router, Redis Pub/Sub ์ „์šฉ ๋…ธ๋“œ ์ถ”๊ฐ€
    • Redis ์ธ์Šคํ„ด์Šค ๋ถ„๋ฆฌ: Streams(๋‚ด๊ตฌ์„ฑ) / Pub/Sub(์‹ค์‹œ๊ฐ„) / Cache(LRU)
    • ๋ถ€ํ•˜ ํ…Œ์ŠคํŠธ ๊ฒ€์ฆ: 50/200/250/300/400/500 VU ํ…Œ์ŠคํŠธ ์™„๋ฃŒ
      • ๋‹จ์ผ ๋…ธ๋“œ(k8s-worker-ai, 2 cores) ๊ธฐ์ค€ 250 VU SLA, 500 VU ํ•œ๊ณ„์  ๋„์ถœ
      • KEDA ์ž๋™ ์Šค์ผ€์ผ๋ง ๊ฒ€์ฆ: scan-worker 1โ†’3 pods, scan-api 1โ†’3 pods

Article

๐Ÿ“ ์ด์ฝ”์—์ฝ”(Ecoยฒ) ๋ฐฑ์—”๋“œ/์ธํ”„๋ผ ๊ฐœ๋ฐœ ๋ธ”๋กœ๊ทธ


Status

v1.0.7 - Event Relay & KEDA

  • โœ… Redis Streams + Pub/Sub + State KV ๊ธฐ๋ฐ˜ Event Relay Layer ์™„๋ฃŒ
  • โœ… Event Router, SSE Gateway ์ปดํฌ๋„ŒํŠธ ๊ฐœ๋ฐœ ์™„๋ฃŒ
  • โœ… KEDA ์ด๋ฒคํŠธ ๋“œ๋ฆฌ๋ธ ์˜คํ† ์Šค์ผ€์ผ๋ง ์ ์šฉ (scan-worker, event-router, character-match-worker)
  • โœ… Celery ๋น„๋™๊ธฐ AI ํŒŒ์ดํ”„๋ผ์ธ ์™„๋ฃŒ (Visionโ†’Ruleโ†’Answerโ†’Reward)
  • โœ… ๋ถ€ํ•˜ ํ…Œ์ŠคํŠธ ์™„๋ฃŒ: 50/200/250/300/400/500 VU ๊ฒ€์ฆ
    • 250 VU (SLA): 99.9% ์™„๋ฃŒ์œจ, 418 req/m, E2E p95 40์ดˆ
    • 500 VU: ๋‹จ์ผ ๋…ธ๋“œ ํ•œ๊ณ„์  (94% ์™„๋ฃŒ์œจ, E2E p95 76์ดˆ)

v1.0.6 - Observability

  • โœ… EFK ๋กœ๊น… ํŒŒ์ดํ”„๋ผ์ธ (Fluent Bit โ†’ Elasticsearch โ†’ Kibana)
  • โœ… ๋ถ„์‚ฐ ํŠธ๋ ˆ์ด์‹ฑ (Jaeger + OpenTelemetry + Kiali)
  • โœ… Alertmanager ์•Œ๋ฆผ ์‹œ์Šคํ…œ (Slack)

v1.0.5 - Service Mesh & Auth Offloading

  • โœ… Istio Service Mesh Migration ์™„๋ฃŒ
  • โœ… gRPC ๋‚ด๋ถ€ ํ†ต์‹  Migration ์™„๋ฃŒ
  • โœ… Auth-Offloading ์™„๋ฃŒ, ๋„๋ฉ”์ธ๋ณ„ ๋…๋ฆฝ์„ฑ ํ™•๋ณด
  • โœ… ext-authz ์„ฑ๋Šฅ ํŠœ๋‹ (Grafana: VU 2500, RPS 1200, p99 200-300ms)

v1.0.0 - Initial Release

  • โœ… Terraform ยท Ansible bootstrap ยท ArgoCD Sync-wave
  • โœ… GitOps Sync-Wave ์žฌ์ •๋ ฌ (00~70) + upstream Helm/CRD ๋ถ„๋ฆฌ
  • โœ… Docker Hub ์ด๋ฏธ์ง€ ํŒŒ์ดํ”„๋ผ์ธ + External Secrets ์šด์˜
  • โœ… API ๊ฐœ๋ฐœ ์™„๋ฃŒ, ํ”„๋ก ํŠธ-๋ฐฑ-AI ์—ฐ๋™ ์™„๋ฃŒ