Version: v1.0.7 | Changelog
- GPT Vision + Rule-based-retrieval ๊ธฐ๋ฐ AI ์ด์์คํดํธ๋ก, ํ๊ธฐ๋ฌผ ์ด๋ฏธ์ง ๋ถ๋ฅยท๋ถ๋ฆฌ๋ฐฐ์ถ ์๋ดยท์ฑ๋ด ๊ธฐ๋ฅ์ ์ ๊ณตํฉ๋๋ค.
- Self-managed Kubernetes 21-Nodes ํด๋ฌ์คํฐ์์ Istio Service Mesh(mTLS, Auth Offloading)์ ArgoCD GitOps๋ก ์ด์ํฉ๋๋ค.
- Redis Streams + Pub/Sub + State KV ๊ธฐ๋ฐ Event Relay Layer๋ก ์ค์๊ฐ SSE ์ด๋ฒคํธ๋ฅผ ์ฒ๋ฆฌํ๊ณ , KEDA๋ก ์ด๋ฒคํธ ๋๋ฆฌ๋ธ ์คํ ์ค์ผ์ผ๋ง์ ์ํํฉ๋๋ค.
- RabbitMQ + Celery ๋น๋๊ธฐ Task Queue๋ก AI ํ์ดํ๋ผ์ธ์ ์ฒ๋ฆฌํ๊ณ , EFK + Jaeger๋ก ๋ก๊น ยทํธ๋ ์ด์ฑ์ ์์งํฉ๋๋ค.
- 7๊ฐ ๋๋ฉ์ธ ๋ง์ดํฌ๋ก์๋น์ค(auth, my, scan, chat, character, location, image)๋ฅผ ๋ชจ๋ ธ๋ ํฌ๋ก ๊ด๋ฆฌํฉ๋๋ค.
- ์ ์ ๋ฐฐํฌ ์ค: https://frontend.dev.growbin.app
Edge Layer : Route 53, AWS ALB, Istio Ingress Gateway
Service Layer : auth, users, my, scan, character, location, chat (w/ Envoy Sidecar)
Integration Layer :
- Event Relay : Redis Streams + Pub/Sub + State KV, Event Router, SSE Gateway
- Worker (Storage) : auth-worker, auth-relay, users-worker, character-worker, my-worker, character-match-worker
- Worker (AI) : scan-worker (VisionโRuleโAnswerโReward)
Persistence Layer : PostgreSQL, Redis (Blacklist/State/Streams/Pub-Sub/Cache)
Platform Layer : ArgoCD, Istiod, KEDA, Prometheus, Grafana, Kiali, Jaeger, EFK Stack๋ณธ ์๋น์ค๋ 5-Layer Architecture๋ก ๊ตฌ์ฑ๋์์ต๋๋ค.
- Edge Layer: AWS ALB๊ฐ SSL Termination์ ์ฒ๋ฆฌํ๊ณ , ํธ๋ํฝ์
Istio Ingress Gateway๋ก ์ ๋ฌํฉ๋๋ค. Gateway๋VirtualService๊ท์น์ ๋ฐ๋ผ North-South ํธ๋ํฝ์ ๋ผ์ฐํ ํฉ๋๋ค. - Service Layer: ๋ชจ๋ ๋ง์ดํฌ๋ก์๋น์ค๋ Istio Service Mesh ๋ด์์ ๋์ํ๋ฉฐ,
Envoy Sidecar๋ฅผ ํตํด mTLS ํต์ , ํธ๋ํฝ ์ ์ด, ๋ฉํธ๋ฆญ ์์ง์ ์ํํฉ๋๋ค.authโusersgRPC ํต์ ์ผ๋ก ๋๋ฉ์ธ ๊ฐ ๋๊ธฐ ํธ์ถ์ ์ฒ๋ฆฌํฉ๋๋ค. - Integration Layer - Event Relay: Redis Streams(๋ด๊ตฌ์ฑ) + Pub/Sub(์ค์๊ฐ) + State KV(๋ณต๊ตฌ) 3-tier ์ด๋ฒคํธ ์ํคํ ์ฒ๋ก SSE ํ์ดํ๋ผ์ธ์ ์ฒ๋ฆฌํฉ๋๋ค. RabbitMQ + Celery ๋น๋๊ธฐ Task Queue๋ก AI ํ์ดํ๋ผ์ธ(VisionโRuleโAnswerโReward)์ ์ฒ๋ฆฌํ๊ณ , KEDA๊ฐ ์ด๋ฒคํธ ๋๋ฆฌ๋ธ ์คํ ์ค์ผ์ผ๋ง์ ์ํํฉ๋๋ค.
- Integration Layer - Worker: Storage Worker(
worker-storage๋ ธ๋)๋ Persistence Layer์ ์ ๊ทผํ์ฌ ๋ฐ์ดํฐ๋ฅผ ๋๊ธฐํํฉ๋๋ค.auth-worker๋ RabbitMQ์์ ๋ธ๋๋ฆฌ์คํธ ์ด๋ฒคํธ๋ฅผ ์๋นํด Redis์ ์ ์ฅํ๊ณ ,auth-relay๋ Redis Outbox ํจํด์ผ๋ก ์คํจ ์ด๋ฒคํธ๋ฅผ ์ฌ๋ฐํํฉ๋๋ค.users-worker๋ Celery Batch๋ก ์บ๋ฆญํฐ ์์ ๊ถ์ PostgreSQL์ UPSERTํฉ๋๋ค. AI Worker(worker-ai๋ ธ๋)๋ OpenAI API์ ํต์ ํ๋ฉฐ,scan-worker๊ฐ VisionโRuleโAnswerโReward ์ฒด์ธ์ gevent pool๋ก ์ฒ๋ฆฌํฉ๋๋ค. - Persistence Layer: ์๋น์ค๋ ์์์ฑ์ ์ํด PostgreSQL, Redis๋ฅผ ์ฌ์ฉํฉ๋๋ค. Redis๋ ์ฉ๋๋ณ๋ก ๋ถ๋ฆฌ(Blacklist/OAuth State/Streams/Pub-Sub/Cache)๋๋ฉฐ, Helm Chart๋ก ๊ด๋ฆฌ๋๋ ๋ ๋ฆฝ์ ์ธ ๋ฐ์ดํฐ ์ธํ๋ผ์ ๋๋ค.
- Platform Layer:
Istiod๊ฐ Service Mesh๋ฅผ ์ ์ดํ๊ณ ,ArgoCD๊ฐ GitOps ๋๊ธฐํ๋ฅผ ๋ด๋นํฉ๋๋ค.KEDA๊ฐ ์ด๋ฒคํธ ๋๋ฆฌ๋ธ ์คํ ์ค์ผ์ผ๋ง์ ์ํํ๊ณ , Observability ์คํ(Prometheus/Grafana/Kiali,Jaeger,EFK Stack)์ด ๋ฉํธ๋ฆญยทํธ๋ ์ด์ฑยท๋ก๊น ์ ํตํฉ ๊ด๋ฆฌํฉ๋๋ค.
๊ฐ ๊ณ์ธต์ ์๋ก ๋ ๋ฆฝ์ ์ผ๋ก ๊ธฐ๋ฅํ๋๋ก ์ค๊ณ๋์์ผ๋ฉฐ, Platform Layer๊ฐ ์ ๊ณ์ธต์ ์ ์ด ๋ฐ ๊ด์ธกํฉ๋๋ค. ํ๋ก๋์ ํ๊ฒฝ์ ์ ์ ๋ก ํ Self-manged Kubernetes ๊ธฐ๋ฐ ํด๋ฌ์คํฐ๋ก ์ปจํ ์ด๋ํ๋ ์ดํ๋ฆฌ์ผ์ด์ ์ ์ค์ผ์คํธ๋ ์ด์ ์ ์ง์ํฉ๋๋ค. Istio Service Mesh๋ฅผ ๋์ ํ์ฌ mTLS ๋ณด์ ํต์ , ํธ๋ํฝ ์ ์ด(VirtualService), ์ธ์ฆ ์์(Auth Offloading)์ ๊ตฌํํ์ต๋๋ค. ํด๋ฌ์คํฐ์ ์์ ์ฑ๊ณผ ์ฑ๋ฅ์ ๋ณด์ฅํ๊ธฐ ์ํด ๋ชจ๋ํฐ๋ง ์์คํ ์ ๋์ , IaC(Infrastructure as Code) ๋ฐ GitOps ํ์ดํ๋ผ์ธ์ ๊ตฌ์ถํด ๋ชจ๋ ธ๋ ํฌ ๊ธฐ๋ฐ ์ฝ๋๋ฒ ์ด์ค๊ฐ SSOT(Single Source Of Truth)๋ก ๊ธฐ๋ฅํ๋๋ก ์ ์๋์์ต๋๋ค.
| ์๋น์ค | ์ค๋ช | ์ด๋ฏธ์ง/ํ๊ทธ |
|---|---|---|
| auth | JWT ์ธ์ฆ/์ธ๊ฐ (RS256) | docker.io/mng990/eco2:auth-{env}-latest |
| my | ์ฌ์ฉ์ ์ ๋ณด | docker.io/mng990/eco2:my-{env}-latest |
| scan | Lite RAG + GPT 5.1 Vision ํ๊ธฐ๋ฌผ ๋ถ๋ฅ | docker.io/mng990/eco2:scan-{env}-latest |
| chat | Lite RAG + GPT 5.1 ์ฑ๋ด | docker.io/mng990/eco2:chat-{env}-latest |
| character | ์บ๋ฆญํฐ ์ ๊ณต | docker.io/mng990/eco2:character-{env}-latest |
| location | ์ง๋/์๊ฑฐํจ ๊ฒ์ | docker.io/mng990/eco2:location-{env}-latest |
| images | ์ด๋ฏธ์ง ์ ๋ก๋ | docker.io/mng990/eco2:image-{env}-latest |
| Worker | ๋ ธ๋ | ์ค๋ช | Queue | Scaling |
|---|---|---|---|---|
| scan-worker | worker-ai |
AI ํ์ดํ๋ผ์ธ ์ฒ๋ฆฌ (VisionโRuleโAnswerโReward) | scan.vision, scan.rule, scan.answer, scan.reward |
KEDA (RabbitMQ) |
| character-match-worker | worker-storage |
์บ๋ฆญํฐ ๋งค์นญ ์ฒ๋ฆฌ | character.match |
KEDA (RabbitMQ) |
| character-worker | worker-storage |
์บ๋ฆญํฐ ์์ ๊ถ ์ ์ฅ (batch) | character.reward |
KEDA (RabbitMQ) |
| my-worker | worker-storage |
๋ง์ดํ์ด์ง ์บ๋ฆญํฐ ๋๊ธฐํ (batch) | my.reward |
KEDA (RabbitMQ) |
| users-worker | worker-storage |
์ ์ ์บ๋ฆญํฐ ์์ ๊ถ PostgreSQL UPSERT (Clean Arch) | users.character |
KEDA (RabbitMQ) |
| celery-beat | worker-storage |
DLQ ์ฌ์ฒ๋ฆฌ ์ค์ผ์ค๋ง | - | ๋จ์ผ ์ธ์คํด์ค |
| Worker | ๋ ธ๋ | ์ค๋ช | ์ ๋ ฅ | ์ถ๋ ฅ |
|---|---|---|---|---|
| auth-worker | worker-storage |
๋ธ๋๋ฆฌ์คํธ ์ด๋ฒคํธ โ Redis ์ ์ฅ | RabbitMQ blacklist.events |
Redis blacklist:{jti} |
| auth-relay | worker-storage |
Redis Outbox โ RabbitMQ ์ฌ๋ฐํ (Outbox Pattern) | Redis outbox:blacklist |
RabbitMQ blacklist.events |
| Component | ์ค๋ช | Scaling |
|---|---|---|
| event-router | Redis Streams โ Pub/Sub Fan-out, State KV ๊ด๋ฆฌ | KEDA (Streams Pending) |
| sse-gateway | Pub/Sub ๊ตฌ๋ โ SSE ํด๋ผ์ด์ธํธ ์ ๋ฌ | KEDA (์ฐ๊ฒฐ ์) |
๊ฐ ๋๋ฉ์ธ์ ๊ณตํต FastAPI ํ ํ๋ฆฟยทDockerfileยทํ ์คํธ๋ฅผ ๊ณต์ ํ๊ณ , Kustomize overlay์์ ์ด๋ฏธ์ง ํ๊ทธ์ ConfigMap/Secret๋ง ๋ถ๊ธฐํฉ๋๋ค.
| ํญ๋ชฉ | ์งํ ๋ด์ฉ (2025-11 ๊ธฐ์ค) |
|---|---|
| Vision ์ธ์ ํ์ดํ๋ผ์ธ | domains/chat/app/core/ImageRecognition.py, vision.py์์ Azure Vision โ OpenAI GPT-5.2/gemini-3.0-flash-preview ์กฐํฉ์ผ๋ก ํ๊ธฐ๋ฌผ ์ด๋ฏธ์ง๋ฅผ ๋ถ๋ฅ. item_class_list.yaml, situation_tags.yaml์ ์นดํ
๊ณ ๋ฆฌ/์ํฉ ํ๊ทธ ์ ์ ํ Prompt์ ์๋ ์ฝ์
. |
| Text/Intent ๋ถ๋ฅ | text_classifier.py, prompts/text_classification_prompt.txt ๊ธฐ๋ฐ์ผ๋ก ์ฌ์ฉ์ ์ง์๋ฅผ intent/priority๋ก ์๋ ๋ถ๋ฅํ์ฌ ๋ต๋ณ ๋ผ์ฐํ
. |
| RAG/์ง์ ๋ฒ ์ด์ค | app/core/source/*.json์ ์์๋ฌผ/์ฌํ์ฉ ํ๋ชฉ๋ณ ์ฒ๋ฆฌ ์ง์นจ์ ๋ค์์ JSON์ผ๋ก ์ถ์ ํ๊ณ , rag.py๊ฐ ๊ฒ์ยท์์ฝํด ๋ต๋ณ์ ์ธ์ฉ. |
| ๋ต๋ณ ์์ฑ Prompt | prompts/answer_generation_prompt.txt, vision_classification_prompt.txt๋ฅผ ํตํด ๋ค์ค ์์ค ๊ฒฐ๊ณผ๋ฅผ ํ๋์ ์น์ ํ ์๋ต์ผ๋ก ๊ตฌ์ฑ. multi-turn ์ปจํ
์คํธ์ tone์ prompt ๋ ๋ฒจ์์ ์ ์ด. |
| API ๊ตฌ์กฐ | domains/chat/app โ FastAPI + chat/app/core/* ์๋น์ค ๊ณ์ธต์ผ๋ก ๋ถ๋ฆฌ. /api/v1/chat ์๋ํฌ์ธํธ๋ text/vision ์์ฒญ์ ์๋ ํ๋ณํ๊ณ OpenAI ํธ์ถ์ ์ถ์ํ. |
| ํ ์คํธ/์ด์ | tests/test_app.py๋ก API ๋ ๋ฒจ smoke test, requirements.txt์ OpenAI/Azure SDK ๊ณ ์ . |
Status: Redis Streams + Pub/Sub + State KV ๊ธฐ๋ฐ Event Relay ์ํคํ ์ฒ ์๋ฃ
flowchart LR
subgraph Worker["๐ง Celery Worker"]
SW["scan-worker"]
end
subgraph Streams["๐ Redis Streams"]
RS[("scan:events:*<br/>(๋ด๊ตฌ์ฑ)")]
end
subgraph Router["๐ Event Router"]
ER["Consumer Group<br/>XREADGROUP"]
end
subgraph State["๐พ State KV"]
SK[("scan:state:*<br/>(๋ณต๊ตฌ/์กฐํ)")]
end
subgraph PubSub["๐ก Redis Pub/Sub"]
PS[("sse:events:*<br/>(์ค์๊ฐ)")]
end
subgraph Gateway["๐ SSE Gateway"]
SG["Pub/Sub ๊ตฌ๋
<br/>State ๋ณต๊ตฌ<br/>Streams Catch-up"]
end
subgraph Client["๐ค Client"]
CL["Browser/App"]
end
SW -->|XADD| RS
RS -->|XREADGROUP| ER
ER -->|SETEX| SK
ER -->|PUBLISH| PS
SK -.->|GET ์ฌ์ ์| SG
PS -->|SUBSCRIBE| SG
SG -->|SSE| CL
classDef worker fill:#fff9c4,stroke:#f9a825,stroke-width:2px,color:#000
classDef streams fill:#ffccbc,stroke:#e64a19,stroke-width:2px,color:#000
classDef router fill:#b3e5fc,stroke:#0288d1,stroke-width:2px,color:#000
classDef state fill:#d1c4e9,stroke:#512da8,stroke-width:2px,color:#000
classDef pubsub fill:#c8e6c9,stroke:#388e3c,stroke-width:2px,color:#000
classDef gateway fill:#b2dfdb,stroke:#00796b,stroke-width:2px,color:#000
classDef client fill:#e1bee7,stroke:#7b1fa2,stroke-width:2px,color:#000
class SW worker
class RS streams
class ER router
class SK state
class PS pubsub
class SG gateway
class CL client
| ์ปดํฌ๋ํธ | ์ญํ | ์ค์ผ์ผ๋ง |
|---|---|---|
| Event Router | Streams โ Pub/Sub Fan-out, State ๊ฐฑ์ , ๋ฉฑ๋ฑ์ฑ ๋ณด์ฅ | KEDA (Pending ๋ฉ์์ง) |
| SSE Gateway | Pub/Sub โ Client, State ๋ณต๊ตฌ, Streams Catch-up | KEDA (์ฐ๊ฒฐ ์) |
| Redis Streams | ์ด๋ฒคํธ ๋ก๊ทธ (๋ด๊ตฌ์ฑ), Consumer Group ์ง์ | ์ค๋ฉ (4 shards) |
| Redis Pub/Sub | ์ค์๊ฐ Fan-out (fire-and-forget) | ์ ์ฉ ์ธ์คํด์ค |
| State KV | ์ต์ ์ํ ์ค๋ ์ท, ์ฌ์ ์ ๋ณต๊ตฌ | Streams Redis ๊ณต์ |
Status: RabbitMQ + Celery + KEDA ์ด๋ฒคํธ ๋๋ฆฌ๋ธ ์คํ ์ค์ผ์ผ๋ง ์๋ฃ
flowchart LR
subgraph Client["๐ค Client"]
CL["Browser/App"]
end
subgraph API["๐ Scan API"]
SA["POST /api/v1/scan<br/>Dispatch Chain"]
end
subgraph MQ["๐ฌ RabbitMQ"]
VQ[("scan.vision")]
RQ[("scan.rule")]
AQ[("scan.answer")]
WQ[("scan.reward")]
end
subgraph Workers["๐ง Celery Workers (gevent)"]
VW["Vision Worker<br/>GPT Vision ๋ถ์"]
RW["Rule Worker<br/>RAG ๊ท์ ๊ฒ์"]
AW["Answer Worker<br/>GPT ๋ต๋ณ ์์ฑ"]
WW["Reward Worker<br/>๋ณด์ ํ์ "]
end
subgraph External["๐ค OpenAI API"]
OAI["GPT-4o Vision<br/>GPT-4o-mini"]
end
subgraph Streams["๐ Redis Streams"]
RS[("scan:events:*<br/>(Event Relay๋ก ์ ๋ฌ)")]
end
subgraph DB["๐พ PostgreSQL"]
PG[("๊ฒฐ๊ณผ ์ ์ฅ")]
end
subgraph Scale["โก KEDA"]
KD["ํ ๊ธธ์ด ๊ธฐ๋ฐ<br/>์คํ ์ค์ผ์ผ๋ง"]
end
CL -->|POST| SA
SA -->|Dispatch| VQ
SA -.->|202 Accepted| CL
VQ --> VW
VW -->|API Call| OAI
VW -->|XADD| RS
VW -->|Chain| RQ
RQ --> RW
RW -->|XADD| RS
RW -->|Chain| AQ
AQ --> AW
AW -->|API Call| OAI
AW -->|XADD| RS
AW -->|Chain| WQ
WQ --> WW
WW -->|Batch Insert| PG
WW -->|XADD stage=done| RS
KD -.->|Monitor| MQ
KD -.->|Scale| Workers
classDef client fill:#e1bee7,stroke:#7b1fa2,stroke-width:2px,color:#000
classDef api fill:#b2dfdb,stroke:#00796b,stroke-width:2px,color:#000
classDef mq fill:#bbdefb,stroke:#1976d2,stroke-width:2px,color:#000
classDef worker fill:#fff9c4,stroke:#f9a825,stroke-width:2px,color:#000
classDef external fill:#ffcc80,stroke:#e65100,stroke-width:2px,color:#000
classDef streams fill:#ffccbc,stroke:#e64a19,stroke-width:2px,color:#000
classDef db fill:#c8e6c9,stroke:#388e3c,stroke-width:2px,color:#000
classDef scale fill:#b3e5fc,stroke:#0288d1,stroke-width:2px,color:#000
class CL client
class SA api
class VQ,RQ,AQ,WQ mq
class VW,RW,AW,WW worker
class OAI external
class RS streams
class PG db
class KD scale
๐ Sequence Diagram (์์ธ ํ๋ฆ)
sequenceDiagram
participant Client
participant ScanAPI as Scan API
participant RabbitMQ
participant KEDA
participant VisionWorker as Vision Worker
participant RuleWorker as Rule Worker
participant AnswerWorker as Answer Worker
participant RewardWorker as Reward Worker
participant RedisStreams as Redis Streams
participant PostgreSQL
Client->>ScanAPI: POST /api/v1/scan
ScanAPI->>RabbitMQ: Dispatch Chain (job_id)
ScanAPI-->>Client: 202 Accepted {job_id}
KEDA->>RabbitMQ: ํ ๊ธธ์ด ๋ชจ๋ํฐ๋ง
KEDA->>VisionWorker: Scale Up (๋ฉ์์ง ์ฆ๊ฐ ์)
RabbitMQ->>VisionWorker: scan.vision queue
VisionWorker->>VisionWorker: GPT Vision ๋ถ์
VisionWorker->>RedisStreams: XADD stage=vision
VisionWorker->>RabbitMQ: Chain โ scan.rule
RabbitMQ->>RuleWorker: scan.rule queue
RuleWorker->>RuleWorker: RAG ๊ท์ ๊ฒ์
RuleWorker->>RedisStreams: XADD stage=rule
RuleWorker->>RabbitMQ: Chain โ scan.answer
RabbitMQ->>AnswerWorker: scan.answer queue
AnswerWorker->>AnswerWorker: GPT ๋ต๋ณ ์์ฑ
AnswerWorker->>RedisStreams: XADD stage=answer
AnswerWorker->>RabbitMQ: Chain โ reward.character
RabbitMQ->>RewardWorker: reward.character queue
RewardWorker->>PostgreSQL (Batch): ๋ณด์ ์ ์ฅ
RewardWorker->>RedisStreams: XADD stage=done
| ์ปดํฌ๋ํธ | ์ญํ | Queue | ์ค์ผ์ผ๋ง |
|---|---|---|---|
| scan-worker | Vision ๋ถ์, RAG ๊ฒ์, ๋ต๋ณ ์์ฑ, ๋ณด์ ํ์ | scan.vision, scan.rule, scan.answer, scan.reward |
KEDA (ํ ๊ธธ์ด) |
| character-match-worker | ์บ๋ฆญํฐ ๋งค์นญ ์ฒ๋ฆฌ | character.match |
KEDA (ํ ๊ธธ์ด) |
| character-worker | ์บ๋ฆญํฐ ์์ ๊ถ ์ ์ฅ (batch) | character.reward |
KEDA (ํ ๊ธธ์ด) |
| my-worker | ๋ง์ดํ์ด์ง ์บ๋ฆญํฐ ๋๊ธฐํ (batch) | my.reward |
KEDA (ํ ๊ธธ์ด) |
| celery-beat | DLQ ์ฌ์ฒ๋ฆฌ ์ค์ผ์ค๋ง (5๋ถ ์ฃผ๊ธฐ) | - | ๋จ์ผ ์ธ์คํด์ค |
| RabbitMQ | AMQP ๋ฉ์์ง ๋ธ๋ก์ปค | vhost: eco2 |
Quorum Queue |
flowchart LR
subgraph Pods["Kubernetes Pods"]
API["API Pods<br/>(auth, scan, chat...)"]
Workers["Celery Workers<br/>(scan, character-match, character, my)"]
Infra["Infra Pods<br/>(istio, argocd...)"]
end
subgraph FluentBit["Fluent Bit (DaemonSet)"]
Tail["Tail Input<br/>(/var/log/containers/*.log)"]
Parser["Parser<br/>(JSON, regex)"]
Filter["Filter<br/>(kubernetes metadata)"]
Output["Output<br/>(es plugin)"]
end
subgraph EFK["EFK Stack"]
ES[("Elasticsearch<br/>(3 nodes)")]
Kibana["Kibana<br/>(UI)"]
end
API -->|stdout/stderr| Tail
Workers -->|stdout/stderr| Tail
Infra -->|stdout/stderr| Tail
Tail --> Parser
Parser --> Filter
Filter --> Output
Output -->|HTTP/9200| ES
ES --> Kibana
classDef pods fill:#326CE5,stroke:#fff,color:white
classDef fluent fill:#009688,stroke:#fff,color:white
classDef efk fill:#FF9800,stroke:#fff,color:white
class API,Workers,Infra pods
class Tail,Parser,Filter,Output fluent
class ES,Kibana efk
| ์ปดํฌ๋ํธ | ์ญํ | ์ค์ |
|---|---|---|
| Fluent Bit | ๋ก๊ทธ ์์ง ๋ฐ ํฌ์๋ฉ (DaemonSet) | /var/log/containers/*.log ์์ง, JSON ํ์ฑ |
| Elasticsearch | ๋ก๊ทธ ์ ์ฅ ๋ฐ ์ธ๋ฑ์ฑ | 3-node cluster, ์ธ๋ฑ์ค: logstash-YYYY.MM.DD |
| Kibana | ๋ก๊ทธ ๊ฒ์ ๋ฐ ์๊ฐํ | Discover, Dashboard, Alerting |
{
"timestamp": "2025-12-22T10:30:00.000Z",
"level": "INFO",
"logger": "scan.vision_task",
"message": "Vision analysis completed",
"task_id": "550e8400-e29b-41d4-a716-446655440000",
"user_id": "123e4567-e89b-12d3-a456-426614174000",
"duration_ms": 2340,
"kubernetes": {
"namespace": "scan",
"pod_name": "scan-worker-5d8f9b7c4-x2k9p",
"container_name": "scan-worker"
}
}Cluster : kubeadm Self-Managed (21 Nodes)
GitOps :
Layer0 - Terraform (AWS ์ธํ๋ผ)
Layer1 - Ansible (kubeadm, CNI)
Layer2 - ArgoCD App-of-Apps Sync-wave + Kustomize/Helm
Layer3 - GitHub Actions + Docker Hub
Architecture :
Edge Layer - Route 53, AWS ALB, Istio Ingress Gateway
Service Layer - auth, users, my, scan, character, location, chat
Integration Layer :
- Event Relay - Redis Streams + Pub/Sub + State KV, Event Router, SSE Gateway
- Worker (Storage) - auth-worker, auth-relay, users-worker, character-worker, my-worker
- Worker (AI) - scan-worker (VisionโRuleโAnswerโReward)
- KEDA (Event-driven Autoscaling)
Persistence Layer - PostgreSQL, Redis (Blacklist/State/Streams/Pub-Sub/Cache ๋ถ๋ฆฌ)
Platform Layer - ArgoCD, Istiod, KEDA, Observability (Prometheus, Grafana, EFK, Jaeger)
Network : Calico CNI + Istio Service Mesh (mTLS)
Node Isolation :
- worker-storage - Taint: domain=worker-storage:NoSchedule (Persistence ์ ๊ทผ Worker ์ ์ฉ)
- worker-ai - Taint: domain=worker-ai:NoSchedule (AI/OpenAI API ํธ์ถ Worker ์ ์ฉ)- Terraform์ผ๋ก AWS ์ธํ๋ผ๋ฅผ ๊ตฌ์ถํฉ๋๋ค.
- Ansible๋ก ๊ตฌ์ถ๋ AWS ์ธํ๋ผ๋ฅผ ์ฎ์ด K8s ํด๋ฌ์คํฐ๋ฅผ ๊ตฌ์ฑํ๊ณ , ArgoCD root-app์ ์ค์นํฉ๋๋ค.
- ๋ชจ๋ ์ปดํฌ๋ํธ๋ ArgoCD root-app๊ณผ sync๋ ์ํ์ด๋ฉฐ, root-app์ develop ๋ธ๋์น๋ฅผ ๋ฐ๋ผ๋ด ๋๋ค.
- develop ๋ธ๋์น์ push๊ฐ ๋ฐ์ํ๋ฉด CI ํ์ดํ๋ผ์ธ์ ๊ฑฐ์ณ ํ ์คํธ, ๋์ปค ์ด๋ฏธ์ง ํจํค์ง, ํ๋ธ ์ ๋ก๋๊น์ง ์ํํฉ๋๋ค.
- ArgoCD root-app์ develop ๋ธ๋์น์ ๋ณ๊ฒฝ์ฌํญ์ด ๊ฐ์ง๋๋ฉด ํด๋น ํํธ๋ฅผ ์ ๋ฐ์ดํธํด ์ฝ๋ ๋ณ๊ฒฝ์ด ํด๋ฌ์คํฐ๋ก ๋ฐ์๋ฉ๋๋ค.
ArgoCD App-of-Apps ํจํด ๊ธฐ๋ฐ GitOps. ๋ชจ๋ ๋ฆฌ์์ค๋ sync-wave๋ก ์์กด์ฑ ์์ ๋ณด์ฅ.
| Wave | ๋ ์ด์ด | ๋ฆฌ์์ค |
|---|---|---|
| 0-10 | ํ๋ซํผ | CRD, Namespace, RBAC, Istio, NetworkPolicy, Secrets |
| 15-32 | ์ธํ๋ผ | ALB, Monitoring, PostgreSQL, Redis, RabbitMQ |
| 35-50 | ์ ํ๋ฆฌ์ผ์ด์ | KEDA, APIs, Workers, Event Router, Routing |
- App-of-Apps: ๋ฃจํธ ์ฑ โ ApplicationSet ์์ฑ โ
sync-wave๊ฐ์ผ๋ก ๋ฐฐํฌ ์์ ๊ฐ์ - Sync Hook: PostSync Job์ผ๋ก DB ๋ง์ด๊ทธ๋ ์ด์ ์๋ ์คํ
- CI/CD: ์ฝ๋ ๋ณ๊ฒฝ โ GitHub Actions โ Docker Hub โ ArgoCD Auto-Sync
- Event Relay Layer + AI ํ์ดํ๋ผ์ธ โ
- Redis Streams(๋ด๊ตฌ์ฑ) + Pub/Sub(์ค์๊ฐ) + State KV(๋ณต๊ตฌ) 3-tier ์ด๋ฒคํธ ์ํคํ ์ฒ ๊ตฌํ
- Event Router: Consumer Group(
XREADGROUP)์ผ๋ก Streams ์๋น, Pub/Sub Fan-out, ๋ฉฑ๋ฑ์ฑ ๋ณด์ฅ - SSE Gateway: Pub/Sub ๊ตฌ๋ ๊ธฐ๋ฐ ์ค์๊ฐ ์ ๋ฌ, State ๋ณต๊ตฌ, Streams Catch-up
- Celery Chain(VisionโRuleโAnswerโReward): GPT 5.1 Vision + GPT 5.1-mini ์กฐํฉ
- gevent pool (100 greenlets) + httpx connection pooling, ๋จ์ผ ์์ฒญ โ 12์ด
- ๋ถํ ํ ์คํธ ๊ฒฐ๊ณผ (๋จ์ผ ๋ ธ๋ ๊ธฐ์ค, ์ด์ Celery Events ๋๋น 2.8๋ฐฐ ํฅ์)
| VU | ์์ฒญ ์ | ์๋ฃ์จ | Throughput | E2E p95 | Scan p95 | ์ํ |
|---|---|---|---|---|---|---|
| 50 | 685 | 99.7% | 198 req/m | 17.7์ด | 93ms | โ ์ฌ์ |
| 200 | 1,649 | 99.8% | 367 req/m | 33.2์ด | 83ms | โ ์์ |
| 250 | 1,754 | 99.9% | 418 req/m | 40.5์ด | 78ms | โญ SLA ๊ธฐ์ค |
| 300 | 1,732 | 99.9% | 402 req/m | 48.5์ด | 83ms | |
| 400 | 1,901 | 98.9% | 422 req/m | 62.2์ด | 207ms | |
| 500 | 1,990 | 94.0% | 438 req/m | 76.4์ด | 154ms | โ ๋จ์ผ ๋ ธ๋ ํ๊ณ |
-
KEDA ์ด๋ฒคํธ ๋๋ฆฌ๋ธ ์คํ ์ค์ผ์ผ๋ง โ
- scan-worker: RabbitMQ ํ ๊ธธ์ด ๊ธฐ๋ฐ ์๋ ์ค์ผ์ผ๋ง (1-3 replicas)
- event-router: Redis Streams pending ๋ฉ์์ง ๊ธฐ๋ฐ ์ค์ผ์ผ๋ง
- character-match-worker: RabbitMQ character.match ํ ๊ธฐ๋ฐ ์ค์ผ์ผ๋ง
- Prometheus Adapter ์ฐ๋์ผ๋ก ์ปค์คํ ๋ฉํธ๋ฆญ ๊ธฐ๋ฐ HPA ๊ตฌํ
-
๋ถํ ํ ์คํธ ๋ฐ ์ค์ผ์ผ๋ง ๊ฒ์ฆ โ
- 21-Node ํด๋ฌ์คํฐ: Event Router, Redis Pub/Sub ์ ์ฉ ๋ ธ๋ ์ถ๊ฐ
- Redis ์ธ์คํด์ค ๋ถ๋ฆฌ: Streams(๋ด๊ตฌ์ฑ) / Pub/Sub(์ค์๊ฐ) / Cache(LRU)
- ๋ถํ ํ
์คํธ ๊ฒ์ฆ: 50/200/250/300/400/500 VU ํ
์คํธ ์๋ฃ
- ๋จ์ผ ๋ ธ๋(k8s-worker-ai, 2 cores) ๊ธฐ์ค 250 VU SLA, 500 VU ํ๊ณ์ ๋์ถ
- KEDA ์๋ ์ค์ผ์ผ๋ง ๊ฒ์ฆ: scan-worker 1โ3 pods, scan-api 1โ3 pods
๐ ์ด์ฝ์์ฝ(Ecoยฒ) ๋ฐฑ์๋/์ธํ๋ผ ๊ฐ๋ฐ ๋ธ๋ก๊ทธ
- โ Redis Streams + Pub/Sub + State KV ๊ธฐ๋ฐ Event Relay Layer ์๋ฃ
- โ Event Router, SSE Gateway ์ปดํฌ๋ํธ ๊ฐ๋ฐ ์๋ฃ
- โ KEDA ์ด๋ฒคํธ ๋๋ฆฌ๋ธ ์คํ ์ค์ผ์ผ๋ง ์ ์ฉ (scan-worker, event-router, character-match-worker)
- โ Celery ๋น๋๊ธฐ AI ํ์ดํ๋ผ์ธ ์๋ฃ (VisionโRuleโAnswerโReward)
- โ
๋ถํ ํ
์คํธ ์๋ฃ: 50/200/250/300/400/500 VU ๊ฒ์ฆ
- 250 VU (SLA): 99.9% ์๋ฃ์จ, 418 req/m, E2E p95 40์ด
- 500 VU: ๋จ์ผ ๋ ธ๋ ํ๊ณ์ (94% ์๋ฃ์จ, E2E p95 76์ด)
- โ EFK ๋ก๊น ํ์ดํ๋ผ์ธ (Fluent Bit โ Elasticsearch โ Kibana)
- โ ๋ถ์ฐ ํธ๋ ์ด์ฑ (Jaeger + OpenTelemetry + Kiali)
- โ Alertmanager ์๋ฆผ ์์คํ (Slack)
- โ Istio Service Mesh Migration ์๋ฃ
- โ gRPC ๋ด๋ถ ํต์ Migration ์๋ฃ
- โ Auth-Offloading ์๋ฃ, ๋๋ฉ์ธ๋ณ ๋ ๋ฆฝ์ฑ ํ๋ณด
- โ ext-authz ์ฑ๋ฅ ํ๋ (Grafana: VU 2500, RPS 1200, p99 200-300ms)
- โ Terraform ยท Ansible bootstrap ยท ArgoCD Sync-wave
- โ GitOps Sync-Wave ์ฌ์ ๋ ฌ (00~70) + upstream Helm/CRD ๋ถ๋ฆฌ
- โ Docker Hub ์ด๋ฏธ์ง ํ์ดํ๋ผ์ธ + External Secrets ์ด์
- โ API ๊ฐ๋ฐ ์๋ฃ, ํ๋ก ํธ-๋ฐฑ-AI ์ฐ๋ ์๋ฃ

