Data Workflow Platform

A comprehensive data orchestration engine designed for SaaS integration. The platform enables users to design, execute, and monitor complex data workflows through a visual interface.

Features

• Visual DAG Builder: Drag-and-drop interface for designing data workflows
• Extensible Node System: Support for SQL, Python, HTTP, and custom node types
• Multi-tenant Architecture: Secure isolation with role-based access control
• Apache Airflow Integration: Production-tested orchestration engine
• Comprehensive Monitoring: Real-time execution tracking, logs, and metrics
• Data Lineage: Track data flow and transformations across workflows
• Secrets Management: Secure credential storage with HashiCorp Vault
• Multiple Execution Backends: Local, Kubernetes, and serverless support

Architecture

The platform consists of:

• Frontend: React + TypeScript + Vite with React Flow for visual DAG building
• Backend: FastAPI + Python with SQLAlchemy for data persistence
• Orchestration: Apache Airflow with custom operators
• Storage: PostgreSQL (metadata), MinIO/S3 (artifacts), Redis (message broker)
• Observability: Prometheus + Grafana for metrics, structured logging

Prerequisites

• Python 3.11+
• Node.js 18+
• Docker and Docker Compose
• Git

Quick Start

1. Clone the Repository

git clone <repository-url>
cd data-workflow-platform

2. Start Infrastructure Services

Start PostgreSQL, Redis, MinIO, Vault, and Airflow using Docker Compose:

docker-compose up -d

Wait for all services to be healthy:

docker-compose ps

3. Set Up Backend

cd backend

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Copy environment file
cp .env.example .env

# Run database migrations
alembic upgrade head

# Start the API server
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

The API will be available at http://localhost:8000

4. Set Up Frontend

cd frontend

# Install dependencies
npm install

# Start development server
npm run dev

The UI will be available at http://localhost:5173

Development

Backend Development

cd backend

# Run tests
pytest

# Run tests with coverage
pytest --cov=app --cov-report=html

# Format code
black .
ruff check . --fix

# Type checking
mypy app

Frontend Development

cd frontend

# Run tests
npm test

# Run tests in watch mode
npm run test:watch

# Lint code
npm run lint

# Format code
npm run format

# Build for production
npm run build

Project Structure

.
├── backend/                 # FastAPI backend
│   ├── app/                # Application code
│   │   ├── api/           # API endpoints
│   │   ├── core/          # Core functionality
│   │   ├── models/        # SQLAlchemy models
│   │   ├── services/      # Business logic
│   │   └── main.py        # Application entry point
│   ├── tests/             # Test suite
│   ├── alembic/           # Database migrations
│   └── requirements.txt   # Python dependencies
│
├── frontend/               # React frontend
│   ├── src/               # Source code
│   │   ├── components/   # React components
│   │   ├── pages/        # Page components
│   │   ├── hooks/        # Custom hooks
│   │   ├── services/     # API clients
│   │   └── main.tsx      # Application entry point
│   ├── public/            # Static assets
│   └── package.json       # Node dependencies
│
├── airflow/               # Airflow configuration
│   ├── dags/             # Generated DAG files
│   ├── plugins/          # Custom operators
│   └── logs/             # Execution logs
│
├── docker-compose.yml     # Local development services
└── README.md             # This file

Service URLs

When running locally:

• Frontend UI: http://localhost:5173
• Backend API: http://localhost:8000
• API Documentation: http://localhost:8000/docs
• Airflow UI: http://localhost:8080 (admin/admin)
• MinIO Console: http://localhost:9001 (minioadmin/minioadmin)
• Vault: http://localhost:8200 (token: dev-token)

Testing

Backend Tests

cd backend
pytest                          # Run all tests
pytest tests/test_main.py      # Run specific test file
pytest -v                       # Verbose output
pytest --cov=app               # With coverage

Frontend Tests

cd frontend
npm test                        # Run all tests
npm run test:watch             # Watch mode

Configuration

Backend Configuration

Edit backend/.env to configure:

• Database connection
• Redis connection
• S3/MinIO settings
• Airflow API settings
• Vault settings
• JWT secrets

Frontend Configuration

The frontend proxies API requests to the backend. Configure the proxy in frontend/vite.config.ts.

Deployment

Building Docker Images

# Backend
docker build -t workflow-backend:latest ./backend

# Frontend
docker build -t workflow-frontend:latest ./frontend

Kubernetes Deployment

Kubernetes manifests will be provided in future releases.

Contributing

1. Create a feature branch
2. Make your changes
3. Run tests and linting
4. Submit a pull request

License

[License information to be added]

Support

For issues and questions, please open a GitHub issue.