Pangolin (Status: Alpha)

A Rust-Based, Multi-Tenant, Iceberg-Compatible Lakehouse Catalog

Pangolin is a high-performance catalog designed for modern lakehouse architectures. It supports Git-style branching, multi-tenancy, federated catalogs, and tracks any lakehouse asset type.

Why Pangolin?

A pangolin is a strong metaphor for a data lakehouse catalog because its defining traits align closely with the core responsibilities of a catalog.

First, a pangolin is covered in layered scales. Each scale is distinct but part of a coherent whole. A lakehouse catalog works the same way. It organizes many independent assets—tables, views, files, models, and metadata—into a single, structured system. Each asset has its own schema, properties, and lineage, yet all are discoverable through one catalog.

Second, pangolins are defensive by design. They protect what matters by curling into a secure form. A catalog plays a similar role in governance. It enforces access controls, tracks ownership, and provides guardrails around sensitive data. Rather than blocking access outright, it enables safe and intentional use.

Third, pangolins are precise and deliberate. They move carefully and use strong claws to uncover food hidden beneath the surface. A lakehouse catalog does the same for data. It helps users uncover datasets buried across object storage, warehouses, and streams, exposing meaning through metadata, classification, and search.

Finally, pangolins are rare and specialized. They exist for a specific purpose and excel at it. A data lakehouse catalog is not a generic system. It is a purpose-built layer focused on clarity, trust, and navigation across complex data environments.

---

🚀 Quick Start

Prerequisites

• Rust 1.92+
• Docker (optional, for MinIO)

Running Locally

cd pangolin
cargo run --bin pangolin_api

API Usage

See Quick Start Guide for detailed setup and example curl commands.

---

✨ Key Features

• Multi-Tenancy: Full tenant isolation with dedicated namespaces and warehouses.
• Iceberg REST Catalog: 100% compliant with Apache Iceberg REST spec.
• Git-like Branching: Branch, tag, and merge catalogs for safe experimentation.
• 3-Way Merging: Intelligent conflict detection with manual and automatic resolution strategies.
• Federated Catalogs: Connect to external Iceberg catalogs as a transparent proxy.
• Service Users: API key authentication for CI/CD, ETL, and automated pipelines.
• Advanced Audit Logging: Comprehensive tracking of 40+ actions across 19 resource types.
• Multi-Cloud Storage: Native support for AWS S3, Azure Blob, and Google Cloud Storage.
• Credential Vending: Securely vends AWS STS, Azure SAS, and GCP downscoped credentials.
• Multiple Backends: Metadata persistence via PostgreSQL, MongoDB, SQLite, or In-Memory.
• Management UI: Modern SvelteKit-based interface for Admins and Data Explorers.

---

📚 Documentation Index

🏁 1. Getting Started

• Installation & Setup - Get running in 5 minutes.
• Auth Modes - Understanding Auth vs No-Auth and OAuth.
• User Scopes - Roles: Root, Tenant Admin, and Tenant User.
• Configuration - Server configuration options.
• Environment Variables - Complete metadata and storage reference.

🏗️ 2. Core Infrastructure

• Warehouses - Managing S3, Azure, and GCS storage.
• Catalogs - Creating Local and Federated catalogs.
• Backend Storage - Metadata persistence with Postgres, Mongo, or SQLite.

🧪 3. Data Management (API, CLI, UI)

• Branching & Versioning - Git-style workflows and auto-add nuances.
• Permissions & RBAC - Asset-level access and cascading grants.
• Business Metadata - Tags, search, and data discovery.
• Audit Logging - Security tracking across all tools.
• Maintenance - Snapshots, orphan files, and storage optimization.

🛠️ 4. Tooling & APIs

• CLI Reference - Full guide for pangolin-admin and pangolin-user.
• API Reference - Iceberg REST and Pangolin Management APIs.
• Management UI - Visual administration and data discovery.
• Client Setup - Connecting PyIceberg, Spark, and Trino.

---

🚦 Project Status

Current Version: Alpha

Production-Ready Features:

• ✅ Iceberg REST Catalog API (100% Compliant)
• ✅ Multi-Tenancy & Tenant Isolation
• ✅ Git-like Branching & Tagging
• ✅ Advanced Audit Logging (UI/CLI/API)
• ✅ Service Users & API Keys
• ✅ PostgreSQL, MongoDB, and SQLite Backends
• ✅ Multi-Cloud Storage (S3, Azure, GCS)
• ✅ Management UI for Admins & Explorers

---

📖 Quick Examples

Create a Catalog (API)

curl -X POST http://localhost:8080/api/v1/catalogs \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
  "name": "production",
  "warehouse_name": "main_s3",
  "storage_location": "s3://my-bucket/warehouse"
}'

Create a Branch (CLI)

pangolin-user create-branch dev --from main --catalog production

Use with PyIceberg

from pyiceberg.catalog import load_catalog

catalog = load_catalog(
    "pangolin",
    **{
        "uri": "http://localhost:8080",
        "warehouse": "production",
        "token": "your-jwt-token",
        "header.X-Iceberg-Access-Delegation": "vended-credentials",
    }
)

# Load a table on the 'dev' branch
table = catalog.load_table("analytics.sales@dev")
df = table.scan().to_pandas()

---

📄 License

MIT License - see LICENSE file for details.

---

📞 Support

• Documentation: See docs/ directory.
• Issues: GitHub Issues.
• Discussions: GitHub Discussions.