API Import Automation for OAK

Automated workflow to import API specifications from multiple sources (Postman collections, documentation, direct OpenAPI URLs), convert them into OpenAPI specs, validate them, and create pull requests for integration into the OAK (OpenAPI Knowledge) repository structure.

Features

• Multi-Source Import: Support for Postman collections, documentation scraping, and direct OpenAPI URL imports
• AI-Powered Conversion: Uses Anthropic Claude for intelligent docs-to-OpenAPI conversion
• OAK Structure: Organizes specs in apis/openapi/vendor/api/version/ structure
• Validation Pipeline: Comprehensive linting, validation, and smoke testing
• GitHub Integration: Trigger imports via GitHub issues with automatic PR creation
• Production Ready: Based on production workflow patterns with error handling

Setup

Prerequisites

• Node.js (for postman-to-openapi and validation tools)
• Python 3.8+ (for import scripts)
• GitHub repository with Actions enabled

Installation

1. Install Node.js dependencies:

npm install

2. Install Python dependencies:

pip install -r requirements.txt

3. Set up environment variables:

cp .env.example .env
# Edit .env with your API keys

Required environment variables:

• POSTMAN_KEY: Your Postman API key
• ANTHROPIC_API_KEY: Your Anthropic API key for AI-powered docs conversion

GitHub Secrets

Add these secrets to your GitHub repository:

• POSTMAN_KEY: Your Postman API key
• ANTHROPIC_API_KEY: Your Anthropic API key

Usage

GitHub Issues (Recommended)

Create a new issue using the "Import API Spec to OAK" template. Fill in:

• Import Type: postman, docs, or openapi_url
• Vendor Name: e.g., "stripe", "github", "openai"
• API Name: e.g., "payments", "rest", "chat"
• Source URL: Collection URL, docs URL, or OpenAPI spec URL

The workflow will automatically:

1. Parse the issue body for import parameters
2. Run the appropriate import script
3. Validate the generated spec
4. Create a pull request with the results

Manual Import

Postman Collection:

python scripts/import-postman.py --vendor stripe --api payments --url "https://api.getpostman.com/collections/1234?apikey=${POSTMAN_KEY}"

Documentation:

python scripts/import-docs.py --vendor github --api rest --url "https://docs.github.com/en/rest"

Direct OpenAPI URL:

python scripts/import-openapi.py --vendor openai --api chat --url "https://api.openai.com/openapi.json"

Output Structure

Generated specs follow the OAK directory structure:

apis/
└── openapi/
    └── vendor/
        └── api/
            ├── meta.json
            └── version/
                └── openapi.yaml

Example:

apis/
└── openapi/
    ├── stripe/
    │   └── payments/
    │       ├── meta.json
    │       └── 1.0.0/
    │           └── openapi.yaml
    └── github/
        └── rest/
            ├── meta.json
            └── 1.0.0/
                └── openapi.yaml

Meta.json Format

Each API includes metadata:

{
  "vendor": "stripe",
  "api": "payments",
  "source": "postman",
  "source_url": "https://api.getpostman.com/collections/...",
  "versions": ["1.0.0"]
}

Validation Pipeline

All imports run through comprehensive validation:

• Lint: OpenAPI linting with openapi CLI
• Validate: Schema validation with swagger-cli
• Smoke Test: API testing with schemathesis

Validation warnings are reported but don't block spec generation.

AI Model

The system uses Anthropic's Claude 3.5 Sonnet model (claude-3-5-sonnet-20241022) for intelligent conversion of documentation to OpenAPI specs. The AI analyzes HTML content and tables to generate structured API specifications.

Workflow

1. Trigger: GitHub issue created with "api-import" label
2. Parse: Extract import type, vendor, API name, and source URL from issue body
3. Import: Run appropriate import script based on type
4. Validate: Run comprehensive validation pipeline
5. Structure: Organize files in OAK directory structure
6. PR: Create pull request with generated spec and metadata
7. Review: Manual review and merge

Import Types

Postman Collections

• Fetches collection JSON from Postman API
• Converts to OpenAPI using postman-to-openapi
• Supports environment variable expansion in URLs

Documentation Scraping

• Scrapes HTML documentation pages
• Extracts tables and content using BeautifulSoup
• Uses Anthropic Claude to generate OpenAPI specs from content

Direct OpenAPI URLs

• Fetches OpenAPI specs directly from URLs
• Supports both JSON and YAML formats
• Automatically converts JSON to YAML for consistency
• Extracts version information from spec when available

Contributing

To add support for a new API:

1. Create a GitHub issue using the import template
2. Specify the import type, vendor, API name, and source URL
3. The workflow will automatically process and create a PR
4. Review and merge the generated PR

For development:

1. Test import scripts locally first
2. Ensure all validation checks pass
3. Verify OAK directory structure is correct