diff --git a/README.md b/README.md index 58908b7c..909a946a 100644 --- a/README.md +++ b/README.md @@ -1,92 +1,207 @@ ![logo light](assets/logo-light.svg#gh-light-mode-only) ![logo dark](assets/logo-dark.svg#gh-dark-mode-only) +[![Docs](https://img.shields.io/badge/docs-latest-success)](https://docs.exosphere.host) +[![Last commit](https://img.shields.io/github/last-commit/exospherehost/exospherehost)](https://github.com/exospherehost/exospherehost/commits/main) +[![PyPI - Version](https://img.shields.io/pypi/v/exospherehost)](https://pypi.org/project/exospherehost/) +[![Coverage](https://img.shields.io/codecov/c/gh/exospherehost/exospherehost)](https://codecov.io/gh/exospherehost/exospherehost) +[![Kubernetes](https://img.shields.io/badge/Kubernetes-native-326ce5?logo=kubernetes&logoColor=white)](https://github.com/orgs/exospherehost/packages?repo_name=exospherehost) +[![Discord](https://badgen.net/discord/members/V8uuA6mmzg)](https://discord.gg/V8uuA6mmzg) +![Stars](https://img.shields.io/github/stars/exospherehost/exospherehost?style=social) + +# ExosphereHost - AI Agents at Scale with Reliability + +Exosphere helps you build and deploy AI Agents that can handle large amounts of data and run for long periods. Create reusable components (nodes) and connect them into powerful AI Agents/workflows. + +## What Exosphere Does + +- **Build once, use anywhere**: Create reusable AI components +- **Scale automatically**: Run infinite parallel agents +- **Never lose data**: Persistent state management across runs +- **Handle failures**: Built-in error recovery and retry logic +- **Monitor everything**: Web dashboard to track your workflows + +## Quick Start + +### 1. Install Exosphere + +```bash +uv add exospherehost +``` + +### 2. Create Your First AI Node + +A node is a reusable AI component. It could be an AI agent, API call, or any code you want to run. + +```python +from exospherehost import BaseNode +from pydantic import BaseModel + +class CityAnalyzer(BaseNode): + class Inputs(BaseModel): + city_name: str + + class Outputs(BaseModel): + description: str + population: str + + async def execute(self) -> Outputs: + # Your AI logic here + description = f"Analyzing {self.inputs.city_name}..." + population = "1000000" # Example data + + return Outputs(description=description, population=population) +``` + +### 3. Set Up State Manager & Dashboard + +Create a `docker-compose.yml` file to run Exosphere services: + +```yaml +version: "3.8" + +services: + # Database for storing workflow data + mongodb: + image: mongo:7.0 + container_name: exosphere-mongodb + restart: unless-stopped + environment: + MONGO_INITDB_ROOT_USERNAME: admin + MONGO_INITDB_ROOT_PASSWORD: password + MONGO_INITDB_DATABASE: exosphere + volumes: + - mongodb_data:/data/db + ports: + - "27017:27017" + networks: + - exosphere-network + + # State Manager - stores and retrieves data + exosphere-state-manager: + image: ghcr.io/exospherehost/exosphere-state-manager:latest + container_name: exosphere-state-manager + restart: unless-stopped + environment: + - MONGO_URI=mongodb://admin:password@mongodb:27017/exosphere?authSource=admin + - STATE_MANAGER_SECRET=your-secret-key + - MONGO_DATABASE_NAME=exosphere + - SECRETS_ENCRYPTION_KEY=your-encryption-key + depends_on: + - mongodb + ports: + - "8000:8000" + networks: + - exosphere-network + + # Dashboard - web interface to monitor workflows + exosphere-dashboard: + image: ghcr.io/exospherehost/exosphere-dashboard:latest + container_name: exosphere-dashboard + restart: unless-stopped + environment: + - NEXT_PUBLIC_EXOSPHERE_STATE_MANAGER_URL=http://exosphere-state-manager:8000 + depends_on: + - exosphere-state-manager + ports: + - "3000:3000" + networks: + - exosphere-network + +volumes: + mongodb_data: + driver: local + +networks: + exosphere-network: + driver: bridge + attachable: true +``` + +Start the services: + +```bash +docker-compose up -d +``` + +### 4. Run Your Node + +```python +from exospherehost import Runtime + +# Set environment variables to connect to State Manager +# EXOSPHERE_STATE_MANAGER_URI=http://localhost:8000 +# EXOSPHERE_API_KEY=your-secret-key + +# Note: Ensure EXOSPHERE_STATE_MANAGER_URI and EXOSPHERE_API_KEY environment variables are set +Runtime( + name="my-runtime", + namespace="my-project", + nodes=[CityAnalyzer] +).start() +``` + +### 5. Register the graph with the State Manager + +Register the graph with the State Manager: + +```python +import asyncio +from exospherehost import StateManager + +asyncio.run(StateManager(namespace="my-project").upsert_graph( + graph_name="city-analysis-workflow", + secrets={}, + graph_nodes=[ + { + "node_name": "CityAnalyzer", + "identifier": "analyze-nyc", + "inputs": { + "city_name": "New York" + }, + "next_nodes": [] + } + ] +)) +``` ---- -Exosphere is open source infrastructure to run AI agents at scale with first party support for large data and long running flows. - -Exosphere lets you define plug and playable nodes that can then be run on a reliable backbone in the form of a workflow, with: -- Dynamic State Creation at runtime -- Infinite parallel agents -- Persistent state management -- Failure handling - -This allows developers to deploy production agents that can scale beautifully to build robust autonomous AI workflows. - - - -## Getting Started - -- ### Installation - ```bash - uv add exospherehost - ``` - -- ### Define your first node - Each node is an atomic reusable unit on Exosphere. Once registered, you can plug it into any workflow going forward. This could be an agent, an api call, or existing code, anything you want to be a unit of your workflow. - ```python - from exospherehost import BaseNode - from pydantic import BaseModel +### 6. Monitor Your Workflows - class MyFirstNode(BaseNode): +Open your browser and go to `http://localhost:3000` to see the dashboard. - class Inputs(BaseModel): - city:str - #Define inputs taken by node +## What Each Service Does - class Outputs(BaseModel): - description:str - #Output fields from this node +- **MongoDB**: Stores all your workflow data and state +- **State Manager**: Handles data storage and retrieval between workflow runs +- **Dashboard**: Web interface to monitor and manage your workflows - async def execute(self) -> Outputs: - return Outputs(descriptor_agent(inputs.city)) - #Execution function: - # >>Agent - # >>Existing Code - # >>Anything else you want to do! - ``` +## Production Deployment - +For production, use the same Docker images with your own configuration: - Create the node and add it to a runtime to enable execution: - ```python - from exospherehost import Runtime +```bash +# Pull latest images +docker pull ghcr.io/exospherehost/exosphere-state-manager:latest +docker pull ghcr.io/exospherehost/exosphere-dashboard:latest - Runtime( - name="my-first-runtime", - namespace="hello-world", - nodes=[ - MyFirstNode - ] - ).start() - ``` - -- ### Define your first flow - - Flows are then described connecting nodes with relationships in json objects. Exosphere runs flows as per defined trigger conditions. See [Flow defintions](docs.exosphere.host) to see more examples. - ```json - { - "secrets": {}, - "nodes": [ - { - "node_name": "MyFirstNode", - "namespace": "hello-world", - "identifier": "describe_city", - "inputs": { - "bucket_name": "initial", - "prefix": "initial", - "files_only": "true", - "recursive": "false" - }, - "next_nodes": ["create_batches"] - }, - ``` +# Run with your settings +docker run -d \ + --name exosphere-state-manager \ + -p 8000:8000 \ + -e MONGO_URI=your-mongodb-connection \ + -e STATE_MANAGER_SECRET=your-secret \ + ghcr.io/exospherehost/exosphere-state-manager:latest +``` ## Documentation - +For detailed guides and examples, visit [docs.exosphere.host](https://docs.exosphere.host) ## Contributing -We welcome community contributions. For guidelines, refer to our [CONTRIBUTING.md](/CONTRIBUTING.md). Further we are thankful to all the contributors helping us to simplify infrastructure starting with the process of building and deploying AI workflows and agents. +We welcome contributions! See [CONTRIBUTING.md](/CONTRIBUTING.md) for guidelines. + +Join our [Discord community](https://discord.gg/V8uuA6mmzg) for discussions and weekly feature talks. + +## Thank You! -Join our Discord: https://discord.gg/msUHahrp for active community discussions. We have weekly community huddle to talk up feature dicsussions, feel free to become a part of the conversation. +![exosphere.host Contributors](https://contrib.rocks/image?repo=exospherehost/exospherehost) diff --git a/READMEold.md b/READMEold.md deleted file mode 100644 index 6fd9f14f..00000000 --- a/READMEold.md +++ /dev/null @@ -1,238 +0,0 @@ -![logo light](assets/logo-light.svg#gh-light-mode-only) -![logo dark](assets/logo-dark.svg#gh-dark-mode-only) - -> We are building a world where creators and innovators can fully dedicate themselves to crafting extraordinary products and services, unburdened by the complexities of the underlying infrastructure. We foresee a future where intelligent systems seamlessly operate behind the scenes, tackling intricate, high-scale challenges with immense computational demands and vast data movements. - -To realize this, we are pioneering an open-source infrastructure layer for background AI workflows and agents that is robust, affordable, and effortless to use, empowering the scalable solutions and transformative tasks of today, tomorrow, and beyond. - -## Core Concepts -To have an intution of the first version of the platform, we would highly recommend watching the video below, this explains using our cluster apis with YML input, we are working on more modalities like pythonic control systems. - - - Cluster API YT - - -### Satellite -Satellites are the core building blocks for exosphere. They are lego blocks designed for a specific purpose: you can connect them together to create complex systems in a matter of minutes without worrying about the underlying infrastructure. - -They are pre-implemented serverless functions highly optimized for workflows and high volume batch processing, optimized for cost, reliability, developer velocity and ease of use. - -Our in-house optimization for workflows and batch processing can lead to significant cost savings, for example you can expect a cost per token saving of about 50-75% on LLMs like DeepSeek R1 70B, Gemma 3 27B, etc. - -Each of these satellites must satisfy the following properties: - -1. Should be idempotent and stateless. -2. Should have a unique identifier of the format `satellite/unique-project-name/satellite-name`, example: `satellite/exospherehost/deepseek-r1-distrill-llama-70b` -3. Should take a `config` parameter as an `object` to control or modify the behaviour. -4. Should be totally independent of any other satellite. -5. Should return a `list` of `objects`. -6. Should have following necessary fields: `parents`, `children`, `identifier`, `status`, `retries`, `delay`. Most of these fields are optional are set by the platform itself. -7. Should provide a clean interface for the user to check the status of the satellite and get output data. - -Further work is being done to allow users to bring their own satellites and use our core infrastructure to manage their lifecycle. - -### Cluster -A Cluster is a collection of satellites connected together to form a complete workflow: a series of satellities working together to achieve a common goal. - -Each of these clusters must satisfy the following properties: - -1. Should be a collection of satellites that are connected together to form a system. -2. Should have a unique identifier of the format `cluster/unique-project-name/cluster-name`, example: `cluster/aikin/structured-json` -3. Should define a necessary parameter of `SLA` denoting the maximum time to complete the cluster, **higher the SLA, lower the cost** as systems have more time to optimize for the task (currently supported: `6h`, `12h`, `24h`) -4. Should have a necessary `trigger` parameter to start the cluster, this can be a `cron` expression, or an `api-call` or other possible events. -5. Each cluster can also define `logs` parameter to configure log forwarding to a specific destination like `NewRelic`, `Kusto`, `CloudWatch` or any other logging service. -6. Each cluster can also define `failure` steps to handle the cluster in case of failure, this could again be a set of satellites to run in case of failure. - -Developers can define their own clusters using our cluster api, which supports cluster creation, deletion, status, logs and other operations. Currently we are supporting cluster creation through `YML` files or our APIs and SDKs. - -### Orbit -Orbit is the core compute platform capable of managing the lifecycle of satellites and clusters optimally across multiple computes including GPUs, CPUs, and other hardware. Further allowing developers to write their own satellites and plug-in with our core exosphere platform. - -## Example -Here is an example of using our cluster api to create a satellite cluster to get structured json from PDF files of quaterly financial reports. The workflow in the image could be represented as the `YML` file below. - -![Example Workflow](assets/example-workflow.png) - -```yaml -# define the version of the exosphere apis -version: 0.0.1b - -# using cluster api to create the cluster -cluster: - # maximum allowed to compute the cluster - # define the sla of the cluster (6h, 12h, 24h) - sla: 6h - - # define the name and description of the cluster for better understanding (optional) - title: Structured JSON from Quarterly Financial Reports - description: This cluster will take a list of PDF files from S3 and return a structured JSON output. - - # define the identifier of the cluster (project-name/cluster-name) - identifier: aikin/strucutred-pdfs - - # trigger for the cluster - trigger: - cron: "0 0 * * *" - - # define retries for each satellite, default is 5 - retries: 3 - - # define the secrets for the cluster, these are stored in a secure vault and only accessible by allowed satellites in this cluster - # still be sure to have minimum required permissions for each secret to avoid any security issues - secrets: - - AWS_ACCESS_KEY: "your-aws-access-key" - - AWS_SECRET_KEY: "your-aws-secret-key" - - API_BEARER_TOKEN: "your-api-bearer-token" - - NEW_RELIC_ACCOUNT_ID: "your-new-relic-account-id" - - NEW_RELIC_API_KEY: "your-new-relic-api-key" - - NEW_RELIC_APPLICATION_ID: "your-new-relic-application-id" - - FAILURE_S3_AWS_ACCESS_KEY: "your-failure-s3-aws-access-key" - - FAILURE_S3_AWS_SECRET_KEY: "your-failure-s3-aws-secret-key" - - # satellites in order to execute, each satellite returns a list of objects hence computational graph is formed and is executed in parallel. - satellites: - - name: Get files from S3 - uses: satellite/exospherehost/get-files-from-s3 - identifier: get-files-from-s3 - config: - bucket: aikin-financial-reports - prefix: sec - recursive: true - extension: pdf - secrets: - - AWS_ACCESS_KEY: ${{ secrets.AWS_ACCESS_KEY }} - - AWS_SECRET_KEY: ${{ secrets.AWS_SECRET_KEY }} - - name: Extract text from PDF - uses: satellite/exospherehost/parse-pdf-with-docling - identifier: parse-pdf-with-docling - config: - language: en - output-format: markdown-string - extract-images: false - - name: Get structured json from markdown using DeepSeek - uses: satellite/exospherehost/deepseek-r1-distrill-llama-70b - identifier: deepseek-r1-distrill-llama-70b - retries: 5 - config: - temperature: 0.5 - max-tokens: 1024 - output-format: json - output-schema: | - { - "company": string, - "quarter": string, - "year": string, - "revenue": number, - "net-income": number, - "gross-profit": number, - "operating-income": number, - "net-income-margin": number, - "gross-profit-margin": number, - "operating-income-margin": number, - } - input: - prompt: | - Parse the following quarterly financial report and return a structured json output as defined in the output-schema, report text is provided below: - ${{satellites.parse-pdf-with-docling.output}} - - name: Call Webhook to send the structured json to Aikin API - uses: satellite/exospherehost/call-webhook - identifier: call-webhook - config: - url: https://api.aikin.com/v1/financial-reports - method: POST - headers: - - Authorization: Bearer ${{ secrets.API_BEARER_TOKEN }} - body: - - data: ${{satellites.deepseek-r1-distrill-llama-70b.output}} - file-path: $${{satellites.get-files-from-s3.output.file-path}} - - name: Delete success file from S3 - uses: satellite/exospherehost/delete-file-from-s3 - identifier: delete-file-from-s3 - config: - bucket: aikin-financial-reports - file-path: $${{satellites.get-files-from-s3.output.file-path}} - secrets: - - AWS_ACCESS_KEY: ${{ secrets.AWS_ACCESS_KEY }} - - AWS_SECRET_KEY: ${{ secrets.AWS_SECRET_KEY }} - - - # define steps to handle logs for this cluster - logs: - satellites: - - name: Send logs to NewRelic - uses: satellite/exospherehost/send-logs-to-new-relic - identifier: send-logs-to-new-relic - config: - account-id: ${{ secrets.NEW_RELIC_ACCOUNT_ID }} - api-key: ${{ secrets.NEW_RELIC_API_KEY }} - application-id: ${{ secrets.NEW_RELIC_APPLICATION_ID }} - log-level: info - - # define steps to handle failure for this cluster - failure: - # run from failed steps from this satellite - from: parse-pdf-with-docling - satellites: - - name: Move to failure bucket - uses: satellite/exospherehost/move-to-s3 - identifier: move-to-failure-bucket - config: - origin-source: s3 - origin-bucket: aikin-financial-reports-failure - origin-file-path: $${{satellites.get-files-from-s3.output.file-path}} - destination-source: s3 - destination-bucket: aikin-financial-reports-failure - destination-file-path: failed/quaterly-financial-reports/$${{satellites.get-files-from-s3.output.file-name}} - secrets: - - ORIGIN_AWS_ACCESS_KEY: ${{ secrets.AWS_ACCESS_KEY }} - - ORIGIN_AWS_SECRET_KEY: ${{ secrets.AWS_SECRET_KEY }} - - DESTINATION_AWS_ACCESS_KEY: ${{ secrets.FAILURE_S3_AWS_ACCESS_KEY }} - - DESTINATION_AWS_SECRET_KEY: ${{ secrets.FAILURE_S3_AWS_SECRET_KEY }} - - name: Send failure notification on PagerDuty - uses: satellite/exospherehost/send-pagerduty-alert - identifier: send-pagerduty-alert - config: - pagerduty-api-key: ${{ secrets.PAGERDUTY_API_KEY }} - pagerduty-service-id: ${{ secrets.PAGERDUTY_SERVICE_ID }} - input: - message: | - Cluster ${{cluster.identifier}} failed at ${{cluster.trigger}} for file $${{satellites.get-files-from-s3.output.file-path}} with error ${{satellites.get-files-from-s3.output.error}}, file has been moved to failure bucket with path $${{satellites.move-to-failure-bucket.output.file-uri}} -``` - -This could also be represented as a pythonic control using our SDK/APIs, checkout [documentation](https://docs.exosphere.host) for more details. - -## Documentation -For more information, please refer to our [documentation](https://docs.exosphere.host). - -### Steps to build the Documentation locally -1. Install UV: Follow the official instructions [here](https://docs.astral.sh/uv/#installation). -2. Clone this repository, use command `git clone https://github.com/exospherehost/exospherehost.git` -3. Install dependencies by navigating to `docs` folder and executing `uv sync` -4. Use the command `uv run mkdocs serve` while `docs` folder being your working path. - -### Contribute to Documentation -We encourage contributions to the documentation page, you can simply add a new PR with the `documentation` label. - -## Open Source Commitment - -We believe that humanity would not have been able to achieve the level of innovation and progress we have today without the support of open source and community, we want to be a part of this movement and support the open source community. In following ways: - -1. We will be open sourcing majority of our codebase for exosphere.host and making it available to the community. We welcome all sort of contributions and feedback from the community and will be happy to collaborate with you. -2. For whatever the profits which we generate from exosphere.host, we will be donating a portion of it to open source projects and communities. If you have any questions, suggestions or ideas. -3. We would further be collaborating with various open source student programs to provide support, encourage and mentor the next generation of open source contributors. - -Please feel free to reach out to us at [nivedit@exosphere.host](mailto:nivedit@exosphere.host). Lets push the boundaries of possibilities for humanity together! - -## Contributing -We welcome community contributions. For guidelines, refer to our [CONTRIBUTING.md](/CONTRIBUTING.md). Further we are thankful to all the contributors helping us to simplify infrastructure starting with the process of building and deploying AI workflows and agents. - -![exosphere.host Contributors](https://contrib.rocks/image?repo=exospherehost/exospherehost) - -## Star History - - - - - Star History Chart - - diff --git a/docs/docs/exosphere/create-runtime.md b/docs/docs/exosphere/create-runtime.md index 430cad7d..ad178f0d 100644 --- a/docs/docs/exosphere/create-runtime.md +++ b/docs/docs/exosphere/create-runtime.md @@ -24,6 +24,7 @@ The `Runtime` class is the core component that manages the execution environment return self.Outputs(result=f"Processed: {self.inputs.data}") # Create and start the runtime + # Note: Ensure EXOSPHERE_STATE_MANAGER_URI and EXOSPHERE_API_KEY environment variables are set Runtime( namespace="MyProject", name="MyRuntime", @@ -52,6 +53,7 @@ The `Runtime` class is the core component that manages the execution environment return self.Outputs(result=f"Processed: {self.inputs.data}") # Create runtime with custom configuration + # Note: Ensure EXOSPHERE_STATE_MANAGER_URI and EXOSPHERE_API_KEY environment variables are set runtime = Runtime( namespace="MyProject", name="MyRuntime", @@ -112,6 +114,7 @@ from exospherehost import Runtime, BaseNode The runtime validates configuration and node classes: ```python +# Note: Ensure EXOSPHERE_STATE_MANAGER_URI and EXOSPHERE_API_KEY environment variables are set runtime = Runtime( namespace="MyProject", name="MyRuntime", @@ -151,6 +154,7 @@ class DataValidatorNode(BaseNode): ... # Register both nodes +# Note: Ensure EXOSPHERE_STATE_MANAGER_URI and EXOSPHERE_API_KEY environment variables are set Runtime( namespace="MyProject", name="DataPipeline", @@ -167,6 +171,7 @@ Choose appropriate values based on your workload: ```python hl_lines="4 13" # For CPU-intensive tasks +# Note: Ensure EXOSPHERE_STATE_MANAGER_URI and EXOSPHERE_API_KEY environment variables are set Runtime( namespace="MyProject", name="CPU", @@ -176,6 +181,7 @@ Runtime( ).start() # For GPU-intensive tasks +# Note: Ensure EXOSPHERE_STATE_MANAGER_URI and EXOSPHERE_API_KEY environment variables are set Runtime( namespace="MyProject", name="GPU", @@ -191,6 +197,7 @@ Adjust based on your latency requirements: ```python hl_lines="6 14" # For real-time processing +# Note: Ensure EXOSPHERE_STATE_MANAGER_URI and EXOSPHERE_API_KEY environment variables are set Runtime( namespace="MyProject", name="RealTime", @@ -199,6 +206,7 @@ Runtime( ).start() # For batch processing +# Note: Ensure EXOSPHERE_STATE_MANAGER_URI and EXOSPHERE_API_KEY environment variables are set Runtime( namespace="MyProject", name="Batch", diff --git a/docs/docs/exosphere/dashboard.md b/docs/docs/exosphere/dashboard.md index 36f2c4ff..8d158b4b 100644 --- a/docs/docs/exosphere/dashboard.md +++ b/docs/docs/exosphere/dashboard.md @@ -41,7 +41,7 @@ Before setting up the dashboard, ensure you have: docker run -d \ --name exosphere-dashboard \ -p 3000:3000 \ - -e NEXT_PUBLIC_STATE_MANAGER_URL="http://localhost:8000" \ + -e NEXT_PUBLIC_EXOSPHERE_STATE_MANAGER_URL="http://localhost:8000" \ -e NEXT_PUBLIC_API_KEY="your-api-key" \ -e NEXT_PUBLIC_NAMESPACE="your-namespace" \ ghcr.io/exospherehost/exosphere-dashboard:latest @@ -65,7 +65,7 @@ Before setting up the dashboard, ensure you have: | Variable | Description | Required | Default | |----------|-------------|----------|---------| - | `NEXT_PUBLIC_STATE_MANAGER_URL` | State manager API endpoint | Yes | - | + | `NEXT_PUBLIC_EXOSPHERE_STATE_MANAGER_URL` | State manager API endpoint | Yes | - | | `NEXT_PUBLIC_API_KEY` | API key for authentication | Yes | - | | `NEXT_PUBLIC_NAMESPACE` | Default namespace | Yes | - | @@ -111,7 +111,7 @@ Before setting up the dashboard, ensure you have: ```bash # State manager API endpoint - NEXT_PUBLIC_STATE_MANAGER_URL=http://localhost:8000 + NEXT_PUBLIC_EXOSPHERE_STATE_MANAGER_URL=http://localhost:8000 # API key for authentication NEXT_PUBLIC_API_KEY=your-api-key diff --git a/docs/docs/exosphere/getting-started.md b/docs/docs/exosphere/getting-started.md index 2e7b6fda..5faea8c9 100644 --- a/docs/docs/exosphere/getting-started.md +++ b/docs/docs/exosphere/getting-started.md @@ -12,14 +12,14 @@ Set up your environment variables for authentication: === ".env File" ```bash - EXOSPHERE_URI=your-state-manager-uri - EXOSPHERE_KEY=your-api-key + EXOSPHERE_STATE_MANAGER_URI=your-state-manager-uri + EXOSPHERE_API_KEY=your-api-key ``` === "Environment Variables" ```bash - export EXOSPHERE_URI="your-state-manager-uri" - export EXOSPHERE_KEY="your-api-key" + export EXOSPHERE_STATE_MANAGER_URI="your-state-manager-uri" + export EXOSPHERE_API_KEY="your-api-key" ``` Refer: [Getting State Manager URI](./state-manager-setup.md) @@ -79,6 +79,7 @@ class SampleNode(BaseNode): ) # Initialize the runtime +# Note: Ensure EXOSPHERE_STATE_MANAGER_URI and EXOSPHERE_API_KEY environment variables are set Runtime( namespace="MyProject", name="DataProcessor", @@ -108,13 +109,13 @@ Now that you have the basics, explore: ## Architecture ``` -┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐ -│ Your Nodes │ │ Runtime │ │ State Manager │ -│ │◄──►│ │◄──►│ │ -│ - Inputs │ │ - Registration │ │ - Orchestration │ -│ - Outputs │ │ - Execution │ │ - State Mgmt │ -│ - Secrets │ │ - Error Handling │ │ - Dashboard │ -└─────────────────┘ └──────────────────┘ └─────────────────┘ +------------------- -------------------- -------------------- +│ Your Nodes │ <-> │ Runtime │ <-> │ State Manager │ +│ │ │ │ │ │ +│ - Inputs │ │ - Registration │ │ - Orchestration │ +│ - Outputs │ │ - Execution │ │ - State Mgmt │ +│ - Secrets │ │ - Error Handling │ │ - Dashboard │ +------------------- -------------------- -------------------- ``` ## Data Model (v1) diff --git a/docs/docs/exosphere/index.md b/docs/docs/exosphere/index.md index 294cb435..00f53de6 100644 --- a/docs/docs/exosphere/index.md +++ b/docs/docs/exosphere/index.md @@ -46,6 +46,7 @@ class HelloWorldNode(BaseNode): ) # Initialize the runtime +# Note: Ensure EXOSPHERE_STATE_MANAGER_URI and EXOSPHERE_API_KEY environment variables are set Runtime( namespace="MyProject", name="HelloWorld", @@ -120,6 +121,7 @@ class DataProcessorNode(BaseNode): ) # Initialize the runtime +# Note: Ensure EXOSPHERE_STATE_MANAGER_URI and EXOSPHERE_API_KEY environment variables are set Runtime( namespace="MyProject", name="DataProcessor", diff --git a/docs/docs/exosphere/register-node.md b/docs/docs/exosphere/register-node.md index 941b58b3..e9ad3d4d 100644 --- a/docs/docs/exosphere/register-node.md +++ b/docs/docs/exosphere/register-node.md @@ -122,6 +122,7 @@ class Node2(BaseNode): # ... node implementation # Register nodes with runtime +# Note: Ensure EXOSPHERE_STATE_MANAGER_URI and EXOSPHERE_API_KEY environment variables are set Runtime( namespace="MyProject", name="MyRuntime", @@ -137,6 +138,7 @@ Organise nodes to a namespace to re-use them across flows in that namespace ```python hl_lines="3 10" # Development namespace +# Note: Ensure EXOSPHERE_STATE_MANAGER_URI and EXOSPHERE_API_KEY environment variables are set Runtime( namespace="dev", name="MyRuntime", @@ -144,6 +146,7 @@ Runtime( ).start() # Production namespace +# Note: Ensure EXOSPHERE_STATE_MANAGER_URI and EXOSPHERE_API_KEY environment variables are set Runtime( namespace="prod", name="MyRuntime", diff --git a/docs/docs/exosphere/state-manager-setup.md b/docs/docs/exosphere/state-manager-setup.md index 67df630e..c27a4128 100644 --- a/docs/docs/exosphere/state-manager-setup.md +++ b/docs/docs/exosphere/state-manager-setup.md @@ -23,11 +23,10 @@ The Exosphere state manager is the core backend service that handles workflow ex docker run -d \ --name exosphere-state-manager \ -p 8000:8000 \ - -e MONGODB_URI="mongodb://localhost:27017/exosphere" \ - -e DATABASE_NAME="exosphere" \ - -e API_KEY="your-secret-api-key" \ - -e NAMESPACE="your-namespace" \ - -e ENCRYPTION_KEY="your-32-character-encryption-key" \ + -e MONGO_URI="mongodb://localhost:27017/exosphere" \ + -e MONGO_DATABASE_NAME="exosphere" \ + -e STATE_MANAGER_SECRET="your-secret-api-key" \ + -e SECRETS_ENCRYPTION_KEY="your-32-character-encryption-key" \ ghcr.io/exospherehost/state-manager:latest ``` @@ -49,11 +48,10 @@ The Exosphere state manager is the core backend service that handles workflow ex | Variable | Description | Required | |----------|-------------|----------| - | `MONGODB_URI` | MongoDB connection string | Yes | - | `DATABASE_NAME` | Database name | Yes | - | `API_KEY` | Secret API key for authentication | Yes | - | `NAMESPACE` | Default namespace for operations | Yes | - | `ENCRYPTION_KEY` | 32-character key for data encryption | Yes | + | `MONGO_URI` | MongoDB connection string | Yes | + | `MONGO_DATABASE_NAME` | Database name | Yes | + | `STATE_MANAGER_SECRET` | Secret API key for authentication | Yes | + | `SECRETS_ENCRYPTION_KEY` | 32-character key for data encryption | Yes | === "Local Development" @@ -80,11 +78,10 @@ The Exosphere state manager is the core backend service that handles workflow ex 3. **Set up environment variables**: ```bash - export MONGODB_URI="mongodb://localhost:27017/exosphere" - export DATABASE_NAME="exosphere" - export API_KEY="your-secret-api-key" - export NAMESPACE="your-namespace" - export ENCRYPTION_KEY="your-32-character-encryption-key" + export MONGO_URI="mongodb://localhost:27017/exosphere" + export MONGO_DATABASE_NAME="exosphere" + export STATE_MANAGER_SECRET="your-secret-api-key" + export SECRETS_ENCRYPTION_KEY="your-32-character-encryption-key" ``` 4. **Run the state manager**: @@ -146,11 +143,10 @@ The Exosphere state manager is the core backend service that handles workflow ex ports: - "8000:8000" environment: - - MONGODB_URI=mongodb://your-mongodb-cluster:27017/exosphere - - DATABASE_NAME=exosphere - - API_KEY=${API_KEY} - - NAMESPACE=${NAMESPACE} - - ENCRYPTION_KEY=${ENCRYPTION_KEY} + - MONGO_URI=mongodb://your-mongodb-cluster:27017/exosphere + - MONGO_DATABASE_NAME=exosphere + - STATE_MANAGER_SECRET=${STATE_MANAGER_SECRET} + - SECRETS_ENCRYPTION_KEY=${SECRETS_ENCRYPTION_KEY} deploy: replicas: 3 update_config: @@ -180,9 +176,8 @@ The Exosphere state manager is the core backend service that handles workflow ex name: state-manager-config namespace: exosphere data: - MONGODB_URI: "mongodb://your-mongodb-cluster:27017/exosphere" - DATABASE_NAME: "exosphere" - NAMESPACE: "your-namespace" + MONGO_URI: "mongodb://your-mongodb-cluster:27017/exosphere" + MONGO_DATABASE_NAME: "exosphere" LOG_LEVEL: "INFO" ``` @@ -195,8 +190,8 @@ The Exosphere state manager is the core backend service that handles workflow ex namespace: exosphere type: Opaque data: - API_KEY: - ENCRYPTION_KEY: + STATE_MANAGER_SECRET: + SECRETS_ENCRYPTION_KEY: ``` 4. **Deploy the state manager**: @@ -212,11 +207,10 @@ The Exosphere state manager is the core backend service that handles workflow ex | Variable | Description | Required | Default | |----------|-------------|----------|---------| -| `MONGODB_URI` | MongoDB connection string | Yes | - | -| `DATABASE_NAME` | Database name | Yes | `exosphere` | -| `API_KEY` | Secret API key for authentication | Yes | - | -| `NAMESPACE` | Default namespace for operations | Yes | - | -| `ENCRYPTION_KEY` | 32-character key for data encryption | Yes | - | +| `MONGO_URI` | MongoDB connection string | Yes | - | +| `MONGO_DATABASE_NAME` | Database name | Yes | `exosphere` | +| `STATE_MANAGER_SECRET` | Secret API key for authentication | Yes | - | +| `SECRETS_ENCRYPTION_KEY` | 32-character key for data encryption | Yes | - | | `LOG_LEVEL` | Logging level (DEBUG, INFO, WARNING, ERROR) | No | `INFO` | | `WORKERS` | Number of worker processes | No | CPU count | | `HOST` | Host to bind to | No | `0.0.0.0` | diff --git a/python-sdk/README.md b/python-sdk/README.md index 7df6c3c2..68ec69a5 100644 --- a/python-sdk/README.md +++ b/python-sdk/README.md @@ -52,6 +52,7 @@ class SampleNode(BaseNode): ) # Initialize the runtime +# Note: Ensure EXOSPHERE_STATE_MANAGER_URI and EXOSPHERE_API_KEY environment variables are set Runtime( namespace="MyProject", name="DataProcessor",