From 05a2af06f9db486747a53c4b129b06bebbd371f7 Mon Sep 17 00:00:00 2001 From: Tim Delisle Date: Wed, 5 Nov 2025 13:39:50 -0500 Subject: [PATCH] initial migration of pages and manual testing code sizeing new batch data insertion versioning & migration next batch next batch fix mdx-rendered and adding next batch of files adding api files fixed some language rendering and added more pages added workflows files added other workflow files streaming files added additional frameworks added new batch added top level files data modeling all files migrated initial changelog addressing cursor comments fix tabs added a bit of padding to the nav added sonner for language change fix TOC fix the build add ts ignore to mux player removed top nav fix TOC challenges added templates to new docs split server & client components moved to static api route instead --- .../content/moosestack/apis/analytics-api.mdx | 1078 +++++++++++++- .../content/moosestack/apis/auth.mdx | 273 +++- .../content/moosestack/apis/index.mdx | 151 +- .../content/moosestack/apis/ingest-api.mdx | 366 ++++- .../content/moosestack/apis/openapi-sdk.mdx | 106 +- .../content/moosestack/apis/trigger-api.mdx | 104 +- .../moosestack/app-api-frameworks/express.mdx | 395 ++++- .../moosestack/app-api-frameworks/fastapi.mdx | 577 +++++++- .../moosestack/app-api-frameworks/fastify.mdx | 388 +++++ .../moosestack/app-api-frameworks/index.mdx | 195 ++- .../moosestack/app-api-frameworks/koa.mdx | 451 ++++++ .../app-api-frameworks/raw-nodejs.mdx | 495 +++++++ .../content/moosestack/changelog.mdx | 482 ++++++- .../content/moosestack/configuration.mdx | 382 ++++- .../content/moosestack/data-modeling.mdx | 911 +++++++++++- .../deploying/configuring-moose-for-cloud.mdx | 128 +- .../deploying-on-an-offline-server.mdx | 269 +++- .../moosestack/deploying/deploying-on-ecs.mdx | 164 ++- .../deploying/deploying-on-kubernetes.mdx | 302 +++- .../deploying-with-docker-compose.mdx | 835 ++++++++++- .../content/moosestack/deploying/index.mdx | 82 +- .../moosestack/deploying/monitoring.mdx | 9 +- .../packaging-moose-for-deployment.mdx | 74 +- .../preparing-clickhouse-redpanda.mdx | 120 +- .../getting-started/from-clickhouse.mdx | 558 +++++++- .../moosestack/getting-started/quickstart.mdx | 762 +++++++++- .../content/moosestack/help/index.mdx | 67 +- .../moosestack/help/minimum-requirements.mdx | 27 +- .../moosestack/help/troubleshooting.mdx | 267 +++- .../content/moosestack/in-your-stack.mdx | 102 +- .../content/moosestack/index.mdx | 13 +- .../moosestack/local-dev-environment.mdx | 411 +++++- .../content/moosestack/metrics.mdx | 168 ++- .../content/moosestack/migrate/index.mdx | 377 ++++- .../content/moosestack/migrate/lifecycle.mdx | 234 ++- .../moosestack/migrate/migration-types.mdx | 358 ++++- .../content/moosestack/moose-cli.mdx | 255 +++- .../content/moosestack/moosedev-mcp.mdx | 593 +++++++- .../moosestack/olap/apply-migrations.mdx | 594 +++++++- .../content/moosestack/olap/db-pull.mdx | 155 +- .../moosestack/olap/external-tables.mdx | 210 ++- .../content/moosestack/olap/indexes.mdx | 80 +- .../content/moosestack/olap/insert-data.mdx | 519 ++++++- .../olap/model-materialized-view.mdx | 991 ++++++++++++- .../content/moosestack/olap/model-table.mdx | 1270 ++++++++++++++++- .../content/moosestack/olap/model-view.mdx | 90 +- .../moosestack/olap/planned-migrations.mdx | 318 ++++- .../content/moosestack/olap/read-data.mdx | 612 +++++++- .../content/moosestack/olap/schema-change.mdx | 199 ++- .../moosestack/olap/schema-optimization.mdx | 483 ++++++- .../moosestack/olap/schema-versioning.mdx | 176 ++- .../moosestack/olap/supported-types.mdx | 861 ++++++++++- .../content/moosestack/olap/ttl.mdx | 75 +- .../content/moosestack/quickstart.mdx | 759 +++++++++- .../content/moosestack/reference/index.mdx | 831 ++++++++++- .../moosestack/reference/py-moose-lib.mdx | 9 - .../moosestack/reference/ts-moose-lib.mdx | 9 - .../moosestack/streaming/connect-cdc.mdx | 11 +- .../streaming/consumer-functions.mdx | 718 +++++++++- .../moosestack/streaming/create-stream.mdx | 516 ++++++- .../streaming/dead-letter-queues.mdx | 656 ++++++++- .../moosestack/streaming/from-your-code.mdx | 227 ++- .../content/moosestack/streaming/index.mdx | 133 +- .../moosestack/streaming/schema-registry.mdx | 125 +- .../moosestack/streaming/sync-to-table.mdx | 274 +++- .../streaming/transform-functions.mdx | 620 +++++++- .../content/moosestack/templates-examples.mdx | 70 +- .../moosestack/workflows/cancel-workflow.mdx | 69 +- .../moosestack/workflows/define-workflow.mdx | 218 ++- .../content/moosestack/workflows/index.mdx | 159 ++- .../workflows/retries-and-timeouts.mdx | 143 +- .../workflows/schedule-workflow.mdx | 315 +++- .../moosestack/workflows/trigger-workflow.mdx | 125 +- apps/framework-docs-v2/package.json | 3 + .../Avatars/Logos-avatar=5-Color-SM.png | Bin 0 -> 853 bytes .../public/ClickHouse-icon.svg | 31 + apps/framework-docs-v2/public/artifact.png | Bin 0 -> 214300 bytes .../framework-docs-v2/public/autocomplete.png | Bin 0 -> 203449 bytes .../public/ch-cloud-creds.png | Bin 0 -> 152783 bytes .../public/ch-cloud-sidebar.png | Bin 0 -> 9076 bytes apps/framework-docs-v2/public/chat.png | Bin 0 -> 92566 bytes .../public/claude-search-and-tools.png | Bin 0 -> 183927 bytes .../public/claude-settings.png | Bin 0 -> 383842 bytes .../public/cursor-mcp-settings.png | Bin 0 -> 143157 bytes apps/framework-docs-v2/public/favicon.ico | Bin 0 -> 1535 bytes apps/framework-docs-v2/public/figjam-arch.jpg | Bin 0 -> 3415353 bytes .../public/github-dashboard.jpeg | Bin 0 -> 713395 bytes apps/framework-docs-v2/public/logo-dark.png | Bin 0 -> 627 bytes apps/framework-docs-v2/public/logo-light.png | Bin 0 -> 490 bytes apps/framework-docs-v2/public/new-chat.png | Bin 0 -> 135266 bytes apps/framework-docs-v2/public/odw-dark.svg | 150 ++ .../public/og-image-fiveonefour.png | Bin 0 -> 107497 bytes .../public/og-image-moose.png | Bin 0 -> 809177 bytes .../public/og-image-sloan.png | Bin 0 -> 2411096 bytes .../public/openapi-request.png | Bin 0 -> 275777 bytes apps/framework-docs-v2/public/openapi-try.png | Bin 0 -> 268688 bytes apps/framework-docs-v2/public/openapi.png | Bin 0 -> 697780 bytes apps/framework-docs-v2/public/sitemap-0.xml | 221 +-- apps/framework-docs-v2/public/sqltools.png | Bin 0 -> 27399 bytes .../public/thunder-client.png | Bin 0 -> 528944 bytes apps/framework-docs-v2/public/ufa-dark.svg | 139 ++ .../public/vscode-install.png | Bin 0 -> 179103 bytes .../scripts/migrate-content.ts | 227 --- .../src/app/[...slug]/page.tsx | 2 +- .../src/app/api/templates/route.ts | 22 + apps/framework-docs-v2/src/app/layout.tsx | 2 + .../src/components/mdx-renderer.tsx | 86 +- .../src/components/mdx/callout.tsx | 13 +- .../src/components/mdx/changelog-category.tsx | 169 +++ .../components/mdx/code-editor-wrapper.tsx | 47 +- .../src/components/mdx/code-snippet.tsx | 11 +- .../src/components/mdx/export-requirement.tsx | 69 + .../src/components/mdx/file-tree.tsx | 37 + .../src/components/mdx/index.ts | 15 + .../src/components/mdx/language-tabs.tsx | 33 +- .../src/components/mdx/mux-video.tsx | 61 + .../src/components/mdx/template-card.tsx | 181 +++ .../src/components/mdx/template-grid.tsx | 257 ++++ .../components/mdx/templates-grid-server.tsx | 49 + .../src/components/navigation/side-nav.tsx | 33 +- .../src/components/navigation/toc-nav.tsx | 172 +-- .../src/components/navigation/top-nav.tsx | 27 +- .../ui/shadcn-io/code-block/index.tsx | 47 +- .../ui/shadcn-io/code-block/server.tsx | 37 +- .../src/components/ui/sonner.tsx | 31 + .../src/config/navigation.ts | 43 +- .../src/lib/template-types.ts | 46 + apps/framework-docs-v2/src/lib/templates.ts | 308 ++++ pnpm-lock.yaml | 91 +- 129 files changed, 27029 insertions(+), 780 deletions(-) create mode 100644 apps/framework-docs-v2/content/moosestack/app-api-frameworks/fastify.mdx create mode 100644 apps/framework-docs-v2/content/moosestack/app-api-frameworks/koa.mdx create mode 100644 apps/framework-docs-v2/content/moosestack/app-api-frameworks/raw-nodejs.mdx delete mode 100644 apps/framework-docs-v2/content/moosestack/reference/py-moose-lib.mdx delete mode 100644 apps/framework-docs-v2/content/moosestack/reference/ts-moose-lib.mdx create mode 100644 apps/framework-docs-v2/public/Avatars/Logos-avatar=5-Color-SM.png create mode 100644 apps/framework-docs-v2/public/ClickHouse-icon.svg create mode 100644 apps/framework-docs-v2/public/artifact.png create mode 100644 apps/framework-docs-v2/public/autocomplete.png create mode 100644 apps/framework-docs-v2/public/ch-cloud-creds.png create mode 100644 apps/framework-docs-v2/public/ch-cloud-sidebar.png create mode 100644 apps/framework-docs-v2/public/chat.png create mode 100644 apps/framework-docs-v2/public/claude-search-and-tools.png create mode 100644 apps/framework-docs-v2/public/claude-settings.png create mode 100644 apps/framework-docs-v2/public/cursor-mcp-settings.png create mode 100644 apps/framework-docs-v2/public/favicon.ico create mode 100644 apps/framework-docs-v2/public/figjam-arch.jpg create mode 100644 apps/framework-docs-v2/public/github-dashboard.jpeg create mode 100644 apps/framework-docs-v2/public/logo-dark.png create mode 100644 apps/framework-docs-v2/public/logo-light.png create mode 100644 apps/framework-docs-v2/public/new-chat.png create mode 100644 apps/framework-docs-v2/public/odw-dark.svg create mode 100644 apps/framework-docs-v2/public/og-image-fiveonefour.png create mode 100644 apps/framework-docs-v2/public/og-image-moose.png create mode 100644 apps/framework-docs-v2/public/og-image-sloan.png create mode 100644 apps/framework-docs-v2/public/openapi-request.png create mode 100644 apps/framework-docs-v2/public/openapi-try.png create mode 100644 apps/framework-docs-v2/public/openapi.png create mode 100644 apps/framework-docs-v2/public/sqltools.png create mode 100644 apps/framework-docs-v2/public/thunder-client.png create mode 100644 apps/framework-docs-v2/public/ufa-dark.svg create mode 100644 apps/framework-docs-v2/public/vscode-install.png delete mode 100644 apps/framework-docs-v2/scripts/migrate-content.ts create mode 100644 apps/framework-docs-v2/src/app/api/templates/route.ts create mode 100644 apps/framework-docs-v2/src/components/mdx/changelog-category.tsx create mode 100644 apps/framework-docs-v2/src/components/mdx/export-requirement.tsx create mode 100644 apps/framework-docs-v2/src/components/mdx/file-tree.tsx create mode 100644 apps/framework-docs-v2/src/components/mdx/mux-video.tsx create mode 100644 apps/framework-docs-v2/src/components/mdx/template-card.tsx create mode 100644 apps/framework-docs-v2/src/components/mdx/template-grid.tsx create mode 100644 apps/framework-docs-v2/src/components/mdx/templates-grid-server.tsx create mode 100644 apps/framework-docs-v2/src/components/ui/sonner.tsx create mode 100644 apps/framework-docs-v2/src/lib/template-types.ts create mode 100644 apps/framework-docs-v2/src/lib/templates.ts diff --git a/apps/framework-docs-v2/content/moosestack/apis/analytics-api.mdx b/apps/framework-docs-v2/content/moosestack/apis/analytics-api.mdx index 4c807397c1..f836e8b858 100644 --- a/apps/framework-docs-v2/content/moosestack/apis/analytics-api.mdx +++ b/apps/framework-docs-v2/content/moosestack/apis/analytics-api.mdx @@ -1,9 +1,1077 @@ --- -title: Expose Analytics -description: Expose analytics data through APIs -order: 3 +title: APIs +description: APIs for Moose +order: 1 +category: apis --- -# Expose Analytics +import { Callout, LanguageTabs, LanguageTabContent, BulletPointsCard, ToggleBlock, CTACards, CTACard, Tabs, TabsList, TabsTrigger, TabsContent } from "@/components/mdx"; -This page is a placeholder. Content migration pending. +# APIs + +## Overview + +APIs are functions that run on your server and automatically exposed as HTTP `GET` endpoints. +They are designed to read data from your OLAP database. Out of the box, these APIs provide: + +- Automatic type validation and type conversion for your query parameters, which are sent in the URL, and response body +- Managed database client connection +- Automatic OpenAPI documentation generation + +Common use cases include: +- Powering user-facing analytics, dashboards and other front-end components +- Enabling AI tools to interact with your data +- Building custom APIs for your internal tools + +### Enabling APIs + +Analytics APIs are enabled by default. To explicitly control this feature in your `moose.config.toml`: + +```toml filename="moose.config.toml" copy +[features] +apis = true +``` + +### Basic Usage + + + +```ts filename="ExampleApi.ts" copy +import { Api } from "@514labs/moose-lib"; +import { SourcePipeline } from "path/to/SourcePipeline"; + +// Define the query parameters +interface QueryParams { + filterField: string; + maxResults: number; +} + +// Model the query result type +interface ResultItem { + id: number; + name: string; + value: number; +} + + +const SourceTable = SourcePipeline.table!; // Use `!` to assert that the table is not null +const cols = SourceTable.columns; + +// Define the result type as an array of the result item type +export const exampleApi = new Api("example_endpoint", + async ({ filterField, maxResults }: QueryParams, { client, sql }) => { + const query = sql` + SELECT + ${cols.id}, + ${cols.name}, + ${cols.value} + FROM ${SourceTable} + WHERE category = ${filterField} + LIMIT ${maxResults}`; + + // Set the result type to the type of the each row in the result set + const resultSet = await client.query.execute(query); + + // Return the result set as an array of the result item type + return await resultSet.json(); + }); +``` + + +```ts filename="SourcePipeline.ts" copy +import { IngestPipeline } from "@514labs/moose-lib"; + +interface SourceSchema { + id: number; + name: string; + value: number; +} + +export const SourcePipeline = new IngestPipeline("Source", { + ingestApi: false, + stream: true, + table: true, +}); +``` + + + + + + execute + execute_raw + + +`execute` is the recommended way to execute queries. It provides a thin wrapper around the ClickHouse Python client so that you can safely pass `OlapTable` and `Column` objects to your query without needing to worry about ClickHouse identifiers: + +```python filename="ExampleApi.py" copy +from moose_lib import Api, MooseClient +from pydantic import BaseModel + +# Import the source pipeline +from app.path.to.SourcePipeline import SourcePipeline + +# Define the query parameters +class QueryParams(BaseModel): + filter_field: str + max_results: int + +# Define the response body +class ResponseBody(BaseModel): + id: int + name: str + value: float + + +SourceTable = SourcePipeline.get_table() + +# Define the route handler function (parameterized) +def run(client: MooseClient, params: QueryParams) -> list[ResponseBody]: + query = """ + SELECT + id, + name, + value + FROM {table} + WHERE category = {category} + LIMIT {limit} + """ + + return client.query.execute(query, {"table": SourceTable, "category": params.filter_field, "limit": params.max_results}) + +# Create the API +example_api = Api[QueryParams, ResponseBody](name="example_endpoint", query_function=run) +``` + + +Use `execute_raw` with parameter binding for safe, typed queries: + +```python filename="ExampleApi.py" copy +from moose_lib import Api, MooseClient +from pydantic import BaseModel + +# Import the source pipeline +from app.path.to.SourcePipeline import SourcePipeline + +# Define the query parameters +class QueryParams(BaseModel): + filterField: str + maxResults: int + +# Define the response body +class ResponseBody(BaseModel): + id: int + name: str + value: float + + +SourceTable = SourcePipeline.get_table() + +# Define the route handler function (using execute_raw with typed parameters) +def run(client: MooseClient, params: QueryParams) -> list[ResponseBody]: + query = """ + SELECT + id, + name, + value + FROM Source + WHERE category = {category:String} + LIMIT {limit:UInt32} + """ + + return client.query.execute_raw(query, {"category": params.filterField, "limit": params.maxResults}) + +# Create the API +example_api = Api[QueryParams, ResponseBody](name="example_endpoint", query_function=run) +``` + + + + +```python filename="SourcePipeline.py" copy +from moose_lib import IngestPipeline, IngestPipelineConfig, Key +from pydantic import BaseModel + +class SourceSchema(BaseModel): + id: Key[int] + name: str + value: float + +SourcePipeline = IngestPipeline[SourceSchema]("Source", IngestPipelineConfig( + ingest_api=False, + stream=True, + table=True, +)) +``` + + + + +The `Api` class takes: +- Route name: The URL path to access your API (e.g., `"example_endpoint"`) +- Handler function: Processes requests with typed parameters and returns the result + +The generic type parameters specify: +- `QueryParams`: The structure of accepted URL parameters +- `ResponseBody`: The exact shape of your API's response data + + +You can name these types anything you want. The first type generates validation for query parameters, while the second defines the response structure for OpenAPI documentation. + + + + +## Type Validation + +You can also model the query parameters and response body as interfaces (TypeScript) or Pydantic models (Python), which Moose will use to provide automatic type validation and type conversion for your query parameters, which are sent in the URL, and response body. + +### Modeling Query Parameters + + + +Define your API's parameters as a TypeScript interface: + +```ts filename="ExampleQueryParams.ts" copy +interface QueryParams { + filterField: string; + maxResults: number; + optionalParam?: string; // Not required for client to provide +} +``` + + +Define your API's parameters as a Pydantic model: + +```python filename="ExampleQueryParams.py" copy +from pydantic import BaseModel +from typing import Optional + +class QueryParams(BaseModel): + filterField: str = Field(..., description="The field to filter by") + maxResults: int = Field(..., description="The maximum number of results to return") + optionalParam: Optional[str] = Field(None, description="An optional parameter") +``` + + + +Moose automatically handles: +- Runtime validation +- Clear error messages for invalid parameters +- OpenAPI documentation generation + + +Complex nested objects and arrays are not supported. Analytics APIs are `GET` endpoints designed to be simple and lightweight. + + +### Adding Advanced Type Validation + + + +Moose uses [Typia](https://typia.io/) to extract type definitions and provide runtime validation. Use Typia's tags for more complex validation: + +```ts filename="ExampleQueryParams.ts" copy +interface QueryParams { + filterField: string; + // Ensure maxResults is a positive integer + maxResults: number & tags.Type<"int64"> & tags.Minimum<"1">; +} +``` + + +Moose uses Pydantic for runtime validation. Use Pydantic's `Field` class for more complex validation: + +```python filename="ExampleQueryParams.py" copy +from pydantic import BaseModel, Field + +class QueryParams(BaseModel): + filterField: str = Field(pattern=r"^(id|name|email)$", description="The field to filter by") ## Only allow valid column names from the UserTable + maxResults: int = Field(gt=0, description="The maximum number of results to return") ## Positive integer +``` + + + +### Common Validation Options + + + +```ts filename="ValidationExamples.ts" copy +interface QueryParams { + // Numeric validations + id: number & tags.Type<"uint32">; // Positive integer (0 to 4,294,967,295) + age: number & tags.Minimum<18> & tags.Maximum<120>; // Range: 18 <= age <= 120 + price: number & tags.ExclusiveMinimum<0> & tags.ExclusiveMaximum<1000>; // Range: 0 < price < 1000 + discount: number & tags.MultipleOf<0.5>; // Must be multiple of 0.5 + + // String validations + username: string & tags.MinLength<3> & tags.MaxLength<20>; // Length between 3-20 characters + email: string & tags.Format<"email">; // Valid email format + zipCode: string & tags.Pattern<"^[0-9]{5}$">; // 5 digits + uuid: string & tags.Format<"uuid">; // Valid UUID + ipAddress: string & tags.Format<"ipv4">; // Valid IPv4 address + + // Date validations + startDate: string & tags.Format<"date">; // YYYY-MM-DD format + + // Literal validation + status: "active" | "pending" | "inactive"; // Must be one of these values + + // Optional parameters + limit?: number & tags.Type<"uint32"> & tags.Maximum<100>; // Optional, if provided: positive integer <= 100 + + // Combined validations + searchTerm?: (string & tags.MinLength<3>) | null; // Either null or string with ≥3 characters +} +``` + +Notice its just regular TypeScript union types. For a full list of validation options, see the [Typia documentation](https://typia.io/api/tags). + +You can derive a safe orderBy union from your actual table columns and use it directly in SQL: +```ts filename="ValidationExamples.ts" copy +interface MyTableSchema { + column1: string; + column2: number; + column3: string; +} + +const MyTable = new OlapTable("my_table"); + +interface QueryParams { + orderByColumn: keyof MyTableSchema; // validates against the column names in "my_table" +} +``` + + +```python filename="ValidationExamples.py" copy +from pydantic import BaseModel, Field + +class QueryParams(BaseModel): + # Numeric validations + id: int = Field(..., gt=0) + age: int = Field(..., gt=0, lt=120) + price: float = Field(..., gt=0, lt=1000) + discount: float = Field(..., gt=0, multiple_of=0.5) + + # String validations + username: str = Field(..., min_length=3, max_length=20) + email: str = Field(..., format="email") + zipCode: str = Field(..., pattern=r"^[0-9]{5}$") + uuid: str = Field(..., format="uuid") + ipAddress: str = Field(..., format="ipv4") + + # Date validations + startDate: str = Field(..., format="date") + + # Enum validation + status: str = Field(..., enum=["active", "pending", "inactive"]) + + # Optional parameters + limit: int = Field(None, gt=0, lt=100) +``` +For a full list of validation options, see the [Pydantic documentation](https://docs.pydantic.dev/latest/concepts/types/#customizing-validation-with-fields). + + + +### Setting Default Values + + + +You can set default values for parameters by setting values for each parameter in the API route handler function signature: + +```ts filename="ExampleQueryParams.ts" copy +interface QueryParams { + filterField: string; + maxResults: number; + optionalParam?: string; // Not required for client to provide +} + +const api = new Api("example_endpoint", + async ({ filterField = "example", maxResults = 10, optionalParam = "default" }, { client, sql }) => { + // Your logic here... + } +); +``` + + +You can set default values for parameters by setting values for each parameter in your Pydantic model: + +```python filename="ExampleQueryParams.py" copy +from pydantic import BaseModel + +class QueryParams(BaseModel): + filterField: str = "example" + maxResults: int = 10 + optionalParam: str | None = "default" +``` + + + +## Implementing Route Handler + +API route handlers are regular functions, so you can implement whatever arbitrary logic you want inside these functions. Most of the time you will be use APIs to expose your data to your front-end applications or other tools: + +### Connecting to the Database + +Moose provides a managed `MooseClient` to your function execution context. This client provides access to the database and other Moose resources, and handles connection pooling/lifecycle management for you: + + + +```ts filename="ExampleApi.ts" copy +import { ApiUtil } from "@514labs/moose-lib"; +import { UserTable } from "./UserTable"; + +async function handler({ client, sql }: ApiUtil) { + const query = sql`SELECT * FROM ${UserTable}`; + const data = await client.query.execute(query); +} +``` + +Pass the type of the result to the `client.query.execute()` method to ensure type safety. + + +```ts filename="UserTable.ts" copy +import { OlapTable } from "@514labs/moose-lib"; + +interface UserSchema { + id: Key + name: string + email: string +} + +export const UserTable = new OlapTable("users") +``` + + + + + + execute + execute_raw + + +```python filename="ExampleApi.py" copy +from moose_lib import MooseClient +from app.UserTable import UserTable + + +def run(client: MooseClient, params: QueryParams): + # You can use a formatted string for simple static query + query = """ + SELECT COUNT(*) FROM {table} + """ + + ## You can optionally pass the table object to the query + return client.query.execute(query, {"table": UserTable}) + +# Create the API +example_api = Api[QueryParams, ResponseBody](name="example_endpoint", query_function=run) +``` + + +Use `execute_raw` with parameter binding: + +```python filename="ExampleApi.py" copy +from moose_lib import MooseClient + +def run(params: QueryParams, client: MooseClient): + # Using execute_raw for safe queries + query = """ + SELECT COUNT(*) FROM {table: Identifier} + """ + + ## Must be the name of the table, not the table object + return client.query.execute_raw(query, {"table": UserTable.name}) + +## Create the API +example_api = Api[QueryParams, ResponseBody](name="example_endpoint", query_function=run) +``` + + + + + +### Constructing Safe SQL Queries + + + +The `sql` template literal in Moose provides type-safe query construction with protection against SQL injection. Below are some examples of common patterns for builing safe queries: + + + + + execute + execute_raw + + +```python filename="SafeQueries.py" copy +from pydantic import BaseModel, Field + +class QueryParams(BaseModel): + min_age: int = Field(ge=0, le=150) + status: str = Field(pattern=r"^(active|inactive)$") + limit: int = Field(default=10, ge=1, le=1000) + search_text: str = Field(pattern=r'^[a-zA-Z0-9\s]*$') + +def run(client: MooseClient, params: QueryParams): + query = """ + SELECT * + FROM users + WHERE age >= {min_age} + AND status = '{status}' + AND name ILIKE '%{search_text}%' + LIMIT {limit} + """ + return client.query.execute(query, {"min_age": params.min_age, "status": params.status, "search_text": params.search_text, "limit": params.limit}) +``` + + +```python filename="SafeQueries.py" copy +from pydantic import BaseModel, Field + +class QueryParams(BaseModel): + min_age: int = Field(ge=0, le=150) + status: str = Field(pattern=r"^(active|inactive)$") + limit: int = Field(default=10, ge=1, le=1000) + search_text: str = Field(pattern=r'^[a-zA-Z0-9\s]*$') + +def run(client: MooseClient, params: QueryParams): + query = """ + SELECT * + FROM users + WHERE age >= {minAge:UInt32} + AND status = {status:String} + AND name ILIKE {searchPattern:String} + LIMIT {limit:UInt32} + """ + return client.query.execute_raw(query, { + "minAge": params.min_age, + "status": params.status, + "searchPattern": f"%{params.search_text}%", + "limit": params.limit + }) +``` + + + + + +#### Basic Query Parameter Interpolation + + + +```ts filename="BasicInterpolation.ts" copy +import { UserTable } from "./UserTable"; + +const minAge = 18; +const userRole = "admin"; + +const query = sql` + SELECT * FROM ${UserTable} + WHERE age > ${minAge} + AND role = ${userRole} +`; + +// MooseClient handles type conversion and escaping +const data = await client.query.execute(query); + +// EXECUTION: SELECT * FROM users WHERE age > 18 AND role = 'admin' +``` + +```ts filename="UserTable.ts" copy +import { OlapTable } from "@514labs/moose-lib"; + +interface UserSchema { + id: Key + name: string + email: string +} + +export const UserTable = new OlapTable("users") +``` + + + +```python filename="ValidatedQueries.py" copy +from moose_lib import Api, MooseClient +from pydantic import BaseModel, Field, constr +from typing import Literal, Optional +from enum import Enum +from app.UserTable import UserTable + +class QueryParams(BaseModel): + # When using f-strings, we need extremely strict validation + column: str = Field(pattern=r"^(id|name|email)$", description="Uses a regex pattern to only allow valid column names") + + search_term: str = Field( + pattern=r'^[\w\s\'-]{1,50}$', # Allows letters, numbers, spaces, hyphens, apostrophes; Does not allow special characters that could be used in SQL injection + strip_whitespace=True, + min_length=1, + max_length=50 + ) + + limit: int = Field( + default=10, + ge=1, + le=100, + description="Number of results to return" + ) + +def run(client: MooseClient, params: QueryParams): + query = """ + SELECT {column} + FROM {table} + WHERE name ILIKE '%{search_term}%' + LIMIT {limit} + """ + + return client.query.execute(query, {"column": UserTable.cols[params.column], "table": UserTable, "search_term": params.search_term, "limit": params.limit}) + +``` + + +```python filename="UserTable.py" copy +from moose_lib import OlapTable, Key +from pydantic import BaseModel + +class UserSchema(BaseModel): + id: Key[int] + name: str + email: str + +UserTable = OlapTable[UserSchema]("users") +``` + + + + +#### Table and Column References + + + +Reference tables and columns directly from your Moose objects as variables in your `sql` template literals: +```ts filename="ExampleTableColumnQuery.ts" copy +import { userTable } from "../tables/userTable"; + +const query = sql` + SELECT + ${UserTable.columns.id}, + ${UserTable.columns.name}, + ${UserTable.columns.email} + FROM ${UserTable} + WHERE ${UserTable.columns.isActive} = true +`; + +// EXECUTION: SELECT id, name, email FROM users WHERE is_active = true +``` + +```ts filename="UserTable.ts" copy +import { OlapTable } from "@514labs/moose-lib"; + +interface UserSchema { + id: Key + name: string + email: string +} + +export const UserTable = new OlapTable("users") +``` + + + +Static type checking ensures you only reference columns that actually exist. + + + + +### Advanced Query Patterns + +#### Dynamic Column & Table Selection + + + +Use `ApiHelpers` to handle dynamic column and table references in your queries: + +```ts filename="DynamicColumns.ts" copy +import { ApiHelpers as CH } from "@514labs/moose-lib"; + +interface QueryParams { + sortBy: string; // Column to sort by + fields: string; // Comma-separated list of columns to select (e.g., "id,name,email") +} + +const queryHandler = async ({ sortBy = "id", fields = "id,name" }: QueryParams, { client, sql }) => { + // Split the comma-separated string into individual fields + const fieldList = fields.split(',').map(f => f.trim()); + + // Build the query by selecting each column individually + const query = sql` + SELECT + ${fieldList.map(field => sql`${CH.column(field)}`).join(', ')} + FROM ${userTable} + ORDER BY ${CH.column(sortBy)} + `; + + // MooseClient converts fieldList to valid ClickHouse identifiers + return client.query.execute(query); + + // EXECUTION: `SELECT id, name FROM users ORDER BY id` +}; +``` + +```ts filename="DynamicTables.ts" copy +import { ApiHelpers as CH } from "@514labs/moose-lib"; + +interface QueryParams { + tableName: string; +} + +const queryHandler = async ({ tableName = "users" }: QueryParams, { client, sql }) => { + const query = sql` + SELECT * FROM ${CH.table(tableName)} + `; + + // MooseClient converts tableName to a valid ClickHouse identifier + return client.query.execute(query); + + // EXECUTION: `SELECT * FROM users` +}; +``` + + +```python filename="DynamicColumns.py" copy +from app.UserTable import UserTable + +class QueryParams(BaseModel): + colName: str = Field(pattern=r"^(id|name|email)$", description="Uses a regex pattern to only allow valid column names from the UserTable") + +class QueryResult(BaseModel): + id: Optional[int] + name: Optional[str] + email: Optional[str] + +def run(client: MooseClient, params: QueryParams): + # Put column and table in the dict for variables + query = "SELECT {column} FROM {table}" + return client.query.execute(query, {"column": UserTable.cols[params.colName], "table": UserTable}) + +# Create the API +bar = Api[QueryParams, QueryResult](name="bar", query_function=run) + +# Call the API +# HTTP Request: GET http://localhost:4000/api/bar?colName=id +# EXECUTED QUERY: SELECT id FROM users +``` + +```python filename="UserTable.py" copy +from moose_lib import OlapTable, Key +from pydantic import BaseModel + +class UserSchema(BaseModel): + id: Key[int] + name: str + email: str + +UserTable = OlapTable[UserSchema]("users") +``` + + + + +#### Conditional `WHERE` Clauses + +Build `WHERE` clauses based on provided parameters: + + + +```ts filename="ConditionalColumns.ts" copy +interface FilterParams { + minAge?: number; + status?: "active" | "inactive"; + searchText?: string; +} + +const buildQuery = ({ minAge, status, searchText }: FilterParams, { sql }) => { + let conditions = []; + + if (minAge !== undefined) { + conditions.push(sql`age >= ${minAge}`); + } + + if (status) { + conditions.push(sql`status = ${status}`); + } + + if (searchText) { + conditions.push(sql`(name ILIKE ${'%' + searchText + '%'} OR email ILIKE ${'%' + searchText + '%'})`); + } + + // Build the full query with conditional WHERE clause + let query = sql`SELECT * FROM ${userTable}`; + + if (conditions.length > 0) { + // Join conditions with AND operator + let whereClause = conditions.join(' AND '); + query = sql`${query} WHERE ${whereClause}`; + } + + query = sql`${query} ORDER BY created_at DESC`; + return query; +}; +``` + + +```python filename="ConditionalColumns.py" copy + +class FilterParams(BaseModel): + min_age: Optional[int] + status: Optional[str] = Field(pattern=r"^(active|inactive)$") + search_text: Optional[str] = Field(pattern=r"^[a-zA-Z0-9\s]+$", description="Alphanumeric search text without special characters to prevent SQL injection") + +class QueryResult(BaseModel): + id: int + name: str + email: str + +def build_query(client: MooseClient, params: FilterParams) -> QueryResult: + # Using f-strings with validated parameters + conditions = [] + if params.min_age: + conditions.append("age >= {min_age}") + parameters["min_age"] = params.min_age + + if params.status: + conditions.append("status = {status}") + parameters["status"] = params.status + + if params.search_text: + conditions.append("(name ILIKE {search_text} OR email ILIKE {search_text})") + parameters["search_text"] = params.search_text + + where_clause = f" WHERE {' AND '.join(conditions)}" if conditions else "" + query = f"""SELECT * FROM users {where_clause} ORDER BY created_at DESC""" + return client.query.execute(query, parameters) + +# Create the API +bar = Api[FilterParams, QueryResult](name="bar", query_function=build_query) + +# Call the API +# HTTP Request: GET http://localhost:4000/api/bar?min_age=20&status=active&search_text=John +# EXECUTED QUERY: SELECT * FROM users WHERE age >= 20 AND status = 'active' AND (name ILIKE '%John%' OR email ILIKE '%John%') ORDER BY created_at DESC +``` + + + +### Adding Authentication + +Moose supports authentication via JSON web tokens (JWTs). When your client makes a request to your Analytics API, Moose will automatically parse the JWT and pass the **authenticated** payload to your handler function as the `jwt` object: + + + +```typescript filename="Authentication.ts" copy +async ( + { orderBy = "totalRows", limit = 5 }, + { client, sql, jwt } +) => { + // Use jwt.userId to filter data for the current user + const query = sql` + SELECT * FROM userReports + WHERE user_id = ${jwt.userId} + LIMIT ${limit} + `; + + return client.query.execute(query); +} +``` + + +```python filename="Authentication.py" copy + +def run(client: MooseClient, params: QueryParams, jwt: dict): + # Use parameter binding with JWT data + query = """SELECT * FROM userReports WHERE user_id = {user_id} LIMIT 5""" + return client.query.execute(query, {"user_id": jwt["userId"]}) +``` + + + + +Moose validates the JWT signature and ensures the JWT is properly formatted. If the JWT authentication fails, Moose will return a `401 Unauthorized error`. + + +## Understanding Response Codes + +Moose automatically provides standard HTTP responses: + +| Status Code | Meaning | Response Body | +|-------------|-------------------------|---------------------------------| +| 200 | Success | Your API's result data | +| 400 | Validation error | `{ "error": "Detailed message"}`| +| 401 | Unauthorized | `{ "error": "Unauthorized"}` | +| 500 | Internal server error | `{ "error": "Internal server error"}` | + +## Post-Processing Query Results + +After executing your database query, you can transform the data before returning it to the client. This allows you to: + + + + + +```ts filename="PostProcessingExample.ts" copy +interface QueryParams { + category: string; + maxResults: number; +} + +interface ResponseBody { + itemId: number; + displayName: string; + formattedValue: string; + isHighValue: boolean; + date: string; +} + +const processDataApi = new Api( + "process_data_endpoint", + async ({ category, maxResults = 10 }, { client, sql }) => { + // 1. Fetch raw data + const query = sql` + SELECT id, name, value, timestamp + FROM data_table + WHERE category = ${category} + LIMIT ${maxResults} + `; + + const rawResults = await client.query.execute<{ + id: number; + name: string; + value: number; + timestamp: string; + }>(query); + + // 2. Post-process the results + return rawResults.map(row => ({ + // Transform field names + itemId: row.id, + displayName: row.name.toUpperCase(), + + // Add derived fields + formattedValue: `$${row.value.toFixed(2)}`, + isHighValue: row.value > 1000, + + // Format dates + date: new Date(row.timestamp).toISOString().split('T')[0] + })); + } +); +``` + + +```python filename="PostProcessingExample.py" copy +from datetime import datetime +from moose_lib import Api +from pydantic import BaseModel + +class QueryParams(BaseModel): + category: str + max_results: int = 10 + +class ResponseItem(BaseModel): + itemId: int + displayName: str + formattedValue: str + isHighValue: bool + date: str + +def run(client: MooseClient, params: QueryParams): + # 1. Fetch raw data using parameter binding + query = """ + SELECT id, name, value, timestamp + FROM data_table + WHERE category = {category} + LIMIT {limit} + """ + + raw_results = client.query.execute(query, {"category": params.category, "limit": params.max_results}) + + # 2. Post-process the results + processed_results = [] + for row in raw_results: + processed_results.append(ResponseItem( + # Transform field names + itemId=row['id'], + displayName=row['name'].upper(), + + # Add derived fields + formattedValue=f"${row['value']:.2f}", + isHighValue=row['value'] > 1000, + + # Format dates + date=datetime.fromisoformat(row['timestamp']).date().isoformat() + )) + + return processed_results + +# Create the API +process_data_api = Api[QueryParams, ResponseItem](name="process_data_endpoint", query_function=run) +``` + + + +### Best Practices + + + + +While post-processing gives you flexibility, remember that database operations are typically more efficient for heavy data manipulation. Reserve post-processing for transformations that are difficult to express in SQL or that involve application-specific logic. + + +## Client Integration + +By default, all API endpoints are automatically integrated with OpenAPI/Swagger documentation. You can integrate your OpenAPI SDK generator of choice to generate client libraries for your APIs. + +Please refer to the [OpenAPI](/moosestack/apis/openapi-sdk) page for more information on how to integrate your APIs with OpenAPI. diff --git a/apps/framework-docs-v2/content/moosestack/apis/auth.mdx b/apps/framework-docs-v2/content/moosestack/apis/auth.mdx index 29dd4a1452..758038e1ca 100644 --- a/apps/framework-docs-v2/content/moosestack/apis/auth.mdx +++ b/apps/framework-docs-v2/content/moosestack/apis/auth.mdx @@ -1,9 +1,272 @@ --- -title: Auth -description: Authentication for APIs -order: 1 +title: API Authentication & Security +description: Secure your Moose API endpoints with JWT tokens or API keys +order: 3 +category: apis --- -# Auth +import { Callout } from "@/components/mdx"; -This page is a placeholder. Content migration pending. +# API Authentication & Security + +Moose supports two authentication mechanisms for securing your API endpoints: + +- **[API Keys](#api-key-authentication)** - Simple, static authentication for internal applications and getting started +- **[JWT (JSON Web Tokens)](#jwt-authentication)** - Token-based authentication for integration with existing identity providers + +Choose the method that fits your use case, or use both together with custom configuration. + +## Do you want to use API Keys? + +API keys are the simplest way to secure your Moose endpoints. They're ideal for: +- Internal applications and microservices +- Getting started quickly with authentication +- Scenarios where you control both client and server + +### How API Keys Work + +API keys use PBKDF2 HMAC SHA256 hashing for secure storage. You generate a token pair (plain-text and hashed) using the Moose CLI, store the hashed version in environment variables, and send the plain-text version in your request headers. + +### Step 1: Generate API Keys + +Generate tokens and hashed keys using the Moose CLI: + +```bash +moose generate hash-token +``` + +**Output:** +- **ENV API Keys**: Hashed key for environment variables (use this in your server configuration) +- **Bearer Token**: Plain-text token for client applications (use this in `Authorization` headers) + + + Use the **hashed key** for environment variables and `moose.config.toml`. Use the **plain-text token** in your `Authorization: Bearer token` headers. + + +### Step 2: Configure API Keys with Environment Variables + +Set environment variables with the **hashed** API keys you generated: +```bash +# For ingest endpoints +export MOOSE_INGEST_API_KEY='your_pbkdf2_hmac_sha256_hashed_key' + +# For analytics endpoints +export MOOSE_CONSUMPTION_API_KEY='your_pbkdf2_hmac_sha256_hashed_key' + +# For admin endpoints +export MOOSE_ADMIN_TOKEN='your_plain_text_token' +``` + +Or set the admin API key in `moose.config.toml`: + +```toml filename="moose.config.toml" +[authentication] +admin_api_key = "your_pbkdf2_hmac_sha256_hashed_key" +``` + + + Storing the `admin_api_key` (which is a PBKDF2 HMAC SHA256 hash) in your `moose.config.toml` file is an acceptable practice, even if the file is version-controlled. This is because the actual plain-text Bearer token (the secret) is not stored. The hash is computationally expensive to reverse, ensuring that your secret is not exposed in the codebase. + + +### Step 3: Make Authenticated Requests + +All authenticated requests require the `Authorization` header with the **plain-text token**: + +```bash +# Using curl +curl -H "Authorization: Bearer your_plain_text_token_here" \ + https://your-moose-instance.com/ingest/YourDataModel + +# Using JavaScript +fetch('https://your-moose-instance.com/api/endpoint', { + headers: { + 'Authorization': 'Bearer your_plain_text_token_here' + } +}) +``` + +## Do you want to use JWTs? + +JWT authentication integrates with existing identity providers and follows standard token-based authentication patterns. Use JWTs when: +- You have an existing identity provider (Auth0, Okta, etc.) +- You need user-specific authentication and authorization +- You want standard OAuth 2.0 / OpenID Connect flows + +### How JWT Works + +Moose validates JWT tokens using RS256 algorithm with your identity provider's public key. You configure the expected issuer and audience, and Moose handles token verification automatically. + +### Step 1: Configure JWT Settings + +#### Option A: Configure in `moose.config.toml` + +```toml filename=moose.config.toml +[jwt] +# Your JWT public key (PEM-formatted RSA public key) +secret = """ +-----BEGIN PUBLIC KEY----- +MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAy... +-----END PUBLIC KEY----- +""" +# Expected JWT issuer +issuer = "https://my-auth-server.com/" +# Expected JWT audience +audience = "my-moose-app" +``` + + + The `secret` field should contain your JWT **public key** used to verify signatures using RS256 algorithm. + + +#### Option B: Configure with Environment Variables + +You can also set these values as environment variables: + +```bash filename=".env" copy +MOOSE_JWT_PUBLIC_KEY=your_jwt_public_key # PEM-formatted RSA public key (overrides `secret` in `moose.config.toml`) +MOOSE_JWT_ISSUER=your_jwt_issuer # Expected JWT issuer (overrides `issuer` in `moose.config.toml`) +MOOSE_JWT_AUDIENCE=your_jwt_audience # Expected JWT audience (overrides `audience` in `moose.config.toml`) +``` + +### Step 2: Make Authenticated Requests + +Send requests with the JWT token in the `Authorization` header: + +```bash +# Using curl +curl -H "Authorization: Bearer your_jwt_token_here" \ + https://your-moose-instance.com/ingest/YourDataModel + +# Using JavaScript +fetch('https://your-moose-instance.com/api/endpoint', { + headers: { + 'Authorization': 'Bearer your_jwt_token_here' + } +}) +``` + +## Want to use both? Here's the caveats + +You can configure both JWT and API Key authentication simultaneously. When both are configured, Moose's authentication behavior depends on the `enforce_on_all_*` flags. + +### Understanding Authentication Priority + +#### Default Behavior (No Enforcement) + +By default, when both JWT and API Keys are configured, Moose tries JWT validation first, then falls back to API Key validation: + +```toml filename="moose.config.toml" +[jwt] +# JWT configuration +secret = "..." +issuer = "https://my-auth-server.com/" +audience = "my-moose-app" +# enforce flags default to false +``` + +```bash filename=".env" +# API Key configuration +MOOSE_INGEST_API_KEY='your_pbkdf2_hmac_sha256_hashed_key' +MOOSE_CONSUMPTION_API_KEY='your_pbkdf2_hmac_sha256_hashed_key' +``` + +**For Ingest Endpoints (`/ingest/*`)**: +- Attempts JWT validation first (RS256 signature check) +- Falls back to API Key validation (PBKDF2 HMAC SHA256) if JWT fails + +**For Analytics Endpoints (`/api/*`)**: +- Same fallback behavior as ingest endpoints + +This allows you to use either authentication method for your clients. + +#### Enforcing JWT Only + +If you want to **only** accept JWT tokens (no API key fallback), set the enforcement flags: + +```toml filename="moose.config.toml" +[jwt] +secret = "..." +issuer = "https://my-auth-server.com/" +audience = "my-moose-app" +# Only accept JWT, no API key fallback +enforce_on_all_ingest_apis = true +enforce_on_all_consumptions_apis = true +``` + +**Result**: When enforcement is enabled, API Key authentication is disabled even if the environment variables are set. Only valid JWT tokens will be accepted. + +### Common Use Cases + +#### Use Case 1: Different Auth for Different Endpoints + +Configure JWT for user-facing analytics endpoints, API keys for internal ingestion: + +```toml filename="moose.config.toml" +[jwt] +secret = "..." +issuer = "https://my-auth-server.com/" +audience = "my-moose-app" +enforce_on_all_consumptions_apis = true # JWT only for /api/* +enforce_on_all_ingest_apis = false # Allow fallback for /ingest/* +``` + +```bash filename=".env" +MOOSE_INGEST_API_KEY='hashed_key_for_internal_services' +``` + +#### Use Case 2: Migration from API Keys to JWT + +Start with both configured, no enforcement. Gradually migrate clients to JWT. Once complete, enable enforcement: + +```toml filename="moose.config.toml" +[jwt] +secret = "..." +issuer = "https://my-auth-server.com/" +audience = "my-moose-app" +# Start with both allowed during migration +enforce_on_all_ingest_apis = false +enforce_on_all_consumptions_apis = false +# Later, enable to complete migration +# enforce_on_all_ingest_apis = true +# enforce_on_all_consumptions_apis = true +``` + +### Admin Endpoints + +Admin endpoints use API key authentication exclusively (configured separately from ingest/analytics endpoints). + +**Configuration precedence** (highest to lowest): +1. `--token` CLI parameter (plain-text token) +2. `MOOSE_ADMIN_TOKEN` environment variable (plain-text token) +3. `admin_api_key` in `moose.config.toml` (hashed token) + +**Example:** + +```bash +# Option 1: CLI parameter +moose remote plan --token your_plain_text_token + +# Option 2: Environment variable +export MOOSE_ADMIN_TOKEN='your_plain_text_token' +moose remote plan + +# Option 3: Config file +# In moose.config.toml: +# [authentication] +# admin_api_key = "your_pbkdf2_hmac_sha256_hashed_key" +``` + +## Security Best Practices + +- **Never commit plain-text tokens to version control** - Always use hashed keys in configuration files +- **Use environment variables for production** - Keep secrets out of your codebase +- **Generate unique tokens for different environments** - Separate development, staging, and production credentials +- **Rotate tokens regularly** - Especially for long-running production deployments +- **Choose the right method for your use case**: + - Use **API Keys** for internal services and getting started + - Use **JWT** when integrating with identity providers or need user-level auth +- **Store hashed keys safely** - The PBKDF2 HMAC SHA256 hash in `moose.config.toml` is safe to version control, but the plain-text token should only exist in secure environment variables or secret management systems + + + Never commit plain-text tokens to version control. Use hashed keys in configuration files and environment variables for production. + diff --git a/apps/framework-docs-v2/content/moosestack/apis/index.mdx b/apps/framework-docs-v2/content/moosestack/apis/index.mdx index 5f981c124b..72b3c6c8c8 100644 --- a/apps/framework-docs-v2/content/moosestack/apis/index.mdx +++ b/apps/framework-docs-v2/content/moosestack/apis/index.mdx @@ -1,9 +1,150 @@ --- -title: APIs -description: Build type-safe ingestion and query endpoints -order: 1 +title: Moose APIs +description: Create type-safe ingestion and analytics APIs for data access and integration +order: 0 +category: apis --- -# APIs +import { Callout, LanguageTabs, LanguageTabContent, BulletPointsCard, CTACards, CTACard, ExportRequirement } from "@/components/mdx"; -This page is a placeholder. Content migration pending. +# Moose APIs + +## Overview + +The APIs module provides standalone HTTP endpoints for data ingestion and analytics. Unlike other modules of the MooseStack, APIs are meant to be paired with other MooseStack modules like OLAP tables and streams. + + + If you'd prefer to use your own API framework of choice, instead of Moose APIs, see the [Bring Your Own API Framework documentation](/moosestack/app-api-frameworks) for comprehensive examples and patterns using frameworks such as Express, Koa, Fastify, or FastAPI. + + +## Core Capabilities + + + + + + + + +## Basic Examples + + + +### Ingestion API + + + +```ts filename="IngestApi.ts" copy +import { IngestApi } from "@514labs/moose-lib"; + +interface UserEvent { + id: string; + userId: string; + timestamp: Date; + eventType: string; +} + +// Create a standalone ingestion API +export const userEventsApi = new IngestApi("user-events", { + destination: eventStream +}); +``` + + +```py filename="IngestApi.py" copy +from moose_lib import IngestApi +from pydantic import BaseModel +from datetime import datetime + +class UserEvent(BaseModel): + id: str + user_id: str + timestamp: datetime + event_type: str + +# Create a standalone ingestion API +user_events_api = IngestApi[UserEvent]("user-events", IngestConfig(destination=event_stream)) + +# No export needed - Python modules are automatically discovered +``` + + + +### Analytics API + + + +```ts filename="AnalyticsApi.ts" copy +import { Api } from "@514labs/moose-lib"; + +interface Params { + userId: string; + limit: number; +} + +interface ResultData { + id: string; + name: string; + email: string; +} + +// Create a standalone analytics API +export const userDataApi = new Api("user-data", + async ({ userId, limit }, { client, sql }) => { + // Query external service or custom logic + return [ + { id: userId, name: "John Doe", email: "john@example.com" } + ]; + } +); +``` + + +```py filename="AnalyticsApi.py" copy +from moose_lib import Api, MooseClient +from pydantic import BaseModel + +class Params(BaseModel): + user_id: str + limit: int + +class ResultData(BaseModel): + id: str + name: str + email: str + +def query_function(client: MooseClient, params: QueryParams) -> list[UserData]: + # Query external service or custom logic using parameter binding + query = "SELECT * FROM user_data WHERE user_id = {user_id} LIMIT {limit}" + return client.query.execute(query, {"user_id": params.user_id, "limit": params.limit}) + +user_data_api = Api[Params, ResultData]("get-data", query_function) + +# No export needed - Python modules are automatically discovered +``` + + diff --git a/apps/framework-docs-v2/content/moosestack/apis/ingest-api.mdx b/apps/framework-docs-v2/content/moosestack/apis/ingest-api.mdx index 4da44c42ec..eed57b9b5c 100644 --- a/apps/framework-docs-v2/content/moosestack/apis/ingest-api.mdx +++ b/apps/framework-docs-v2/content/moosestack/apis/ingest-api.mdx @@ -1,9 +1,367 @@ --- -title: Ingest New Data -description: Create APIs to ingest new data +title: Ingestion APIs +description: Ingestion APIs for Moose order: 2 +category: apis --- -# Ingest New Data +import { Callout, LanguageTabs, LanguageTabContent, BulletPointsCard, ZoomImg, CompareBulletPointsCard, CTACard, CTACards, Tabs, TabsList, TabsTrigger, TabsContent } from "@/components/mdx"; -This page is a placeholder. Content migration pending. +# Ingestion APIs + +## Overview + +Moose Ingestion APIs are the entry point for getting data into your Moose application. They provide a fast, reliable, and type-safe way to move data from your sources into streams and tables for analytics and processing. + +## When to Use Ingestion APIs + +Ingestion APIs are most useful when you want to implement a push-based pattern for getting data from your data sources into your streams and tables. Common use cases include: +- Instrumenting external client applications +- Receiving webhooks from third-party services +- Integrating with ETL or data pipeline tools that push data + +## Why Use Moose's APIs Over Your Own? + +Moose's ingestion APIs are purpose-built for high-throughput data pipelines, offering key advantages over other more general-purpose frameworks: +- **Built-in schema validation:** Ensures only valid data enters your pipeline. +- **Direct connection to streams/tables:** Instantly link HTTP endpoints to Moose data infrastructure to route incoming data to your streams and tables without any glue code. +- **Dead Letter Queue (DLQ) support:** Invalid records are automatically captured for review and recovery. +- **OpenAPI auto-generation:** Instantly generate client SDKs and docs for all endpoints, including example data. +- **Rust-powered performance:** Far higher throughput and lower latency than typical Node.js or Python APIs. + +## Validation + +Moose validates all incoming data against your interface (TypeScript) or Pydantic model (Python). If a record fails validation, Moose can automatically route it to a Dead Letter Queue (DLQ) for later inspection and recovery. + + + +```typescript filename="ValidationExample.ts" copy +interface ExampleModel { + id: string; + userId: string; + timestamp: Date; + properties?: { + device?: string; + version?: number; + } +} + +export const api = new IngestApi("your-api-route", { + destination: new Stream("your-stream-name"), + deadLetterQueue: new DeadLetterQueue("your-dlq-name") +}); +``` + + +```python filename="ValidationExample.py" copy +from moose_lib import IngestPipeline, IngestPipelineConfig, IngestConfig +from pydantic import BaseModel + +class Properties(BaseModel): + device: Optional[str] + version: Optional[int] + +class ExampleModel(BaseModel): + id: str + userId: str + timestamp: datetime + properties: Properties + +api = IngestApi[ExampleModel]("your-api-route", IngestConfig( + destination=Stream[ExampleModel]("your-stream-name"), + dead_letter_queue=DeadLetterQueue[ExampleModel]("your-dlq-name") +)) +``` + + + + +If your IngestPipeline's schema marks a field as optional but annotates a ClickHouse default, Moose treats: + +- API request and Stream message: field is optional (you may omit it) +- ClickHouse table storage: field is required with a DEFAULT clause + +Behavior: When the API/stream inserts into ClickHouse and the field is missing, ClickHouse sets it to the configured default value. This keeps request payloads simple while avoiding Nullable columns in storage. + +Example: + + +`field?: number & ClickHouseDefault<"18">` or `WithDefault` + + +`Annotated[int, clickhouse_default("18")]` (or equivalent annotation) + + + + + + + Valid Event + Invalid: Missing Field + Invalid: Bad Date + + +Send a valid event - routed to the destination stream + + +```typescript filename="ValidEvent.ts" copy +fetch("http://localhost:4000/ingest/your-api-route", { + method: "POST", + body: JSON.stringify({ + id: "event1", + userId: "user1", + timestamp: "2023-05-10T15:30:00Z" + }) +}) +// ✅ Accepted and routed to the destination stream +// API returns 200 and { success: true } +``` + + +```python filename="ValidEvent.py" copy +requests.post("http://localhost:4000/ingest/your-api-route", json={ + "id": "event1", + "userId": "user1", + "timestamp": "2023-05-10T15:30:00Z" + }) +# ✅ Accepted and routed to the destination stream +# API returns 200 and { success: true } +``` + + + + +Send an invalid event (missing required field) - routed to the DLQ + + +```typescript filename="InvalidEventMissingField.ts" copy +fetch("http://localhost:4000/ingest/your-api-route", { + method: "POST", + body: JSON.stringify({ + id: "event1" + }) +}) +// ❌ Routed to DLQ, because it's missing a required field +// API returns 400 response +``` + + +```python filename="InvalidEventMissingField.py" copy +requests.post("http://localhost:4000/ingest/your-api-route", json={ + "id": "event1", + }) +# ❌ Routed to DLQ, because it's missing a required field +# API returns 400 response +``` + + + + +Send an invalid event (bad date format) - routed to the DLQ + + +```typescript filename="InvalidEventBadDate.ts" copy +fetch("http://localhost:4000/ingest/your-api-route", { + method: "POST", + body: JSON.stringify({ + id: "event1", + userId: "user1", + timestamp: "not-a-date" + }) +}) +// ❌ Routed to DLQ, because the timestamp is not a valid date +// API returns 400 response +``` + + +```python filename="InvalidEventBadDate.py" copy +requests.post("http://localhost:4000/ingest/your-api-route", json={ + "id": "event1", + "userId": "user1", + "timestamp": "not-a-date" + }) +# ❌ Routed to DLQ, because the timestamp is not a valid date +# API returns 400 response +``` + + + + + +## Creating Ingestion APIs + +You can create ingestion APIs in two ways: +- **High-level:** Using the `IngestPipeline` class (recommended for most use cases) +- **Low-level:** Manually configuring the `IngestApi` component for more granular control + +### High-level: IngestPipeline (Recommended) + +The `IngestPipeline` class provides a convenient way to set up ingestion endpoints, streams, and tables with a single declaration: + + + +```typescript filename="AnalyticsPipeline.ts" copy +import { IngestPipeline } from "@514labs/moose-lib"; + +interface ExampleModel { + id: string; + name: string; + value: number; + timestamp: Date; +} + +const examplePipeline = new IngestPipeline("example-name", { + ingestApi: true, // Creates a REST API endpoint + stream: true, // Connects to a stream + table: true +}); +``` + + +```python filename="IngestPipeline.py" copy +from moose_lib import Key, IngestPipeline, IngestPipelineConfig +from pydantic import BaseModel + +class ExampleSchema(BaseModel): + id: Key[str] + name: str + value: int + timestamp: datetime + +example_pipeline = IngestPipeline[ExampleSchema]( + name="example-name", + config=IngestPipelineConfig( + ingest_api=True, + stream=True, + table=True + ) +) +``` + + + +### Low-level: Standalone IngestApi + +For more granular control, you can manually configure the `IngestApi` component: + + + +```typescript filename="AnalyticsPipelineManual.ts" copy +interface ExampleRecord { + id: string; + name: string; + value: number; + timestamp: Date; +} + +// Create the ClickHouse table +const exampleTable = new OlapTable("example-table-name"); + +// Create the stream with specific settings +const exampleStream = new Stream("example-stream-name", { + destination: exampleTable // Connect stream to table +}); + +// Create the ingestion API +const exampleApi = new IngestApi("example-api-route", { + destination: exampleStream, // Connect API to stream +}); +``` + + +```python filename="AnalyticsPipelineManual.py" copy +# Python example would go here +``` + + + + +The types of the destination `Stream` and `Table` must match the type of the `IngestApi`. + + + + +## Configuration Reference + +Configuration options for both high-level and low-level ingestion APIs are provided below. + + + + IngestPipeline + IngestApi + + + + +```typescript filename="IngestPipelineConfig.ts" copy +interface IngestPipelineConfig { + table?: boolean | OlapConfig; + stream?: boolean | Omit, "destination">; + ingestApi?: boolean | Omit, "destination">; + deadLetterQueue?: boolean | Omit, "destination">; + version?: string; + metadata?: { + description?: string; + }; + lifeCycle?: LifeCycle; +} +``` + + +```python filename="IngestPipelineConfig.py" copy +class IngestPipelineConfig(BaseModel): + table: bool | OlapConfig = True + stream: bool | StreamConfig = True + ingest_api: bool | IngestConfig = True + dead_letter_queue: bool | StreamConfig = True + version: Optional[str] = None + metadata: Optional[dict] = None + life_cycle: Optional[LifeCycle] = None +``` + + + + + + +```typescript filename="IngestConfig.ts" copy +interface IngestConfig { + destination: Stream; + deadLetterQueue?: DeadLetterQueue; + version?: string; + metadata?: { + description?: string; + }; +} +``` + + +```python filename="IngestConfig.py" copy +@dataclass +class IngestConfigWithDestination[T: BaseModel]: + destination: Stream[T] + dead_letter_queue: Optional[DeadLetterQueue[T]] = None + version: Optional[str] = None + metadata: Optional[dict] = None +``` + + + + diff --git a/apps/framework-docs-v2/content/moosestack/apis/openapi-sdk.mdx b/apps/framework-docs-v2/content/moosestack/apis/openapi-sdk.mdx index e6ff98309f..e3ac964f23 100644 --- a/apps/framework-docs-v2/content/moosestack/apis/openapi-sdk.mdx +++ b/apps/framework-docs-v2/content/moosestack/apis/openapi-sdk.mdx @@ -1,9 +1,105 @@ --- -title: OpenAPI SDK -description: Generate OpenAPI SDKs for your APIs -order: 5 +title: OpenAPI SDK Generation +description: Generate type-safe client SDKs from your Moose APIs +order: 6 +category: apis --- -# OpenAPI SDK +import { Callout, LanguageTabs, LanguageTabContent, ToggleBlock } from "@/components/mdx"; -This page is a placeholder. Content migration pending. +# OpenAPI SDK Generation + +Moose automatically generates OpenAPI specifications for all your APIs, enabling you to create type-safe client SDKs in any language. This allows you to integrate your Moose APIs into any application with full type safety and IntelliSense support. + +## Overview + +While `moose dev` is running, Moose emits an OpenAPI spec at `.moose/openapi.yaml` covering: + +- **Ingestion endpoints** with request/response schemas +- **Analytics APIs** with query parameters and response types + +Every time you make a change to your Moose APIs, the OpenAPI spec is updated automatically. + +## Generating Typed SDKs from OpenAPI + +You can use your preferred generator to create a client from that spec. Below are minimal, tool-agnostic examples you can copy into your project scripts. + +### Setup + + + +The following example uses Kubb to generate the SDK. Kubb can be installed into your project without any dependencies on the Java runtime (unlike the OpenAPI Generator which requires Java). + +Follow the setup instructions for Kubb [here](https://kubb.dev/docs/getting-started/installation). + +Then, in your project's package.json, add the following script: +```json filename="package.json" copy +{ + "scripts": { + "generate-sdk": "kubb generate" + } +} +``` +Finally, configure the `on_reload_complete_script` hook in your `moose.config.toml`: + +```toml filename="moose.config.toml" copy +[http_server_config] +on_reload_complete_script = "npm run generate-sdk" +``` + +This will trigger the generation CLI command after each reload. + + +The following example uses `openapi-python-client` to generate the SDK. Follow the setup instructions here: [openapi-python-client on PyPI](https://pypi.org/project/openapi-python-client/). + +Add a generation script in your repository: +```bash filename="scripts/generate_python_sdk.sh" copy +#!/usr/bin/env bash +set -euo pipefail +openapi-python-client generate --path .moose/openapi.yaml --output ./generated/python --overwrite +``` + +Then configure Moose to run it after each dev reload: +```toml filename="moose.config.toml" copy +[http_server_config] +on_reload_complete_script = "bash scripts/generate_python_sdk.sh" +``` + +This will regenerate the Python client from the live spec on every reload. + + + +### Hooks for automatic SDK generation + +The `on_reload_complete_script` hook is available in your `moose.config.toml` file. It runs after each dev server reload when code/infra changes have been fully applied. This allows you to keep your SDKs continuously up to date as you make changes to your Moose APIs. + +Notes: +- The script runs in your project root using your `$SHELL` (falls back to `/bin/sh`). +- Paths like `.moose/openapi.yaml` and `./generated/...` are relative to the project root. +- You can combine multiple generators with `&&` (as shown) or split into a shell script if preferred. + + +These hooks only affect local development (`moose dev`). The reload hook runs after Moose finishes applying your changes, ensuring `.moose/openapi.yaml` is fresh before regeneration. + + +## Integration + +Import from the output path your generator writes to (see your chosen example repo). The Moose side is unchanged: the spec lives at `.moose/openapi.yaml` during `moose dev`. + +## Generators + +Use any OpenAPI-compatible generator: + +### TypeScript projects +- [OpenAPI Generator (typescript-fetch)](https://github.com/OpenAPITools/openapi-generator) — mature, broad options; generates Fetch-based client +- [Kubb](https://github.com/kubb-project/kubb) — generates types + fetch client with simple config +- [Orval](https://orval.dev/) — flexible output (client + schemas), good DX +- [openapi-typescript](https://github.com/openapi-ts/openapi-typescript) — generates types only (pair with your own client) +- [swagger-typescript-api](https://github.com/acacode/swagger-typescript-api) — codegen for TS clients from OpenAPI +- [openapi-typescript-codegen](https://github.com/ferdikoomen/openapi-typescript-codegen) — TS client + models +- [oazapfts](https://github.com/oazapfts/oazapfts) — minimal TS client based on fetch +- [openapi-zod-client](https://github.com/astahmer/openapi-zod-client) — Zod schema-first client generation + +### Python projects +- [openapi-python-client](https://pypi.org/project/openapi-python-client/) — modern typed client for OpenAPI 3.0/3.1 +- [OpenAPI Generator (python)](https://github.com/OpenAPITools/openapi-generator) — multiple Python generators (python, python-nextgen) diff --git a/apps/framework-docs-v2/content/moosestack/apis/trigger-api.mdx b/apps/framework-docs-v2/content/moosestack/apis/trigger-api.mdx index f4c9a5a624..5339fd5101 100644 --- a/apps/framework-docs-v2/content/moosestack/apis/trigger-api.mdx +++ b/apps/framework-docs-v2/content/moosestack/apis/trigger-api.mdx @@ -1,9 +1,103 @@ --- -title: Trigger Workflows -description: Trigger workflows via API calls -order: 4 +title: Trigger APIs +description: Create APIs that trigger workflows and other processes +order: 5 +category: apis --- -# Trigger Workflows +import { Callout, LanguageTabs, LanguageTabContent, BulletPointsCard } from "@/components/mdx"; -This page is a placeholder. Content migration pending. +# Trigger APIs + +## Overview + +You can create APIs to initiate workflows, data processing jobs, or other automated processes. + +## Basic Usage + + + +```typescript filename="app/apis/trigger_workflow.ts" copy +import { Api } from "@514labs/moose-lib"; + +interface WorkflowParams { + inputValue: string; + priority?: string; +} + +interface WorkflowResponse { + workflowId: string; + status: string; +} + +const triggerApi = new Api( + "trigger-workflow", + async ({ inputValue, priority = "normal" }: WorkflowParams, { client }) => { + // Trigger the workflow with input parameters + const workflowExecution = await client.workflow.execute("data-processing", { + inputValue, + priority, + triggeredAt: new Date().toISOString() + } + ); + + return { + workflowId: workflowExecution.id, + status: "started" + }; + } +); + +export default triggerApi; +``` + + +```python filename="app/apis/trigger_workflow.py" copy +from moose_lib import MooseClient, Api +from pydantic import BaseModel, Field +from datetime import datetime + +class WorkflowParams(BaseModel): + input_value: str + priority: str = Field(default="normal") + +class WorkflowResponse(BaseModel): + workflow_id: str + status: str + +def run(params: WorkflowParams, client: MooseClient) -> WorkflowResponse: + # Trigger the workflow with input parameters + workflow_execution = client.workflow.execute( + workflow="data-processing", + params={ + "input_value": params.input_value, + "priority": params.priority, + "triggered_at": datetime.now().isoformat() + } + ) + + return WorkflowResponse( + workflow_id=workflow_execution.id, + status="started" + ) + +api = Api[WorkflowParams, WorkflowResponse]("trigger-workflow", run) +``` + + + +## Using the Trigger API + +Once deployed, you can trigger workflows via HTTP requests: + +```bash filename="Terminal" copy +curl "http://localhost:4000/api/trigger-workflow?inputValue=process-user-data&priority=high" +``` + +Response: +```json +{ + "workflowId": "workflow-12345", + "status": "started" +} +``` diff --git a/apps/framework-docs-v2/content/moosestack/app-api-frameworks/express.mdx b/apps/framework-docs-v2/content/moosestack/app-api-frameworks/express.mdx index f43460d0cf..aa6059fb31 100644 --- a/apps/framework-docs-v2/content/moosestack/app-api-frameworks/express.mdx +++ b/apps/framework-docs-v2/content/moosestack/app-api-frameworks/express.mdx @@ -1,14 +1,399 @@ --- -title: Express -description: Integrating MooseStack with Express applications -order: 2 +title: Express with MooseStack +description: Use Express framework with MooseStack +order: 1 category: integration --- -# Express +import { Callout } from "@/components/mdx"; -This page is a placeholder. Content migration pending. +# Express with MooseStack +Mount Express applications within your MooseStack project using the `WebApp` class. Express is the most popular Node.js web framework with a rich ecosystem of middleware. + + - Already run Express elsewhere? Keep it outside your MooseStack project and query data with the MooseStack client. The [Querying Data guide](/moosestack/olap/read-data) shows how to use the SDK. + - Want to mount Express in your MooseStack project? Follow the steps below with `WebApp` for unified deployment and access to MooseStack utilities. + +## Basic Example +```ts filename="app/apis/expressApp.ts" copy +import express from "express"; +import { WebApp, expressMiddleware, getMooseUtils } from "@514labs/moose-lib"; +import { MyTable } from "../tables/MyTable"; + +const app = express(); + +app.use(express.json()); +app.use(expressMiddleware()); // Required for Express + +app.get("/health", (req, res) => { + res.json({ status: "ok" }); +}); + +app.get("/data", async (req, res) => { + const moose = getMooseUtils(req); + if (!moose) { + return res.status(500).json({ error: "Moose utilities not available" }); + } + + const { client, sql } = moose; + const limit = parseInt(req.query.limit as string) || 10; + + try { + const query = sql` + SELECT + ${MyTable.columns.id}, + ${MyTable.columns.name}, + ${MyTable.columns.createdAt} + FROM ${MyTable} + ORDER BY ${MyTable.columns.createdAt} DESC + LIMIT ${limit} + `; + const result = await client.query.execute(query); + const data = await result.json(); + res.json(data); + } catch (error) { + res.status(500).json({ error: String(error) }); + } +}); + +export const expressApp = new WebApp("expressApp", app, { + mountPath: "/express", + metadata: { description: "Express API with custom middleware" } +}); +``` + +**Access your API:** +- `GET http://localhost:4000/express/health` +- `GET http://localhost:4000/express/data?limit=20` + + +Express applications must use `expressMiddleware()` to access Moose utilities: + +```ts +import { expressMiddleware } from "@514labs/moose-lib"; +app.use(expressMiddleware()); +``` + +This middleware injects MooseStack utilities into the request object. + + +## Complete Example with Features + +```ts filename="app/apis/advancedExpressApp.ts" copy +import express, { Request, Response, NextFunction } from "express"; +import { WebApp, expressMiddleware, getMooseUtils } from "@514labs/moose-lib"; +import { UserEvents } from "../tables/UserEvents"; +import { UserProfile } from "../tables/UserProfile"; + +const app = express(); + +// Middleware setup +app.use(express.json()); +app.use(expressMiddleware()); // Required! + +// Custom logging middleware +app.use((req: Request, res: Response, next: NextFunction) => { + console.log(`${req.method} ${req.path}`); + next(); +}); + +// Error handling middleware +const asyncHandler = (fn: Function) => (req: Request, res: Response, next: NextFunction) => { + Promise.resolve(fn(req, res, next)).catch(next); +}; + +// Health check endpoint +app.get("/health", (req, res) => { + res.json({ + status: "ok", + timestamp: new Date().toISOString() + }); +}); + +// GET endpoint with query parameters +app.get("/users/:userId/events", asyncHandler(async (req: Request, res: Response) => { + const moose = getMooseUtils(req); + if (!moose) { + return res.status(500).json({ error: "Moose utilities not available" }); + } + + const { client, sql } = moose; + const { userId } = req.params; + const limit = parseInt(req.query.limit as string) || 10; + const eventType = req.query.eventType as string; + + const cols = UserEvents.columns; + const query = sql` + SELECT + ${cols.id}, + ${cols.event_type}, + ${cols.timestamp} + FROM ${UserEvents} + WHERE ${cols.user_id} = ${userId} + ${eventType ? sql`AND ${cols.event_type} = ${eventType}` : sql``} + ORDER BY ${cols.timestamp} DESC + LIMIT ${limit} + `; + + const result = await client.query.execute(query); + const events = await result.json(); + + res.json({ + userId, + count: events.length, + events + }); +})); + +// POST endpoint +app.post("/users/:userId/profile", asyncHandler(async (req: Request, res: Response) => { + const moose = getMooseUtils(req); + if (!moose) { + return res.status(500).json({ error: "Moose utilities not available" }); + } + + const { userId } = req.params; + const { name, email } = req.body; + + // Validation + if (!name || !email) { + return res.status(400).json({ error: "Name and email are required" }); + } + + // Handle POST logic here + res.json({ + success: true, + userId, + profile: { name, email } + }); +})); + +// Protected endpoint with JWT +app.get("/protected", asyncHandler(async (req: Request, res: Response) => { + const moose = getMooseUtils(req); + + if (!moose?.jwt) { + return res.status(401).json({ error: "Unauthorized" }); + } + + const userId = moose.jwt.sub; + res.json({ + message: "Authenticated", + userId, + claims: moose.jwt + }); +})); + +// Error handling +app.use((err: Error, req: Request, res: Response, next: NextFunction) => { + console.error(err); + res.status(500).json({ + error: "Internal Server Error", + message: err.message + }); +}); + +// Register as WebApp +export const advancedExpressApp = new WebApp("advancedExpress", app, { + mountPath: "/api/v1", + metadata: { + description: "Advanced Express API with routing and middleware" + } +}); +``` + +## WebApp Configuration + +```ts +new WebApp(name, app, config) +``` + +**Parameters:** +- `name` (string): Unique identifier for your WebApp +- `app`: Your Express application instance +- `config` (WebAppConfig): Configuration object + +**WebAppConfig:** +```ts +interface WebAppConfig { + mountPath: string; // Required: URL path (e.g., "/api/v1") + metadata?: { description?: string }; // Optional: Documentation metadata + injectMooseUtils?: boolean; // Optional: Inject utilities (default: true) +} +``` + +## Accessing Moose Utilities + +```ts +import { getMooseUtils } from "@514labs/moose-lib"; + +app.get("/data", async (req, res) => { + const moose = getMooseUtils(req); + if (!moose) { + return res.status(500).json({ error: "Utilities not available" }); + } + + const { client, sql, jwt } = moose; + // Use client and sql for database queries +}); +``` + +**Available utilities:** +- `client`: MooseClient for database queries +- `sql`: Template tag for safe SQL queries +- `jwt`: Parsed JWT payload (when authentication is configured) + +## Middleware Integration + +Express middleware works seamlessly with MooseStack: + +```ts +import helmet from "helmet"; +import compression from "compression"; +import rateLimit from "express-rate-limit"; + +const app = express(); + +// Security +app.use(helmet()); + +// Compression +app.use(compression()); + +// Rate limiting +const limiter = rateLimit({ + windowMs: 15 * 60 * 1000, // 15 minutes + max: 100 // limit each IP to 100 requests per windowMs +}); +app.use(limiter); + +// Body parsing +app.use(express.json()); + +// Moose utilities (must be after body parsing) +app.use(expressMiddleware()); +``` + +## Router Pattern + +Organize routes using Express Router: + +```ts filename="app/apis/routers/usersRouter.ts" +import { Router } from "express"; +import { getMooseUtils } from "@514labs/moose-lib"; +import { UserProfile } from "../../tables/UserProfile"; + +export const usersRouter = Router(); + +usersRouter.get("/:userId", async (req, res) => { + const moose = getMooseUtils(req); + if (!moose) { + return res.status(500).json({ error: "Utilities not available" }); + } + + const { client, sql } = moose; + const { userId } = req.params; + + const query = sql` + SELECT + ${UserProfile.columns.id}, + ${UserProfile.columns.name}, + ${UserProfile.columns.email}, + ${UserProfile.columns.createdAt} + FROM ${UserProfile} + WHERE ${UserProfile.columns.id} = ${userId} + `; + + const result = await client.query.execute(query); + const users = await result.json(); + + if (users.length === 0) { + return res.status(404).json({ error: "User not found" }); + } + + res.json(users[0]); +}); +``` + +```ts filename="app/apis/mainApp.ts" +import express from "express"; +import { WebApp, expressMiddleware } from "@514labs/moose-lib"; +import { usersRouter } from "./routers/usersRouter"; + +const app = express(); + +app.use(express.json()); +app.use(expressMiddleware()); + +// Mount routers +app.use("/users", usersRouter); + +export const mainApp = new WebApp("mainApp", app, { + mountPath: "/api", + metadata: { description: "Main API with routers" } +}); +``` + +## Authentication with JWT + +```ts +app.get("/protected", async (req, res) => { + const moose = getMooseUtils(req); + + if (!moose?.jwt) { + return res.status(401).json({ error: "Unauthorized" }); + } + + const userId = moose.jwt.sub; + const userRole = moose.jwt.role; + + // Check permissions + if (userRole !== "admin") { + return res.status(403).json({ error: "Forbidden" }); + } + + res.json({ message: "Authenticated", userId }); +}); +``` + +See [Authentication documentation](/moosestack/apis/auth) for JWT configuration. + +## Best Practices + +1. **Always use expressMiddleware()**: Required for accessing Moose utilities +2. **Check for moose utilities**: Always verify `getMooseUtils(req)` returns a value +3. **Use async error handling**: Wrap async routes with error handler +4. **Organize with routers**: Split large applications into multiple routers +5. **Apply middleware in order**: Body parsing before expressMiddleware +6. **Use TypeScript types**: Import Request, Response types from Express +7. **Handle errors globally**: Use Express error handling middleware + +## Troubleshooting + +### "Moose utilities not available" + +**Solution:** Ensure `expressMiddleware()` is added after body parsing: +```ts +app.use(express.json()); +app.use(expressMiddleware()); // Must come after body parsers +``` + +### TypeScript errors with getMooseUtils + +**Solution:** The utilities may be undefined, always check: +```ts +const moose = getMooseUtils(req); +if (!moose) { + return res.status(500).json({ error: "Utilities not available" }); +} +// Now moose.client, moose.sql are safely accessible +``` + +### Mount path conflicts + +**Solution:** Ensure your mount path doesn't conflict with reserved paths: +- Avoid: `/api`, `/admin`, `/consumption`, `/health`, `/ingest`, `/mcp` +- Use: `/myapi`, `/v1`, `/custom` diff --git a/apps/framework-docs-v2/content/moosestack/app-api-frameworks/fastapi.mdx b/apps/framework-docs-v2/content/moosestack/app-api-frameworks/fastapi.mdx index 935fe16c04..61da15ade9 100644 --- a/apps/framework-docs-v2/content/moosestack/app-api-frameworks/fastapi.mdx +++ b/apps/framework-docs-v2/content/moosestack/app-api-frameworks/fastapi.mdx @@ -1,14 +1,583 @@ --- -title: FastAPI -description: Integrating MooseStack with FastAPI applications +title: FastAPI with MooseStack +description: Use FastAPI framework with MooseStack order: 3 category: integration --- -# FastAPI +import { Callout } from "@/components/mdx"; -This page is a placeholder. Content migration pending. +# FastAPI with MooseStack +Mount FastAPI applications within your MooseStack project using the `WebApp` class. FastAPI is a modern, fast Python framework with automatic API documentation and async support. + + - Already operating FastAPI outside MooseStack? Keep it separate and call MooseStack data with the client SDK. The [Querying Data guide](/moosestack/olap/read-data) includes Python examples. + - Want to mount FastAPI in your MooseStack project? Use the `WebApp` flow below for unified deployment and access to MooseStack utilities. + +## Basic Example +```python filename="app/apis/fastapi_app.py" copy +from fastapi import FastAPI, Request, HTTPException +from moose_lib.dmv2 import WebApp, WebAppConfig, WebAppMetadata +from moose_lib.dmv2.web_app_helpers import get_moose_utils +from app.tables.my_table import MyTable + +app = FastAPI() + +@app.get("/health") +async def health(): + return {"status": "ok"} + +@app.get("/data") +async def get_data(request: Request, limit: int = 10): + moose = get_moose_utils(request) + if not moose: + raise HTTPException( + status_code=500, + detail="Moose utilities not available" + ) + + try: + query = f""" + SELECT + {MyTable.columns.id}, + {MyTable.columns.name}, + {MyTable.columns.created_at} + FROM {MyTable} + ORDER BY {MyTable.columns.created_at} DESC + LIMIT {{limit}} + """ + + result = moose.client.query.execute_raw(query, { + "limit": limit + }) + + return {"success": True, "data": result} + except Exception as error: + raise HTTPException(status_code=500, detail=str(error)) + +# Register as WebApp +fastapi_app = WebApp( + "fastApiApp", + app, + WebAppConfig( + mount_path="/fastapi", + metadata=WebAppMetadata(description="FastAPI application") + ) +) +``` + +**Access your API:** +- `GET http://localhost:4000/fastapi/health` +- `GET http://localhost:4000/fastapi/data?limit=20` + +## Complete Example with Features + +```python filename="app/apis/advanced_fastapi_app.py" copy +from fastapi import FastAPI, Request, HTTPException, Depends, BackgroundTasks +from fastapi.responses import JSONResponse +from fastapi.middleware.cors import CORSMiddleware +from moose_lib.dmv2 import WebApp, WebAppConfig, WebAppMetadata +from moose_lib.dmv2.web_app_helpers import get_moose_utils, ApiUtil +from app.tables.user_events import UserEvents +from app.tables.user_profile import UserProfile +from pydantic import BaseModel, Field +from datetime import datetime +from typing import Optional + +app = FastAPI( + title="Advanced API", + description="Advanced FastAPI application with MooseStack", + version="1.0.0" +) + +# CORS middleware +app.add_middleware( + CORSMiddleware, + allow_origins=["*"], + allow_credentials=True, + allow_methods=["*"], + allow_headers=["*"], +) + +# Custom middleware +@app.middleware("http") +async def log_requests(request: Request, call_next): + start_time = datetime.now() + response = await call_next(request) + duration = (datetime.now() - start_time).total_seconds() + print(f"{request.method} {request.url.path} - {duration:.3f}s") + return response + +# Request/Response models +class EventQuery(BaseModel): + limit: int = Field(10, ge=1, le=100) + event_type: Optional[str] = None + +class EventData(BaseModel): + event_type: str = Field(..., min_length=1) + data: dict + +class EventResponse(BaseModel): + id: str + event_type: str + timestamp: datetime + +# Health check +@app.get("/health") +async def health(): + return { + "status": "ok", + "timestamp": datetime.now().isoformat() + } + +# GET with path and query parameters +@app.get("/users/{user_id}/events", response_model=dict) +async def get_user_events( + request: Request, + user_id: str, + limit: int = 10, + event_type: Optional[str] = None +): + moose = get_moose_utils(request) + if not moose: + raise HTTPException(status_code=500, detail="Moose utilities not available") + + query = """ + SELECT + id, + event_type, + timestamp + FROM {table} + WHERE user_id = {user_id} + {event_filter} + ORDER BY timestamp DESC + LIMIT {limit} + """ + + event_filter = "AND event_type = {event_type}" if event_type else "" + params = { + "table": UserEvents, + "user_id": user_id, + "limit": limit + } + if event_type: + params["event_type"] = event_type + + try: + result = moose.client.query.execute( + query.format(event_filter=event_filter), + params + ) + return { + "user_id": user_id, + "count": len(result), + "events": result + } + except Exception as error: + raise HTTPException(status_code=500, detail=str(error)) + +# POST with validated body +@app.post("/users/{user_id}/events", status_code=201) +async def create_event( + request: Request, + user_id: str, + body: EventData, + background_tasks: BackgroundTasks +): + moose = get_moose_utils(request) + if not moose: + raise HTTPException(status_code=500, detail="Moose utilities not available") + + # Background task + def log_event_creation(user_id: str, event_type: str): + print(f"Event created: {user_id} - {event_type}") + + background_tasks.add_task(log_event_creation, user_id, body.event_type) + + return { + "success": True, + "user_id": user_id, + "event_type": body.event_type, + "data": body.data + } + +# Protected endpoint with dependency injection +async def require_auth(request: Request) -> ApiUtil: + moose = get_moose_utils(request) + if not moose or not moose.jwt: + raise HTTPException(status_code=401, detail="Unauthorized") + return moose + +@app.get("/protected") +async def protected(moose: ApiUtil = Depends(require_auth)): + return { + "message": "Authenticated", + "user": moose.jwt.get("sub"), + "claims": moose.jwt + } + +# Admin endpoint with role check +async def require_admin(moose: ApiUtil = Depends(require_auth)) -> ApiUtil: + role = moose.jwt.get("role") + if role != "admin": + raise HTTPException(status_code=403, detail="Forbidden") + return moose + +@app.get("/admin/stats") +async def admin_stats(moose: ApiUtil = Depends(require_admin)): + return {"message": "Admin access granted"} + +# WebSocket support +from fastapi import WebSocket + +@app.websocket("/ws") +async def websocket_endpoint(websocket: WebSocket): + await websocket.accept() + try: + while True: + data = await websocket.receive_text() + await websocket.send_text(f"Echo: {data}") + except Exception: + await websocket.close() + +# Global exception handler +@app.exception_handler(Exception) +async def global_exception_handler(request: Request, exc: Exception): + return JSONResponse( + status_code=500, + content={"error": "Internal Server Error", "message": str(exc)} + ) + +# Register as WebApp +fastapi_app = WebApp( + "advancedFastApi", + app, + WebAppConfig( + mount_path="/api/v1", + metadata=WebAppMetadata( + description="Advanced FastAPI application with middleware and dependencies" + ) + ) +) +``` + +## WebApp Configuration + +```python +WebApp(name, app, config) +``` + +**Parameters:** +- `name` (str): Unique identifier for your WebApp +- `app` (FastAPI): Your FastAPI application instance +- `config` (WebAppConfig): Configuration object + +**WebAppConfig:** +```python +@dataclass +class WebAppConfig: + mount_path: str # Required: URL path + metadata: Optional[WebAppMetadata] = None # Optional: Documentation + inject_moose_utils: bool = True # Optional: Inject utilities + +@dataclass +class WebAppMetadata: + description: Optional[str] = None +``` + +## Accessing Moose Utilities + +### Direct Access + +```python +from moose_lib.dmv2.web_app_helpers import get_moose_utils + +@app.get("/data") +async def get_data(request: Request): + moose = get_moose_utils(request) + if not moose: + raise HTTPException(status_code=500, detail="Utilities not available") + + client = moose.client + jwt = moose.jwt +``` + +### Dependency Injection + + +Use FastAPI's dependency injection for cleaner code: + +```python +from moose_lib.dmv2.web_app_helpers import get_moose_dependency, ApiUtil +from fastapi import Depends + +@app.get("/data") +async def get_data(moose: ApiUtil = Depends(get_moose_dependency())): + # moose is automatically injected and guaranteed to exist + result = moose.client.query.execute(...) + return result +``` + + +## Middleware + +FastAPI middleware works seamlessly: + +```python +from fastapi import FastAPI +from fastapi.middleware.cors import CORSMiddleware +from fastapi.middleware.gzip import GZipMiddleware +from starlette.middleware.sessions import SessionMiddleware + +app = FastAPI() + +# CORS +app.add_middleware( + CORSMiddleware, + allow_origins=["https://example.com"], + allow_credentials=True, + allow_methods=["*"], + allow_headers=["*"], +) + +# Compression +app.add_middleware(GZipMiddleware, minimum_size=1000) + +# Sessions +app.add_middleware(SessionMiddleware, secret_key="your-secret-key") + +# Custom middleware +@app.middleware("http") +async def add_custom_header(request: Request, call_next): + response = await call_next(request) + response.headers["X-Custom-Header"] = "Value" + return response +``` + +## Dependency Patterns + +### Reusable Dependencies + +```python +from fastapi import Depends, HTTPException +from typing import Optional + +# Auth dependency +async def get_current_user(moose: ApiUtil = Depends(get_moose_dependency())) -> dict: + if not moose.jwt: + raise HTTPException(status_code=401, detail="Not authenticated") + + user_id = moose.jwt.get("sub") + # Fetch user from database + return {"id": user_id} + +# Admin dependency +async def require_admin_user(user: dict = Depends(get_current_user)) -> dict: + if user.get("role") != "admin": + raise HTTPException(status_code=403, detail="Not authorized") + return user + +# Use dependencies +@app.get("/user/profile") +async def get_profile(user: dict = Depends(get_current_user)): + return user + +@app.get("/admin/dashboard") +async def admin_dashboard(user: dict = Depends(require_admin_user)): + return {"message": f"Welcome admin {user['id']}"} +``` + +### Dependency Classes + +```python +from fastapi import Depends +from app.tables.user_table import UserTable + +class Pagination: + def __init__(self, page: int = 1, size: int = 10): + self.page = page + self.size = size + self.skip = (page - 1) * size + +@app.get("/users") +async def list_users( + pagination: Pagination = Depends(), + moose: ApiUtil = Depends(get_moose_dependency()) +): + # Using parameterized query for user input values + query = f""" + SELECT + {UserTable.columns.id}, + {UserTable.columns.name}, + {UserTable.columns.email} + FROM {UserTable} + WHERE {UserTable.columns.status} = 'active' + LIMIT {{size:UInt32}} + OFFSET {{skip:UInt32}} + """ + result = moose.client.query.execute_raw(query, { + "size": pagination.size, + "skip": pagination.skip + }) + return result +``` + +## Request Validation + +FastAPI uses Pydantic for powerful validation: + +```python +from pydantic import BaseModel, Field, validator +from datetime import datetime +from typing import Optional, Literal + +class UserEventCreate(BaseModel): + event_type: Literal["click", "view", "purchase"] + timestamp: datetime = Field(default_factory=datetime.now) + properties: dict = Field(default_factory=dict) + value: Optional[float] = Field(None, ge=0, le=1000000) + + @validator('properties') + def validate_properties(cls, v): + if len(v) > 50: + raise ValueError('Too many properties') + return v + +@app.post("/events") +async def create_event( + event: UserEventCreate, + moose: ApiUtil = Depends(get_moose_dependency()) +): + # event is fully validated + return {"success": True, "event": event} +``` + +## Background Tasks + +```python +from fastapi import BackgroundTasks + +def send_notification(user_id: str, message: str): + # Expensive operation + print(f"Sending notification to {user_id}: {message}") + +@app.post("/notify/{user_id}") +async def notify_user( + user_id: str, + message: str, + background_tasks: BackgroundTasks +): + # Add task to run after response is sent + background_tasks.add_task(send_notification, user_id, message) + return {"message": "Notification queued"} +``` + +## Authentication with JWT + +```python +# Manual check +@app.get("/protected") +async def protected(request: Request): + moose = get_moose_utils(request) + if not moose or not moose.jwt: + raise HTTPException(status_code=401, detail="Unauthorized") + + user_id = moose.jwt.get("sub") + return {"message": "Authenticated", "user_id": user_id} + +# Dependency pattern (recommended) +async def require_auth(request: Request) -> ApiUtil: + moose = get_moose_utils(request) + if not moose or not moose.jwt: + raise HTTPException(status_code=401, detail="Unauthorized") + return moose + +@app.get("/protected") +async def protected(moose: ApiUtil = Depends(require_auth)): + return {"message": "Authenticated", "user": moose.jwt.get("sub")} +``` + +See [Authentication documentation](/moosestack/apis/auth) for JWT configuration. + +## Best Practices + +1. **Use dependency injection**: Leverage `Depends()` for cleaner code +2. **Validate with Pydantic**: Use `BaseModel` and `Field()` for validation +3. **Use response models**: Specify `response_model` for automatic validation +4. **Handle async properly**: Use `async def` for I/O operations +5. **Add type hints**: FastAPI uses types for validation and documentation +6. **Use background tasks**: For operations that don't need to complete before response +7. **Document with docstrings**: FastAPI includes docstrings in OpenAPI docs +8. **Organize with routers**: Split large applications into APIRouter instances + +## Router Organization + +```python filename="app/apis/routers/users.py" +from fastapi import APIRouter, Depends +from moose_lib.dmv2.web_app_helpers import get_moose_dependency, ApiUtil + +router = APIRouter(prefix="/users", tags=["users"]) + +@router.get("/{user_id}") +async def get_user( + user_id: str, + moose: ApiUtil = Depends(get_moose_dependency()) +): + return {"user_id": user_id} +``` + +```python filename="app/apis/main_app.py" +from fastapi import FastAPI +from moose_lib.dmv2 import WebApp, WebAppConfig +from .routers import users + +app = FastAPI() + +# Include routers +app.include_router(users.router) + +webapp = WebApp("mainApp", app, WebAppConfig(mount_path="/api")) +``` + +## Troubleshooting + +### "Moose utilities not available" + +**Solution:** Verify `inject_moose_utils` is enabled (default): +```python +WebAppConfig(mount_path="/myapi", inject_moose_utils=True) +``` + +### Routes not responding + +**Solution:** Pass the FastAPI app instance, not a router: +```python +# Correct +app = FastAPI() +webapp = WebApp("name", app, config) + +# Incorrect +router = APIRouter() +webapp = WebApp("name", router, config) # Won't work +``` + +### Dependency injection errors + +**Solution:** Ensure dependencies return correct types: +```python +# Correct +async def get_moose(request: Request) -> ApiUtil: + moose = get_moose_utils(request) + if not moose: + raise HTTPException(500, "Not available") + return moose # Return ApiUtil + +# Usage +@app.get("/") +async def handler(moose: ApiUtil = Depends(get_moose)): + # moose is ApiUtil type + pass +``` diff --git a/apps/framework-docs-v2/content/moosestack/app-api-frameworks/fastify.mdx b/apps/framework-docs-v2/content/moosestack/app-api-frameworks/fastify.mdx new file mode 100644 index 0000000000..2d173f39bf --- /dev/null +++ b/apps/framework-docs-v2/content/moosestack/app-api-frameworks/fastify.mdx @@ -0,0 +1,388 @@ +--- +title: Fastify with MooseStack +description: Use Fastify framework with MooseStack +order: 2 +category: integration +--- + +import { Callout } from "@/components/mdx"; + +# Fastify with MooseStack + +Mount Fastify applications within your MooseStack project using the `WebApp` class. Fastify is a fast and low overhead web framework with powerful schema-based validation. + + + - Already running Fastify elsewhere? Keep it outside your MooseStack project and query data with the MooseStack client. The [Querying Data guide](/moosestack/olap/read-data) walks through the SDK. + - Want to mount Fastify in your MooseStack project? Follow the setup below with `WebApp` for unified deployment and access to MooseStack utilities. + + +## Basic Example + +```ts filename="app/apis/fastifyApp.ts" copy +import Fastify from "fastify"; +import { WebApp, getMooseUtils } from "@514labs/moose-lib"; +import { MyTable } from "../tables/MyTable"; + +const app = Fastify({ logger: true }); + +app.get("/health", async (request, reply) => { + return { status: "ok" }; +}); + +app.get("/data", async (request, reply) => { + const moose = getMooseUtils(request.raw); + if (!moose) { + reply.code(500); + return { error: "Moose utilities not available" }; + } + + const { client, sql } = moose; + const limit = parseInt((request.query as any).limit || "10"); + + try { + const query = sql` + SELECT + ${MyTable.columns.id}, + ${MyTable.columns.name}, + ${MyTable.columns.createdAt} + FROM ${MyTable} + ORDER BY ${MyTable.columns.createdAt} DESC + LIMIT ${limit} + `; + const result = await client.query.execute(query); + return await result.json(); + } catch (error) { + reply.code(500); + return { error: String(error) }; + } +}); + +// Must call ready() before passing to WebApp +await app.ready(); + +export const fastifyApp = new WebApp("fastifyApp", app, { + mountPath: "/fastify", + metadata: { description: "Fastify API" } +}); +``` + +**Access your API:** +- `GET http://localhost:4000/fastify/health` +- `GET http://localhost:4000/fastify/data?limit=20` + + +Fastify apps must call `.ready()` before passing to WebApp. + + +## Complete Example with Schema Validation + +```ts filename="app/apis/advancedFastifyApp.ts" copy +import Fastify, { FastifyRequest } from "fastify"; +import { WebApp, getMooseUtils } from "@514labs/moose-lib"; +import { UserEvents } from "../tables/UserEvents"; + +const app = Fastify({ + logger: true, + ajv: { + customOptions: { + removeAdditional: "all", + coerceTypes: true + } + } +}); + +// Schema definitions +const getUserEventsSchema = { + querystring: { + type: "object", + properties: { + limit: { type: "integer", minimum: 1, maximum: 100, default: 10 }, + eventType: { type: "string" } + } + }, + params: { + type: "object", + required: ["userId"], + properties: { + userId: { type: "string", pattern: "^[a-zA-Z0-9-]+$" } + } + }, + response: { + 200: { + type: "object", + properties: { + userId: { type: "string" }, + count: { type: "integer" }, + events: { type: "array" } + } + } + } +}; + +// GET with schema validation +app.get<{ + Params: { userId: string }; + Querystring: { limit?: number; eventType?: string }; +}>("/users/:userId/events", { + schema: getUserEventsSchema +}, async (request, reply) => { + const moose = getMooseUtils(request.raw); + if (!moose) { + reply.code(500); + return { error: "Moose utilities not available" }; + } + + const { client, sql } = moose; + const { userId } = request.params; + const { limit = 10, eventType } = request.query; + + const cols = UserEvents.columns; + const query = sql` + SELECT + ${cols.id}, + ${cols.event_type}, + ${cols.timestamp} + FROM ${UserEvents} + WHERE ${cols.user_id} = ${userId} + ${eventType ? sql`AND ${cols.event_type} = ${eventType}` : sql``} + ORDER BY ${cols.timestamp} DESC + LIMIT ${limit} + `; + + const result = await client.query.execute(query); + const events = await result.json(); + + return { + userId, + count: events.length, + events + }; +}); + +// POST with schema validation +const createEventSchema = { + body: { + type: "object", + required: ["eventType", "data"], + properties: { + eventType: { type: "string", minLength: 1 }, + data: { type: "object" } + } + } +}; + +app.post<{ + Params: { userId: string }; + Body: { eventType: string; data: object }; +}>("/users/:userId/events", { + schema: createEventSchema +}, async (request, reply) => { + const { userId } = request.params; + const { eventType, data } = request.body; + + // Handle POST logic + return { + success: true, + userId, + eventType, + data + }; +}); + +// Protected route with JWT +app.get("/protected", async (request, reply) => { + const moose = getMooseUtils(request.raw); + + if (!moose?.jwt) { + reply.code(401); + return { error: "Unauthorized" }; + } + + return { + message: "Authenticated", + userId: moose.jwt.sub + }; +}); + +// Error handler +app.setErrorHandler((error, request, reply) => { + request.log.error(error); + reply.code(500).send({ + error: "Internal Server Error", + message: error.message + }); +}); + +await app.ready(); + +export const advancedFastifyApp = new WebApp("advancedFastify", app, { + mountPath: "/api/v1", + metadata: { + description: "Advanced Fastify API with schema validation" + } +}); +``` + +## Accessing Moose Utilities + +Use `request.raw` to access the underlying Node.js request: + +```ts +const moose = getMooseUtils(request.raw); +if (!moose) { + reply.code(500); + return { error: "Utilities not available" }; +} + +const { client, sql, jwt } = moose; +``` + +## Plugins and Decorators + +Fastify plugins work seamlessly: + +```ts +import Fastify from "fastify"; +import cors from "@fastify/cors"; +import helmet from "@fastify/helmet"; +import rateLimit from "@fastify/rate-limit"; +import { getMooseUtils } from "@514labs/moose-lib"; +import { MyTable } from "../tables/MyTable"; + +const app = Fastify({ logger: true }); + +// CORS +await app.register(cors, { + origin: true +}); + +// Security headers +await app.register(helmet); + +// Rate limiting +await app.register(rateLimit, { + max: 100, + timeWindow: "15 minutes" +}); + +// Custom decorator +app.decorate("utility", { + formatResponse: (data: any) => ({ + success: true, + timestamp: new Date().toISOString(), + data + }) +}); + +app.get("/data", async (request, reply) => { + const moose = getMooseUtils(request.raw); + if (!moose) { + reply.code(500); + return { error: "Utilities not available" }; + } + + const { client, sql } = moose; + const result = await client.query.execute(sql` + SELECT + ${MyTable.columns.id}, + ${MyTable.columns.name}, + ${MyTable.columns.status} + FROM ${MyTable} + WHERE ${MyTable.columns.status} = 'active' + LIMIT 10 + `); + const data = await result.json(); + + return app.utility.formatResponse(data); +}); + +await app.ready(); +``` + +## Type-Safe Routes + +Leverage TypeScript for type-safe routes: + +```ts +interface UserQueryParams { + limit?: number; + offset?: number; + status?: "active" | "inactive"; +} + +interface UserResponse { + id: string; + name: string; + email: string; +} + +app.get<{ + Querystring: UserQueryParams; + Reply: UserResponse[] +}>("/users", async (request, reply) => { + const { limit = 10, offset = 0, status } = request.query; + + // TypeScript knows the shape of query params + const moose = getMooseUtils(request.raw); + // ... query logic + + // Return type is checked + return [ + { id: "1", name: "John", email: "john@example.com" } + ]; +}); +``` + +## WebApp Configuration + +```ts +new WebApp(name, app, config) +``` + +**WebAppConfig:** +```ts +interface WebAppConfig { + mountPath: string; + metadata?: { description?: string }; + injectMooseUtils?: boolean; // default: true +} +``` + +## Best Practices + +1. **Call .ready() before WebApp**: Always await `app.ready()` before creating WebApp +2. **Use request.raw for utilities**: Access Moose utilities via `getMooseUtils(request.raw)` +3. **Define schemas**: Use Fastify's JSON Schema validation for request/response +4. **Type your routes**: Use TypeScript generics for type-safe route handlers +5. **Leverage plugins**: Use Fastify's rich plugin ecosystem +6. **Handle errors**: Use `setErrorHandler` for global error handling +7. **Enable logging**: Use Fastify's built-in logger for debugging + +## Troubleshooting + +### "Moose utilities not available" + +**Solution:** Use `request.raw` to access the underlying request: +```ts +const moose = getMooseUtils(request.raw); // Not request! +``` + +### App not responding after mounting + +**Solution:** Ensure you called `.ready()`: +```ts +await app.ready(); // Must call before WebApp +export const fastifyApp = new WebApp("name", app, config); +``` + +### Schema validation errors + +**Solution:** Match your TypeScript types with JSON schemas: +```ts +// JSON Schema +{ querystring: { type: "object", properties: { limit: { type: "integer" } } } } + +// TypeScript type +interface Query { limit?: number } +``` + diff --git a/apps/framework-docs-v2/content/moosestack/app-api-frameworks/index.mdx b/apps/framework-docs-v2/content/moosestack/app-api-frameworks/index.mdx index 3a85dfafd4..f8c0b51e25 100644 --- a/apps/framework-docs-v2/content/moosestack/app-api-frameworks/index.mdx +++ b/apps/framework-docs-v2/content/moosestack/app-api-frameworks/index.mdx @@ -1,14 +1,199 @@ --- -title: App / API frameworks -description: Integrating MooseStack with your application and API frameworks -order: 2 +title: Bring Your Own API Framework +description: Use Express, Koa, Fastify, or FastAPI with MooseStack +order: 0 category: integration --- -# App / API frameworks +import { Callout, LanguageTabs, LanguageTabContent } from "@/components/mdx"; -This page is a placeholder. Content migration pending. +# Bring Your Own API Framework +MooseStack provides flexible approaches for building HTTP APIs to expose your data. You can use Moose's built-in [`Api` class](/moosestack/apis/analytics-api) for simple GET endpoints, or bring popular web frameworks like [Express](/moosestack/app-api-frameworks/express), [Koa](/moosestack/app-api-frameworks/typescript-koa), [Fastify](/moosestack/app-api-frameworks/typescript-fastify), or [FastAPI](/moosestack/app-api-frameworks/fastapi) for advanced features and custom middleware. +These framework integrations give you two complementary integration paths: +- **Embed supported frameworks inside your MooseStack project.** Mount Express, Koa, Fastify, or raw Node.js with `WebApp` so MooseStack deploys everything together and injects auth, `client`, and `sql` helpers on every request. +- **Call MooseStack from any framework using the client.** Keep an existing app (Next.js, Rails, etc.) in its current runtime and use the MooseStack TypeScript or Python client to query ClickHouse or trigger workflows. +## Overview + +### Native APIs + +Moose's built-in [`Api` class](/moosestack/apis/analytics-api) provides simple GET endpoints with automatic type validation and OpenAPI documentation. This is ideal for straightforward data queries without complex routing or middleware requirements. + +**Use Native `Api` class when you need:** +- Simple GET endpoints with query parameters +- Automatic type validation and conversion +- Automatic OpenAPI documentation generation +- Minimal setup and boilerplate + +### Embed a supported framework inside a MooseStack project + +Use the `WebApp` class to mount your framework in your MooseStack project alongside tables, streams, and workflows. + +**Why embed the framework:** +- One deployment pipeline managed by the MooseStack CLI (dev hot reload + production config) +- Access MooseStack utilities (`client`, `sql`, `jwt`) in every handler through `getMooseUtils` +- Optionally share auth, logging, and observability defaults with the rest of your MooseStack modules +- Compose MooseStack APIs and framework routes under the same hostname + +### Use the MooseStack client from any framework + +If you already have an application server, call MooseStack directly with the client SDKs—no MooseStack deployment required. Works great for Next.js, Rails, Python microservices, or anything else that can make HTTP requests. + +Here's the client pattern: + + + +```ts filename="lib/moose.ts" copy +import { MooseClient, sql } from "@514labs/moose-lib"; +import { UserTable } from "./tables/UserTable"; + +const client = new MooseClient(); + +export async function listUsers(limit = 25) { + const result = await client.query.execute( + sql` + SELECT + ${UserTable.columns.id}, + ${UserTable.columns.name}, + ${UserTable.columns.email} + FROM ${UserTable} + WHERE ${UserTable.columns.status} = 'active' + LIMIT ${limit} + ` + ); + return await result.json(); +} +``` + + +```py filename="lib/moose.py" copy +from moose_lib import MooseClient +from app.tables.user_table import UserTable + +client = MooseClient() + +def list_users(limit: int = 25): + query = f""" + SELECT + {UserTable.columns.id}, + {UserTable.columns.name}, + {UserTable.columns.email} + FROM {UserTable} + WHERE {UserTable.columns.status} = 'active' + LIMIT {{limit:UInt32}} + """ + return client.query.execute_raw(query, {"limit": limit}) +``` + + + + + For more examples, see the [Querying Data guide](/moosestack/olap/read-data). + + +## Supported Frameworks + + + For basic GET endpoints that query your OLAP tables, MooseStack provides built-in [Analytics APIs](/moosestack/apis/analytics-api) using the `Api` class. + + + + `WebApp` currently ships adapters for [Express](/moosestack/app-api-frameworks/express), [Koa](/moosestack/app-api-frameworks/koa), [Fastify](/moosestack/app-api-frameworks/fastify), [raw Node.js](/moosestack/app-api-frameworks/raw-nodejs), and [FastAPI](/moosestack/app-api-frameworks/fastapi). Frameworks like Next.js can call MooseStack today through the client pattern above. + + +### TypeScript + +
+ +
Express
+
+ Most popular Node.js web framework +
+ + + +
Fastify
+
+ Fast and low overhead web framework +
+ + + +
Koa
+
+ Expressive middleware framework by Express team +
+ + + +
Raw Node.js
+
+ Use raw HTTP handlers without any framework +
+ +
+ +### Python + +
+ +
FastAPI
+
+ Modern, fast Python web framework +
+ +
+ +## Key Concepts + +### WebApp Class + +The `WebApp` class is the bridge between your web framework and MooseStack. It handles: + +- Mounting your framework at a custom URL path +- Injecting MooseStack utilities (database clients, SQL helpers, JWT) +- Validating JWT tokens when authentication is configured +- Managing the request/response lifecycle + +### Accessing Data + +All frameworks can access your OLAP database through MooseStack's injected utilities: + + + +```ts +import { getMooseUtils } from "@514labs/moose-lib"; + +const moose = getMooseUtils(req); +const { client, sql, jwt } = moose; +``` + + +```python +from moose_lib.dmv2.web_app_helpers import get_moose_utils + +moose = get_moose_utils(request) +client = moose.client +jwt = moose.jwt +``` + + + +### Mount Paths + +Each WebApp must specify a unique mount path where it will be accessible. Mount paths have specific rules: + +- Cannot be `/` (root path) +- Cannot end with a trailing slash +- Cannot start with reserved paths: `/admin`, `/api`, `/consumption`, `/health`, `/ingest`, `/mcp`, `/moose`, `/ready`, `/workflows` + +**Valid:** `/myapi`, `/v1/analytics`, `/custom/endpoint` + +**Invalid:** `/`, `/myapi/`, `/api/myendpoint` + + + MooseStack's native `Api` class provides built-in validation and OpenAPI documentation. See the [Analytics API documentation](/moosestack/apis/analytics-api) for a guided walkthrough. + diff --git a/apps/framework-docs-v2/content/moosestack/app-api-frameworks/koa.mdx b/apps/framework-docs-v2/content/moosestack/app-api-frameworks/koa.mdx new file mode 100644 index 0000000000..6e2965e265 --- /dev/null +++ b/apps/framework-docs-v2/content/moosestack/app-api-frameworks/koa.mdx @@ -0,0 +1,451 @@ +--- +title: Koa with MooseStack +description: Use Koa framework with MooseStack +order: 4 +category: integration +--- + +import { Callout } from "@/components/mdx"; + +# Koa with MooseStack + +Mount Koa applications within your MooseStack project using the `WebApp` class. Koa is an expressive, minimalist framework by the Express team, designed for modern async/await patterns. + + + - Already running Koa outside MooseStack? Keep it separate and call MooseStack data with the client SDK. The [Querying Data guide](/moosestack/olap/read-data) has TypeScript examples. + - Want to mount Koa in your MooseStack project? Continue below with `WebApp` for unified deployment and access to MooseStack utilities. + + +## Basic Example + +```ts filename="app/apis/koaApp.ts" copy +import Koa from "koa"; +import Router from "@koa/router"; +import bodyParser from "koa-bodyparser"; +import { WebApp, getMooseUtils } from "@514labs/moose-lib"; +import { MyTable } from "../tables/MyTable"; + +const app = new Koa(); +const router = new Router(); + +app.use(bodyParser()); + +router.get("/health", (ctx) => { + ctx.body = { status: "ok" }; +}); + +router.get("/data", async (ctx) => { + const moose = getMooseUtils(ctx.req); + if (!moose) { + ctx.status = 500; + ctx.body = { error: "Moose utilities not available" }; + return; + } + + const { client, sql } = moose; + const limit = parseInt((ctx.query.limit as string) || "10"); + + try { + const query = sql` + SELECT + ${MyTable.columns.id}, + ${MyTable.columns.name}, + ${MyTable.columns.createdAt} + FROM ${MyTable} + ORDER BY ${MyTable.columns.createdAt} DESC + LIMIT ${limit} + `; + const result = await client.query.execute(query); + ctx.body = await result.json(); + } catch (error) { + ctx.status = 500; + ctx.body = { error: String(error) }; + } +}); + +app.use(router.routes()); +app.use(router.allowedMethods()); + +export const koaApp = new WebApp("koaApp", app, { + mountPath: "/koa", + metadata: { description: "Koa API" } +}); +``` + +**Access your API:** +- `GET http://localhost:4000/koa/health` +- `GET http://localhost:4000/koa/data?limit=20` + +## Complete Example with Middleware + +```ts filename="app/apis/advancedKoaApp.ts" copy +import Koa, { Context, Next } from "koa"; +import Router from "@koa/router"; +import bodyParser from "koa-bodyparser"; +import logger from "koa-logger"; +import { WebApp, getMooseUtils } from "@514labs/moose-lib"; +import { UserEvents } from "../tables/UserEvents"; +import { UserProfile } from "../tables/UserProfile"; + +const app = new Koa(); +const router = new Router(); + +// Middleware +app.use(logger()); +app.use(bodyParser()); + +// Custom error handling middleware +app.use(async (ctx: Context, next: Next) => { + try { + await next(); + } catch (error) { + ctx.status = error.status || 500; + ctx.body = { + error: error.message || "Internal Server Error" + }; + ctx.app.emit("error", error, ctx); + } +}); + +// Custom logging middleware +app.use(async (ctx: Context, next: Next) => { + const start = Date.now(); + await next(); + const ms = Date.now() - start; + console.log(`${ctx.method} ${ctx.url} - ${ms}ms`); +}); + +// Health check +router.get("/health", (ctx) => { + ctx.body = { + status: "ok", + timestamp: new Date().toISOString() + }; +}); + +// GET with params and query +router.get("/users/:userId/events", async (ctx) => { + const moose = getMooseUtils(ctx.req); + if (!moose) { + ctx.throw(500, "Moose utilities not available"); + } + + const { client, sql } = moose; + const { userId } = ctx.params; + const limit = parseInt((ctx.query.limit as string) || "10"); + const eventType = ctx.query.eventType as string; + + const cols = UserEvents.columns; + const query = sql` + SELECT + ${cols.id}, + ${cols.event_type}, + ${cols.timestamp} + FROM ${UserEvents} + WHERE ${cols.user_id} = ${userId} + ${eventType ? sql`AND ${cols.event_type} = ${eventType}` : sql``} + ORDER BY ${cols.timestamp} DESC + LIMIT ${limit} + `; + + const result = await client.query.execute(query); + const events = await result.json(); + + ctx.body = { + userId, + count: events.length, + events + }; +}); + +// POST endpoint +router.post("/users/:userId/events", async (ctx) => { + const moose = getMooseUtils(ctx.req); + if (!moose) { + ctx.throw(500, "Moose utilities not available"); + } + + const { userId } = ctx.params; + const { eventType, data } = ctx.request.body as any; + + // Validation + if (!eventType || !data) { + ctx.throw(400, "eventType and data are required"); + } + + // Handle POST logic + ctx.body = { + success: true, + userId, + eventType, + data + }; + ctx.status = 201; +}); + +// Protected route with JWT +router.get("/protected", async (ctx) => { + const moose = getMooseUtils(ctx.req); + + if (!moose?.jwt) { + ctx.throw(401, "Unauthorized"); + } + + const userId = moose.jwt.sub; + const userRole = moose.jwt.role; + + ctx.body = { + message: "Authenticated", + userId, + role: userRole + }; +}); + +// Multiple route handlers (middleware chain) +const checkAuth = async (ctx: Context, next: Next) => { + const moose = getMooseUtils(ctx.req); + if (!moose?.jwt) { + ctx.throw(401, "Unauthorized"); + } + await next(); +}; + +router.get("/admin/stats", checkAuth, async (ctx) => { + const moose = getMooseUtils(ctx.req); + // moose.jwt is guaranteed to exist here + ctx.body = { stats: "admin stats" }; +}); + +app.use(router.routes()); +app.use(router.allowedMethods()); + +// Error listener +app.on("error", (err, ctx) => { + console.error("Server error:", err); +}); + +export const advancedKoaApp = new WebApp("advancedKoa", app, { + mountPath: "/api/v1", + metadata: { + description: "Advanced Koa API with middleware chain" + } +}); +``` + +## Accessing Moose Utilities + +Use `ctx.req` to access the underlying Node.js request: + +```ts +const moose = getMooseUtils(ctx.req); +if (!moose) { + ctx.throw(500, "Utilities not available"); +} + +const { client, sql, jwt } = moose; +``` + +## Middleware Patterns + +### Composition + +```ts +import compose from "koa-compose"; + +const authMiddleware = async (ctx: Context, next: Next) => { + const moose = getMooseUtils(ctx.req); + if (!moose?.jwt) { + ctx.throw(401, "Unauthorized"); + } + await next(); +}; + +const adminMiddleware = async (ctx: Context, next: Next) => { + const moose = getMooseUtils(ctx.req); + if (!moose?.jwt || moose.jwt.role !== "admin") { + ctx.throw(403, "Forbidden"); + } + await next(); +}; + +// Compose middleware +const requireAdmin = compose([authMiddleware, adminMiddleware]); + +router.get("/admin", requireAdmin, async (ctx) => { + ctx.body = { message: "Admin access granted" }; +}); +``` + +### Custom Context Extensions + +```ts +import Koa, { Context } from "koa"; +import { MyTable } from "../tables/MyTable"; + +// Extend context type +interface CustomContext extends Context { + formatResponse: (data: any) => { success: boolean; data: any }; +} + +const app = new Koa(); + +// Add custom method to context +app.context.formatResponse = function(data: any) { + return { + success: true, + timestamp: new Date().toISOString(), + data + }; +}; + +router.get("/data", async (ctx: CustomContext) => { + const moose = getMooseUtils(ctx.req); + if (!moose) { + ctx.throw(500, "Utilities not available"); + } + + const { client, sql } = moose; + const result = await client.query.execute(sql` + SELECT + ${MyTable.columns.id}, + ${MyTable.columns.name}, + ${MyTable.columns.status} + FROM ${MyTable} + WHERE ${MyTable.columns.status} = 'active' + LIMIT 10 + `); + const data = await result.json(); + + ctx.body = ctx.formatResponse(data); +}); +``` + +## Error Handling + +Koa uses try-catch for error handling: + +```ts +// Error middleware +app.use(async (ctx, next) => { + try { + await next(); + } catch (err) { + // Custom error handling + ctx.status = err.statusCode || err.status || 500; + ctx.body = { + error: { + message: err.message, + status: ctx.status + } + }; + + // Emit error event + ctx.app.emit("error", err, ctx); + } +}); + +// Error listener +app.on("error", (err, ctx) => { + console.error("Error:", err); +}); +``` + +## Router Nesting + +Organize routes with nested routers: + +```ts filename="app/apis/routers/usersRouter.ts" +import Router from "@koa/router"; +import { getMooseUtils } from "@514labs/moose-lib"; + +export const usersRouter = new Router({ prefix: "/users" }); + +usersRouter.get("/:userId", async (ctx) => { + const moose = getMooseUtils(ctx.req); + if (!moose) { + ctx.throw(500, "Utilities not available"); + } + + const { userId } = ctx.params; + // Query logic + ctx.body = { userId }; +}); +``` + +```ts filename="app/apis/mainApp.ts" +import Koa from "koa"; +import Router from "@koa/router"; +import { WebApp } from "@514labs/moose-lib"; +import { usersRouter } from "./routers/usersRouter"; + +const app = new Koa(); +const mainRouter = new Router(); + +// Nest routers +mainRouter.use("/api", usersRouter.routes(), usersRouter.allowedMethods()); + +app.use(mainRouter.routes()); +app.use(mainRouter.allowedMethods()); + +export const mainApp = new WebApp("mainApp", app, { + mountPath: "/v1", + metadata: { description: "Main API with nested routers" } +}); +``` + +## WebApp Configuration + +```ts +new WebApp(name, app, config) +``` + +**WebAppConfig:** +```ts +interface WebAppConfig { + mountPath: string; + metadata?: { description?: string }; + injectMooseUtils?: boolean; // default: true +} +``` + +## Best Practices + +1. **Use ctx.req for utilities**: Access Moose utilities via `getMooseUtils(ctx.req)` +2. **Use ctx.throw()**: Koa's built-in error throwing for cleaner code +3. **Leverage async/await**: Koa is designed for modern async patterns +4. **Compose middleware**: Use `koa-compose` for reusable middleware chains +5. **Handle errors globally**: Use error middleware at the top of middleware stack +6. **Type your context**: Extend Context type for custom properties +7. **Organize with routers**: Split large applications into nested routers + +## Troubleshooting + +### "Moose utilities not available" + +**Solution:** Use `ctx.req` not `ctx.request`: +```ts +const moose = getMooseUtils(ctx.req); // Correct +const moose = getMooseUtils(ctx.request); // Wrong +``` + +### Middleware order issues + +**Solution:** Apply middleware in correct order: +```ts +app.use(logger()); // 1. Logging +app.use(bodyParser()); // 2. Body parsing +app.use(errorHandler); // 3. Error handling +app.use(router.routes()); // 4. Routes +``` + +### TypeScript errors with Context + +**Solution:** Import and use correct types: +```ts +import { Context, Next } from "koa"; + +router.get("/path", async (ctx: Context, next: Next) => { + // ctx is properly typed +}); +``` + diff --git a/apps/framework-docs-v2/content/moosestack/app-api-frameworks/raw-nodejs.mdx b/apps/framework-docs-v2/content/moosestack/app-api-frameworks/raw-nodejs.mdx new file mode 100644 index 0000000000..e3153ce47b --- /dev/null +++ b/apps/framework-docs-v2/content/moosestack/app-api-frameworks/raw-nodejs.mdx @@ -0,0 +1,495 @@ +--- +title: Raw Node.js with MooseStack +description: Use raw Node.js HTTP handlers with MooseStack +order: 5 +category: integration +--- + +import { Callout } from "@/components/mdx"; + +# Raw Node.js with MooseStack + +Use raw Node.js HTTP handlers without any framework. This gives you maximum control and minimal dependencies, ideal for performance-critical applications or when you want to avoid framework overhead. + + + - Running standalone Node.js elsewhere? Keep it separate and query Moose with the client SDK—see the [Querying Data guide](/moosestack/olap/read-data) for examples. + - Want to mount your raw handlers in your MooseStack project? Follow the `WebApp` approach below so your endpoints deploy alongside the rest of your Moose project. + + +## Basic Example + +```ts filename="app/apis/rawApp.ts" copy +import { WebApp, getMooseUtils } from "@514labs/moose-lib"; +import { MyTable } from "../tables/MyTable"; +import { IncomingMessage, ServerResponse } from "http"; +import { parse as parseUrl } from "url"; + +const handler = async (req: IncomingMessage, res: ServerResponse) => { + const url = parseUrl(req.url || "", true); + const pathname = url.pathname || "/"; + + if (pathname === "/health" && req.method === "GET") { + res.writeHead(200, { "Content-Type": "application/json" }); + res.end(JSON.stringify({ status: "ok" })); + return; + } + + if (pathname === "/data" && req.method === "GET") { + const moose = getMooseUtils(req); + if (!moose) { + res.writeHead(500, { "Content-Type": "application/json" }); + res.end(JSON.stringify({ error: "Moose utilities not available" })); + return; + } + + const { client, sql } = moose; + const limit = parseInt((url.query.limit as string) || "10"); + + try { + const query = sql` + SELECT + ${MyTable.columns.id}, + ${MyTable.columns.name}, + ${MyTable.columns.createdAt} + FROM ${MyTable} + ORDER BY ${MyTable.columns.createdAt} DESC + LIMIT ${limit} + `; + const result = await client.query.execute(query); + const data = await result.json(); + + res.writeHead(200, { "Content-Type": "application/json" }); + res.end(JSON.stringify(data)); + } catch (error) { + res.writeHead(500, { "Content-Type": "application/json" }); + res.end(JSON.stringify({ error: String(error) })); + } + return; + } + + res.writeHead(404, { "Content-Type": "application/json" }); + res.end(JSON.stringify({ error: "Not found" })); +}; + +export const rawApp = new WebApp("rawApp", handler, { + mountPath: "/raw", + metadata: { description: "Raw Node.js handler" } +}); +``` + +**Access your API:** +- `GET http://localhost:4000/raw/health` +- `GET http://localhost:4000/raw/data?limit=20` + +## Complete Example with Advanced Features + +```ts filename="app/apis/advancedRawApp.ts" copy +import { WebApp, getMooseUtils } from "@514labs/moose-lib"; +import { UserEvents } from "../tables/UserEvents"; +import { IncomingMessage, ServerResponse } from "http"; +import { parse as parseUrl } from "url"; + +// Helper to parse request body +const parseBody = (req: IncomingMessage): Promise => { + return new Promise((resolve, reject) => { + let body = ""; + req.on("data", chunk => body += chunk.toString()); + req.on("end", () => { + try { + resolve(body ? JSON.parse(body) : {}); + } catch (error) { + reject(new Error("Invalid JSON")); + } + }); + req.on("error", reject); + }); +}; + +// Helper to send JSON response +const sendJSON = (res: ServerResponse, status: number, data: any) => { + res.writeHead(status, { + "Content-Type": "application/json", + "X-Powered-By": "MooseStack" + }); + res.end(JSON.stringify(data)); +}; + +// Helper to send error +const sendError = (res: ServerResponse, status: number, message: string) => { + sendJSON(res, status, { error: message }); +}; + +// Route handlers +const handleHealth = (req: IncomingMessage, res: ServerResponse) => { + sendJSON(res, 200, { + status: "ok", + timestamp: new Date().toISOString() + }); +}; + +const handleGetUserEvents = async ( + req: IncomingMessage, + res: ServerResponse, + userId: string, + query: any +) => { + const moose = getMooseUtils(req); + if (!moose) { + return sendError(res, 500, "Moose utilities not available"); + } + + const { client, sql } = moose; + const limit = parseInt(query.limit || "10"); + const eventType = query.eventType; + + try { + const cols = UserEvents.columns; + const querySQL = sql` + SELECT + ${cols.id}, + ${cols.event_type}, + ${cols.timestamp} + FROM ${UserEvents} + WHERE ${cols.user_id} = ${userId} + ${eventType ? sql`AND ${cols.event_type} = ${eventType}` : sql``} + ORDER BY ${cols.timestamp} DESC + LIMIT ${limit} + `; + + const result = await client.query.execute(querySQL); + const events = await result.json(); + + sendJSON(res, 200, { + userId, + count: events.length, + events + }); + } catch (error) { + sendError(res, 500, String(error)); + } +}; + +const handleCreateEvent = async ( + req: IncomingMessage, + res: ServerResponse, + userId: string +) => { + try { + const body = await parseBody(req); + const { eventType, data } = body; + + if (!eventType || !data) { + return sendError(res, 400, "eventType and data are required"); + } + + // Handle POST logic + sendJSON(res, 201, { + success: true, + userId, + eventType, + data + }); + } catch (error) { + sendError(res, 400, "Invalid request body"); + } +}; + +const handleProtected = (req: IncomingMessage, res: ServerResponse) => { + const moose = getMooseUtils(req); + + if (!moose?.jwt) { + return sendError(res, 401, "Unauthorized"); + } + + sendJSON(res, 200, { + message: "Authenticated", + userId: moose.jwt.sub, + claims: moose.jwt + }); +}; + +// Main handler with routing +const handler = async (req: IncomingMessage, res: ServerResponse) => { + const url = parseUrl(req.url || "", true); + const pathname = url.pathname || "/"; + const method = req.method || "GET"; + + // CORS headers + res.setHeader("Access-Control-Allow-Origin", "*"); + res.setHeader("Access-Control-Allow-Methods", "GET, POST, OPTIONS"); + res.setHeader("Access-Control-Allow-Headers", "Content-Type, Authorization"); + + // Handle preflight + if (method === "OPTIONS") { + res.writeHead(204); + res.end(); + return; + } + + // Route matching + if (pathname === "/health" && method === "GET") { + return handleHealth(req, res); + } + + // Match /users/:userId/events + const userEventsMatch = pathname.match(/^\/users\/([^\/]+)\/events$/); + if (userEventsMatch) { + const userId = userEventsMatch[1]; + + if (method === "GET") { + return handleGetUserEvents(req, res, userId, url.query); + } + + if (method === "POST") { + return handleCreateEvent(req, res, userId); + } + + return sendError(res, 405, "Method not allowed"); + } + + if (pathname === "/protected" && method === "GET") { + return handleProtected(req, res); + } + + // 404 + sendError(res, 404, "Not found"); +}; + +export const advancedRawApp = new WebApp("advancedRaw", handler, { + mountPath: "/api/v1", + metadata: { + description: "Advanced raw Node.js handler with routing" + } +}); +``` + +## Pattern Matching for Routes + +```ts +// Simple pattern matching +const matchRoute = (pathname: string, pattern: string): { [key: string]: string } | null => { + const patternParts = pattern.split("/"); + const pathParts = pathname.split("/"); + + if (patternParts.length !== pathParts.length) { + return null; + } + + const params: { [key: string]: string } = {}; + + for (let i = 0; i < patternParts.length; i++) { + if (patternParts[i].startsWith(":")) { + const paramName = patternParts[i].slice(1); + params[paramName] = pathParts[i]; + } else if (patternParts[i] !== pathParts[i]) { + return null; + } + } + + return params; +}; + +// Usage +const handler = async (req: IncomingMessage, res: ServerResponse) => { + const url = parseUrl(req.url || "", true); + const pathname = url.pathname || "/"; + + const userParams = matchRoute(pathname, "/users/:userId"); + if (userParams) { + const { userId } = userParams; + // Handle user route + return; + } + + const eventParams = matchRoute(pathname, "/users/:userId/events/:eventId"); + if (eventParams) { + const { userId, eventId } = eventParams; + // Handle event route + return; + } +}; +``` + +## Streaming Responses + +```ts +const handleStreamData = async (req: IncomingMessage, res: ServerResponse) => { + const moose = getMooseUtils(req); + if (!moose) { + return sendError(res, 500, "Utilities not available"); + } + + const { client, sql } = moose; + + res.writeHead(200, { + "Content-Type": "application/x-ndjson", + "Transfer-Encoding": "chunked" + }); + + const query = sql` + SELECT + ${MyTable.columns.id}, + ${MyTable.columns.name}, + ${MyTable.columns.data} + FROM ${MyTable} + ORDER BY ${MyTable.columns.createdAt} DESC + LIMIT 1000 + `; + const result = await client.query.execute(query); + const data = await result.json(); + + // Stream data in chunks + for (const row of data) { + res.write(JSON.stringify(row) + "\n"); + await new Promise(resolve => setTimeout(resolve, 10)); + } + + res.end(); +}; +``` + +## WebApp Configuration + +```ts +new WebApp(name, handler, config) +``` + +**WebAppConfig:** +```ts +interface WebAppConfig { + mountPath: string; + metadata?: { description?: string }; + injectMooseUtils?: boolean; // default: true +} +``` + +## Accessing Moose Utilities + +```ts +const moose = getMooseUtils(req); +if (!moose) { + res.writeHead(500, { "Content-Type": "application/json" }); + res.end(JSON.stringify({ error: "Utilities not available" })); + return; +} + +const { client, sql, jwt } = moose; +``` + +## Best Practices + +1. **Parse URL with url module**: Use `url.parse()` for query parameters and pathname +2. **Set Content-Type headers**: Always set appropriate response headers +3. **Handle errors gracefully**: Wrap async operations in try-catch +4. **Use helper functions**: Extract common patterns (sendJSON, parseBody) +5. **Implement routing logic**: Use pattern matching for dynamic routes +6. **Handle CORS**: Set CORS headers if needed for browser clients +7. **Stream large responses**: Use chunked encoding for large datasets + +## Middleware Pattern + +Create your own middleware pattern: + +```ts +type Middleware = ( + req: IncomingMessage, + res: ServerResponse, + next: () => Promise +) => Promise; + +const createMiddlewareChain = (...middlewares: Middleware[]) => { + return async (req: IncomingMessage, res: ServerResponse) => { + let index = 0; + + const next = async (): Promise => { + if (index < middlewares.length) { + const middleware = middlewares[index++]; + await middleware(req, res, next); + } + }; + + await next(); + }; +}; + +// Example middleware +const loggerMiddleware: Middleware = async (req, res, next) => { + console.log(`${req.method} ${req.url}`); + await next(); +}; + +const authMiddleware: Middleware = async (req, res, next) => { + const moose = getMooseUtils(req); + if (!moose?.jwt) { + sendError(res, 401, "Unauthorized"); + return; + } + await next(); +}; + +const routeHandler: Middleware = async (req, res, next) => { + sendJSON(res, 200, { message: "Success" }); +}; + +// Create handler with middleware chain +const handler = createMiddlewareChain( + loggerMiddleware, + authMiddleware, + routeHandler +); +``` + +## When to Use Raw Node.js + +**Ideal for:** +- Maximum control over request/response +- Performance-critical applications +- Minimal dependencies +- Custom protocols or streaming +- Learning HTTP fundamentals + +**Not ideal for:** +- Rapid development (frameworks are faster) +- Complex routing (use Express/Koa instead) +- Large teams (frameworks provide structure) +- Standard REST APIs (frameworks have better DX) + +## Troubleshooting + +### Response not closing + +**Solution:** Always call `res.end()`: +```ts +res.writeHead(200, { "Content-Type": "application/json" }); +res.end(JSON.stringify(data)); // Don't forget this! +``` + +### Query parameters not parsing + +**Solution:** Use `url.parse()` with `true` for query parsing: +```ts +import { parse as parseUrl } from "url"; +const url = parseUrl(req.url || "", true); // true enables query parsing +const limit = url.query.limit; +``` + +### POST body not available + +**Solution:** Manually parse the request stream: +```ts +const parseBody = (req: IncomingMessage): Promise => { + return new Promise((resolve, reject) => { + let body = ""; + req.on("data", chunk => body += chunk.toString()); + req.on("end", () => { + try { + resolve(body ? JSON.parse(body) : {}); + } catch (error) { + reject(new Error("Invalid JSON")); + } + }); + req.on("error", reject); + }); +}; +``` + diff --git a/apps/framework-docs-v2/content/moosestack/changelog.mdx b/apps/framework-docs-v2/content/moosestack/changelog.mdx index e06aa72bd1..3621fb2c8e 100644 --- a/apps/framework-docs-v2/content/moosestack/changelog.mdx +++ b/apps/framework-docs-v2/content/moosestack/changelog.mdx @@ -4,8 +4,486 @@ description: MooseStack changelog and release notes order: 1 --- -# Changelog +import { + ReleaseHighlights, + Added, + Changed, + Deprecated, + Fixed, + Security, + BreakingChanges, +} from "@/components/mdx"; -This page is a placeholder. Content migration pending. +# Moose Changelog +## What is this page? +This changelog tracks all meaningful changes to Moose. Each entry includes the PR link and contributor credit, organized by date (newest first). Use this page to stay informed about new features, fixes, and breaking changes that might affect your projects. +## How to read this changelog: + +
+ + +Key features, enhancements, or fixes for each release. + + + +New features and capabilities. + + + +Updates to existing functionality or improvements. + + + +Features that are no longer recommended for use and may be removed in the future. + + + +Bug fixes and reliability improvements. + + + +Changes that require user action or may break existing usage. + +
+ +--- + +## 2025-08-21 + + + - **Analytics API Standardization** — Standardized naming of classes and utilities for analytics APIs + - **Complete S3Queue Engine Support** — Full implementation of ClickHouse S3Queue engine with comprehensive parameter support, modular architecture, and generic settings framework. + + + + - **S3Queue Engine Configuration** — Added complete support for all ClickHouse S3Queue engine parameters including authentication (`aws_access_key_id`, `aws_secret_access_key`), compression, custom headers, and NOSIGN for public buckets. [PR #2674](https://github.com/514-labs/moosestack/pull/2674) + - **Generic Settings Framework** — Introduced a flexible settings system that allows any engine to use configuration settings, laying groundwork for future engine implementations. + - **Enhanced Documentation** — Added comprehensive documentation for OlapTable S3Queue configuration in both TypeScript and Python SDKs. + + + + - **Improved Architecture** — Moved ClickHouse-specific types from core infrastructure to ClickHouse module for better separation of concerns. + - **Settings Location** — Engine-specific settings are now properly encapsulated within their respective engine configurations (e.g., `s3QueueEngineConfig.settings` for S3Queue). + - **API Consistency** — Unified configuration APIs across TypeScript and Python SDKs for S3Queue engine. + + + + - **Compilation Issues** — Fixed struct patterns to handle new S3Queue parameter structure correctly. + - **Diff Strategy** — Enhanced diff strategy to properly handle S3Queue parameter changes. + + + + - Renamed `ConsumptionApi` to `Api` + - Renamed `EgressConfig` to `ApiConfig` + - Renamed `ConsumptionUtil` to `ApiUtil` + - Renamed `ConsumptionHelpers` to `ApiHelpers` + + *[#2676](https://github.com/514-labs/moosestack/pull/2676) by [camelCasedAditya](https://github.com/camelCasedAditya)* + + +--- + +## 2025-08-20 + + + - **Improved IngestPipeline API Clarity** — The confusing `ingest` parameter has been renamed to `ingestApi` (TypeScript) and `ingest_api` (Python) for better clarity. The old parameter names are still supported with deprecation warnings. + + + + - **IngestPipeline Parameter Renamed** — The `ingest` parameter in IngestPipeline configurations has been renamed for clarity: + - **TypeScript**: `ingest: true` → `ingestApi: true` + - **Python**: `ingest=True` → `ingest_api=True` + + The old parameter names continue to work with deprecation warnings to ensure backwards compatibility. *[Current PR]* + + + + - **IngestPipeline `ingest` parameter** — The `ingest` parameter in IngestPipeline configurations is deprecated: + - **TypeScript**: Use `ingestApi` instead of `ingest` + - **Python**: Use `ingest_api` instead of `ingest` + + The old parameter will be removed in a future major version. Please update your code to use the new parameter names. *[Current PR]* + + + + None - Full backwards compatibility maintained + + +--- +## 2025-06-12 + + + - **Enhanced TypeScript Workflow Types** — Improved type safety for Tasks with optional input/output parameters, supporting `null` types for better flexibility. + + + + - TypeScript workflow Task types now properly support optional input/output with `null` types, enabling more flexible task definitions like `Task` and `Task`. + *[#2442](https://github.com/514-labs/moose/pull/2442) by [DatGuyJonathan](https://github.com/DatGuyJonathan)* + + + + None + + +--- + +## 2025-06-10 + + + - **OlapTable Direct Insert API** — New comprehensive insert API with advanced error handling, typia validation, and multiple failure strategies. Enables direct data insertion into ClickHouse tables with production-ready reliability features. + - **Python Workflows V2** — Replaces static file-based routing with explicit `Task` and `Workflow` classes, enabling dynamic task composition and programmatic workflow orchestration. No more reliance on `@task` decorators or file naming conventions. + + + + - OlapTable direct insert API with `insert()` method supporting arrays and Node.js streams. Features comprehensive typia-based validation, three error handling strategies (`fail-fast`, `discard`, `isolate`), configurable error thresholds, memoized ClickHouse connections, and detailed insertion results with failed record tracking. + *[#2437](https://github.com/514-labs/moose/pull/2437) by [callicles](https://github.com/callicles)* + - Enhanced typia validation integration for OlapTable and IngestPipeline with `validateRecord()`, `isValidRecord()`, and `assertValidRecord()` methods providing compile-time type safety and runtime validation. + *[#2437](https://github.com/514-labs/moose/pull/2437) by [callicles](https://github.com/callicles)* + - Python Workflows V2 with `Task[InputType, OutputType]` and `Workflow` classes for dynamic workflow orchestration. Replaces the legacy `@task` decorator approach with explicit task definitions, enabling flexible task composition, type-safe chaining via `on_complete`, retries, timeouts, and scheduling with cron expressions. + *[#2439](https://github.com/514-labs/moose/pull/2439) by [DatGuyJonathan](https://github.com/DatGuyJonathan)* + + + + + None + + +--- + +## 2025-06-06 + + + - **TypeScript Workflows V2** — Replaces static file-based routing with explicit `Task` and `Workflow` classes, enabling dynamic task composition and programmatic workflow orchestration. No more reliance on file naming conventions for task execution order. + + + + - TypeScript Workflows V2 with `Task` and `Workflow` classes for dynamic workflow orchestration. Replaces the legacy file-based routing approach with explicit task definitions, enabling flexible task composition, type-safe chaining via `onComplete`, configurable retries and timeouts, and flexible scheduling with cron expressions. + *[#2421](https://github.com/514-labs/moose/pull/2421) by [DatGuyJonathan](https://github.com/DatGuyJonathan)* + + + + None + + +--- + +## 2025-05-23 + + + - **TypeScript `DeadLetterQueue` support** — Handle failed streaming function messages with type-safe dead letter queues in TypeScript. + - **Improved Python `DeadLetterModel` API** — Renamed `as_t` to `as_typed` for better clarity. + + + + - TypeScript `DeadLetterQueue` class with type guards and transform methods for handling failed streaming function messages with full type safety. + *[#2356](https://github.com/514-labs/moose/pull/2356) by [phiSgr](https://github.com/phiSgr)* + + + + - Renamed `DeadLetterModel.as_t()` to `DeadLetterModel.as_typed()` in Python for better API clarity and consistency. + *[#2356](https://github.com/514-labs/moose/pull/2356) by [phiSgr](https://github.com/phiSgr)* + + + + - `DeadLetterModel.as_t()` method renamed to `as_typed()` in Python. Update your code to use the new method name. + *[#2356](https://github.com/514-labs/moose/pull/2356) by [phiSgr](https://github.com/phiSgr)* + + +--- + +## 2025-05-22 + + + - **Refactored CLI 'peek' command** — Now supports peeking into both tables and streams with unified parameters. + - **Simplified CLI experience** — Removed unused commands and routines for a cleaner interface. + + + + - Updated CLI 'peek' command to use a unified 'name' parameter and new flags (`--table`, `--stream`) to specify resource type. Default is table. Documentation updated to match. + *[#2361](https://github.com/514-labs/moose/pull/2361) by [callicles](https://github.com/callicles)* + + + + - Removed unused CLI commands and routines including `Function`, `Block`, `Consumption`, `DataModel`, and `Import`. CLI is now simpler and easier to maintain. + *[#2360](https://github.com/514-labs/moose/pull/2360) by [callicles](https://github.com/callicles)* + + + + None + + +--- + +## 2025-05-21 + + + - **Infrastructure state sync** — Auto-syncs DB state before changes, handling manual modifications and failed DDL runs. + - **Fixed nested data type support** — Use objects and arrays in your Moose models. + + + + - State reconciliation for infrastructure planning — Moose now checks and updates its in-memory infra map to match the real database state before planning changes. Makes infra planning robust to manual DB changes and failed runs. + *[#2341](https://github.com/514-labs/moose/pull/2341) by [callicles](https://github.com/callicles)* + + + + - Handling of nested data structures in Moose models for correct support of complex objects and arrays. + *[#2357](https://github.com/514-labs/moose/pull/2357) by [georgevanderson](https://github.com/georgevanderson)* + + + + + None + + +--- + +## 2025-05-27 + + + - **IPv4 and IPv6 Type Support** — Added native support for IP address types in ClickHouse data models, enabling efficient storage and querying of network data. + + + + - IPv4 and IPv6 data types for ClickHouse integration, supporting native IP address storage and operations. + *[#2373](https://github.com/514-labs/moose/pull/2373) by [phiSgr](https://github.com/phiSgr)* + - Enhanced type parser to handle IP address types across the Moose ecosystem. + *[#2374](https://github.com/514-labs/moose/pull/2374) by [phiSgr](https://github.com/phiSgr)* + + + + None + + +--- + +## 2025-05-20 + + + - **ClickHouse `Date` type support** — Store and query native date values in your schemas. + + + + - ClickHouse `Date` column support for native date types in Moose schemas and ingestion. + *[#2352](https://github.com/514-labs/moose/pull/2352), [#2351](https://github.com/514-labs/moose/pull/2351) by [phiSgr](https://github.com/phiSgr)* + + + + None + + +--- + +## 2025-05-19 + + + - **Metadata map propagation** — Metadata is now tracked and available in the infra map for both Python and TypeScript. Improves LLM accuracy and reliability when working with Moose objects. + + + + - Metadata map propagation to infra map for consistent tracking and availability in both Python and TypeScript. + *[#2326](https://github.com/514-labs/moose/pull/2326) by [georgevanderson](https://github.com/georgevanderson)* + + + + None + + +--- + +## 2025-05-16 + + + - **New `list[str]` support to Python `AggregateFunction`** — Enables more flexible aggregation logic in Materialized Views. + - **Python `DeadLetterQueue[T]` alpha release** — Automatically route exceptions to a dead letter queue in streaming functions. + + + + - AggregateFunction in Python now accepts `list[str]` for more expressive and type-safe aggregations. + *[#2321](https://github.com/514-labs/moose/pull/2321) by [phiSgr](https://github.com/phiSgr)* + + + + - Python dead letter queues for handling and retrying failed messages in Python streaming functions. + *[#2324](https://github.com/514-labs/moose/pull/2324) by [phiSgr](https://github.com/phiSgr)* + + + + None + + +--- + +## 2025-05-15 + + + - **Hotfix — casing fix for `JSON` columns in TypeScript. + + + + - TypeScript JSON columns to have consistent casing, avoiding confusion and errors in your code. + *[#2320](https://github.com/514-labs/moose/pull/2320) by [phiSgr](https://github.com/phiSgr)* + + + + None + + +--- + +## 2025-05-14 + + + - **Introduced TypeScript JSON columns** — Use `Record` for type-safe JSON fields. + - **Ingestion config simplified** — Less config needed for ingestion setup. + - **Python `enum` support improved** — More robust data models. + + + + - TypeScript ClickHouse JSON columns to use `Record` for type-safe JSON fields. + *[#2317](https://github.com/514-labs/moose/pull/2317) by [phiSgr](https://github.com/phiSgr)* + - Pydantic mixin for parsing integer enums by name for more robust Python data models. + *[#2316](https://github.com/514-labs/moose/pull/2316) by [phiSgr](https://github.com/phiSgr)* + - Better Python enum handling in data models for easier enum usage. + *[#2315](https://github.com/514-labs/moose/pull/2315) by [phiSgr](https://github.com/phiSgr)* + + + + - `IngestionFormat` from `IngestApi` config for simpler ingestion setup. + *[#2306](https://github.com/514-labs/moose/pull/2306) by [georgevanderson](https://github.com/georgevanderson)* + + + + None + + +--- + +## 2025-05-13 + + + - **New `refresh` CLI command** — Quickly reload data and schema changes from changes applied directly to your database outside of Moose. + - **Python: `LowCardinality` type support** — Better performance for categorical data. + + + + - `refresh` command to reload data and schema with a single command. + *[#2309](https://github.com/514-labs/moose/pull/2309) by [phiSgr](https://github.com/phiSgr)* + - Python support for `LowCardinality(T)` to improve performance for categorical columns. + *[#2313](https://github.com/514-labs/moose/pull/2313) by [phiSgr](https://github.com/phiSgr)* + + + + None + + +--- + +## 2025-05-10 + + + - **Dependency-based execution order for Materialized Views** — Reduces migration errors and improves reliability. + + + + - Order changes for materialized views based on dependency to ensure correct execution order for dependent changes. + *[#2294](https://github.com/514-labs/moose/pull/2294) by [callicles](https://github.com/callicles)* + + + + None + + +--- + +## 2025-05-07 + + + - **Python `datetime64` support** - Enables more precise datetime handling in Python data models. + - **Type mapping in Python `QueryClient`** - Automatically maps ClickHouse query result rows to the correct Pydantic model types. + + + + - Row parsing in QueryClient with type mapping for Python. + *[#2299](https://github.com/514-labs/moose/pull/2299) by [phiSgr](https://github.com/phiSgr)* + + + + - `datetime64` parsing and row parsing in QueryClient for more reliable data handling in Python. + *[#2299](https://github.com/514-labs/moose/pull/2299) by [phiSgr](https://github.com/phiSgr)* + + + + None + + +--- + +## 2025-05-06 + + + - **`uint` type support in TypeScript** — Enables type safety for unsigned integer fields in Typescript data models. + + + + - uint type support in TypeScript for unsigned integers in Moose models. + *[#2295](https://github.com/514-labs/moose/pull/2295) by [phiSgr](https://github.com/phiSgr)* + + + + None + + +--- + +## 2025-05-01 + + + - **Explicit dependency tracking for materialized views** — Improves data lineage, migration reliability, and documentation. + + + + - Explicit dependency tracking for materialized views to make migrations and data lineage more robust and easier to understand. + *[#2282](https://github.com/514-labs/moose/pull/2282) by [callicles](https://github.com/callicles)* + + + + - Required `selectTables` field in `MaterializedView` config that must specify an array of `OlapTable` objects for the source tables. + *[#2282](https://github.com/514-labs/moose/pull/2282) by [callicles](https://github.com/callicles)* + + +--- + +## 2025-04-30 + + + - **More flexible `JSON_ARRAY` configuration for `IngestApi`** — Now accepts both arrays and single elements. Default config is now `JSON_ARRAY`. + - **Python rich ClickHouse type support** — Added support for advanced types in Python models: + - `Decimal`: `clickhouse_decimal(precision, scale)` + - `datetime` with precision: `clickhouse_datetime64(precision)` + - `date`: `date` + - `int` with size annotations: `Annotated[int, 'int8']`, `Annotated[int, 'int32']`, etc. + - `UUID`: `UUID` + + + + - `JSON_ARRAY` to allow both array and single element ingestion for more flexible data handling. + *[#2285](https://github.com/514-labs/moose/pull/2285) by [phiSgr](https://github.com/phiSgr)* + + + + - Python rich ClickHouse type support with: + - `Decimal`: `clickhouse_decimal(precision, scale)` + - `datetime` with precision: `clickhouse_datetime64(precision)` + - `date`: `date` + - `int` with size annotations: `Annotated[int, 'int8']`, `Annotated[int, 'int32']`, etc. + - `UUID`: `UUID` + for more expressive data modeling. + *[#2284](https://github.com/514-labs/moose/pull/2284) by [phiSgr](https://github.com/phiSgr)* + + + + None + diff --git a/apps/framework-docs-v2/content/moosestack/configuration.mdx b/apps/framework-docs-v2/content/moosestack/configuration.mdx index f2a0929702..28702959eb 100644 --- a/apps/framework-docs-v2/content/moosestack/configuration.mdx +++ b/apps/framework-docs-v2/content/moosestack/configuration.mdx @@ -4,10 +4,390 @@ description: Configure your MooseStack project order: 1 --- +import { Callout } from "@/components/mdx"; + # Project Configuration -This page is a placeholder. Content migration pending. +Moose provides flexible configuration through multiple sources, allowing you to customize your application for different environments while keeping sensitive data secure. + +## Configuration Precedence + +Moose loads configuration from multiple sources in the following order (highest to lowest priority): + +1. **System environment variables** (`MOOSE_*`) - Highest priority, never overwritten +2. **`.env.local`** - Local development secrets (gitignored, only loaded in dev mode) +3. **`.env.{environment}`** - Environment-specific files (`.env.dev`, `.env.prod`) +4. **`.env`** - Base environment defaults (committed to git) +5. **`moose.config.toml`** - Structured configuration file +6. **Default values** - Built-in defaults from Moose + +This precedence allows you to: +- Store safe defaults in `moose.config.toml` +- Override with environment-specific `.env` files +- Keep local secrets in `.env.local` (development only) +- Use system env vars for production secrets + + +See the configuration loading implementation in the source code: +- dotenv.rs - See `load_dotenv_files` function +- project.rs - See `load` function in the `Project` impl block + + +## Environment Variables & .env Files + +### Overview + +Moose automatically loads `.env` files based on the command you run: + +- `moose dev` → Loads `.env` → `.env.dev` → `.env.local` +- `moose prod` → Loads `.env` → `.env.prod` (`.env.local` is NOT loaded) +- `moose build` → Loads `.env` → `.env.prod` (production mode) + + +`.env.local` is only loaded in development mode. In production, use system environment variables or `.env.prod` for configuration. This prevents accidentally exposing local development credentials in production. + + +### File Naming Convention + +| File | Purpose | Committed? | When Loaded | +|:-----|:--------|:-----------|:------------| +| `.env` | Base defaults for all environments | ✅ Yes | Always | +| `.env.dev` | Development-specific config | ✅ Yes | `moose dev` | +| `.env.prod` | Production-specific config | ✅ Yes | `moose prod`, `moose build` | +| `.env.local` | Local secrets and overrides | ❌ No (gitignored) | `moose dev` only | + +### Setting Up .env Files + +**1. Create a `.env` file** with safe, non-secret defaults: + +```bash +# .env - Committed to git +MOOSE_HTTP_SERVER_CONFIG__PORT=4000 +MOOSE_CLICKHOUSE_CONFIG__DB_NAME=local +``` + +**2. Create environment-specific files:** + +```bash +# .env.dev - Development settings +MOOSE_LOGGER__LEVEL=debug +MOOSE_FEATURES__WORKFLOWS=false +``` + +```bash +# .env.prod - Production settings +MOOSE_LOGGER__LEVEL=info +MOOSE_LOGGER__FORMAT=Json +MOOSE_FEATURES__WORKFLOWS=true +``` + +**3. Create `.env.local`** for your local secrets (gitignored): + +```bash +# .env.local - NOT committed to git +MOOSE_CLICKHOUSE_CONFIG__PASSWORD=my-local-password +MOOSE_REDIS_CONFIG__URL=redis://localhost:6380 +``` + +**4. Update `.gitignore`:** + +```gitignore +# Environment files with secrets +.env.local +``` + +### Environment Variable Naming + +All Moose environment variables use the `MOOSE_` prefix with double underscores (`__`) to separate nested configuration sections: + +```bash +MOOSE_
__=value +``` + +**Examples:** + +| Config in moose.config.toml | Environment Variable | +|:----------------------------|:---------------------| +| `clickhouse_config.host` | `MOOSE_CLICKHOUSE_CONFIG__HOST` | +| `clickhouse_config.port` | `MOOSE_CLICKHOUSE_CONFIG__PORT` | +| `http_server_config.port` | `MOOSE_HTTP_SERVER_CONFIG__PORT` | +| `features.workflows` | `MOOSE_FEATURES__WORKFLOWS` | +| `redis_config.url` | `MOOSE_REDIS_CONFIG__URL` | + +### Complete Example + +**File structure:** +``` +my-moose-project/ +├── .env # Base config +├── .env.dev # Dev overrides +├── .env.prod # Prod overrides +├── .env.local # Local secrets (gitignored) +└── moose.config.toml # Structured config +``` + +**.env** (committed): +```bash +# Base configuration for all environments +MOOSE_HTTP_SERVER_CONFIG__PORT=4000 +MOOSE_CLICKHOUSE_CONFIG__DB_NAME=moose_db +``` + +**.env.dev** (committed): +```bash +# Development environment +MOOSE_LOGGER__LEVEL=debug +MOOSE_CLICKHOUSE_CONFIG__HOST=localhost +MOOSE_REDIS_CONFIG__URL=redis://localhost:6379 +``` + +**.env.prod** (committed): +```bash +# Production environment +MOOSE_LOGGER__LEVEL=info +MOOSE_LOGGER__FORMAT=Json +MOOSE_CLICKHOUSE_CONFIG__USE_SSL=true +``` + +**.env.local** (gitignored): +```bash +# Local development secrets +MOOSE_CLICKHOUSE_CONFIG__PASSWORD=my-dev-password +MOOSE_TEMPORAL_CONFIG__API_KEY=my-temporal-key +``` + +**Usage:** +```bash +# Development - loads .env → .env.dev → .env.local +moose dev + +# Production - loads .env → .env.prod (NOT .env.local) +moose prod +``` + +## moose.config.toml Reference + + +The `moose.config.toml` file is the primary way to configure all Moose infrastructure including ClickHouse, Redpanda, Redis, Temporal, and HTTP servers. + +Do not use docker-compose overrides to modify Moose-managed services. See [Development Mode](/moosestack/local-dev#extending-docker-infrastructure) for guidelines on when to use docker-compose extensions. + + +```toml +# Programming language used in the project (`Typescript` or `Python`) +language = "Typescript" + +# Map of supported old versions and their locations (Default: {}) +# supported_old_versions = { "0.1.0" = "path/to/old/version" } + +#Telemetry configuration for usage tracking and metrics +[telemetry] +# Whether telemetry collection is enabled +enabled = true +# Whether to export metrics to external systems +export_metrics = true +# Flag indicating if the user is a Moose developer +is_moose_developer = false + +# Redpanda streaming configuration (also aliased as `kafka_config`) +[redpanda_config] +# Broker connection string (e.g., "host:port") (Default: "localhost:19092") +broker = "localhost:19092" +# Confluent Schema Registry URL (optional) +# schema_registry_url = "http://localhost:8081" +# Message timeout in milliseconds (Default: 1000) +message_timeout_ms = 1000 +# Default retention period in milliseconds (Default: 30000) +retention_ms = 30000 +# Replication factor for topics (Default: 1) +replication_factor = 1 +# SASL username for authentication (Default: None) +# sasl_username = "user" +# SASL password for authentication (Default: None) +# sasl_password = "password" +# SASL mechanism (e.g., "PLAIN", "SCRAM-SHA-256") (Default: None) +# sasl_mechanism = "PLAIN" +# Security protocol (e.g., "SASL_SSL", "PLAINTEXT") (Default: None) +# security_protocol = "SASL_SSL" +# Namespace for topic isolation (Default: None) +# namespace = "my_namespace" + +# ClickHouse database configuration +[clickhouse_config] +# Database name (Default: "local") +db_name = "local" +# ClickHouse user (Default: "panda") +user = "panda" +# ClickHouse password (Default: "pandapass") +password = "pandapass" +# Whether to use SSL for connection (Default: false) +use_ssl = false +# ClickHouse host (Default: "localhost") +host = "localhost" +# ClickHouse HTTP port (Default: 18123) +host_port = 18123 +# ClickHouse native protocol port (Default: 9000) +native_port = 9000 +# Optional host path to mount as the ClickHouse data volume (uses Docker volume if None) (Default: None) +# host_data_path = "/path/on/host/clickhouse_data" +# Optional list of additional databases to create on startup (Default: []) +# additional_databases = ["analytics", "staging"] + +# HTTP server configuration for local development +[http_server_config] +# Host to bind the webserver to (Default: "localhost") +host = "localhost" +# Port for the main API server (Default: 4000) +port = 4000 +# Port for the management server (Default: 5001) +management_port = 5001 +# Optional path prefix for all routes (Default: None) +# path_prefix = "api" +# Number of worker processes for consumption API cluster (TypeScript only) (Default: Auto-calculated - 70% of CPU cores) +# Python projects always use 1 worker regardless of this setting +# api_workers = 2 + +# Redis configuration +[redis_config] +# Redis connection URL (Default: "redis://127.0.0.1:6379") +url = "redis://127.0.0.1:6379" +# Namespace prefix for all Redis keys (Default: "MS") +key_prefix = "MS" + +# State storage configuration for migrations +[state_config] +# Storage backend: "clickhouse" (default) or "redis" +# - "clickhouse": Store state in ClickHouse _MOOSE_STATE table (requires KeeperMap) +# - "redis": Store state in Redis (best for existing Redis infra or multi-tenant setups) +storage = "clickhouse" + +# Git configuration +[git_config] +# Name of the main branch (Default: "main") +main_branch_name = "main" + +# Temporal workflow configuration +[temporal_config] +# Temporal database user (Default: "temporal") +db_user = "temporal" +# Temporal database password (Default: "temporal") +db_password = "temporal" +# Temporal database port (Default: 5432) +db_port = 5432 +# Temporal server host (Default: "localhost") +temporal_host = "localhost" +# Temporal server port (Default: 7233) +temporal_port = 7233 +# Temporal server scheme - "http" or "https" (Default: auto-detect based on host) +# temporal_scheme = "https" +# Temporal server version (Default: "1.22.3") +temporal_version = "1.22.3" +# Temporal admin tools version (Default: "1.22.3") +admin_tools_version = "1.22.3" +# Temporal UI version (Default: "2.21.3") +ui_version = "2.21.3" +# Temporal UI port (Default: 8080) +ui_port = 8080 +# Temporal UI CORS origins (Default: "http://localhost:3000") +ui_cors_origins = "http://localhost:3000" +# Temporal dynamic config path (Default: "config/dynamicconfig/development-sql.yaml") +config_path = "config/dynamicconfig/development-sql.yaml" +# PostgreSQL version for Temporal database (Default: "13") +postgresql_version = "13" +# Path to Temporal client certificate (mTLS) (Default: "") +client_cert = "" +# Path to Temporal client key (mTLS) (Default: "") +client_key = "" +# Path to Temporal CA certificate (mTLS) (Default: "") +ca_cert = "" +# API key for Temporal Cloud connection (Default: "") +api_key = "" + +# JWT (JSON Web Token) authentication configuration (Optional) +[jwt] +# Enforce JWT on all consumption APIs (Default: false) +enforce_on_all_consumptions_apis = false +# Enforce JWT on all ingestion APIs (Default: false) +enforce_on_all_ingest_apis = false +# Secret key for JWT signing (Required if jwt section is present) +# secret = "your-jwt-secret" +# JWT issuer (Required if jwt section is present) +# issuer = "your-issuer-name" +# JWT audience (Required if jwt section is present) +# audience = "your-audience-name" + +# General authentication configuration +[authentication] +# Optional hashed admin API key for auth (Default: None) +# admin_api_key = "hashed_api_key" + +# Migration configuration +[migration_config] +# Operations to ignore during migration plan generation and drift detection +# Useful for managing TTL changes outside of Moose or when you don't want +# migration failures due to TTL drift +# ignore_operations = ["ModifyTableTtl", "ModifyColumnTtl"] + +# Feature flags +[features] +# Enable the streaming engine (Default: true) +streaming_engine = true +# Enable Temporal workflows (Default: false) +workflows = false +# Enable OLAP database (Default: true) +olap = true +# Enable Analytics APIs server (Default: true) +apis = true +``` + +## Common Environment Variables + +Here are the most commonly used environment variables for overriding `moose.config.toml` settings: + +### HTTP Server +```bash +MOOSE_HTTP_SERVER_CONFIG__HOST=0.0.0.0 +MOOSE_HTTP_SERVER_CONFIG__PORT=4000 +MOOSE_HTTP_SERVER_CONFIG__MANAGEMENT_PORT=5001 +``` + +### ClickHouse +```bash +MOOSE_CLICKHOUSE_CONFIG__HOST=localhost +MOOSE_CLICKHOUSE_CONFIG__PORT=9000 +MOOSE_CLICKHOUSE_CONFIG__DB_NAME=local +MOOSE_CLICKHOUSE_CONFIG__USER=panda +MOOSE_CLICKHOUSE_CONFIG__PASSWORD=pandapass +MOOSE_CLICKHOUSE_CONFIG__USE_SSL=false +``` + +### Redis +```bash +MOOSE_REDIS_CONFIG__URL=redis://localhost:6379 +MOOSE_REDIS_CONFIG__KEY_PREFIX=MS +``` +### Redpanda/Kafka +```bash +MOOSE_REDPANDA_CONFIG__BROKER=localhost:19092 +MOOSE_REDPANDA_CONFIG__NAMESPACE=my_namespace +MOOSE_REDPANDA_CONFIG__SASL_USERNAME=user +MOOSE_REDPANDA_CONFIG__SASL_PASSWORD=password +``` +### Feature Flags +```bash +MOOSE_FEATURES__STREAMING_ENGINE=true +MOOSE_FEATURES__WORKFLOWS=false +MOOSE_FEATURES__OLAP=true +MOOSE_FEATURES__APIS=true +``` +### Logging +```bash +MOOSE_LOGGER__LEVEL=info +MOOSE_LOGGER__STDOUT=true +MOOSE_LOGGER__FORMAT=Json +``` +For a complete list of all available configuration options, see the [moose.config.toml Reference](#mooseconfigtoml-reference) above. diff --git a/apps/framework-docs-v2/content/moosestack/data-modeling.mdx b/apps/framework-docs-v2/content/moosestack/data-modeling.mdx index 424ec84d25..7a32d04400 100644 --- a/apps/framework-docs-v2/content/moosestack/data-modeling.mdx +++ b/apps/framework-docs-v2/content/moosestack/data-modeling.mdx @@ -4,10 +4,919 @@ description: Learn how to model data in MooseStack order: 5 --- +import { Callout, LanguageTabs, LanguageTabContent, CTACards, CTACard, BulletPointsCard, FileTree } from "@/components/mdx"; + # Data Modeling -This page is a placeholder. Content migration pending. + + + + + +## Overview + +In Moose, data models are just TypeScript interfaces (TypeScript) or Pydantic models (Python) that become the authoritative source for your infrastructure schemas. + +Data Models are used to define: +- [OLAP Tables and Materialized Views](/moosestack/olap) (automatically generated DDL) +- [Redpanda/Kafka Streams](/moosestack/streaming) (schema registry and topic validation) +- [API Contracts](/moosestack/apis) (request/response validation and OpenAPI specs) +- [Workflow Task Input and Output Types](/moosestack/workflows) (typed function inputs/outputs) + +## Philosophy + +### Problem: Analytical Backends are Prone to Schema Drift +Analytical backends are unique in that they typically have to coordinate schemas across multiple systems that each have their own type systems and constraints. + +Consider a typical pipeline for ingesting events into a ClickHouse table. + + + +```ts +// What you're building: +// API endpoint → Kafka topic → ClickHouse table → Analytics API + +// Traditional approach: Define schema 4 times +// 1. API validation schema +const apiSchema = z.object({ + userId: z.string(), + eventType: z.enum(["click", "view", "purchase"]), + timestamp: z.string().datetime() +}); + +// 2. Kafka schema (Avro/JSON Schema) +const kafkaSchema = { + type: "record", + fields: [ + { name: "userId", type: "string" }, + { name: "eventType", type: "string" }, + { name: "timestamp", type: "string" } + ] +}; + +// 3. ClickHouse DDL +// CREATE TABLE events ( +// user_id String, +// event_type LowCardinality(String), +// timestamp DateTime +// ) ENGINE = MergeTree() + +// 4. Analytics API response type +interface EventsResponse { + userId: string; + eventType: string; + timestamp: Date; +} +``` + + +```python +# What you're building: +# API endpoint → Kafka topic → ClickHouse table → Analytics API + +# Traditional approach: Define schema 4 times +# 1. API validation with Pydantic +class APIEvent(BaseModel): + user_id: str + event_type: Literal["click", "view", "purchase"] + timestamp: datetime + +# 2. Kafka schema registration +kafka_schema = { + "type": "record", + "fields": [ + {"name": "user_id", "type": "string"}, + {"name": "event_type", "type": "string"}, + {"name": "timestamp", "type": "string"} + ] +} + +# 3. ClickHouse DDL +# CREATE TABLE events ( +# user_id String, +# event_type LowCardinality(String), +# timestamp DateTime +# ) ENGINE = MergeTree() + +# 4. Analytics API response +class EventsResponse(BaseModel): + user_id: str + event_type: str + timestamp: datetime +``` + + + +**The Problem:** When you add a field or change a type, you must update it in multiple places. Miss one, and you get: +- Silent data loss (Kafka → ClickHouse sync fails) +- Runtime errors +- Data quality issues (validation gaps) + +### Solution: Model In Code, Reuse Everywhere + +With Moose you define your schemas in native language types with optional metadata. This lets you reuse your schemas across multiple systems: + + + +```ts filename="app/index.ts" copy +// 1. Define your schema (WHAT your data looks like) +interface MyDataModel { + primaryKey: Key; + someString: string & LowCardinality; + someNumber: number; + someDate: Date; + someJson: Record; +} + +// This single interface can be reused across multiple systems: +const pipeline = new IngestPipeline("MyDataPipeline", { + ingestApi: true, // POST API endpoint + stream: true, // Kafka topic + table: { // ClickHouse table + orderByFields: ["primaryKey", "someDate"] + } +}); +``` + + +```python filename="app/main.py" +# 1. Define your schema (WHAT your data looks like) +class MyFirstDataModel(BaseModel): + id: Key[str] + some_string: Annotated[str, "LowCardinality"] + some_number: int + some_date: datetime + some_json: Any + +# This single model can be reused across multiple systems: +my_first_pipeline = IngestPipeline[MyFirstDataModel]("my_first_pipeline", IngestPipelineConfig( + ingest_api=True, # POST API endpoint + stream=True, # Kafka topic + table=True # ClickHouse table +)) +``` + + + + + +## How It Works + + + +The key idea is leveraging Union Types to extend base TypeScript types with "metadata" that represents specific optimizations and details on how to either: +- map that type in ClickHouse +- validate the data at runtime + +```ts +interface Event { + // Base type: string + // ClickHouse: String with primary key + id: Key; + + // Base type: string + // ClickHouse: Decimal(10,2) for precise money + amount: string & ClickHouseDecimal<10, 2>; + + // Base type: string + // ClickHouse: LowCardinality(String) for enums + status: string & LowCardinality; + + // Base type: Date + // ClickHouse: DateTime + createdAt: Date; +} + +// In your application code: +const event: Event = { + id: "id_123", + amount: "99.99", // Just a string in TypeScript + status: "completed", // Just a string in TypeScript + createdAt: new Date() +}; + +// In ClickHouse: +// CREATE TABLE events ( +// id String, +// amount Decimal(10,2), +// status LowCardinality(String), +// created_at DateTime +// ) ENGINE = MergeTree() +// ORDER BY transaction_id +``` + + +The key idea is leveraging Annotated types to extend base Python types with "metadata" that represents specific optimizations and details on how to map that type in ClickHouse: + +```python +from moose_lib import Key, clickhouse_decimal +from typing import Annotated + +class Model(BaseModel): + # Base type: str + # ClickHouse: String with primary key + id: Key[str] + + # Base type: Decimal + # ClickHouse: Decimal(10,2) for precise money + amount: clickhouse_decimal(10, 2) + + # Base type: str + # ClickHouse: LowCardinality(String) for enums + status: Annotated[str, "LowCardinality"] + + # Base type: datetime + # ClickHouse: DateTime + created_at: datetime + +events = OlapTable[Event]("events") + +# In your application code: +tx = Event( + id="id_123", + amount=Decimal("99.99"), # Regular Decimal in Python + status="completed", # Regular string in Python + created_at=datetime.now() +) + +# In ClickHouse: +# CREATE TABLE events ( +# id String, +# amount Decimal(10,2), +# status LowCardinality(String), +# created_at DateTime +# ) ENGINE = MergeTree() +# ORDER BY transaction_id +``` + + + +**The metadata annotations are compile-time only** - they don't affect your runtime code. Your application works with regular strings and numbers, while Moose uses the metadata to generate optimized infrastructure. + +## Building Data Models: From Simple to Complex + +Let's walk through how to model data for different infrastructure components and see how types behave across them. + +### Simple Data Model Shared Across Infrastructure + +A basic data model that works identically across all infrastructure components: + + + +```ts filename="app/datamodels/SimpleShared.ts" +export interface SimpleShared { + id: string; + name: string; + value: number; + timestamp: Date; +} + +// This SAME model creates all infrastructure +const pipeline = new IngestPipeline("simple_shared", { + ingestApi: true, // Creates: POST /ingest/simple_shared + stream: true, // Creates: Kafka topic + table: true // Creates: ClickHouse table +}); + +// The exact same types work everywhere: +// - API validates: { id: "123", name: "test", value: 42, timestamp: "2024-01-01T00:00:00Z" } +// - Kafka stores: { id: "123", name: "test", value: 42, timestamp: "2024-01-01T00:00:00Z" } +// - ClickHouse table: id String, name String, value Float64, timestamp DateTime +``` + + +```python filename="app/datamodels/simple_shared.py" +from pydantic import BaseModel +from datetime import datetime + +class SimpleShared(BaseModel): + id: str + name: str + value: float + timestamp: datetime + +# This SAME model creates all infrastructure +pipeline = IngestPipeline[SimpleShared]("simple_shared", IngestPipelineConfig( + ingest_api=True, # Creates: POST /ingest/simple_shared + stream=True, # Creates: Kafka topic + table=True # Creates: ClickHouse table +)) + +# The exact same types work everywhere: +# - API validates: { "id": "123", "name": "test", "value": 42, "timestamp": "2024-01-01T00:00:00Z" } +# - Kafka stores: { "id": "123", "name": "test", "value": 42, "timestamp": "2024-01-01T00:00:00Z" } +# - ClickHouse table: id String, name String, value Float64, timestamp DateTime +``` + + + +**Key Point**: One model definition creates consistent schemas across all systems. + +### Composite Types Shared Across Infrastructure + +Complex types including nested objects, arrays, and enums work seamlessly across all components: + + + +```ts filename="app/datamodels/CompositeShared.ts" +import { Key } from "@514labs/moose-lib"; + +export interface CompositeShared { + id: Key; // Primary key + status: "active" | "pending" | "completed"; // Enum + + // Nested object + metadata: { + category: string; + priority: number; + tags: string[]; + }; + + // Arrays and maps + values: number[]; + attributes: Record; + + // Optional field + description?: string; + createdAt: Date; +} + +// Using in IngestPipeline - all types preserved +const pipeline = new IngestPipeline("composite_shared", { + ingestApi: true, + stream: true, + table: true +}); + +// How the types map: +// - API validates nested structure and enum values +// - Kafka preserves the exact JSON structure +// - ClickHouse creates: +// - id String (with PRIMARY KEY) +// - status Enum8('active', 'pending', 'completed') +// - metadata.category String, metadata.priority Float64, metadata.tags Array(String) +// - values Array(Float64) +// - attributes String (JSON) +// - description Nullable(String) +// - createdAt DateTime +``` + + +```python filename="app/datamodels/composite_shared.py" +from moose_lib import Key +from pydantic import BaseModel +from typing import List, Dict, Any, Optional, Literal +from datetime import datetime + +class Metadata(BaseModel): + category: str + priority: float + tags: List[str] + +class CompositeShared(BaseModel): + id: Key[str] # Primary key + status: Literal["active", "pending", "completed"] # Enum + + # Nested object + metadata: Metadata + + # Arrays and maps + values: List[float] + attributes: Dict[str, Any] + + # Optional field + description: Optional[str] = None + created_at: datetime + +# Using in IngestPipeline - all types preserved +pipeline = IngestPipeline[CompositeShared]("composite_shared", IngestPipelineConfig( + ingest_api=True, + stream=True, + table=True +)) + +# How the types map: +# - API validates nested structure and enum values +# - Kafka preserves the exact JSON structure +# - ClickHouse creates: +# - id String (with PRIMARY KEY) +# - status Enum8('active', 'pending', 'completed') +# - metadata.category String, metadata.priority Float64, metadata.tags Array(String) +# - values Array(Float64) +# - attributes String (JSON) +# - description Nullable(String) +# - created_at DateTime +``` + + + +**Key Point**: Complex types including nested objects and arrays work consistently across all infrastructure. + +### ClickHouse-Specific Types (Standalone vs IngestPipeline) + +ClickHouse type annotations optimize database performance but are **transparent to other infrastructure**: + + + +```ts filename="app/datamodels/ClickHouseOptimized.ts" +import { Key, Decimal, ClickHouseDecimal, LowCardinality, ClickHouseNamedTuple } from "@514labs/moose-lib"; + +export interface ClickHouseOptimized { + id: Key; + + // ClickHouse-specific type annotations + amount: Decimal<10, 2>; // Decimal(10,2) in ClickHouse + // Alternative: amount: string & ClickHouseDecimal<10, 2>; // Verbose syntax still works + category: string & LowCardinality; // LowCardinality(String) in ClickHouse + + // Optimized nested type + details: { + name: string; + value: number; + } & ClickHouseNamedTuple; // NamedTuple in ClickHouse + + timestamp: Date; +} + +// SCENARIO 1: Standalone OlapTable - gets all optimizations +const table = new OlapTable("optimized_table", { + orderByFields: ["id", "timestamp"] +}); +// Creates ClickHouse table with: +// - amount Decimal(10,2) +// - category LowCardinality(String) +// - details Tuple(name String, value Float64) + +// SCENARIO 2: IngestPipeline - optimizations ONLY in ClickHouse +const pipeline = new IngestPipeline("optimized_pipeline", { + ingestApi: true, + stream: true, + table: true +}); + +// What happens at each layer: +// 1. API receives/validates: { amount: "123.45", category: "electronics", ... } +// - Sees amount as string, category as string (annotations ignored) +// 2. Kafka stores: { amount: "123.45", category: "electronics", ... } +// - Plain JSON, no ClickHouse types +// 3. ClickHouse table gets optimizations: +// - amount stored as Decimal(10,2) +// - category stored as LowCardinality(String) +// - details stored as NamedTuple +``` + + +```python filename="app/datamodels/clickhouse_optimized.py" +from moose_lib import Key, clickhouse_decimal, OlapTable, IngestPipeline, IngestPipelineConfig +from typing import Annotated +from pydantic import BaseModel +from datetime import datetime + +class Details(BaseModel): + name: str + value: float + +class ClickHouseOptimized(BaseModel): + id: Key[str] + + # ClickHouse-specific type annotations + amount: clickhouse_decimal(10, 2) # Decimal(10,2) in ClickHouse + category: Annotated[str, "LowCardinality"] # LowCardinality(String) in ClickHouse + + # Optimized nested type + details: Annotated[Details, "ClickHouseNamedTuple"] # NamedTuple in ClickHouse + + timestamp: datetime + +# SCENARIO 1: Standalone OlapTable - gets all optimizations +table = OlapTable[ClickHouseOptimized]("optimized_table", { + "order_by_fields": ["id", "timestamp"] +}) +# Creates ClickHouse table with: +# - amount Decimal(10,2) +# - category LowCardinality(String) +# - details Tuple(name String, value Float64) + +# SCENARIO 2: IngestPipeline - optimizations ONLY in ClickHouse +pipeline = IngestPipeline[ClickHouseOptimized]("optimized_pipeline", IngestPipelineConfig( + ingest_api=True, + stream=True, + table=True +)) + +# What happens at each layer: +# 1. API receives/validates: { "amount": "123.45", "category": "electronics", ... } +# - Sees amount as str, category as str (annotations ignored) +# 2. Kafka stores: { "amount": "123.45", "category": "electronics", ... } +# - Plain JSON, no ClickHouse types +# 3. ClickHouse table gets optimizations: +# - amount stored as Decimal(10,2) +# - category stored as LowCardinality(String) +# - details stored as NamedTuple +``` + + + +**Key Point**: ClickHouse annotations are metadata that ONLY affect the database schema. Your application code and other infrastructure components see regular TypeScript/Python types. + +### API Contracts with Runtime Validators + +APIs use runtime validation to ensure query parameters meet your requirements: + + + +```ts filename="app/apis/consumption_with_validation.ts" +import { tags, Api } from "@514labs/moose-lib"; + +// Query parameters with runtime validation +interface SearchParams { + // Date range validation + startDate: string & tags.Format<"date">; // Must be YYYY-MM-DD + endDate: string & tags.Format<"date">; + + // Numeric constraints + minValue?: number & tags.Minimum<0>; // Optional, but if provided >= 0 + maxValue?: number & tags.Maximum<1000>; // Optional, but if provided <= 1000 + + // String validation + category?: string & tags.MinLength<2> & tags.MaxLength<50>; + + // Pagination + page?: number & tags.Type<"int32"> & tags.Minimum<1>; + limit?: number & tags.Type<"int32"> & tags.Minimum<1> & tags.Maximum<100>; +} + +// Response data model +interface SearchResult { + id: string; + name: string; + value: number; + category: string; + timestamp: Date; +} + +// Create validated API endpoint +const searchAPI = new Api( + "search", + async (params, { client }) => { + // Params are already validated when this runs + const query = ` + SELECT * FROM data_table + WHERE timestamp >= {startDate: Date} + AND timestamp <= {endDate: Date} + ${params.minValue ? `AND value >= {minValue: Float64}` : ''} + ${params.maxValue ? `AND value <= {maxValue: Float64}` : ''} + ${params.category ? `AND category = {category: String}` : ''} + LIMIT {limit: UInt32} + OFFSET {offset: UInt32} + `; + + return client.query(query, { + startDate: params.startDate, + endDate: params.endDate, + minValue: params.minValue, + maxValue: params.maxValue, + category: params.category, + limit: params.limit || 10, + offset: ((params.page || 1) - 1) * (params.limit || 10) + }); + } +); + +// API Usage Examples: +// ✅ Valid: GET /api/search?startDate=2024-01-01&endDate=2024-01-31 +// ✅ Valid: GET /api/search?startDate=2024-01-01&endDate=2024-01-31&minValue=100&limit=50 +// ❌ Invalid: GET /api/search?startDate=Jan-1-2024 (wrong date format) +// ❌ Invalid: GET /api/search?startDate=2024-01-01&endDate=2024-01-31&limit=200 (exceeds max) +``` + + +```python filename="app/apis/consumption_with_validation.py" +from moose_lib import Api +from pydantic import BaseModel, Field +from datetime import datetime +from typing import Optional, List + +# Query parameters with runtime validation +class SearchParams(BaseModel): + # Date range validation + start_date: str = Field(..., regex="^\\d{4}-\\d{2}-\\d{2}$") # Must be YYYY-MM-DD + end_date: str = Field(..., regex="^\\d{4}-\\d{2}-\\d{2}$") + + # Numeric constraints + min_value: Optional[float] = Field(None, ge=0) # Optional, but if provided >= 0 + max_value: Optional[float] = Field(None, le=1000) # Optional, but if provided <= 1000 + + # String validation + category: Optional[str] = Field(None, min_length=2, max_length=50) + + # Pagination + page: Optional[int] = Field(None, ge=1) + limit: Optional[int] = Field(None, ge=1, le=100) + +# Response data model +class SearchResult(BaseModel): + id: str + name: str + value: float + category: str + timestamp: datetime + +# Create validated API endpoint +async def search_handler(params: SearchParams, client: MooseClient) -> List[SearchResult]: + # Params are already validated when this runs + # Build a parameterized query safely + clauses = [ + "timestamp >= {startDate}", + "timestamp <= {endDate}" + ] + params_dict = { + "startDate": params.start_date, + "endDate": params.end_date, + "limit": params.limit or 10, + "offset": ((params.page or 1) - 1) * (params.limit or 10) + } + if params.min_value is not None: + clauses.append("value >= {minValue}") + params_dict["minValue"] = params.min_value + if params.max_value is not None: + clauses.append("value <= {maxValue}") + params_dict["maxValue"] = params.max_value + if params.category is not None: + clauses.append("category = {category}") + params_dict["category"] = params.category + + where_clause = " AND ".join(clauses) + query = f""" + SELECT * FROM data_table + WHERE {where_clause} + LIMIT {limit} + OFFSET {offset} + """ + + results = await client.query.execute(query, params=params_dict) + return [SearchResult(**row) for row in results] + +search_api = Api[SearchParams, List[SearchResult]]( + "search", + handler=search_handler +) + +# API Usage Examples: +# ✅ Valid: GET /api/search?start_date=2024-01-01&end_date=2024-01-31 +# ✅ Valid: GET /api/search?start_date=2024-01-01&end_date=2024-01-31&min_value=100&limit=50 +# ❌ Invalid: GET /api/search?start_date=Jan-1-2024 (wrong date format) +# ❌ Invalid: GET /api/search?start_date=2024-01-01&end_date=2024-01-31&limit=200 (exceeds max) +``` + + + +**Key Point**: Runtime validators ensure API consumers provide valid data, returning clear error messages for invalid requests before any database queries run. + + +## Additional Data Modeling Patterns + +### Modeling for Stream Processing + +When you need to process data in real-time before it hits the database: + + + +```ts filename="app/datamodels/StreamExample.ts" +import { Key, LowCardinality } from "@514labs/moose-lib"; + +// Raw data from external source +interface RawData { + id: Key; + timestamp: Date; + rawPayload: string; + sourceType: string & LowCardinality; +} + +// Processed data after transformation +interface ProcessedData { + id: Key; + timestamp: Date; + field1: string; + field2: string & LowCardinality; + numericValue: number; + attributes: Record; +} + +// Create stream with transformation +const rawStream = new Stream("raw-stream"); +const processedStream = new Stream("processed-stream"); + +// Transform raw data to processed +rawStream.addConsumer(async (raw: RawData) => { + const parsed = JSON.parse(raw.rawPayload); + + const processed: ProcessedData = { + id: raw.id, + timestamp: raw.timestamp, + field1: parsed.field_1, + field2: parsed.field_2, + numericValue: parseFloat(parsed.value) || 0, + attributes: parsed.attributes || {} + }; + + await processedStream.publish(processed); +}); + +// Sink to ClickHouse +const table = new OlapTable("processed_data", { + stream: processedStream, + orderByFields: ["id", "timestamp"] +}); +``` + + +```python filename="app/datamodels/stream_example.py" +from moose_lib import Key, Stream, OlapTable +from pydantic import BaseModel +from typing import Dict, Any, Annotated +from datetime import datetime +import json + +# Raw data from external source +class RawData(BaseModel): + id: Key[str] + timestamp: datetime + raw_payload: str + source_type: Annotated[str, "LowCardinality"] + +# Processed data after transformation +class ProcessedData(BaseModel): + id: Key[str] + timestamp: datetime + field1: str + field2: Annotated[str, "LowCardinality"] + numeric_value: float + attributes: Dict[str, Any] + +# Create streams +raw_stream = Stream[RawData]("raw-stream") + + +processed_table = OlapTable[ProcessedData]("processed_data", OlapConfig( + order_by_fields = ["id", "timestamp"] +)) + +processed_stream = Stream[ProcessedData]("processed-stream", StreamConfig( + destination=processed_table +)) + +# Transform raw data +async def process_data(raw: RawData): + parsed = json.loads(raw.raw_payload) + + processed = ProcessedData( + id=raw.id, + timestamp=raw.timestamp, + field1=parsed["field_1"], + field2=parsed["field_2"], + numeric_value=float(parsed.get("value", 0)), + attributes=parsed.get("attributes", {}) + ) + +raw_stream.add_transform(processed_stream, process_data) + +``` + + + +### Modeling for Workflow Tasks + +Define strongly-typed inputs and outputs for async jobs: + + + +```ts filename="app/workflows/task_example.ts" +import { Task, tags } from "@514labs/moose-lib"; + +// Input validation with constraints +interface TaskInput { + id: string & tags.Format<"uuid">; + items: string[]; + taskType: "typeA" | "typeB" | "typeC"; + options?: { + includeMetadata: boolean; + maxItems?: number & tags.Minimum<1> & tags.Maximum<100>; + }; +} + +// Structured output +interface TaskOutput { + id: string; + processedAt: Date; + resultA?: { + category: string; + score: number; + details: Record; + }; + resultB?: { + values: string[]; + metrics: number[]; + }; + resultC?: { + field1: string; + field2: string; + field3: number; + }; +} + +// Create workflow task +const exampleTask = new Task( + "example-task", + { + run: async (ctx) => { + // Process data based on task type + const output: TaskOutput = { + id: ctx.input.id, + processedAt: new Date() + }; + + if (ctx.input.taskType === "typeA") { + output.resultA = await processTypeA(ctx.input); + } + + return output; + }, + + retries: 3, + timeout: 30000 // 30 seconds + } +); +``` + + +```python filename="app/workflows/task_example.py" +from moose_lib import Task, TaskContext +from pydantic import BaseModel, Field +from typing import Optional, List, Literal, Dict, Any +from datetime import datetime + +# Input validation with constraints +class TaskOptions(BaseModel): + include_metadata: bool + max_items: Optional[int] = Field(None, ge=1, le=100) + +class TaskInput(BaseModel): + id: str = Field(..., regex="^[0-9a-f-]{36}$") + items: List[str] + task_type: Literal["typeA", "typeB", "typeC"] + options: Optional[TaskOptions] = None + +# Structured output +class ResultA(BaseModel): + category: str + score: float + details: Dict[str, Any] + +class ResultB(BaseModel): + values: List[str] + metrics: List[float] + +class ResultC(BaseModel): + field1: str + field2: str + field3: float + +class TaskOutput(BaseModel): + id: str + processed_at: datetime + result_a: Optional[ResultA] = None + result_b: Optional[ResultB] = None + result_c: Optional[ResultC] = None +# Create workflow task +async def run_task(ctx: TaskContext[TaskInput]) -> TaskOutput: + # Process data based on task type + output = TaskOutput( + id=ctx.input.id, + processed_at=datetime.now() + ) + if ctx.input.task_type == "typeA": + output.result_a = await process_type_a(ctx.input) + return output +example_task = Task[TaskInput, TaskOutput]( + "example-task", + run_function=run_task, + retries=3, + timeout=30 # seconds +) +``` + + diff --git a/apps/framework-docs-v2/content/moosestack/deploying/configuring-moose-for-cloud.mdx b/apps/framework-docs-v2/content/moosestack/deploying/configuring-moose-for-cloud.mdx index 28dfb87b95..0d0d315d3d 100644 --- a/apps/framework-docs-v2/content/moosestack/deploying/configuring-moose-for-cloud.mdx +++ b/apps/framework-docs-v2/content/moosestack/deploying/configuring-moose-for-cloud.mdx @@ -2,8 +2,132 @@ title: Cloud Configuration description: Configure MooseStack for cloud deployment order: 3 +category: deployment --- -# Cloud Configuration +# Configuring Moose for cloud environments -This page is a placeholder. Content migration pending. +In the [Packaging Moose for deployment](/moosestack/deploying/packaging-moose-for-deployment) page, we looked at how to package your moose +application into Docker containers (using the `moose build --docker` command), and you've pushed them to your container repository. + +We can connect and configure your container image with remote ClickHouse and Redis-hosted services. You can also optionally +use Redpanda for event streaming and Temporal for workflow orchestration. + +The methods used to accomplish this are generally similar, but the specific details depend on your target cloud infrastructure. +So, we'll look at the overarching concepts and provide some common examples. + +## Specifying your repository container + +Earlier, we created two local containers and pushed them to a docker repository. + +```txt filename="Terminal" copy +>docker images +REPOSITORY TAG IMAGE ID CREATED SIZE +moose-df-deployment-aarch64-unknown-linux-gnu 0.3.175 c50674c7a68a About a minute ago 155MB +moose-df-deployment-x86_64-unknown-linux-gnu 0.3.175 e5b449d3dea3 About a minute ago 163MB +``` + +We pushed the containers to the `514labs` Docker Hub account. So, we have these two containers available for use: + +``` +514labs/moose-df-deployment-aarch64-unknown-linux-gnu:0.3.175 +514labs/moose-df-deployment-x86_64-unknown-linux-gnu:0.3.175 +``` + +In later examples, we'll use an AMD64 (x86_64) based machine, so we'll stick to using the following container image: `514labs/moose-df-deployment-x86_64-unknown-linux-gnu:0.3.175` + +We'll also examine how the container image name can be used in various cloud providers and scenarios. + +## General overview + +The general approach is to use a cloud provider that supports specifying a container image to launch your application. Examples include the Google Kubernetes Engine (GKE), Amazon's Elastic Kubernetes Service (EKS), and Elastic Container Service (ECS). Each provider also offers a way of configuring container environment variables that your container application will have access to. + +## Essential Environment Variables + +Based on our production deployments, here are the essential environment variables you'll need to configure for your Moose application in cloud environments: + +### Logging and Telemetry + +``` +# Logger configuration +MOOSE_LOGGER__LEVEL=Info +MOOSE_LOGGER__STDOUT=true +MOOSE_LOGGER__FORMAT=Json + +# Telemetry configuration +MOOSE_TELEMETRY__ENABLED=false +MOOSE_TELEMETRY__EXPORT_METRICS=true + +# For debugging +RUST_BACKTRACE=1 +``` + +### HTTP Server Configuration + +``` +# HTTP server settings +MOOSE_HTTP_SERVER_CONFIG__HOST=0.0.0.0 +MOOSE_HTTP_SERVER_CONFIG__PORT=4000 +``` + +### External Service Connections + +For detailed configuration of the external services, refer to the [Preparing ClickHouse and Redpanda](/moosestack/deploying/preparing-clickhouse-redpanda) page. + +#### ClickHouse + +``` +MOOSE_CLICKHOUSE_CONFIG__DB_NAME= +MOOSE_CLICKHOUSE_CONFIG__USER= +MOOSE_CLICKHOUSE_CONFIG__PASSWORD= +MOOSE_CLICKHOUSE_CONFIG__HOST= +MOOSE_CLICKHOUSE_CONFIG__HOST_PORT=8443 +MOOSE_CLICKHOUSE_CONFIG__USE_SSL=1 +MOOSE_CLICKHOUSE_CONFIG__NATIVE_PORT=9440 +``` + +#### Redis + +Moose requires Redis for caching and message passing: + +``` +MOOSE_REDIS_CONFIG__URL= +MOOSE_REDIS_CONFIG__KEY_PREFIX= +``` + +#### Redpanda (Optional) + +If you choose to use Redpanda for event streaming: + +``` +MOOSE_REDPANDA_CONFIG__BROKER= +MOOSE_REDPANDA_CONFIG__NAMESPACE= +MOOSE_REDPANDA_CONFIG__MESSAGE_TIMEOUT_MS=10043 +MOOSE_REDPANDA_CONFIG__SASL_USERNAME= +MOOSE_REDPANDA_CONFIG__SASL_PASSWORD= +MOOSE_REDPANDA_CONFIG__SASL_MECHANISM=SCRAM-SHA-256 +MOOSE_REDPANDA_CONFIG__SECURITY_PROTOCOL=SASL_SSL +MOOSE_REDPANDA_CONFIG__REPLICATION_FACTOR=3 +``` + +#### Temporal (Optional) + +If you choose to use Temporal for workflow orchestration: + +``` +MOOSE_TEMPORAL_CONFIG__CA_CERT=/etc/ssl/certs/ca-certificates.crt +MOOSE_TEMPORAL_CONFIG__API_KEY= +MOOSE_TEMPORAL_CONFIG__TEMPORAL_HOST=.tmprl.cloud +``` + +## Securing Sensitive Information + +When deploying to cloud environments, it's important to handle sensitive information like passwords and API keys securely. Each cloud provider offers mechanisms for this: + +- **Kubernetes**: Use Secrets to store sensitive data. See our [Kubernetes deployment guide](/moosestack/deploying/deploying-on-kubernetes) for examples. +- **Amazon ECS**: Use AWS Secrets Manager or Parameter Store to securely inject environment variables. +- **Other platforms**: Use the platform's recommended secrets management approach. + +Never hardcode sensitive values directly in your deployment configuration files. + +Please share your feedback about Moose monitoring capabilities through [our GitHub repository](https://github.com/514-labs/moose/issues/new?title=Feedback%20for%20%E2%80%9CMonitoring%E2%80%9D&labels=feedback). diff --git a/apps/framework-docs-v2/content/moosestack/deploying/deploying-on-an-offline-server.mdx b/apps/framework-docs-v2/content/moosestack/deploying/deploying-on-an-offline-server.mdx index 7706fc295e..ea0d773aac 100644 --- a/apps/framework-docs-v2/content/moosestack/deploying/deploying-on-an-offline-server.mdx +++ b/apps/framework-docs-v2/content/moosestack/deploying/deploying-on-an-offline-server.mdx @@ -2,8 +2,273 @@ title: Offline Deployment description: Deploy MooseStack on offline servers order: 6 +category: deployment --- -# Offline Deployment +import { Callout, LanguageTabs, LanguageTabContent } from "@/components/mdx"; -This page is a placeholder. Content migration pending. +# Building and Deploying Moose Applications + +This guide will walk you through the process of building a Moose application and deploying it to a server that does not have internet access. +We'll cover both the build environment setup and the deployment environment requirements. + +## Build Environment Setup + +### Prerequisites + +Before you can build a Moose application, you need to set up your build environment with the following dependencies: + +OS: +- Debian 10+ +- Ubuntu 18.10+ +- Fedora 29+ +- CentOS/RHEL 8+ +- Amazon Linux 2023+ +- Mac OS 13+ + +Common CLI utilities: +- zip +- curl (optional, for installing the Moose CLI) + + + +Typescript build environment requirements: +1. Node.js 20 or later +2. npm, yarn or pnpm package manager +3. Moose CLI + + +Python build environment requirements: +1. Python 3.12 or later (we recommend using pyenv for Python version management) +2. pip + + + +### Setting up the Build Environment + + + +First, ensure you have Node.js installed. We recommend using nvm (Node Version Manager): + +```bash +# Install nvm +curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash + +# Reload shell configuration +source ~/.bashrc # or ~/.zshrc + +# Install and use Node.js 20 +nvm install 20 +nvm use 20 +``` + +Verify the installation: +```bash +node --version +npm --version # or yarn --version +``` + + +First, install the required system dependencies: + +```bash +sudo apt update +sudo apt install build-essential libssl-dev zlib1g-dev libbz2-dev \ + libreadline-dev libsqlite3-dev curl git libncursesw5-dev xz-utils \ + tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev +``` + +Install pyenv and configure your shell: + +```bash +curl -fsSL https://pyenv.run | bash +``` + +Add the following to your `~/.bashrc` or `~/.zshrc`: + +```bash +export PYENV_ROOT="$HOME/.pyenv" +command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH" +eval "$(pyenv init -)" +``` + +Install and set Python 3.12: + +```bash +pyenv install 3.12 +pyenv global 3.12 +``` + +Verify the installation: +```bash +python --version +``` + + + +### Installing Moose CLI (Optional) + +You can install the Moose CLI using the official installer: + +```bash +curl -SfsL https://fiveonefour.com/install.sh | bash -s -- moose +source ~/.bashrc # Or restart your terminal +``` + +or + + + +```bash +pip install moose-cli +``` + + +```bash +npm install -g @514labs/moose-cli +``` + + + +## Building Your Application + +### 1. Initialize a New Project (Optional) +This step is optional if you already have a Moose project. +Create a new Moose project: + + + +```bash +moose init your-project-name ts +cd your-project-name +``` + +Install dependencies: +```bash +npm install # or yarn install +``` + + +```bash +moose init your-project-name py +cd your-project-name +``` + + + +### 2. Build the Application + + + Make sure you have the `zip` utility installed (`sudo apt install zip`) before building your application. + + +If you installed the moose cli to be available globally, you can build the application with the following command: +```bash +moose build +``` +Or if you installed the moose cli to be available locally, you can build the application with the following command: + + + +Create the deployable package: +```bash +npm run moose-cli build +``` + + +The build process will create a deployable package: + +```bash +moose build +``` + + + +This will create a zip file in your project directory with a timestamp, for example: `your-project-name-YYYY-MM-DD.zip` + +## Deployment Environment Setup + +### Deployment Prerequisites + +The deployment server requires: + + + +1. Node.js 20 or later +2. Unzip utility + + +1. Python 3.12 or later +2. Unzip utility + + + +### Setting up the Deployment Environment + +1. Install the runtime environment: + + + +Follow the Node.js installation steps from the build environment setup section. + + +Follow the Python installation steps from the build environment setup section. + + + +2. Install the unzip utility: +```bash +sudo apt install unzip +``` + +## Deploying Your Application + +1. Copy your built application package to the deployment server + +2. Extract the application: +```bash +unzip your-project-name-YYYY-MM-DD.zip -d ./app +cd ./app/packager +``` + +3. Start your application: + + + +```bash +# If using npm scripts +npm run moose-cli prod + +# Or directly with moose +moose prod +``` + + +```bash +moose prod +``` + + + + + Ensure all required environment variables and configurations are properly set before starting your application. + + +## Troubleshooting + + + + +1. Verify that Node.js is properly installed using `node --version` +2. Check that your application's dependencies are properly listed in `package.json` +3. If you encounter TypeScript compilation errors, ensure your `tsconfig.json` is properly configured + + + + +1. Verify that Python is properly installed using `python --version` +2. Check that your application's dependencies are properly listed in `requirements.txt` +3. If you encounter Python import errors, ensure your `PYTHONPATH` is properly set + + + diff --git a/apps/framework-docs-v2/content/moosestack/deploying/deploying-on-ecs.mdx b/apps/framework-docs-v2/content/moosestack/deploying/deploying-on-ecs.mdx index 7f5df0b78f..8d693ecd34 100644 --- a/apps/framework-docs-v2/content/moosestack/deploying/deploying-on-ecs.mdx +++ b/apps/framework-docs-v2/content/moosestack/deploying/deploying-on-ecs.mdx @@ -2,8 +2,168 @@ title: AWS ECS Deployment description: Deploy MooseStack on AWS ECS order: 5 +category: deployment --- -# AWS ECS Deployment +import { Callout } from "@/components/mdx"; -This page is a placeholder. Content migration pending. +# Deploying on Amazon ECS + +Moose can be deployed to Amazon's Elastic Container Service (ECS). ECS offers a managed container orchestrator at a fraction of the complexity of managing a Kubernetes cluster. + +If you're relatively new to ECS we recommend the following resources: + +- [Amazon Elastic Container Service (ECS) with a Load Balancer | AWS Tutorial with New ECS Experience](https://www.youtube.com/watch?v=rUgZNXKbsrY) +- [Tutorial: Deploy NGINX Containers On ECS Fargate with Load Balancer](https://bhaveshmuleva.hashnode.dev/tutorial-deploy-nginx-containers-on-ecs-fargate-with-load-balancer) +- [How to configure target groups ports with listeners and tasks](https://stackoverflow.com/questions/66275574/how-to-configure-target-groups-ports-with-listeners-and-tasks) + +The first step is deciding whether you'll host your Moose container on Docker Hub or Amazon's Elastic Container Registry (ECR). +Amazon ECR is straightforward and is designed to work out of the box with ECS. Using Docker Hub works if your moose container is publicly available; however, + if your container is private, you'll need to do a bit more work to provide ECS with your Docker credentials. + +> See: [Authenticating with Docker Hub for AWS Container Services](https://aws.amazon.com/blogs/containers/authenticating-with-docker-hub-for-aws-container-services/) + +Here is an overview of the steps required: + +1. You'll first need to create or use an existing ECS cluster. +2. Then, you'll need to create an ECS `Task definition.` This is where you'll specify whether you want to use AWS Fargate or AWS EC2 instances. + You'll also have options for selecting your OS and Architecture. Specify `Linux/X86-64` or `Linux/ARM-64`. This is important as you'll also need to + specify a matching moose container image, such as `moose-df-deployment-x86_64-unknown-linux-gnu:0.3.175` or `moose-df-deployment-aarch64-unknown-linux-gnu:0.3.175` +3. As with all AWS services, if you're using secrets to store credentials, you will need to specify an IAM role with an `AmazonECSTaskExecutionRolePolicy` and `SecretsManagerReadWrite` + policy. +4. Under the Container section, specify the name of your moose deployment and provide the container image name you're using. +5. Next, specify the Container Port as 4000. + +## Configuring container environment variables + +While still in the Amazon ECS Task definition section, you'll need to provide the environment variables on which your Moose application depends. +Scroll down to the Environment variables section and fill in each of the following variables. + + + ClickHouse and Redis are required components for Moose. Redpanda and Temporal are optional - configure them only if you're using these components in your application. + + +> Note: if you prefer, you can provide the environment variables below via an env file hosted on S3 or using AWS Secrets Manager for sensitive values. + +### Core Configuration + +| Key | Description | Example Value | +|-----|-------------|---------------| +| MOOSE_LOGGER__LEVEL | Log level | Info | +| MOOSE_LOGGER__STDOUT | Enable stdout logging | true | +| MOOSE_LOGGER__FORMAT | Log format | Json | +| RUST_BACKTRACE | Enable backtraces for debugging | 1 | + +### HTTP Server Configuration + +| Key | Description | Example Value | +|-----|-------------|---------------| +| MOOSE_HTTP_SERVER_CONFIG__HOST | Your moose network binding address | 0.0.0.0 | +| MOOSE_HTTP_SERVER_CONFIG__PORT | The network port your moose server is using | 4000 | + +### ClickHouse Configuration (Required) + +| Key | Description | Example Value | +|-----|-------------|---------------| +| MOOSE_CLICKHOUSE_CONFIG__DB_NAME | The name of your ClickHouse database | moose_production | +| MOOSE_CLICKHOUSE_CONFIG__USER | The database user name | clickhouse_user | +| MOOSE_CLICKHOUSE_CONFIG__PASSWORD | The password to your ClickHouse database | (use AWS Secrets Manager) | +| MOOSE_CLICKHOUSE_CONFIG__HOST | The hostname for your ClickHouse database | your-clickhouse.cloud.example.com | +| MOOSE_CLICKHOUSE_CONFIG__HOST_PORT | The HTTPS port for your ClickHouse database | 8443 | +| MOOSE_CLICKHOUSE_CONFIG__USE_SSL | Whether your database connection requires SSL | 1 | +| MOOSE_CLICKHOUSE_CONFIG__NATIVE_PORT | The native port for your ClickHouse database | 9440 | + +### Redis Configuration (Required) + +| Key | Description | Example Value | +|-----|-------------|---------------| +| MOOSE_REDIS_CONFIG__URL | Redis connection URL | redis://user:password@redis.example.com:6379 | +| MOOSE_REDIS_CONFIG__KEY_PREFIX | Prefix for Redis keys to isolate namespaces | moose_production | + +### Redpanda Configuration (Optional) + +| Key | Description | Example Value | +|-----|-------------|---------------| +| MOOSE_REDPANDA_CONFIG__BROKER | The hostname for your Redpanda instance | seed-5fbcae97.example.redpanda.com:9092 | +| MOOSE_REDPANDA_CONFIG__NAMESPACE | Namespace for isolation | moose_production | +| MOOSE_REDPANDA_CONFIG__MESSAGE_TIMEOUT_MS | The message timeout delay in milliseconds | 10043 | +| MOOSE_REDPANDA_CONFIG__SASL_USERNAME | Your Redpanda user name | redpanda_user | +| MOOSE_REDPANDA_CONFIG__SASL_PASSWORD | Your Redpanda password | (use AWS Secrets Manager) | +| MOOSE_REDPANDA_CONFIG__SASL_MECHANISM | SASL mechanism | SCRAM-SHA-256 | +| MOOSE_REDPANDA_CONFIG__SECURITY_PROTOCOL | The Redpanda security protocol | SASL_SSL | +| MOOSE_REDPANDA_CONFIG__REPLICATION_FACTOR | Topic replication factor | 3 | + +### Temporal Configuration (Optional) + +| Key | Description | Example Value | +|-----|-------------|---------------| +| MOOSE_TEMPORAL_CONFIG__CA_CERT | Path to CA certificate | /etc/ssl/certs/ca-certificates.crt | +| MOOSE_TEMPORAL_CONFIG__API_KEY | Temporal Cloud API key | (use AWS Secrets Manager) | +| MOOSE_TEMPORAL_CONFIG__TEMPORAL_HOST | Temporal Cloud namespace host | your-namespace.tmprl.cloud | + +Consider using a value of greater than 1000ms (1 second) for the Redpanda message timeout delay if you're using a hosted Redpanda cloud service. + +Review other options on the Task Creation page and press the `Create` button when ready. + +## Using AWS Secrets Manager + +For sensitive information like passwords and API keys, we recommend using AWS Secrets Manager. To configure a secret: + +1. Go to AWS Secrets Manager and create a new secret +2. Choose "Other type of secret" and add key-value pairs for your secrets +3. Name your secret appropriately (e.g., `moose/production/credentials`) +4. In your ECS task definition, reference the secret: + - For environment variables, select "ValueFrom" and enter the ARN of your secret with the key name + - Example: `arn:aws:secretsmanager:region:account:secret:moose/production/credentials:MOOSE_CLICKHOUSE_CONFIG__PASSWORD::` + +## Building an ECS Service + +Once you've completed creating an ECS Task, you're ready to create an ECS Service. An ECS Service is a definition that allows you to specify how your cluster will be managed. +Navigate to your cluster's Service page and press the `Create` button to create your new Moose service. +The section we're interested in is the `Deployment configuration` section. There, you'll specify the Task Definition you created earlier. You can also specify the name +of your service—perhaps something creative like `moose-service`—and the number of tasks to launch. + + + Note at this time, we recommend that you only launch a single instance of + Moose in your cluster. We're currently developing for multi-instance + concurrent usage. + + +The remaining sections on the create service page allow you to specify networking considerations and whether you'll use a load balancer. + +You can press the `Create` button to launch an instance of your new ECS Moose service. + +## Setting up health checks + +Your generated Moose containers include a health check endpoint at `/health` that should be configured in your ECS service. We recommend configuring the following health check settings: + +### Container-level Health Check + +In your task definition's container configuration: + +``` +healthCheck: + command: ["CMD-SHELL", "curl -f http://localhost:4000/health || exit 1"] + interval: 30 + timeout: 5 + retries: 3 + startPeriod: 60 +``` + +### Load Balancer Health Check + +If you're using an Application Load Balancer: + +1. Create a target group for your service +2. Set the health check path to `/health` +3. Configure appropriate health check settings: + - Health check protocol: HTTP + - Health check port: 4000 + - Health check path: /health + - Healthy threshold: 2 + - Unhealthy threshold: 2 + - Timeout: 5 seconds + - Interval: 15 seconds + - Success codes: 200 + +These health check configurations ensure that your Moose service is properly monitored and that traffic is only routed to healthy containers. diff --git a/apps/framework-docs-v2/content/moosestack/deploying/deploying-on-kubernetes.mdx b/apps/framework-docs-v2/content/moosestack/deploying/deploying-on-kubernetes.mdx index 6c32291383..5431f9e4b9 100644 --- a/apps/framework-docs-v2/content/moosestack/deploying/deploying-on-kubernetes.mdx +++ b/apps/framework-docs-v2/content/moosestack/deploying/deploying-on-kubernetes.mdx @@ -2,8 +2,306 @@ title: Kubernetes Deployment description: Deploy MooseStack on Kubernetes order: 4 +category: deployment --- -# Kubernetes Deployment +import { Callout } from "@/components/mdx"; -This page is a placeholder. Content migration pending. +# Deploying on Kubernetes + +Moose applications can be deployed to Kubernetes clusters, whether it's your own on-prem +cluster or through a cloud service like Google's Kubernetes Engine (GKE) or Amazon's +Elastic Kubernetes Service (EKS). + + + Note at this time, we recommend that you only launch a single instance of + moose in one cluster. We're currently developing for multi-instance concurrent + usage. + + +Essentially you'll need to create a moose-deployment YAML file. Here is an example: + +```yaml filename="moose-deployment.yaml-fragment" copy +apiVersion: apps/v1 +kind: Deployment +metadata: + name: moosedeployment +spec: + replicas: 1 + selector: + matchLabels: + app: moosedeploy + template: + metadata: + labels: + app: moosedeploy + spec: + containers: + - name: moosedeploy + image: 514labs/moose-df-deployment-x86_64-unknown-linux-gnu:latest + ports: + - containerPort: 4000 +``` + +> Make sure to update the image key above with the location of your repository and image tag. + +You may also need to configure a load balancer to route external traffic to your moose ingest points. + +```yaml filename="moose-lb-service.yaml" copy +apiVersion: v1 +kind: Service +metadata: + name: moose-service +spec: + selector: + app: moosedeploy + ports: + - protocol: TCP + port: 4000 + targetPort: 4000 + type: LoadBalancer +``` + +Another approach would be to use a service type of `ClusterIP`: + +```yaml filename="moose-service.yaml" copy +apiVersion: v1 +kind: Service +metadata: + name: moose-service +spec: + selector: + app: moosedeploy + type: ClusterIP + ports: + - protocol: TCP + port: 4000 + targetPort: 4000 +``` + +The approach you decide on will depend on your specific Kubernetes networking requirements. + +## Setting up health checks and probes + +Your generated Moose docker containers feature a health check endpoint at `/health` that can be used by Kubernetes to monitor the health of your application. Based on our production deployment, we recommend configuring the following probes: + +```yaml +# Startup probe - gives Moose time to initialize before accepting traffic +startupProbe: + httpGet: + path: /health + port: 4000 + initialDelaySeconds: 60 + timeoutSeconds: 3 + periodSeconds: 5 + failureThreshold: 30 + successThreshold: 3 + +# Readiness probe - determines when the pod is ready to receive traffic +readinessProbe: + httpGet: + path: /health + port: 4000 + initialDelaySeconds: 5 + timeoutSeconds: 3 + periodSeconds: 3 + failureThreshold: 2 + successThreshold: 5 + +# Liveness probe - restarts the pod if it becomes unresponsive +livenessProbe: + httpGet: + path: /health + port: 4000 + initialDelaySeconds: 5 + timeoutSeconds: 3 + periodSeconds: 5 + failureThreshold: 5 + successThreshold: 1 +``` + +## Zero-downtime deployments with lifecycle hooks + +For production deployments, we recommend configuring a preStop lifecycle hook to ensure graceful pod termination during updates: + +```yaml +lifecycle: + preStop: + exec: + command: ["/bin/sleep", "60"] +``` + +This gives the pod time to finish processing in-flight requests before termination. You should also set an appropriate + `terminationGracePeriodSeconds` value (we recommend 70 seconds) to work with this hook. + +## Resource requirements + +Based on our production deployments, we recommend the following resource allocation for a standard Moose deployment: + +```yaml +resources: + requests: + cpu: "1000m" + memory: "8Gi" +``` + +You can adjust these values based on your application's specific needs and workload. + +## Configuring container environment variables + +Inside your `moose-deployment.yaml` file, you will need to add an `env` section for environment variables. + + + The example below includes actual sample values for clarity. In production deployments, you should use Kubernetes secrets for sensitive information as shown in the second example. + + + + Note that both Redpanda and Temporal are optional. If you're not using these components, you can omit their respective configuration sections. + + +### Example with hardcoded values (for development/testing only): + + + The example below includes actual sample values for clarity. In production deployments, you should use Kubernetes secrets for sensitive information as shown in the second example. + + + + Note that both Redpanda and Temporal are optional. If you're not using these components, you can omit their respective configuration sections. + + +```yaml filename="moose-deployment-dev.yaml" copy +apiVersion: apps/v1 +kind: Deployment +metadata: + name: moosedeployment +spec: + # For zero-downtime deployments + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 1 + maxUnavailable: 0 + replicas: 1 + selector: + matchLabels: + app: moosedeploy + template: + metadata: + labels: + app: moosedeploy + spec: + # For graceful shutdowns + terminationGracePeriodSeconds: 70 + containers: + - name: moosedeploy + image: 514labs/moose-df-deployment-x86_64-unknown-linux-gnu:latest + ports: + - containerPort: 4000 + # Lifecycle hook to delay pod shutdown + lifecycle: + preStop: + exec: + command: ["/bin/sleep", "60"] + # Startup probe + startupProbe: + httpGet: + path: /health + port: 4000 + initialDelaySeconds: 60 + timeoutSeconds: 3 + periodSeconds: 5 + failureThreshold: 30 + successThreshold: 3 + # Readiness probe + readinessProbe: + httpGet: + path: /health + port: 4000 + initialDelaySeconds: 5 + timeoutSeconds: 3 + periodSeconds: 3 + failureThreshold: 2 + successThreshold: 5 + # Liveness probe + livenessProbe: + httpGet: + path: /health + port: 4000 + initialDelaySeconds: 5 + timeoutSeconds: 3 + periodSeconds: 5 + failureThreshold: 5 + successThreshold: 1 + # Resource requirements + resources: + requests: + cpu: "1000m" + memory: "8Gi" + env: + # Logger configuration + - name: MOOSE_LOGGER__LEVEL + value: "Info" + - name: MOOSE_LOGGER__STDOUT + value: "true" + - name: MOOSE_LOGGER__FORMAT + value: "Json" + # Telemetry configuration + - name: MOOSE_TELEMETRY__ENABLED + value: "true" + - name: MOOSE_TELEMETRY__EXPORT_METRICS + value: "true" + # Debugging + - name: RUST_BACKTRACE + value: "1" + # HTTP server configuration + - name: MOOSE_HTTP_SERVER_CONFIG__HOST + value: "0.0.0.0" + - name: MOOSE_HTTP_SERVER_CONFIG__PORT + value: "4000" + # ClickHouse configuration + - name: MOOSE_CLICKHOUSE_CONFIG__DB_NAME + value: "moose_production" + - name: MOOSE_CLICKHOUSE_CONFIG__USER + value: "clickhouse_user" + - name: MOOSE_CLICKHOUSE_CONFIG__PASSWORD + value: "clickhouse_password_example" + - name: MOOSE_CLICKHOUSE_CONFIG__HOST + value: "your-clickhouse.cloud.example.com" + - name: MOOSE_CLICKHOUSE_CONFIG__HOST_PORT + value: "8443" + - name: MOOSE_CLICKHOUSE_CONFIG__USE_SSL + value: "1" + - name: MOOSE_CLICKHOUSE_CONFIG__NATIVE_PORT + value: "9440" + # Redis configuration + - name: MOOSE_REDIS_CONFIG__URL + value: "redis://redis_user:redis_password_example@redis.example.com:6379" + - name: MOOSE_REDIS_CONFIG__KEY_PREFIX + value: "moose_production" + # Redpanda configuration (Optional) + - name: MOOSE_REDPANDA_CONFIG__BROKER + value: "seed-5fbcae97.example.redpanda.com:9092" + - name: MOOSE_REDPANDA_CONFIG__NAMESPACE + value: "moose_production" + - name: MOOSE_REDPANDA_CONFIG__MESSAGE_TIMEOUT_MS + value: "10043" + - name: MOOSE_REDPANDA_CONFIG__SASL_USERNAME + value: "redpanda_user" + - name: MOOSE_REDPANDA_CONFIG__SASL_PASSWORD + value: "redpanda_password_example" + - name: MOOSE_REDPANDA_CONFIG__SASL_MECHANISM + value: "SCRAM-SHA-256" + - name: MOOSE_REDPANDA_CONFIG__SECURITY_PROTOCOL + value: "SASL_SSL" + - name: MOOSE_REDPANDA_CONFIG__REPLICATION_FACTOR + value: "3" + # Temporal configuration (Optional) + - name: MOOSE_TEMPORAL_CONFIG__CA_CERT + value: "/etc/ssl/certs/ca-certificates.crt" + - name: MOOSE_TEMPORAL_CONFIG__API_KEY + value: "temporal_api_key_example" + - name: MOOSE_TEMPORAL_CONFIG__TEMPORAL_HOST + value: "your-namespace.tmprl.cloud" + imagePullSecrets: + - name: moose-docker-repo-credentials +``` diff --git a/apps/framework-docs-v2/content/moosestack/deploying/deploying-with-docker-compose.mdx b/apps/framework-docs-v2/content/moosestack/deploying/deploying-with-docker-compose.mdx index 2e1fb4148b..0c8e17aa82 100644 --- a/apps/framework-docs-v2/content/moosestack/deploying/deploying-with-docker-compose.mdx +++ b/apps/framework-docs-v2/content/moosestack/deploying/deploying-with-docker-compose.mdx @@ -2,8 +2,839 @@ title: Docker Compose Deployment description: Deploy MooseStack with Docker Compose order: 7 +category: deployment --- -# Docker Compose Deployment +import { Callout } from "@/components/mdx"; -This page is a placeholder. Content migration pending. +# Deploying with Docker Compose + +Deploying a Moose application with all its dependencies can be challenging and time-consuming. You need to properly configure multiple services, + ensure they communicate with each other, and manage their lifecycle. + + Docker Compose solves this problem by allowing you to deploy your entire stack with a single command. + +This guide shows you how to set up a production-ready Moose environment on a single server using Docker Compose, with proper security, +monitoring, and maintenance practices. + + + This guide describes a single-server deployment. For high availability (HA) deployments, you'll need to: + - Deploy services across multiple servers + - Configure service replication and redundancy + - Set up load balancing + - Implement proper failover mechanisms + + +We are also offering an HA managed deployment option for Moose called [Boreal](https://fiveonefour.com/boreal). + +## Prerequisites + +Before you begin, you'll need: + +- Ubuntu 24 or above (for this guide) +- Docker and Docker Compose (minimum version 2.23.1) +- Access to a server with at least 8GB RAM and 4 CPU cores + +The Moose stack consists of: +- Your Moose Application +- [ClickHouse](https://clickhouse.com) (required) +- [Redis](https://redis.io) (required) +- [Redpanda](https://redpanda.com) (optional for event streaming) +- [Temporal](https://temporal.io) (optional for workflow orchestration) + +## Setting Up a Production Server + +### Installing Required Software + +First, install Docker on your Ubuntu server: + +```bash +# Update the apt package index +sudo apt-get update + +# Install packages to allow apt to use a repository over HTTPS +sudo apt-get install -y \ + apt-transport-https \ + ca-certificates \ + curl \ + gnupg \ + lsb-release + +# Add Docker's official GPG key +curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg + +# Set up the stable repository +echo \ + "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \ + $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null + +# Update apt package index again +sudo apt-get update + +# Install Docker Engine +sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin +``` + +Next, install Node.js or Python depending on your Moose application: + +```bash +# For Node.js applications +curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash - +sudo apt-get install -y nodejs + +# OR for Python applications +sudo apt-get install -y python3.12 python3-pip +``` + +### Configure Docker Log Size Limits + +To prevent Docker logs from filling up your disk space, configure log rotation: + +```bash +sudo mkdir -p /etc/docker +sudo vim /etc/docker/daemon.json +``` + +Add the following configuration: + +```json +{ + "log-driver": "json-file", + "log-opts": { + "max-size": "100m", + "max-file": "3" + } +} +``` + +Restart Docker to apply the changes: + +```bash +sudo systemctl restart docker +``` + +### Enable Docker Non-Root Access + +To run Docker commands without sudo: + +```bash +# Add your user to the docker group +sudo usermod -aG docker $USER + +# Apply the changes (log out and back in, or run this) +newgrp docker +``` + +### Setting Up GitHub Actions Runner (Optional) + +If you want to set up CI/CD automation, you can install a GitHub Actions runner: + +1. Navigate to your GitHub repository +2. Go to Settings > Actions > Runners +3. Click "New self-hosted runner" +4. Select Linux and follow the instructions shown + +To configure the runner as a service (to run automatically): + +```bash +cd actions-runner +sudo ./svc.sh install +sudo ./svc.sh start +``` + +## Setting up a Foo Bar Moose Application (Optional) + +If you already have a Moose application, you can skip this section. +You should copy the moose project to your server and then build the application with the `--docker` flag and get the built image +on the server. + +### Install Moose CLI + +```bash +bash -i <(curl -fsSL https://fiveonefour.com/install.sh) moose +``` + +### Create a new Moose Application + +Please follow the initialization instructions for your language. + +```bash +moose init test-ts typescript +cd test-ts +npm install +``` + +or + +```bash +moose init test-py python +cd test-py +pip install -r requirements.txt +``` + +### Build the application on AMD64 + +```bash +moose build --docker --amd64 +``` + +### Build the application on ARM64 + +```bash +moose build --docker --arm64 +``` + +### Confirm the image was built + +```bash +docker images +``` + + +For more information on packaging Moose for deployment, see the full packaging guide. + + +## Preparing for Deployment + +### Create Environment Configuration + +First, create a file called `.env` in your project directory to specify component versions: + +```bash +# Create and open the .env file +vim .env +``` + +Add the following content to the `.env` file: + +``` +# Version configuration for components +POSTGRESQL_VERSION=14.0 +TEMPORAL_VERSION=1.22.0 +TEMPORAL_UI_VERSION=2.20.0 +REDIS_VERSION=7 +CLICKHOUSE_VERSION=25.4 +REDPANDA_VERSION=v24.3.13 +REDPANDA_CONSOLE_VERSION=v3.1.0 +``` + +Additionally, create a `.env.prod` file for your Moose application-specific secrets and configuration: + +```bash +# Create and open the .env.prod file +vim .env.prod +``` + +Add your application-specific environment variables: + +``` +# Application-specific environment variables +APP_SECRET=your_app_secret +# Add other application variables here +``` + +## Deploying with Docker Compose + +Create a file called `docker-compose.yml` in the same directory: + +```bash +# Create and open the docker-compose.yml file +vim docker-compose.yml +``` + +Add the following content to the file: + +```yaml file=./docker-compose.yml +name: moose-stack +volumes: + # Required volumes + clickhouse-0-data: null + clickhouse-0-logs: null + redis-0: null + # Optional volumes + redpanda-0: null + postgresql-data: null + +configs: + temporal-config: + # Using the "content" property to inline the config + content: | + limit.maxIDLength: + - value: 255 + constraints: {} + system.forceSearchAttributesCacheRefreshOnRead: + - value: true # Dev setup only. Please don't turn this on in production. + constraints: {} + +services: + # REQUIRED SERVICES + + # ClickHouse - Required analytics database + clickhouse-0: + container_name: clickhouse-0 + restart: always + image: clickhouse/clickhouse-server:${CLICKHOUSE_VERSION} + volumes: + - clickhouse-0-data:/var/lib/clickhouse/ + - clickhouse-0-logs:/var/log/clickhouse-server/ + environment: + # Enable SQL-driven access control and user management + CLICKHOUSE_ALLOW_INTROSPECTION_FUNCTIONS: 1 + # Default admin credentials + CLICKHOUSE_USER: admin + CLICKHOUSE_PASSWORD: adminpassword + # Disable default user + CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: 1 + # Database setup + CLICKHOUSE_DB: moose + # Uncomment this if you want to access clickhouse from outside the docker network + # ports: + # - 8123:8123 + # - 9000:9000 + healthcheck: + test: wget --no-verbose --tries=1 --spider http://localhost:8123/ping || exit 1 + interval: 30s + timeout: 5s + retries: 3 + start_period: 30s + ulimits: + nofile: + soft: 262144 + hard: 262144 + networks: + - moose-network + + # Redis - Required for caching and pub/sub + redis-0: + restart: always + image: redis:${REDIS_VERSION} + volumes: + - redis-0:/data + command: redis-server --save 20 1 --loglevel warning + healthcheck: + test: ["CMD", "redis-cli", "ping"] + interval: 10s + timeout: 5s + retries: 5 + networks: + - moose-network + + # OPTIONAL SERVICES + + # --- BEGIN REDPANDA SERVICES (OPTIONAL) --- + # Remove this section if you don't need event streaming + redpanda-0: + restart: always + command: + - redpanda + - start + - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092 + # Address the broker advertises to clients that connect to the Kafka API. + # Use the internal addresses to connect to the Redpanda brokers' + # from inside the same Docker network. + # Use the external addresses to connect to the Redpanda brokers' + # from outside the Docker network. + - --advertise-kafka-addr internal://redpanda-0:9092,external://localhost:19092 + - --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:18082 + # Address the broker advertises to clients that connect to the HTTP Proxy. + - --advertise-pandaproxy-addr internal://redpanda-0:8082,external://localhost:18082 + - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:18081 + # Redpanda brokers use the RPC API to communicate with each other internally. + - --rpc-addr redpanda-0:33145 + - --advertise-rpc-addr redpanda-0:33145 + # Mode dev-container uses well-known configuration properties for development in containers. + - --mode dev-container + # Tells Seastar (the framework Redpanda uses under the hood) to use 1 core on the system. + - --smp 1 + - --default-log-level=info + image: docker.redpanda.com/redpandadata/redpanda:${REDPANDA_VERSION} + container_name: redpanda-0 + volumes: + - redpanda-0:/var/lib/redpanda/data + networks: + - moose-network + healthcheck: + test: ["CMD-SHELL", "rpk cluster health | grep -q 'Healthy:.*true'"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 30s + + # Optional Redpanda Console for visualizing the cluster + redpanda-console: + restart: always + container_name: redpanda-console + image: docker.redpanda.com/redpandadata/console:${REDPANDA_CONSOLE_VERSION} + entrypoint: /bin/sh + command: -c 'echo "$$CONSOLE_CONFIG_FILE" > /tmp/config.yml; /app/console' + environment: + CONFIG_FILEPATH: /tmp/config.yml + CONSOLE_CONFIG_FILE: | + kafka: + brokers: ["redpanda-0:9092"] + # Schema registry config moved outside of kafka section + schemaRegistry: + enabled: true + urls: ["http://redpanda-0:8081"] + redpanda: + adminApi: + enabled: true + urls: ["http://redpanda-0:9644"] + ports: + - 8080:8080 + depends_on: + - redpanda-0 + healthcheck: + test: ["CMD", "wget", "--spider", "--quiet", "http://localhost:8080/admin/health"] + interval: 30s + timeout: 5s + retries: 3 + start_period: 10s + networks: + - moose-network + # --- END REDPANDA SERVICES --- + + # --- BEGIN TEMPORAL SERVICES (OPTIONAL) --- + # Remove this section if you don't need workflow orchestration + + # Temporal PostgreSQL database + postgresql: + container_name: temporal-postgresql + environment: + POSTGRES_PASSWORD: temporal + POSTGRES_USER: temporal + image: postgres:${POSTGRESQL_VERSION} + restart: always + volumes: + - postgresql-data:/var/lib/postgresql/data + healthcheck: + test: ["CMD-SHELL", "pg_isready -U temporal"] + interval: 10s + timeout: 5s + retries: 3 + networks: + - moose-network + + # Temporal server + # For initial setup, use temporalio/auto-setup image + # For production, switch to temporalio/server after first run + temporal: + container_name: temporal + depends_on: + postgresql: + condition: service_healthy + environment: + # Database configuration + - DB=postgres12 + - DB_PORT=5432 + - POSTGRES_USER=temporal + - POSTGRES_PWD=temporal + - POSTGRES_SEEDS=postgresql + # Namespace configuration + - DEFAULT_NAMESPACE=moose-workflows + - DEFAULT_NAMESPACE_RETENTION=72h + # Auto-setup options - set to false after initial setup + - AUTO_SETUP=true + - SKIP_SCHEMA_SETUP=false + # Service configuration - all services by default + # For high-scale deployments, run these as separate containers + # - SERVICES=history,matching,frontend,worker + # Logging and metrics + - LOG_LEVEL=info + # Addresses + - TEMPORAL_ADDRESS=temporal:7233 + - DYNAMIC_CONFIG_FILE_PATH=/etc/temporal/config/dynamicconfig/development-sql.yaml + # For initial deployment, use the auto-setup image + image: temporalio/auto-setup:${TEMPORAL_VERSION} + # For production, after initial setup, switch to server image: + # image: temporalio/server:${TEMPORAL_VERSION} + restart: always + ports: + - 7233:7233 + # Volume for dynamic configuration - essential for production + configs: + - source: temporal-config + target: /etc/temporal/config/dynamicconfig/development-sql.yaml + mode: 0444 + networks: + - moose-network + healthcheck: + test: ["CMD", "tctl", "--ad", "temporal:7233", "cluster", "health", "|", "grep", "-q", "SERVING"] + interval: 30s + timeout: 5s + retries: 3 + start_period: 30s + + # Temporal Admin Tools - useful for maintenance and debugging + temporal-admin-tools: + container_name: temporal-admin-tools + depends_on: + - temporal + environment: + - TEMPORAL_ADDRESS=temporal:7233 + - TEMPORAL_CLI_ADDRESS=temporal:7233 + image: temporalio/admin-tools:${TEMPORAL_VERSION} + restart: "no" + networks: + - moose-network + stdin_open: true + tty: true + + # Temporal Web UI + temporal-ui: + container_name: temporal-ui + depends_on: + - temporal + environment: + - TEMPORAL_ADDRESS=temporal:7233 + - TEMPORAL_CORS_ORIGINS=http://localhost:3000 + image: temporalio/ui:${TEMPORAL_UI_VERSION} + restart: always + ports: + - 8081:8080 + networks: + - moose-network + healthcheck: + test: ["CMD", "wget", "--spider", "--quiet", "http://localhost:8080/health"] + interval: 30s + timeout: 5s + retries: 3 + start_period: 10s + # --- END TEMPORAL SERVICES --- + + # Your Moose application + moose: + image: moose-df-deployment-x86_64-unknown-linux-gnu:latest # Update with your image name + depends_on: + # Required dependencies + - clickhouse-0 + - redis-0 + # Optional dependencies - remove if not using + - redpanda-0 + - temporal + restart: always + environment: + # Logging and debugging + RUST_BACKTRACE: "1" + MOOSE_LOGGER__LEVEL: "Info" + MOOSE_LOGGER__STDOUT: "true" + + # Required services configuration + # ClickHouse configuration + MOOSE_CLICKHOUSE_CONFIG__DB_NAME: "moose" + MOOSE_CLICKHOUSE_CONFIG__USER: "moose" + MOOSE_CLICKHOUSE_CONFIG__PASSWORD: "your_moose_password" + MOOSE_CLICKHOUSE_CONFIG__HOST: "clickhouse-0" + MOOSE_CLICKHOUSE_CONFIG__HOST_PORT: "8123" + + # Redis configuration + MOOSE_REDIS_CONFIG__URL: "redis://redis-0:6379" + MOOSE_REDIS_CONFIG__KEY_PREFIX: "moose" + + # Optional services configuration + # Redpanda configuration (remove if not using Redpanda) + MOOSE_REDPANDA_CONFIG__BROKER: "redpanda-0:9092" + MOOSE_REDPANDA_CONFIG__MESSAGE_TIMEOUT_MS: "1000" + MOOSE_REDPANDA_CONFIG__RETENTION_MS: "30000" + MOOSE_REDPANDA_CONFIG__NAMESPACE: "moose" + + # Temporal configuration (remove if not using Temporal) + MOOSE_TEMPORAL_CONFIG__TEMPORAL_HOST: "temporal:7233" + MOOSE_TEMPORAL_CONFIG__NAMESPACE: "moose-workflows" + + # HTTP Server configuration + MOOSE_HTTP_SERVER_CONFIG__HOST: 0.0.0.0 + ports: + - 4000:4000 + env_file: + - path: ./.env.prod + required: true + networks: + - moose-network + healthcheck: + test: ["CMD-SHELL", "curl -s http://localhost:4000/health | grep -q '\"unhealthy\": \\[\\]' && echo 'Healthy'"] + interval: 30s + timeout: 5s + retries: 10 + start_period: 60s + +# Define the network for all services +networks: + moose-network: + driver: bridge +``` + +At this point, don't start the services yet. First, we need to configure the individual services for production use as described in the following sections. + +## Configuring Services for Production + +### Configuring ClickHouse Securely (Required) + +For production ClickHouse deployment, we'll use environment variables to configure users and access control +(as recommended in the [official Docker image documentation](https://hub.docker.com/r/clickhouse/clickhouse-server)): + +1. First, start the ClickHouse container: + +```bash +# Start just the ClickHouse container +docker compose up -d clickhouse-0 +``` + +2. After ClickHouse has started, connect to create additional users: + +```bash +# Connect to ClickHouse with the admin user +docker exec -it clickhouse-0 clickhouse-client --user admin --password adminpassword + +# Create moose application user +CREATE USER moose IDENTIFIED BY 'your_moose_password'; +GRANT ALL ON moose.* TO moose; + +# Create read-only user for BI tools (optional) +CREATE USER power_bi IDENTIFIED BY 'your_powerbi_password' SETTINGS PROFILE 'readonly'; +GRANT SHOW TABLES, SELECT ON moose.* TO power_bi; +``` + +3. To exit the ClickHouse client, type `\q` and press Enter. + +4. Update your Moose environment variables to use the new moose user: + +```bash +vim docker-compose.yml +``` + +```yaml +MOOSE_CLICKHOUSE_CONFIG__USER: "moose" +MOOSE_CLICKHOUSE_CONFIG__PASSWORD: "your_moose_password" +``` + +5. Remove the following environement variables from the clickhouse service in the docker-compose.yml file: + +```yaml +MOOSE_CLICKHOUSE_CONFIG__USER: "admin" +MOOSE_CLICKHOUSE_CONFIG__PASSWORD: "adminpassword" +``` + +6. For additional security in production, consider using Docker secrets for passwords. + +7. Restart the ClickHouse container to apply the changes: + +```bash +docker compose restart clickhouse-0 +``` + +8. Verify that the new configuration works by connecting with the newly created user: + +```bash +# Connect with the new moose user +docker exec -it moose-stack-clickhouse-0-1 clickhouse-client --user moose --password your_moose_password + +# Test access by listing tables +SHOW TABLES FROM moose; + +# Exit the clickhouse client +\q +``` + +If you can connect successfully and run commands with the new user, your ClickHouse configuration is working properly. + +### Securing Redpanda (Optional) + +For production, it's recommended to restrict external access to Redpanda: + +1. Modify your Docker Compose file to remove external access: + - Use only internal network access for production + - If needed, use a reverse proxy with authentication for external access + +2. For this simple deployment, we'll keep Redpanda closed to the external world with no authentication required, + as it's only accessible from within the Docker network. + +### Configuring Temporal (Optional) + +If your Moose application uses Temporal for workflow orchestration, the configuration above includes all necessary services based on the +[official Temporal Docker Compose examples](https://github.com/temporalio/docker-compose). + +If you're not using Temporal, simply remove the Temporal-related services (postgresql, temporal, temporal-ui) +and environment variables from the docker-compose.yml file. + +#### Temporal Deployment Process: From Setup to Production + +Deploying Temporal involves a two-phase process: initial setup followed by production operation. Here are step-by-step instructions for each phase: + +##### Phase 1: Initial Setup + +1. **Start the PostgreSQL database**: + +```bash +docker compose up -d postgresql +``` + +2. **Wait for PostgreSQL to be healthy** (check the status): + +```bash +docker compose ps postgresql +``` + +Look for `healthy` in the output before proceeding. + +3. **Start Temporal with auto-setup**: + +```bash +docker compose up -d temporal +``` + +During this phase, Temporal's auto-setup will: +- Create the necessary PostgreSQL databases +- Initialize the schema tables +- Register the default namespace (moose-workflows) + +4. **Verify Temporal server is running**: + +```bash +docker compose ps temporal +``` + +5. **Start the Admin Tools and UI**: + +```bash +docker compose up -d temporal-admin-tools temporal-ui +``` + +6. **Create the namespace manually**: + +```bash +# Register the moose-workflows namespace with a 3-day retention period +docker compose exec temporal-admin-tools tctl namespace register --retention 72h moose-workflows +``` + +Verify that the namespace was created: + +```bash +# List all namespaces +docker compose exec temporal-admin-tools tctl namespace list + +# Describe your namespace +docker compose exec temporal-admin-tools tctl namespace describe moose-workflows +``` + +You should see details about the namespace including its retention policy. + +##### Phase 2: Transition to Production + +After successful initialization, modify your configuration for production use: + +1. **Stop Temporal services**: + +```bash +docker compose stop temporal temporal-ui temporal-admin-tools +``` + +2. **Edit your docker-compose.yml file** to: + - Change image from `temporalio/auto-setup` to `temporalio/server` + - Set `SKIP_SCHEMA_SETUP=true` + +Example change: +```yaml +# From: +image: temporalio/auto-setup:${TEMPORAL_VERSION} + +# To: +image: temporalio/server:${TEMPORAL_VERSION} + +# And change: +- AUTO_SETUP=true +- SKIP_SCHEMA_SETUP=false + +# To: +- AUTO_SETUP=false +- SKIP_SCHEMA_SETUP=true +``` + +3. **Restart services with production settings**: + +```bash +docker compose up -d temporal temporal-ui temporal-admin-tools +``` + +4. **Verify services are running with new configuration**: + +```bash +docker compose ps +``` + +## Starting and Managing the Service + +### Starting the Services + +Start all services with Docker Compose: + +```bash +docker compose up -d +``` + +### Setting Up Systemd Service for Docker Compose + +For production, create a systemd service to ensure Docker Compose starts automatically on system boot: + +1. Create a systemd service file: + +```bash +sudo vim /etc/systemd/system/moose-stack.service +``` + +2. Add the following configuration (adjust paths as needed): + +``` +[Unit] +Description=Moose Stack +Requires=docker.service +After=docker.service + +[Service] +Type=oneshot +RemainAfterExit=yes +WorkingDirectory=/path/to/your/compose/directory +ExecStart=/usr/bin/docker compose up -d +ExecStop=/usr/bin/docker compose down +TimeoutStartSec=0 + +[Install] +WantedBy=multi-user.target +``` + +3. Enable and start the service: + +```bash +sudo systemctl enable moose-stack.service +sudo systemctl start moose-stack.service +``` + +## Deployment Workflow + +You get a smooth deployment process with these options: + +### Automated Deployment with CI/CD + +1. Set up a CI/CD pipeline using GitHub Actions (if runner is configured) +2. When code is pushed to your repository: + - The GitHub Actions runner builds your Moose application + - Updates the Docker image + - Deploys using Docker Compose + +### Manual Deployment + +Alternatively, for manual deployment: + +1. Copy the latest version of the code to the machine +2. Run `moose build` +3. Update the Docker image tag in your docker-compose.yml +4. Restart the stack with `docker compose up -d` + +## Monitoring and Maintenance + +No more worrying about unexpected outages or performance issues. Set up proper monitoring: + +- Set up log monitoring with a tool like [Loki](https://grafana.com/oss/loki/) +- Regularly backup your volumes (especially ClickHouse data) +- Monitor disk space usage +- Set up alerting for service health diff --git a/apps/framework-docs-v2/content/moosestack/deploying/index.mdx b/apps/framework-docs-v2/content/moosestack/deploying/index.mdx index 3226a14f59..ae8d5f14be 100644 --- a/apps/framework-docs-v2/content/moosestack/deploying/index.mdx +++ b/apps/framework-docs-v2/content/moosestack/deploying/index.mdx @@ -1,9 +1,87 @@ --- title: Deploy description: Deploy MooseStack to production -order: 1 +order: 0 +category: deployment --- +import { Callout, LanguageTabs, LanguageTabContent } from "@/components/mdx"; + # Deploy -This page is a placeholder. Content migration pending. +## Overview + +Once you've finished developing your Moose application locally, the next step is to deploy your Moose app into production. You have two options: + +- Self-host your Moose application on your own servers +- Use the [Boreal Cloud hosting platform](https://www.fiveonefour.com/boreal) (from the makers of the Moose Stack) + + +Want this managed in production for you? Check out Boreal Cloud (from the makers of the Moose Stack). + + +## Getting Started With Self-Hosting + +Moose makes it easy to package and deploy your applications, whether you're deploying to a server with or without internet access. The deployment process is designed to be flexible and can accommodate both containerized and non-containerized environments. + +### Deployment Options + +1. **Kubernetes Deployment**: Deploy your application to Kubernetes clusters (GKE, EKS, AKS, or on-premises) +2. **Standard Server Deployment**: Deploy your application to a server with internet access +3. **Containerized Cloud Deployment**: Deploy to cloud services like AWS ECS or Google Cloud Run +4. **Offline Server Deployment**: Deploy to an environment without internet access + +### Key Deployment Steps + +There are three main aspects to deploying a Moose application: + + + +1. Setting up your build environment with Node.js and the Moose CLI +2. Building your application using `moose build` +3. Setting up your deployment environment with the necessary runtime dependencies (Node.js, Docker) and configuration + + +1. Setting up your build environment with Python and the Moose CLI +2. Building your application using `moose build` +3. Setting up your deployment environment with the necessary runtime dependencies (Python, Docker) and configuration + + + +## Configuring Your Deployment + +Based on our production experience, we recommend the following best practices for deploying Moose applications: + +### Health Monitoring + +Configure comprehensive health checks to ensure your application remains available: +- Startup probes to handle initialization +- Readiness probes for traffic management +- Liveness probes to detect and recover from deadlocks + +### Zero-Downtime Deployments + +Implement graceful termination and rolling updates: +- Pre-stop hooks to handle in-flight requests +- Appropriate termination grace periods +- Rolling update strategies that maintain service availability + +### Resource Allocation + +Properly size your deployments based on workload: +- CPU and memory requests tailored to your application +- Replicas scaled according to traffic patterns +- Horizontal scaling for high availability + +### Environment Configuration + +For any deployment type, you'll need to configure: + +1. Runtime environment variables for logging, telemetry, and application settings +2. External service connections (ClickHouse, Redpanda, Redis) +3. Network settings and security configurations +4. Application-specific configurations + +## Detailed Guides + +The following pages provide detailed guides for each deployment scenario, including step-by-step instructions for both Python and TypeScript applications and production-ready configuration templates. diff --git a/apps/framework-docs-v2/content/moosestack/deploying/monitoring.mdx b/apps/framework-docs-v2/content/moosestack/deploying/monitoring.mdx index f45e541316..39f84519d2 100644 --- a/apps/framework-docs-v2/content/moosestack/deploying/monitoring.mdx +++ b/apps/framework-docs-v2/content/moosestack/deploying/monitoring.mdx @@ -2,8 +2,13 @@ title: Monitoring (moved) description: Monitor MooseStack deployments order: 8 +category: deployment --- -# Monitoring (moved) +import { Callout } from "@/components/mdx"; -This page is a placeholder. Content migration pending. +# Monitoring + + +Monitoring documentation has been consolidated into the unified [Observability](/moosestack/metrics) page, which covers observability across both development and production environments, including metrics console, health checks, Prometheus, OpenTelemetry, logging, and error tracking. + diff --git a/apps/framework-docs-v2/content/moosestack/deploying/packaging-moose-for-deployment.mdx b/apps/framework-docs-v2/content/moosestack/deploying/packaging-moose-for-deployment.mdx index 7a8d192e95..caa54e5334 100644 --- a/apps/framework-docs-v2/content/moosestack/deploying/packaging-moose-for-deployment.mdx +++ b/apps/framework-docs-v2/content/moosestack/deploying/packaging-moose-for-deployment.mdx @@ -2,8 +2,78 @@ title: Packaging Moose for deployment description: Package MooseStack for deployment order: 1 +category: deployment --- -# Packaging Moose for deployment +# Packaging Moose for Deployment -This page is a placeholder. Content migration pending. +Once you've developed your Moose application locally, you can package it for deployment to your on-prem or cloud infrastructure. + +The first step is to navigate (`cd`) to your moose project in your terminal. + +```bash filename="Terminal" copy +cd my-moose-project +``` + +The Moose CLI you've used to build your Moose project also has a handy flag that will automate the packaging and building of your project into docker images. + +```bash filename="Terminal" copy +moose build --docker +``` + +After the above command completes you can view your newly created docker files by running the `docker images` command: + +```bash filename="Terminal" copy +>docker images +REPOSITORY TAG IMAGE ID CREATED SIZE +moose-df-deployment-aarch64-unknown-linux-gnu latest c50674c7a68a About a minute ago 155MB +moose-df-deployment-x86_64-unknown-linux-gnu latest e5b449d3dea3 About a minute ago 163MB +``` + +> Notice that you get two `moose-df-deployment` containers, one for the `aarch64` (ARM64) architecture and another for the `x86_64` architecture. This is necessary to allow you to choose the version that matches your cloud or on-prem machine architecture. + +You can then use standard docker commands to push your new project images to your container repository of choice. + +First tag your local images: + +```bash filename="Terminal" copy +docker tag moose-df-deployment-aarch64-unknown-linux-gnu:latest {your-repo-user-name}/moose-df-deployment-aarch64-unknown-linux-gnu:latest +docker tag moose-df-deployment-x86_64-unknown-linux-gnu:latest {your-repo-user-name}/moose-df-deployment-x86_64-unknown-linux-gnu:latest +``` + +Then `push` your files to your container repository. + +```bash filename="Terminal" copy +docker push {your-repo-user-name}/moose-df-deployment-aarch64-unknown-linux-gnu:latest +docker push {your-repo-user-name}/moose-df-deployment-x86_64-unknown-linux-gnu:latest +``` + +You can also use the following handy shell script to automate the steps above. + +```bash filename="push.sh" copy +#!/bin/bash + +version=$2 + +if [ -z "$1" ] +then + echo "You must specify the dockerhub repository as an argument. Example: ./push.sh container-repo-name" + echo "Note: you can also provide a second argument to supply a specific version tag - otherwise this script will use the same version as the latest moose-cli on Github." + exit 1 +fi + +if [ -z "$2" ] +then + output=$(npx @514labs/moose-cli -V) + version=$(echo "$output" | sed -n '2p' | awk '{print $2}') +fi + +echo "Using version: $version" +arch="moose-df-deployment-aarch64-unknown-linux-gnu" +docker tag $arch:$version $1/$arch:$version +docker push $1/$arch:$version + +arch="moose-df-deployment-x86_64-unknown-linux-gnu" +docker tag $arch:$version $1/$arch:$version +docker push $1/$arch:$version +``` diff --git a/apps/framework-docs-v2/content/moosestack/deploying/preparing-clickhouse-redpanda.mdx b/apps/framework-docs-v2/content/moosestack/deploying/preparing-clickhouse-redpanda.mdx index 7fbe3a5096..de8463bdb5 100644 --- a/apps/framework-docs-v2/content/moosestack/deploying/preparing-clickhouse-redpanda.mdx +++ b/apps/framework-docs-v2/content/moosestack/deploying/preparing-clickhouse-redpanda.mdx @@ -2,8 +2,124 @@ title: Preparing Infrastructure description: Prepare ClickHouse and Redpanda infrastructure order: 2 +category: deployment --- -# Preparing Infrastructure +import { Callout } from "@/components/mdx"; -This page is a placeholder. Content migration pending. +# Preparing access to ClickHouse, Redis, Temporal and Redpanda + +Your hosted Moose application requires access to hosted ClickHouse and Redis service instances. You can also optionally use Redpanda for event streaming. +You can stand up open source versions of these applications within your environments or opt to use cloud-hosted versions available at: + +- [ClickHouse Cloud](https://clickhouse.com) +- [Redis Cloud](https://redis.com) +- [Redpanda Cloud](https://redpanda.com) +- [Temporal Cloud](https://temporal.io) + +## ClickHouse Configuration + + +If you're using `state_config.storage = "clickhouse"` in your config (serverless mode without Redis), your ClickHouse instance must support the **KeeperMap** table engine. This is used for migration state storage and distributed locking. + +✅ **ClickHouse Cloud**: Supported by default +✅ **`moose dev` / `moose prod`**: Already configured in our Docker setup +⚠️ **Self-hosted ClickHouse**: See [ClickHouse KeeperMap documentation](https://clickhouse.com/docs/en/engines/table-engines/special/keeper-map) for setup requirements + +If you're using Redis for state storage (`state_config.storage = "redis"`), you don't need KeeperMap. + + +For ClickHouse, you'll need the following information: + +| Parameter | Description | Default Value | +|-----------|-------------|---------------| +| DB_NAME | Database name to use | Your branch or application ID | +| USER | Username for authentication | - | +| PASSWORD | Password for authentication | - | +| HOST | Hostname or IP address | - | +| HOST_PORT | HTTPS port | 8443 | +| USE_SSL | Whether to use SSL (1 for true, 0 for false) | 1 | +| NATIVE_PORT | Native protocol port | 9440 | + +These values are used to configure the Moose application's connection to ClickHouse through environment variables following this pattern: + +``` +MOOSE_CLICKHOUSE_CONFIG__= +``` + +For example: +``` +MOOSE_CLICKHOUSE_CONFIG__DB_NAME=myappdb +MOOSE_CLICKHOUSE_CONFIG__HOST=myclickhouse.example.com +MOOSE_CLICKHOUSE_CONFIG__USE_SSL=1 +MOOSE_CLICKHOUSE_CONFIG__HOST_PORT=8443 +MOOSE_CLICKHOUSE_CONFIG__NATIVE_PORT=9440 +``` + +## Redis Configuration + +Moose requires Redis for caching and as a message broker. You'll need the following configuration: + +| Parameter | Description | +|-----------|-------------| +| URL | Redis connection URL | +| KEY_PREFIX | Prefix for Redis keys to isolate namespaces | + +These values are configured through: +``` +MOOSE_REDIS_CONFIG__URL=redis://username:password@redis.example.com:6379 +MOOSE_REDIS_CONFIG__KEY_PREFIX=myapp +``` + +## Temporal Configuration (Optional) + +Temporal is an optional workflow orchestration platform that can be used with Moose. If you choose to use Temporal, you'll need the following configuration: + +| Parameter | Description | Default Value | +|-----------|-------------|---------------| +| CA_CERT | Path to CA certificate | /etc/ssl/certs/ca-certificates.crt | +| API_KEY | Temporal Cloud API key | - | +| TEMPORAL_HOST | Temporal Cloud namespace host | Your namespace + .tmprl.cloud | + +These values are configured through: +``` +MOOSE_TEMPORAL_CONFIG__CA_CERT=/etc/ssl/certs/ca-certificates.crt +MOOSE_TEMPORAL_CONFIG__API_KEY=your-temporal-api-key +MOOSE_TEMPORAL_CONFIG__TEMPORAL_HOST=your-namespace.tmprl.cloud +``` + +## Redpanda Configuration (Optional) + +Redpanda is an optional component that can be used for event streaming. If you choose to use Redpanda, you'll need the following information: + +| Parameter | Description | Default Value | +|-----------|-------------|---------------| +| BROKER | Bootstrap server address | - | +| NAMESPACE | Namespace for isolation (often same as branch or app ID) | - | +| MESSAGE_TIMEOUT_MS | Message timeout in milliseconds | 10043 | +| SASL_USERNAME | SASL username for authentication | - | +| SASL_PASSWORD | SASL password for authentication | - | +| SASL_MECHANISM | SASL mechanism | SCRAM-SHA-256 | +| SECURITY_PROTOCOL | Security protocol | SASL_SSL | +| REPLICATION_FACTOR | Topic replication factor | 3 | + +These values are used to configure the Moose application's connection to Redpanda through environment variables following this pattern: + +``` +MOOSE_REDPANDA_CONFIG__= +``` + +For example: +``` +MOOSE_REDPANDA_CONFIG__BROKER=seed-5fbcae97.example.redpanda.com:9092 +MOOSE_REDPANDA_CONFIG__NAMESPACE=myapp +MOOSE_REDPANDA_CONFIG__SECURITY_PROTOCOL=SASL_SSL +MOOSE_REDPANDA_CONFIG__SASL_MECHANISM=SCRAM-SHA-256 +MOOSE_REDPANDA_CONFIG__REPLICATION_FACTOR=3 +``` + +## Using Environment Variables in Deployment + +When deploying your Moose application, you'll need to pass these configurations as environment variables. +Refer to the deployment guides for your specific platform (Kubernetes, ECS, etc.) for details on how to securely +provide these values to your application. diff --git a/apps/framework-docs-v2/content/moosestack/getting-started/from-clickhouse.mdx b/apps/framework-docs-v2/content/moosestack/getting-started/from-clickhouse.mdx index aced50046c..e27b9c258d 100644 --- a/apps/framework-docs-v2/content/moosestack/getting-started/from-clickhouse.mdx +++ b/apps/framework-docs-v2/content/moosestack/getting-started/from-clickhouse.mdx @@ -2,12 +2,566 @@ title: Use with Existing ClickHouse description: Use MooseStack with your existing ClickHouse database order: 2 +category: getting-started --- -# Use with Existing ClickHouse +import { Callout, ToggleBlock, ZoomImg, MuxVideo, BulletPointsCard, IconBadge, CTACards, CTACard, LanguageTabs, LanguageTabContent, FileTree, Tabs, TabsList, TabsTrigger, TabsContent } from "@/components/mdx"; +import { PathConfig } from "@/lib/path-config"; -This page is a placeholder. Content migration pending. +# Use Moose with Your Existing ClickHouse +
+ + + +
+## What This Guide Does +This guide sets up a local ClickHouse development environment that mirrors your production database and enables code-first schema management: +1. **Introspect** your remote ClickHouse tables and generate TypeScript/Python data models +2. **Create** a local ClickHouse instance with your exact table schemas +3. **Seed** your local database with production data (optional) +4. **Build** APIs and pipelines on top of your ClickHouse data in code + +## How It Works + +**Local Development:** +- Your production ClickHouse remains untouched +- You get a local ClickHouse instance that copies your remote table schemas +- All development happens locally with hot-reload + +**Production Deployment:** +- When you deploy your code, it connects to your remote ClickHouse +- Any new tables, materialized views, or schema changes you create in code are automatically migrated to your target database +- Your existing data and tables remain intact + + + +## Prerequisites + + + + + + + + + + +
+ +## Step 1: Install Moose +Install the Moose CLI globally to your system: + +```bash filename="Terminal" copy +bash -i <(curl -fsSL https://fiveonefour.com/install.sh) moose +``` + + +After installation, you'll use `moose init` to create a new project that automatically connects to your ClickHouse and generates all the code you need. + + +## Step 2: Create Your Project + + +Use the ClickHouse Playground tab to try it out! + + + + +Your own ClickHouse +ClickHouse Playground + + + + + +```bash filename="Initialize new project" copy +# Option 1: Provide connection string directly +moose init my-project --from-remote --language typescript + +# Option 2: Run without connection string for interactive setup +moose init my-project --from-remote --language typescript +``` + + +```bash filename="Initialize new project" copy +# Option 1: Provide connection string directly +moose init my-project --from-remote --language python + +# Option 2: Run without connection string for interactive setup +moose init my-project --from-remote --language python +``` + + + +**Connection String Format:** +``` +https://username:password@host:port/?database=database_name +``` + + +If you don't provide a connection string, Moose will guide you through an interactive setup process where you'll be prompted to enter: +- **Host and port** (e.g., `https://your-service-id.region.clickhouse.cloud:8443`) +- **Username** (usually `default`) +- **Password** (your ClickHouse password) +- **Database name** (optional, defaults to `default`) + +This is perfect if you're not sure about your connection details or prefer a guided experience. + + + +Moose will create a complete project structure with: +- **Data models**: TypeScript/Python classes for every table in your ClickHouse +- **Type definitions**: Full type safety for all your data +- **Development environment**: Local ClickHouse instance that mirrors your production schema +- **Build tools**: Everything configured and ready to go + + +- Make sure you are using the `HTTPS` connection string, not the `HTTP` connection string. +- Make sure the port is correct. For `HTTPS` the default is `8443` +- The default username is `default` + + + + +See the section: Connect to your remote ClickHouse. + + + + + + + +```bash filename="Try using the ClickHouse Playground!" copy +moose init my-project --from-remote https://explorer:@play.clickhouse.com:443/?database=default --language typescript +``` + + +```bash filename="Initialize new project" copy +# Generate code models from your existing ClickHouse tables +moose init my-project --from-remote https://explorer:@play.clickhouse.com:443/?database=default --language python +``` + + + + + + +
+ + + +```bash filename="Install dependencies" copy +cd my-project +npm install +``` + + +```bash filename="Install dependencies" copy +cd my-project +python3 -m venv .venv +source .venv/bin/activate +pip install -r requirements.txt +``` + + + +You should see: `Successfully generated X models from ClickHouse tables` + +### Explore Your Generated Models + + +Check what Moose created from your tables in the `app/main.py` file: + + + + + + + +
Your generated table models are imported here so Moose can detect them.
+
Learn more about export pattern: local development / hosted
+ + + +Check what Moose created from your tables in the `app/index.ts` file: + + + + + + + +
Your generated table models are exported here so Moose can detect them.
+
Learn more about export pattern: local development / hosted
+ + + + +### If your database includes ClickPipes/PeerDB (CDC) tables + +As noted above, when you use `moose init --from-remote`, Moose introspects your database. If it detects CDC‑managed tables (e.g., PeerDB/ClickPipes with fields like `_peerdb_synced_at`, `_peerdb_is_deleted`, `_peerdb_version`), it marks those as `EXTERNALLY_MANAGED` and writes them into a dedicated external models file. Your root file is updated to load these models automatically. + + + + + + + + + + + + + + + + + + + + +This separation is a best‑effort by the CLI to keep clearly CDC‑owned tables external. For other tables you don't want Moose to manage, set the lifecycle to external and move them into the external file. See: +- [External Tables](/moosestack/olap/external-tables) documentation for more information on how external tables work. +- [DB Pull](/moosestack/olap/db-pull) for keeping models in sync with the remote schema. + +
+ +## Step 3: Start Development + +Start your development server. This spins up a local ClickHouse instance that perfectly mirrors your production schema: + +```bash filename="Start your dev server" copy +moose dev +``` + +**What happens when you run `moose dev`:** +- 🏗️ Creates a local ClickHouse instance with your exact table schemas in your project code +- 🔄 Hot-reloads migrations to your local infrastructure as you save code changes +- 🚀 Starts a web server for building APIs + + +Your production ClickHouse remains completely untouched. This is a separate, local development environment. + + + +```txt + Created docker compose file +⡗ Starting local infrastructure + Successfully started containers + Validated clickhousedb-1 docker container + Validated redpanda-1 docker container + Successfully validated red panda cluster + Validated temporal docker container + Successfully ran local infrastructure + Node Id: my-analytics-app::b15efaca-0c23-42b2-9b0c-642105f9c437 + Starting development mode + Watching "/path/to/my-analytics-app/app" + Started Webserver. + + + Next Steps + +💻 Run the moose 👉 `ls` 👈 command for a bird's eye view of your application and infrastructure + +📥 Send Data to Moose + Your local development server is running at: http://localhost:4000/ingest +``` + +Don't see this output? [Check out the troubleshooting section](#troubleshooting) + + +### Seed Your Local Database (Optional) +Copy real data from your production ClickHouse to your local development environment. This gives you realistic data to work with during development. + +**Why seed?** Your local database starts empty. Seeding copies real data so you can: +- Test with realistic data volumes +- Debug with actual production data patterns +- Develop features that work with real data structures + + + +From your own ClickHouse +From the ClickHouse Playground +Using env var + + +```bash filename="Terminal" copy +moose seed clickhouse --connection-string --limit 100000 +``` + +**Connection String Format:** +The connection string must use ClickHouse native protocol: +```bash +# ClickHouse native protocol (secure connection) +clickhouse://username:password@host:9440/database +``` + +**Note:** Data transfer uses ClickHouse's native TCP protocol via `remoteSecure()`. The remote server must have the native TCP port accessible. The command automatically handles table mismatches gracefully. + + +```bash filename="Terminal" copy +moose seed clickhouse --connection-string clickhouse://explorer:@play.clickhouse.com:9440/default --limit 100000 +``` + + +```bash filename="Terminal" copy +# You can omit --connection-string by setting an env var +export MOOSE_SEED_CLICKHOUSE_URL='clickhouse://username:password@host:9440/database' + +# copy a limited number of rows (batched under the hood) +moose seed clickhouse --limit 100000 +``` + + + + +- `--limit` and `--all` are mutually exclusive +- `--all` can be used to copy the entire table(s), use with caution as it can be very slow and computationally intensive. +- Large copies are automatically batched to avoid remote limits; you'll see per-batch progress. +- If you stop with Ctrl+C, the current batch finishes and the command exits gracefully. + +**Expected Output:** +```bash +✓ Database seeding completed +Seeded 'local_db' from 'remote_db' + +✓ table1: copied from remote +⚠️ table2: skipped (not found on remote) +✓ table3: copied from remote +``` + +**Troubleshooting:** +- Tables that don't exist on remote are automatically skipped with warnings +- Use `--table ` to seed a specific table that exists in both databases +- Check `moose ls table` to see your local tables + + +
+ +## Step 4: Build Your First API +Now that you have your data models, let's build something useful! You can create APIs, materialized views, and applications with full type safety. + + +- **REST APIs** that expose your ClickHouse data to frontend applications +- **Materialized Views** for faster queries and aggregations +- **Streaming pipelines** for real-time data processing +- **Full-stack applications** with your ClickHouse data as the backend + + +### Add APIs + +Build REST APIs on top of your existing tables to expose your data to your user-facing apps. This is a great way to get started with Moose without changing any of your existing pipelines. + + + +```ts copy +import { TargetSchema, materializedView } from "./myMaterializedView"; + + +export interface QueryParams { + date: Date; + limit?: number; +} + +const api = new Api("get-aggregated-data", + async ({ date, limit = 10 }: QueryParams, { client, sql }) => { + const query = sql` + SELECT * FROM ${materializedView.targetTable} + WHERE date = ${date} + LIMIT ${limit}`; + + const result = await client.query.execute(query); // Type-safe query execution + return result.json(); + } +); +``` + + + + +Check out the MooseAPI module for more information on building APIs with Moose. + + +### Build Materialized Views +Build materialized views on top of your existing tables to improve query performance. If you have Materialized Views in your ClickHouse, you can use Moose to build new Materialized Views on top of your existing tables, or to migrate your existing Materialized Views to Moose. + + + +```ts copy +// Data Model for your Materialized View- use type throughout your codebase +export interface TargetSchema { + date: Date; + count: number; +} +// Creates both the target table and the materialized view +export const materializedView = new MaterializedView({ + // Raw SQL with type-safe table references + selectStatement: sql`SELECT count(*) as count, date FROM ${myTable} GROUP BY date`, + selectTables: [myTable], // Automatic dependency resolution for cascading materialized views + engine: ClickHouseEngines.MergeTree, // Can use which ever engine you want + orderByFields: ["date"], + tableName: "myAggregation", + materializedViewName: "myAggregation_MV", +}); +``` + + + + + +Check out the MooseOLAP module for more information on building Materialized Views with Moose. + + + +
+ +## Known Limitations + + +Some advanced ClickHouse features may not be fully supported yet. Join the Moose Slack and let us know if you have any issues, feedback, or requests. + + +**What we're working on:** +- **Selective table import** (currently imports all tables) +- **Default value annotations** + +
+ +## Troubleshooting +### Error: Failed to connect to ClickHouse + +This guide shows exactly where to find your host, port, username, and password, and how to construct a valid HTTPS connection string. + + + +ClickHouse Cloud +Self-Hosted or Docker + + +1. Log into your [ClickHouse Cloud console](https://clickhouse.cloud/) +2. Open your service details page +3. Click "Connect" in the sidebar +
+ +
+4. Select the `HTTPS` tab and copy the values shown + - **Host**: e.g. `your-service-id.region.clickhouse.cloud` + - **Port**: usually `8443` + - **Username**: usually `default` + - **Password**: the password you configured +
+ +
+5. Build your connection string: + +```txt +https://USERNAME:PASSWORD@HOST:PORT/?database=DATABASE_NAME +``` + +6. Example (with placeholders): + +```txt +https://default:your_password@your-service-id.region.clickhouse.cloud:8443/?database=default +``` + +7. Optional: Test with curl + +```bash +curl --user "USERNAME:PASSWORD" --data-binary "SELECT 1" https://HOST:PORT +``` + + +### Self-hosted or Docker +- Check your server config (usually `/etc/clickhouse-server/config.xml`) + - `` default: `8123` + - `` default: `8443` +- Check users in `/etc/clickhouse-server/users.xml` or `users.d/` +- For Docker, check environment variables in your compose/run config: + - `CLICKHOUSE_USER`, `CLICKHOUSE_PASSWORD`, `CLICKHOUSE_DB` + +Build the HTTPS connection string with your values: + +```txt +https://USERNAME:PASSWORD@HOST:8443/?database=DB +``` + +If you only have HTTP enabled, enable HTTPS or use an HTTPS proxy; Moose init expects an HTTPS URL for remote introspection. + + + +### `moose dev` fails to start + +Double check Docker is running and you do not have any port conflicts. + +- ClickHouse local runs on port `18123` +- Your local webserver runs on port `4000` +- Your local management API runs on port `5001` + +## What's Next? + +Ready to build more? Check out the [APIs](/moosestack/apis) and [OLAP](/moosestack/olap) documentation to learn how to build powerful features on top of your ClickHouse data. diff --git a/apps/framework-docs-v2/content/moosestack/getting-started/quickstart.mdx b/apps/framework-docs-v2/content/moosestack/getting-started/quickstart.mdx index 453134bbc5..25cd1836c5 100644 --- a/apps/framework-docs-v2/content/moosestack/getting-started/quickstart.mdx +++ b/apps/framework-docs-v2/content/moosestack/getting-started/quickstart.mdx @@ -1,13 +1,771 @@ --- title: 5-Minute Quickstart -description: Quick start guide for MooseStack +description: Build your first analytical backend with Moose in 5 minutes order: 1 +category: getting-started --- + +import { Callout, ToggleBlock, MuxVideo, BulletPointsCard, IconBadge, CTACards, CTACard, LanguageTabs, LanguageTabContent, FileTree, Tabs, TabsList, TabsTrigger, TabsContent } from "@/components/mdx"; +import { PathConfig } from "@/lib/path-config"; + # 5-Minute Quickstart -This page is a placeholder. Content migration pending. +
+ + + +
+ + + + + + + + + + +## Prerequisites +Check that your pre-requisites are installed by running the following commands: + + + + + +```bash filename="Terminal" copy +node --version +``` + +```bash filename="Terminal" copy +docker ps +``` + + + + + +```bash filename="Terminal" copy +python --version +``` + +```bash filename="Terminal" copy +docker ps +``` + + + + +Make sure Docker Desktop has at least **2.5GB of memory allocated**. To check or change this setting, open Docker Desktop, go to Settings → Resources → Memory, and adjust the slider if needed. [Learn more about Docker Desktop settings →](https://docs.docker.com/desktop/settings/) + + + +Skip the tutorial and add Moose as a layer on top of your existing database + + +
+ +## Step 1: Install Moose (30 seconds) + + +### Run the installation script + +```bash filename="Terminal" copy +bash -i <(curl -fsSL https://fiveonefour.com/install.sh) moose +``` + +You should see this message: `Moose vX.X.X installed successfully!` (note that X.X.X is the actual version number) + +If you see an error instead, check [Troubleshooting](#need-help) below. + +### Reload your shell configuration + +**This step is required.** Your current terminal doesn't know about the `moose` command yet. + + + +Zsh (most macOS) +Bash (most Linux) + + + +```bash filename="Terminal" copy +source ~/.zshrc +``` + + + + +If `echo $SHELL` showed `/bin/bash` or `/usr/bin/bash`: + +```bash filename="Terminal" copy +source ~/.bashrc +``` + + + + +### Verify moose command works + +```bash filename="Terminal" copy +moose --version +``` + +You should see: +```txt +moose X.X.X +``` + + + +**Try these steps in order:** + +1. Re-run the correct `source` command for your shell +2. Close this terminal completely and open a new terminal window +3. Run `moose --version` again +4. If still failing, see [Troubleshooting](#need-help) + + + + +You should see the moose version number. Do not proceed to Step 2 until `moose --version` works. + + + + +
+ +## Step 2: Create Your Project (1 minute) + + +### Initialize your project + + + +```bash filename="Terminal" copy +moose init my-analytics-app typescript +``` + +You should see output like: +```txt +✓ Created my-analytics-app +✓ Initialized TypeScript project +``` + + +```bash filename="Terminal" copy +moose init my-analytics-app python +``` + +You should see output like: +```txt +✓ Created my-analytics-app +✓ Initialized Python project +``` + + + +### Navigate to your project directory + +```bash filename="Terminal" copy +cd my-analytics-app +``` + + + + +A virtual environment isolates your project's dependencies. We recommend creating one for your project. + + +**Create a virtual environment (Recommended)** + +```bash filename="Terminal" copy +python3 -m venv .venv +``` + +**activate your virtual environment(Recommended)** +```bash filename="Terminal" copy +source .venv/bin/activate +``` + +This creates a `.venv` folder and activates it. Your terminal prompt should now look something like this: + +```txt +(.venv) username@computer my-analytics-app % +``` + + +### Install dependencies + + + + +```bash filename="Terminal" copy +npm install +``` + +**Wait for installation to complete.** + + +Dependencies installed successfully with no errors. + + + +```bash filename="Terminal" copy +pip install -r requirements.txt +``` + +**Wait for installation to complete.** You should see successful installation messages ending with: +```txt +Successfully installed [list of packages] +``` + + +You should see `(.venv)` in your prompt and dependencies installed with no errors. + + + + +### Start your development environment + +```bash filename="Terminal" copy +moose dev +``` + + +Moose is: +- Downloading Docker images for ClickHouse, Redpanda, and Temporal +- Starting containers +- Initializing databases +- Starting the development server + + +Do not proceed until you see the "Started Webserver" message. + +```txt + Created docker compose file +⡗ Starting local infrastructure + Successfully started containers + Validated clickhousedb-1 docker container + Validated redpanda-1 docker container + Successfully validated red panda cluster + Validated temporal docker container + Successfully ran local infrastructure + Node Id: my-analytics-app::b15efaca-0c23-42b2-9b0c-642105f9c437 + Starting development mode + Watching "/path/to/my-analytics-app/app" + Started Webserver. 👈 WAIT FOR THIS + + + Next Steps + +💻 Run the moose 👉 `ls` 👈 command for a bird's eye view of your application and infrastructure + +📥 Send Data to Moose + Your local development server is running at: http://localhost:4000/ingest +``` + + +Keep this terminal running. This is your Moose development server. You'll open a new terminal for the next step. + + + + +## Step 3: Understand Your Project (1 minute) + +Your project includes a complete example pipeline: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +**Important:** While your pipeline objects are defined in the child folders, they **must be imported** into the root `index.ts` (TypeScript) or `main.py` (Python) file for the Moose CLI to discover and use them. + + + +```ts filename="app/index.ts" +export * from "./ingest/models"; // Data models & pipelines +export * from "./ingest/transforms"; // Transformation logic +export * from "./apis/bar"; // API endpoints +export * from "./views/barAggregated"; // Materialized views +export * from "./workflows/generator"; // Background workflows +``` + + +```python filename="app/main.py" +from app.ingest.models import * # Data models & pipelines +from app.ingest.transform import * # Transformation logic +from app.apis.bar import * # API endpoints +from app.views.bar_aggregated import * # Materialized views +from app.workflows.generator import * # Background workflows +``` + + + + +## Step 4: Test Your Pipeline (2 minutes) + + +**Keep your `moose dev` terminal running.** You need a second terminal for the next commands. + + + +**macOS Terminal:** +- Press `Cmd+N` for a new window, or +- Right-click Terminal icon in dock → New Window + +**VSCode:** +- Click the `+` button in the terminal panel, or +- Press `Ctrl+Shift+` ` (backtick) + +**Linux Terminal:** +- Press `Ctrl+Shift+N`, or +- Use your terminal's File → New Window menu + + + + + +### Navigate to your project in the new terminal + +In your **new terminal window** (not the one running `moose dev`): + +```bash filename="Terminal 2 (New Window)" copy +cd my-analytics-app +``` + + + +If not automatically activated, activate the virtual environment: +```bash filename="Terminal 2 (New Window)" copy +source .venv/bin/activate +``` + + + + +### Run the data generator workflow + +Your project comes with a pre-built [Workflow](/moosestack/workflows) called `generator` that acts as a **data simulator**: + +```bash filename="Terminal 2 (New Window)" copy +moose workflow run generator +``` + +You should see: +```txt +Workflow 'generator' triggered successfully +``` + + + + +- Generates 1000 fake records with realistic data (using the Faker library) +- Sends each record to your ingestion API via HTTP POST +- Runs as a background task managed by Temporal +- Helps you test your entire pipeline without needing real data + +You can see the code in the `/workflows/generator.ts` (TypeScript) or `/workflows/generator.py` (Python) file. + + + + +### Watch for data processing logs + +**Switch to your first terminal** (where `moose dev` is running). You should see new logs streaming: + +```txt +POST ingest/Foo +[POST] Data received at ingest API sink for Foo +Received Foo_0_0 -> Bar_0_0 1 message(s) +[DB] 17 row(s) successfully written to DB table (Bar) +``` + + +These logs show your pipeline working: Workflow generates data → Ingestion API receives it → Data transforms → Writes to ClickHouse + + +**If you don't see logs after 30 seconds:** +- Verify `moose dev` is still running in Terminal 1 +- Check Terminal 2 for error messages from the workflow command +- Run `docker ps` to verify containers are running + + +The workflow runs in the background, powered by [Temporal](https://temporal.io). You can see workflow status at `http://localhost:8080`. + + + + +```bash filename="Terminal" copy +moose peek Bar --limit 5 # This queries your Clickhouse database to show raw data; useful for debugging / verification +``` + +You should see output like: +```txt +┌─primaryKey─────────────────────────┬─utcTimestamp────────┬─hasText─┬─textLength─┐ +│ 123e4567-e89b-12d3-a456-426614174000 │ 2024-01-15 10:30:00 │ 1 │ 42 │ +│ 987fcdeb-51a2-43d1-b789-123456789abc │ 2024-01-15 10:31:00 │ 0 │ 0 │ +└────────────────────────────────────┴─────────────────────┴─────────┴────────────┘ +``` + +If you see 0 rows, wait a few seconds for the workflow to process data, then try again. + + + +### Query your data + +Your application has a pre-built [API](/moosestack/apis) that reads from your database. The API runs on `localhost:4000`. + +**In Terminal 2**, call the API with `curl`: + +```bash filename="Terminal 2 (New Window)" copy +curl "http://localhost:4000/api/bar" +``` + +You should see JSON data like: +```json +[ + { + "dayOfMonth": 15, + "totalRows": 67, + "rowsWithText": 34, + "maxTextLength": 142, + "totalTextLength": 2847 + }, + { + "dayOfMonth": 14, + "totalRows": 43, + "rowsWithText": 21, + "maxTextLength": 98, + "totalTextLength": 1923 + } +] +``` + + +You should see JSON data with analytics results. Your complete data pipeline is working! + + +**Try query parameters:** +```bash filename="Terminal 2 - Add filters and limits" copy +curl "http://localhost:4000/api/bar?limit=5&orderBy=totalRows" +``` + + +- **Port 4000**: Your Moose application webserver (all APIs are running on this port) +- **Port 8080**: Temporal UI dashboard (workflow management) +- **Port 18123**: ClickHouse HTTP interface (direct database access) + + + +**If the workflow command doesn't work:** +- Make sure you're in the project directory (`cd my-analytics-app`) +- Verify `moose dev` is still running in your first terminal +- Check that Docker containers are running: `docker ps` + +**If curl returns an error:** +- Verify the URL is `http://localhost:4000` (not 8080) +- Make sure the workflow has had time to generate data (wait 30-60 seconds) +- Check your `moose dev` terminal for error messages + +**If you get HTML instead of JSON:** +- You might be hitting the wrong port - use 4000, not 8080 +- Port 8080 serves the Temporal UI (workflow dashboard), not your API + +**If `moose peek Bar` shows 0 rows:** +- Wait for the workflow to complete (it processes 1000 records) +- Check the workflow is running: look for "Ingested X records..." messages +- Verify no errors in your `moose dev` terminal logs + +**If you see connection refused:** +- Restart `moose dev` and wait for "Started Webserver" message +- Check if another process is using port 4000: `lsof -i :4000` + + + + +1. Install the [OpenAPI (Swagger) Viewer extension](https://marketplace.cursorapi.com/items?itemName=42Crunch.vscode-openapi) in your IDE +2. Open `.moose/openapi.yaml` in your IDE +3. Click the "Preview" icon to launch the interactive API explorer +4. Test the `POST /ingest/Foo` and `GET /api/bar` endpoints + + + + + + + +## Step 5: Hot Reload Schema Changes (1 minute) + +1. Open `app/ingest/models.ts` (TypeScript) or `app/ingest/models.py` (Python) +2. Add a new field to your data model: + + + +```ts filename="app/ingest/models.ts" {5} copy +/** Analyzed text metrics derived from Foo */ +export interface Bar { + primaryKey: Key; // From Foo.primaryKey + utcTimestamp: DateTime; // From Foo.timestamp + hasText: boolean; // From Foo.optionalText? + textLength: number; // From Foo.optionalText.length + newField?: string; // Add this new optional field +} +``` + + +```python filename="app/ingest/models.py" {16} copy +from moose_lib import Key, StringToEnumMixin +from typing import Optional, Annotated +from enum import IntEnum, auto +from pydantic import BaseModel + +class Baz(StringToEnumMixin, IntEnum): + QUX = auto() + QUUX = auto() + + +class Bar(BaseModel): + primary_key: Key[str] + utc_timestamp: datetime + baz: Baz + has_text: bool + text_length: int + new_field: Optional[str] = None # New field +``` + + + +3. Save the file and watch your terminal + +**Switch to Terminal 1** (where `moose dev` is running). You should see Moose automatically update your infrastructure: + + + +```txt +⠋ Processing Infrastructure changes from file watcher + ~ Table Bar: + Column changes: + + newField: String +``` + + +```txt +⠋ Processing Infrastructure changes from file watcher + ~ Table Bar: + Column changes: + + new_field: String +``` + + + + +You should see the column change logged. Your API, database schema, and streaming topic all updated automatically! + + +**Try it yourself:** Add another field with a different data type and watch the infrastructure update in real-time. + +## Recap + +You've built a complete analytical backend with: + + + +## Need Help? + + + + +Docker Issues +Version Issues +Port Conflicts +Permission Issues + + +**Docker not running:** +```bash filename="Terminal" copy +# macOS +open -a Docker + +# Linux +sudo systemctl start docker + +# Verify Docker is running +docker ps +``` + +**Docker out of space:** +```bash filename="Terminal" copy +docker system prune -a +``` + + + + + +**Node.js version too old:** +```bash filename="Terminal" copy +# Check version +node -v + +# Install Node 20+ with nvm +curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash +nvm install 20 +nvm use 20 +``` + + +**Python version too old:** +```bash filename="Terminal" copy +# Check version +python3 --version + +# Install Python 3.12+ with pyenv +curl https://pyenv.run | bash +pyenv install 3.12 +pyenv local 3.12 +``` + + + + + +**Port 4000 already in use:** +```bash filename="Terminal" copy +# Find what's using port 4000 +lsof -i :4000 + +# Kill the process (replace PID) +kill -9 + +# Or use a different port +moose dev --port 4001 +``` + + +**Permission denied:** +```bash filename="Terminal" copy +# Fix Docker permissions (Linux) +sudo usermod -aG docker $USER +newgrp docker +# Fix file permissions +chmod +x ~/.moose/bin/moose +``` + + +**Still stuck?** Join our [Slack community](https://join.slack.com/t/moose-community/shared_invite/zt-2fjh5n3wz-cnOmM9Xe9DYAgQrNu8xKxg) or [open an issue](https://github.com/514-labs/moose/issues). + diff --git a/apps/framework-docs-v2/content/moosestack/help/index.mdx b/apps/framework-docs-v2/content/moosestack/help/index.mdx index 9834ae5341..8805f05203 100644 --- a/apps/framework-docs-v2/content/moosestack/help/index.mdx +++ b/apps/framework-docs-v2/content/moosestack/help/index.mdx @@ -1,9 +1,72 @@ --- title: Help description: Get help with MooseStack -order: 1 +order: 0 +category: help --- +import { CTACards, CTACard } from "@/components/mdx"; + # Help -This page is a placeholder. Content migration pending. +Need assistance with MooseStack? You've come to the right place. This section provides resources to help you get started, troubleshoot issues, and find answers to common questions. + +## Quick Links + + + + + + +## Common Topics + +### Development Environment +- Port conflicts and how to resolve them +- Docker setup and configuration +- Missing dependencies troubleshooting + +### Data Ingestion +- Data validation errors +- Stream processing issues +- Database connectivity problems + +### Performance +- Slow query optimization +- Stream parallelism tuning +- Database indexing strategies + +### Deployment +- Configuration validation +- Resource allocation +- Migration lock issues + +## Community Resources + +- **Slack Community**: Join our [Moose community Slack channel](https://join.slack.com/t/moose-community/shared_invite/zt-2fjh5n3wz-cnOmM9Xe9DYAgQrNu8xKxg) for real-time help +- **GitHub Issues**: Search or create issues on [GitHub](https://github.com/514-labs/moose/issues) +- **Documentation**: Browse our comprehensive documentation for detailed guides + +## Still Need Help? + +If you can't find what you're looking for: + +1. Check the [Troubleshooting Guide](/moosestack/help/troubleshooting) for common issues +2. Search existing [GitHub issues](https://github.com/514-labs/moose/issues) for similar problems +3. Join our [Slack community](https://join.slack.com/t/moose-community/shared_invite/zt-2fjh5n3wz-cnOmM9Xe9DYAgQrNu8xKxg) and ask for help +4. Open a new GitHub issue with: + - Moose version (`moose --version`) + - Error messages and logs + - Steps to reproduce + - Expected vs. actual behavior diff --git a/apps/framework-docs-v2/content/moosestack/help/minimum-requirements.mdx b/apps/framework-docs-v2/content/moosestack/help/minimum-requirements.mdx index 8291904fb9..ccd211afe7 100644 --- a/apps/framework-docs-v2/content/moosestack/help/minimum-requirements.mdx +++ b/apps/framework-docs-v2/content/moosestack/help/minimum-requirements.mdx @@ -2,8 +2,33 @@ title: Minimum Requirements description: Minimum system requirements for MooseStack order: 2 +category: help --- # Minimum Requirements -This page is a placeholder. Content migration pending. +## Development Setup + +The development setup has higher requirements because Moose runs locally along with all its dependencies (Redpanda, ClickHouse, Temporal, Redis). + +- **CPU:** 12 cores +- **Memory:** 18GB +- **Disk:** >500GB SSD +- **OS:** + - Windows with Linux subsystem (Ubuntu preferred) + - Linux (Debian 10+, Ubuntu 18.10+, Fedora 29+, CentOS/RHEL 8+) + - Mac OS 13+ +- **Runtime:** Python 3.12+ or Node.js 20+, Docker 24.0.0+, and Docker Compose 2.23.1+ + +## Production Setup + +The production setup has lower requirements, as external components (Redpanda, ClickHouse, Redis, and Temporal) are assumed to be deployed separately. + +- **CPU:** 1vCPU +- **Memory:** 6GB +- **Disk:** >30GB SSD +- **OS:** + - Windows with Linux subsystem (Ubuntu preferred) + - Linux (Debian 10+, Ubuntu 18.10+, Fedora 29+, CentOS/RHEL 8+) + - Mac OS 13+ +- **Runtime:** Python 3.12+ or Node.js 20+ diff --git a/apps/framework-docs-v2/content/moosestack/help/troubleshooting.mdx b/apps/framework-docs-v2/content/moosestack/help/troubleshooting.mdx index 6015b42cbe..6df84df345 100644 --- a/apps/framework-docs-v2/content/moosestack/help/troubleshooting.mdx +++ b/apps/framework-docs-v2/content/moosestack/help/troubleshooting.mdx @@ -2,8 +2,273 @@ title: Troubleshooting description: Troubleshoot common issues order: 1 +category: help --- +import { Callout, LanguageTabs, LanguageTabContent } from "@/components/mdx"; + # Troubleshooting -This page is a placeholder. Content migration pending. +Common issues and their solutions when working with Moose. + +## Development Environment + +### Issue: `moose dev` fails to start + +**Possible causes and solutions:** + +1. **Port conflicts** + - Check if ports 4000-4002 are already in use + - Solution: Kill the conflicting processes or configure different ports + + ```bash + # Find processes using ports + lsof -i :4000-4002 + + # Kill process by PID + kill + ``` + +2. **Missing dependencies** + - Solution: Ensure all dependencies are installed + + + + ```bash + npm install + ``` + + + ```bash + pip install . + ``` + + + +3. **Docker not running** + - Solution: Start Docker Desktop or Docker daemon + + ```bash + # Check Docker status + docker info + + # Start Docker on Linux + sudo systemctl start docker + ``` + +## Data Ingestion + +### Issue: Data not appearing in tables + +1. **Validation errors** + - Check logs for validation failures + - Solution: Ensure data matches schema + + ```bash filename="Terminal" copy + moose logs + ``` + +2. **Stream processing errors** + - Solution: Check transform functions for errors + + ```bash filename="Terminal" copy + moose logs --filter functions + ``` + +3. **Database connectivity** + - Solution: Verify database credentials in `.moose/config.toml` + +```toml filename=".moose/config.toml" copy +[clickhouse_config] +db_name = "local" +user = "panda" +password = "pandapass" +use_ssl = false +host = "localhost" +host_port = 18123 +native_port = 9000 +``` + +## Stream Processing + +### Issue: High processing latency + +1. **Insufficient parallelism** + - Solution: Increase stream parallelism + + + + ```typescript + const stream = new Stream("high_volume", { + parallelism: 8 // Increase from default + }); + ``` + + + ```python + from moose_lib import Stream, StreamConfig + + stream = Stream[Data]("high_volume", StreamConfig(parallelism=8)) + ``` + + + +### Issue: Data transformations not working + +1. **Transform function errors** + - Solution: Debug transformation logic + + + + ```typescript + // Add logging to transform + stream.addTransform(outputStream, (record) => { + console.log('Processing record:', record.id); + try { + // Your transformation logic + return transformedRecord; + } catch (error) { + console.error('Transform error:', error); + return undefined; // Skip record on error + } + }); + ``` + + + ```python + # Add logging to transform + def transform(record: Data) -> Data: + print(f"Processing record: {record.id}") + try: + # Your transformation logic + return transformed_record + except Exception as e: + print(f"Transform error: {e}") + return None # Skip record on error + ``` + + + +## Database Issues + +### Issue: Slow queries + +1. **Missing or improper indexes** + - Solution: Check orderByFields configuration + + + + ```typescript + const table = new OlapTable("slow_table", { + orderByFields: ["frequently_queried_field", "timestamp"] + }); + ``` + + + ```python + from moose_lib import OlapTable, OlapConfig + + table = OlapTable[Data]("slow_table", OlapConfig( + order_by_fields=["frequently_queried_field", "timestamp"] + )) + ``` + + + +2. **Large result sets** + - Solution: Add limits and pagination + + + + ```typescript + // In query API + const results = await client.query.execute(sql` + SELECT * FROM large_table + WHERE category = 'example' + LIMIT 100 + `); + ``` + + + ```python + # In query API + results = client.query.execute( + # not an f-string, the values are provided in the dict + """ + SELECT * FROM large_table + WHERE category = {category} + LIMIT {limit} + """, {"category": "example", "limit": 100} + ) + ``` + + + +## Deployment Issues + +### Issue: Deployment fails + +1. **Configuration errors** + - Solution: Check deployment configuration + + ```bash + # Validate configuration + moose validate --config + ``` + +2. **Resource limitations** + - Solution: Increase resource allocation + + ```yaml + # In kubernetes manifest + resources: + requests: + memory: "1Gi" + cpu: "500m" + limits: + memory: "2Gi" + cpu: "1000m" + ``` + +3. **Permission issues** + - Solution: Verify service account permissions + + ```bash + # Check permissions + moose auth check + ``` + +### Issue: Migration stuck with "Migration already in progress" + +**Cause:** A previous migration was interrupted without releasing its lock. + +**Solution:** + +1. **Wait 5 minutes** - locks expire automatically + +2. **Or manually clear the lock:** + ```sql + DELETE FROM _MOOSE_STATE WHERE key = 'migration_lock'; + ``` + +3. **Verify it worked:** + ```sql + SELECT * FROM _MOOSE_STATE WHERE key = 'migration_lock'; + -- Should return no rows + ``` + + +The `_MOOSE_STATE` table uses ClickHouse's KeeperMap engine for distributed locking, ensuring only one migration runs at a time across multiple deployments. + + +## Getting Help + +If you can't resolve an issue: + +1. Ask for help on the [Moose community Slack channel](https://join.slack.com/t/moose-community/shared_invite/zt-2fjh5n3wz-cnOmM9Xe9DYAgQrNu8xKxg) +2. Search existing [GitHub issues](https://github.com/514-labs/moose/issues) +3. Open a new issue with: + - Moose version (`moose --version`) + - Error messages and logs + - Steps to reproduce + - Expected vs. actual behavior diff --git a/apps/framework-docs-v2/content/moosestack/in-your-stack.mdx b/apps/framework-docs-v2/content/moosestack/in-your-stack.mdx index 7ff67da379..0003160259 100644 --- a/apps/framework-docs-v2/content/moosestack/in-your-stack.mdx +++ b/apps/framework-docs-v2/content/moosestack/in-your-stack.mdx @@ -4,10 +4,108 @@ description: Integrating MooseStack into your application stack order: 6 --- -# In Your Stack +import { ZoomImg, Callout } from "@/components/mdx"; -This page is a placeholder. Content migration pending. +# Moose In Your Dev Stack +Moose handles the analytical layer of your application stack. The [Area Code](https://github.com/514-labs/area-code) repository contains two working implementations that show how to integrate Moose with existing applications. +## User Facing Analytics (UFA) + +UFA shows how to add a dedicated analytics microservice to an existing application without impacting your primary database. + + +View the open source repository to see the full implementation and clone it on your own machine. + + +### UFA Data Flow + +1. Application writes to Supabase (transactional backend) +2. Supabase Realtime streams changes to Analytical Backend and Retrieval Backend +3. Moose ingest pipeline syncs change events from Redpanda into ClickHouse +4. Frontend queries analytics APIs for dashboards + +### UFA Architecture Components + +The UFA template demonstrates a microservices architecture with specialized components for different data access patterns: + +
+ + +The user interface for dashboards and application interactions + +Technologies: [Vite](https://vite.dev), [React](https://react.dev), [TanStack Query](https://tanstack.com/query), [TanStack Router](https://tanstack.com/router), [Tailwind CSS](https://tailwindcss.com) + + + +Handles CRUD operations and maintains application state + +Technologies: [Supabase](https://supabase.com), [Fastify](https://fastify.dev), [Drizzle ORM](https://orm.drizzle.team/) + + + +Fast text search and complex queries across large datasets + +Technologies: [Elasticsearch](https://www.elastic.co/) + [Fastify](https://fastify.dev) + + + +High-performance analytical queries and aggregations + +Technologies: [ClickHouse](https://clickhouse.com/) + [Moose OLAP](/moosestack/olap), [Redpanda](https://redpanda.com/) + [Moose Streaming](/moosestack/streaming), [Moose APIs](/moosestack/apis) + + + +Keep data synchronized between transactional, retrieval, and analytics systems + +Technologies: [Supabase Realtime](https://supabase.com/docs/guides/realtime), [Temporal](https://temporal.io/) + [Moose Workflows](/moosestack/workflows) + +
+ +## Operational Data Warehouse (ODW) + + + +ODW shows how to build a centralized data platform that ingests from multiple sources for business intelligence and reporting. + + +View the open source repository to see the full implementation and clone it on your own machine. + + +### ODW Data Flow + +1. Sources send data to Moose ingestion endpoints +2. Streaming functions validate and transform data +3. Data lands in ClickHouse tables +4. BI tools query via generated APIs or direct SQL + +### ODW Architecture Components + +
+ + +Handles incoming data from push-based sources (webhooks, application logs) with validation and transformation + +Technologies: [Moose APIs](/moosestack/apis), [Redpanda](https://redpanda.com/) + [Moose Streaming](/moosestack/streaming) + + + +Connects to your existing databases, object storage, or third-party APIs + +Technologies: [Temporal](https://temporal.io/) + [Moose Workflows](/moosestack/workflows) + + + +Centralized analytical database for raw and transformed data + +Technologies: [ClickHouse](https://clickhouse.com/) + [Moose OLAP](/moosestack/olap) + + + +Query interface for business intelligence and reporting + +Technologies: [Streamlit](https://streamlit.io/) dashboards, [Moose APIs](/moosestack/apis), [ClickHouse Connect](https://clickhouse.com/docs/en/interfaces/http/connect) + +
diff --git a/apps/framework-docs-v2/content/moosestack/index.mdx b/apps/framework-docs-v2/content/moosestack/index.mdx index 997becc4fb..9bd73b7bda 100644 --- a/apps/framework-docs-v2/content/moosestack/index.mdx +++ b/apps/framework-docs-v2/content/moosestack/index.mdx @@ -5,21 +5,16 @@ order: 1 category: getting-started --- -import { IconBadge } from "@/components/mdx/icon-badge"; -import { CTACard, CTACards } from "@/components/mdx/cta-card"; -import { StaggeredCard, StaggeredCards, StaggeredContent, StaggeredCode } from "@/components/mdx/staggered-card"; -import { Callout } from "@/components/mdx/callout"; -import { LanguageTabs, LanguageTabContent } from "@/components/mdx/language-tabs"; -import { CodeEditorWrapper } from "@/components/mdx/code-editor-wrapper"; -import { PathConfig } from "@/lib/path-config"; +import { IconBadge, CTACard, CTACards, StaggeredCard, StaggeredCards, StaggeredContent, StaggeredCode, Callout, LanguageTabs, LanguageTabContent, CodeEditorWrapper } from "@/components/mdx"; import { Separator } from "@/components/ui/separator"; +import { PathConfig } from "@/lib/path-config"; # MooseStack
- +
Type-safe, code-first tooling for building real-time analytical backends--OLAP Databases, Data Streaming, ETL Workflows, Query APIs, and more. @@ -51,7 +46,7 @@ bash -i <(curl -fsSL https://fiveonefour.com/install.sh) moose title="Bootstrap with a Reference Application" description="Start with a reference application for integrating Moose with complex microservices architectures" ctaLink="https://github.com/514-labs/area-code" - Icon="Github" + Icon="BrandGithub" variant="default" orientation="horizontal" /> diff --git a/apps/framework-docs-v2/content/moosestack/local-dev-environment.mdx b/apps/framework-docs-v2/content/moosestack/local-dev-environment.mdx index 6cb105a07b..3c9334db43 100644 --- a/apps/framework-docs-v2/content/moosestack/local-dev-environment.mdx +++ b/apps/framework-docs-v2/content/moosestack/local-dev-environment.mdx @@ -1,13 +1,416 @@ --- -title: Local Dev Environment -description: Set up and configure your local development environment +title: Development Mode +description: Local development environment with hot reload and automatic infrastructure management order: 3 --- -# Local Dev Environment +import { LanguageTabs, LanguageTabContent, Callout, FileTree } from "@/components/mdx"; -This page is a placeholder. Content migration pending. +# Setting Up Your Development Environment + + + + +Development mode (`moose dev`) provides a full-featured local environment optimized for rapid iteration and debugging. It automatically manages Docker containers, provides hot reload capabilities, and includes enhanced debugging features. +## Getting Started +```bash +# Start development environment +moose dev + +# View your running infrastructure +moose ls +``` + +## Container Management + +Development mode automatically manages Docker containers for your infrastructure: + +- **ClickHouse** (when `olap` feature is enabled) +- **Redpanda** (when `streaming_engine` feature is enabled) +- **Temporal** (when `workflows` feature is enabled) +- **Analytics APIs Server** (when `apis` feature is enabled) +- **Redis** (always enabled) +- **MCP Server** (always enabled) - Enables AI-assisted development. [Learn more](/moosestack/moosedev-mcp) + +### Container Configuration + +Control which containers start with feature flags: + +```toml copy +# moose.config.toml +[features] +olap = true # Enables ClickHouse +streaming_engine = true # Enables Redpanda +workflows = false # Controls Temporal startup +apis = true # Enables Analytics APIs server +``` + +### Extending Docker Infrastructure + +You can extend Moose's Docker Compose configuration with custom services by creating a `docker-compose.dev.override.yaml` file in your project root. This allows you to add additional infrastructure (databases, monitoring tools, etc.) that runs alongside your Moose development environment. + + +**Do not use docker-compose.dev.override.yaml to modify Moose-managed services** (ClickHouse, Redpanda, Redis, Temporal). The Docker Compose merge behavior makes it difficult to override existing configuration correctly, often leading to conflicts. + +Instead, use `moose.config.toml` to configure Moose infrastructure. See [Configuration](/moosestack/configuration) for all available options including database connections, ports, volumes, and service-specific settings. + +Use the override file **only for adding new services** that complement your Moose environment (e.g., PostgreSQL for application data, monitoring tools). + + +**How it works:** + +When you run `moose dev`, Moose automatically detects and merges your override file with the generated Docker Compose configuration. The files are merged using Docker Compose's [standard merge behavior](https://docs.docker.com/compose/how-tos/multiple-compose-files/merge/). + +**Example: Adding PostgreSQL for Application Data** + +Create a `docker-compose.dev.override.yaml` file in your project root: + +```yaml copy filename="docker-compose.dev.override.yaml" +services: + postgres: + image: postgres:16 + environment: + POSTGRES_USER: myapp + POSTGRES_PASSWORD: mypassword + POSTGRES_DB: myapp_db + ports: + - "5432:5432" + volumes: + - postgres-data:/var/lib/postgresql/data + +volumes: + postgres-data: +``` + +Now when you run `moose dev`, PostgreSQL will start alongside your other infrastructure. You'll see a message confirming the override file is being used: + +``` +[moose] Using docker-compose.dev.override.yaml for custom infrastructure +``` + +**Recommended Use Cases:** + +- **Add databases**: PostgreSQL, MySQL, MongoDB for application data +- **Add monitoring**: Grafana, Prometheus for metrics visualization +- **Add custom services**: Additional message queues, caching layers, or development tools + +**Not Recommended:** + +- Modifying Moose-managed services (ClickHouse, Redpanda, Redis, Temporal) +- Overriding ports, volumes, or environment variables for Moose infrastructure +- Attempting to change database credentials or connection settings + +For any Moose infrastructure configuration, use `moose.config.toml` instead. See [Configuration](/moosestack/configuration). + +**Example: Adding Grafana for Monitoring** + +```yaml copy filename="docker-compose.dev.override.yaml" +services: + grafana: + image: grafana/grafana:latest + ports: + - "3001:3000" + environment: + - GF_SECURITY_ADMIN_PASSWORD=admin + - GF_USERS_ALLOW_SIGN_UP=false + volumes: + - grafana-data:/var/lib/grafana + +volumes: + grafana-data: +``` + + +When merging files, Docker Compose follows these rules: +- **Services**: Merged by name with values from the override file taking precedence +- **Environment variables**: Appended (both files' values are used) +- **Volumes**: Appended +- **Ports**: Appended (use `!override` tag to replace instead of merge) + +See [Docker's merge documentation](https://docs.docker.com/reference/compose-file/merge/) for complete details. + + + +The override file is only used in development mode (`moose dev`). For production deployments, configure your infrastructure separately using your deployment platform's tools. + + +## Hot Reloading Development + +The development runtime includes a file watcher that provides near-instantaneous feedback when you save code changes. + +### Watched Files + +The file watcher recursively monitors your entire `app/` directory structure and only rebuilds the components that actually changed. + + + + + + + + + + + + + + + + + + + + + + + +Only the root file in your `app/` directory is run when changes are detected. In order for your tables/streams/apis/workflows to be detected, you must export them from your root file (`app/index.ts`). If you change a file in your app directory and it is a dependency of your root file, then those changes WILL be detected. + + +Only the root file in your `app/` directory is run when changes are detected. In order for your tables/streams/apis/workflows to be detected, you must import them in your root file (`main.py`). If you change a file in your app directory and it is a dependency of your root file, then those changes WILL be detected. + + + + +## Quick Example + + + +**❌ Doesn't work - No export from root:** +```ts file="app/tables/users.ts" +import { UserSchema } from './schemas/user'; + +const usersTable = new OlapTable("Users"); +// Moose can't see this - not exported from root +``` + +**✅ Works - Export from root file:** +```ts file="app/tables/users.ts" {3} +import { UserSchema } from './schemas/user'; + +export const usersTable = new OlapTable("Users"); +``` + +```ts file="app/index.ts" +export * from './tables/users'; // Moose sees this +``` + +Now because we exported the table from the root file, Moose will detect the change and rebuild the table. + + +**❌ Doesn't work - No export from root:** +```py file="app/tables/users.py" +from schemas.user import UserSchema + +users_table = OlapTable[UserSchema]("Users") +# Moose can't see this - not imported in main.py +``` + +**✅ Works - Import in main file:** +```py file="app/tables/users.py" {3} +from schemas.user import UserSchema + +users_table = OlapTable[UserSchema]("Users") # No export needed - Python modules are automatically discovered +``` + +```py file="app/main.py" +from tables.users import users_table # Moose sees this +``` + +Now because we imported the table in the main file, Moose will detect the change and rebuild the table. + + + + +**✅ Works - Change dependency:** + + +```ts file="app/schemas/user.ts" {5} +export interface UserSchema { + id: string; + name: string; + email: string; + age: number; // Adding this triggers migration +} +``` +*Moose detects this because `UserSchema` is imported in the root file via the dependency chain.* + + +```py file="app/schemas/user.py" {5} +class UserSchema(BaseModel): + id: str + name: str + email: str + age: int # Adding this triggers migration +``` +*Moose detects this because `UserSchema` is imported in the root file via the dependency chain.* + + + + +Learn more about how Moose handles migrations. + + +## Script Execution Hooks + +You can configure your dev server to run your own shell commands automatically during development. Use these hooks to keep generated artifacts in sync (e.g., refreshing external models, regenerating OpenAPI SDKs). + +### Available hooks + +- `on_first_start_script`: runs once when the dev server first starts in this process +- `on_reload_complete_script`: runs after each dev server reload when code/infra changes have been fully applied + +Configure these in `moose.config.toml` under the `http_server_config` section: + +```toml copy +# moose.config.toml +[http_server_config] +# One-time on first start +on_first_start_script = "echo 'dev started'" + +# After every code/infra reload completes +on_reload_complete_script = "echo 'reload complete'" +``` + +Notes: + +- Scripts run from your project root using your `$SHELL` (falls back to `/bin/sh`). +- Use `&&` to chain multiple commands or point to a custom script. +- Prefer passing credentials via environment variables or your secret manager. + +### Use case: keep external models in sync (DB Pull) + +Refresh `EXTERNALLY_MANAGED` table models from a remote ClickHouse on dev start so your local code matches the live schema. + +```bash filename="Terminal" copy +export REMOTE_CLICKHOUSE_URL="https://username:password@host:8443/?database=default" +``` + +```toml copy +# moose.config.toml +[http_server_config] +on_first_start_script = "moose db pull --connection-string $REMOTE_CLICKHOUSE_URL" +``` + +See the full guide: [DB Pull](/moosestack/olap/db-pull) + +### Use case: regenerate OpenAPI SDKs on reload + +Automatically regenerate client SDKs after Moose finishes applying code/infra changes so `.moose/openapi.yaml` is fresh. + +```toml copy +# moose.config.toml +[http_server_config] +on_first_start_script = "command -v openapi-generator-cli >/dev/null 2>&1 || npm i -g @openapitools/openapi-generator-cli" +on_reload_complete_script = "openapi-generator-cli generate -i .moose/openapi.yaml -g typescript-fetch -o ./generated/ts" +``` + +More examples: [OpenAPI SDK](/moosestack/apis/openapi-sdk) + +## Local Infrastructure + +### Port Allocation + +Development mode uses the following default ports: + +- **4000**: Main API server +- **5001**: Management API (health checks, metrics, admin, OpenAPI docs) + +### Service URLs + +Access your development services at: + + + +```bash +# Main application +http://localhost:4000 + +# Management interface +http://localhost:5001/metrics + +# OpenAPI documentation +http://localhost:5001/openapi.yaml +``` + + +```bash +# Main application +http://localhost:4000 + +# Management interface +curl http://localhost:5001/metrics + +# OpenAPI documentation +http://localhost:5001/openapi.yaml +``` + + + +### Container Networking + +All containers run in an isolated Docker network with automatic service discovery: + +- Containers communicate using service names +- Port mapping only for external access +- Automatic DNS resolution between services + +### MCP Server for AI-Assisted Development + +Development mode includes a built-in Model Context Protocol (MCP) server that lets AI assistants interact with your local infrastructure through natural language. + +**What you can do:** +- Query your ClickHouse database with natural language +- Inspect streaming topics and messages +- Search and filter development logs +- Explore your infrastructure map + +**Quick setup:** + +The MCP server runs automatically at `http://localhost:4000/mcp`. For Claude Code, just run: + +```bash copy +claude mcp add --transport http moose-dev http://localhost:4000/mcp +``` + +For other AI clients (Windsurf, VS Code, Cursor, Claude Desktop), see the [full setup guide](/moosestack/moosedev-mcp). + +**Example prompts:** +- *"What errors are in the logs?"* +- *"What tables exist in my project?"* +- *"Show me the schema of all tables"* +- *"Sample 5 messages from the Foo stream"* + + +See the complete guide for all available tools, detailed configuration for each AI client, and example workflows. + + +## Troubleshooting + +### Common Issues + +**Container Startup Failures** +```bash +# Check Docker is running +docker info + +# View container logs +moose logs + +``` + +**Port Conflicts** +```bash +# Check what's using your ports +lsof -i :4000 +lsof -i :5001 + +# Use custom ports +export MOOSE_HTTP_PORT=4040 +export MOOSE_MANAGEMENT_PORT=5010 +moose dev +``` diff --git a/apps/framework-docs-v2/content/moosestack/metrics.mdx b/apps/framework-docs-v2/content/moosestack/metrics.mdx index e8b1f03f87..4cdfb6182a 100644 --- a/apps/framework-docs-v2/content/moosestack/metrics.mdx +++ b/apps/framework-docs-v2/content/moosestack/metrics.mdx @@ -1,13 +1,177 @@ --- title: Observability -description: Observability and metrics for MooseStack +description: Unified observability for Moose across development and production—metrics console, health checks, Prometheus, OpenTelemetry, logging, and error tracking order: 1 --- +import { Callout } from "@/components/mdx"; + # Observability -This page is a placeholder. Content migration pending. +This page consolidates Moose observability for both local development and production environments. + +## Local Development + +### Metrics Console + +Moose provides a console to view live metrics from your Moose application. To launch the console, run: + +```bash filename="Terminal" copy +moose metrics +``` + + +Use the arrow keys to move up and down rows in the endpoint table and press Enter to view more details about that endpoint. + + + +#### Endpoint Metrics + +Aggregated metrics for all endpoints: + +| Metric | Description | +| :-------------------- | :---------------------------------------------------------------------------------- | +| `AVERAGE LATENCY` | Average time in milliseconds it takes for a request to be processed by the endpoint | +| `TOTAL # OF REQUESTS` | Total number of requests made to the endpoint | +| `REQUESTS PER SECOND` | Average number of requests made per second to the endpoint | +| `DATA IN` | Average number of bytes of data sent to all `/ingest` endpoints per second | +| `DATA OUT` | Average number of bytes of data sent to all `/api` endpoints per second | + +Individual endpoint metrics: + +| Metric | Description | +| :---------------------------- | :---------------------------------------------------------------------------------- | +| `LATENCY` | Average time in milliseconds it takes for a request to be processed by the endpoint | +| `# OF REQUESTS RECEIVED` | Total number of requests made to the endpoint | +| `# OF MESSAGES SENT TO KAFKA` | Total number of messages sent to the Kafka topic | + +#### Stream → Table Sync Metrics + +| Metric | Description | +| :---------- | :-------------------------------------------------------------------------------------------------- | +| `MSG READ` | Total number of messages sent from `/ingest` API endpoint to the Kafka topic | +| `LAG` | The number of messages that have been sent to the consumer but not yet received | +| `MSG/SEC` | Average number of messages sent from `/ingest` API endpoint to the Kafka topic per second | +| `BYTES/SEC` | Average number of bytes of data received by the ClickHouse consumer from the Kafka topic per second | + +#### Streaming Transformation Metrics +For each streaming transformation: + +| Metric | Description | +| :------------ | :---------------------------------------------------------------------------- | +| `MSG IN` | Total number of messages passed into the streaming function | +| `MSG IN/SEC` | Average number of messages passed into the streaming function per second | +| `MSG OUT` | Total number of messages returned by the streaming function | +| `MSG OUT/SEC` | Average number of messages returned by the streaming function per second | +| `BYTES/SEC` | Average number of bytes of data returned by the streaming function per second | + +--- + +## Production + +### Health Monitoring + +Moose applications expose a health check endpoint at `/health` that returns a 200 OK response when the application is operational. This endpoint is used by container orchestration systems like Kubernetes to determine the health of your application. + +In production environments, we recommend configuring three types of probes: + +1. Startup Probe: Gives Moose time to initialize before receiving traffic +2. Readiness Probe: Determines when the application is ready to receive traffic +3. Liveness Probe: Detects when the application is in a deadlocked state and needs to be restarted + + +Learn more about how to configure health checks in your Kubernetes deployment. + + +### Prometheus Metrics + +Moose applications expose metrics in Prometheus format at the `/metrics` endpoint. These metrics include: + +- HTTP request latency histograms for each endpoint +- Request counts and error rates +- System metrics for the Moose process + +Example metrics output: + +``` +# HELP latency Latency of HTTP requests. +# TYPE latency histogram +latency_sum{method="POST",path="ingest/UserActivity"} 0.025 +latency_count{method="POST",path="ingest/UserActivity"} 2 +latency_bucket{le="0.001",method="POST",path="ingest/UserActivity"} 0 +latency_bucket{le="0.01",method="POST",path="ingest/UserActivity"} 0 +latency_bucket{le="0.02",method="POST",path="ingest/UserActivity"} 1 +latency_bucket{le="0.05",method="POST",path="ingest/UserActivity"} 1 +latency_bucket{le="0.1",method="POST",path="ingest/UserActivity"} 1 +latency_bucket{le="0.25",method="POST",path="ingest/UserActivity"} 1 +latency_bucket{le="0.5",method="POST",path="ingest/UserActivity"} 1 +latency_bucket{le="1.0",method="POST",path="ingest/UserActivity"} 1 +latency_bucket{le="5.0",method="POST",path="ingest/UserActivity"} 1 +latency_bucket{le="10.0",method="POST",path="ingest/UserActivity"} 1 +latency_bucket{le="30.0",method="POST",path="ingest/UserActivity"} 1 +latency_bucket{le="60.0",method="POST",path="ingest/UserActivity"} 1 +latency_bucket{le="120.0",method="POST",path="ingest/UserActivity"} 1 +latency_bucket{le="240.0",method="POST",path="ingest/UserActivity"} 1 +latency_bucket{le="+Inf",method="POST",path="ingest/UserActivity"} 1 +``` + +You can scrape these metrics using a Prometheus server or any compatible monitoring system. + +### OpenTelemetry Integration + +In production deployments, Moose can export telemetry data using OpenTelemetry. Enable via environment variables: + +``` +MOOSE_TELEMETRY__ENABLED=true +MOOSE_TELEMETRY__EXPORT_METRICS=true +``` + +When running in Kubernetes with an OpenTelemetry operator, you can configure automatic sidecar injection by adding annotations to your deployment: + +```yaml +metadata: + annotations: + "sidecar.opentelemetry.io/inject": "true" +``` + +### Logging + +Configure structured logging via environment variables: + +``` +MOOSE_LOGGER__LEVEL=Info +MOOSE_LOGGER__STDOUT=true +MOOSE_LOGGER__FORMAT=Json +``` + +The JSON format is ideal for log aggregation systems (ELK Stack, Graylog, Loki, or cloud logging solutions). + +### Production Monitoring Stack + +Recommended components: + +1. Metrics Collection: Prometheus or cloud-native monitoring services +2. Log Aggregation: ELK Stack, Loki, or cloud logging solutions +3. Distributed Tracing: Jaeger or other OpenTelemetry-compatible backends +4. Alerting: Alertmanager or cloud provider alerting + +### Error Tracking + +Integrate with systems like Sentry via environment variables: + +``` +SENTRY_DSN=https://your-sentry-dsn +RUST_BACKTRACE=1 +``` + + +Want this managed in production for you? Check out Boreal Cloud (from the makers of the Moose Stack). + +## Feedback + +Join our Slack community to share feedback and get help with Moose. + diff --git a/apps/framework-docs-v2/content/moosestack/migrate/index.mdx b/apps/framework-docs-v2/content/moosestack/migrate/index.mdx index a8750cb55c..28bdfee785 100644 --- a/apps/framework-docs-v2/content/moosestack/migrate/index.mdx +++ b/apps/framework-docs-v2/content/moosestack/migrate/index.mdx @@ -1,9 +1,376 @@ --- -title: Migrate -description: Database migration tools and workflows -order: 1 +title: Migrations & Planning +description: How Moose handles infrastructure migrations and planning +order: 0 +category: migrate --- -# Migrate +import { LanguageTabs, LanguageTabContent, Callout } from "@/components/mdx"; -This page is a placeholder. Content migration pending. +# Moose Migrate + +Moose's migration system works like version control for your infrastructure. It automatically detects changes in your code and applies them to your data infrastructure with confidence. + + +Moose tracks changes across: +- OLAP Tables and Materialized Views +- Streaming Topics +- API Endpoints +- Workflows + + + +## How It Works + +Moose collects all objects defined in your main file (`index.ts` for TypeScript or `main.py` for Python) and automatically generates infrastructure operations to match your code: + + + + +```ts file="app/index.ts" +interface UserSchema { + id: string; + name: string; + email: string; +} + +export const usersTable = new OlapTable("Users"); +export const userEvents = new Stream("Users"); +``` + + + + +```python file="app/main.py" +from pydantic import BaseModel +from moose_lib import OlapTable, Stream + +class UserSchema(BaseModel): + id: str + name: str + email: str + +users_table = OlapTable[UserSchema]("Users") +user_events = Stream[UserSchema]("Users") +``` + + + + +When you add these objects, Moose automatically creates: +- A ClickHouse table named `Users` with the `UserSchema` +- A Redpanda topic named `Users` with the `UserSchema` + +## Development Workflow + +When running your code in development mode, Moose will automatically hot-reload migrations to your local infrastructure as you save code changes. + +### Quick Start + +Start your development environment: + +```bash filename="Terminal" copy +moose dev +``` + +This automatically: +1. Recursively watches your `/app` directory for code changes +2. Parses objects defined in your main file +3. Compares the new objects with the current infrastructure state Moose stores internally +4. Generates and applies migrations in real-time based on the differences +5. Provides immediate feedback on any errors or warnings +6. Updates the internal state of your infrastructure to reflect the new state + +### Example: Adding a New Table + + + + +```ts file="app/index.ts" {6} copy +// Before +export const usersTable = new OlapTable("Users"); + +// After (add analytics table) +export const usersTable = new OlapTable("Users"); +export const analyticsTable = new OlapTable("Analytics"); +``` + + + + +```python file="app/main.py" {6} copy +# Before +users_table = OlapTable[UserSchema]("Users") + +# After (add analytics table) +users_table = OlapTable[UserSchema]("Users") +analytics_table = OlapTable[AnalyticsSchema]("Analytics") +``` + + + + +**What happens:** +- Moose detects the new `analyticsTable` object +- Compares: "No Analytics table exists" +- Generates migration: "Create Analytics table" +- Applies migration automatically +- Updates internal state + +In your terminal, you will see a log that shows the new table being created: +```bash +⠋ Processing Infrastructure changes from file watcher + + Table: Analytics Version None - id: String, number: Int64, status: String - - deduplicate: false +``` +### Example: Schema Changes + + + + +```ts file="app/index.ts" {8} copy +import { Key } from "@514labs/moose-lib"; + +// After (add age field) +interface UserSchema { + id: Key; + name: string; + email: string; + age: number; // New field +} +``` + + + + +```python file="app/main.py" {8} copy +from moose_lib import Key + +# After (add age field) +class UserSchema(BaseModel): + id: Key[str] + name: str + email: str + age: int # New field +``` + + + + +**What happens:** +- Moose detects the new `age` field +- Generates migration: "Add age column to Users table" +- Applies migration +- Existing rows get NULL/default values + +## Production Workflow + +Moose supports two deployment patterns: **Moose Server** and **Serverless**. + +### Moose Server Deployments + +For deployments with a running Moose server, preview changes before applying: + +```bash filename="Terminal" copy +moose plan --url https://your-production-instance --token +``` + + +Remote planning requires authentication: + +1. Generate a token: `moose generate hash-token` +2. Configure your server: +```toml filename="moose.config.toml" copy +[authentication] +admin_api_key = "your-hashed-token" +``` +3. Use the token with `--token` flag + + +**Deployment Flow:** +1. **Develop locally** with `moose dev` +2. **Test changes** in local environment +3. **Plan against production**: `moose plan --url --token ` +4. **Review changes** carefully +5. **Deploy** - Moose applies migrations automatically on startup + +### Serverless Deployments + +For serverless deployments (no Moose server), use the ClickHouse connection directly: + +```bash filename="Terminal" copy +# Step 1: Generate migration files +moose generate migration --clickhouse-url --save + +# Step 2: Preview changes in PR +moose plan --clickhouse-url clickhouse://user:pass@host:port/database + +# Step 3: Execute migration after merge +moose migrate --clickhouse-url +``` + +**Deployment Flow:** +1. **Develop locally** with `moose dev` +2. **Generate migration plan**: `moose generate migration --clickhouse-url --save` +3. **Create PR** with `plan.yaml`, `remote_state.json`, `local_infra_map.json` +4. **PR validation**: Run `moose plan --clickhouse-url ` in CI to preview changes +5. **Review** migration files and plan output +6. **Merge PR** +7. **Execute migration**: Run `moose migrate --clickhouse-url ` in CI/CD + + +Requires `state_config.storage = "clickhouse"` in `moose.config.toml`: +```toml filename="moose.config.toml" copy +[state_config] +storage = "clickhouse" + +[features] +olap = true +data_models_v2 = true +``` + + + +Your ClickHouse instance needs the KeeperMap engine for state storage and migration locking. + +✅ **ClickHouse Cloud**: Works out of the box +✅ **`moose dev` or `moose prod`**: Already configured +⚠️ **Self-hosted ClickHouse**: See [ClickHouse KeeperMap documentation](https://clickhouse.com/docs/en/engines/table-engines/special/keeper-map) for setup requirements + + +### State Storage Options + +Moose migrations require storing infrastructure state and coordinating locks. You can choose between two backends: + +**ClickHouse State Storage (Default)** +Uses the `_MOOSE_STATE` KeeperMap table. Best for: +- ClickHouse Cloud (works out of the box) +- Self-hosted with ClickHouse Keeper already configured + +**Redis State Storage** +Uses Redis for state and locking. Best for: +- Existing Redis infrastructure +- Multi-tenant deployments (isolated by `key_prefix`) +- When ClickHouse Keeper isn't available + +**Configuration:** +```toml filename="moose.config.toml" copy +[state_config] +storage = "redis" # or "clickhouse" (default) +``` + +**Usage with Redis:** +```bash filename="Terminal" copy +# With environment variable (recommended) +export MOOSE_REDIS_CONFIG__URL="redis://host:port" +moose migrate --clickhouse-url clickhouse://... + +# Or with CLI flag +moose migrate \ + --clickhouse-url clickhouse://... \ + --redis-url redis://host:port +``` + + +The ClickHouse URL is always required, even when using Redis for state storage. + + +### Understanding Plan Output + +Moose shows exactly what will change: + +```bash + + Table: Analytics Version None - id: String, number: Int64, status: String - - deduplicate: false + + Table: Users Version None - id: String, name: String, email: String - - deduplicate: false +``` + +## Migration Types + +| Change Type | Infrastructure Impact | Data Impact | +|-------------|----------------------|-------------| +| **Add new object** | New table/stream/API created | No impact | +| **Remove object** | Table/stream/API dropped | All data lost | +| **Add field** | New column created | Existing rows get NULL/default | +| **Remove field** | Column dropped | Data permanently lost | +| **Change type** | Column altered | Data converted if compatible | + +For detailed examples of each migration type, see [Migration Types](/moosestack/migrate/migration-types). + +## Viewing Infrastructure State + +### Via CLI +```bash +# Check current infrastructure objects +moose ls + +# View migration logs +moose logs +``` + +### Via Direct Connection + +Connect to your local infrastructure using details from `moose.config.toml`: + +```toml file="moose.config.toml" +[features] +olap = true # ClickHouse for analytics +streaming_engine = true # Redpanda for streaming +workflows = false # Temporal for workflows + +[clickhouse_config] +host = "localhost" +host_port = 18123 +native_port = 9000 +db_name = "local" +user = "panda" +password = "pandapass" + +[redpanda_config] +broker = "localhost:19092" +message_timeout_ms = 1000 +retention_ms = 30000 +replication_factor = 1 +``` + +## Best Practices + +### Development +- Use `moose dev` for all local development +- Monitor plan outputs for warnings +- Test schema changes with sample data + +### Production +- Always use remote planning before deployments +- Review changes carefully in production plans +- Maintain proper authentication +- Test migrations in staging first + +### Managing TTL Outside Moose + +If you're managing ClickHouse TTL settings through other tools or want to avoid migration failures from TTL drift, you can configure Moose to ignore TTL changes: + +```toml filename="moose.config.toml" copy +[migration_config] +ignore_operations = ["ModifyTableTtl", "ModifyColumnTtl"] +``` + +This tells Moose to: +- Skip generating TTL change operations in migration plans +- Ignore TTL differences during drift detection + +You'll still get migrations for all other schema changes (adding tables, modifying columns, etc.), but TTL changes won't block your deployments. + +## Troubleshooting + +### Authentication Errors +- Verify your authentication token +- Generate a new token: `moose generate hash-token` +- Check server configuration in `moose.config.toml` + +### Migration Issues +- Check `moose logs` for detailed error messages +- Verify object definitions in your main file +- Ensure all required fields are properly typed +- **Stuck migration lock**: If you see "Migration already in progress" but no migration is running, wait 5 minutes for automatic expiry or manually clear it: + ```sql + DELETE FROM _MOOSE_STATE WHERE key = 'migration_lock'; + ``` diff --git a/apps/framework-docs-v2/content/moosestack/migrate/lifecycle.mdx b/apps/framework-docs-v2/content/moosestack/migrate/lifecycle.mdx index 38f1bdf241..fbc537281a 100644 --- a/apps/framework-docs-v2/content/moosestack/migrate/lifecycle.mdx +++ b/apps/framework-docs-v2/content/moosestack/migrate/lifecycle.mdx @@ -1,9 +1,239 @@ --- title: Lifecycle Management -description: Manage migration lifecycle +description: Control how Moose manages database and streaming resources when your code changes order: 2 +category: migrate --- +import { Callout, BulletPointsCard, LanguageTabs, LanguageTabContent } from "@/components/mdx"; + # Lifecycle Management -This page is a placeholder. Content migration pending. + + + + + +## Overview + +The `LifeCycle` enum controls how Moose manages the lifecycle of database/streaming resources when your code changes. +This feature gives you fine-grained control over whether Moose automatically updates your database schema or +leaves it under external/manual control. + + + +## LifeCycle Modes + +### `FULLY_MANAGED` (Default) + +This is the default behavior where Moose has complete control over your database resources. When you change your data models, Moose will automatically: + +- Add new columns or tables +- Remove columns or tables that no longer exist in your code +- Modify existing column types and constraints + + +This mode can perform destructive operations. Data may be lost if you remove fields from your data models or if you perform operations that require a destroy and recreate to be effective, like changing the `order_by_fields` (Python) or `orderByFields` (TypeScript) field. + + + + + +```ts filename="FullyManagedExample.ts" copy +import { OlapTable, LifeCycle } from "@514labs/moose-lib"; + +interface UserData { + id: string; + name: string; + email: string; +} + +// Default behavior - fully managed +const userTable = new OlapTable("users"); + +// Explicit fully managed configuration +const explicitTable = new OlapTable("users", { + orderByFields: ["id"], + lifeCycle: LifeCycle.FULLY_MANAGED +}); +``` + + + + +```py filename="FullyManagedExample.py" copy +from moose_lib import OlapTable, OlapConfig, LifeCycle +from pydantic import BaseModel + +class UserData(BaseModel): + id: str + name: str + email: str + +# Default behavior - fully managed +user_table = OlapTable[UserData]("users") + +# Explicit fully managed configuration +explicit_table = OlapTable[UserData]("users", OlapConfig( + order_by_fields=["id"], + life_cycle=LifeCycle.FULLY_MANAGED +)) +``` + + + + +### `DELETION_PROTECTED` + +This mode allows Moose to automatically add new database structures but prevents it from removing existing ones. +Perfect for production environments where you want to evolve your schema safely without risking data loss. + +**What Moose will do:** +- Add new columns, tables +- Modify column types (if compatible) +- Update non-destructive configurations + +**What Moose won't do:** +- Drop columns or tables +- Perform destructive schema changes + + + + +```ts filename="DeletionProtectedExample.ts" copy +import { IngestPipeline, LifeCycle } from "@514labs/moose-lib"; + +interface ProductEvent { + id: string; + productId: string; + timestamp: Date; + action: string; +} + +const productAnalytics = new IngestPipeline("product_analytics", { + table: { + orderByFields: ["timestamp", "productId"], + engine: ClickHouseEngines.ReplacingMergeTree, + }, + stream: { + parallelism: 4, + }, + ingestApi: true, + // automatically applied to the table and stream + lifeCycle: LifeCycle.DELETION_PROTECTED +}); +``` + + + + +```py filename="DeletionProtectedExample.py" copy +from moose_lib import IngestPipeline, IngestPipelineConfig, OlapConfig, StreamConfig, LifeCycle +from pydantic import BaseModel +from datetime import datetime + +class ProductEvent(BaseModel): + id: str + product_id: str + timestamp: datetime + action: str + +product_analytics = IngestPipeline[ProductEvent]("product_analytics", IngestPipelineConfig( + table=OlapConfig( + order_by_fields=["timestamp", "product_id"], + engine=ClickHouseEngines.ReplacingMergeTree, + ), + stream=StreamConfig( + parallelism=4, + ), + ingest_api=True, + # automatically applied to the table and stream + life_cycle=LifeCycle.DELETION_PROTECTED +)) +``` + + + + +### `EXTERNALLY_MANAGED` + +This mode tells Moose to completely hands-off your resources. +You become responsible for creating and managing the database schema. This is useful when: + +- You have existing database tables managed by another team +- You're integrating with another system (e.g. PeerDB) +- You have strict database change management processes + + +With externally managed resources, you must ensure your database schema matches your data models exactly, or you may encounter runtime errors. + + + + + +```ts filename="ExternallyManagedExample.ts" copy +import { Stream, OlapTable, LifeCycle, Key } from "@514labs/moose-lib"; + +interface ExternalUserData { + userId: Key; + fullName: string; + emailAddress: string; + createdAt: Date; +} + +// Connect to existing database table +const legacyUserTable = new OlapTable("legacy_users", { + lifeCycle: LifeCycle.EXTERNALLY_MANAGED +}); + +// Connect to existing Kafka topic +const legacyStream = new Stream("legacy_user_stream", { + lifeCycle: LifeCycle.EXTERNALLY_MANAGED, + destination: legacyUserTable +}); +``` + + + + +```py filename="ExternallyManagedExample.py" copy +from moose_lib import Stream, OlapTable, OlapConfig, StreamConfig, LifeCycle, Key +from pydantic import BaseModel +from datetime import datetime + +class ExternalUserData(BaseModel): + user_id: Key[str] + full_name: str + email_address: str + created_at: datetime + +# Connect to existing database table +legacy_user_table = OlapTable[ExternalUserData]("legacy_users", OlapConfig( + life_cycle=LifeCycle.EXTERNALLY_MANAGED +)) + +# Connect to existing Kafka topic +legacy_stream = Stream[ExternalUserData]("legacy_user_stream", StreamConfig( + life_cycle=LifeCycle.EXTERNALLY_MANAGED, + destination=legacy_user_table +)) +``` + + + diff --git a/apps/framework-docs-v2/content/moosestack/migrate/migration-types.mdx b/apps/framework-docs-v2/content/moosestack/migrate/migration-types.mdx index 5f2888993e..8f4ba681c5 100644 --- a/apps/framework-docs-v2/content/moosestack/migrate/migration-types.mdx +++ b/apps/framework-docs-v2/content/moosestack/migrate/migration-types.mdx @@ -1,9 +1,363 @@ --- title: Migration Types -description: Types of database migrations +description: Detailed migration examples and advanced development topics order: 1 +category: migrate --- +import { LanguageTabs, LanguageTabContent, Callout } from "@/components/mdx"; + # Migration Types -This page is a placeholder. Content migration pending. + + + + + +This guide provides detailed examples of different migration types. For the complete workflow overview, see [Migrations & Planning](/moosestack/migrate). + +## Adding New Infrastructure Components + + +Keep in mind that only the modules that you have enabled in your `moose.config.toml` will be included in your migrations. + + +```toml file="moose.config.toml" +[features] +olap = true +streaming_engine = true +workflows = true +``` + +### New OLAP Table or Materialized View + + + + +```ts file="app/index.ts" +interface AnalyticsSchema { + id: string; + event_type: string; + timestamp: Date; + user_id: string; + value: number; +} + +export const analyticsTable = new OlapTable("Analytics"); +``` + + + + +```python file="app/main.py" +from pydantic import BaseModel +from datetime import datetime +from moose_lib import OlapTable + +class AnalyticsSchema(BaseModel): + id: str + event_type: str + timestamp: datetime + user_id: str + value: float + +analytics_table = OlapTable[AnalyticsSchema]("Analytics") +``` + + + + +**Migration Result:** Creates ClickHouse table `Analytics` with all fields from `AnalyticsSchema` + + + +If you have not enabled the `olap` feature flag, you will not be able to create new OLAP tables. + +```toml file="moose.config.toml" +[features] +olap = true +``` + + + +Check out the OLAP migrations guide to learn more about the different migration modes. + + + +### New Stream + + + + +```ts file="app/index.ts" +export const userEvents = new Stream("UserEvents"); +export const systemEvents = new Stream("SystemEvents"); +``` + + + + +```python file="app/main.py" +user_events = Stream[UserSchema]("UserEvents") +system_events = Stream[SystemEventSchema]("SystemEvents") +``` + + + + +**Migration Result:** Creates Redpanda topics `UserEvents` and `SystemEvents` + + +If you have not enabled the `streaming_engine` feature flag, you will not be able to create new streaming topics. + +```toml file="moose.config.toml" +[features] +streaming_engine = true +``` + + +## Schema Modifications + +### Adding Fields + + + + +```ts file="app/index.ts" +// Before +interface UserSchema { + id: string; + name: string; + email: string; +} + +// After +interface UserSchema { + id: string; + name: string; + email: string; + age: number; + created_at: Date; + is_active: boolean; +} +``` + + + + +```python file="app/main.py" +# Before +class UserSchema(BaseModel): + id: str + name: str + email: str + +# After +class UserSchema(BaseModel): + id: str + name: str + email: str + age: int + created_at: datetime + is_active: bool +``` + + + + +**Migration Result:** Adds `age`, `created_at`, and `is_active` columns to existing table + +### Removing Fields + + + + +```ts file="app/index.ts" +// Before +interface UserSchema { + id: string; + name: string; + email: string; + age: number; + deprecated_field: string; // Will be removed +} + +// After +interface UserSchema { + id: string; + name: string; + email: string; + age: number; +} +``` + + + + +```python file="app/main.py" +# Before +class UserSchema(BaseModel): + id: str + name: str + email: str + age: int + deprecated_field: str # Will be removed + +# After +class UserSchema(BaseModel): + id: str + name: str + email: str + age: int +``` + + + + +**Migration Result:** Drops `deprecated_field` column (data permanently lost) + +### Type Changes + + + + +```ts file="app/index.ts" +// Before +interface UserSchema { + id: string; + name: string; + email: string; + score: number; // Will change to string +} + +// After +interface UserSchema { + id: string; + name: string; + email: string; + score: string; // Changed from number +} +``` + + + + +```python file="app/main.py" +# Before +class UserSchema(BaseModel): + id: str + name: str + email: str + score: float # Will change to str + +# After +class UserSchema(BaseModel): + id: str + name: str + email: str + score: str # Changed from float +``` + + + + +**Migration Result:** Alters `score` column type (data converted if compatible) + +## Removing Infrastructure + + + + +```ts file="app/index.ts" +// Before +export const usersTable = new OlapTable("Users"); +export const analyticsTable = new OlapTable("Analytics"); +export const deprecatedTable = new OlapTable("Deprecated"); + +// After (remove deprecated table) +export const usersTable = new OlapTable("Users"); +export const analyticsTable = new OlapTable("Analytics"); +``` + + + + +```python file="app/main.py" +# Before +users_table = OlapTable[UserSchema]("Users") +analytics_table = OlapTable[AnalyticsSchema]("Analytics") +deprecated_table = OlapTable[DeprecatedSchema]("Deprecated") + +# After (remove deprecated table) +users_table = OlapTable[UserSchema]("Users") +analytics_table = OlapTable[AnalyticsSchema]("Analytics") +``` + + + + +**Migration Result:** Drops `Deprecated` table (all data lost) + + +## Working with Local Infrastructure + +There are two main ways to inspect your local infrastructure to see how your migrations are applied: + +### Using the CLI + +Run `moose ls` to see the current state of your infrastructure: + +```bash +# Verify object definitions +moose ls +``` + +### Connecting to your local infrastructure +You can also connect directly to your local infrastructure to see the state of your infrastructure. + + +All credentials for your local infrastructure are located in your project config file (`moose.config.toml`). + + +#### Connecting to ClickHouse + +```bash +# Using clickhouse-client +clickhouse-client --host localhost --port 18123 --user panda --password pandapass --database local + +# Using connection string +clickhouse-client "clickhouse://panda:pandapass@localhost:18123/local" +``` + +#### Connecting to Redpanda + +```bash +# Using kafka-console-consumer +kafka-console-consumer --bootstrap-server localhost:19092 --topic UserEvents --from-beginning + +# Using kafka-console-producer +kafka-console-producer --bootstrap-server localhost:19092 --topic UserEvents +``` + +#### Viewing Temporal Workflows + +Navigate to `http://localhost:8080` to view the Temporal UI and see registered workflows. + + + +## Gotchas: + +Your dev server must be running to connect to your local infrastructure. + +```bash +moose dev +``` + +Only the modules that you have enabled in your `moose.config.toml` will be included in your migrations: + +```toml file="moose.config.toml" +[features] +olap = true # Required for OLAP Tables and Materialized Views +streaming_engine = true # Required for Streams +workflows = true # Required for Workflows and Tasks +``` diff --git a/apps/framework-docs-v2/content/moosestack/moose-cli.mdx b/apps/framework-docs-v2/content/moosestack/moose-cli.mdx index df69c28161..6f03230427 100644 --- a/apps/framework-docs-v2/content/moosestack/moose-cli.mdx +++ b/apps/framework-docs-v2/content/moosestack/moose-cli.mdx @@ -4,10 +4,261 @@ description: Command-line interface for MooseStack order: 1 --- -# CLI +# Moose CLI Reference -This page is a placeholder. Content migration pending. +## Installation +```bash filename="Terminal" copy +bash -i <(curl -fsSL https://fiveonefour.com/install.sh) moose +``` +## Core Commands +### Init +Initializes a new Moose project. +```bash +moose init --template