diff --git a/Tutorial/docs/assets/webapp/create_dataset.png b/Tutorial/docs/assets/webapp/create_dataset.png new file mode 100644 index 0000000..32cfc63 Binary files /dev/null and b/Tutorial/docs/assets/webapp/create_dataset.png differ diff --git a/Tutorial/docs/assets/webapp/create_dataset_2.png b/Tutorial/docs/assets/webapp/create_dataset_2.png new file mode 100644 index 0000000..3832018 Binary files /dev/null and b/Tutorial/docs/assets/webapp/create_dataset_2.png differ diff --git a/Tutorial/docs/assets/webapp/create_scenario.png b/Tutorial/docs/assets/webapp/create_scenario.png new file mode 100644 index 0000000..983cfc6 Binary files /dev/null and b/Tutorial/docs/assets/webapp/create_scenario.png differ diff --git a/Tutorial/docs/assets/webapp/created_dataset.png b/Tutorial/docs/assets/webapp/created_dataset.png new file mode 100644 index 0000000..be0d06f Binary files /dev/null and b/Tutorial/docs/assets/webapp/created_dataset.png differ diff --git a/Tutorial/docs/assets/webapp/created_scenario.png b/Tutorial/docs/assets/webapp/created_scenario.png new file mode 100644 index 0000000..67b8fe7 Binary files /dev/null and b/Tutorial/docs/assets/webapp/created_scenario.png differ diff --git a/Tutorial/docs/assets/webapp/empty_dataset.png b/Tutorial/docs/assets/webapp/empty_dataset.png new file mode 100644 index 0000000..bf9b94b Binary files /dev/null and b/Tutorial/docs/assets/webapp/empty_dataset.png differ diff --git a/Tutorial/docs/assets/webapp/empty_scenario.png b/Tutorial/docs/assets/webapp/empty_scenario.png new file mode 100644 index 0000000..8c440af Binary files /dev/null and b/Tutorial/docs/assets/webapp/empty_scenario.png differ diff --git a/Tutorial/docs/assets/webapp/running_scenario.png b/Tutorial/docs/assets/webapp/running_scenario.png new file mode 100644 index 0000000..7f21268 Binary files /dev/null and b/Tutorial/docs/assets/webapp/running_scenario.png differ diff --git a/Tutorial/docs/tutorials/hidden/adx-database.md b/Tutorial/docs/tutorials/hidden/adx-database.md new file mode 100644 index 0000000..f2a0ae0 --- /dev/null +++ b/Tutorial/docs/tutorials/hidden/adx-database.md @@ -0,0 +1,73 @@ +# Azure Data Explorer (ADX) + +Azure Data Explorer (ADX) is a fast, fully managed data analytics service optimized for real-time analysis of large volumes of data. + +## Purpose in CosmoTech Platform + +ADX serves as the **data warehouse** for CosmoTech simulation results, storing: + +- **ProbesMeasures** - Time-series data from simulation probes +- **ScenarioRun** - Execution details and run metadata +- **ScenarioMetadata** - Scenario configurations and parameters +- **SimulationTotalFacts** - Aggregated metrics and KPIs + +## Key Benefits + +- **High Performance** - Query billions of records in seconds +- **Real-time Ingestion** - Data flows from Event Hub to ADX automatically +- **Time-Series Optimized** - Built for temporal data analysis +- **KQL Language** - Powerful Kusto Query Language for data exploration +- **Power BI Integration** - Direct connection for visualization +- **Scalability** - Handles massive simulation datasets efficiently + +## Data Flow + +1. **Simulation runs** and generates output data +2. **Event Hub** receives and streams the data +3. **ADX** ingests data in real-time via connectors +4. **Data is stored** in structured tables with compression +5. **Users query** the data using KQL or visualize in Power BI + +## Common Use Cases + +1. **Analyze Simulation Results** - Query and compare scenario outcomes +2. **Track Metrics Over Time** - Monitor KPIs across multiple runs +3. **Generate Reports** - Create custom analytics dashboards +4. **Data Exploration** - Investigate simulation behavior and patterns +5. **Performance Analysis** - Identify bottlenecks and optimization opportunities + +## Example Query + +```kql +ProbesMeasures +| where SimulationRun == "run-abc123" +| where Timestamp >= ago(1h) +| summarize avg(Value) by bin(Timestamp, 5m), ProbeName +| render timechart +``` + +This query retrieves probe measurements from the last hour and displays average values in a time chart. + +## Database Configuration + +ADX databases are automatically created when a workspace is deployed with the following naming convention: + +``` +- +``` + +Configuration includes: +- **Retention policies** - Default 365 days +- **Permissions** - User access controls +- **Initialization scripts** - Table schemas and mappings +- **Event Hub connectors** - Automatic data ingestion + +## Integration with Power BI + +ADX connects directly to Power BI, enabling: +- Real-time dashboard updates +- Interactive data exploration +- Custom visualizations +- Scenario comparison reports + +Connect using the Azure Data Explorer connector in Power BI Desktop with your cluster URL and database name. \ No newline at end of file diff --git a/Tutorial/docs/tutorials/hidden/babylon.md b/Tutorial/docs/tutorials/hidden/babylon.md new file mode 100644 index 0000000..9d1e477 --- /dev/null +++ b/Tutorial/docs/tutorials/hidden/babylon.md @@ -0,0 +1,284 @@ +# Babylon + +Babylon is CosmoTech's proprietary command-line tool for deploying, configuring, and managing platform resources. + +## Overview + +Babylon is a deployment orchestration tool developed by CosmoTech that simplifies the process of creating and managing solutions, workspaces, datasets, and other platform resources. It uses declarative YAML manifests to define infrastructure and configurations, similar to Kubernetes. + +## Key Features + +- **Declarative Configuration** - Define resources in YAML manifests +- **State Management** - Track and manage resource states across environments +- **Idempotent Operations** - Apply configurations multiple times safely +- **Multi-Environment Support** - Manage dev, staging, and production environments +- **Version Control Friendly** - Store configurations in Git repositories +- **Template Support** - Use variables and templates for reusable configurations +- **Dependency Management** - Handle relationships between resources automatically + +## Why Babylon? + +### Before Babylon + +Without Babylon, deploying CosmoTech resources required: +- Manual API calls for each resource +- Complex scripts to coordinate deployments +- Difficult tracking of resource states +- Error-prone configuration management +- No easy way to replicate environments + +### With Babylon + +Babylon provides: +- Single command deployment of entire environments +- Automatic handling of resource dependencies +- Consistent configuration across environments +- Easy rollback and version management +- Template-based configuration for reusability + +## Core Concepts + +### Namespaces + +Namespaces organize and isolate resources: + +- **`remote`** - Environment identifier (dev, staging, production) +- **`state_id`** - Unique identifier for this deployment state +- **`context`** - Logical grouping of related resources +- **`platform`** - Target CosmoTech platform URL + +### Payload + +The payload contains the actual resource configuration that will be sent to the CosmoTech API. + +## Resource Types + +### Solution + +Define and deploy solutions: + +```yaml +kind: Solution +namespace: + remote: production + state_id: "solution-v1" + context: "manufacturing" + platform: https://api.cosmotech.com +metadata: + selector: + organization_id: o-a1b2c3d4e5f6g7 +spec: + payload: + key: manufacturing-sim + name: Manufacturing Simulation + repository: myregistry.azurecr.io/manufacturing-sim + version: 2.1.0 + sdkVersion: "11.3" + tags: + - manufacturing + - optimization + parameters: + - id: ProductionRate + varType: number + runTemplates: + - id: Simulation + name: Run Simulation + computeSize: basic +``` + +### Workspace + +Define workspaces with all configurations: + +```yaml +kind: Workspace +namespace: + remote: production + state_id: "workspace-main" + context: "manufacturing" + platform: https://api.cosmotech.com +metadata: + selector: + organization_id: o-a1b2c3d4e5f6g7 + solution_id: s-a1b2c3d4e5f6g7 +spec: + sidecars: + azure: + adx: + database: + create: true + retention: 365 + eventhub: + create: true + powerbi: + workspace: + name: Manufacturing Analytics + payload: + key: main-workspace + name: Main Workspace + description: Production workspace for manufacturing + useDedicatedEventHubNamespace: true + datasetCopy: false +``` + +### Dataset + +Manage datasets: + +```yaml +kind: Dataset +namespace: + remote: production + state_id: "dataset-reference" + context: "manufacturing" + platform: https://api.cosmotech.com +metadata: + selector: + organization_id: o-a1b2c3d4e5f6g7 +spec: + payload: + name: Reference Dataset + description: Base data for simulations + connector: + id: c-ADTConnector + parametersValues: + AZURE_DIGITAL_TWINS_URL: https://myadt.api.weu.digitaltwins.azure.net +``` + +## Using Templates and Variables + +### Variable Substitution + +Babylon supports Jinja2-style variable substitution: + +```yaml +spec: + payload: + key: {{workspace_key}} + name: {{workspace_name}} + description: {{workspace_description}} + solution: + solutionId: {{services['api.solution_id']}} +``` + +### Variable Files + +Define variables in separate files: + +**variables.yaml:** +```yaml +workspace_key: production-workspace +workspace_name: Production Workspace +workspace_description: Main production environment +organization_id: o-a1b2c3d4e5f6g7 +powerbi_workspace_name: Analytics Workspace +``` + +### Environment-Specific Variables + +Organize variables by environment: + +``` +config/ +├── common.yaml +├── dev/ +│ └── variables.yaml +├── staging/ +│ └── variables.yaml +└── production/ + └── variables.yaml +``` +## State Management + +### State Files + +Babylon maintains state files to track deployed resources: + +``` +.babylon/ +├── states/ +│ ├── production/ +│ │ ├── solution-v1.state +│ │ └── workspace-main.state +│ └── dev/ +│ ├── solution-v1.state +│ └── workspace-dev.state +``` + +### State Commands + +```bash +# List states +babylon state list + +# Show state details +babylon state show production/solution-v1 + +# Remove state (without deleting resource) +babylon state rm production/solution-v1 + +# Import existing resource into state +babylon state import solution my-solution --organization o-a1b2c3d4e5f6g7 +``` + +## Sidecars + +Sidecars are Azure resource configurations managed alongside platform resources: + +### Azure Data Explorer (ADX) + +```yaml +sidecars: + azure: + adx: + database: + create: true + retention: 365 + permissions: + - email: user@company.com + role: Admin + scripts: + - id: init + name: 00-Initialization.kql + path: "adx/scripts" +``` + +### Event Hub + +```yaml +sidecars: + azure: + eventhub: + create: true + consumers: + - displayName: adx + entity: ProbesMeasures + connectors: + - table_name: ProbesMeasures + consumer_group: adx + connection_name: ProbesMeasures + database_target: "{{organization_id}}-{{workspace_key}}" + format: JSON + compression: Gzip +``` + +### Power BI + +```yaml +sidecars: + azure: + powerbi: + workspace: + name: Analytics Workspace + reports: + - name: Dashboard + type: scenario + path: "powerbi/Dashboard.pbix" + tag: main + parameters: + - id: ADX_Cluster + value: "https://mycluster.westeurope.kusto.windows.net" + permissions: + - email: user@company.com + role: Admin +``` \ No newline at end of file diff --git a/Tutorial/docs/tutorials/hidden/docker.md b/Tutorial/docs/tutorials/hidden/docker.md new file mode 100644 index 0000000..f36dcc5 --- /dev/null +++ b/Tutorial/docs/tutorials/hidden/docker.md @@ -0,0 +1,78 @@ +# Docker + +Docker is a containerization platform that packages your simulation code and dependencies into portable, self-contained units called containers. + +## Purpose in CosmoTech Platform + +Docker enables **consistent and isolated execution** of simulations across different environments by: + +- Packaging your solution code with all dependencies +- Ensuring simulations run identically everywhere +- Isolating each scenario run in its own container +- Managing different solution versions through image tags + +## Key Benefits + +- **Reproducibility** - Same container image always produces the same results +- **Portability** - Run anywhere that supports Docker (local, cloud, Kubernetes) +- **Dependency Management** - All libraries and tools packaged together +- **Version Control** - Different versions coexist using different tags +- **Scalability** - Easily replicate containers for parallel execution +- **Isolation** - Each simulation runs independently without conflicts + +## How It Works + +1. **Build** - Create a Docker image containing your solution code +2. **Tag** - Version your image (e.g., `my-solution:1.0.0`) +3. **Push** - Upload to Azure Container Registry +4. **Register** - Reference the image in your CosmoTech solution +5. **Execute** - Platform pulls and runs the container for each scenario + +## Example Dockerfile + +```dockerfile +FROM python:3.11-slim + +WORKDIR /app + +# Install dependencies +COPY requirements.txt . +RUN pip install --no-cache-dir -r requirements.txt + +# Copy solution code +COPY src/ ./src/ + +# Run simulation +ENTRYPOINT ["python", "src/main.py"] +``` + +## Image Registry + +Solutions reference Docker images stored in Azure Container Registry: + +```yaml +spec: + payload: + repository: myregistry.azurecr.io/my-solution + version: 1.0.0 + alwaysPull: false +``` + +## Best Practices + +- **Use semantic versioning** - Tag images as `MAJOR.MINOR.PATCH` +- **Keep images small** - Use minimal base images (e.g., `-slim`, `-alpine`) +- **Don't hardcode secrets** - Use environment variables +- **Test locally first** - Run containers locally before deploying +- **Version alignment** - Solution version must match image tag + +## Container Execution + +When a scenario runs: +1. Kubernetes pulls the Docker image from the registry +2. Container starts with scenario parameters as environment variables +3. Your simulation code executes inside the container +4. Results are sent to Event Hub and stored in ADX +5. Container is cleaned up after completion + +Docker ensures your simulation runs consistently and reliably, regardless of where it's executed. \ No newline at end of file diff --git a/Tutorial/docs/tutorials/hidden/frontend.md b/Tutorial/docs/tutorials/hidden/frontend.md new file mode 100644 index 0000000..ef5dd93 --- /dev/null +++ b/Tutorial/docs/tutorials/hidden/frontend.md @@ -0,0 +1,143 @@ +# Frontend + +Learn about the CosmoTech web application interface and how to create datasets and scenarios to run simulations. + +## Overview + +The CosmoTech web application provides an intuitive interface for: +- Creating and managing datasets +- Configuring and running scenarios +- Monitoring simulation execution +- Visualizing results through Power BI dashboards + +This guide walks you through the essential steps to get started with your first simulation. + +## Creating a Dataset + +Datasets contain the input data that your scenarios will use for simulations. Before running any scenario, you need to create at least one dataset. + +### Step 1: Navigate to the Dataset View + +When you first access a workspace with no datasets, you'll see an empty dataset view. + +![No Dataset](../../assets/webapp/empty_dataset.png) + +### Step 2: Enter Dataset Information + +Click the **Create Dataset** button and provide the basic information: + +- **Name** - A descriptive name for your dataset +- **Description** - Optional details about the dataset contents +- **Tags** - Optional labels for organization + +![Creating Dataset - Basic Information](../../assets/webapp/create_dataset.png) + +### Step 3: Configure Data Source + +Select your data source type and configure the connection parameters: + +- **Source Type** - Choose from available connectors (ADT, Azure Storage, Local File, etc.) +- **Connection Parameters** - Provide authentication and location details +- **Data Mapping** - Configure how data should be imported + +![Creating Dataset - Source Configuration](../../assets/webapp/create_dataset_2.png) + +### Step 4: Dataset Import + +The web application will begin importing your data. Depending on the data size, this may take a few moments. + +![Dataset Created Successfully](../../assets/webapp/created_dataset.png) + +Once the import is complete, your dataset will be ready to use in scenarios. + +## Creating a Scenario + +Scenarios define the simulation parameters and execution settings. Each scenario is linked to a dataset and a run template. + +### Step 1: Navigate to the Scenario View + +When you first access a workspace with no scenarios, you'll see an empty scenario view. + +![No Scenario](../../assets/webapp/empty_scenario.png) + +### Step 2: Configure Scenario Details + +Click the **Create Scenario** button and provide the configuration: + +- **Name** - A descriptive name for your scenario +- **Description** - Optional details about what this scenario simulates +- **Run Template** - Select the type of simulation to execute +- **Dataset** - Choose the dataset created in the previous steps +- **Parameters** - Configure scenario-specific parameters based on the selected run template + +![Create Scenario](../../assets/webapp/create_scenario.png) + +### Step 3: Scenario Created + +Once created, your scenario is ready to run. You can modify parameters or execute the simulation. + +![Created Scenario](../../assets/webapp/created_scenario.png) + +## Running a Scenario + +### Execute the Simulation + +Click the **Run** button to start the simulation. The scenario will be submitted to the execution queue. + +### Monitor Progress + +The web application displays the execution status in real-time: +- **Pending** - Waiting for resources +- **Running** - Simulation in progress +- **Successful** - Completed successfully +- **Failed** - Execution encountered an error + +![Running Scenario](../../assets/webapp/running_scenario.png) + +### View Results + +Once the scenario has finished running successfully, you can: +- View results in the **Dashboards** tab (Power BI reports) +- Analyze data directly in **Azure Data Explorer** +- Compare results with other scenarios +- Export data for further analysis + +The simulation output data is automatically sent to Event Hub, ingested into ADX, and made available for visualization in Power BI dashboards. + +## Next Steps + +After running your first scenario: + +1. **Explore Results** - Navigate to the Dashboards tab to view Power BI reports +2. **Create Variations** - Clone your scenario and adjust parameters to explore different outcomes +3. **Compare Scenarios** - Use comparison features to analyze differences +4. **Query Raw Data** - Access ADX directly for custom data analysis +5. **Share Insights** - Export reports and share findings with your team + +## Tips and Best Practices + +- **Naming Conventions** - Use clear, descriptive names for datasets and scenarios +- **Dataset Reuse** - Multiple scenarios can share the same dataset +- **Parameter Documentation** - Add descriptions to help others understand your scenario configuration +- **Version Control** - Create scenario clones before making significant changes +- **Monitor Resources** - Check execution status and logs if a scenario fails +- **Validate Data** - Ensure datasets are correctly imported before running scenarios + +## Troubleshooting + +### Dataset Import Issues +- Verify connection parameters and authentication +- Check data source accessibility +- Review error messages in the dataset details + +### Scenario Execution Failures +- Confirm the dataset is valid and complete +- Check parameter values are within acceptable ranges +- Review execution logs for error details +- Verify the Docker image is available in the registry + +### Dashboard Not Loading +- Ensure the scenario has completed successfully +- Verify Power BI workspace permissions +- Check that ADX contains the expected data +- Refresh the dashboard view \ No newline at end of file diff --git a/Tutorial/docs/tutorials/hidden/power-bi.md b/Tutorial/docs/tutorials/hidden/power-bi.md new file mode 100644 index 0000000..e07eb82 --- /dev/null +++ b/Tutorial/docs/tutorials/hidden/power-bi.md @@ -0,0 +1,118 @@ +# Power BI + +Embed and use Power BI dashboards in your web application to visualize and analyze data stored in Azure Data Explorer (ADX). + +## Overview + +Power BI is a powerful business analytics tool that enables you to create interactive visualizations and reports from your data. When integrated with Azure Data Explorer (ADX), Power BI provides a robust solution for understanding and representing large volumes of simulation data generated by the CosmoTech platform. + +## Why Use Power BI with ADX? + +The combination of Power BI and ADX offers several advantages: + +- **Real-time Analytics** - Query and visualize large datasets stored in ADX with minimal latency +- **Interactive Dashboards** - Create dynamic, interactive reports that allow users to explore data through filtering and drill-down capabilities +- **Scalability** - Handle massive amounts of time-series and telemetry data efficiently +- **Custom Visualizations** - Build tailored charts, graphs, and KPIs specific to your simulation scenarios +- **Embedded Reports** - Seamlessly integrate Power BI reports directly into your web application + +## Common Data Tables in ADX + +The CosmoTech platform typically stores the following data in ADX: + +- **ProbesMeasures** - Time-series data from simulation probes and measurements +- **ScenarioMetadata** - Information about scenarios (name, description, parameters) +- **ScenarioRun** - Execution details and results of scenario runs +- **ScenarioRunMetadata** - Metadata associated with each scenario run +- **SimulationTotalFacts** - Aggregated facts and metrics from simulations + +## Creating Visualizations + +### Basic Steps + +1. **Import Data** - Connect to ADX and load relevant tables +2. **Transform Data** - Use Power Query Editor to clean and shape your data +3. **Create Relationships** - Define relationships between tables (e.g., linking runs to scenarios) +4. **Build Visuals** - Add charts, tables, and graphs to your report canvas +5. **Add Interactivity** - Configure filters, slicers, and drill-through actions +6. **Publish** - Publish your report to Power BI Service for sharing and embedding + +### Example Visualizations + +- **Line Charts** - Display time-series trends from ProbesMeasures +- **Bar Charts** - Compare metrics across different scenarios +- **KPI Cards** - Show key performance indicators at a glance +- **Tables** - Present detailed scenario parameters and results +- **Maps** - Visualize geographic data if applicable to your simulation + +## Using KQL in Power BI + +You can write Kusto Query Language (KQL) queries directly in Power BI to: + +- Filter and aggregate data before importing +- Perform complex calculations and transformations +- Optimize query performance for large datasets + +Example KQL query: +```kql +ProbesMeasures +| where SimulationRun == "run-123" +| summarize avg(Value) by bin(Timestamp, 1h), ProbeName +| order by Timestamp asc +``` + +## Embedding Reports in the Web Application + +Power BI reports can be embedded directly into the CosmoTech web application: + +1. **Publish to Power BI Service** - Upload your report to a Power BI workspace +2. **Configure Workspace** - Set up the workspace in your CosmoTech configuration +3. **Define Report Settings** - Specify display options, filters, and navigation preferences +4. **Apply Dynamic Filters** - Link report filters to scenario data for context-aware visualizations + +The workspace configuration includes settings for: +- Report tags and display names +- Navigation pane options +- Filter pane visibility +- Dynamic filters based on scenario context + +## Best Practices + +- **Optimize Queries** - Use aggregations and filters to reduce data volume +- **Incremental Refresh** - Configure incremental refresh for large datasets +- **Limit Visuals** - Keep reports focused with a reasonable number of visualizations +- **Use Bookmarks** - Create bookmarks for different views and perspectives +- **Test Performance** - Ensure reports load quickly even with large datasets +- **Document Calculations** - Add descriptions to measures and calculated columns + +## Dynamic Filtering + +Power BI reports in the CosmoTech web application support dynamic filtering based on the current scenario context. This allows users to: + +- Automatically view data relevant to the selected scenario +- Compare results across different scenario runs +- Filter by specific parameters or time ranges + +Dynamic filters are configured in the workspace settings and can link Power BI report filters to scenario metadata stored in ADX. + +## Troubleshooting + +### Common Issues + +- **Connection Errors** - Verify cluster URI and database name are correct +- **Authentication Failures** - Ensure proper permissions are set in ADX and Power BI +- **Slow Performance** - Optimize KQL queries and consider data aggregation +- **Empty Reports** - Check that data exists in ADX tables and queries are correct + +### Performance Tips + +- Use DirectQuery mode for real-time data or Import mode for better performance +- Create aggregation tables in ADX for frequently accessed metrics +- Limit the date range of data being queried +- Use materialized views in ADX for complex calculations + +## Additional Resources + +- [Power BI Documentation](https://docs.microsoft.com/power-bi/) +- [Azure Data Explorer Documentation](https://docs.microsoft.com/azure/data-explorer/) +- [KQL Query Language Reference](https://docs.microsoft.com/azure/data-explorer/kusto/query/)