Main 3d

modules/accelerators/contextualization/cdf_three_dimenssion/README.md

@@ -0,0 +1,109 @@
+# 3D contextualization (CDF Toolkit)
+
+This CDF Toolkit module deploys a **data modeling (DM)–first** pipeline that contextualizes 3D CAD nodes with Cognite Asset instances: manual and rule-based input, ML matching, quality review, and RAW outputs for iterative tuning.
+
+It is parameterized by **`default_location`** and **`source_name`** (for example `clov` / `navisworks`), so resource external IDs follow the pattern `…_{{default_location}}_{{source_name}}` after build.
+
+## Prerequisites
+
+- A **DM-only or hybrid** CDF project with the 3D chain and views required for Industrial Tools / CAD contextualization. Follow **[docs/3d_contextualization_dm_only_guide.md](docs/3d_contextualization_dm_only_guide.md)** for spaces, `Cognite3DObject` / `CADNode` linking, scene configuration, and common pitfalls.
+- After configuring **`asset_instance_space`**, **view** (`assetView*` / `asset_view_*`), and pipeline IDs for your project, use **[docs/CONFIGURATION_VERIFICATION.md](docs/CONFIGURATION_VERIFICATION.md)** to verify live CDF pipeline config and instance vs view spaces.
+- A **3D model** ingested and processed in CDF (file upload via UI or the file extractor pattern in this module).
+- Toolkit **`variables`** set in your environment config (see `default.config.dev.yaml` and `env.template`).
+
+Optional reference data (for demos): small asset / OID-style examples can be adapted; the module itself is not tied to a single field like Valhall.
+
+## Processing workflow
+
+High-level contextualization flow (manual input → ML → good/bad RAW → retune):
+
+![Processing workflow](https://github.com/cognitedata/toolkit/assets/31886431/b29522f8-7f4b-4e23-b06a-f3ffffde103c)
+
+## Managed resources
+
+Names below use toolkit variables; **deployed** external IDs use your configured `default_location` and `source_name`.
+
+### 1. Auth groups
+
+| Pattern | Purpose |
+|--------|---------|
+| `gp_3d_{{default_location}}_extractor` | Extractor: RAW, files, extraction runs for the 3D dataset and source pipeline |
+| `gp_3d_{{default_location}}_processing` | Runs contextualization functions: RAW, DM instances, files, 3D, sessions, pipelines |
+| `gp_3d_{{default_location}}_read` | Read access to 3D / files for consumers |
+
+Source IDs for each group come from your IdP (`3d_location_*_group_source_id` in config).
+
+### 2. Data set
+
+| External ID | Role |
+|-------------|------|
+| `ds_3d_{{default_location}}` | Lineage for extraction pipelines, functions, RAW, and files |
+
+### 3. Extraction pipelines
+
+| External ID pattern | Role |
+|---------------------|------|
+| `ep_src_3d_{{default_location}}_{{source_name}}` | File extractor: upload 3D files from local disk, SharePoint, etc. |
+| `ep_ctx_3d_{{default_location}}_{{source_name}}_annotation` | Main **3D ↔ asset** contextualization (CDF Function) |
+| `ep_ctx_3d_{{default_location}}_{{source_name}}_annotation_quality_check` | Quality check on contextualization results |
+| `ep_ctx_3d_{{default_location}}_{{source_name}}_upload_manual_mappings` | Load **CSV** mappings into RAW (`3DId`, `assetId`, …) |
+| `ep_ctx_3d_{{default_location}}_{{source_name}}_upload_asset_hierarchy` | Load **asset hierarchy** CSV into DM asset instances |
+
+Pipeline documentation and **runtime parameters** (RAW DB, DM asset/CAD spaces, model name, thresholds) live in the corresponding `*.config.Config.yaml` next to each pipeline.
+
+### 4. Functions
+
+| External ID pattern | Role |
+|---------------------|------|
+| `fn_context_3d_{{default_location}}_{{source_name}}_asset` | Main annotation / contextualization |
+| `fn_context_3d_{{default_location}}_{{source_name}}_quality_check` | Post-run quality checks |
+| `fn_context_3d_{{default_location}}_{{source_name}}_upload_manual_mappings` | CSV → `contextualization_manual_input` |
+| `fn_context_3d_{{default_location}}_{{source_name}}_upload_asset_hierarchy` | CSV → DM assets |
+
+The main contextualization function, in normal operation, follows the pipeline described on the annotation extraction pipeline (read manual RAW, apply overrides, match assets using DM configuration, write **good** / **bad** tables for workflow and tuning).
+
+### 5. RAW database and tables
+
+| Database | Tables |
+|----------|--------|
+| `3d_{{default_location}}_{{source_name}}` | `contextualization_good`, `contextualization_bad`, `contextualization_manual_input`, `contextualization_rule` (optional rule-based mapping before ML) |
+
+### Illustrations
+
+**3D data pipeline (time series / context):**
+
+![3D data pipeline](https://github.com/cognitedata/toolkit/assets/31886431/f1129181-bab0-42cb-8366-860e8fb30d7e)
+
+**Contextualization workflow** (good/bad tables, manual and rule modules):
+
+![Contextualization workflow](https://github.com/cognitedata/toolkit/assets/31886431/0e990b47-0c06-4040-b680-7e2dddcdccee)
+
+## Variables
+
+Set module variables in **`default.config.yaml`** (merged by the Toolkit). Values such as DM spaces, model names, and `function_space` are read from **environment variables** (see **`env.template`**): `FUNCTION_SPACE`, `THREE_D_MODEL_NAME`, `CAD_MODEL_NAME`, `DEFAULT_CAD_SPACE`, `DEFAULT_DM_SPACE`, `DATA_MODEL_SPACE`, `DM_EXT_ID`, `DM_VERSION`, `ASSET_INSTANCE_SPACE`, `CAD_NODE_INSTANCE_SPACE`, plus `CDF_PROJECT`, `CDF_CLUSTER`, IDP and CI/CD vars as already referenced in `default.config.yaml`. Copy `env.template` → `.env` and fill in real values before `cdf build`.
+
+| Variable | Description |
+|----------|-------------|
+| `default_location` | Short location key used in resource names (e.g. `clov`) |
+| `source_name` | Source system key (e.g. `navisworks`) |
+| `3d_model_name` | 3D / CAD model identifier in CDF (from `.env`: `THREE_D_MODEL_NAME`) |
+| `3d_dataset` | Data set external ID for 3D resources (typically `ds_3d_<location>`) |
+| `raw_db`, `raw_table_manual` | RAW database and manual-input table (for scripts / docs) |
+| `asset_instance_space`, `cad_node_instance_space` | DM instance spaces (from `.env`: `ASSET_INSTANCE_SPACE`, `CAD_NODE_INSTANCE_SPACE`) |
+| `function_space` | Space for Cognite Function code artifacts (from `.env`: `FUNCTION_SPACE`) |
+| `default_dm_space`, `dm_ext_id`, `dm_version` | Data model reference (spaces / ext id from `.env` via `${DM_EXT_ID}` / `${DM_VERSION}`) |
+| `default_cad_space`, `default_scene_space` | CAD and scene configuration spaces (`DEFAULT_CAD_SPACE` in `.env`) |
+| `required_views`, `cad_model_view`, … | View references for DM 3D chain (see dev config) |
+| `file_extractor_watch_path` | Local path for file extractor config (if used) |
+| `3d_location_extractor_group_source_id`, `3d_location_processing_group_source_id`, `3d_location_read_group_source_id` | IdP object IDs for the three auth groups |
+| `cicd_clientId`, `cicd_clientSecret` | Optional: schedules / automation |
+
+Legacy variable **`external_root_id_asset`** may appear in older templates; DM deployments use **`asset_instance_space`** / **`cad_node_instance_space`** in pipeline config instead.
+
+## Usage
+
+Copy this module into your project’s `custom_modules` (or reference it from `modules/`), adjust **`variables`** for your project and IdP groups, then add it under **`selected_modules_and_packages`** in your `config.<env>.yaml` and deploy with the CDF Toolkit.
+
+See [Using Templates](https://developer.cognite.com/sdks/toolkit/templates).
+
+**Note:** Cognite Functions have **time and memory limits**. Very large asset or node volumes may need batching or a different runtime than a single function invocation.

modules/accelerators/contextualization/cdf_three_dimenssion/auth/3d.Group.yaml

@@ -0,0 +1,95 @@
+# Single group for all 3D pipeline actors: file extractor, contextualization functions, and read-only users.
+# Merged from 3d.extractor.Group, 3d.processing.Group, 3d.read.Group for simplified dev/small-team setup.
+name: 'gp_3d_{{default_location}}'
+sourceId: '{{3d_location_group_source_id}}'
+metadata:
+  origin: 'cdf-project-templates'
+capabilities:
+  # RAW: read/write (extractor + processing)
+  - rawAcl:
+      actions:
+        - READ
+        - WRITE
+      scope:
+        tableScope:
+          dbsToTables:
+            3d_{{default_location}}_{{source_name}}: []
+
+  # Files: read/write (extractor uploads, processing reads, users view)
+  - filesAcl:
+      actions:
+        - READ
+        - WRITE
+      scope:
+        datasetScope:
+          ids: ['ds_3d_{{default_location}}']
+
+  # 3D models: read (processing + users)
+  - threedAcl:
+      actions:
+        - READ
+      scope:
+        datasetScope:
+          ids: ['ds_3d_{{default_location}}']
+
+  # Data Model instances: read/write (processing writes AssetExtension nodes)
+  - dataModelInstancesAcl:
+      actions:
+        - READ
+        - WRITE
+      scope:
+        spaceIdScope:
+          spaceIds:
+            - '{{data_model_space}}'
+            - cdf_cdm
+
+  # Extraction pipelines: read configs, read/write runs
+  - extractionConfigsAcl:
+      actions:
+        - READ
+        - WRITE
+      scope:
+        datasetScope:
+          ids: ['ds_3d_{{default_location}}']
+  - extractionRunsAcl:
+      actions:
+        - READ
+        - WRITE
+      scope:
+        datasetScope:
+          ids: ['ds_3d_{{default_location}}']
+
+  # Sessions: needed for CDF Functions
+  - sessionsAcl:
+      actions:
+        - LIST
+        - CREATE
+        - DELETE
+      scope:
+        all: {}
+
+  # Entity matching + functions (contextualization processing)
+  - entitymatchingAcl:
+      actions:
+        - READ
+        - WRITE
+      scope:
+        all: {}
+  - functionsAcl:
+      actions:
+        - READ
+        - WRITE
+      scope:
+        all: {}
+
+  # Annotations + labels: read (users)
+  - annotationsAcl:
+      actions:
+        - READ
+      scope:
+        all: {}
+  - labelsAcl:
+      actions:
+        - READ
+      scope:
+        all: {}

modules/accelerators/contextualization/cdf_three_dimenssion/data_sets/dataset.yaml

@@ -0,0 +1,14 @@
+externalId: ds_3d_{{default_location}}
+name:  3d:{{default_location}}
+description: 3D data for {{default_location}} uploaded from sharepoint
+metadata:
+  consoleSource:
+    names:
+      - '{{source_name}}'
+  rawTables:
+    - databaseName: 3d_{{default_location}}_{{source_name}}
+      tableName: 'contextualization_good'
+    - databaseName: '3d_{{default_location}}_{{source_name}}'
+      tableName: 'contextualization_bad'
+    - databaseName: '3d_{{default_location}}_{{source_name}}'
+      tableName: 'contextualization_manual_input'

modules/accelerators/contextualization/cdf_three_dimenssion/default.config.yaml

@@ -0,0 +1,137 @@
+environment:
+  name: dev
+  project: shivam-qs
+  validation-type: dev
+  selected:
+  - modules/
+
+variables:
+  modules:
+    accelerators:
+      contextualization:
+        cdf_three_dimenssion:
+          default_location: ${default_location}
+          source_name: ${source_name}
+          raw_db: ${RAW_DB}          # raw database name for manual mappings
+          raw_table_manual: contextualization_manual_input
+          raw_table_asset_staging: asset_hierarchy_staging
+          function_space: ${FUNCTION_SPACE}
+          3d_model_name: ${3d_model_name}   # 3d_model_name is the "name" property of cognite3dmodel
+          cad_model_name: cdf_cdm
+          cad_model_type: CAD
+          default_cad_space: cdf_cdm
+          default_dm_space: cdf_cdm
+          data_model_space: cdf_cdm
+          dm_ext_id: ${DM_EXT_ID}  # data model external id for the project
+          dm_version: ${DM_VERSION}
+          default_scene_space: scene
+          cad_contextualization_batch_size: 100
+          required_views:
+          - space: cdf_cdm
+            external_id: Cognite3DObject
+            version: v1
+
+          - space: cdf_cdm
+            external_id: CogniteCADNode
+            version: v1
+
+          - space: cdf_cdm
+            external_id: CogniteCADRevision
+            version: v1
+
+          - space: cdf_cdm
+            external_id: CogniteCADModel
+            version: v1
+
+          - space: cdf_cdm
+            external_id: Cognite3DRevision
+            version: v1
+
+          - space: cdf_cdm
+            external_id: Cognite3DModel
+            version: v1
+
+          - space: cdf_cdm
+            external_id: Cognite3DTransformation
+            version: v1
+
+          - space: cdf_cdm
+            external_id: CogniteVisualizable
+            version: v1
+
+          - space: cdf_cdm
+            external_id: Cognite360Image
+            version: v1
+
+          - space: cdf_cdm
+            external_id: Cognite360ImageAnnotation
+            version: v1
+
+          - space: cdf_cdm
+            external_id: CogniteAnnotation
+            version: v1
+
+          - space: cdf_cdm
+            external_id: Cognite360ImageCollection
+            version: v1
+
+          - space: cdf_cdm
+            external_id: Cognite360ImageStation
+            version: v1
+
+          - space: cdf_cdm
+            external_id: CognitePointCloudVolume
+            version: v1
+
+          - space: cdf_cdm
+            external_id: CognitePointCloudRevision
+            version: v1
+
+          - space: scene
+            external_id: SceneConfiguration
+            version: v1
+
+          - space: scene
+            external_id: RevisionProperties
+            version: v1
+
+          - space: cdf_3d_schema
+            external_id: Cdf3dModel
+            version: '1'
+
+          asset_instance_space: ${ASSET_INSTANCE_SPACE}
+          cad_node_instance_space: ${CAD_NODE_INSTANCE_SPACE}
+          asset_view_ext_id: ${ASSET_VIEW_EXT_ID}          #view where asset are available for contextualization
+          asset_view_version: ${ASSET_VIEW_VERSION}          #version of the view where asset are available for contextualization
+          file_extractor_watch_path: c:/tmp/files
+          3d_location_group_source_id: ${3d_location_group_source_id}   #group source id for the project
+          cicd_clientId: ${cicd_clientId}
+          cicd_clientSecret: ${cicd_clientSecret}
+          cicd_tokenUri: ${IDP_TOKEN_URL}
+          cdf_project_name: ${CDF_PROJECT}
+          cdf_cluster: ${CDF_CLUSTER}
+          external_root_id_asset: ${EXTERNAL_ROOT_ID_ASSET}  #optional: only used for legacy behavior
+          cad_model_view:
+            space: cdf_cdm
+            external_id: CogniteCADModel
+            version: v1
+
+          cad_revision_view:
+            space: cdf_cdm
+            external_id: CogniteCADRevision
+            version: v1
+
+          scene_config_view:
+            space: scene
+            external_id: SceneConfiguration
+            version: v1
+
+          scene_model_view:
+            space: cdf_3d_schema
+            external_id: Cdf3dModel
+            version: '1'
+
+          rev_props_view:
+            space: scene
+            external_id: RevisionProperties
+            version: v1