diff --git a/ai-sdk-context/README.md b/ai-sdk-context/README.md deleted file mode 100644 index 3c9bcb9..0000000 --- a/ai-sdk-context/README.md +++ /dev/null @@ -1,2 +0,0 @@ -# ai-sdk-context - diff --git a/ai-sdk-context/add-3d-models/SKILL.md b/ai-sdk-context/add-3d-models/SKILL.md new file mode 100644 index 0000000..d41719c --- /dev/null +++ b/ai-sdk-context/add-3d-models/SKILL.md @@ -0,0 +1,233 @@ +--- +name: add-3d-models +description: Add 3D models (.glb/.gltf) to a Decentraland scene using GltfContainer. Covers loading, positioning, scaling, colliders, parenting, and browsing 2,700+ free assets from the Creator Hub catalog and 991 CC0 models. Use when the user wants to add models, import GLB files, find free 3D assets, or set up model colliders. Do NOT use for materials/textures (see advanced-rendering) or model animations (see animations-tweens). +--- + +# Adding 3D Models to Decentraland Scenes + +## RULE: Use composite for initial models + +**Always add models that exist at scene load to `assets/scene/main.composite`, not in TypeScript.** + +Only use TypeScript (`engine.addEntity()` + `GltfContainer.create()`) for models spawned dynamically at runtime (e.g., a bullet instantiated on fire, an NPC summoned by an event). + +For initial/static models, define them in the composite using `core::GltfContainer` and `core::Transform`. See `{baseDir}/../composites/composite-reference.md` for the full format. + +```json +{ + "name": "core::GltfContainer", + "data": { + "512": { + "json": { + "src": "assets/asset-packs/tree_forest_01/Tree_Forest_01.glb", + "visibleMeshesCollisionMask": 0, + "invisibleMeshesCollisionMask": 3 + } + } + } +} +``` + +To add behavior to a model placed in the composite, fetch it in `index.ts` by name or tag — do NOT re-create it in code. See the **composites/composite-reference** for `getEntityOrNullByName` and `getEntitiesByTag` patterns. + +--- + +## Loading a 3D Model in TypeScript (dynamic entities only) + +Use `GltfContainer` to load `.glb` or `.gltf` files for entities spawned at runtime: + +```typescript +import { engine, Transform, GltfContainer } from '@dcl/sdk/ecs' +import { Vector3, Quaternion } from '@dcl/sdk/math' + +const model = engine.addEntity() +Transform.create(model, { + position: Vector3.create(8, 0, 8), + rotation: Quaternion.fromEulerDegrees(0, 0, 0), + scale: Vector3.create(1, 1, 1), +}) +GltfContainer.create(model, { + src: 'assets/scene/Models/myModel.glb', +}) +``` + +## File Organization + +Place model files in the `assets/scene/Models/` directory at the project root: + +``` +project/ +├── assets/ +│ └── scene/ +│ └── Models/ +│ ├── building.glb +│ ├── tree.glb +│ └── furniture/ +│ ├── chair.glb +│ └── table.glb +├── src/ +│ └── index.ts +└── scene.json +``` + +## Colliders + +### Using Model's Built-in Colliders + +Models exported with collision meshes work automatically. Set the collision mask: + +```typescript +GltfContainer.create(model, { + src: 'assets/scene/Models/building.glb', + visibleMeshesCollisionMask: + ColliderLayer.CL_PHYSICS | ColliderLayer.CL_POINTER, + invisibleMeshesCollisionMask: ColliderLayer.CL_PHYSICS, +}) +``` + +### Adding Simple Colliders + +For basic shapes, add `MeshCollider`: + +```typescript +import { MeshCollider } from '@dcl/sdk/ecs' +MeshCollider.setBox(model) // Box collider +MeshCollider.setSphere(model) // Sphere collider +``` + +## Common Model Operations + +### Scaling + +```typescript +Transform.create(model, { + position: Vector3.create(8, 0, 8), + scale: Vector3.create(2, 2, 2), // 2x size +}) +``` + +### Rotation + +```typescript +Transform.create(model, { + position: Vector3.create(8, 0, 8), + rotation: Quaternion.fromEulerDegrees(0, 90, 0), // Rotate 90° on Y axis +}) +``` + +### Parenting (Attach to Another Entity) + +```typescript +const parent = engine.addEntity() +Transform.create(parent, { position: Vector3.create(8, 0, 8) }) + +const child = engine.addEntity() +Transform.create(child, { + position: Vector3.create(0, 2, 0), // 2m above parent + parent: parent, +}) +GltfContainer.create(child, { src: 'assets/scene/Models/hat.glb' }) +``` + +### Get Global (World-Space) Position and Rotation + +When an entity is parented, `Transform.get(entity).position` returns the **local** position relative to the parent. Use `getWorldPosition` and `getWorldRotation` to get the actual world-space values: + +```typescript +import { getWorldPosition, getWorldRotation } from '@dcl/sdk/ecs' + +const worldPos = getWorldPosition(engine, childEntity) +console.log(worldPos.x, worldPos.y, worldPos.z) + +const worldRot = getWorldRotation(engine, childEntity) +console.log(worldRot.x, worldRot.y, worldRot.z, worldRot.w) +``` + +Both functions traverse the parent hierarchy to compute the final result. They return a zero vector / identity quaternion if the entity has no `Transform`. + +## Free 3D Models + +Always check the scene's local asset folder first. + +IMPORTANT: Only fetch models from the free catalogs below if the prompt explicitly asks to add new models. Confirm with the user always if they wish to add new models to their scene. + +### Creator Hub Asset Packs (2,700+ models) + +Read `{baseDir}/../context/asset-packs-catalog.md` for official Decentraland models across 12 themed packs (Cyberpunk, Fantasy, Gallery, Sci-fi, Western, Pirates, etc.) with furniture, structures, decorations, nature, and more. + +To use a Creator Hub model: + +```bash +# Download from catalog +mkdir -p assets/scene/Models +curl -o assets/scene/Models/arcade_machine.glb "https://builder-items.decentraland.org/contents/bafybei..." +``` + +```typescript +// Reference in code — must be a local file path +GltfContainer.create(entity, { src: 'assets/scene/Models/arcade_machine.glb' }) +``` + +### Open Source CC0 Models (991 models) + +Read `{baseDir}/../context/open-source-3d-assets.md` for free CC0-licensed models from Polygonal Mind, organized by 18 themed collections (MomusPark, Medieval Fair, Cyberpunk, Sci-fi, etc.) with direct GitHub download URLs. + +```bash +curl -o assets/scene/Models/tree.glb "https://raw.githubusercontent.com/ToxSam/cc0-models-Polygonal-Mind/main/projects/MomusPark/Tree_01_Art.glb" +``` + +### How to suggest models + +1. Read both catalog files +2. Search for models matching the user's description/theme +3. Suggest specific models with download commands +4. Download selected models into the scene's `assets/scene/Models/` directory +5. Reference them in code with local paths + +> **Important**: `GltfContainer` only works with **local files**. Never use external URLs for the model `src` field. Always download models into `models/` first. + +### Checking Model Load State + +Use `GltfContainerLoadingState` to check if a model has finished loading: + +```typescript +import { + GltfContainer, + GltfContainerLoadingState, + LoadingState, +} from '@dcl/sdk/ecs' + +engine.addSystem(() => { + const state = GltfContainerLoadingState.getOrNull(modelEntity) + if (state && state.currentState === LoadingState.FINISHED) { + console.log('Model loaded successfully') + } else if (state && state.currentState === LoadingState.FINISHED_WITH_ERROR) { + console.log('Model failed to load') + } +}) +``` + +## Troubleshooting + +| Problem | Cause | Solution | +| -------------------------------- | --------------------------------- | ----------------------------------------------------------------------------------------------------------- | +| Model not visible | Wrong file path | Verify the file exists at the exact path relative to project root (e.g., `assets/scene/Models/myModel.glb`) | +| Model not visible | Position outside scene boundaries | Check Transform position is within 0-16 per parcel. Center of 1-parcel scene is (8, 0, 8) | +| Model not visible | Scale is 0 or very small | Check `Transform.scale` — default is (1,1,1). Try larger values if model was exported very small | +| Model not visible | Behind the camera | Move the avatar or rotate to look in the model's direction | +| Model loads but looks wrong | Y-up vs Z-up mismatch | Decentraland uses Y-up. Re-export from Blender with "Y Up" checked | +| "FINISHED_WITH_ERROR" load state | Corrupted or unsupported .glb | Re-export the model. Use `.glb` (binary GLTF) format. Ensure no unsupported extensions | +| Clicking model does nothing | Missing collider | Add `visibleMeshesCollisionMask: ColliderLayer.CL_POINTER` to `GltfContainer` or add `MeshCollider` | + +> **Need to optimize models for scene limits?** See the **optimize-scene** skill for triangle budgets and LOD patterns. +> **Need animations from your model?** See the **animations-tweens** skill for playing GLTF animation clips with Animator. + +## Model Best Practices + +- Keep models under 50MB per file for good loading times +- Use `.glb` format (binary GLTF) — smaller than `.gltf` +- Optimize triangle count: aim for under 1,500 triangles per model for small props +- Use texture atlases when possible to reduce draw calls +- Models with embedded animations can be played with the `Animator` component +- Test model orientation — Decentraland uses Y-up coordinate system +- Materials in models should use PBR (physically-based rendering) for best results diff --git a/ai-sdk-context/add-interactivity/SKILL.md b/ai-sdk-context/add-interactivity/SKILL.md new file mode 100644 index 0000000..2824c99 --- /dev/null +++ b/ai-sdk-context/add-interactivity/SKILL.md @@ -0,0 +1,364 @@ +--- +name: add-interactivity +description: Add click handlers, hover effects, pointer events, trigger areas, raycasting, and global input to Decentraland scene entities. Use when the user wants to make objects clickable, add hover effects, detect player proximity, handle E/F key actions, or cast rays. Do NOT use for advanced input patterns like movement restriction, cursor lock, or WASD control (see advanced-input). Do NOT use for screen-space UI buttons (see build-ui). +--- + +# Adding Interactivity to Decentraland Scenes + +## RULE: Fetch composite entities — never re-create them + +If the entity to make interactive was defined in `assets/scene/main.composite`, **look it up by name or tag in `index.ts`**. Do NOT call `engine.addEntity()` + component create — that would create a duplicate. + +```typescript +import { engine, pointerEventsSystem, InputAction } from '@dcl/sdk/ecs' +import { EntityNames } from '../assets/scene/entity-names' + +export function main() { + // By name (type-safe via auto-generated EntityNames enum) + const door = engine.getEntityOrNullByName(EntityNames.Door_1) + if (door) { + pointerEventsSystem.onPointerDown( + { entity: door, opts: { button: InputAction.IA_PRIMARY, hoverText: 'Open' } }, + () => { /* open door */ } + ) + } + + // By tag (batch operations on groups of composite entities) + const crystals = engine.getEntitiesByTag('Crystal') + for (const crystal of crystals) { + pointerEventsSystem.onPointerDown( + { entity: crystal, opts: { button: InputAction.IA_PRIMARY, hoverText: 'Collect' } }, + () => { /* collect crystal */ } + ) + } +} +``` + +These lookups must happen inside `main()` or functions called after `main()` — composite entities are not instantiated before that point. + +--- + +## Decision Tree + +| Need | Approach | API | +|------|----------|-----| +| Click/hover on a specific entity | Pointer events | `pointerEventsSystem.onPointerDown()` | +| Detect player entering an area | Trigger area | `TriggerArea` + `triggerAreaEventsSystem` | +| Poll key state every frame | Global input | `inputSystem.isTriggered()` / `isPressed()` | +| Detect objects in a direction | Raycasting | `raycastSystem` or `Raycast` component | +| Read cursor position / lock state | Cursor state | `PointerLock`, `PrimaryPointerInfo` | + +--- + +## Pointer Events (Click / Hover) + +### Using the Helper System (Recommended) +```typescript +import { engine, Transform, MeshRenderer, pointerEventsSystem, InputAction } from '@dcl/sdk/ecs' +import { Vector3 } from '@dcl/sdk/math' + +const cube = engine.addEntity() +Transform.create(cube, { position: Vector3.create(8, 1, 8) }) +MeshRenderer.setBox(cube) + +// Add click handler +pointerEventsSystem.onPointerDown( + { + entity: cube, + opts: { + button: InputAction.IA_POINTER, // Left click + hoverText: 'Click me!', + maxDistance: 10 + } + }, + (event) => { + console.log('Cube clicked!', event.hit?.position) + } +) +``` + +### All Input Actions +```typescript +InputAction.IA_POINTER // Left mouse button +InputAction.IA_PRIMARY // E key +InputAction.IA_SECONDARY // F key +InputAction.IA_ACTION_3 // 1 key +InputAction.IA_ACTION_4 // 2 key +InputAction.IA_ACTION_5 // 3 key +InputAction.IA_ACTION_6 // 4 key +InputAction.IA_JUMP // Space key +InputAction.IA_FORWARD // W key +InputAction.IA_BACKWARD // S key +InputAction.IA_LEFT // A key +InputAction.IA_RIGHT // D key +InputAction.IA_WALK // Shift key +``` + +### All Event Types +```typescript +PointerEventType.PET_DOWN // Button pressed +PointerEventType.PET_UP // Button released +PointerEventType.PET_HOVER_ENTER // Cursor enters entity +PointerEventType.PET_HOVER_LEAVE // Cursor leaves entity +``` + +### Pointer Up (Release) +```typescript +pointerEventsSystem.onPointerDown( + { entity: cube, opts: { button: InputAction.IA_POINTER, hoverText: 'Hold me' } }, + () => { console.log('Pressed!') } +) + +pointerEventsSystem.onPointerUp( + { entity: cube, opts: { button: InputAction.IA_POINTER } }, + () => { console.log('Released!') } +) +``` + +### Removing Handlers +```typescript +pointerEventsSystem.removeOnPointerDown(cube) +pointerEventsSystem.removeOnPointerUp(cube) +``` + +### Important: Colliders Required +Pointer events only work on entities with a **collider**, using the `ColliderLayer.CL_POINTER` layer. Add one if your entity doesn't have a mesh: +```typescript +import { MeshCollider } from '@dcl/sdk/ecs' +MeshCollider.setBox(entity) // Invisible box collider +``` + +For GLTF models, set the collision mask: +```typescript +GltfContainer.create(entity, { + src: 'models/button.glb', + visibleMeshesCollisionMask: ColliderLayer.CL_POINTER +}) +``` + +--- + +## Trigger Areas (Proximity Detection) + +Detect when the player enters, exits, or stays inside an area: + +```typescript +import { engine, Transform, TriggerArea } from '@dcl/sdk/ecs' +import { triggerAreaEventsSystem } from '@dcl/sdk/ecs' +import { Vector3 } from '@dcl/sdk/math' + +const area = engine.addEntity() +TriggerArea.setBox(area) // or TriggerArea.setSphere(area) +Transform.create(area, { + position: Vector3.create(8, 0, 8), + scale: Vector3.create(4, 4, 4) // Size the area via Transform.scale +}) + +// Register enter/exit/stay events +triggerAreaEventsSystem.onTriggerEnter(area, (event) => { + console.log('Entity entered trigger:', event.trigger.entity) +}) + +triggerAreaEventsSystem.onTriggerExit(area, () => { + console.log('Entity exited trigger') +}) + +triggerAreaEventsSystem.onTriggerStay(area, () => { + // Called every frame while an entity is inside +}) +``` + +By default, trigger areas react to the player layer. Use `ColliderLayer` to restrict which entities activate the area: + +```typescript +import { ColliderLayer, MeshCollider } from '@dcl/sdk/ecs' + +// Area that only reacts to custom layers +TriggerArea.setBox(area, ColliderLayer.CL_CUSTOM1 | ColliderLayer.CL_CUSTOM2) + +// Mark a moving entity to activate the area +const mover = engine.addEntity() +Transform.create(mover, { position: Vector3.create(8, 0, 8) }) +MeshCollider.setBox(mover, ColliderLayer.CL_CUSTOM1) +``` + +--- + +## Raycasting + +### Raycast Direction Types + +Four direction modes are available: + +```typescript +// 1. Local direction — relative to entity rotation +{ $case: 'localDirection', localDirection: Vector3.Forward() } + +// 2. Global direction — world-space, ignores entity rotation +{ $case: 'globalDirection', globalDirection: Vector3.Down() } + +// 3. Global target — aim at a world position +{ $case: 'globalTarget', globalTarget: Vector3.create(10, 0, 10) } + +// 4. Target entity — aim at another entity +{ $case: 'targetEntity', targetEntity: entityId } +``` + +### Callback-Based Raycasting (Recommended) + +```typescript +import { raycastSystem, RaycastQueryType, ColliderLayer } from '@dcl/sdk/ecs' + +// Local direction raycast +raycastSystem.registerLocalDirectionRaycast( + { entity: myEntity, opts: { queryType: RaycastQueryType.RQT_HIT_FIRST, direction: Vector3.Forward(), maxDistance: 16, collisionMask: ColliderLayer.CL_POINTER } }, + (result) => { + if (result.hits.length > 0) { + console.log('Hit:', result.hits[0].entityId) + } + } +) + +// Global direction raycast +raycastSystem.registerGlobalDirectionRaycast( + { entity: myEntity, opts: { queryType: RaycastQueryType.RQT_HIT_FIRST, direction: Vector3.Down(), maxDistance: 20 } }, + (result) => { /* handle hits */ } +) + +// Target position raycast +raycastSystem.registerGlobalTargetRaycast( + { entity: myEntity, opts: { globalTarget: Vector3.create(8, 0, 8), maxDistance: 20 } }, + (result) => { /* handle result */ } +) + +// Target entity raycast +raycastSystem.registerTargetEntityRaycast( + { entity: sourceEntity, opts: { targetEntity: targetEntity, maxDistance: 15 } }, + (result) => { /* handle result */ } +) + +// Remove raycast from entity +raycastSystem.removeRaycasterEntity(myEntity) +``` + +### Component-Based Raycasting + +```typescript +import { engine, Raycast, RaycastResult, RaycastQueryType } from '@dcl/sdk/ecs' +import { Vector3 } from '@dcl/sdk/math' + +const rayEntity = engine.addEntity() +Raycast.create(rayEntity, { + direction: { $case: 'localDirection', localDirection: Vector3.Forward() }, + maxDistance: 16, + queryType: RaycastQueryType.RQT_HIT_FIRST, + continuous: false // Set true for continuous raycasting +}) + +// Check results +engine.addSystem(() => { + const result = RaycastResult.getOrNull(rayEntity) + if (result && result.hits.length > 0) { + const hit = result.hits[0] + console.log('Hit entity:', hit.entityId, 'at', hit.position) + } +}) +``` + +### Camera Raycast + +Cast a ray from the camera to detect what the player is looking at: + +```typescript +raycastSystem.registerGlobalDirectionRaycast( + { + entity: engine.CameraEntity, + opts: { + direction: Vector3.rotate(Vector3.Forward(), Transform.get(engine.CameraEntity).rotation), + maxDistance: 16 + } + }, + (result) => { + if (result.hits.length > 0) console.log('Looking at:', result.hits[0].entityId) + } +) +``` + +--- + +## Global Input Handling + +Listen for key presses anywhere (not entity-specific): + +```typescript +import { inputSystem, InputAction, PointerEventType } from '@dcl/sdk/ecs' + +engine.addSystem(() => { + // Check if E key was just pressed this frame + if (inputSystem.isTriggered(InputAction.IA_PRIMARY, PointerEventType.PET_DOWN)) { + console.log('E key pressed!') + } + + // Check if a key is currently held down + if (inputSystem.isPressed(InputAction.IA_SECONDARY)) { + console.log('F key is held!') + } + + // Entity-specific input via system + const clickData = inputSystem.getInputCommand( + InputAction.IA_POINTER, + PointerEventType.PET_DOWN, + myEntity + ) + if (clickData) { + console.log('Entity clicked via system:', clickData.hit.entityId) + } +}) +``` + +## Cursor State + +```typescript +import { PointerLock, PrimaryPointerInfo } from '@dcl/sdk/ecs' + +// Check if cursor is locked +const isLocked = PointerLock.get(engine.CameraEntity).isPointerLocked + +// Get cursor position and world ray +const pointerInfo = PrimaryPointerInfo.get(engine.RootEntity) +console.log('Cursor position:', pointerInfo.screenCoordinates) +console.log('World ray direction:', pointerInfo.worldRayDirection) +``` + +--- + +## Toggle Pattern (Click to Switch States) + +Common pattern for toggleable objects: + +```typescript +let doorOpen = false + +pointerEventsSystem.onPointerDown( + { entity: door, opts: { button: InputAction.IA_POINTER, hoverText: 'Toggle door' } }, + () => { + doorOpen = !doorOpen + const mutableTransform = Transform.getMutable(door) + mutableTransform.rotation = doorOpen + ? Quaternion.fromEulerDegrees(0, 90, 0) + : Quaternion.fromEulerDegrees(0, 0, 0) + } +) +``` + +## Best Practices + +- Always set `maxDistance` on pointer events (8-16m is typical) +- Always set `hoverText` so users know what outcome their interaction will have +- Clean up handlers when entities are removed +- Use `MeshCollider` for invisible trigger surfaces +- For complex interactions, use a system with state tracking +- Set `continuous: false` on raycasts unless you need per-frame results +- Design for both desktop and mobile — mobile has no keyboard, rely on pointer and on-screen buttons + +For the full input action list and advanced patterns, see `{baseDir}/references/input-reference.md`. diff --git a/ai-sdk-context/add-interactivity/references/input-reference.md b/ai-sdk-context/add-interactivity/references/input-reference.md new file mode 100644 index 0000000..f86286b --- /dev/null +++ b/ai-sdk-context/add-interactivity/references/input-reference.md @@ -0,0 +1,365 @@ +# Input System Reference + +## All Input Actions + +| Action | Key Binding | Constant | +| ----------------- | ----------------- | -------------------------- | +| Left mouse button | Mouse click / tap | `InputAction.IA_POINTER` | +| Primary action | E key | `InputAction.IA_PRIMARY` | +| Secondary action | F key | `InputAction.IA_SECONDARY` | +| Action 3 | 1 key | `InputAction.IA_ACTION_3` | +| Action 4 | 2 key | `InputAction.IA_ACTION_4` | +| Action 5 | 3 key | `InputAction.IA_ACTION_5` | +| Action 6 | 4 key | `InputAction.IA_ACTION_6` | +| Jump | Space key | `InputAction.IA_JUMP` | +| Forward | W key | `InputAction.IA_FORWARD` | +| Backward | S key | `InputAction.IA_BACKWARD` | +| Left | A key | `InputAction.IA_LEFT` | +| Right | D key | `InputAction.IA_RIGHT` | +| Walk | Shift key | `InputAction.IA_WALK` | + +**Notes:** + +- Mouse wheel is **not available** as an input +- Always design for both desktop and mobile — mobile has no keyboard, rely on pointer and on-screen buttons +- Set `maxDistance` on pointer events (8-10 meters typical) to prevent interactions from across the scene +- Use `hoverText` to communicate what an interaction does before the player commits + +## All Pointer Event Types + +```typescript +PointerEventType.PET_DOWN // Button/key pressed +PointerEventType.PET_UP // Button/key released +PointerEventType.PET_HOVER_ENTER // Cursor enters entity bounds +PointerEventType.PET_HOVER_LEAVE // Cursor leaves entity bounds +``` + +## Declarative Pointer Events Component + +Instead of the callback system, you can use the `PointerEvents` component directly: + +```typescript +import { PointerEvents, PointerEventType, InputAction } from '@dcl/sdk/ecs' + +PointerEvents.create(entity, { + pointerEvents: [ + { + eventType: PointerEventType.PET_DOWN, + eventInfo: { + button: InputAction.IA_POINTER, + hoverText: 'Click me', + showFeedback: true, + maxDistance: 10, + }, + }, + ], +}) +``` + +Then read results in a system using `inputSystem.getInputCommand()`. + +## Proximity Interactions + +Proximity interactions detect button events when a player is near and roughly facing an entity, **without requiring them to aim their cursor at it**. Unlike pointer events (which use raycasting), proximity events check for entities within a wide triangular slice of a sphere projecting forward from the player's position. + +Key distinction: avatar facing direction matters, independently of where the camera is pointing. + +### onProximityDown / onProximityUp + +```typescript +import { pointerEventsSystem, InputAction } from '@dcl/sdk/ecs' + +pointerEventsSystem.onProximityDown( + { + entity: myEntity, + opts: { + button: InputAction.IA_PRIMARY, + hoverText: 'Press E', + maxPlayerDistance: 5, + }, + }, + function () { + console.log('Player pressed button near entity') + } +) + +pointerEventsSystem.onProximityUp( + { + entity: myEntity, + opts: { + button: InputAction.IA_PRIMARY, + hoverText: 'Release E', + maxPlayerDistance: 5, + }, + }, + function () { + console.log('Player released button near entity') + } +) +``` + +> **Note:** Only one `onProximityDown` and one `onProximityUp` can be registered per entity. Once added, they keep listening until removed. Do not call these inside a system loop — that would keep rewriting the behavior. + +### onProximityEnter / onProximityLeave + +Fires when the player walks into or out of an entity's proximity range. Use this to play sounds, trigger animations, or show hints when the player approaches. + +```typescript +pointerEventsSystem.onProximityEnter( + { + entity: myEntity, + opts: { + button: InputAction.IA_POINTER, + hoverText: 'Nearby', + maxPlayerDistance: 5, + }, + }, + function () { + console.log('Player entered proximity') + } +) + +pointerEventsSystem.onProximityLeave( + { + entity: myEntity, + opts: { + button: InputAction.IA_POINTER, + hoverText: 'Nearby', + maxPlayerDistance: 5, + }, + }, + function () { + console.log('Player left proximity') + } +) +``` + +### Options + +| Option | Description | +| ------------------- | ------------------------------------------------------------------------------------------------------------------------ | +| `button` | Which button to listen for (`InputAction.IA_PRIMARY`, `IA_SECONDARY`, `IA_POINTER`, etc.) | +| `maxPlayerDistance` | Max distance from the player's **avatar** to the entity (meters). This is the most relevant option for proximity events. | +| `maxDistance` | Max distance from the player's **camera** to the entity (meters). | +| `hoverText` | Text shown in the UI when the player is in range. | +| `showHighlight` | Show an edge highlight on the entity when player is in range. Default: `true`. | +| `showFeedback` | Show hover feedback around the center of the entity. Default: `true`. | +| `priority` | Conflict resolution when multiple entities are in range. Higher values respond first. | + +### Priority + +When multiple entities are within range and could respond to the same input, only the closest one responds by default. Use `priority` to control which takes precedence — higher values win. + +Pointer interactions (cursor aimed at entity) **always take priority** over proximity interactions, regardless of priority values. + +```typescript +// Door has higher priority than floor when both are in range +pointerEventsSystem.onProximityDown( + { + entity: doorEntity, + opts: { + button: InputAction.IA_PRIMARY, + hoverText: 'Open door', + maxPlayerDistance: 5, + priority: 2, + }, + }, + () => { + console.log('Door activated') + } +) + +pointerEventsSystem.onProximityDown( + { + entity: floorEntity, + opts: { + button: InputAction.IA_PRIMARY, + hoverText: 'Step here', + maxPlayerDistance: 5, + priority: 1, + }, + }, + () => { + console.log('Floor activated') + } +) +``` + +### Remove Callbacks + +```typescript +pointerEventsSystem.removeOnProximityDown(myEntity) +pointerEventsSystem.removeOnProximityUp(myEntity) +pointerEventsSystem.removeOnProximityEnter(myEntity) +pointerEventsSystem.removeOnProximityLeave(myEntity) +``` + +### System-Based Proximity (PointerEvents Component) + +For the system-based approach, use `PET_PROXIMITY_ENTER` and `PET_PROXIMITY_LEAVE` in the `PointerEvents` component, and `InteractionType.IT_PROXIMITY` for proximity button presses: + +```typescript +import { PointerEvents, PointerEventType, InputAction } from '@dcl/sdk/ecs' + +PointerEvents.create(myEntity, { + pointerEvents: [ + { + eventType: PointerEventType.PET_PROXIMITY_ENTER, + eventInfo: { + button: InputAction.IA_PRIMARY, + hoverText: 'Approach', + maxDistance: 5, + }, + }, + { + eventType: PointerEventType.PET_PROXIMITY_LEAVE, + eventInfo: { + button: InputAction.IA_PRIMARY, + maxDistance: 5, + }, + }, + ], +}) +``` + +Then read results in a system using `inputSystem.getInputCommand()` with `InteractionType.IT_PROXIMITY`. + +### Example: Proximity Door + +Opens or closes a door when the player presses E while nearby, without needing to aim at it: + +```typescript +import { engine, Transform, GltfContainer, Tween } from '@dcl/sdk/ecs' +import { Vector3, Quaternion } from '@dcl/sdk/math' +import { pointerEventsSystem, InputAction } from '@dcl/sdk/ecs' + +const doorPivot = engine.addEntity() +Transform.create(doorPivot, { position: Vector3.create(3, 0, 4) }) + +const door = engine.addEntity() +GltfContainer.create(door, { src: 'assets/door.glb' }) +Transform.create(door, { + position: Vector3.create(-1, 0, 0), + parent: doorPivot, +}) + +let isDoorOpen = false +const closedRot = Quaternion.fromEulerDegrees(0, 0, 0) +const openRot = Quaternion.fromEulerDegrees(0, 90, 0) + +pointerEventsSystem.onProximityDown( + { + entity: door, + opts: { + button: InputAction.IA_PRIMARY, + hoverText: 'Open / Close', + maxPlayerDistance: 5, + priority: 1, + }, + }, + function () { + if (isDoorOpen) { + Tween.setRotate(doorPivot, openRot, closedRot, 700) + isDoorOpen = false + } else { + Tween.setRotate(doorPivot, closedRot, openRot, 700) + isDoorOpen = true + } + } +) +``` + +## Raycast Direction Types + +```typescript +// 1. Local direction — relative to entity rotation +{ $case: 'localDirection', localDirection: Vector3.Forward() } + +// 2. Global direction — world-space direction, ignores entity rotation +{ $case: 'globalDirection', globalDirection: Vector3.Down() } + +// 3. Global target — aim at a specific world position +{ $case: 'globalTarget', globalTarget: Vector3.create(10, 0, 10) } + +// 4. Target entity — aim at another entity dynamically +{ $case: 'targetEntity', targetEntity: entityId } +``` + +### Raycast Options + +```typescript +{ + direction: Vector3.Forward(), + maxDistance: 16, + queryType: RaycastQueryType.RQT_HIT_FIRST, // or RQT_QUERY_ALL + originOffset: Vector3.create(0, 0.5, 0), // offset from entity origin + collisionMask: ColliderLayer.CL_PHYSICS | ColliderLayer.CL_CUSTOM1, + continuous: false // true = every frame, false = one-shot +} +``` + +### Camera Raycast + +Cast a ray from the camera to detect what the player is looking at: + +```typescript +raycastSystem.registerGlobalDirectionRaycast( + { + entity: engine.CameraEntity, + opts: { + direction: Vector3.rotate( + Vector3.Forward(), + Transform.get(engine.CameraEntity).rotation + ), + maxDistance: 16, + }, + }, + (result) => { + if (result.hits.length > 0) + console.log('Looking at:', result.hits[0].entityId) + } +) +``` + +## Avatar Modifier Areas + +Modify how avatars appear or behave in a region: + +```typescript +import { AvatarModifierArea, AvatarModifierType } from '@dcl/sdk/ecs' + +AvatarModifierArea.create(entity, { + area: { box: Vector3.create(4, 3, 4) }, + modifiers: [AvatarModifierType.AMT_HIDE_AVATARS], + excludeIds: ['0x123...abc'], // Optional +}) + +// Available modifiers: +// AMT_HIDE_AVATARS — Hide all avatars in the area +// AMT_DISABLE_PASSPORTS — Disable clicking on avatars to see profiles +// AMT_DISABLE_JUMPING — Prevent jumping in the area +``` + +## Cursor State + +```typescript +// Check if cursor is locked (pointer lock mode) +const isLocked = PointerLock.get(engine.CameraEntity).isPointerLocked + +// Get cursor position and world ray +const pointerInfo = PrimaryPointerInfo.get(engine.RootEntity) +console.log('Cursor screen position:', pointerInfo.screenCoordinates) +console.log('World ray direction:', pointerInfo.worldRayDirection) +``` + +## Trigger Area Callback Fields + +The trigger area event callback provides: + +- `triggeredEntity` — the entity that activated the area +- `eventType` — ENTER, EXIT, or STAY +- `trigger.entity` — the trigger area entity +- `trigger.layer` — the collider layer +- `trigger.position` — position of the triggered entity +- `trigger.rotation` — rotation of the triggered entity +- `trigger.scale` — scale of the triggered entity diff --git a/ai-sdk-context/advanced-input/SKILL.md b/ai-sdk-context/advanced-input/SKILL.md new file mode 100644 index 0000000..fe4af0e --- /dev/null +++ b/ai-sdk-context/advanced-input/SKILL.md @@ -0,0 +1,272 @@ +--- +name: advanced-input +description: Advanced input handling in Decentraland. PointerLock (cursor capture state), InputModifier (freeze/restrict player movement), PrimaryPointerInfo (cursor position and world ray), WASD keyboard patterns, and action bar slots. Use when the user wants movement restriction, cursor control, FPS controls, input polling, or cutscene freezing. Do NOT use for basic click/hover events on entities (see add-interactivity). +--- + +# Advanced Input Handling in Decentraland + +For basic click/hover events, see the `add-interactivity` skill. This skill covers advanced input patterns. + +## Pointer Lock State + +Detect whether the cursor is captured (first-person mode) or free: + +```typescript +import { engine, PointerLock } from '@dcl/sdk/ecs' + +function checkPointerLock() { + const isLocked = PointerLock.get(engine.CameraEntity).isPointerLocked + + if (isLocked) { + // Cursor is captured — player is in first-person control + } else { + // Cursor is free — player can click UI elements + } +} + +engine.addSystem(checkPointerLock) +``` + +### Pointer Lock Change Detection + +```typescript +PointerLock.onChange(engine.CameraEntity, (pointerLock) => { + if (pointerLock?.isPointerLocked) { + console.log('Cursor locked') + } else { + console.log('Cursor unlocked') + } +}) +``` + +## Cursor Position and World Ray + +Get the cursor's screen position and the ray it casts into the 3D world: + +```typescript +import { engine, PrimaryPointerInfo } from '@dcl/sdk/ecs' + +function readPointer() { + const pointerInfo = PrimaryPointerInfo.get(engine.RootEntity) + console.log('Cursor position:', pointerInfo.screenCoordinates) + console.log('Cursor delta:', pointerInfo.screenDelta) + console.log('World ray direction:', pointerInfo.worldRayDirection) +} + +engine.addSystem(readPointer) +``` + +## Input Polling with inputSystem + +### Per-Entity Input Commands + +Check if a specific input action occurred on a specific entity: + +```typescript +import { engine, inputSystem, InputAction, PointerEventType } from '@dcl/sdk/ecs' + +function myInputSystem() { + // Check for click on a specific entity + const clickData = inputSystem.getInputCommand( + InputAction.IA_POINTER, + PointerEventType.PET_DOWN, + myEntity + ) + + if (clickData) { + console.log('Entity clicked via system:', clickData.hit.entityId) + } +} + +engine.addSystem(myInputSystem) +``` + +Best practice: use the Tag component to mark all entities that share a same interaction, then iterate over them in a system. + +```typescript +import { engine, inputSystem, InputAction, PointerEventType, Tags } from '@dcl/sdk/ecs' + +function myInputSystem() { + + // fetch entities with a particular tag + const taggedEntities = engine.getEntitiesByTag('myTag') + + // iterate over those entities + for (const entity of taggedEntities) { + // Check for click on a specific entity + const clickData = inputSystem.getInputCommand( + InputAction.IA_POINTER, + PointerEventType.PET_DOWN, + entity + ) + + if (clickData) { + console.log('Entity clicked via system:', clickData.hit.entityId) + } + } + +} + +engine.addSystem(myInputSystem) +``` + + + +### Global Input Checks + +Check if a specific key was pressed, regardless of if the player's cursor was pointing at an entity or not. + +```typescript +function globalInputSystem() { + // Was the key just pressed this frame? + if (inputSystem.isTriggered(InputAction.IA_PRIMARY, PointerEventType.PET_DOWN)) { + console.log('E key pressed!') + } + + // Is the key currently held down? + if (inputSystem.isPressed(InputAction.IA_SECONDARY)) { + console.log('F key is held!') + } +} + +engine.addSystem(globalInputSystem) +``` + +## All InputAction Values + +| InputAction | Key/Button | +|-------------|-----------| +| `IA_POINTER` | Left mouse button | +| `IA_PRIMARY` | E key | +| `IA_SECONDARY` | F key | +| `IA_ACTION_3` | 1 key | +| `IA_ACTION_4` | 2 key | +| `IA_ACTION_5` | 3 key | +| `IA_ACTION_6` | 4 key | +| `IA_JUMP` | Space key | +| `IA_FORWARD` | W key | +| `IA_BACKWARD` | S key | +| `IA_LEFT` | A key | +| `IA_RIGHT` | D key | +| `IA_WALK` | Shift key | + +## Event Types + +```typescript +PointerEventType.PET_DOWN // Button/key pressed +PointerEventType.PET_UP // Button/key released +PointerEventType.PET_HOVER_ENTER // Cursor enters entity +PointerEventType.PET_HOVER_LEAVE // Cursor leaves entity +``` + +## InputModifier (Movement Restriction) + +Restrict or freeze the player's movement: + +```typescript +import { engine, InputModifier } from '@dcl/sdk/ecs' + +// Freeze player completely +InputModifier.create(engine.PlayerEntity, { + mode: InputModifier.Mode.Standard({ disableAll: true }) +}) + +// Restrict specific movement +InputModifier.createOrReplace(engine.PlayerEntity, { + mode: InputModifier.Mode.Standard({ + disableRun: true, + disableJump: true, + disableEmote: true, + disableDoubleJump: true, + disableGliding: true + }) +}) + +// Restore normal movement +InputModifier.deleteFrom(engine.PlayerEntity) +``` + +**Important:** InputModifier only works in the DCL 2.0 desktop client. It has no effect in the web browser explorer. + +### Cutscene Pattern + +Freeze the player during a cinematic sequence: + +```typescript +function startCutscene() { + // Freeze player + InputModifier.create(engine.PlayerEntity, { + mode: InputModifier.Mode.Standard({ disableAll: true }) + }) + + // ... play cinematic with VirtualCamera ... + + // After cutscene ends, restore movement + // InputModifier.deleteFrom(engine.PlayerEntity) +} +``` + +## WASD Movement Pattern + +Poll movement keys to control custom entities: + +```typescript +import { engine, inputSystem, InputAction, PointerEventType, Transform } from '@dcl/sdk/ecs' +import { Vector3 } from '@dcl/sdk/math' + +const MOVE_SPEED = 5 + +function customMovementSystem(dt: number) { + const transform = Transform.getMutable(controllableEntity) + let moveX = 0 + let moveZ = 0 + + if (inputSystem.isPressed(InputAction.IA_FORWARD)) moveZ += 1 + if (inputSystem.isPressed(InputAction.IA_BACKWARD)) moveZ -= 1 + if (inputSystem.isPressed(InputAction.IA_LEFT)) moveX -= 1 + if (inputSystem.isPressed(InputAction.IA_RIGHT)) moveX += 1 + + transform.position.x += moveX * MOVE_SPEED * dt + transform.position.z += moveZ * MOVE_SPEED * dt +} + +engine.addSystem(customMovementSystem) +``` + +## Combining Input Patterns + +### Action Bar with Number Keys + +```typescript +function actionBarSystem() { + if (inputSystem.isTriggered(InputAction.IA_ACTION_3, PointerEventType.PET_DOWN)) { + console.log('Slot 1 activated') + useAbility(1) + } + if (inputSystem.isTriggered(InputAction.IA_ACTION_4, PointerEventType.PET_DOWN)) { + console.log('Slot 2 activated') + useAbility(2) + } + if (inputSystem.isTriggered(InputAction.IA_ACTION_5, PointerEventType.PET_DOWN)) { + console.log('Slot 3 activated') + useAbility(3) + } + if (inputSystem.isTriggered(InputAction.IA_ACTION_6, PointerEventType.PET_DOWN)) { + console.log('Slot 4 activated') + useAbility(4) + } +} + +engine.addSystem(actionBarSystem) +``` + +## Best Practices + +- Use `isTriggered()` for one-shot actions (fire weapon, open door) — it returns true only on the frame the key is first pressed +- Use `isPressed()` for continuous actions (movement, holding a shield) — it returns true every frame while held +- `getInputCommand()` gives hit data (position, entity) — use it when you need to know what was clicked +- Prefer `pointerEventsSystem.onPointerDown()` for simple entity clicks — use `inputSystem` for complex multi-key or polling patterns +- InputModifier only works in the DCL 2.0 desktop client — test with the desktop client if your scene relies on it +- WASD keys (`IA_FORWARD`, etc.) also control player movement — polling them reads the movement state but doesn't override it + +For basic pointer events and click handlers, see the `add-interactivity` skill. diff --git a/ai-sdk-context/advanced-rendering/SKILL.md b/ai-sdk-context/advanced-rendering/SKILL.md new file mode 100644 index 0000000..a3dffc8 --- /dev/null +++ b/ai-sdk-context/advanced-rendering/SKILL.md @@ -0,0 +1,366 @@ +--- +name: advanced-rendering +description: Advanced rendering in Decentraland scenes. Billboard (face camera), TextShape (3D world text), PBR materials (metallic, roughness, transparency, emissive glow), GltfNodeModifiers (per-node shadow/material overrides), VisibilityComponent (show/hide entities), and texture modes. Use when the user wants billboards, floating labels, 3D text, material effects, glow, transparency, or model node control. Do NOT use for screen-space UI (see build-ui) or loading 3D models (see add-3d-models). +--- + +# Advanced Rendering in Decentraland + +## When to Use Which Rendering Feature + +| Need | Component | When | +|------|-----------|------| +| Entity faces the camera | `Billboard` | Name tags, signs, sprite-like objects | +| Text in the 3D world | `TextShape` | Labels, signs, floating text above entities | +| Custom material appearance | `Material.setPbrMaterial` | Metallic, rough, transparent, emissive surfaces | +| Show/hide without removing | `VisibilityComponent` | LOD systems, toggling objects, conditional display | +| Modify GLTF model nodes | `GltfNodeModifiers` | Override materials or shadow casting on specific mesh nodes | + +**Decision flow:** +1. Need text on screen? → Use **build-ui** (React-ECS Label) instead +2. Need text in 3D space? → `TextShape` (+ `Billboard` to face camera) +3. Need glowing/transparent materials? → `Material.setPbrMaterial` with emissive/transparency +4. Need to override material on a model node? → `GltfNodeModifiers` with `modifiers` array + +## Billboard (Face the Camera) + +Make entities always rotate to face the player's camera: + +```typescript +import { engine, Transform, Billboard, BillboardMode, MeshRenderer } from '@dcl/sdk/ecs' +import { Vector3 } from '@dcl/sdk/math' + +const sign = engine.addEntity() +Transform.create(sign, { position: Vector3.create(8, 2, 8) }) +MeshRenderer.setPlane(sign) + +// Rotate only on Y axis (most common — stays upright) +Billboard.create(sign, { + billboardMode: BillboardMode.BM_Y +}) +``` + +### Billboard Modes + +```typescript +BillboardMode.BM_Y // Rotate on Y axis only (stays upright) — most common +BillboardMode.BM_ALL // Rotate on all axes (fully faces camera) +BillboardMode.BM_X // Rotate on X axis only +BillboardMode.BM_Z // Rotate on Z axis only +BillboardMode.BM_NONE // No billboard rotation +``` + +- Prefer `BM_Y` over `BM_ALL` for most use cases — it looks more natural and is cheaper to render. +- `BM_ALL` is useful for particles or effects that should always directly face the camera. + +## TextShape (3D Text) + +Render text directly in 3D space: + +```typescript +import { engine, Transform, TextShape, TextAlignMode } from '@dcl/sdk/ecs' +import { Vector3, Color4 } from '@dcl/sdk/math' + +const label = engine.addEntity() +Transform.create(label, { position: Vector3.create(8, 3, 8) }) + +TextShape.create(label, { + text: 'Hello World!', + fontSize: 24, + textColor: Color4.White(), + outlineColor: Color4.Black(), + outlineWidth: 0.1, + textAlign: TextAlignMode.TAM_MIDDLE_CENTER +}) +``` + +### Text Alignment Options + +```typescript +TextAlignMode.TAM_TOP_LEFT +TextAlignMode.TAM_TOP_CENTER +TextAlignMode.TAM_TOP_RIGHT +TextAlignMode.TAM_MIDDLE_LEFT +TextAlignMode.TAM_MIDDLE_CENTER +TextAlignMode.TAM_MIDDLE_RIGHT +TextAlignMode.TAM_BOTTOM_LEFT +TextAlignMode.TAM_BOTTOM_CENTER +TextAlignMode.TAM_BOTTOM_RIGHT +``` + +### Floating Label (Billboard + TextShape) + +Combine Billboard and TextShape to create labels that always face the player: + +```typescript +const floatingLabel = engine.addEntity() +Transform.create(floatingLabel, { position: Vector3.create(8, 4, 8) }) + +TextShape.create(floatingLabel, { + text: 'NPC Name', + fontSize: 16, + textColor: Color4.White(), + outlineColor: Color4.Black(), + outlineWidth: 0.08, + textAlign: TextAlignMode.TAM_BOTTOM_CENTER +}) + +Billboard.create(floatingLabel, { + billboardMode: BillboardMode.BM_Y +}) +``` + +## Advanced PBR Materials + +### Metallic and Roughness + +```typescript +import { engine, Transform, MeshRenderer, Material, MaterialTransparencyMode } from '@dcl/sdk/ecs' +import { Color4, Color3 } from '@dcl/sdk/math' + +// Shiny metal +Material.setPbrMaterial(entity, { + albedoColor: Color4.create(0.8, 0.8, 0.9, 1), + metallic: 1.0, + roughness: 0.1 +}) + +// Rough stone +Material.setPbrMaterial(entity, { + albedoColor: Color4.create(0.5, 0.5, 0.5, 1), + metallic: 0.0, + roughness: 0.9 +}) +``` + +### Transparency + +```typescript +// Alpha blend — smooth transparency +Material.setPbrMaterial(entity, { + albedoColor: Color4.create(1, 0, 0, 0.5), // 50% transparent red + transparencyMode: MaterialTransparencyMode.MTM_ALPHA_BLEND +}) + +// Alpha test — cutout (binary visible/invisible based on threshold) +Material.setPbrMaterial(entity, { + texture: Material.Texture.Common({ src: 'assets/scene/Images/cutout.png' }), + transparencyMode: MaterialTransparencyMode.MTM_ALPHA_TEST, + alphaTest: 0.5 +}) +``` + +### Emissive (Glow Effects) + +```typescript +// Glowing material (emissiveColor uses Color3, not Color4) +Material.setPbrMaterial(entity, { + albedoColor: Color4.create(0, 0, 0, 1), + emissiveColor: Color3.create(0, 1, 0), // Green glow + emissiveIntensity: 2.0 +}) + +// Emissive with texture +Material.setPbrMaterial(entity, { + texture: Material.Texture.Common({ src: 'assets/scene/Images/diffuse.png' }), + emissiveTexture: Material.Texture.Common({ src: 'assets/scene/Images/emissive.png' }), + emissiveIntensity: 1.0, + emissiveColor: Color3.White() +}) +``` + +### Texture Maps + +```typescript +Material.setPbrMaterial(entity, { + texture: Material.Texture.Common({ src: 'assets/scene/Images/diffuse.png' }), + bumpTexture: Material.Texture.Common({ src: 'assets/scene/Images/normal.png' }), + emissiveTexture: Material.Texture.Common({ src: 'assets/scene/Images/emissive.png' }) +}) +``` + +## GltfContainer Collision Masks + +Use collision masks to control which collision layers respond to the different mesh layers in a GLTF model. GLTF models have two mesh layers: visible meshes (what players see rendered), and invisible layers (collider meshes, named internally with _collider): + +```typescript +import { engine, Transform, GltfContainer, ColliderLayer } from '@dcl/sdk/ecs' +import { Vector3 } from '@dcl/sdk/math' + +const model = engine.addEntity() +Transform.create(model, { position: Vector3.create(4, 0, 4) }) + +GltfContainer.create(model, { + src: 'models/myModel.glb', + visibleMeshesCollisionMask: ColliderLayer.CL_PHYSICS | ColliderLayer.CL_POINTER, + invisibleMeshesCollisionMask: ColliderLayer.CL_PHYSICS +}) +``` + +## VisibilityComponent + +Show or hide entities without removing them: + +```typescript +import { engine, VisibilityComponent } from '@dcl/sdk/ecs' + +// Hide an entity +VisibilityComponent.create(entity, { visible: false }) + +// Toggle visibility +const visibility = VisibilityComponent.getMutable(entity) +visibility.visible = !visibility.visible + +// Useful for LOD (Level of Detail) +function lodSystem() { + const playerPos = Transform.get(engine.PlayerEntity).position + + for (const [entity, transform] of engine.getEntitiesWith(Transform, MeshRenderer)) { + const distance = Vector3.distance(playerPos, transform.position) + + if (distance > 30) { + VisibilityComponent.createOrReplace(entity, { visible: false }) + } else { + VisibilityComponent.createOrReplace(entity, { visible: true }) + } + } +} + +engine.addSystem(lodSystem) +``` + +### propagateToChildren + +Set `propagateToChildren: true` on a `VisibilityComponent` to apply visibility to all children in the hierarchy at once. This avoids having to mark every child entity individually: + +```typescript +VisibilityComponent.create(parentEntity, { visible: false, propagateToChildren: true }) +``` + +Rules: +- If a child has its **own** `VisibilityComponent`, that overrides what the parent propagates. +- If a child has **no** `VisibilityComponent`, it inherits from the nearest ancestor with `propagateToChildren: true`. + +### Per-Node Modifiers (GltfNodeModifiers) + +Override material or shadow casting on specific nodes within a GLTF model: + +```typescript +import { GltfNodeModifiers } from '@dcl/sdk/ecs' + +GltfNodeModifiers.create(entity, { + modifiers: [ + { + path: 'RootNode/Armor', // GLTF hierarchy path + castShadows: false // Disable shadow casting for this node + } + ] +}) +``` + +To override the materials or shadow casting of the entire model, set the path to ''. + +```typescript +import { GltfNodeModifiers } from '@dcl/sdk/ecs' + +GltfNodeModifiers.create(entity, { + modifiers: [ + { + path: '', + material: { + material: { + $case: 'pbr', + pbr: { + albedoColor: Color4.Red(), + }, + }, + }, + } + ] +}) +``` + + +### Avatar Texture + +Generate a texture from a player's avatar portrait: + +```typescript +Material.setPbrMaterial(portraitFrame, { + texture: Material.Texture.Avatar({ userId: '0x...' }) +}) +``` + +This will fetch a thumbnail image with a closeup of the player's face, wearing the wearables that this player currently has on. + + +### Texture Modes + +Control how textures are filtered and wrapped: + +```typescript +import { TextureFilterMode, TextureWrapMode } from '@dcl/sdk/ecs' + +Material.setPbrMaterial(entity, { + texture: Material.Texture.Common({ + src: 'assets/scene/Images/pixel-art.png', + filterMode: TextureFilterMode.TFM_POINT, // crisp pixels (no smoothing) + wrapMode: TextureWrapMode.TWM_REPEAT // tile the texture + }) +}) +``` + +Filter modes: `TFM_POINT` (pixelated), `TFM_BILINEAR` (smooth), `TFM_TRILINEAR` (smoothest). +Wrap modes: `TWM_REPEAT` (tile), `TWM_CLAMP` (stretch edges), `TWM_MIRROR` (mirror tile). + +## Texture tweens + +You can use tweens to make a texture slide sideways or shrink or zoom in, this can be used to achieve very cool effects. + +```typescript +Material.setPbrMaterial(myEntity, { + texture: Material.Texture.Common({ + src: 'materials/water.png', + wrapMode: TextureWrapMode.TWM_REPEAT, + }), +}) + +// move continuously +Tween.setTextureMoveContinuous(myEntity, Vector2.create(0, 1), 1) +``` + +You can also make a texture move once, lasting a specific duration + +```typescript +// slide once, for 1 second +Tween.setTextureMove(myEntity, Vector2.create(0, 0), Vector2.create(0, 1), 1000) +``` + + +## FlatMaterial Accessors + +The `Material` component provides shortcut methods that skip the nested union structure, making material access more ergonomic: + +| Method | Returns | Throws if no material? | +|---|---|---| +| `Material.getFlat(entity)` | Read-only `FlatMaterial` | Yes | +| `Material.getFlatOrNull(entity)` | Read-only `FlatMaterial \| null` | No | +| `Material.getFlatMutable(entity)` | Read/write `FlatMaterial` | Yes | +| `Material.getFlatMutableOrNull(entity)` | Read/write `FlatMaterial \| null` | No | + +```typescript +// Read a property safely +const src = Material.getFlatOrNull(entity)?.texture?.src + +// Mutate a texture in-place without knowing PBR vs Basic +Material.getFlatMutableOrNull(entity)!.texture = Material.Texture.Common({ src: 'assets/scene/Images/new.png' }) +``` + +## Best Practices + +- Use `BillboardMode.BM_Y` instead of `BM_ALL` — looks more natural and renders faster +- Keep `fontSize` readable (16-32 for in-world text) +- Add `outlineColor` and `outlineWidth` to TextShape for legibility against any background +- Use `emissiveColor` with a dark `albedoColor` for maximum glow visibility +- `MTM_ALPHA_TEST` is cheaper than `MTM_ALPHA_BLEND` — use cutout when smooth transparency isn't needed +- Combine Billboard + TextShape for floating name labels above NPCs or objects +- Use VisibilityComponent for LOD systems instead of removing/re-adding entities diff --git a/ai-sdk-context/animations-tweens/SKILL.md b/ai-sdk-context/animations-tweens/SKILL.md new file mode 100644 index 0000000..2f05eb3 --- /dev/null +++ b/ai-sdk-context/animations-tweens/SKILL.md @@ -0,0 +1,293 @@ +--- +name: animations-tweens +description: Animate objects in Decentraland scenes. Play GLTF model animations with Animator, create procedural motion with Tween (move/rotate/scale), and chain sequences with TweenSequence. Use when the user wants to animate, move, rotate, spin, slide, bob, or create motion effects. Do NOT use for audio/video playback (see audio-video). +--- + +# Animations and Tweens in Decentraland + +## When to Use Which Animation Approach + +| Need | Approach | When | +|------|----------|------| +| Play animation baked into a .glb model | `Animator` | Character walks, door opens, flag waves — any animation created in Blender/Maya | +| Move/rotate/scale an entity smoothly | `Tween` | Sliding doors, floating platforms, growing objects — procedural A-to-B motion | +| Chain multiple animations in sequence | `TweenSequence` | Patrol paths, multi-step doors, complex choreography | +| Continuous per-frame control | `engine.addSystem()` | Physics-like motion, following a target, custom easing | + +**Decision flow:** +1. Does the .glb model already have the animation? → `Animator` +2. Is it a simple move/rotate/scale between two values? → `Tween` +3. Do you need frame-by-frame control or custom math? → System with `dt` + +## GLTF Animations (Animator) + +Play animations embedded in .glb models: + +```typescript +import { engine, Transform, GltfContainer, Animator } from '@dcl/sdk/ecs' +import { Vector3 } from '@dcl/sdk/math' + +const character = engine.addEntity() +Transform.create(character, { position: Vector3.create(8, 0, 8) }) +GltfContainer.create(character, { src: 'models/character.glb' }) + +// Set up animation states +Animator.create(character, { + states: [ + { clip: 'idle', playing: true, loop: true, speed: 1 }, + { clip: 'walk', playing: false, loop: true, speed: 1 }, + { clip: 'attack', playing: false, loop: false, speed: 1.5 } + ] +}) + +// Play a specific animation +Animator.playSingleAnimation(character, 'walk') + +// Stop all animations +Animator.stopAllAnimations(character) +``` + +### Switching Animations +```typescript +function playAnimation(entity: Entity, clipName: string) { + const animator = Animator.getMutable(entity) + // Stop all + for (const state of animator.states) { + state.playing = false + } + // Play the desired one + const state = animator.states.find(s => s.clip === clipName) + if (state) { + state.playing = true + } +} +``` + +## Tweens (Code-Based Animation) + +Animate entity properties smoothly over time: + +### Move +```typescript +import { engine, Transform, Tween, EasingFunction } from '@dcl/sdk/ecs' +import { Vector3 } from '@dcl/sdk/math' + +const box = engine.addEntity() +Transform.create(box, { position: Vector3.create(2, 1, 8) }) + +Tween.create(box, { + mode: Tween.Mode.Move({ + start: Vector3.create(2, 1, 8), + end: Vector3.create(14, 1, 8) + }), + duration: 2000, // milliseconds + easingFunction: EasingFunction.EF_EASESINE +}) +``` + +### Rotate +```typescript +Tween.create(box, { + mode: Tween.Mode.Rotate({ + start: Quaternion.fromEulerDegrees(0, 0, 0), + end: Quaternion.fromEulerDegrees(0, 360, 0) + }), + duration: 3000, + easingFunction: EasingFunction.EF_LINEAR +}) +``` + +You can also use tweens for a continuous rotation: + +```typescript +Tween.setRotateContinuous(myEntity, + Quaternion.fromEulerDegrees(0, -1, 0), + 700 +) +``` + + +### Scale +```typescript +Tween.create(box, { + mode: Tween.Mode.Scale({ + start: Vector3.create(1, 1, 1), + end: Vector3.create(2, 2, 2) + }), + duration: 1000, + easingFunction: EasingFunction.EF_EASEOUTBOUNCE +}) +``` + +### Multiple transformations + +If an entity needs to tween in any combination of position, scale, or rotation, you can achieve multiple simultaneous changes using `Tween.setMoveRotateScale`. +An entity can only have one Tween compoent at a time. + +```typescript +Tween.setMoveRotateScale(mrsEntity, { + position: { start: Vector3.create(14, 1, 2), end: Vector3.create(14, 3, 2) }, + rotation: { start: Quaternion.fromEulerDegrees(0, 0, 0), end: Quaternion.fromEulerDegrees(0, 180, 90) }, + scale: { start: Vector3.One(), end: Vector3.create(2, 0.5, 2) }, + duration: 2000 +}) +``` + + +## Tween Sequences (Chained Animations) + +Chain multiple tweens to play one after another: + +```typescript +import { TweenSequence, TweenLoop } from '@dcl/sdk/ecs' + +// First tween +Tween.create(box, { + mode: Tween.Mode.Move({ + start: Vector3.create(2, 1, 8), + end: Vector3.create(14, 1, 8) + }), + duration: 2000, + easingFunction: EasingFunction.EF_EASESINE +}) + +// Chain sequence +TweenSequence.create(box, { + sequence: [ + // Second: move back + { + mode: Tween.Mode.Move({ + start: Vector3.create(14, 1, 8), + end: Vector3.create(2, 1, 8) + }), + duration: 2000, + easingFunction: EasingFunction.EF_EASESINE + } + ], + loop: TweenLoop.TL_RESTART // Loop the entire sequence +}) +``` + +## Easing Functions + +Available easing functions from `EasingFunction`: +- `EF_LINEAR` — Constant speed +- `EF_EASEINQUAD` / `EF_EASEOUTQUAD` / `EF_EASEQUAD` — Quadratic +- `EF_EASEINSINE` / `EF_EASEOUTSINE` / `EF_EASESINE` — Sinusoidal (smooth) +- `EF_EASEINEXPO` / `EF_EASEOUTEXPO` / `EF_EASEEXPO` — Exponential +- `EF_EASEINELASTIC` / `EF_EASEOUTELASTIC` / `EF_EASEELASTIC` — Elastic bounce +- `EF_EASEOUTBOUNCE` / `EF_EASEINBOUNCE` / `EF_EASEBOUNCE` — Bounce effect +- `EF_EASEINBACK` / `EF_EASEOUTBACK` / `EF_EASEBACK` — Overshoot +- `EF_EASEINCUBIC` / `EF_EASEOUTCUBIC` / `EF_EASECUBIC` — Cubic +- `EF_EASEINQUART` / `EF_EASEOUTQUART` / `EF_EASEQUART` — Quartic +- `EF_EASEINQUINT` / `EF_EASEOUTQUINT` / `EF_EASEQUINT` — Quintic +- `EF_EASEINCIRC` / `EF_EASEOUTCIRC` / `EF_EASECIRC` — Circular + +## Custom Animation Systems + +For complex animations, create a system: + +```typescript +// Continuous rotation system +function spinSystem(dt: number) { + for (const [entity] of engine.getEntitiesWith(Transform, Spinner)) { + const transform = Transform.getMutable(entity) + const spinner = Spinner.get(entity) + // Rotate around Y axis + const currentRotation = Quaternion.toEulerAngles(transform.rotation) + transform.rotation = Quaternion.fromEulerDegrees( + currentRotation.x, + currentRotation.y + spinner.speed * dt, + currentRotation.z + ) + } +} + +engine.addSystem(spinSystem) +``` + +### Tween Helper Methods + +Use shorthand helpers that create or replace the Tween component directly on the entity: + +```typescript +import { Tween, EasingFunction } from '@dcl/sdk/ecs' + +// Move — signature: Tween.setMove(entity, start, end, duration, easingFunction?) +Tween.setMove(entity, + Vector3.create(0, 1, 0), Vector3.create(0, 3, 0), + 1500, EasingFunction.EF_EASEINBOUNCE +) + +// Rotate — signature: Tween.setRotate(entity, start, end, duration, easingFunction?) +Tween.setRotate(entity, + Quaternion.fromEulerDegrees(0, 0, 0), Quaternion.fromEulerDegrees(0, 180, 0), + 2000, EasingFunction.EF_EASEOUTQUAD +) + +// Scale — signature: Tween.setScale(entity, start, end, duration, easingFunction?) +Tween.setScale(entity, + Vector3.One(), Vector3.create(2, 2, 2), + 1000, EasingFunction.EF_LINEAR +) +``` + +### Yoyo Loop Mode + +`TL_YOYO` reverses the tween sequence at each end instead of restarting: + +```typescript +TweenSequence.create(entity, { + sequence: [{ duration: 1000, ... }], + loop: TweenLoop.TL_YOYO +}) +``` + +### Detecting Tween Completion + +Use `tweenSystem.tweenCompleted()` to check if a tween finished this frame: + +```typescript +engine.addSystem(() => { + if (tweenSystem.tweenCompleted(entity)) { + console.log('Tween finished on', entity) + } +}) +``` + +### Animator Extras + +Additional `Animator` features: + +```typescript +// Get a specific clip to modify +const clip = Animator.getClip(entity, 'Walk') + +// shouldReset: restart animation from beginning when re-triggered +Animator.playSingleAnimation(entity, 'Attack', true) // resets to start + +// weight: blend between animations (0.0 to 1.0) +const anim = Animator.getMutable(entity) +anim.states[0].weight = 0.5 // blend walk at 50% +anim.states[1].weight = 0.5 // blend idle at 50% +``` + +## Troubleshooting + +| Problem | Cause | Solution | +|---------|-------|----------| +| GLTF animation not playing | Wrong clip name in `Animator.states` | Open the .glb in a viewer (e.g., Blender) to find exact clip names — they are case-sensitive | +| Animator component has no effect | Entity missing `GltfContainer` | `Animator` only works on entities that have a loaded GLTF model | +| Tween doesn't move | Start and end positions are the same | Verify `start` and `end` values differ in `Tween.Mode.Move()` | +| Tween plays once then stops | No `TweenSequence` with loop | Add `TweenSequence.create(entity, { sequence: [], loop: TweenLoop.TL_YOYO })` for back-and-forth | +| Animation jitters or stutters | Creating new Tween every frame | Only create Tween once, not inside a system — use `tweenSystem.tweenCompleted()` to chain | + +> **Need 3D models to animate?** See the **add-3d-models** skill for loading GLTF models that contain animation clips. + +## Best Practices + +- Use Tweens for simple A-to-B animations (doors, platforms, UI elements) +- Use Animator for character/model animations baked into GLTF files +- Use Systems for continuous user control or physics-based animations +- Tween durations are in **milliseconds** (1000 = 1 second) +- For looping: use `TweenSequence` with `loop: TweenLoop.TL_RESTART` diff --git a/ai-sdk-context/audio-video/SKILL.md b/ai-sdk-context/audio-video/SKILL.md new file mode 100644 index 0000000..7e4580c --- /dev/null +++ b/ai-sdk-context/audio-video/SKILL.md @@ -0,0 +1,408 @@ +--- +name: audio-video +description: Add sound effects, music, audio streaming, and video players to Decentraland scenes. Covers AudioSource (local files), AudioStream (streaming URLs), VideoPlayer (video surfaces), video events, and media permissions. Use when the user wants sound, music, audio, video screens, radio, or media playback. +--- + +# Audio and Video in Decentraland + +## When to Use Which Media Component + +| Need | Component | Key Difference | +| ----------------------------------------------------- | ---------------------------------------- | ---------------------------------------- | +| Sound effect from a file (click, explosion, footstep) | `AudioSource` | Local file, spatial, one-shot or looping | +| Background music or radio stream | `AudioStream` | External URL, non-spatial, continuous | +| Video on a surface (screen, billboard) | `VideoPlayer` + `Material.Texture.Video` | Requires a mesh to display on | + +**Decision flow:** + +1. Is it a local audio file? → `AudioSource` +2. Is it a streaming URL (radio, live audio)? → `AudioStream` +3. Is it video content? → `VideoPlayer` on a plane/mesh + +## Audio Source (Sound Effects & Music) + +Play audio clips from files: + +```typescript +import { engine, Transform, AudioSource } from '@dcl/sdk/ecs' +import { Vector3 } from '@dcl/sdk/math' + +const speaker = engine.addEntity() +Transform.create(speaker, { position: Vector3.create(8, 1, 8) }) + +AudioSource.create(speaker, { + audioClipUrl: 'assets/scene/Audio/music.mp3', + playing: true, + loop: true, + volume: 0.5, // 0 to 1 + pitch: 1.0, // Playback speed (0.5 = half speed, 2.0 = double) +}) +``` + +### Supported Formats + +- `.mp3` (recommended) +- `.ogg` +- `.wav` + +### File Organization + +``` +project/ +├── assets/ +│ └── scene/ +│ └── Audio/ +│ ├── click.mp3 +│ ├── background-music.mp3 +│ └── explosion.ogg +├── src/ +│ └── index.ts +└── scene.json +``` + +### Play/Stop/Toggle + +```typescript +// Play +AudioSource.getMutable(speaker).playing = true + +// Stop +AudioSource.getMutable(speaker).playing = false + +// Toggle +const audio = AudioSource.getMutable(speaker) +audio.playing = !audio.playing +``` + +### Play on Click + +```typescript +import { pointerEventsSystem, InputAction } from '@dcl/sdk/ecs' + +const button = engine.addEntity() +// ... set up transform and mesh ... + +const audioEntity = engine.addEntity() +Transform.create(audioEntity, { position: Vector3.create(8, 1, 8) }) +AudioSource.create(audioEntity, { + audioClipUrl: 'assets/scene/Audio/click.mp3', + playing: false, + loop: false, + volume: 0.8, +}) + +pointerEventsSystem.onPointerDown( + { + entity: button, + opts: { button: InputAction.IA_POINTER, hoverText: 'Play sound' }, + }, + () => { + // Reset and play + const audio = AudioSource.getMutable(audioEntity) + audio.playing = false + audio.playing = true + } +) +``` + +## Audio Streaming + +Stream audio from a URL (radio, live streams): + +```typescript +import { engine, Transform, AudioStream } from '@dcl/sdk/ecs' +import { Vector3 } from '@dcl/sdk/math' + +const radio = engine.addEntity() +Transform.create(radio, { position: Vector3.create(8, 1, 8) }) + +AudioStream.create(radio, { + url: 'https://example.com/stream.mp3', + playing: true, + volume: 0.3, +}) +``` + +## Video Player + +Play video on a surface: + +```typescript +import { + engine, + Transform, + VideoPlayer, + Material, + MeshRenderer, +} from '@dcl/sdk/ecs' +import { Vector3 } from '@dcl/sdk/math' + +// Create a screen +const screen = engine.addEntity() +Transform.create(screen, { + position: Vector3.create(8, 3, 15.9), + scale: Vector3.create(8, 4.5, 1), // 16:9 ratio +}) +MeshRenderer.setPlane(screen) + +// Add video player +VideoPlayer.create(screen, { + src: 'https://example.com/video.mp4', + playing: true, + loop: true, + volume: 0.5, + playbackRate: 1.0, + position: 0, // Start time in seconds +}) + +// Create video texture +const videoTexture = Material.Texture.Video({ videoPlayerEntity: screen }) + +// Basic material (recommended — better performance) +Material.setBasicMaterial(screen, { + texture: videoTexture, +}) +``` + +### Video Controls + +```typescript +// Play +VideoPlayer.getMutable(screen).playing = true + +// Pause +VideoPlayer.getMutable(screen).playing = false + +// Change volume +VideoPlayer.getMutable(screen).volume = 0.8 + +// Change source +VideoPlayer.getMutable(screen).src = 'https://example.com/other.mp4' +``` + +### Enhanced Video Material (PBR) + +Ideally video screens should use basic (unlit) materials, that way they're always bright and crisp. + +```typescript +const videoTexture = Material.Texture.Video({ videoPlayerEntity: screen }) + +Material.setBasicMaterial(screen, { + texture: videoTexture, +}) +``` + +For a brighter, emissive video screen: + +```typescript +import { Color3 } from '@dcl/sdk/math' + +const videoTexture = Material.Texture.Video({ videoPlayerEntity: screen }) +Material.setPbrMaterial(screen, { + texture: videoTexture, + roughness: 1.0, + specularIntensity: 0, + metallic: 0, + emissiveTexture: videoTexture, + emissiveIntensity: 0.6, + emissiveColor: Color3.White(), +}) +``` + +### Video Events + +Monitor video playback state: + +```typescript +import { videoEventsSystem, VideoState } from '@dcl/sdk/ecs' + +videoEventsSystem.registerVideoEventsEntity(screen, (videoEvent) => { + switch (videoEvent.state) { + case VideoState.VS_PLAYING: + console.log('Video started playing') + break + case VideoState.VS_PAUSED: + console.log('Video paused') + break + case VideoState.VS_READY: + console.log('Video ready to play') + break + case VideoState.VS_ERROR: + console.log('Video error occurred') + break + } +}) +``` + +## Spatial Audio + +Audio from the `AudioSource` component in Decentraland is **spatial by default** — it gets louder as the player approaches the audio source entity and quieter as they move away. The position is determined by the entity's `Transform`. You can change this by setting the `global` property to true. + +```typescript +AudioSource.create(sourceEntity, { + audioClipUrl: 'assets/scene/Audio/music.mp3', + playing: true, + global: true, +}) +``` + +Audio from the `VideoPlayer` component and the `AudioStream` component is global by default. Set it to spatial by setting the `spatial` to true. You can also change these properties: + +- spatialMinDistance: The minimum distance at which audio becomes spatial. If the player is closer, the audio will be heard at full volume. 0 by default. + +- spatialMaxDistance: The maximum distance at which the audio is heard. If the player is further away, the audio will be heard at 0 volume. 60 by default + +```typescript +VideoPlayer.create(videoPlayerEntity, { + src: 'https://player.vimeo.com/progressive_redirect/playback/1145666916/rendition/540p/file.mp4%20%28540p%29.mp4?loc=external&signature=db1cd6946851313cb8f7be60d1f6c30af0902bcc46fdae0ba2a06e5fdf44c329', + playing: true, + spatial: true, + spatialMinDistance: 5, + spatialMaxDistance: 10, +}) + +AudioStream.create(audioStreamEntity, { + url: 'https://radioislanegra.org/listen/up/stream', + playing: true, + volume: 1.0, + spatial: true, + spatialMinDistance: 5, + spatialMaxDistance: 10, +}) +``` + +## Free Audio Files + +Always check the audio catalog before creating placeholder sound file references. It contains 50 free sounds from the Creator Hub asset packs. + +Read `{baseDir}/../context/audio-catalog.md` for music tracks (ambient, dance, medieval, sci-fi, etc.), ambient sounds (birds, city, factory, etc.), interaction sounds (buttons, doors, levers, chests), sound effects (explosions, sirens, bells), and game mechanic sounds (win/lose, heal, respawn, damage). + +To use a catalog sound: + +```bash +# Download from catalog +mkdir -p assets/scene/Audio +curl -o assets/scene/Audio/ambient_1.mp3 "https://builder-items.decentraland.org/contents/bafybeic4faewxkdqx67dloyw57ikgaeibc2e2dbx34hwjubl3gfvs2r4su" +``` + +```typescript +// Reference in code — must be a local file path +AudioSource.create(entity, { + audioClipUrl: 'assets/scene/Audio/ambient_1.mp3', + playing: true, + loop: true, +}) +``` + +### How to suggest audio + +1. Read the audio catalog file +2. Search for sounds matching the user's description/theme +3. Suggest specific sounds with download commands +4. Download selected sounds into the scene's `assets/scene/Audio/` directory +5. Reference them in code with local paths + +> **Important**: `AudioSource` only works with **local files**. Never use external URLs for the `audioClipUrl` field. Always download audio into `assets/scene/Audio/` first. + +### Video State Polling + +Check video playback state programmatically: + +```typescript +import { videoEventsSystem, VideoState } from '@dcl/sdk/ecs' + +engine.addSystem(() => { + const state = videoEventsSystem.getVideoState(videoEntity) + if (state) { + console.log('Video state:', state.state) // VideoState.VS_PLAYING, VS_PAUSED, etc. + console.log('Current time:', state.currentOffset) + } +}) +``` + +### Audio Playback Events + +Use the `AudioEvent` component to detect audio state changes: + +```typescript +import { AudioEvent } from '@dcl/sdk/ecs' + +engine.addSystem(() => { + const event = AudioEvent.getOrNull(audioEntity) + if (event) { + console.log('Audio state:', event.state) // playing, paused, finished + } +}) +``` + +### Permission for External Media + +External audio/video URLs require the `ALLOW_MEDIA_HOSTNAMES` permission in scene.json: + +```json +{ + "requiredPermissions": ["ALLOW_MEDIA_HOSTNAMES"], + "allowedMediaHostnames": ["stream.example.com", "cdn.example.com"] +} +``` + +### Multiple Video Surfaces + +Share one VideoPlayer across multiple screens by referencing the same `videoPlayerEntity`: + +```typescript +Material.setPbrMaterial(screen1, { + texture: Material.Texture.Video({ videoPlayerEntity: videoEntity }), +}) +Material.setPbrMaterial(screen2, { + texture: Material.Texture.Video({ videoPlayerEntity: videoEntity }), +}) +``` + +### Play a video on a glTF model + +You may want to play a video on a shape that is not a primitive, to have curved screens or exotic shapes. Use `GltfNodeModifiers` to swap the material of a GLTF model. + +```typescript +VideoPlayer.create(myEntity, { + src: 'https://player.vimeo.com/external/552481870.m3u8?s=c312c8533f97e808fccc92b0510b085c8122a875', + playing: true, +}) + +GltfNodeModifiers.create(myEntity, { + modifiers: [ + { + path: '', + material: { + material: { + $case: 'pbr', + pbr: { + texture: Material.Texture.Video({ + videoPlayerEntity: myEntity, + }), + }, + }, + }, + }, + ], +}) +``` + +### Video Limits & Tips + +- **Simultaneous videos**: Always avoid playing multiple videos at once. Only play more than 1 simultaneous video if explicitly requested. Maximum 5 simultaneous videos. +- **Distance-based control**: Pause video when player is far away to save bandwidth +- **Supported formats**: `.mp4` (H.264), `.webm`, HLS (`.m3u8`) for live streaming +- **Live streaming**: Use HLS (`.m3u8`) URLs — most reliable across clients + +For full component field details, supported formats, and advanced patterns, see `{baseDir}/references/media-reference.md`. + +## Important Notes + +- Audio files must be in the project's directory (relative paths from project root) +- Video requires HTTPS URLs — HTTP won't work +- Players must interact with the scene (click) before audio can play (browser autoplay policy) +- Keep audio files small — large files increase scene load time +- Use `.mp3` for music and `.ogg` for sound effects (smaller file sizes) +- For live video streaming, use HLS (.m3u8) URLs when possible +- If an audio file needs to be ready to play as the player interacts, use the `AssetLoad` component to pre-load the asset. diff --git a/ai-sdk-context/audio-video/references/media-reference.md b/ai-sdk-context/audio-video/references/media-reference.md new file mode 100644 index 0000000..4ff0768 --- /dev/null +++ b/ai-sdk-context/audio-video/references/media-reference.md @@ -0,0 +1,184 @@ +# Media Components Reference + +## AudioSource — Full Fields + +```typescript +import { AudioSource } from '@dcl/sdk/ecs' + +AudioSource.create(entity, { + audioClipUrl: 'sounds/effect.mp3', // Path to local audio file (required) + playing: false, // Start/stop playback + loop: false, // Loop when finished + volume: 1.0, // Volume 0.0 to 1.0 + pitch: 1.0 // Playback speed (0.5 = half, 2.0 = double) +}) +``` + +**Supported formats:** `.mp3` (recommended), `.ogg`, `.wav` + +Audio is spatial by default — volume decreases with distance from the entity. Place the entity where the sound should originate. + +### Playback Control + +```typescript +const audio = AudioSource.getMutable(entity) +audio.playing = true // Play +audio.playing = false // Stop +audio.volume = 0.5 // Adjust volume +audio.pitch = 1.5 // Speed up +``` + +### Reset and Replay + +To replay a sound effect from the beginning: +```typescript +const audio = AudioSource.getMutable(entity) +audio.playing = false +audio.playing = true // Restarts from beginning +``` + +## AudioStream — Full Fields + +```typescript +import { AudioStream } from '@dcl/sdk/ecs' + +AudioStream.create(entity, { + url: 'https://stream.example.com/radio.mp3', // Streaming URL (required) + playing: true, // Start/stop stream + volume: 0.5 // Volume 0.0 to 1.0 +}) +``` + +**Supported stream formats:** HTTP/HTTPS audio streams (`.mp3`, `.ogg`, `.aac`) + +AudioStream is NOT spatial — it plays at the same volume regardless of player distance. Best for background music or radio. + +## VideoPlayer — Full Fields + +```typescript +import { VideoPlayer } from '@dcl/sdk/ecs' + +VideoPlayer.create(entity, { + src: 'videos/clip.mp4', // Local file or external URL (required) + playing: true, // Start/stop playback + loop: false, // Loop when finished + volume: 1.0, // Volume 0.0 to 1.0 + playbackRate: 1.0, // Playback speed + position: 0 // Start time in seconds +}) +``` + +**Supported formats:** +- `.mp4` (H.264) — most compatible +- `.webm` — good quality, smaller files +- `.ogg` — open format +- `.m3u8` (HLS) — live streaming, most reliable for streams + +### Video Texture Setup + +VideoPlayer alone doesn't display video. You must create a video texture and apply it to a mesh: + +```typescript +// 1. Create mesh surface +MeshRenderer.setPlane(entity) + +// 2. Create video texture referencing the VideoPlayer entity +const videoTexture = Material.Texture.Video({ videoPlayerEntity: entity }) + +// 3. Apply as basic material (best performance) +Material.setBasicMaterial(entity, { texture: videoTexture }) + +// OR as PBR material with emissive (self-lit screen) +Material.setPbrMaterial(entity, { + texture: videoTexture, + roughness: 1.0, + specularIntensity: 0, + metallic: 0, + emissiveTexture: videoTexture, + emissiveIntensity: 0.6, + emissiveColor: Color3.White() +}) +``` + +### Live Streaming + +```typescript +// HLS stream +VideoPlayer.create(entity, { + src: 'https://example.com/stream.m3u8', + playing: true +}) + +// LiveKit video stream +VideoPlayer.create(entity, { + src: 'livekit-video://current-stream', + playing: true +}) +``` + +### Video Events + +```typescript +import { videoEventsSystem, VideoState } from '@dcl/sdk/ecs' + +videoEventsSystem.registerVideoEventsEntity(entity, (event) => { + console.log('State:', event.state) // VideoState enum + console.log('Time:', event.currentOffset) // Current playback time + console.log('Length:', event.videoLength) // Total duration +}) + +// Poll current state +const state = videoEventsSystem.getVideoState(entity) +``` + +**VideoState values:** `VS_READY`, `VS_PLAYING`, `VS_PAUSED`, `VS_ERROR`, `VS_BUFFERING`, `VS_SEEKING`, `VS_NONE` + +### Multiple Screens, One Video + +```typescript +// One VideoPlayer, shared across screens +VideoPlayer.create(screen1, { src: 'videos/shared.mp4', playing: true }) +const tex = Material.Texture.Video({ videoPlayerEntity: screen1 }) +Material.setBasicMaterial(screen1, { texture: tex }) +Material.setBasicMaterial(screen2, { texture: tex }) +``` + +### Video Limits + +| Quality Setting | Max Simultaneous Videos | +|----------------|------------------------| +| Low | 1 | +| Medium | 5 | +| High | 10 | + +### Media Permissions in scene.json + +External audio/video URLs require permissions: + +```json +{ + "requiredPermissions": ["ALLOW_MEDIA_HOSTNAMES"], + "allowedMediaHostnames": ["stream.example.com", "cdn.example.com"] +} +``` + +## AudioAnalysis (Advanced) + +Real-time frequency and amplitude data from audio sources: + +```typescript +import { AudioAnalysis, AudioAnalysisView } from '@dcl/sdk/ecs' + +// Enable analysis on an audio source entity +AudioAnalysis.create(audioEntity, {}) + +// Read analysis data in a system +engine.addSystem(() => { + const view = AudioAnalysisView.getOrNull(audioEntity) + if (view) { + // Use frequency/amplitude data for visualizers, beat detection, etc. + } +}) +``` + +Used for music visualizers, reactive environments, and beat-synced animations. diff --git a/ai-sdk-context/authoritative-server/SKILL.md b/ai-sdk-context/authoritative-server/SKILL.md new file mode 100644 index 0000000..003d6b0 --- /dev/null +++ b/ai-sdk-context/authoritative-server/SKILL.md @@ -0,0 +1,352 @@ +--- +name: authoritative-server +description: Build multiplayer Decentraland scenes with a headless authoritative server. Covers isServer() branching, registerMessages() for client-server communication, validateBeforeChange() for server-only state, Storage (world and player persistence), EnvVar (environment variables), and project structure. Use when the user wants authoritative multiplayer, anti-cheat, server-side validation, persistent storage, or server messages. Do NOT use for basic CRDT multiplayer without a server (see multiplayer-sync). +--- + +# Authoritative Server Pattern + +**IMPORTANT**: The authoritative server is feature in BETA. Always notify the user and ask them if they want to proceed in using this feature before adding it to the scene. + +Build multiplayer Decentraland scenes where a **headless server** controls game state, validates changes, and prevents cheating. The same codebase runs on both server and client, with the server having full authority. + +For basic CRDT multiplayer (no server), see the `multiplayer-sync` skill instead. + +## Setup + +### 1. Install the auth-server SDK branch (MANDATORY) + +You **must** use the `auth-server` tag — the standard `@dcl/sdk` does NOT include authoritative server APIs (`isServer`, `registerMessages`, `Storage`, `EnvVar`, etc.): + +```bash +npm install @dcl/sdk@auth-server +``` + +### 2. Configure scene.json + +Your `scene.json` **must** include these properties: + +- **`authoritativeMultiplayer: true`** — enables the authoritative server runtime. +- **`worldConfiguration.name`** — identifies the world for deployment, Storage, and EnvVar. +- **`logsPermissions`** — array of wallet addresses allowed to see server logs in the console. Without this, server `console.log()` output is hidden. + +```json +{ + "authoritativeMultiplayer": true, + "worldConfiguration": { + "name": "my-world-name.dcl.eth" + }, + "logsPermissions": ["0xYourWalletAddress"] +} +``` + +### 3. Run the scene + +Just use the normal preview — it automatically starts the authoritative server in the background when `authoritativeMultiplayer: true` is set in scene.json. + +> **Debugging note (do NOT tell the user to run this):** Under the hood, the preview runs `npx @dcl/hammurabi-server@next`. If the auth server isn't starting, check that the hammurabi process is running and look for errors in its output. + +## Server/Client Branching + +Use `isServer()` to branch logic in a single codebase: + +```typescript +import { isServer } from '@dcl/sdk/network' + +export async function main() { + if (isServer()) { + // Server-only: game logic, validation, state management + const { server } = await import('./server/server') + server() + return + } + + // Client-only: UI, input, message sending + setupClient() + setupUi() +} +``` + +The server runs your scene code headlessly (no rendering). It has access to all player positions via `PlayerIdentityData` and manages all authoritative game state. + +## Synced Components with Validation + +Define custom components that sync from server to all clients. **Always** use `validateBeforeChange()` to prevent clients from modifying server-authoritative state. + +### Custom Components (Global Validation) + +```typescript +import { engine, Schemas } from '@dcl/sdk/ecs' +import { AUTH_SERVER_PEER_ID } from '@dcl/sdk/network/message-bus-sync' + +export const GameState = engine.defineComponent('game:State', { + phase: Schemas.String, + score: Schemas.Number, + timeRemaining: Schemas.Number, +}) + +// Restrict ALL modifications to server only +GameState.validateBeforeChange((value) => { + return value.senderAddress === AUTH_SERVER_PEER_ID +}) +``` + +### Built-in Components (Per-Entity Validation) + +For built-in components like `Transform` and `GltfContainer`, use per-entity validation so you don't block client-side transforms on the player's own entities: + +```typescript +import { Entity, Transform, GltfContainer } from '@dcl/sdk/ecs' +import { AUTH_SERVER_PEER_ID } from '@dcl/sdk/network/message-bus-sync' + +type ComponentWithValidation = { + validateBeforeChange: ( + entity: Entity, + cb: (value: { senderAddress: string }) => boolean + ) => void +} + +function protectServerEntity( + entity: Entity, + components: ComponentWithValidation[] +) { + for (const component of components) { + component.validateBeforeChange(entity, (value) => { + return value.senderAddress === AUTH_SERVER_PEER_ID + }) + } +} + +// Usage: after creating a server-managed entity +const entity = engine.addEntity() +Transform.create(entity, { position: Vector3.create(10, 5, 10) }) +GltfContainer.create(entity, { src: 'assets/model.glb' }) +protectServerEntity(entity, [Transform, GltfContainer]) +``` + +### Syncing Entities + +After creating and protecting an entity, sync it to all clients: + +```typescript +import { syncEntity } from '@dcl/sdk/network' + +syncEntity(entity, [Transform.componentId, GameState.componentId]) +``` + +## Messages + +Use `registerMessages()` for client-to-server and server-to-client communication: + +### Define Messages + +```typescript +import { Schemas } from '@dcl/sdk/ecs' +import { registerMessages } from '@dcl/sdk/network' + +export const Messages = { + // Client -> Server + playerJoin: Schemas.Map({ displayName: Schemas.String }), + playerAction: Schemas.Map({ + actionType: Schemas.String, + data: Schemas.Number, + }), + + // Server -> Client + gameEvent: Schemas.Map({ + eventType: Schemas.String, + playerName: Schemas.String, + }), +} + +export const room = registerMessages(Messages) +``` + +### Send Messages + +```typescript +// Client sends to server +room.send('playerJoin', { displayName: 'Alice' }) + +// Server sends to ALL clients +room.send('gameEvent', { eventType: 'ROUND_START', playerName: '' }) + +// Server sends to ONE client +room.send( + 'gameEvent', + { eventType: 'YOU_WIN', playerName: 'Alice' }, + { to: [playerAddress] } +) +``` + +### Receive Messages + +```typescript +// Server receives from client +room.onMessage('playerJoin', (data, context) => { + if (!context) return + const playerAddress = context.from // Wallet address of sender + console.log(`[Server] Player joined: ${data.displayName} (${playerAddress})`) +}) + +// Client receives from server +room.onMessage('gameEvent', (data) => { + console.log(`Event: ${data.eventType}`) +}) +``` + +### Wait for State Sync + +Before sending messages from the client, wait until state is synchronized: + +```typescript +import { isStateSyncronized } from '@dcl/sdk/network' + +engine.addSystem(() => { + if (!isStateSyncronized()) return + + // Safe to send messages now + room.send('playerJoin', { displayName: 'Player' }) +}) +``` + +## Server Reading Player Positions + +The server can read **actual** player positions — critical for anti-cheat: + +```typescript +import { engine, PlayerIdentityData, Transform } from '@dcl/sdk/ecs' + +engine.addSystem(() => { + for (const [entity, identity] of engine.getEntitiesWith(PlayerIdentityData)) { + const transform = Transform.getOrNull(entity) + if (!transform) continue + + const address = identity.address + const position = transform.position + // Use actual server-verified position, not client-reported data + } +}) +``` + +Never trust client-reported positions. Always read `PlayerIdentityData` + `Transform` on the server. + +## Storage + +Persist data across server restarts. **Server-only** — guard with `isServer()`. + +```typescript +import { Storage } from '@dcl/sdk/server' +``` + +### World Storage (Global) + +Shared across all players: + +```typescript +// Store +await Storage.world.set('leaderboard', JSON.stringify(leaderboardData)) + +// Retrieve +const data = await Storage.world.get('leaderboard') +if (data) { + const leaderboard = JSON.parse(data) +} + +// Delete +await Storage.world.delete('oldKey') +``` + +### Player Storage (Per-Player) + +Keyed by player wallet address: + +```typescript +// Store +await Storage.player.set(playerAddress, 'highScore', String(score)) + +// Retrieve +const saved = await Storage.player.get(playerAddress, 'highScore') +const highScore = saved ? parseInt(saved) : 0 + +// Delete +await Storage.player.delete(playerAddress, 'highScore') +``` + +Storage only accepts strings. Use `JSON.stringify()`/`JSON.parse()` for objects and `String()`/`parseInt()` for numbers. + +Local development storage is at `node_modules/@dcl/sdk-commands/.runtime-data/server-storage.json`. + +## Environment Variables + +Configure your scene without hardcoding values. **Server-only** — guard with `isServer()`. + +```typescript +import { EnvVar } from '@dcl/sdk/server' + +// Read a variable with default +const maxPlayers = parseInt((await EnvVar.get('MAX_PLAYERS')) || '4') +const debugMode = ((await EnvVar.get('DEBUG')) || 'false') === 'true' +``` + +### Local Development + +Create a `.env` file in your project root: + +``` +MAX_PLAYERS=8 +GAME_DURATION=300 +DEBUG=true +``` + +Add `.env` to your `.gitignore`. + +### Deploy to Production + +```bash +# Set a variable +npx sdk-commands deploy-env MAX_PLAYERS --value 8 + +# Delete a variable +npx sdk-commands deploy-env OLD_VAR --delete +``` + +Deployed env vars take precedence over `.env` file values. + +## Recommended Project Structure + +``` +src/ +├── index.ts # Entry point — isServer() branching +├── client/ +│ ├── setup.ts # Client initialization, message handlers +│ └── ui.tsx # React ECS UI reading synced state +├── server/ +│ ├── server.ts # Server init, systems, message handlers +│ └── gameState.ts # Server state management class +└── shared/ + ├── schemas.ts # Synced component definitions + validateBeforeChange + └── messages.ts # Message definitions via registerMessages() +``` + +Put synced components and messages in `shared/` so both server and client import the same definitions. Keep server logic (Storage, EnvVar, game systems) in `server/`. Keep UI and client input in `client/`. + +## Testing & Debugging + +- **Log prefixes**: Use `[Server]` and `[Client]` prefixes in `console.log()` to distinguish server and client output in the terminal. +- **Stale CRDT files**: If you see "Outside of the bounds of written data" errors, delete `main.crdt` and `main1.crdt` files and restart. +- **Storage inspection**: Check `node_modules/@dcl/sdk-commands/.runtime-data/server-storage.json` to inspect persisted data during local development. +- **Timers**: `setTimeout`/`setInterval` are available via runtime polyfill. For game logic, prefer `engine.addSystem()` with a delta-time accumulator to stay in sync with the frame loop. +- **Entity sync issues**: Verify you call `syncEntity(entity, [componentIds])` with the correct component IDs (`MyComponent.componentId`). + +## Important Notes + +- **Use `Schemas.Int64` for timestamps**: `Schemas.Number` corrupts large numbers (13+ digits). Always use `Schemas.Int64` for values like `Date.now()`. +- **State sync readiness**: Clients must wait for `isStateSyncronized()` (from `@dcl/sdk/network`) to return `true` before sending messages. Note the intentional SDK typo: "Syncronized" not "Synchronized". +- **Custom vs built-in validation**: Custom components use global `validateBeforeChange((value) => ...)`. Built-in components (Transform, GltfContainer) use per-entity `validateBeforeChange(entity, (value) => ...)`. +- **Single codebase**: Both server and client run the same `index.ts` entry point. Use `isServer()` to branch. +- **No Node.js APIs**: The DCL runtime uses sandboxed QuickJS — no `fs`, `http`, etc. `setTimeout`/`setInterval` are supported. Use SDK-provided APIs (Storage, EnvVar, engine systems) for server-side operations. +- **SDK branch (MANDATORY)**: The auth-server pattern requires `npm install @dcl/sdk@auth-server`, not the standard `@dcl/sdk`. Without it, `isServer()`, `registerMessages()`, `Storage`, and `EnvVar` are unavailable. +- **scene.json required fields**: `authoritativeMultiplayer: true` must be set, and `logsPermissions: ["0xWalletAddress"]` must list wallet addresses that should see server logs. +- For basic CRDT multiplayer without a server, see the `multiplayer-sync` skill. + +For complete server setup examples, authentication flow, state reconciliation, Storage patterns, and EnvVar usage, see `{baseDir}/references/server-patterns.md`. diff --git a/ai-sdk-context/authoritative-server/references/server-patterns.md b/ai-sdk-context/authoritative-server/references/server-patterns.md new file mode 100644 index 0000000..5d2c3ef --- /dev/null +++ b/ai-sdk-context/authoritative-server/references/server-patterns.md @@ -0,0 +1,251 @@ +# Authoritative Server Patterns Reference + +## Complete Server Setup + +### Project Structure + +``` +src/ +├── index.ts # Entry point — isServer() branching +├── client/ +│ ├── setup.ts # Client init, input handlers, message senders +│ └── ui.tsx # React ECS UI (reads synced state, sends messages) +├── server/ +│ ├── server.ts # Server init, game loop, message handlers +│ └── gameState.ts # Server state management +└── shared/ + ├── schemas.ts # Custom component definitions + validateBeforeChange + └── messages.ts # Message definitions via registerMessages() +``` + +### Entry Point (index.ts) + +```typescript +import { isServer } from '@dcl/sdk/network' + +export async function main() { + if (isServer()) { + const { initServer } = await import('./server/server') + initServer() + return + } + + const { initClient } = await import('./client/setup') + const { setupUi } = await import('./client/ui') + initClient() + setupUi() +} +``` + +### Shared Schemas (shared/schemas.ts) + +```typescript +import { engine, Schemas, Entity } from '@dcl/sdk/ecs' +import { AUTH_SERVER_PEER_ID } from '@dcl/sdk/network/message-bus-sync' + +// Custom synced component +export const GameState = engine.defineComponent('game:State', { + phase: Schemas.String, + score: Schemas.Int, + timeRemaining: Schemas.Int +}) + +// Global validation — only server can modify +GameState.validateBeforeChange((value) => { + return value.senderAddress === AUTH_SERVER_PEER_ID +}) + +// For built-in components, use per-entity validation +type ComponentWithValidation = { + validateBeforeChange: (entity: Entity, cb: (value: { senderAddress: string }) => boolean) => void +} + +export function protectServerEntity(entity: Entity, components: ComponentWithValidation[]) { + for (const component of components) { + component.validateBeforeChange(entity, (value) => { + return value.senderAddress === AUTH_SERVER_PEER_ID + }) + } +} +``` + +### Shared Messages (shared/messages.ts) + +```typescript +import { Schemas } from '@dcl/sdk/ecs' +import { registerMessages } from '@dcl/sdk/network' + +export const Messages = { + // Client → Server + playerReady: Schemas.Map({ displayName: Schemas.String }), + playerAction: Schemas.Map({ action: Schemas.String, targetId: Schemas.Int }), + + // Server → Client + gameStarted: Schemas.Map({ roundNumber: Schemas.Int }), + playerScored: Schemas.Map({ playerName: Schemas.String, points: Schemas.Int }), + gameEnded: Schemas.Map({ winnerId: Schemas.String }) +} + +export const room = registerMessages(Messages) +``` + +### Server Logic (server/server.ts) + +```typescript +import { engine, PlayerIdentityData, Transform } from '@dcl/sdk/ecs' +import { syncEntity } from '@dcl/sdk/network' +import { room } from '../shared/messages' +import { GameState, protectServerEntity } from '../shared/schemas' + +export function initServer() { + // Create server-managed entities + const stateEntity = engine.addEntity() + GameState.create(stateEntity, { phase: 'lobby', score: 0, timeRemaining: 60 }) + protectServerEntity(stateEntity, [Transform]) + syncEntity(stateEntity, [GameState.componentId], 1) + + // Handle client messages + room.onMessage('playerReady', (data, context) => { + if (!context) return + console.log(`[Server] ${data.displayName} ready (${context.from})`) + }) + + room.onMessage('playerAction', (data, context) => { + if (!context) return + // Validate action on server + const playerPos = getPlayerPosition(context.from) + if (isValidAction(data.action, playerPos)) { + applyAction(data) + } + }) + + // Game loop + engine.addSystem(gameLoopSystem) +} +``` + +## Authentication Flow + +The auth server automatically provides player identity via `PlayerIdentityData`: + +```typescript +// Server reads actual player positions +engine.addSystem(() => { + for (const [entity, identity] of engine.getEntitiesWith(PlayerIdentityData)) { + const transform = Transform.getOrNull(entity) + if (!transform) continue + + // identity.address = wallet address (verified by server) + // transform.position = actual player position (not client-reported) + console.log(`[Server] ${identity.address} at`, transform.position) + } +}) +``` + +Never trust client-reported positions. The server sees real positions via `PlayerIdentityData` + `Transform`. + +## State Reconciliation + +When server state diverges from client state, the server always wins: + +```typescript +// Server-side: apply authoritative state +function reconcileState() { + const state = GameState.getMutable(stateEntity) + + // Server calculates correct state + state.timeRemaining = Math.max(0, state.timeRemaining - 1) + + if (state.timeRemaining <= 0 && state.phase === 'active') { + state.phase = 'ended' + room.send('gameEnded', { winnerId: findWinner() }) + } +} +``` + +Because `validateBeforeChange` blocks client writes, clients can only read the state and send messages. The server is the single source of truth. + +## Storage Patterns + +### World Storage (Global Data) + +```typescript +import { Storage } from '@dcl/sdk/server' + +// Save leaderboard +await Storage.world.set('leaderboard', JSON.stringify([ + { name: 'Alice', score: 100 }, + { name: 'Bob', score: 85 } +])) + +// Load leaderboard +const data = await Storage.world.get('leaderboard') +const leaderboard = data ? JSON.parse(data) : [] + +// Delete +await Storage.world.delete('leaderboard') +``` + +### Player Storage (Per-Player Data) + +```typescript +import { Storage } from '@dcl/sdk/server' + +// Save player progress +await Storage.player.set(playerAddress, 'progress', JSON.stringify({ + level: 5, + coins: 250, + achievements: ['first_kill', 'speedrun'] +})) + +// Load player progress +const saved = await Storage.player.get(playerAddress, 'progress') +const progress = saved ? JSON.parse(saved) : { level: 1, coins: 0, achievements: [] } +``` + +**Note:** Storage only accepts strings. Always `JSON.stringify()` objects and `String()` numbers. + +**Local dev storage location:** `node_modules/@dcl/sdk-commands/.runtime-data/server-storage.json` + +## Environment Variables + +```typescript +import { EnvVar } from '@dcl/sdk/server' + +// Read with defaults +const maxPlayers = parseInt((await EnvVar.get('MAX_PLAYERS')) || '4') +const gameDuration = parseInt((await EnvVar.get('GAME_DURATION')) || '300') +const debugMode = ((await EnvVar.get('DEBUG')) || 'false') === 'true' +``` + +### Local Development (.env file) + +``` +MAX_PLAYERS=8 +GAME_DURATION=300 +DEBUG=true +``` + +### Production Deployment + +```bash +npx sdk-commands deploy-env MAX_PLAYERS --value 8 +npx sdk-commands deploy-env GAME_DURATION --value 300 +npx sdk-commands deploy-env OLD_VAR --delete +``` + +## scene.json Required Fields + +```json +{ + "authoritativeMultiplayer": true, + "worldConfiguration": { + "name": "my-world.dcl.eth" + }, + "logsPermissions": ["0xYourWalletAddress"] +} +``` + +- `authoritativeMultiplayer: true` — enables the headless server runtime +- `worldConfiguration.name` — identifies the world (required for Storage and deploy) +- `logsPermissions` — wallet addresses that can see `console.log()` from the server diff --git a/ai-sdk-context/build-ui/SKILL.md b/ai-sdk-context/build-ui/SKILL.md new file mode 100644 index 0000000..38b31d8 --- /dev/null +++ b/ai-sdk-context/build-ui/SKILL.md @@ -0,0 +1,480 @@ +--- +name: build-ui +description: Build 2D screen-space UI for Decentraland scenes using React-ECS (JSX). Create HUDs, menus, health bars, scoreboards, dialogs, buttons, inputs, and dropdowns. Use when the user wants screen overlays, on-screen UI, HUD elements, menus, or form inputs. Do NOT use for 3D in-world text (see advanced-rendering) or clickable 3D objects (see add-interactivity). +--- + +# Building UI with React-ECS + +Decentraland SDK7 uses a React-like JSX system for 2D UI overlays. + +## When to Use Which UI Approach + +| Need | Approach | Component | +|------|----------|-----------| +| Screen-space HUD, menus, buttons | React-ECS (this skill) | `UiEntity`, `Label`, `Button`, `Input`, `Dropdown` | +| 3D text floating in the world | TextShape + Billboard | See **advanced-rendering** skill | +| Open a web page | `openExternalUrl` | See **scene-runtime** skill | +| Clickable objects in 3D space | Pointer events | See **add-interactivity** skill | + +Use React-ECS for any 2D overlay: scoreboards, health bars, dialogs, inventories, settings menus. Use TextShape for labels above NPCs or objects in the 3D world. + +## Setup + +### File: src/ui.tsx +```tsx +import ReactEcs, { ReactEcsRenderer, UiEntity, Label, Button } from '@dcl/sdk/react-ecs' + +const MyUI = () => ( + + +) + +export function setupUi() { + ReactEcsRenderer.setUiRenderer(MyUI, { virtualWidth: 1920, virtualHeight: 1080 }) +} +``` + +### File: src/index.ts +```typescript +import { setupUi } from './ui' + +export function main() { + setupUi() +} +``` + +### tsconfig.json (already configured by /init) + +The SDK template already includes the required JSX settings — do NOT modify tsconfig.json: +- `"jsx": "react-jsx"` +- `"jsxImportSource": "@dcl/sdk/react-ecs-lib"` + +## Core Components + +### UiEntity (Container) +```tsx +import { Color4 } from '@dcl/sdk/math' + + +``` + +### Label (Text) +```tsx +import { Color4 } from '@dcl/sdk/math' + +