projectNEWM · AndrewWestberg · Jan 31, 2026 · Jan 30, 2026
diff --git a/.agent/skills/git-commit-formatter/SKILL.md b/.agent/skills/git-commit-formatter/SKILL.md
@@ -0,0 +1,30 @@
+---
+name: git-commit-formatter
+description: Formats git commit messages according to Conventional Commits specification. Use this when the user asks to commit changes or write a commit message.
+---
+
+# Git Commit Formatter Skill
+
+When writing a git commit message, you MUST follow the Conventional Commits specification.
+
+## Format
+`<type>[optional scope]: <description>`
+
+## Allowed Types
+- **feat**: A new feature
+- **fix**: A bug fix
+- **docs**: Documentation only changes
+- **style**: Changes that do not affect the meaning of the code (white-space, formatting, etc)
+- **refactor**: A code change that neither fixes a bug nor adds a feature
+- **perf**: A code change that improves performance
+- **test**: Adding missing tests or correcting existing tests
+- **chore**: Changes to the build process or auxiliary tools and libraries such as documentation generation
+
+## Instructions
+1. Analyze the changes to determine the primary `type`.
+2. Identify the `scope` if applicable (e.g., specific component or file).
+3. Write a concise `description` in imperative mood (e.g., "add feature" not "added feature").
+4. If there are breaking changes, add a footer starting with `BREAKING CHANGE:`.
+
+## Example
+`feat(auth): implement login with google`
diff --git a/.agent/task/newm-chain-db-improvement/newm-chain-db-performance-prd.md b/.agent/task/newm-chain-db-improvement/newm-chain-db-performance-prd.md
@@ -0,0 +1,104 @@
+# Chain Sync Initial-Load Performance Improvements
+
+## Overview
+Improve full initial sync performance for newm-chain/newm-chain-db without changing external behavior or weakening transactional guarantees. Focus on reducing write amplification, eliminating N+1 query patterns, and shortening transaction duration while keeping the system runnable and correct after every incremental change.
+
+## Requirements
+- [x] Initial sync throughput improves measurably without regressions in correctness.
+- [x] No changes to external APIs or observable ledger semantics.
+- [x] Transactional guarantees remain intact (no partial or inconsistent state).
+- [x] Each incremental change is safe, isolated, and testable.
+- [x] Rollback handling remains correct and reliable.
+- [x] Sync logic remains stable and recoverable after crashes/restarts.
+
+## Technical Design
+
+### Components Affected
+- Module: newm-chain (BlockDaemon)
+  - Guard rollback logic to run only on actual rollbacks.
+  - Tune block batching behavior for initial sync.
+  - Optional config-gated fast sync for non-critical metadata/logs.
+- Module: newm-chain-db (ChainRepositoryImpl)
+  - Batch insert chain blocks.
+  - Compute etaV in-memory for batches to reduce DB reads.
+- Module: newm-chain-db (LedgerRepositoryImpl)
+  - Replace N+1 per-UTXO/asset queries with batch operations.
+  - Batch spend updates.
+  - Reduce repeated metadata/log reads within a block.
+
+### Database Changes
+No schema changes required. Existing indexes already cover key lookups:
+- chain: unique indexes on slot_number and block_number.
+- ledger_utxos: indexes on block_spent, block_created, (tx_id, tx_ix), ledger_id, transaction_spent.
+- ledger_assets: (policy, name).
+- ledger_utxo_assets: ledger_utxo_id, ledger_asset_id.
+- logs: address_tx_log(address, tx_id), block_number indexes; native_asset_log(block_number).
+- stake: stake_delegations and stake_registrations indexes.
+
+### API Changes
+None.
+
+## Implementation Steps
+1. Make rollbacks explicit and rare
+   - Only perform rollback deletes when a RollBackward actually occurs. A RollBackward is always sent when the connection to Kogmios is made, so normal forward syncs should not trigger rollback logic.
+   - Keep rollback semantics intact; avoid delete/rollback on normal forward batches.
+
+2. Batch chain inserts and etaV computation
+   - Implement batch insert for chain blocks in ChainRepositoryImpl.insertAll.
+   - Precompute etaV sequentially in memory per batch using the last known etaV.
+
+3. Batch ledger UTXO creation
+   - Preload ledger ids for all addresses in a block.
+   - Batch insert missing ledger rows.
+   - Batch insert ledger_utxos.
+   - Resolve asset ids in a single query and batch insert missing assets.
+   - Batch insert ledger_utxo_assets rows.
+
+4. Batch spend updates
+   - Replace per-UTXO updates with a batch update keyed by (tx_id, tx_ix).
+
+5. Optimize metadata/log reads within a block
+   - Cache asset and metadata lookups per block to avoid repeated DB reads.
+
+6. Stabilize batch sizing for initial sync
+   - Cap blockBufferSize and use deterministic batch sizes for catch-up.
+
+7. Add performance instrumentation
+   - Time per repository method and per batch.
+   - Capture query counts where possible.
+
+## Testing Strategy
+- Replay a representative block range and verify:
+  - Chain table continuity and etaV correctness.
+  - Ledger UTXO counts and balances vs. baseline.
+  - Stake registrations/delegations match baseline.
+  - Native asset metadata/log outputs unchanged (or match when fast sync is off).
+- Integration test rollback scenario to ensure rollback correctness.
+- Measure sync throughput before and after each change.
+- Verify idempotency and crash recovery across mid-sync restarts.
+
+## Rollout Plan
+- Implement changes incrementally in separate PRs or commits.
+- Validate each step with a small sync range and a rollback test.
+- Only enable optional fast sync mode after validation and with a backfill plan.
+
+## Open Questions
+- What is the acceptable tradeoff between throughput and metadata/log latency during initial sync?
+
+---
+
+Status: ✅ Completed
+Date: 2026-01-30
+Author: OpenCode
+
+## Progress
+- [x] Task-001: Rollback only on RollBackward (gate rollback deletes to rollback events and clear forward buffer on rollback).
+- [x] Task-002: Cap batch size during catch-up (cap block buffer growth for non-tip batches).
+- [x] Task-003: Batch insert chain blocks (batch chain inserts and compute etaV in-memory per batch).
+- [x] Task-004: Batch ledger lookup/insert (preload ledger ids by address and batch insert missing ledger rows).
+- [x] Task-005: Batch insert ledger_utxos (insert created UTXOs in batch and return ids for asset join).
+- [x] Task-006: Batch resolve and insert ledger_assets (resolve asset ids in one query and batch insert missing assets).
+- [x] Task-007: Batch insert ledger_utxo_assets (insert asset-to-utxo rows in a batch).
+- [x] Task-008: Batch spend updates (batch update spent UTXOs by block).
+- [x] Task-009: Per-block metadata lookup cache (cache ledger asset/metadata lookups within a block to reduce repeated reads).
+- [~] Step-007: Performance instrumentation (deferred).
diff --git a/.agent/task/newm-chain-db-improvement/newm-chain-db-performance-tasks.json b/.agent/task/newm-chain-db-improvement/newm-chain-db-performance-tasks.json
@@ -0,0 +1,228 @@
+{
+  "metadata": {
+    "title": "NEWM Chain DB Initial Sync Performance Tasks",
+    "created": "2026-01-30",
+    "updated": "2026-01-30",
+    "version": "1.0.0",
+    "totalTasks": 9,
+    "prdReference": ".agent/task/newm-chain-db-improvement/newm-chain-db-performance-prd.md",
+    "description": "Incremental, safe performance improvements for initial chain sync without changing external behavior or transactional guarantees",
+    "importantNote": "Each task must keep the system runnable and preserve existing rollback and transactional semantics. Validate after every change."
+  },
+  "phases": [
+    {
+      "id": "phase-1",
+      "name": "Rollback & Batch Safety",
+      "description": "Remove unnecessary rollback work during forward sync and establish safe batch boundaries",
+      "riskLevel": "medium",
+      "tasks": [
+        {
+          "id": "task-001",
+          "title": "Rollback only on RollBackward",
+          "description": "Stop running rollback deletes on every forward batch; perform rollbacks only when RollBackward is received",
+          "status": "completed",
+          "priority": "high",
+          "estimatedHours": 2,
+          "dependencies": [],
+          "targetPath": "newm-chain/src/main/kotlin/io/newm/chain/daemon/BlockDaemon.kt",
+          "acceptanceCriteria": [
+            "Forward-only sync no longer triggers rollback deletes",
+            "Rollback events still cleanly revert chain/ledger/log tables"
+          ],
+          "testPlan": [
+            "Sync forward over N blocks and verify no rollback deletes",
+            "Simulate rollback and verify state consistency"
+          ]
+        },
+        {
+          "id": "task-002",
+          "title": "Cap batch size during catch-up",
+          "description": "Introduce a safe max blockBufferSize for initial sync while preserving tip syncing behavior",
+          "status": "completed",
+          "priority": "medium",
+          "estimatedHours": 1,
+          "dependencies": ["task-001"],
+          "targetPath": "newm-chain/src/main/kotlin/io/newm/chain/daemon/BlockDaemon.kt",
+          "acceptanceCriteria": [
+            "blockBufferSize does not grow unbounded during catch-up",
+            "Tip sync latency unchanged"
+          ],
+          "testPlan": [
+            "Observe batch size during long catch-up",
+            "Verify tip syncing remains responsive"
+          ]
+        }
+      ]
+    },
+    {
+      "id": "phase-2",
+      "name": "Chain Inserts",
+      "description": "Reduce per-block DB work for chain table writes",
+      "riskLevel": "medium",
+      "tasks": [
+        {
+          "id": "task-003",
+          "title": "Batch insert chain blocks",
+          "description": "Replace per-block inserts with batchInsert and compute etaV in-memory per batch",
+          "status": "completed",
+          "priority": "high",
+          "estimatedHours": 3,
+          "dependencies": ["task-001"],
+          "targetPath": "newm-chain-db/src/main/kotlin/io/newm/chain/database/repository/ChainRepositoryImpl.kt",
+          "acceptanceCriteria": [
+            "Chain rows match baseline for same range",
+            "etaV continuity preserved across batch boundaries"
+          ],
+          "testPlan": [
+            "Compare chain table rows vs baseline",
+            "Validate etaV sequence for a multi-batch range"
+          ]
+        }
+      ]
+    },
+    {
+      "id": "phase-3",
+      "name": "Ledger UTXO Writes",
+      "description": "Eliminate N+1 queries for UTXO creation and asset joins",
+      "riskLevel": "high",
+      "tasks": [
+        {
+          "id": "task-004",
+          "title": "Batch ledger lookup/insert",
+          "description": "Preload ledger ids by address and batch insert missing ledger rows",
+          "status": "completed",
+          "priority": "high",
+          "estimatedHours": 3,
+          "dependencies": ["task-003"],
+          "targetPath": "newm-chain-db/src/main/kotlin/io/newm/chain/database/repository/LedgerRepositoryImpl.kt",
+          "acceptanceCriteria": [
+            "Ledger rows match baseline",
+            "No duplicate ledger rows created"
+          ],
+          "testPlan": [
+            "Compare ledger table counts and distinct addresses",
+            "Verify existing addresses are reused"
+          ]
+        },
+        {
+          "id": "task-005",
+          "title": "Batch insert ledger_utxos",
+          "description": "Insert created UTXOs in batch and return ids for asset join",
+          "status": "completed",
+          "priority": "high",
+          "estimatedHours": 3,
+          "dependencies": ["task-004"],
+          "targetPath": "newm-chain-db/src/main/kotlin/io/newm/chain/database/repository/LedgerRepositoryImpl.kt",
+          "acceptanceCriteria": [
+            "UTXO rows match baseline",
+            "UTXO fields (datum, scripts, creds) preserved"
+          ],
+          "testPlan": [
+            "Compare UTXO counts and sample rows vs baseline",
+            "Verify datum/script fields for script outputs"
+          ]
+        },
+        {
+          "id": "task-006",
+          "title": "Batch resolve and insert ledger_assets",
+          "description": "Resolve asset ids in one query and batch insert missing assets",
+          "status": "completed",
+          "priority": "high",
+          "estimatedHours": 2,
+          "dependencies": ["task-005"],
+          "targetPath": "newm-chain-db/src/main/kotlin/io/newm/chain/database/repository/LedgerRepositoryImpl.kt",
+          "acceptanceCriteria": [
+            "Ledger assets match baseline",
+            "No duplicate assets for same policy/name"
+          ],
+          "testPlan": [
+            "Compare ledger_assets counts and sample rows",
+            "Verify policy/name uniqueness"
+          ]
+        },
+        {
+          "id": "task-007",
+          "title": "Batch insert ledger_utxo_assets",
+          "description": "Insert all asset-to-utxo rows in batch",
+          "status": "completed",
+          "priority": "high",
+          "estimatedHours": 2,
+          "dependencies": ["task-006"],
+          "targetPath": "newm-chain-db/src/main/kotlin/io/newm/chain/database/repository/LedgerRepositoryImpl.kt",
+          "acceptanceCriteria": [
+            "UTXO asset links match baseline",
+            "Amounts preserved"
+          ],
+          "testPlan": [
+            "Compare asset balances per policy/name vs baseline",
+            "Verify UTXO asset counts for sample transactions"
+          ]
+        }
+      ]
+    },
+    {
+      "id": "phase-4",
+      "name": "Spend & Metadata Optimization",
+      "description": "Reduce per-input updates and repeated metadata reads",
+      "riskLevel": "medium",
+      "tasks": [
+        {
+          "id": "task-008",
+          "title": "Batch spend updates",
+          "description": "Replace per-spent-UTXO updates with a batch update",
+          "status": "completed",
+          "priority": "medium",
+          "estimatedHours": 2,
+          "dependencies": ["task-005"],
+          "targetPath": "newm-chain-db/src/main/kotlin/io/newm/chain/database/repository/LedgerRepositoryImpl.kt",
+          "acceptanceCriteria": [
+            "Spent UTXOs match baseline",
+            "No unspent UTXOs incorrectly marked"
+          ],
+          "testPlan": [
+            "Compare spent UTXO counts after block range",
+            "Verify random sample of unspent UTXOs remain unspent"
+          ]
+        },
+        {
+          "id": "task-009",
+          "title": "Per-block metadata lookup cache",
+          "description": "Cache ledger asset and metadata queries within a block to reduce repeated reads",
+          "status": "completed",
+          "priority": "medium",
+          "estimatedHours": 2,
+          "dependencies": ["task-007"],
+          "targetPath": "newm-chain/src/main/kotlin/io/newm/chain/daemon/BlockDaemon.kt",
+          "acceptanceCriteria": [
+            "Native asset log output unchanged",
+            "Fewer metadata DB queries per block"
+          ],
+          "testPlan": [
+            "Compare log outputs for sample blocks",
+            "Measure query count per block"
+          ]
+        }
+      ]
+    }
+  ],
+  "summary": {
+    "totalPhases": 4,
+    "totalTasks": 9,
+    "estimatedTotalHours": 20,
+    "criticalPath": ["task-001", "task-003", "task-004", "task-005", "task-006", "task-007"],
+    "keyFiles": [
+      "newm-chain/src/main/kotlin/io/newm/chain/daemon/BlockDaemon.kt",
+      "newm-chain-db/src/main/kotlin/io/newm/chain/database/repository/ChainRepositoryImpl.kt",
+      "newm-chain-db/src/main/kotlin/io/newm/chain/database/repository/LedgerRepositoryImpl.kt"
+    ],
+    "deferred": [
+      "step-007: performance instrumentation"
+    ],
+    "safetyNotes": [
+      "No API changes",
+      "Keep rollback semantics identical",
+      "Each change must keep sync runnable and consistent",
+      "Validate after every incremental change"
+    ]
+  }
+}
diff --git a/.opencode b/.opencode
@@ -0,0 +1 @@
+.agent
diff --git a/buildSrc/src/main/kotlin/Versions.kt b/buildSrc/src/main/kotlin/Versions.kt
@@ -25,7 +25,7 @@ object Versions {
     const val J_AUDIO_TAGGER = "3.0.1"
     const val KOIN = "4.1.0-Beta8"
     const val KOIN_TEST = "4.1.1"
-    const val KOGMIOS = "2.6.1"
+    const val KOGMIOS = "2.7.1"
     const val KOTLINX_SERIALIZATION = "1.9.0"
     const val KOTLIN_LOGGING = "7.0.13"
     const val KOTLIN_PLUGIN = "2.3.0"

diff --git a/gradle/wrapper/gradle-wrapper.properties b/gradle/wrapper/gradle-wrapper.properties
@@ -1,6 +1,6 @@
 distributionBase=GRADLE_USER_HOME
 distributionPath=wrapper/dists
-distributionUrl=https\://services.gradle.org/distributions/gradle-9.2.1-bin.zip
+distributionUrl=https\://services.gradle.org/distributions/gradle-9.3.1-bin.zip
 networkTimeout=10000
 validateDistributionUrl=true
 zipStoreBase=GRADLE_USER_HOME