Open
Conversation
6-agent swarm research covering: - HyFire2 master: 3 profiling classes, 5 monitors, 20+ scripts, 2 Python analyzers - HyFire2 analysis/docs branches (11): 3-month retrospective, hotspot analyses, automation playbooks - HyFire2 feature branches (18): monitoring UIs, flame charts, mobile perf, stress testers - HyFire2 perf/fix branches (27): optimization techniques (caching, pooling, deferral) - HYTOPIA SDK: Telemetry, PerformanceMetricsManager, DebugPanel, Stats classes, 8 gaps identified - Headless/automation: Puppeteer scripts, Chrome trace parsers, ARM64 sim, CI/CD roadmap - SYNTHESIS: 965-line framework spec with architecture, scenarios, metrics, 4-phase roadmap Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
SDK Module (server/src/metrics/ + server/src/bots/): - PerformanceMonitor: singleton profiler with CircularBuffer tick history, per-operation p50/p95/p99 percentiles, spike detection, entity profiling - @monitor decorator + monitorBlock/monitorAsyncBlock helpers - NetworkMetrics: bandwidth/packet/serialization tracking singleton - CpuProfiler: V8 Inspector CPU profile + heap snapshot capture - WorldLoop integration: per-tick phase timing (entities, physics, network) - EntityManager integration: opt-in per-entity cost tracking - Telemetry.ts: dual-path Sentry + PerformanceMonitor when enabled - Bot system: BotPlayer, BotManager, 4 behaviors (Idle/RandomWalk/Chase/Interact) Benchmark CLI (packages/perf-tools/): - ScenarioLoader: YAML/JSON benchmark scenario parser - BenchmarkRunner: orchestrates scenario execution with phases - MetricCollector: aggregates server + client snapshots - HeadlessClient: Puppeteer-based headless browser for client metrics - BaselineComparer: regression detection with warning/fail thresholds - ConsoleReporter + JsonReporter - 5 built-in presets (idle, stress, large-world, many-players, combined) Trace Analysis: - TraceParser: Chrome DevTools trace → frame timings, long tasks, GC events - CpuProfileAnalyzer: V8 .cpuprofile → hot functions, call tree, WASM mapping - SpikeCorrelator: correlate tick spikes with entity spawns, GC, phase overruns - NoiseFilter: IQR outlier removal, moving average, change point detection CI/CD: - perf-gate.yml: PR performance gate with baseline comparison - perf-baseline-update.yml: baseline update on master push Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All benchmarks PASS on all thresholds. Results: - idle: avg=0.05ms p99=0.15ms mem=40.7MB (empty server) - stress: avg=0.26ms p99=1.34ms mem=47.0MB (100 bots: walk/chase/interact) - entity-density: avg=0.39ms p99=0.79ms mem=43.7MB (500 dynamic entities) - block-churn: avg=0.75ms p99=1.66ms mem=53.6MB (10 clients, continuous edits) - large-world: avg=0.35ms p99=0.60ms mem=43.5MB (boilerplate map + 20 bots) - blocks-500k: avg=0.16ms p99=1.60ms mem=51.6MB (500K blocks, 20 WS clients) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…neck Results from 5 heavy stress tests: - combined (100 bots + 10 clients): ALL PASS, avg 1.73ms tick - many-players (50 WS + 50 bots): ALL PASS, avg 0.96ms tick - join-storm (100 clients at once): FAIL p99 71.54ms — serialize_packets bottleneck - blocks-1m-multi-world (4x1M blocks + 40 clients): ALL PASS, avg 0.04ms tick - blocks-10m-dense (10M blocks + 1 client): ALL PASS, avg 0.16ms tick Key finding: join-storm exposes a chunk serialization bottleneck when 100+ clients connect simultaneously. serialize_packets avg=16.15ms, p95=45.91ms with a max tick spike of 1253ms. This is the only scenario that exceeded thresholds (p99 71.54ms vs 30ms limit). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ngle core) CPU throttled to 30% of 1 core via taskset + cpulimit, simulating a ~1.5GHz single-vCPU cloud VM (vs dev machine: 9800X3D 16-thread 5GHz). Throttled results vs unthrottled: - idle: 0.07ms avg (was 0.05ms) — PASS - stress (100 bots): 0.47ms avg (was 0.26ms) — PASS - combined (100 bots + 10 clients): 2.22ms avg (was 1.73ms) — PASS - many-players (50 WS + 50 bots): 1.46ms/11.73ms p99 (was 0.96ms/3.58ms) — PASS - block-churn (10 clients + edits): 1.79ms/32.30ms p99 (was 0.75ms/1.66ms) — FAIL p99 - blocks-10m-dense (10M blocks): 0.37ms avg (was 0.16ms) — PASS - join-storm (100 clients): 14.46ms avg/376ms p99 (was 4.40ms/71ms) — FAIL avg+p99 Key findings on weak hardware: 1. join-storm is catastrophic: avg tick=14.46ms (basically at budget), p99=376ms, serialize_packets p95=633ms. 100 concurrent joins on a weak server = unplayable for everyone. 2. block-churn now fails p99 (32ms vs 30ms threshold) — on-the-edge. 3. combined/stress/many-players still pass comfortably. 4. Serialization is the #1 bottleneck across all failing tests. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- ProcessMonitor: reads /proc/<pid>/stat + status for CPU%, RSS, threads, FDs - BenchmarkRunner: integrates ProcessMonitor, graceful PerfHarness fallback, log capture - MetricCollector: process snapshot collection support - ConsoleReporter: process metrics display with CPU% threshold warnings - CLI: --no-perf-api and --log-file options - Scripts: link-sdk.sh and setup-game.sh for external game SDK linkage - Presets: hyfire2-bots.yaml and zoo-game-bots.yaml Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The server spawns with shell=true+detached=true, so the PID is the shell process, not the actual node worker. Fixed by scanning /proc for all PIDs sharing the same process group and summing their CPU/RSS/threads/FDs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Server changes: - Restore WorldMapCodec, WorldMapChunkCacheCodec, WorldMapFileLoader, WorldMapArtifactsGenerator from feature/map-compression branch - Export all map codec types from SDK barrel (index.ts) - World.loadMap() accepts string paths + compressed formats with auto-detect Benchmark results: - HyFire2 5v5 bots: avg tick 0.61ms, p99 1.34ms, 431MB heap, 1.2GB RSS - Zoo Game (OS-only): avg CPU 7.3%, 746MB RSS, 24 threads Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…erfHarness benchmark Added startModelLoopedAnimations, startModelOneshotAnimations, setModelNodeEmissiveColor, setModelNodeEmissiveIntensity to Entity.ts for Zoo Game SDK compatibility. Zoo Game now benchmarks with full PerfHarness: avg tick 0.25ms, p99 0.85ms, 313MB heap. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds PerfBridge to the client that exposes window.__HYTOPIA_PERF__ with rich snapshots (FPS, draw calls, triangles, entities, chunks, GLTF stats, JS heap). BenchmarkRunner now optionally launches a headless browser via --with-client to collect client metrics alongside server metrics. Includes client threshold support and rich console output. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Changed waitUntil from 'networkidle2' to 'load' since the game client maintains persistent WebSocket connections that prevent network idle. Added try-catch around HeadlessClient launch in BenchmarkRunner so client failures don't crash the whole benchmark. Verified: idle benchmark with --with-client shows client metrics (FPS, draw calls, triangles, entities, chunks, JS heap). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix HeadlessClient: ignoreHTTPSErrors, CDP cert bypass, SwiftShader GL - Patch fetch() to strip unsupported targetAddressSpace (Chrome PNA API) - Add warmCert step for self-signed HTTPS certs - Fix BaselineComparer: null-safe operations, nested JSON format support - Add console log forwarding from headless browser for debugging - A/B results: blob shadows add +1 draw call, +6.7% frame time, +0.9% triangles Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix SwiftShader: use --enable-unsafe-swiftshader (old flag deprecated) - Fix HTTPS certs: ignoreHTTPSErrors + CDP Security.setIgnoreCertificateErrors - Fix fetch: patch targetAddressSpace (Chrome PNA API not available) - Fix warmCert: pre-accept self-signed HTTPS cert before client navigation - Add player entity spawning in perf-harness (was spectator-only) - Add set_camera, throttle_cpu, walk_player (sendMovement via network packets) - Add wait_for_entities action for entity load synchronization - Add stress-walkthrough preset: 200 idle entities, deterministic positions - Expose __HYTOPIA_GAME__ on window for headless camera/input control - Fix BaselineComparer null-safety for operations + nested format support A/B results (PR #2 blob shadows, 16 visible entities, SwiftShader): - Frame time: 83ms → 152ms (+82% FAIL) - Max draw calls: 25 → 82 (+228%) - Max triangles: 263k → 584k (+121%) - Bug found: TransparentSortData missing on shadow meshes (error spam) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ted mobile Mobile simulation via CDP Emulation.setCPUThrottlingRate (rate=4). Desktop FPS 12.8 → Mobile FPS 11.0 (baseline), 11.8 (blob shadows). Frame time: 113ms → 123ms (+8.5% WARN) on mobile with shadows. Server metrics unaffected (throttle is client-side only). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Performance Benchmark Resultsidle-baseline
stress-test
Generated by @hytopia/perf-tools |
Performance Benchmark Resultsidle-baseline
stress-test
Generated by @hytopia/perf-tools |
Performance Benchmark Resultsidle-baseline
stress-test
Generated by @hytopia/perf-tools |
Performance Benchmark Resultsidle-baseline
stress-test
Generated by @hytopia/perf-tools |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
What is this?
A general-purpose performance framework for ANY HYTOPIA game. Two components:
PerformanceMonitor.instance.enable()- one line to start profiling@hytopia/perf-tools):hytopia-bench run --preset stress- repeatable benchmarksTest plan
hytopia-bench run --preset idleagainst sdk-example🤖 Generated with Claude Code