Skip to content

Conversation

Copy link
Contributor

Copilot AI commented Dec 20, 2025

Introduces a V2 telemetry schema to address storage inefficiencies (60-80% reduction expected) while preserving wire compatibility with existing V1 format.

Changes

New proto file: proto/telemetry/v1/telemetry_compact.proto

  • EnvelopeV2 - Streaming wrapper for V2 messages
  • TelemetryHeaderV2 - base_time_unix_nanos + typed metadata (server_id, mode, level); deltas from base per-frame instead of repeated Timestamp
  • SessionStatic - Rarely-changing session data (session_id, map_name, client_name) moved from per-frame to header
  • LobbySessionStateFrameV2 - sint32 delta_nanos from base time (±2.1s range, ~11ms/frame typical)
  • CompactSessionState - float32 with packed encoding for all vectors (disc position/velocity, game state); enums for pause_state and match_type
  • CompactTeam/Player/Stats - Float32 transforms (head/body/hand pos/forward/left/up), all vectors length-3 validated
  • CompactPlayerBones - Keyframe interval + delta frames; changed_mask bitmask (little-endian bit ordering); translations (3×N floats), orientations (4×N quaternions)

Storage Impact

Optimization V1 V2
Time per frame Timestamp (10+ bytes) sint32 delta (1-5 bytes)
Vectors double (8 bytes × 3) float packed (4 bytes × 3)
Enums string (5-20 bytes) varint (1-2 bytes)
Bones Full per frame Keyframe + delta + mask

Usage

// Producer
TelemetryHeaderV2 {
  capture_id: "uuid"
  base_time_unix_nanos: 1703001234000000000
  timestep_nanos: 11111111  // 90 Hz
  static: { session_id: "...", map_name: "arena", match_type: ARENA_PUBLIC }
}

LobbySessionStateFrameV2 {
  frame_index: 42
  // delta_nanos omitted when fixed timestep
  session: { disc_position: [0.0, 1.5, 0.0], ... }
  player_bones: { frame_modulus: 5, frames: [...] }
}

Wire Compatibility

Existing Envelope, TelemetryHeader, LobbySessionStateFrame remain unchanged. Consumers can adopt V2 progressively alongside V1.

Out of Scope

  • rtapi integration (future PR)
  • int16/int24 quantization (future PR)
  • Weapon/ordnance enum tables (future PR)

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • buf.build
    • Triggering command: /tmp/buf /tmp/buf lint (dns block)
  • esm.ubuntu.com
    • Triggering command: /usr/lib/apt/methods/https /usr/lib/apt/methods/https (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Original prompt

Goal
Add a compact telemetry schema (V2) alongside existing telemetry to
address storage inefficiencies without breaking wire compatibility.

Primary inefficiencies observed

  • Per-frame google.protobuf.Timestamp repeated in every
    LobbySessionStateFrame. Solution: base time in header + per-frame delta
    varint.
  • Full enginehttp.SessionResponse embedded per frame; many fields are
    static or infrequently changing. Solution: split static vs dynamic;
    static in header; dynamic in frames; or use change events.
  • Orientation/position/velocity stored as repeated double (64-bit).
    Solution: float32 and/or quantization (packed repeated) in compact
    messages.
  • String enums (paused_state, team, goal_type, match_type, map_name,
    team_name, weapon/ordnance/tac_mod, etc.). Solution: enums or ID table
    in header.
  • Player bones repeated fully per frame. Solution: keyframe + delta frames
    with bitmask of changed indices; store float32.
  • Metadata map in header has overhead. Solution: promote common keys to
    typed fields and keep a small extra map.

Scope of this PR

  • Introduce a new proto file with V2 compact messages. Do not modify tag
    numbers of existing messages. Keep existing telemetry intact.
  • Provide EnvelopeV2 which carries TelemetryHeaderV2 and
    LobbySessionStateFrameV2.
  • Compact representations for session state, teams, players, stats, and
    bones.
  • No runtime integration changes in rtapi yet; producers/consumers can
    adopt V2 progressively.

Acceptance criteria

  • New proto file compiles with protoc (no build breakage in repo).
  • Messages include documentation for migration.
  • Wire compatibility preserved: existing telemetry remains untouched.

Proposed changes (new file)

syntax = "proto3";
package telemetry.v1;

import "google/protobuf/timestamp.proto"; // kept for possible interop

option csharp_namespace = "Nevr.Telemetry.Protobuf";
option go_package = "github.com/echotools/nevr-common/v4/gen/go/telemetry/v1;telemetry";
option java_multiple_files = true;
option java_outer_classname = "NevrTelemetryCompact";
option java_package = "com.echotools.nevr.telemetry.v1";

// EnvelopeV2 wraps compact telemetry messages for streaming.
message EnvelopeV2 {
  oneof message {
    TelemetryHeaderV2 header = 1;
    LobbySessionStateFrameV2 frame = 2;
  }
}

// Compact header: base time + typed metadata; small extra map
message TelemetryHeaderV2 {
  string capture_id = 1;
  uint64 base_time_unix_nanos = 2;   // replaces created_at
  uint32 timestep_nanos = 3;         // fixed step (optional)
  uint64 server_id = 4;              // promoted metadata
  string mode = 5;                   // promoted metadata
  string level = 6;                  // promoted metadata
  map<string,string> extra = 7;      // remaining metadata
  SessionStatic static = 8;          // session strings that rarely change
}

message SessionStatic {
  string session_id = 1;
  string map_name = 2;
  MatchType match_type = 3;
  string client_name = 4;
}

enum MatchType {
  MATCH_TYPE_UNSPECIFIED = 0;
  MATCH_TYPE_ARENA_PUBLIC = 1;
  MATCH_TYPE_ARENA_PRIVATE = 2;
  MATCH_TYPE_COMBAT_PUBLIC = 3;
  MATCH_TYPE_COMBAT_PRIVATE = 4;
}

// Compact frame: delta time + compact session + compact bones
message LobbySessionStateFrameV2 {
  uint32 frame_index = 1;
  // varint delta from base_time; omit when derivable from frame_index
  sint32 delta_nanos = 2;
  repeated LobbySessionEvent events = 3; // reuse existing event types
  CompactSessionState session = 4;
  CompactPlayerBones player_bones = 5;
}

// Compact session replaces repeated double with float, strings with enums
message CompactSessionState {
  // Disc
  repeated float disc_position = 1;  // len=3, packed
  repeated float disc_forward = 2;   // len=3, packed
  repeated float disc_left = 3;      // len=3, packed
  repeated float disc_up = 4;        // len=3, packed
  repeated float disc_velocity = 5;  // len=3, packed

  // Teams
  repeated CompactTeam teams = 10;

  // Pause
  PauseStateEnum pause_state = 20;
  float paused_timer = 21;
  float unpaused_timer = 22;

  // Game clock + payload
  float game_clock = 30;
  float payload_multiplier = 31;
  float payload_distance = 32;
  float payload_speed = 33;
  int32 payload_checkpoint = 34;
  uint32 blue_points = 35;
  uint32 orange_points = 36;
}

enum PauseStateEnum {
  PAUSE_STATE_UNSPECIFIED = 0;
  PAUSE_STATE_PAUSED = 1;
  PAUSE_STATE_UNPAUSED = 2;
  PAUSE_STATE_PENDING = 3;
}

message CompactTeam {
  uint32 team_id = 1;        // enum or small ID
  bool has_possession = 2;
  CompactTeamStats stats = 3;
  repeated CompactPlayer players = 4;
}

message CompactTeamStats {
  float possession_time = 1;
  uint32 points = 2;
  uint32 saves = 3;
  uint32 goals = 4;
  uint32 stuns = 5;
  uint32 passes = 6;
  uint32 catches = 7;
  uint32 steals = 8;
  uint32 blocks = 9;
  uint32 interceptions = 10;
  uint32 assists = 11;
  uint32 shots_taken = 12;
}

message CompactPlayer {
  ...

</details>



<!-- START COPILOT CODING AGENT SUFFIX -->

*This pull request was created from Copilot chat.*
>

<!-- START COPILOT CODING AGENT TIPS -->
---

✨ Let Copilot coding agent [set things up for you](https://github.com/EchoTools/nevr-common/issues/new?title=✨+Set+up+Copilot+instructions&body=Configure%20instructions%20for%20this%20repository%20as%20documented%20in%20%5BBest%20practices%20for%20Copilot%20coding%20agent%20in%20your%20repository%5D%28https://gh.io/copilot-coding-agent-tips%29%2E%0A%0A%3COnboard%20this%20repo%3E&assignees=copilot) — coding agent works faster and does higher quality work when set up for your repo.

Copilot AI and others added 4 commits December 20, 2025 08:06
Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>
Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>
Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>
Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>
Copilot AI changed the title [WIP] Add compact telemetry schema V2 for storage efficiency Add compact telemetry schema V2 for storage efficiency Dec 20, 2025
Copilot AI requested a review from thesprockee December 20, 2025 08:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants