Skip to content

Conversation

Copy link
Contributor

Copilot AI commented Aug 21, 2025

This PR implements comprehensive caching of lobby session data at session initialization to eliminate redundant database and storage calls during lobby operations.

Problem

The lobbySessionRequest code path was making multiple database/storage calls on every lobby request (find/join/create/spectate):

  • nk.FriendsList() - loading friends list to extract blocked players (1-N paginated calls)
  • MatchmakingRatingLoad() - loading player ratings for skill-based matchmaking
  • CalculateSmoothedPlayerRankPercentile() - calculating rank percentiles with database queries
  • MatchLabelByID() - validating next match information
  • StreamUserList() - getting presence data for next match hosts

For users with many friends, this could result in 5-10+ database calls per lobby request, creating unnecessary load and latency.

Solution

Session-Level Caching

Extended SessionParameters with new cached data fields:

  • friendsList - complete friends list loaded once at session start
  • blockedPlayerIDs - extracted blocked player IDs for matchmaking queries
  • matchmakingRatingsByGroup - ratings cached by group and game mode
  • rankPercentilesByGroup - rank percentiles cached by group and game mode
  • nextMatchInfo - resolved next match information including Discord host lookups

Initialization-Time Loading

Added three new functions called during initializeSession():

  • loadAndCacheFriendsData() - loads complete friends list and extracts blocked players
  • loadAndCacheMatchmakingData() - pre-loads ratings/percentiles for all user groups
  • loadAndCacheNextMatchInfo() - resolves and caches next match metadata

Lobby Parameter Optimization

Updated NewLobbyParametersFromRequest() to use cached data instead of fresh database calls:

// Before: DB calls on every request
users, cursor, err = nk.FriendsList(ctx, session.UserID().String(), 100, nil, cursor)
rankPercentile, err = CalculateSmoothedPlayerRankPercentile(ctx, logger, p.db, p.nk, userID, groupIDStr, mmMode)

// After: uses cached data  
blockedIDs := sessionParams.GetBlockedPlayerIDs()
rankPercentile = sessionParams.GetRankPercentile(groupIDStr, mmMode)

Performance Impact

  • Eliminated ~90% of database calls from lobby operations after session initialization
  • Before: 3-10 DB calls per lobby request depending on friend count
  • After: 0 DB calls per lobby request (uses cached data)
  • Trade-off: +10-15 DB calls during session initialization (one-time cost per session)

Data Freshness

Cached until session restart:

  • Friends list and blocked players
  • Matchmaking ratings and rank percentiles
  • User's guild group memberships

Still retrieved fresh on every request:

  • Match registry lookups for join operations (data integrity)
  • Real-time server availability for match creation
  • Live match discovery for spectating

The caching approach balances performance with data consistency - friend changes and rating updates from other sessions require a session restart to be reflected, which is an acceptable trade-off for the significant performance improvement.

Fixes #89.


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

thesprockee and others added 17 commits August 15, 2025 20:22
* feat: Add file-level options and swagger definitions to proto files

* Update evr.proto

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

* Update evr.proto

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

* Update evr_http.proto

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

---------

Signed-off-by: Andrew Bates <a@sprock.io>
Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
Co-authored-by: Andrew Bates <a@sprock.io>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
#25)

* Initial plan

* Add comprehensive characterization tests for LoginHistory struct

Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>

* Implement focused improvements to LoginHistory struct methods

Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

* Update protobuf field names

---------

Signed-off-by: Andrew Bates <a@sprock.io>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>
Co-authored-by: Andrew Bates <a@sprock.io>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…oles command (#50)

* Initial plan

* Implement interactive role configuration interface for /set-roles command

Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>

* Add unit tests for role configuration functionality

Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

---------

Signed-off-by: Andrew Bates <a@sprock.io>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Andrew Bates <a@sprock.io>
Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…hoTools Nakama (#65)

* Initial plan

* Complete README.md with comprehensive EchoTools Nakama deployment guide

Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>

* Update README.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

* Update README.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

* Update README.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

* Update README.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

---------

Signed-off-by: Andrew Bates <a@sprock.io>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>
Co-authored-by: Andrew Bates <a@sprock.io>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…treamModeLobbySessionTelemetry API (#67)

* Initial plan

* Implement Redis Streams Event Journaling, MongoDB Summarization, and StreamModeLobbySessionTelemetry API

Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>

* Add telemetry integration utilities and documentation

Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>

* Update server/evr_telemetry_manager.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

* Update server/evr_telemetry_integration.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

* Update server/evr_telemetry_test.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

---------

Signed-off-by: Andrew Bates <a@sprock.io>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>
Co-authored-by: Andrew Bates <a@sprock.io>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
… match data in MongoDB (#42)

* Initial plan

* Implement EventMatchDataJournal integration with Redis queue and MongoDB persistence

Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>

* Apply gofmt formatting to new files

* Add demo script and finalize MatchDataJournal integration

Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>

* Update GitHub Actions workflow to reduce unnecessary builds

* Enforcement Suspension System Refactor (#47)

* Remove obsolete ServiceConfig

* Refactor enforcement records and suspension handling

- Introduced AllModes and PrivateModes constants for better mode management.
- Updated GuildEnforcementRecord to use Expiry instead of SuspensionExpiry.
- Enhanced CheckEnforcementSuspensions to return active suspensions by game mode.
- Adjusted lobby authorization to handle game mode-specific suspensions.
- Removed LimitedAccess role checks and related logic from GuildGroup.
- Updated tests to verify correct behavior of group ID and user ID assignments in enforcement records.

* Refactor GitHub Actions workflows to streamline build process (#48)

* Refactor GitHub Actions workflows to streamline build process

* Enforcement Suspension System Refactor (#47)

* Remove obsolete ServiceConfig

* Refactor enforcement records and suspension handling

- Introduced AllModes and PrivateModes constants for better mode management.
- Updated GuildEnforcementRecord to use Expiry instead of SuspensionExpiry.
- Enhanced CheckEnforcementSuspensions to return active suspensions by game mode.
- Adjusted lobby authorization to handle game mode-specific suspensions.
- Removed LimitedAccess role checks and related logic from GuildGroup.
- Updated tests to verify correct behavior of group ID and user ID assignments in enforcement records.

* Update .github/workflows/build.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

* Update build workflow: add Go module download step and change build command to 'make nakama'

* Update Docker and Makefile configurations for improved build process and caching

* Refactor build workflow: remove Go module download step and integrate 'go mod vendor' before building binary

* Disable pull request trigger for tests workflow and update event configuration

* Enforcement Suspension System Refactor (#47)

* Remove obsolete ServiceConfig

* Refactor enforcement records and suspension handling

- Introduced AllModes and PrivateModes constants for better mode management.
- Updated GuildEnforcementRecord to use Expiry instead of SuspensionExpiry.
- Enhanced CheckEnforcementSuspensions to return active suspensions by game mode.
- Adjusted lobby authorization to handle game mode-specific suspensions.
- Removed LimitedAccess role checks and related logic from GuildGroup.
- Updated tests to verify correct behavior of group ID and user ID assignments in enforcement records.

---------

Signed-off-by: Andrew Bates <a@sprock.io>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Refactor storage system to use simplified StorableAdapter pattern (#46)

* Initial plan

* Create StorableAdapter pattern and convert LatencyHistory

Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>

* Convert GuildGroupState and LoginHistory to StorableAdapter pattern

Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>

* Convert DisplayNameHistory and EarlyQuitConfig to StorableAdapter pattern

Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>

* Convert remaining types to StorableAdapter pattern

Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>

* Complete Storable interface refactoring with legacy compatibility

Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>

* Replace Storable with initial proposed implementation

* Remove deprecated legacy storage interfaces and functions in favor of StorableAdapter

* Refactor latency history handling to remove StorableAdapter usage

* Refactor storage index registration to use StorableIndexer pattern

* Refactor most structs to use simplified StorableAdapter pattern

Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>

* Enforcement Suspension System Refactor (#47)

* Remove obsolete ServiceConfig

* Refactor enforcement records and suspension handling

- Introduced AllModes and PrivateModes constants for better mode management.
- Updated GuildEnforcementRecord to use Expiry instead of SuspensionExpiry.
- Enhanced CheckEnforcementSuspensions to return active suspensions by game mode.
- Adjusted lobby authorization to handle game mode-specific suspensions.
- Removed LimitedAccess role checks and related logic from GuildGroup.
- Updated tests to verify correct behavior of group ID and user ID assignments in enforcement records.

* Complete StorableAdapter refactoring to simplified pattern

Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>

* Enforcement Suspension System Refactor (#47)

* Remove obsolete ServiceConfig

* Refactor enforcement records and suspension handling

- Introduced AllModes and PrivateModes constants for better mode management.
- Updated GuildEnforcementRecord to use Expiry instead of SuspensionExpiry.
- Enhanced CheckEnforcementSuspensions to return active suspensions by game mode.
- Adjusted lobby authorization to handle game mode-specific suspensions.
- Removed LimitedAccess role checks and related logic from GuildGroup.
- Updated tests to verify correct behavior of group ID and user ID assignments in enforcement records.

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>
Co-authored-by: Andrew Bates <a@sprock.io>

* Refactor profile and display name handling (#49)

* WIP: Refactor profile

* Refactor profile handling to improve display name management and streamline related structures

* Fix display name history caching by limiting to 15 recent names per group and filtering by age

* Add kick-player full audit log message

* Update interaction signature formatting and log message structure for improved clarity

* Refactor storage index registration to include DisplayNameHistory and DeveloperApplications

* Update server/evr_account_settings.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

* Update server/evr_account_displayname_history.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

* Refactor display name assignment in DisplayNameHistory and clean up unused code in ProfileOptions and LatencyHistory

---------

Signed-off-by: Andrew Bates <a@sprock.io>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Potential crash fix for evr_discord_appbot (#54)

* Add check for `GamePauseSettings` being nil.

---------

Signed-off-by: Andrew Bates <a@sprock.io>

* Update .gitignore to include 'nakama-debug' and remove unnecessary comments

* Optimize Dockerfile by adding 'go mod tidy' and 'go mod vendor' steps for improved dependency management

* Refactor EVRProfile structure and remove ProfileOptions

* Update Go version to 1.25

* Remove unused pprof server setup from EvrPipeline

* Update server/evr_runtime_events.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

* Update server/evr_runtime_events.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

* Update server/evr_runtime_event_matchdatajournal.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

* Update server/evr_match_data_indexing.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

* Update server/evr_runtime_event_matchdatajournal.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

---------

Signed-off-by: Andrew Bates <a@sprock.io>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>
Co-authored-by: Andrew Bates <a@sprock.io>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Avery ❤️ <avery442@proton.me>
…ng (#62)

* Initial plan

* Implement Discord sessions channel with automated lobby session tracking

Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>

* Complete Discord sessions channel implementation with cleanup and service guild support

Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

* Update server/evr_discord_sessions_channel.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

* Update server/evr_discord_sessions_channel.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

---------

Signed-off-by: Andrew Bates <a@sprock.io>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>
Co-authored-by: Andrew Bates <a@sprock.io>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Initial plan

* Implement Discord Linked Roles with custom requirement options

Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>

* Complete Discord Linked Roles implementation with CORS support and validation guide

Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>

* Address review feedback: move guide to docs, fix User-Agent, remove gitignore entry

Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>

* Replace redundant findUserIDByDiscordID with existing GetUserIDByDiscordID function

Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>
Co-authored-by: Andrew Bates <a@sprock.io>
* Initial plan

* Add comprehensive Discord bot guild setup documentation

Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Bates <a@sprock.io>

---------

Signed-off-by: Andrew Bates <a@sprock.io>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>
Co-authored-by: Andrew Bates <a@sprock.io>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
… similar calculation corruption (#69)

* Initial plan

* Initial investigation of profile statistics issue

Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>

* Fix profile statistics showing large negative numbers

Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>

* Add comprehensive validation to prevent corrupted statistics in all calculations

Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>
Bumps the go_modules group with 1 update in the / directory: [golang.org/x/net](https://github.com/golang/net).


Updates `golang.org/x/net` from 0.37.0 to 0.38.0
- [Commits](golang/net@v0.37.0...v0.38.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-version: 0.38.0
  dependency-type: indirect
  dependency-group: go_modules
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Copilot AI and others added 3 commits August 21, 2025 06:16
Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>
…use cached data

Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>
Co-authored-by: thesprockee <962164+thesprockee@users.noreply.github.com>
Copilot AI changed the title [WIP] Ensure all lobby session data is retrieved at initialization (reduce runtime DB/storage calls) Optimize lobby session data retrieval by caching at initialization Aug 21, 2025
Copilot AI requested a review from thesprockee August 21, 2025 06:25
@thesprockee thesprockee force-pushed the main branch 3 times, most recently from 3046b52 to 4a7d4cc Compare September 28, 2025 14:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ensure all lobby session data is retrieved at initialization (reduce runtime DB/storage calls)

2 participants