Skip to content

feat: refactor GTFS realtime data storage to use a FeedData struct per feed with granular locking#480

Open
mann-patwa wants to merge 2 commits intoOneBusAway:mainfrom
mann-patwa:per-feed-lock
Open

feat: refactor GTFS realtime data storage to use a FeedData struct per feed with granular locking#480
mann-patwa wants to merge 2 commits intoOneBusAway:mainfrom
mann-patwa:per-feed-lock

Conversation

@mann-patwa
Copy link
Contributor

Fixes #479

Description

This PR refactors the GTFS Manager to use fine-grained, per-feed locking for realtime data ingestion.

Previously, a global reader/writer lock (realTimeMutex) was held for the entire duration of a feed update. In environments with multiple realtime feeds updating concurrently, this caused significant lock contention. A slower feed update would block all other feeds from writing and could potentially block readers trying to access the current state.

This change introduces a localized FeedData struct with its own sync.RWMutex. This allows individual feeds to process and store their realtime data independently and concurrently. The global lock is now only acquired briefly during the final aggregation step (buildMergedRealtime), where pointers to the updated feed data are merged into the global system view.


Changes Proposed

1. Introduced FeedData Struct

Replaced the separate:

  • feedTrips
  • feedVehicles
  • feedAlerts
  • feedVehicleLastSeen

With a single:

map[string]*FeedData

This bundles a feed's realtime state into a cohesive structure, improving maintainability and encapsulation.


2. Added Per-Feed Locks

  • Added a sync.RWMutex inside FeedData.
  • Data extraction, parsing, transformation, and assignment for a specific feed now hold only that feed’s lock.
  • Independent feeds no longer block each other during ingestion.

This enables true parallel processing of realtime feeds.


3. Map Synchronization

  • Introduced a feedMapMutex.
  • Ensures safe read/write access to the top-level map[string]*FeedData.
  • Allows new feeds to be registered safely without race conditions.

This guarantees thread-safe feed lifecycle management.


4. Optimized Merging (buildMergedRealtime)

a) Merge Serialization

  • Introduced a mergeMutex to serialize concurrent merge operations.
  • Prevents overlapping global aggregation steps.

b) Feed-Level Locking During Merge

  • Iterates over sorted feeds.
  • Acquires each feed’s RWMutex only long enough to copy its data slices.
  • Releases immediately after copying.

This ensures minimal blocking at the feed level.


5. Updated Tests

  • Refactored existing unit tests and benchmarks to align with the new FeedData architecture.
  • Updated mocks and test setup where necessary to support per-feed locking.

6. Added Test Coverage

New test cases were added to validate:

  • Retaining previous realtime data on HTTP failures.
  • Correctly handling empty payload responses.
  • Proper expiration of stale vehicles based on last-seen timestamps.
  • Safe concurrent execution of feed updates.

@mann-patwa mann-patwa force-pushed the per-feed-lock branch 2 times, most recently from d7751eb to 00d8539 Compare February 25, 2026 15:51
@mann-patwa mann-patwa marked this pull request as draft February 25, 2026 15:53
Comment on lines +253 to +261
manager.feedMapMutex.Lock()
feed := manager.feedData[feedID]
if feed == nil {
feed = &FeedData{
VehicleLastSeen: make(map[string]time.Time),
}
manager.feedData[feedID] = feed
}
manager.feedMapMutex.Unlock()
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we acquire the feedMapMutex to take a snapshot of the feedData and make new FeedData struct if this is a new feed

@mann-patwa mann-patwa marked this pull request as ready for review February 27, 2026 04:35
@mann-patwa
Copy link
Contributor Author

@aaronbrethorst , need a review!

@mann-patwa
Copy link
Contributor Author

@aaronbrethorst would love a review!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement Per-Feed Locking for GTFS Realtime Updates

1 participant