Video relevance is not getting updated for some videos #361
Description
The problem
Currently VideoRelevanceManager takes 2 parameters called fullUpdateLoopTime and scheduledUpdateLoopTime, but it initializes the same update loop using both of those intervals.
Probably the original idea was to have shorter interval (scheduledUpdateLoopTime) updates for videos that were recently viewed / commented on etc. and a longer interval (fullUpdateLoopTime) to update the relevance of all videos, as it changes with the passage of time (since video's age is one of the factors).
This 2nd kind of update is currently not being executed, so videos that have not received any new views, comments or reactions recently will have a stale relevance (higher than they should have).
The solution
The simplest solution would be to just run full update in fullUpdateLoopTime intervals, but this is a very heavy operation which could block other important updates from happening in the meantime.
It would probably be a better idea to split the full update into smaller chunks that are executed at smaller intervals.
For example, we could imagine VideoRelevanceManager.channelsToUpdate as a priority queue.
Every 24 hours we add all existing channels to this queue with priority=0.
Every time a channel's video is affected by a change in views / comments / reactions we add this channel to the queue with priority=1
We can then run updates at a fixed interval (for example, every 30 minutes), but limit the number of channels we retrieve from the queue to perform a single update (to 1000 for example).
It's important to implement this solution in a way that avoids causing very long locks or even deadlocks on the database, considering that orion processor and graphql server may try to run other updates at the same time.