CircularQueue with a custom index

The most important parameter for CircularQueue is definitly the maximum size.

const circularQueue = new CircularQueue<number>({
	maxSize: 4,
});

I have a use case, where I want to make it circular on two indexes !! Let me explain why this could be an amazing addition.

My case:

I have a stream of data that I want to limit analysis on say to 10k elements for processing time and the memory not to blow. (first index)
I also would like to take time into consideration, I want to limit also the size of the queue by the distance between minIndex and maxIndex where this new index is simply epoch. (my case is to only keep elements added since last week, earlier elements are not important at all).

I hope I could make it clear, and I would like to thank you Toreda.

These are simply two types of indexes:

1. maxSize is the size of the queue. (grows on each insert by +1)
2. maxElapsedTime is time delta between the last and earliest element in queue (grows on each insert by maxIndex.time - minIndex.time , if it hits maxElapsedTime, earliest element is removed, and maxElapsedTime is recalculated recursively until maxIndex.time - minIndex.time < maxElapsedTime)

[mbrich]

Good comments here. The use case makes sense and I can see other uses benefiting from it, given time-based logic is a pretty common need in any sort of time or TTL based cache/buffer.

I'm thinking about possible implementations and wondering whether we want to implement the actual time-based logic, or whether it would be better to simply let the caller provide their own validator function which is called each time an item is added. We use this type of approach in @toreda/strong-types and it works pretty well. The data structure remains generic while users can customize the validation logic without having to spend time extending the class.

To make sure I understand, let me repeat what I think your ask is:

• add element overwrites the earliest item when maxIndex - minIndex exceeds a target value when buffer is full.
• add element overwrites the earliest item when maxIndex.time - minIndex.time exceeds some value.

In both cases, I'm assuming you only want to overwrite when the buffer is full? Do the cases apply when the buffer is not yet at max capacity?

Did I get those two cases right?

Hello @mbrich
Thank you so much for your consideration.

Indeed this is exactly TTL. Except that I verify on insert in my case.

We can imagine ensuring this constraint onRead, onWrite, or another event (continuous CRON...)

For my case, I have worked my example around the datastrucure, not the best, you could tell:
https://github.com/bacloud14/topK_example/blob/ad39206e8d908ef76c5f1a5c552da3403b2aa814/index.js#L77

I added a date key in items, and purge recursively on insertion for items older that one month. Code is scrambled, I will rework and comment it.

For the context, indeed this is very common use case. For my case:

I'm running topK algorithm to data mine popular searches on the circular queue (topK is supposed to work on endless streams theoretically, but I want something like: top keywords searched in the last month, so I purge earlier items, and rerun topK on the queue each time we are half the way the queue length )