Created new window pipe for sliding window aggregations#3
Created new window pipe for sliding window aggregations#3itsnotapt wants to merge 16 commits intoendgameinc:masterfrom
Conversation
Pipes now reset their state after PIPE_EOF. Added new window pipe for time windowing streamed events. Fixed bug in walk__time_range reporting str instead of node.
… results. Added documentation for window pipe.
|
@rw-access I removed the tight coupling with host_key associated with pipes, as this seems very environment specific and breaks some common scenarios. I had to "reset" state after the pipe receives I don't have your test data, so I wasn't able to do a full test. My biggest concern is cases where I missed an object that should be immutable e.g. 37ee552. |
|
Other things to consider, the analytic will technically hold onto the buffer longer than necessary, e.g. imagine we get a burst of events in a timespan, then no events. The buffer will retain this spike until another event in the future triggers the buffer cleanup. I thought about adding a garbage cleanup routine, but I feel it's not a significant issue given it's unlikely to cause a relevant memory issue. |
Rolled back host_key removal.
# docs/query-guide/pipes.rst # eql/ast.py # eql/engine.py # eql/parser.py # setup.cfg # tests/test_data.json # tests/test_eql.py # tests/test_python_engine.py
|
@rw-access I've updated this code for EQL 0.7.0 and rolled back the host_key changes to make it easier for you to review. I made a change to the |
� eql/etc/eql.ebnf � eql/etc/test_queries.toml � eql/pipes.py � tests/test_python_engine.py
The concept of pipes doesn't work very well in context of streaming events. In order to allow pipes to function as expected in the streaming scenario, I've added a new "window" pipe that will buffer events into a sliding window, allowing all pipes to function as expected even when streaming.
A typical example of this feature would be the following:
In this example, the query is effective in both streaming and one off batches.