Smeared workflow execution

[danielnorberg]

We would like to smear execution of scheduled workflows over time to achieve a smoother less bursty resource usage.

Would this imply smearing out the actual triggering (i.e. creation of active states) or would it make more sense to smear/throttle the actual execution of the workflows by e.g. some additional "smear-wait" state?

@rouzwawi WDYT?

[rouzwawi]

Without having thought too much about it, I would say it would make more sense to smear/throttle
the dequeueing in Scheduler. The trigger handler should only be doing metadata bookkeeping and not care about actual executions.

From our monitoring, it looks like the bursts appear on trigger boundaries (hours, days), but then also on a 10 minute interval, caused by the fixed 10 minute delay on "missing dependencies". By just throttling the dequeu event, we would solve both of those bursts right?

[danielnorberg]

Throttling/smearing dequeue sounds good to me 👍

[fabriziodemaria]

#70 should have addressed this issue. Close? @danielnorberg

[danielnorberg]

@fabriziodemaria no #70 is about rate limiting pos submission in order to safe-guard the health of the k8s master and docker daemon.

This issue is about smearing execution of workflows and by extension the data processing jobs they execute in order for users like Spotify to achieve smoother resource usage across the data processing platform.