Relaxed Swarm implementation from Hive

[gposluns]

This adds the relaxed Swarm implementation from the Hive paper.

The config option relaxedOrder causes all tasks to use softPrio, which is implemented so that tasks are ordered by softTs if their timestamps are equal, and all tasks with softPrio get timestamp 0 and softTs = appTs.

The result is that tasks get the same app-level TS that they would otherwise and continue to dequeue in order, but can commit out of order, while retaining atomicity. Aborts can still happen between simultaneous conflicting tasks similarly to TM.

[VictorYing]

@gposluns @markcjeffrey I'm wondering if you guys considered if there should be a guarantee that ROB::removeUntiedTaskInRange() returns soft-prioritized tasks in strictly decreasing softTs order. If, after a spiller (a.k.a. coalescer) removes a task with softTs 42 from ROB, the ROB receives a new task with softTs 100, then can the same spiller can remove the task with softTs 100 next? removeUntiedTaskInRange() is carefully constructed to avoid this in the case of traditional hard timestamps.

If there is no such restriction, I'm not sure the usual rule of giving the requeuer (a.k.a. splitter) a (soft) timestamp equal to the next task it will requeue achieves the goals that it usually does in our normal Swarm hard-prioritization world: traditional Swarm sets requeuer timestamps to ensure that that once a ROB has run out of ordinary application tasks with timestamp <= 42, if a task with timestamp 42 was spilled, then there must be a requeuer with timestamp 42 (or less) that will become the min task in the runQueue soon, that will requeue the spilled timestamp-42 task before the ROB gets too far inverting priorities by dequeuing other ordinary application tasks with timestamp > 42. But if spillers can get softTs tasks in unrestricted order, then it can create a requeuer that will requeue the softTs 100 task first before requeuing the softTs 42 task, and this means, if I'm understanding the current code correctly, the requeuer can itself be assigned softTs 100, meaning the softTs 42 task can become "hidden" in the spill buffer managed by an idle softTs 100 requeuer that won't run until the ROB has finished dequeuing all tasks with softTs < 100... this seems like a problem in that spills can cause soft-prioritized tasks to end up executing far out of order as if they're unordered, greatly reducing work efficiency in the application.

[gposluns]

We didn't consider that while writing this. Reading what you wrote I can see how doing so might reduce priority inversions (though at the cost of making it harder to make space in the queues, and spilling more urgent tasks when we do spill). I will note that other relaxed HW schedulers that spill extra tasks to memory (eg Polygraph) have a similar priority inversion when spilling that you noticed here, so this behaviour is similar to what those systems do. If we want to make spillers go in strictly decreasing softPrio order we can do so, though whether or not this improves performance will depend on how much of an issue this priority inversion is compared to the one created by spilling the lower softTS tasks before the higher ones.

[VictorYing]

Okay cool, thanks for the comparison to Polygraph! I have no experimental results to tell me to what degree any of this is important, but it's a good point that most work on relaxed priority scheduling seems tolerate various inversions and delays.