statemachine: Error messages should release (optionally)? #132
Description
Problem
There are a number of circumstances where a task cannot be executed on a node that has claimed it:
- A new task type is being added via a rolling deploy and a node without the necessary code has claimed the a new task type.
- A task cannot complete due to a node-specific condition: eg a node has insufficient diskspace for a task to use as scratch.
Possible Solution 1: Errors always release
Seems like the worst case scenario to Error messages always causing releases is extra broker load. Probably worth it.
Possible Solution 2: Make Release metadata on a Message
Allow any message to cause a release via a Release bool field on the statemachine.Message struct.
This might actually simplify other code... not sure.