Setup for impure code

[caderek]

@kirbysayshi suggested:

As I'm looking through the other cases for setup, the common pattern seems to be that the setup is run once, completely outside of benchmark.js (this makes sense since it doesn't support async setup). But there could be a deeper concern here too: only running the setup function once could result in inconsistent start state for the clocked function:

add('array truncation', async () => {
  const arr = (new Array(1000)).fill(true);
  return async () => {
    // on the first run, this will actually truncate the array.
    // on subsequent runs, the length would already be zero since the setup state is shared.
    arr.length = 0;
  }
})

I open this issue to reconsider this behavior for impure functions that can mutate setup.

[ncovercash]

I agree; I'd really like more fine-grained control over the lifecycle. Maybe this could be implemented like:

benny.add('reconcile only', (ctx) => {
  const data = // something to create a data set
  ctx.start();
  reconcile(data);
  ctx.end();
});

With something like this, we could control specifically which part of the function is counted in the timing. This would be particularly useful for me, since I'm looking for something that can separate out the steps of a two-stage upload process and tell me what amount of time is spent on each, but currently I have no way to do that. Perhaps something like:

benny.add('reconcile only', async (ctx) => {
  await ctx.step(() => doPart1());
  await ctx.step(() => doPart2());
});

With this, the final result could be the total time of the entire function, but there'd be an additional property containing statistics for each step individually.