Naive attempt at keeping crc cache around between runs

[evs-chris]

I needed a good distraction for an hour or two, so I thought I'd take a swing at #25. It did knock a "cold" build of Ractive from ~15 seconds to ~4 on my machine. I feel like I shouldn't make gobble keep .cache directories around, but without a bigger refactor than I'm capable of at this point, I couldn't see a way around it.

Anyways, I figure this is at least a fair proof of concept.

[Rich-Harris]

@evs-chris did you mean to close this? Seems worth following up

[evs-chris]

Yeah. I don't think keeping the .cache around is the best approach. I ran with it for a little while, and it caused some weirdness here and there. After pondering a bit, I think a better approach is:

• keep a map file of relative filename to {crc,target} in each gobble stage directory
• when starting a file transformer run, run through the input and check the files against the map collecting the differences and then process them and building a new map
• run through the output and check for files that should no longer be present and remove them
• save the new map to the stage directory

I think with this, directory transforms could possibly also opt in to crc tracking to make skipping directory transforms possible.

[evs-chris]

I've pondered this a bit more, and when I get some breathing room (probably a month or three), I'll try roughly this:

• Don't automatically clean output directories on start
• Create an observer node that watches source(s) and maintains a (cached to disk) map of input files to CRCs.
• When the observer is triggered, it checks for changes and calls its given function with what has changed (lsr with change flags), its input dir, and options
• The observer function can request an output dir, which turns the observer into a source
  • Either way, the observer can be an input node, it will just make downstream nodes wait for it to finish if it has no output (as far as gobble is concerned, anyways)
• Transformers then become specialized observers that automatically request an output dir and automatically receive an lsr with info on what has changed based on CRC
• File transforms use their one-to-one mapping knowledge to automatically maintain their output dir as a cache, selectively updating and removing files based on the cache info

That would open up an avenue for directory transformers to take advantage of caching if they so desired. It would also open up some possibilities for non-source-transforming code to run with input into the build process. It's also pretty easy to write an observer that just dumps its input files to an external directory for inspection.

The basis for my line of thought here is that the observer is the simplest unit of action rather than the transform.

Does any of that sound at all reasonable?

[Rich-Harris]

It sounds more than reasonable. I ported everything to ES6 recently (because Babel has spoiled me, and my fingers can no longer bear to write code that Node understands natively) but didn't change the architecture in any fundamental way, because I wasn't sure I could see a clear path. But I think this ('the observer is the simplest unit of action') could be that path.

With fast cold builds and non-transforming observer nodes, I think we go from 'broccoli, but with a nicer API and more Windows bugs' to something that genuinely fills an underserved niche.