Investigate caching buildkit cache volume

[ingwarsw]

After @kevinschoonover added caching in #2 maybe it would be possible to cache in GHA whole buildkit cache volume.

That should give huge speed boost cause it would work more/less as perfect case of advanced docker caching in earthly.
And speed of GH cache should be really good.

[shepherdjerred]

This would be nice to have. Long-term, I'd like to move over to one of Earthly's paid CI features (Satellites or Earthly CI), but I can't do that until I've convinced my team how great Earthly is :)

[theomessin]

I've made a little proof-of-concept for this using actions/cache here:
https://github.com/theomessin/example/blob/master/.github/workflows/push.yaml

I think it works great!

[shepherdjerred]

That looks great!!

Any idea how this interacts with GitHub's 10GB per-repo cache limit? If the volume is >10GB, will the entire cache be evicted, or just individual layers/files

[theomessin]

@shepherdjerred I'm pretty sure the entire cache would be evicted. However, I think that's okay, it's just a cache after all. I don't mind clearing the cache and re-running all steps every once in a while. Now if your earthly-cache is more than 10GB after running earthly once, I can't help you 😆

[alexcb]

this looks cool!

I wonder if /var/lib/docker/volumes/earthly-cache could be backed without the use of a docker run? I've never looked into if it's possible.

[shepherdjerred]

Now if your earthly-cache is more than 10GB after running earthly once, I can't help you 😆

The cache size is 10GB per repo, not workflow :/

Unfortunately, I think this would lead to a lot of evictions making the cache not very effective. The evictions might be tolerable if it's per layer, but if the entire volume is evicted then I think it'll just be wasted bandwidth, at least in my case.

Great work though! I'm sure this would be useful for many others.

[theomessin]

Unfortunately, I think this would lead to a lot of evictions making the cache not very effective. The evictions might be tolerable if it's per layer, but if the entire volume is evicted then I think it'll just be wasted bandwidth, at least in my case.

I've no idea how the earthly-cache is structured, but nothing's stopping us from saving multiple separate caches. I'll try to find some time to test this 😃