Repo size growth

[nikitakuklev]

A naïve git clone now produces a 1.3GB(!) folder because each commit triggers a refresh of the notebook rendering, changing the base64 encoded images. While mitigations like checkout depth are possible, this growth needs to stop.

Action items:

• Do a git history rewrite to clean up old notebook versions
• Use a separate branch going forward or offload (rendered) notebooks completely from git

[roussel-ryan]

I agree, we will discuss this during the software meeting today and hopefully @pluflou can take a look at migrating the notebooks to a separate repo in xopt-org

[pluflou]

I looked at the largest files in the repo, and at least 1GB of the size was due to assets from the docs site deployment that were being stored in the gh-pages branch. Up until 7 months ago, we were deploying the docs from a branch, but now we use the actions/deploy-pages to do so, which uploads artifacts that are generated during the workflow to GitHub’s internal infrastructure responsible for serving GitHub Pages.

I can confirm that the last time these files were generated was September 2024, before we migrated to the actions deployment workflow. See below for the two largest files (the JSON site map and the ZIP file of all the executed Jupyter Notebooks):

git rev-list --all | while read commit; do
  git ls-tree -r $commit | grep "search_index.json" | while read mode hash filename; do
    echo "$(git show -s --format="%ai" $commit) $hash $filename"
  done
done
2024-09-10 18:24:03 -0700 blob 45f6d3f419125802ea10eafba774d91d7f3bee7d search/search_index.json
2024-09-10 17:55:14 -0700 blob 45f6d3f419125802ea10eafba774d91d7f3bee7d search/search_index.json

and

git rev-list --all | while read commit; do
  git ls-tree -r $commit | grep "search_index.json" | while read mode hash filename; do
    echo "$(git show -s --format="%ai" $commit) $hash $filename"
  done
done
2024-09-10 18:24:03 -0700 blob 45f6d3f419125802ea10eafba774d91d7f3bee7d search/search_index.json
2024-09-10 17:55:14 -0700 blob 45f6d3f419125802ea10eafba774d91d7f3bee7d search/search_index.json

I removed these two files from the commit history of the repo, and it brought it down to 350MB.

My suggestion to resolve this issue is to do the same with all docs deployment related objects (HTML files are the next largest in the repo, along with .ipynb files), and force push a cleaned up commit history (someone with admin privileges will need to do this). Since we are no longer deploying from a branch, and we clear all notebooks before committing, we should not see any more crazy repo size growth after that. This would also avoid us having to move the docs to a separate repo (at least until there’s another reason to do so).

The only drawback is that we’d be rewriting the commit history, but this seems like a necessary step anyway to shave off the repo size.

[roussel-ryan]

Thanks for the detailed report @pluflou , if there are no objections I think we should proceed with this plan, probably when I get back to the US later this month

[roussel-ryan]

@pluflou and I just made an initial pass on reducing the repo size, git clone is now down to ~400MB. We will continue this effort over the next week or so