Skip to content

Refactor: Maximum of 1000 entries per json file.#1

Open
tuxx wants to merge 4 commits intomasterfrom
refactor/handle_many_files
Open

Refactor: Maximum of 1000 entries per json file.#1
tuxx wants to merge 4 commits intomasterfrom
refactor/handle_many_files

Conversation

@tuxx
Copy link
Copy Markdown
Owner

@tuxx tuxx commented Apr 8, 2025

Problem: Loading all file information (potentially hundreds of thousands) into memory at once in both the Go backend and the JavaScript frontend was identified as a major performance bottleneck and potential source of memory issues or crashes.

Fixes

Go backend

  • Modified scanDirectory in localpics.go to process files and write data out incrementally.
  • Instead of one large JSON file per category (e.g., image.json), it now creates numbered page files (e.g., image_1.json, image_2.json, ...) with a fixed number of entries (filesPerPage = 1000).
  • Added generation of a meta.json file containing metadata about each category (total file count, pages, files per page).

Frontend

  • Updated fetchAllStats in static/js/fileLoader.js to first fetch meta.json to get the overall counts and pagination details. Added fallback logic in case meta.json doesn't exist (for small directories or older structures).
  • Introduced loadCategory to initialize the view for a selected category based on meta.json.
  • Introduced loadPage to fetch a specific page's JSON file (e.g., image_2.json).
  • Introduced renderPage to append the files from a newly loaded page to the DOM.
  • Modified the infinite scroll (onscroll) and checkForMoreContent logic to call loadPage for the next page number instead of rendering slices of an existing large array.
  • Removed the old load and render functions that assumed all data was loaded upfront.
  • Updated navigation links in template/index.html to call the new loadCategory function.

Possible issues or things that need addressing:

  • While pagination helps load data incrementally, all loaded items currently stay in the DOM. For extremely large categories, this can still slow down the browser. Implementing DOM virtualization (only rendering visible elements) would significantly improve performance in those cases, although it adds complexity.
  • Complementary to DOM virtualization, we could implement a strategy to remove file data from the data Map in JS for pages that are far scrolled away, freeing up JS memory. This needs careful handling for features like the image modal.

[auto] prettier applied to JavaScript, CSS, and HTML files

Tuxx added 4 commits April 8, 2025 23:46
**Problem**: Loading all file information (potentially hundreds of thousands) into memory at once in both the Go backend and the JavaScript frontend was identified as a major performance bottleneck and potential source of memory issues or crashes.

- Modified scanDirectory in localpics.go to process files and write data out incrementally.
- Instead of one large JSON file per category (e.g., image.json), it now creates numbered page files (e.g., image_1.json, image_2.json, ...) with a fixed number of entries (filesPerPage = 1000).
- Added generation of a meta.json file containing metadata about each category (total file count, pages, files per page).

- Updated fetchAllStats in static/js/fileLoader.js to first fetch meta.json to get the overall counts and pagination details. Added fallback logic in case meta.json doesn't exist (for small directories or older structures).

- Introduced loadCategory to initialize the view for a selected category based on meta.json.
- Introduced loadPage to fetch a specific page's JSON file (e.g., image_2.json).
- Introduced renderPage to append the files from a newly loaded page to the DOM.
- Modified the infinite scroll (onscroll) and checkForMoreContent logic to call loadPage for the next page number instead of rendering slices of an existing large array.
- Removed the old load and render functions that assumed all data was loaded upfront.

- Updated navigation links in template/index.html to call the new loadCategory function.

While pagination helps load data incrementally, all loaded items currently stay in the DOM. For extremely large categories, this can still slow down the browser. Implementing DOM virtualization (only rendering visible elements) would significantly improve performance in those cases, although it adds complexity.

Complementary to DOM virtualization, we could implement a strategy to remove file data from the data Map in JS for pages that are far scrolled away, freeing up JS memory. This needs careful handling for features like the image modal.

[auto] prettier applied to JavaScript, CSS, and HTML files
[auto] prettier applied to JavaScript, CSS, and HTML files
Changed the eventlistener from `canplay` to `canplaythrough` to ensure the browser waits until it has a more substantial buffer before attempting playback, leading to a smoother startup.

[auto] prettier applied to JavaScript, CSS, and HTML files
[auto] prettier applied to JavaScript, CSS, and HTML files
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant