Open
Conversation
added 4 commits
April 8, 2025 23:46
**Problem**: Loading all file information (potentially hundreds of thousands) into memory at once in both the Go backend and the JavaScript frontend was identified as a major performance bottleneck and potential source of memory issues or crashes. - Modified scanDirectory in localpics.go to process files and write data out incrementally. - Instead of one large JSON file per category (e.g., image.json), it now creates numbered page files (e.g., image_1.json, image_2.json, ...) with a fixed number of entries (filesPerPage = 1000). - Added generation of a meta.json file containing metadata about each category (total file count, pages, files per page). - Updated fetchAllStats in static/js/fileLoader.js to first fetch meta.json to get the overall counts and pagination details. Added fallback logic in case meta.json doesn't exist (for small directories or older structures). - Introduced loadCategory to initialize the view for a selected category based on meta.json. - Introduced loadPage to fetch a specific page's JSON file (e.g., image_2.json). - Introduced renderPage to append the files from a newly loaded page to the DOM. - Modified the infinite scroll (onscroll) and checkForMoreContent logic to call loadPage for the next page number instead of rendering slices of an existing large array. - Removed the old load and render functions that assumed all data was loaded upfront. - Updated navigation links in template/index.html to call the new loadCategory function. While pagination helps load data incrementally, all loaded items currently stay in the DOM. For extremely large categories, this can still slow down the browser. Implementing DOM virtualization (only rendering visible elements) would significantly improve performance in those cases, although it adds complexity. Complementary to DOM virtualization, we could implement a strategy to remove file data from the data Map in JS for pages that are far scrolled away, freeing up JS memory. This needs careful handling for features like the image modal. [auto] prettier applied to JavaScript, CSS, and HTML files
[auto] prettier applied to JavaScript, CSS, and HTML files
Changed the eventlistener from `canplay` to `canplaythrough` to ensure the browser waits until it has a more substantial buffer before attempting playback, leading to a smoother startup. [auto] prettier applied to JavaScript, CSS, and HTML files
[auto] prettier applied to JavaScript, CSS, and HTML files
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem: Loading all file information (potentially hundreds of thousands) into memory at once in both the Go backend and the JavaScript frontend was identified as a major performance bottleneck and potential source of memory issues or crashes.
Fixes
Go backend
Frontend
Possible issues or things that need addressing:
[auto] prettier applied to JavaScript, CSS, and HTML files