feat: Implementing parallelization for unzipping files #1114
+9
−3
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Vincy1230
approved these changes
Feb 11, 2026
Vincy1230
left a comment
Vincy1230
left a comment
There was a problem hiding this comment.
I noticed the same issue as you - some servers may require up to ten or more hours for the initial download stage when only utilizing one CPU core.
Deeply grateful for your pull request.
Vincy1230
reviewed
Feb 11, 2026
Vincy1230
left a comment
Vincy1230
left a comment
There was a problem hiding this comment.
I noticed that there is still room for optimization in your changes: you added parallel to enable gzip to run concurrently, but doing so loses the ability of find ... {} + to pass as many files as possible at once, which causes the new script to incur the overhead of starting and stopping gzip frequently.
How about adding a --xargs option as well, so we can get the best of both approaches?
| find "${RAW_DIR}/" -type f -iname "*.gz" -exec gunzip {} + | ||
| if command -v parallel >/dev/null 2>&1 | ||
| then | ||
| find "${RAW_DIR}/" -type f -iname "*.gz" -print0 | parallel -0 -j -1 gunzip |
There was a problem hiding this comment.
Suggested change
| find "${RAW_DIR}/" -type f -iname "*.gz" -print0 | parallel -0 -j -1 gunzip | |
| find "${RAW_DIR}/" -type f -iname "*.gz" -print0 | parallel -0 -j -1 --xargs gunzip |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR implements GNU Parallel-based unzipping for 200,000+ *.cif.gz files in the AlphaFold pipeline.
The Problem:
The current unzipping process is strictly serial, which is extremely slow for large datasets. This limits efficiency and delays downstream processing.
The Fix:
Added a check for GNU Parallel availability.
Automatically detects the number of CPU cores, leaving one core free for I/O-bound tasks and using the remaining cores for parallel unzipping.
Falls back to the existing serial method if GNU Parallel is not installed.
Updated README.md to reflect the new parallelization option and usage instructions.
Fixes: #1075