Open
Conversation
migration.py: Updated to check if target directory and possibly target files already exists.
…king if the user is properly signed into Steam.
README: Updated to include CSV processing requirements
…d. Login ID us used for session data, and if this script is run on a PC that has Steam running, then they will be in conflict and cause the earlier session to log out.
… client is connected.
Requirements: Set a specific commit for the steam library
README: Explains need to uninstall the steam python library and installing from requirement.txt to ensure proper version. Added instructions for Login ID and why it's important. login: added logic to change lastuser.txt based on who signed in.
…s properly set to None. Moved the creation of directories for each depot to after the downloading of a manifest. This is to ensure that if a manifest is unavailable, such as access denied, the depot directory wiill not be created. Also moved depot key acquisition to happen only after a manifest is downloaded, again as to ensure if the manifest can't be downloaded, the depot key shouldn't be downloaded or created either, otherwise it might result in a 0-length file.
…t is a 0-length file, it counts as a failed download.
…fic to app, not depot, so has to be saved in a different directory. Migration: Added check for moving depot key files.
…to a CSV file. Added in a catch for exceptions and keyboard interruptions to gracefully close everything out. Add in a bulk pack function for chunkstore to manage directly handling file lists.
depot_archiver: refactored chunkstore creation. depot_extractor: fixed dry_run. depot_validator: updated chunk validation and refactored validation with chunkstores.
…stead of appending to it. Made get_chunk multithread safe.
…tions. depot_validator: restored printing bad chunks from chunkstore.
…. Chunkstore is slower. chunkstore: made reading from CSD cross-thread capable.
…gle sql query for chunks to combine into a file. Chunkstore processing time for file extraction is now fast.
…ltiple threads reading from the same chunkstore file. Added comments about what functions do, mostly sorted functions by name. Multithreaded Unpacking of chunks. Added thread tracker for thread_local sql connections, as to properly close any multithreaded sql connections. diff_manifest: Fixed depot key variable, updated to new location of depot key. pack_sis: Updated to pass chunk paths to chunkstore, allowing the bulk of the processing to be done from chunkstore itself. Added option to only pack files belonging to the manifest, if one is included.
…ebug your chunkstores. unpack_sis: Updated to use chunkstore's built in processing. All chunks are extracted. generate_json_from_csm: refactored to use chunkstore's exporter. pack_sis: attempt to fix writing sku.sis data.
… provided. depot_validator: Just added something to say if no bad files are found. unpack_sis: added depot id option to pass, if one wants to.
repack_sis: Shrunk file get_appinfo: updated to get app info of owned apps. Commented old method as it always failed, but might work with some changes.
…uch as private depots for things like DLC. Added logic to bypass.
…e-compressed by Valve for smaller storage. This is due to some newer chunks using ZSTD instead of LZMA.
…anged new file detection to leave out the name of the first file not found in the chunkstore.
pack_sis: removed unused argument unpack_sis: fixed chunkstore location processing.
… check of Branch is excluded. This ensures read consistency. CSV headers must be "AppID","DepotID","ManifestID", and can include "Branch". Any additonal headers will be ignored.
This reverts commit 95f3c65.
…dentification of ZSTD files.
…for normal usage.
Could have come in handy when Gmod lost all those Mario items.
Depot_archiver: Reads the outputted file created from workshop_scraper to bulk download workshop items,
requirements.txt: added requirements for workshop_scraper to work.
…lters due to limit in how many items can be scraped per filter. README: Added workshop_scraper.
…least 11 available threads. 10 for scraping and 1 for OS, for example.
…ilters, should items created in a specific range count above 50010 (1667 pages).
…UGC) workshop items, and added in processors for examining and extracting the legacy content. Currently only tested with Garry's Mod Workshop items.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changed how Migration handles existing folders and files. Previously if the target folder existed, it would error. So updated to account for the possibility that target may already exists, and went as far as checked if any of the target files also already exist. So regardless if target folders exist or not, migration will succeed.