Open
Conversation
- don't query files torrents are not inserted - use basic types as much as possible - run sqlite in batch instead of window
but use it after computing details, it will filter path with only nul bytes
Contributor
Author
|
I've added support for pgcopy. The code is somewhat more complicated, but performance are nearly doubled. |
Contributor
Author
|
Using pgcopy for file insertion removed the detection of duplicated file that are all empty. I don't know if I can restore that easily, since it's done in a generator. |
Contributor
Author
"empty" is a database without any entries. "full" is the database with all entries existing. This is useful for re-importing entries with only a few changes, e.g. merging divergent data. The benchmark was run with:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
It changes the SQL a lot, but I kept all existing checks
Here are my results, ran on AMD Ryzen Pro 4750G + 16GB RAM on SSD, with a 29 millions magnetico database. Arguments list:
--source-name magnetico --add-files --add-files-limit 200 --insert-contentSo it's a huge win for pre-filled postgresql target, since this will never query.
Please double check that I didn't miss something.
I didn't added #10 in this, but I ran my tests with it.