You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Fixed bugs caused by schema change.
And also fixed memory overloading problem in solr recording data. Recordings table has more than 10 million records and the sql query without LIMIT may cause memory overloading.
Sorry for not merging this earlier. I just noticed it, when I was looking for making the Solr export working again. I'm not sure I understand the memory problem. What is allocating that much of memory? The Python code is all based on iterators, so it should only process one row at a time.
The memory problem is not caused by Python. SELECT r.gid,rn.name, an.name FROM recording r JOIN track_name rn ON r.name = rn.id JOIN artist_credit ac ON r.artist_credit = ac.id OIN artist_name an ON ac.name = an.id returns more than 10m records and Postgresql consumes ~1.5gb memory for this query. If the server does not have enough memory to run this query, Postgresql kills the process that created by Postgresql for the query and throws an exception. It means the query returns an empty result and if it occurs Python can't create index file for recordings.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixed bugs caused by schema change.
And also fixed memory overloading problem in solr recording data. Recordings table has more than 10 million records and the sql query without LIMIT may cause memory overloading.