Skip to content

Fixed small bugs in Solr import#11

Open
buremba wants to merge 6 commits intolalinsky:masterfrom
buremba:master
Open

Fixed small bugs in Solr import#11
buremba wants to merge 6 commits intolalinsky:masterfrom
buremba:master

Conversation

@buremba
Copy link
Copy Markdown

@buremba buremba commented Jun 2, 2012

Fixed bugs caused by schema change.
And also fixed memory overloading problem in solr recording data. Recordings table has more than 10 million records and the sql query without LIMIT may cause memory overloading.

@lalinsky
Copy link
Copy Markdown
Owner

lalinsky commented Oct 8, 2012

Sorry for not merging this earlier. I just noticed it, when I was looking for making the Solr export working again. I'm not sure I understand the memory problem. What is allocating that much of memory? The Python code is all based on iterators, so it should only process one row at a time.

@buremba
Copy link
Copy Markdown
Author

buremba commented Oct 8, 2012

The memory problem is not caused by Python. SELECT r.gid,rn.name, an.name FROM recording r JOIN track_name rn ON r.name = rn.id JOIN artist_credit ac ON r.artist_credit = ac.id OIN artist_name an ON ac.name = an.id returns more than 10m records and Postgresql consumes ~1.5gb memory for this query. If the server does not have enough memory to run this query, Postgresql kills the process that created by Postgresql for the query and throws an exception. It means the query returns an empty result and if it occurs Python can't create index file for recordings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants