Database connections not released until failure. #91
Description
I can only perform 4 Full updates until read exception. I query my database(oracle or postgres) and connections are not released.
After Full import 1: 25 connections
After Full import 2: 50 connections
After Full import 3: 73 connections
After Full import 4: 90 connections
Exception about reading, that is no doubt related to lack of cleanup of database connections
When I shut down solr the connections are cleared. Starting solr and performing 4 full updates
produces the same connection count increase until failure.
In contrast during the full import while running 8.4.1 the connection count goes up to 25 and
then when the status of full import is complete the connections goes back to zero.
This exception is just an artifact of running out of connections I assume. Re starting solr allows full imports again and the query is not the problem.
This is the exception while using the oracle database.
at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:69)
org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to execute query: select ac.asset_id, to_char(modify_timestamp, 'yyyy-MM-dd hh24:mi:ss') MODIFY_TIMESTAMP, to_char(catalog_timestamp , 'yyyy-MM-dd hh24:mi:ss') CATALOG_TIMESTAMP, ac.client_id, asset_type_id, dig_format_id, deletion_status, asset_name, asset_descr, ludf.code, security_level_id, lusl.code LU_SECURITY_LEVEL, lupf.code LU_PHYS_FORMAT, luat.code LU_ASSET_TYPE, CASE is_physical WHEN 0 THEN 'Digital' ELSE 'Physical' END IS_DIGITAL FROM asset_core ac LEFT OUTER JOIN lu_dig_format ludf ON ac.dig_format_id=ludf.id LEFT OUTER JOIN lu_security_level lusl ON ac.security_level_id=lusl.id LEFT OUTER JOIN lu_phys_format lupf ON ac.phys_format_id=lupf.id LEFT OUTER JOIN lu_asset_type luat ON ac.asset_type_id=luat.id WHERE ac.deletion_status is null AND ac.client_id=11 AND ac.asset_type_id=1 Processing Document # 1
at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:69) ~[?:?]
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:326) ~[?:?]
at org.apache.solr.handler.dataimport.JdbcDataSource.createResultSetIterator(JdbcDataSource.java:287) ~[?:?]
at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:282) ~[?:?]
at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:51) ~[?:?]
at org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:59) ~[?:?]
at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73) ~[?:?]
at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:267) ~[?:?]
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:494) ~[?:?]
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:433) ~[?:?]
at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:348) ~[?:?]
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:236) ~[?:?]
at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:424) ~[?:?]
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483) ~[?:?]
at org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:466) ~[?:?]
at java.base/java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.sql.SQLRecoverableException: IO Error: Got minus one from a read call
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:854) ~[?:?]
at oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:793) ~[?:?]
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:57) ~[?:?]
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:747) ~[?:?]
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:562) ~[?:?]
at org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:191) ~[?:?]
at org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:171) ~[?:?]
at org.apache.solr.handler.dataimport.JdbcDataSource.getConnection(JdbcDataSource.java:527) ~[?:?]
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:316) ~[?:?]
... 14 more
Caused by: oracle.net.ns.NetException: Got minus one from a read call
at oracle.net.ns.NSProtocolNIO.doSocketRead(NSProtocolNIO.java:557) ~[?:?]
at oracle.net.ns.NIOPacket.readNIOPacket(NIOPacket.java:403) ~[?:?]
at oracle.net.ns.NSProtocolNIO.negotiateConnection(NSProtocolNIO.java:127) ~[?:?]
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:340) ~[?:?]
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1596) ~[?:?]
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:588) ~[?:?]
at oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:793) ~[?:?]
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:57) ~[?:?]
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:747) ~[?:?]
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:562) ~[?:?]
at org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:191) ~[?:?]
at org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:171) ~[?:?]
at org.apache.solr.handler.dataimport.JdbcDataSource.getConnection(JdbcDataSource.java:527) ~[?:?]
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:316) ~[?:?]
... 14 more
This is the exception while using a postgres database.
14:53:40.799, assetCategory=Source]) => org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to execute query: select parent_asset_id FROM dme_db_schema.asset_source_hierarchy WHERE source_relate_type_id=2 AND child_asset_id=475 Processing Document # 1
at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:69)
org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to execute query: select parent_asset_id FROM dme_db_schema.asset_source_hierarchy WHERE source_relate_type_id=2 AND child_asset_id=475 Processing Document # 1
at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:69) ~[?:?]
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:326) ~[?:?]
at org.apache.solr.handler.dataimport.JdbcDataSource.createResultSetIterator(JdbcDataSource.java:287) ~[?:?]
at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:282) ~[?:?]
at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:51) ~[?:?]
at org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:59) ~[?:?]
at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73) ~[?:?]
at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:267) ~[?:?]
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:494) ~[?:?]
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:535) ~[?:?]
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:433) ~[?:?]
at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:348) ~[?:?]
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:236) ~[?:?]
at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:424) ~[?:?]
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483) ~[?:?]
at org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:466) ~[?:?]
at java.base/java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.postgresql.util.PSQLException: FATAL: remaining connection slots are reserved for non-replication superuser connections
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2713) ~[?:?]
at org.postgresql.core.v3.QueryExecutorImpl.readStartupMessages(QueryExecutorImpl.java:2825) ~[?:?]
at org.postgresql.core.v3.QueryExecutorImpl.(QueryExecutorImpl.java:175) ~[?:?]
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:313) ~[?:?]
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:54) ~[?:?]
at org.postgresql.jdbc.PgConnection.(PgConnection.java:263) ~[?:?]
at org.postgresql.Driver.makeConnection(Driver.java:443) ~[?:?]
at org.postgresql.Driver.connect(Driver.java:297) ~[?:?]
at org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:191) ~[?:?]
at org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:171) ~[?:?]
at org.apache.solr.handler.dataimport.JdbcDataSource.getConnection(JdbcDataSource.java:527) ~[?:?]
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:316) ~[?:?]
... 15 more
These errors are being output when solr is stopped:
2025-02-14 09:56:25.011 ERROR (Finalizer) [c: s: r: x: t:] o.a.s.h.d.JdbcDataSource JdbcDataSource was not closed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
2025-02-14 09:56:25.011 ERROR (Finalizer) [c: s: r: x: t:] o.a.s.h.d.JdbcDataSource JdbcDataSource was not closed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
2025-02-14 09:56:25.013 ERROR (Finalizer) [c: s: r: x: t:] o.a.s.h.d.JdbcDataSource JdbcDataSource was not closed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
2025-02-14 09:56:25.015 ERROR (Finalizer) [c: s: r: x: t:] o.a.s.h.d.JdbcDataSource JdbcDataSource was not closed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
2025-02-14 09:56:25.016 ERROR (Finalizer) [c: s: r: x: t:] o.a.s.h.d.JdbcDataSource JdbcDataSource was not closed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
2025-02-14 09:56:25.017 ERROR (Finalizer) [c: s: r: x: t:] o.a.s.h.d.JdbcDataSource JdbcDataSource was not closed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
2025-02-14 09:56:25.018 ERROR (Finalizer) [c: s: r: x: t:] o.a.s.h.d.JdbcDataSource JdbcDataSource was not closed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
2025-02-14 09:56:25.020 ERROR (Finalizer) [c: s: r: x: t:] o.a.s.h.d.JdbcDataSource JdbcDataSource was not closed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!