Improve Oracle DB, Trino, Preso support, extend tests#45
Improve Oracle DB, Trino, Preso support, extend tests#45vmatt wants to merge 0 commit intoerezsh:masterfrom
Conversation
sqeleton/databases/oracle.py
Outdated
| # Convert %s style parameters to :1, :2, :3 style for Oracle to support queries built by sqeleton (tbl.create()) | ||
| if sql_code.args: | ||
| # Replace %s with :1, :2, :3, etc. | ||
| oracle_sql = sql_code.code | ||
| for i in range(len(sql_code.args)): | ||
| oracle_sql = oracle_sql.replace('%s', f':{i+1}', 1) | ||
| sql_code = CompiledCode(oracle_sql, sql_code.args, sql_code.type) | ||
|
|
There was a problem hiding this comment.
@erezsh - Would ARG_SYMBOL work here? I'm afraid that because parameters in oracle incrementing, it will not work with it.
There was a problem hiding this comment.
From a quick glance, looks like the options are either to set ARG_SYMBOL = None, and use a custom implementation, or update eval_template() to support an incrementing ARG_SYMBOL, in addition to a constant one.
tests/common.py
Outdated
| TEST_BIGQUERY_CONN_STRING: str = os.environ.get("BIGQUERY_URI") or None | ||
| TEST_REDSHIFT_CONN_STRING: str = os.environ.get("REDSHIFT_URI") or None | ||
| TEST_ORACLE_CONN_STRING: str = None | ||
| TEST_ORACLE_CONN_STRING: str = "oracle://oracle:Password1@localhost/app" or None |
There was a problem hiding this comment.
TODO: Oracle requires client libraries to be installed
|
@vmatt Looks like this would be better as 3 or maybe even 4 separate Pull Requests, so that we can discuss each one in its own thread, and advance them separately. |
Not sure what that means. Doesn't look like the decorator has changed at all. |
2 participants
This PR improves Oracle database support by allowing custom ports to be defined, Presto and Trino timestamp normalization, optimizes Presto container configuration, and enhances test coverage across multiple databases.
Changes
Oracle Database Support
%sto:1, :2, ...format to support sqeleton query builderPresto Improvements
jvm.config:-Xmx16G→-Xmx8Gconfig.properties: Reduced query memory limits (5GB → 2GB, 1GB → 512MB, 2GB → 1GB)-Djdk.attach.allowAttachSelf=trueprestodb/presto:latestimagetimestampinstead oftimestamp(N)as Presto doesn't support precision in cast expressionsTrino Improvements
timestamp(N)precision in castsTest Infrastructure
@test_each_databasedecorator toTestDatabaseandTestMD5test classesTestNormalizeValuetest class with:test_normalize_uuid: Tests UUID normalization across databasestest_normalize_timestamp: Tests timestamp normalization with different precisionstest_normalize_number: Tests numeric value normalization with various precisionsTestThreePartIdsto exclude DuckDB (doesn't support catalog-scoped information_schema queries)TestQueriesto use UTC and handle Presto/Trino/Dremio precision limitationsConfiguration Updates
versionfield from docker-compose.yml (deprecated in newer Docker Compose versions)Testing
All changes have been tested with the updated test suite. The new
@test_each_databasedecorator ensures consistent testing across all supported databases.Breaking Changes
None. All changes are backwards compatible.
TODO / Notes
cx_oracledeprecated package, might worth updating it to oracledb Python package