Skip to content

Replace NullPool with QueuePool and consolidate database engines#621

Open
Dagonite wants to merge 2 commits intomainfrom
use-queuepool
Open

Replace NullPool with QueuePool and consolidate database engines#621
Dagonite wants to merge 2 commits intomainfrom
use-queuepool

Conversation

@Dagonite
Copy link
Collaborator

@Dagonite Dagonite commented Mar 10, 2026

Closes None.

Issue

Both fia_api/core/session.py and fia_api/core/repositories.py create independent SQLAlchemy engines with NullPool, meaning every database query opens a new TCP connection and closes it immediately. This is a big performance bottleneck in the API. Additionally, having two separate engines with identical connection strings is redundant.

Fix

Replaced NullPool with QueuePool in session.py, enabling connection reuse across requests instead of creating a new TCP connection per query.

Removed the duplicate ENGINE and SESSION from repositories.py and consolidate into session.py as the single source of truth

Moved ensure_db_connection from repositories.py to session.py alongside the engine it depends on.

Dagonite and others added 2 commits March 10, 2026 11:28
Both session.py and repositories.py were creating independent SQLAlchemy engines with NullPool, meaning every database query opened a new TCP connection and tore it down immediately. This replaces NullPool with QueuePool and consolidates the duplicate engine into session.py as the single source of truth.
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves database performance and consistency by consolidating SQLAlchemy engine/session creation into fia_api/core/session.py and enabling connection pooling so queries can reuse TCP connections instead of reconnecting for every operation.

Changes:

  • Switch engine configuration to use pooled connections (instead of NullPool) and add pool health/recycle settings.
  • Remove duplicate engine/session definitions from repositories.py and move ensure_db_connection into session.py.
  • Update routers and tests to import ENGINE/SESSION/ensure_db_connection from fia_api.core.session.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
fia_api/core/session.py Centralizes engine/session setup, adds pooling config, and hosts ensure_db_connection.
fia_api/core/repositories.py Removes duplicated engine/session and DB connectivity helper, leaving repository logic.
fia_api/routers/health.py Updates imports to use consolidated DB session utilities.
test/utils.py Updates test DB setup imports to the new session module.
test/e2e/conftest.py Updates imports to the new session module for fixtures.
test/e2e/test_autoreduction_routes.py Updates imports to use SESSION from the new location.
test/core/test_repositories_integration.py Updates imports and mock patch target for moved ensure_db_connection.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +34 to +35
with db as session:
session.execute(select(1))
Copy link

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ensure_db_connection uses with db as session: on a Session instance that is typically owned/closed by the caller (e.g., FastAPI dependency). Entering the context manager will close the passed-in session on exit, which is surprising and can cause the caller to operate on a closed session (and also results in double-close with get_db_session()). Prefer executing directly on the provided Session without wrapping it in a context manager, or have this helper create/own its own Session instead.

Suggested change
with db as session:
session.execute(select(1))
db.execute(select(1))

Copilot uses AI. Check for mistakes.
Comment on lines 12 to 18
ENGINE = create_engine(
DB_URL,
poolclass=NullPool,
pool_size=5,
max_overflow=10,
pool_pre_ping=True,
pool_recycle=300,
)
Copy link

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The connection pool sizing/tuning is hard-coded (pool_size/max_overflow/pool_recycle). This can cause unexpected connection counts in production (e.g., per-worker pools) and makes it hard to tune without a redeploy. Consider sourcing these values from environment variables (with sensible defaults) so deployments can adjust based on DB limits and worker count.

Copilot uses AI. Check for mistakes.
)

SessionLocal = sessionmaker(bind=ENGINE, autoflush=False, autocommit=False)
SESSION = sessionmaker(ENGINE)
Copy link

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are now two session factories bound to the same ENGINE (SessionLocal and SESSION) with different configuration (SessionLocal sets autoflush/autocommit, SESSION does not). This duplication can lead to inconsistent behavior depending on which one is imported. Consider exporting a single configured sessionmaker (or making SESSION an alias of SessionLocal) and using it consistently.

Suggested change
SESSION = sessionmaker(ENGINE)
SESSION = SessionLocal

Copilot uses AI. Check for mistakes.
@codecov
Copy link

codecov bot commented Mar 10, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.36%. Comparing base (dc11fea) to head (5eb95e7).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #621      +/-   ##
==========================================
- Coverage   96.37%   96.36%   -0.02%     
==========================================
  Files          48       48              
  Lines        1961     1955       -6     
==========================================
- Hits         1890     1884       -6     
  Misses         71       71              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants