Fix DuckDB SQL queries hanging under concurrent requests (#418)

[Rish-it]

Summary

• Fixes SQL Query with duckdb just hanging #418 — DuckDB SQL queries hanging when executed from the chat UI
• Root cause: os.chdir() in the execute_sql handler is process-global, so concurrent requests stomp each other's working directory, causing DuckDB to resolve its relative file path against the wrong location
• Fix: resolve relative DB paths (e.g. data.duckdb) to absolute before connecting, and offload blocking DB calls to asyncio.to_thread() to keep the event loop responsive

Changes

• apps/backend/fastapi/main.py — Remove os.chdir() from handler, resolve relative paths to absolute, wrap execute_sql in asyncio.to_thread()

Test plan

• Reproduced the race condition with concurrent os.chdir() + relative DuckDB path
• Verified fix works under 10 concurrent requests with continuous cwd stomping
• Verified asyncio.to_thread() keeps event loop alive during blocking queries
• Verified :memory: and absolute paths are not modified
• All 6 test scenarios passed consistently across 3 runs (18/18)
• Python linter (ruff, ty) and TypeScript linter (tsc, eslint) pass

[cubic-dev-ai]

No issues found across 1 file

[Copilot]

Pull request overview

Fixes a concurrency bug in the FastAPI /execute_sql endpoint where process-global working directory changes could cause DuckDB (and other file-based DBs) to resolve relative database paths incorrectly under concurrent requests.

Changes:

• Removes request-time os.chdir() usage by resolving the project folder to an absolute path.
• Converts relative database file paths to absolute paths before executing SQL.
• Runs blocking execute_sql work in asyncio.to_thread() to keep the event loop responsive.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Path(db_config.path) treats values like ~/data.duckdb as a relative path, so this code would rewrite it to <project_path>/~/data.duckdb and likely break connections. Consider expanding ~ (e.g., Path(...).expanduser()) and then re-checking is_absolute() / resolving before prefixing with project_path.

Suggested change

	db_path = Path(db_config.path)
	if not db_path.is_absolute():
	db_config.path = str(project_path / db_path)
	db_path = Path(db_config.path).expanduser()
	if not db_path.is_absolute():
	db_config.path = str(project_path / db_path)
	else:
	db_config.path = str(db_path)

[Copilot]

The inline comment mentions "concurrent requests changing cwd via os.chdir()", but this handler no longer calls os.chdir(). Updating the comment to describe the actual risk (process-global cwd affecting relative DB paths, and why paths are made absolute) would avoid confusion for future readers.

Suggested change

	# Resolve relative DB paths (e.g. DuckDB) to absolute before executing,
	# so concurrent requests changing cwd via os.chdir() can't cause races.
	# Resolve relative DB paths (e.g. DuckDB) to absolute based on the project
	# root, so changes to the process-global working directory (or starting the
	# server from a different cwd) do not change which database file is used.

[Bl3f]

Small change.

[Bl3f]

How did you successfully reproduced the error?

[Rish-it]

simulated it programmatically: two concurrent async tasks calling os.chdir() to different directories while one tries to open a relative DuckDB path. This consistently triggers IO Error: Cannot open database, confirming the root cause.

os.chdir(project_dir)              # Request A — correct dir, data.duckdb exists
os.chdir(wrong_dir)                # Request B stomps cwd
ibis.duckdb.connect("data.duckdb") # Request A — resolves against wrong dir, fails

Works in CLI because it runs one query at a time. In the app, FastAPI handles concurrent requests on one event loop — os.chdir() is process-global so they stomp each other.

Edge cases I tested

Test	What it verifies
Bug reproduction	Concurrent os.chdir() + relative path causes IO Error: Cannot open database
Path resolution fix	Query succeeds despite concurrent os.chdir() after resolving to absolute path
asyncio.to_thread()	Event loop stays responsive during blocking DuckDB query
Full integration	10 concurrent requests with continuous cwd stomping — all succeed
:memory: path	Skips path resolution for in-memory databases
Absolute path	Skips path resolution when path is already absolute