Skip to content

Use duckdb to speed up dependency management #518

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
hagenw wants to merge 9 commits into from
Closed

Use duckdb to speed up dependency management #518

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
7c25782
Use duckdb to speed up dependency management
hagenw Aug 4, 2025
a495329
Try to fix potential errors
hagenw Aug 6, 2025
4cd0093
Simplify code
hagenw Dec 30, 2025
e0a0801
Don't catch errors
hagenw Dec 30, 2025
c039124
Store deps file in db_root
hagenw Dec 30, 2025
5719cd7
Update api.py
hagenw Dec 30, 2025
5be8bca
Update dependencies.py
hagenw Dec 30, 2025
acf3f70
Update load_to.py
hagenw Dec 30, 2025
6f4e9b7
Update dependency benchmark script (#530)
hagenw Dec 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 12 additions & 6 deletions audb/core/api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -293,12 +293,12 @@ def dependencies(
version,
cache_root=cache_root,
)
cached_deps_file = os.path.join(db_root, define.CACHED_DEPENDENCY_FILE)
deps_file = os.path.join(db_root, define.DEPENDENCY_FILE)

with FolderLock(db_root):
try:
deps = Dependencies()
deps.load(cached_deps_file)
deps.load(deps_file)
except Exception: # does not catch KeyboardInterupt
# If loading cached file fails, load again from backend
#
Expand All @@ -312,9 +312,12 @@ def dependencies(
# See https://github.com/audeering/audb/pull/507
#
backend_interface = utils.lookup_backend(name, version)
deps = download_dependencies(backend_interface, name, version, verbose)
# Store as pickle in cache
deps.save(cached_deps_file)
deps = download_dependencies(
db_root, backend_interface, name, version, verbose
)
# Store as parquet in cache
if not os.path.exists(deps_file):
deps.save(deps_file)

return deps

Expand Down Expand Up @@ -513,9 +516,12 @@ def remove_media(

for version in versions(name):
backend_interface = utils.lookup_backend(name, version)
deps = download_dependencies(backend_interface, name, version, verbose)

with tempfile.TemporaryDirectory() as db_root:
deps = download_dependencies(
db_root, backend_interface, name, version, verbose
)

# Track if we need to upload the dependency table again
upload = False

Expand Down
Loading
Loading