Implement bypass_cache flag for Tier1 and other fixes#137
Conversation
Codecov Report❌ Patch coverage is
🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
Adds cache-bypass capabilities to Tier 1 Elasticsearch metadata/subclass-mapping retrieval and query execution, primarily to mitigate stale-cache issues and related test flakiness (Issue #45).
Changes:
- Add
bypass_cachesupport to Tier 1 ES query execution by clearing ES index caches before running queries when requested. - Add
bypass_cachesupport to ubergraph subclass mapping retrieval (skip Redis metadata cache when requested). - Update Tier 1 ES driver tests to use cache bypass and add a new cache-bypass test.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
src/retriever/data_tiers/tier_1/elasticsearch/driver.py |
Adds bypass_cache handling for query execution and passes bypass flag through subclass-mapping retrieval. |
src/retriever/data_tiers/tier_1/elasticsearch/meta.py |
Adds bypass_cache option to ubergraph info retrieval to skip cached values when requested. |
tests/data_tiers/tier_1/elasticsearch_tests/test_tier1_driver.py |
Updates integration tests to bypass cache for metadata/subclass mapping and adds a cache-bypass query test. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
tests/data_tiers/tier_1/elasticsearch_tests/test_tier1_driver.py
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
tests/data_tiers/tier_1/elasticsearch_tests/test_tier1_driver.py
Outdated
Show resolved
Hide resolved
|
@shuchenliu I've opened a new pull request, #138, to work on those changes. Once the pull request is ready, I'll request review from you. |
…pendency (#138) test: mock clear_cache and run_single_query in test_cache_bypass Co-authored-by: shuchenliu <5944967+shuchenliu@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: shuchenliu <5944967+shuchenliu@users.noreply.github.com>
…dd associated tests
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@shuchenliu Just checking, this now does a timestamp check instead of purging the cache, and is ready for review? |
Yes, we no longer purge the cache. A timestamp check payload is no added to every query |
This PR solves
tier1part of #45 and added other fixesFixed flaky test in
metadata_retrievalResolved an issue where stale data in the Redis instance could cause test failures when new indices were deployed.
Enabled
bypass_cacheforget_subclass_mappingAdded support for the
bypass_cacheflag in thedriver.get_subclass_mappingmethod to address similar cache-related issues.Implemented query-level
bypass_cachesupportAdded support for the TRAPI-level
bypass_cacheflag, allowing clients to purge Elasticsearch’squery,request, andfielddatacaches whenbypass_cache: trueis specified in the original TRAPI query, by updatingdriver.run_querymethod to accept an additionalbypass_cachekeyword parameter.