WIP: VCF Ingestion Tests #727
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This replaces hard-coded strings used when reading/writing the manifest array.
Ingest now distinguishes between smaples that are ready to load and samples that have been loaded using a new "ready" status. The status of a sample is changed from "ready" / "missing index" to "ok" upon successful ingestion, allowing failed ingestions to be resumed.
This includes renaming some variables to add context to the messages.
This ensures that a member's value is saved to the ingestion manifest, rather than the member's name.
This allows the function to be blocking, if need be. The default value is False to preserve the previous unparameterized behavior.
This allows the function to be used externally.
The class provides common setup, log capturing during ingestion, and tests common to all ingestion types.
This class extends the base class for VCF ingestion, implementing ingestion via search URI and search-specific test cases.
This class extends the base class for VCF ingestion, implementing ingestion via sample list and list-specific test cases.
This class extends the base class for VCF ingestion, implementing ingestion via metadata array and metadata-specific test cases.
…class This allows the base class configure, setup, and teardown to be leveraged by classes that don't need to run the common tests.
This class extends the base class for VCF ingestion, implementing ingestions that simulate a failed ingestion with subsequent (non-)resume ingestions, as well as resume-specific test cases. Note that causing an ingestion to fail programmatically is not support by VCF; ingestion failure is enabled for this class by monkey patching the sample ingestion DAG function, allowing failures to be simulated deterministically.
This is to prevent them from being run by CI.
This is to prevent an import error during CI since TileDB-VCF is not installed in the CI environment.
…cleary/vcf-15/vcf-ingestion-tests
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Tests are run against data in s3://tiledb-unittest/vcf-ingestion-test/. Specifically, there's 6
.vcf.gzfiles, each corresponding to a different success/failure code path in VCF ingestion. Additionally, there's a TileDB "metadata" array and a file containing the.vcf.gzURIs. These, and the.vcf.gzfiles themselves, are used to exercise all 3 ways VCF URIs can be provided during ingestion.These tests won't be runnable via CI because the unittest user doesn't have adequate privileges. However, these tests can be run locally, which is better than nothing!
Currently looking for any/all feedback.
Note, this is a branch of alancleary/vcf-12/ingestion-failure-handling and is intended to be merged after PR #723.