Klondike v0.3.0

[IanRFerguson]

• Move off of Poetry and onto UV
• Adds a basic GoogleCloudStorage object (including unit tests)
• Adds a basic "CSV Streaming" utility script to load data into BigQuery

[Copilot]

Pull request overview

This PR releases v0.3.0 by modernizing packaging/tooling (Poetry → uv + Hatchling), adding a Google Cloud Storage connector with tests, and introducing a CSV → database batching script intended for loading data into BigQuery (and other connectors).

Changes:

• Migrate packaging to PEP 621 [project] metadata + Hatchling build backend; update CI/Makefile/pre-commit to use uv, ruff-format, mypy, pytest.
• Add CloudStorageConnector (GCS) plus unit tests.
• Add stream_csv_to_database script and tests for batched CSV uploads.

Reviewed changes

Copilot reviewed 21 out of 26 changed files in this pull request and generated 18 comments.

Show a summary per file

File	Description
pyproject.toml	Moves to PEP 621 metadata + Hatchling; defines deps, ruff/mypy/pytest config.
poetry.lock	Removed as part of Poetry → uv migration.
uv.lock	New lockfile to support uv sync in CI (existing file in repo).
Makefile	Adds uv-based ruff/pytest/mypy targets.
.pre-commit-config.yaml	Updates ruff hook version and adds ruff-format.
.github/workflows/pr_checks.yml	New PR CI jobs using uv (ruff, format, mypy, pytest).
.github/workflows/publish.yml	New PyPI publish workflow (currently broken; see comments).
.github/workflows/automated_tests.yml	Removes legacy Poetry-based CI workflow.
klondike/base/abc_klondike.py	Renames/expands ABCs into database vs storage connector interfaces.
klondike/gcp/cloud_storage.py	Implements CloudStorageConnector (list/get/put blobs, list buckets).
tests/gcp/test_cloud_storage.py	Adds unit tests for CloudStorageConnector behaviors.
klondike/scripts/stream_csv_to_database.py	Adds CSV batching uploader script (currently not truly streaming; see comments).
klondike/scripts/__init__.py	Exposes stream_csv_to_database.
tests/scripts/test_stream_csv_to_bigquery.py	Adds tests around the CSV batching script.
klondike/gcp/bigquery.py	Refactors BigQueryConnector typing/base class and query return behavior.
klondike/snowflake/snowflake.py	Refactors SnowflakeConnector typing/base class; adds kwargs plumbing and result normalization.
klondike/utilities/utilities.py	Adds get_env_or_value; improves typing/docstrings.
klondike/utilities/logger.py	Introduces shared colored logger implementation.
klondike/__init__.py	Switches to shared logger and dynamic exports (contains a broken export; see comments).
klondike/aws/redshift.py	Updates base class name to KlondikeBaseDatabaseConnector.
klondike/aws/s3.py	Updates base class name to KlondikeBaseStorageConnector.
klondike/postgres/postgres.py	Updates base class name to KlondikeBaseDatabaseConnector.
README.md	Updates intro and adds logo.
.gitignore	Adds development/.

Comments suppressed due to low confidence (1)

klondike/gcp/bigquery.py:133

• __setup_google_app_creds is broken for dict credentials: it writes JSON to a TemporaryFile (which typically has no usable filesystem path) and then sets the env var to the JSON string (creds), not to a filename. GOOGLE_APPLICATION_CREDENTIALS must point to a JSON file path. Use NamedTemporaryFile(delete=False) (or similar), write the JSON there, and set the env var to that file path (and ensure the temp file lifecycle is handled).

    def __setup_google_app_creds(
        self, app_creds: Union[str, dict[str, Any]], env_variable: str
    ) -> None:
        "Sets runtime environment variable for Google SDK"

        if isinstance(app_creds, dict):
            creds = json.dumps(app_creds)
            tempp_file = tempfile.TemporaryFile(suffix=".json")

            with open(tempp_file.name, "w") as f:
                f.write(creds)

        elif isinstance(app_creds, str):
            creds = app_creds

        os.environ[env_variable] = creds

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.