Add fetcher, papers, annotator pipeline & CLI

[iskoldt-X]

This pull request introduces a new AI annotation module for extracting GPCR structural data from scientific papers, along with significant enhancements to the CLI and annotation pipeline. The main changes include the addition of a Gemini API client with rate limiting, a Ghostscript-based PDF compressor, a robust post-processing pipeline for annotation results, and new CLI commands for batch annotation and paper fetching. The pyproject.toml has also been updated to include new dependencies and extras for annotation and paper processing.

Major new annotation module and pipeline:

• Added a new AI annotation module in src/gpcr_tools/annotator/ with the following components:
  • gemini_client.py: Implements a Gemini API client with per-key sliding-window rate limiting, supporting both new and legacy API key environment variables.
  • pdf_compressor.py: Provides Ghostscript-based PDF compression for large files before Gemini upload, ensuring compliance with API size limits.
  • post_processor.py: Adds a post-processing pipeline for Gemini annotation responses, including unwrapping protobuf objects, cleaning up empty fields, standardizing auxiliary protein names, and normalizing UniProt entry names.
  • cli_handler.py: Implements a CLI handler for the new annotate command, supporting both single and batch annotation modes, with robust input resolution and logging.
  • Added module docstring to __init__.py.

CLI and workflow enhancements:

• Extended the main CLI in src/gpcr_tools/__main__.py to add new commands:
  • fetch: Download PDB metadata from RCSB and enrich with UniProt/PubChem data.
  • fetch-papers: Download open-access papers for enriched PDB entries.
  • annotate: Run Gemini AI annotation in both single and batch modes, with options for prompt customization, model selection, and batch recovery/status checking. [1] [2]

Dependency and configuration updates:

• Updated pyproject.toml:
  • Bumped version to 0.1.3.
  • Added new extras for [annotate] (Gemini) and [papers] (PyMuPDF, lxml).
  • Included gpcr-annotation-tools[annotate,papers] in the development dependencies for easier local testing. [1] [2]

These changes collectively enable automated, scalable annotation of GPCR structures using AI, with robust input/output handling and improved developer ergonomics.Introduce end-to-end pipeline components and CLI commands for fetching, paper acquisition, and AI annotation. Adds new packages: fetcher (RCSB GraphQL client, enricher, cache, targets, runner), papers (downloader, watcher, runner), and annotator (Gemini client with sliding-window rate limiting, prompt builder, PDF compressor, post-processor, schema and runner/CLI glue). Extend CLI (main.py) with fetch / fetch-papers / annotate commands (single + batch modes, prompt/model/runs flags and batch control). Update packaging (pyproject.toml bumped to 0.1.3, extras for annotate/papers), expand README with pipeline docs and usage, add many unit/integration tests, and small config/workspace/validator adjustments to integrate the new stages.

[Copilot]

Pull request overview

This PR adds an end-to-end pipeline for fetching PDB metadata, downloading papers, and running Gemini-based AI annotation (single + batch), plus supporting workspace/config updates, docs, and tests.

Changes:

• Added new pipeline stages: fetcher/ (RCSB + enrichment), papers/ (OA download + paywall watcher), annotator/ (Gemini client, prompt building, compression, post-processing, runner).
• Extended the CLI with fetch, fetch-papers, and annotate commands.
• Updated configuration/constants, workspace initialization, dependencies/extras, and expanded documentation/testing.

Reviewed changes

Copilot reviewed 37 out of 37 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
src/gpcr_tools/config.py	Adds centralized URLs/timeouts/sleeps + new workspace path fields used by new pipeline stages
src/gpcr_tools/workspace.py	Adds targets.txt creation + switches to raw_pdb_json_dir
src/gpcr_tools/fetcher/*	Implements target parsing, RCSB GraphQL fetch, enrichment logic, and JSON caching
src/gpcr_tools/papers/*	Implements tiered paper download, download logging, and manual PDF watcher
src/gpcr_tools/annotator/*	Implements Gemini client + schema, prompt builder, PDF compression, post-processing, and runners
src/gpcr_tools/__main__.py	Adds new CLI subcommands and annotation orchestration logic
src/gpcr_tools/validator/*	Switches hardcoded URLs/timeouts/sleeps to config constants
README.md	Documents new pipeline, CLI usage, and architecture
pyproject.toml	Version bump + new extras/deps
tests/unit/*, tests/integration/*	Adds unit tests and new “live API” integration tests

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

prompt_builder is reading polymer/nonpolymer entities from the top-level enriched_data, but the enricher writes them under enriched_data["data"]["entry"]. As a result, chain inventory and simplified metadata will be empty/"UNKNOWN" for real enriched JSON, which will significantly degrade annotation quality. Update the prompt builder to consistently dereference data.entry (and use rcsb_id as the PDB ID) before extracting polymer_entities, nonpolymer_entities, exptl/refine, etc.

[Copilot]

enhanced_simplify_pdb_json currently uses enriched_data.get("entry"), but the enriched files produced by fetcher/enricher.py are full GraphQL responses with the payload under data.entry. This makes pdb_id, method, resolution, and release_date resolution incorrect for real runs. Consider normalizing by setting entry = (enriched_data.get("data") or {}).get("entry") or {} and reading fields from there.

Suggested change

	pdb_id = (enriched_data.get("entry") or {}).get("id") or "UNKNOWN"
	entry = enriched_data.get("entry") or {}
	entry = (enriched_data.get("data") or {}).get("entry") or enriched_data.get("entry") or {}
	pdb_id = entry.get("id") or "UNKNOWN"

[Copilot]

Sibling PDB IDs are stored as data.entry.sibling_pdbs by the enricher, but the prompt builder looks for enriched_data.get("sibling_pdbs"). This prevents the sibling warning from ever being included for real enriched JSON. Please read sibling_pdbs from the same normalized entry object used for the rest of the prompt.

[Copilot]

Auto-discovery for annotate treats any existing ai_results/{pdb_id}/ directory as "done". This will incorrectly skip PDBs that have only partial run_*.json outputs (e.g., interrupted runs). Align this logic with annotator/cli_handler.py (or a shared helper) by checking whether the expected run_1..run_N.json files exist before considering a PDB complete.

[Copilot]

annotator/cli_handler.py appears to be unused (no references found) and duplicates substantial CLI logic that already exists in gpcr_tools/__main__.py. This increases maintenance burden and can lead to behavior drift. Either wire __main__.py to call run_annotate(...) or remove the handler module to avoid dead code.

[Copilot]

_standardize_auxiliary_name is annotated as accepting/returning str, but it is called in tests with None and will return None at runtime. This is likely to trip mypy and makes the API misleading. Update the signature to name: str | None (and return type accordingly) to match actual usage.

[Copilot]

This import statement exceeds the configured line length and bypasses Ruff's auto-formatting style. Please run ruff format or wrap the import across multiple lines to match the project's formatting conventions.

Suggested change

	from gpcr_tools.config import OVERRIDE_VARS, SUPPORTED_CONTRACT_VERSION, WorkspaceConfig, get_config
	from gpcr_tools.config import (
	OVERRIDE_VARS,
	SUPPORTED_CONTRACT_VERSION,
	WorkspaceConfig,
	get_config,
	)

[Copilot]

Including gpcr-annotation-tools[annotate,papers] inside the package's own dev extra makes the project depend on itself. In editable installs this can cause pip to resolve the published package from PyPI (or fail if it's not available), and it also obscures which deps are actually needed for tests. Instead, add the extra deps directly to dev (e.g., google-genai, PyMuPDF, lxml) or document installing with pip install -e '.[dev,annotate,papers]'.

Suggested change

	"gpcr-annotation-tools[annotate,papers]",
	"google-genai",
	"PyMuPDF",
	"lxml",

[Copilot]

The README claims there are "no live network calls in the test suite", but this PR adds live integration tests under tests/integration/ (e.g., test_fetch_live.py). Either adjust the documentation to describe how live tests are gated/skipped, or change the tests to be opt-in so the statement remains true.

Suggested change

	- **Mock HTTP** for all external APIs — no live network calls in the test suite
	- **Mock HTTP** for external APIs in the default test suite; any live network integration tests are gated/opt-in and skipped unless explicitly enabled

[Copilot]

This file performs live network calls (RCSB/UniProt/PubChem) but is not skipped/marked, so it will run under the default pytest configuration (testpaths = ['tests']). That can make CI runs slow and flaky due to external rate limits/outages. Make these tests opt-in (e.g., @pytest.mark.skipif(not os.getenv('GPCR_RUN_LIVE_TESTS'), ...)) or add a network marker and configure pytest to exclude it by default.

[Copilot]

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated 5 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

enhanced_simplify_pdb_json() derives pdb_id from entry.get("id"), but the RCSB GraphQL payload puts the identifier at entry["rcsb_id"] (and a nested entry["entry"]["id"]). With real enriched data this will often become UNKNOWN, degrading prompt grounding. Prefer entry.get("rcsb_id") (fallback to ((entry.get("entry") or {}).get("id"))) so prompts consistently include the actual PDB ID.

Suggested change

	pdb_id = entry.get("id") or "UNKNOWN"
	pdb_id = entry.get("rcsb_id") or ((entry.get("entry") or {}).get("id")) or "UNKNOWN"

[Copilot]

The CLI falls back to reading cfg.default_prompt_file ({workspace}/prompts/v5.txt), but init-workspace/ensure_runtime_dirs do not create prompts/ or the default prompt file, and there is no prompts/v5.txt in-repo. On a fresh workspace, gpcr-tools annotate will crash with FileNotFoundError. Either create/populate the default prompt in init_workspace (and add prompts to the contract dirs), or ship the prompt as a package resource and reference it from the installed package rather than the workspace.

[Copilot]

These tests run live network calls whenever GPCR_EMAIL_FOR_APIS is set (via the skipif(not _EMAIL, ...) gate). This is easier to satisfy in CI than the explicit GPCR_RUN_LIVE_TESTS=1 gate used elsewhere (e.g. test_fetch_live.py), which can unintentionally enable slow/flaky live tests. Consider gating on GPCR_RUN_LIVE_TESTS (and optionally also requiring GPCR_EMAIL_FOR_APIS).

[Copilot]

Pull request overview

Copilot reviewed 44 out of 44 changed files in this pull request and generated 5 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

check_batch_status() builds raw_out_file using the raw job_name, but Gemini batch job names commonly contain slashes (e.g. batchJobs/...). This turns the output path into nested directories (and recover_batch() won’t find it via raw_output_*.jsonl). Sanitize the job name for filenames (e.g. replace / with _ or use the last path segment) and ensure the parent directory exists before writing.

[Copilot]

run_single_pdb() defaults model_name to the module-level GEMINI_MODEL_NAME, which is a static constant (and the config comment suggests using get_gemini_model_name() for fresh env reads). Consider defaulting model_name to None and resolving it at runtime (or defaulting via get_gemini_model_name()), so programmatic callers get the same env override behavior as the CLI.

[Copilot]

In do_run(), generic exceptions are retried with backoff but the underlying exception is never logged (only a final “failed after retries” message). This makes it hard to diagnose schema mismatches, tool-calling failures, or transient SDK issues. Log the exception details at least once per run (e.g., on the first failure or on the final retry) before sleeping.

[Copilot]

Unpaywall API calls typically require an email query parameter; currently _fetch_unpaywall_pdf_url() calls GET {UNPAYWALL_API_URL}/{doi} without passing the user email, even though GPCR_EMAIL_FOR_APIS is required and _build_session() sets polite headers. If Unpaywall rejects requests without ?email=..., this tier will always fail. Consider adding the email as a query param (and/or threading it into this function).

[Copilot]

init_workspace() creates the prompts/ directory and config defaults default_prompt_file to workspace/prompts/v5.txt, but no default prompt file is created here. As a result, gpcr-tools annotate will error immediately in a freshly-initialized workspace unless the user manually creates v5.txt. Consider writing a starter prompts/v5.txt alongside targets.txt (or updating the default path).

[Copilot]

Pull request overview

Copilot reviewed 45 out of 47 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (2)

src/gpcr_tools/csv_generator/review_engine.py:66

• isinstance() does not accept PEP 604 union types (e.g., list | dict) at runtime; this will raise TypeError: isinstance() argument 2 cannot be a union and break edits in the review UI. Use a tuple of types instead (e.g., isinstance(original, (list, dict))).

    if isinstance(original, list | dict):
        try:
            parsed = json.loads(new_str)
        except (json.JSONDecodeError, ValueError):
            parsed = None
        if isinstance(parsed, type(original)):
            return parsed
    return new_str

src/gpcr_tools/csv_generator/review_engine.py:250

• isinstance(value, dict | list) will raise TypeError at runtime because isinstance() cannot take a union type. Use isinstance(value, (dict, list)).

        def format_value(value: Any) -> str:
            if isinstance(value, dict | list):
                try:
                    return json.dumps(value, sort_keys=True)
                except (TypeError, ValueError):
                    return str(value)
            return str(value)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

isinstance(conf, int | float) will raise TypeError because isinstance() cannot take a union type. Use isinstance(conf, (int, float)) (or handle the new string-based confidence values explicitly).

[Copilot]

isinstance(data, str | bytes | list | tuple) will raise TypeError because isinstance() cannot take a PEP 604 union type. Replace with a tuple of types (e.g., (str, bytes, list, tuple)) to avoid crashing post-processing.

[Copilot]

isinstance(val, dict | list | str) will raise TypeError at runtime because isinstance() cannot take a union type. Use isinstance(val, (dict, list, str)).

[Copilot]

isinstance(cids, int | float) will raise TypeError at runtime because isinstance() cannot take a PEP 604 union type. Use isinstance(cids, (int, float)) to avoid breaking PubChem CID resolution.