diff --git a/INTEGRATION_CHECKLIST.md b/INTEGRATION_CHECKLIST.md index 8e02a34..807f8da 100644 --- a/INTEGRATION_CHECKLIST.md +++ b/INTEGRATION_CHECKLIST.md @@ -46,6 +46,7 @@ The repo's `ruff.toml` enforces rules `E` (pycodestyle errors), `F` (pyflakes), |------|-----------|-----| | `__init__.py` | `F401` (unused import) | Import-and-re-export is the expected pattern | | `tests/context.py` | `F401`, `E402` | Import-after-path-setup is the expected pattern | +| `tests/conftest.py` | `E402` | Import-after-path-setup is the expected pattern (pytest alternative to `context.py`) | Other files that need intentional "unused" imports must use `# noqa: F401` inline. diff --git a/LOCAL_DEVELOPMENT.md b/LOCAL_DEVELOPMENT.md index 2516c51..98a5801 100644 --- a/LOCAL_DEVELOPMENT.md +++ b/LOCAL_DEVELOPMENT.md @@ -31,7 +31,7 @@ Run this **first** — it catches structural problems before you waste time on c - `config.json` has the required fields and valid schema - `__init__.py` is minimal (only import + `__all__`) - `requirements.txt` includes `autohive-integrations-sdk` -- `tests/` folder has `__init__.py`, `context.py`, and at least one `test_*.py` +- `tests/` folder has `__init__.py`, `context.py` or `conftest.py`, and at least one `test_*.py` - Icon is exactly 512x512 pixels - OAuth scopes are actually used (heuristic) @@ -133,19 +133,46 @@ python scripts/check_version_bump.py origin/main $DIRS Once everything passes, run `validate_integration.py` for a final structure check before pushing. +## Running Tests + +Unit tests and integration tests are run separately. See the integrations repo's [CONTRIBUTING.md](https://github.com/autohive-ai/autohive-integrations/blob/master/CONTRIBUTING.md#running-tests) for full details. + +### Unit tests (CI + local) + +```bash +# Via the tooling runner (installs deps per-integration, runs with coverage) +python scripts/run_tests.py my-integration + +# Or directly via pytest (from the integrations repo root) +pytest my-integration/ +``` + +### Integration tests (local only) + +Integration tests (`test_*_integration.py`) call real APIs and are **never** run in CI. They must be invoked by passing the file path explicitly: + +```bash +pytest my-integration/tests/test_my_integration_integration.py -m integration +``` + +They are excluded from auto-discovery by two mechanisms: +1. `python_files` in `pyproject.toml` only matches `test_*_unit.py` +2. `addopts` includes `-m unit`, which deselects `@pytest.mark.integration` tests + ## What CI Runs on Your PR -The `validate-integration.yml` workflow uses the composite action defined in `action.yml` to run three checks on every PR: +The `validate-integration.yml` workflow uses the composite action defined in `action.yml` to run these checks on every PR: | Step | Script | What It Does | |------|--------|-------------| | 1 | `get_changed_dirs.py` | Detects which integration folders changed | | 2 | `validate_integration.py` | Structure and config validation | | 3 | `check_code.py` | Syntax, imports, JSON, lint, format, security, deps, config sync | -| 4 | `check_readme.py` | Checks that the main README.md was updated for new integrations | -| 5 | `check_version_bump.py` | Checks that config.json version was incremented, recommends bump level | +| 4 | `run_tests.py` | Installs each integration's deps, runs `test_*_unit.py` files (unit tests only) | +| 5 | `check_readme.py` | Checks that the main README.md was updated for new integrations | +| 6 | `check_version_bump.py` | Checks that config.json version was incremented, recommends bump level | -If no integration directories changed (only `scripts/`, `tests/`, etc.), steps 2–5 are skipped. +If no integration directories changed (only `scripts/`, `tests/`, etc.), steps 2–6 are skipped. Results are posted as a sticky PR comment showing ✅ Passed, ⚠️ Passed with warnings, or ❌ Failed for each check. diff --git a/README.md b/README.md index f02f07a..aa37b6a 100644 --- a/README.md +++ b/README.md @@ -198,7 +198,11 @@ python scripts/run_tests.py hackernews bitly notion python scripts/run_tests.py ``` -Integrations without `test_*_unit.py` files are skipped with a warning. The test infrastructure (`pyproject.toml`, `conftest.py`, `requirements-test.txt`) lives in the integrations repo — see its `CONTRIBUTING.md` for how to write and run tests locally. +Integrations without `test_*_unit.py` files are skipped with a warning. + +> **Note:** This script only runs unit tests. Integration tests (`test_*_integration.py`) require real API credentials and are run locally by developers — never in CI. See the integrations repo's `CONTRIBUTING.md` for details. + +The test infrastructure (`pyproject.toml`, `conftest.py`, `requirements-test.txt`) lives in the integrations repo — see its `CONTRIBUTING.md` for how to write and run tests locally. ## Integration Requirements @@ -211,7 +215,7 @@ See `INTEGRATION_CHECKLIST.md` for full details. - `requirements.txt` - Dependencies (must include `autohive-integrations-sdk`) - `README.md` - Documentation - `icon.png` or `icon.svg` - Integration icon (512x512 pixels) -- `tests/` - Test folder with `__init__.py`, `context.py`, and `test_*.py` +- `tests/` - Test folder with `__init__.py`, `context.py` or `conftest.py`, and `test_*.py` ## Integrations diff --git a/scripts/docs/run_tests.md b/scripts/docs/run_tests.md index b86f9db..7be763b 100644 --- a/scripts/docs/run_tests.md +++ b/scripts/docs/run_tests.md @@ -132,6 +132,16 @@ my-integration/ ⚠️ No unit tests to run ``` +## Unit Tests Only + +This script only runs **unit tests** (`test_*_unit.py`). Integration tests (`test_*_integration.py`) are a separate concern: + +- They require real API credentials and must never run in CI. +- They are not auto-discovered by pytest (`python_files` restricts discovery to `test_*_unit.py`). +- Developers run them locally by passing the file path explicitly: `pytest /tests/test_*_integration.py -m integration` + +See the integrations repo's `CONTRIBUTING.md` for full details on running both test types. + ## Integration with CI Called by the composite action in `action.yml`: diff --git a/scripts/docs/validate_integration.md b/scripts/docs/validate_integration.md index 1803800..8947f83 100644 --- a/scripts/docs/validate_integration.md +++ b/scripts/docs/validate_integration.md @@ -155,7 +155,7 @@ Allowed patterns: |-------|----------|-------------| | `tests/` directory exists | Error | Test directory required | | `tests/__init__.py` exists | Error | Test package init | -| `tests/context.py` exists | Error | Test import setup | +| `tests/context.py` or `tests/conftest.py` exists | Error | Test import/fixture setup (either one satisfies this check) | | `tests/test_*.py` exists | Error | At least one test file | ### 7. Main Python File (`_check_main_python_file`) diff --git a/scripts/validate_integration.py b/scripts/validate_integration.py index 0290cdf..114de07 100755 --- a/scripts/validate_integration.py +++ b/scripts/validate_integration.py @@ -367,14 +367,12 @@ def _check_tests_folder(self): return # Check required test files - required_test_files = [ - ('__init__.py', 'Test package init (can be empty)'), - ('context.py', 'Test context/import setup'), - ] + if not (tests_path / '__init__.py').exists(): + self.add_error("Missing tests/__init__.py (Test package init — can be empty)") - for filename, description in required_test_files: - if not (tests_path / filename).exists(): - self.add_error(f"Missing tests/{filename} ({description})") + # Accept either context.py (legacy import setup) or conftest.py (pytest fixture setup) + if not (tests_path / 'context.py').exists() and not (tests_path / 'conftest.py').exists(): + self.add_error("Missing tests/context.py or tests/conftest.py (test import/fixture setup)") # Check for at least one test file test_files = list(tests_path.glob('test_*.py'))