[WIP] Phase 2 Scaffold and Observability Foundations

[Copilot]

Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan and make progress.

Original description:

Phase 2 Scaffold and Observability Foundations

Summary

Introduce Phase 2 scaffold structure for the hybrid recommendation system, feedback event ingestion, experiment & flag framework, PWA offline shell, observability assets, model integrity workflows, and governance (ADRs + design docs). This PR adds directory layout, placeholder/stub modules, CI workflows (build/test, security, SBOM, performance, experiment manifest validation, model artifact validation, canary deploy), ADR stubs (002–006), Grafana dashboard JSON stubs, planning document, and feature store design skeleton. All new functional endpoints remain behind feature flags (implementation stubs only) to ensure safe progressive rollout. No production model logic is included yet.

Scope of Changes

• docs/planning/PHASE2_SCAFFOLD_PR_PLAN.md (planning & rollout strategy)
• docs/adr/ADR-002-hybrid-recommendation-architecture.md
• docs/adr/ADR-003-event-feedback-ingestion.md
• docs/adr/ADR-004-feature-flags-and-experiments.md
• docs/adr/ADR-005-observability-strategy.md
• docs/adr/ADR-006-model-artifact-integrity.md
• docs/feature-store/FEATURE_STORE_DESIGN.md
• observability/dashboards/ (5 Grafana dashboard JSON stubs)
• .github/workflows/ build-test.yml, security-scan.yml, sbom.yml, performance-benchmark.yml, experiment-gating.yml, model-validation.yml, canary-deploy.yml
• (If missing) scripts/ placeholders referenced by workflows (gate scripts, checksum verify, drift stub, canary metrics snapshot, perf gate, coverage gate, warmup). If scripts folder does not yet exist, create stub python scripts with TODO markers so workflows do not fail.
• Placeholder directory structure for reco/, events/, experiments/, pwa/, observability/ with minimal init.py or README stubs to anchor future implementation.

Non-Goals

• Implementing actual recommendation algorithms
• Shipping live PWA offline caching logic beyond basic manifest/service worker placeholder
• Production experiment evaluation logic beyond assignment hashing stub

Rationale

Early establishment of consistent structure, governance, and guardrails reduces integration friction, accelerates subsequent feature development, and enables observability-first approach before traffic.

Testing & Quality

• Coverage threshold gate set to 70% for new code (scripts/coverage_gate.py stub supports threshold flag)
• Lint, type check workflows integrated
• Contract / performance workflows scaffolded but tolerant (non-blocking initially except build & basic tests)

Security & Integrity

• Dependency & SAST scans (pip-audit, Bandit, Trivy)
• Model artifact integrity (checksum + shape validation stubs)
• SBOM + license inventory

Follow-up (Will Become Separate Issues After Merge)

1. Implement collaborative filtering trainer
2. Implement content feature embedder
3. Hybrid blending algorithm (production version) + re-ranking diversity/recency
4. Trending detector job
5. Feedback event enrichment & DLQ handling
6. Full experiment multi-metric evaluation + guardrails
7. PWA push notifications & background sync logic
8. Redis (or chosen store) feature store partial implementation & benchmarking

Rollout Plan

Merge with all flags disabled. Enable dark traffic for recommendation endpoint after instrumentation verified. Gradually roll up traffic percentages. Introduce feedback ingestion after baseline stability.

Acceptance Criteria for This PR

• All listed files present and syntactically valid
• Workflows run successfully (stubs do not fail) on PR branch
• ADRs marked Proposed
• Dashboards loadable (schemaVersion & version provided)
• No runtime activation of unfinished features without flags

Risks & Mitigations

Risk	Mitigation
Workflow failures due to missing scripts	Provide stub scripts returning success / placeholder output
Over-scoped initial PR	Keep only scaffolding & stubs; defer real logic to follow-up issues
Broken dashboards due to metric name drift	Metric naming convention documented in ADR-005

Notes

If repository already contains conflicting workflow/file names, adjust naming or merge content accordingly during implementation stage.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[vercel]

Deployment failed with the following error:

If `rewrites`, `redirects`, `headers`, `cleanUrls` or `trailingSlash` are used, then `routes` cannot be present.

Learn More: https://vercel.link/mix-routing-props

[vercel]

Deployment failed with the following error:

There is no GitHub account connected to this Vercel account.

[netlify]

❌ Deploy Preview for rococo-croquembouche-5ab5cd failed. Why did it fail? →

Name	Link
🔨 Latest commit	5608a0e
🔍 Latest deploy log	https://app.netlify.com/projects/rococo-croquembouche-5ab5cd/deploys/68acf6599be8e50008dcecc7

[bito-code-review]

Bito Automatic Review Skipped - Draft PR

Bito didn't auto-review because this pull request is in draft status.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change draft PR review settings here, or contact your Bito workspace admin at dxp51037@hawkmail.hacc.edu.