diff --git a/.agents/planning/00-index.md b/.agents/planning/00-index.md new file mode 100644 index 000000000..d06d93ca0 --- /dev/null +++ b/.agents/planning/00-index.md @@ -0,0 +1,143 @@ +# UI Framework Migration Planning + +## Overview + +Planning documents for migrating the Antenna UI from Vite + React Router to a more opinionated framework. Two options are evaluated: **Next.js** and **Remix**. + +**Recommendation**: Remix is better aligned with our motivations (reducing boilerplate, prescribed patterns, DRF-like experience). + +--- + +## Document Index + +### Context & Motivations + +| # | Document | Description | +|---|----------|-------------| +| 01 | [Current State](./01-current-state.md) | Analysis of current UI and API architecture | +| 08 | [**Motivations**](./08-motivations.md) | Why we're considering migration (small team, patterns, maintenance) | +| 09 | [**E2E Testing Strategy**](./09-e2e-testing-strategy.md) | Playwright setup, test generation strategies, analytics-driven prioritization | + +### Next.js Assessment + +| # | Document | Description | +|---|----------|-------------| +| 02 | [Next.js Benefits](./02-nextjs-benefits.md) | What Next.js offers and can replace | +| 03 | [Required Changes](./03-required-changes.md) | Everything that needs to change for Next.js | +| 04 | [Custom/Bespoke](./04-custom-bespoke.md) | What stays custom (Next.js) | +| 05 | [Migration Steps](./05-migration-steps.md) | Phased implementation plan (Next.js) | +| 06 | [Questions & Risks](./06-questions-risks.md) | Risks, alternatives, benefits (Next.js) | +| 07 | [Testing & Evaluation](./07-testing-evaluation.md) | Testing plan (Next.js) | + +### Remix Assessment (Recommended) + +| # | Document | Description | +|---|----------|-------------| +| R1 | [Remix Benefits](./remix/01-remix-benefits.md) | What Remix offers - loaders, actions, forms | +| R2 | [Required Changes](./remix/02-required-changes.md) | Everything that needs to change for Remix | +| R3 | [Custom/Bespoke](./remix/03-custom-bespoke.md) | What stays custom (Remix) | +| R4 | [Migration Steps](./remix/04-migration-steps.md) | Phased implementation plan (Remix) | +| R5 | [Questions & Risks](./remix/05-questions-risks.md) | Risks, alternatives, benefits (Remix) | +| R6 | [Testing & Evaluation](./remix/06-testing-evaluation.md) | Testing plan (Remix) | +| R7 | [**AI-Driven Migration**](./remix/07-ai-driven-migration.md) | How Claude Code executes migration in 1-2 hours | + +--- + +## Quick Comparison + +### Current Stack +- **Build**: Vite 4.5.3 +- **Routing**: React Router v6.8.2 +- **State**: React Query + Context +- **Styling**: Tailwind CSS + SCSS +- **Components**: Radix UI + nova-ui-kit + +### Option A: Next.js +- **Build**: Next.js 14 (App Router) +- **Routing**: File-based (app/) +- **State**: React Query + Context (keep) +- **Data loading**: Flexible (server components, client) +- **Forms**: Manual (Server Actions basic) + +### Option B: Remix (Recommended) +- **Build**: Remix 2.x (Vite-based) +- **Routing**: File-based (routes/) +- **State**: Minimal (loaders handle server state) +- **Data loading**: Prescribed (loaders) +- **Forms**: First-class (`
`, actions) + +--- + +## Comparison Matrix + +| Factor | Current | Next.js | Remix | +|--------|---------|---------|-------| +| Boilerplate | High | Medium | **Low** | +| Prescribed Patterns | None | Some | **Many** | +| Form Handling | Manual | Basic | **Built-in** | +| Data Flow Opinions | None | Low | **High** | +| Files to Remove | - | ~10 | **~43** | +| DRF-like Feel | No | Somewhat | **Yes** | +| Learning Curve | - | Medium | Medium | +| Community Size | - | **Largest** | Medium | +| Migration Time | - | 6-8 weeks | 6-7 weeks | + +--- + +## Key Metrics + +### Next.js Migration +| Aspect | Value | +|--------|-------| +| Estimated Duration | 6-8 weeks | +| Routes to Migrate | ~25 | +| Files to Change | ~150 | +| Custom Code Preserved | ~85% | +| Boilerplate Reduction | Low | + +### Remix Migration +| Aspect | Value | +|--------|-------| +| Estimated Duration | 6-7 weeks | +| Routes to Migrate | ~25 | +| Files to Delete | ~43 (hooks → loaders) | +| Custom Code Preserved | ~85% | +| Boilerplate Reduction | **High** | + +--- + +## Recommendation + +**Go with Remix** because: + +1. **Loaders replace ~22 data hooks** - prescribed pattern for data fetching +2. **Actions replace ~15 mutation hooks** - prescribed pattern for mutations +3. **URL state is built-in** - no more useFilters, useSort, usePagination +4. **Forms are first-class** - `` with automatic pending states +5. **More opinionated** - closer to "DRF for frontend" philosophy +6. **Same migration effort** - ~6-7 weeks vs 6-8 weeks for Next.js + +### Suggested Approach + +1. **POC (1 week)**: Migrate login + projects list + one occurrence page +2. **Evaluate**: Does it reduce boilerplate? Is the pattern clear? +3. **Decide**: Continue or reconsider +4. **Full migration (5-6 weeks)**: If POC successful + +--- + +## Team Context + +- **2 developers** + periodic interns + AI agents +- Need **prescribed patterns** for fast onboarding +- Want to **reduce custom code** maintenance burden +- Looking for **DRF-like productivity** on frontend + +See [Motivations](./08-motivations.md) for full context. + +--- + +## Created +- **Date**: 2026-02-04 +- **Branch**: claude/nextjs-migration-planning-nx9Cd +- **Updated**: 2026-02-04 (added Remix assessment) diff --git a/.agents/planning/01-current-state.md b/.agents/planning/01-current-state.md new file mode 100644 index 000000000..961580099 --- /dev/null +++ b/.agents/planning/01-current-state.md @@ -0,0 +1,191 @@ +# Current State of UI and API + +## Overview + +The Antenna UI is a React 18 single-page application (SPA) built with Vite, serving as the frontend for the Automated Monitoring of Insects ML Platform. It interfaces with a Django REST Framework backend via a versioned JSON API. + +--- + +## UI Architecture + +### Build System +- **Bundler**: Vite 4.5.3 +- **Language**: TypeScript 4.4.2 (strict mode) +- **Output**: Static bundle deployed to `./build` +- **Dev Server**: Port 3000 with HMR +- **Proxy**: Dev server proxies `/api` and `/media` to Django backend + +### Routing +- **Library**: React Router v6.8.2 +- **Pattern**: Client-side SPA routing +- **Structure**: Nested routes with `` for layout composition +- **Modal Pattern**: Detail dialogs controlled via URL params (e.g., `/occurrences/:id?`) + +**Route Hierarchy:** +``` +/ +├── /auth +│ ├── /login +│ ├── /reset-password +│ └── /reset-password-confirm +├── /projects (list) +└── /projects/:projectId + ├── /summary + ├── /deployments/:id? + ├── /sessions/:id? + ├── /captures + ├── /occurrences/:id? + ├── /taxa/:id? + ├── /jobs/:id? + ├── /collections + ├── /exports/:id? + ├── /pipelines/:id? + ├── /algorithms/:id? + ├── /processing-services/:id? + ├── /sites, /devices, /general, /team + ├── /default-filters, /storage, /processing +├── /terms-of-service +└── /code-of-conduct +``` + +### State Management + +**Server State**: TanStack React Query v4.29.5 +- Automatic caching and background refetching +- Query invalidation on mutations +- DevTools for debugging + +**Client State**: React Context API +- `UserContext` - Authentication token and login state +- `UserInfoContext` - Current user profile +- `UserPreferencesContext` - UI preferences (localStorage-backed) +- `BreadcrumbContext` - Navigation breadcrumbs +- `CookieConsentContext` - GDPR compliance + +### Data Fetching + +**Pattern**: Custom hooks wrapping React Query + +```typescript +// List queries +useOccurrences(params?: FetchParams) → { occurrences, total, isLoading } +useJobs(params?: FetchParams) → { jobs, total, isLoading } + +// Detail queries +useOccurrenceDetails(id) → { occurrence, isLoading } +useProjectDetails(projectId) → { project, isLoading } + +// Mutations +useCreateJob(onSuccess) → { createJob, isLoading, error } +useCreateIdentification(onSuccess) → { createIdentification, isLoading } +``` + +**HTTP Client**: Axios with auth header injection + +**Query Key Pattern**: `[API_ROUTE, params]` + +### Component Organization + +``` +ui/src/ +├── components/ # Reusable UI components (filtering, forms, gallery) +├── design-system/ # Custom primitives (34 component directories) +│ ├── components/ # Button, Dialog, Table, etc. +│ └── map/ # Geospatial components +├── pages/ # Route-level components (21 page directories) +├── data-services/ # API integration +│ ├── hooks/ # React Query hooks (22 domain directories) +│ ├── models/ # Client-side model classes +│ └── constants.ts # API routes +└── utils/ # Shared utilities and contexts +``` + +### Styling +- **Primary**: Tailwind CSS v3.4.14 +- **Secondary**: SCSS modules for component-specific styles +- **Design Tokens**: `nova-ui-kit` library for colors and variables +- **Breakpoints**: SM (720px), MD (1024px), LG (1440px) + +### Key Dependencies +| Category | Libraries | +|----------|-----------| +| UI Components | Radix UI (7 packages), nova-ui-kit, lucide-react | +| Forms | react-hook-form v7.43.9 | +| Maps | leaflet + react-leaflet | +| Charts | plotly.js + react-plotly.js | +| Utilities | lodash, date-fns, classnames | +| Monitoring | @sentry/react | + +### Authentication +- **Method**: Token-based (djoser) +- **Storage**: localStorage +- **Header**: `Authorization: Token {token}` +- **Auto-logout**: On 403 responses + +--- + +## API Architecture + +### Base Configuration +- **URL**: `/api/v2/` +- **Framework**: Django REST Framework +- **Schema**: OpenAPI via drf-spectacular +- **Docs**: Swagger UI at `/api/v2/docs/` + +### Versioning +- Namespace versioning (`/api/v2/`) +- Single version currently deployed + +### Authentication Endpoints +``` +POST /api/v2/auth/token/login/ # Get token (email + password) +POST /api/v2/auth/token/logout/ # Logout +GET /api/v2/users/me/ # Current user +POST /api/v2/users/reset_password/ # Password reset +``` + +### REST Patterns +Standard CRUD for all resources: +``` +GET /api/v2/{resource}/ # List (paginated) +POST /api/v2/{resource}/ # Create +GET /api/v2/{resource}/{id}/ # Retrieve +PUT /api/v2/{resource}/{id}/ # Update +DELETE /api/v2/{resource}/{id}/ # Delete +``` + +### Key Resources +| Resource | Endpoint | Custom Actions | +|----------|----------|----------------| +| Projects | `/projects/` | `charts/` | +| Deployments | `/deployments/` | `sync/` | +| Events | `/events/` | `timeline/` | +| Captures | `/captures/` | `star/`, `unstar/` | +| Occurrences | `/occurrences/` | `add/`, `remove/`, `suggest/` | +| Jobs | `/jobs/` | `run/`, `cancel/`, `retry/`, `tasks/`, `result/` | +| Pipelines | `/ml/pipelines/` | `test_process/` | +| Processing Services | `/ml/processing_services/` | `status/`, `register_pipelines/` | + +### Pagination +- **Type**: Limit-Offset +- **Default Page Size**: 10 +- **Response**: `{ count, next, previous, results, user_permissions }` + +### Filtering +- Field filtering: `?field=value` +- Ordering: `?ordering=field` or `?ordering=-field` +- Search: `?search=query` +- Custom threshold filters for numeric comparisons + +### Nested Resources +``` +/api/v2/projects/{project_id}/members/ +``` + +--- + +## Current Pain Points + +needs rewriting - pain points are about where to put X, and relying more on external opensource community rather than writing custom aspects. +2. **No Edge/Middleware**: Can't intercept requests at edge +3. **Manual Caching**: No built-in data caching beyond React Query diff --git a/.agents/planning/02-nextjs-benefits.md b/.agents/planning/02-nextjs-benefits.md new file mode 100644 index 000000000..11e6bd537 --- /dev/null +++ b/.agents/planning/02-nextjs-benefits.md @@ -0,0 +1,294 @@ +# What Next.js Can Offer + +## Overview + +Next.js 14+ (App Router) provides a full-stack React framework that can replace multiple custom solutions with battle-tested, optimized built-ins. This document outlines what Next.js brings and what it can replace. + +--- + +## Stock Modules That Replace Custom Solutions + +### 1. Routing System + +**Current**: React Router v6 with manual configuration + +**Next.js Replacement**: File-based App Router + +| Feature | Current (React Router) | Next.js App Router | +|---------|----------------------|-------------------| +| Route Definition | Explicit in `app.tsx` | Folder structure (`app/`) | +| Nested Layouts | `` component | `layout.tsx` files | +| Route Groups | Manual organization | `(folder)` syntax | +| Parallel Routes | Not supported | `@folder` slots | +| Intercepting Routes | Complex modal patterns | `(.)` syntax | +| Loading States | Manual with React Query | `loading.tsx` files | +| Error Boundaries | Manual setup | `error.tsx` files | +| Not Found | Manual 404 handling | `not-found.tsx` files | + +**Migration Example:** +``` +# Current structure +src/pages/occurrences/occurrences.tsx +src/pages/occurrence-details/occurrence-details.tsx + +# Next.js structure +app/projects/[projectId]/occurrences/page.tsx +app/projects/[projectId]/occurrences/[id]/page.tsx +app/projects/[projectId]/occurrences/loading.tsx +app/projects/[projectId]/occurrences/error.tsx +``` + +### 2. Data Fetching + +**Current**: React Query + Axios + custom hooks + +**Next.js Options**: + +| Pattern | Use Case | Benefit | +|---------|----------|---------| +| Server Components | Initial data load | No client bundle, faster TTI | +| `fetch()` with caching | Server-side requests | Automatic deduplication | +| Server Actions | Mutations | Type-safe, no API routes needed | +| React Query (kept) | Real-time updates | Client-side caching | + +**Hybrid Approach** (Recommended): +```typescript +// Server Component for initial data +async function OccurrencesPage({ params }) { + const occurrences = await fetchOccurrences(params.projectId) + return +} + +// Client Component for interactivity +'use client' +function OccurrencesList({ initialData }) { + const { data } = useOccurrences({ initialData }) + // React Query handles updates, initial data from server +} +``` + +### 3. Image Optimization + +**Current**: Manual `` tags, no optimization + +**Next.js Replacement**: `next/image` component + +| Feature | Benefit | +|---------|---------| +| Automatic WebP/AVIF | Smaller file sizes | +| Lazy loading | Built-in with blur placeholder | +| Responsive images | `sizes` prop generates srcset | +| Remote optimization | Configurable `remotePatterns` | +| Priority loading | LCP optimization | + +**For Antenna**: Critical for gallery views with many insect images + +### 4. Metadata & SEO + +**Current**: react-helmet-async for dynamic titles + +**Next.js Replacement**: Metadata API + +```typescript +// Static metadata +export const metadata = { + title: 'Occurrences | Antenna', + description: 'Browse species occurrences', +} + +// Dynamic metadata +export async function generateMetadata({ params }) { + const project = await getProject(params.projectId) + return { title: `${project.name} | Antenna` } +} +``` + +### 5. Environment Variables + +**Current**: Vite's `VITE_` prefix, manual setup + +**Next.js Replacement**: Built-in env handling + +| Prefix | Visibility | Use Case | +|--------|------------|----------| +| `NEXT_PUBLIC_` | Client + Server | Public config | +| (none) | Server only | API keys, secrets | + +### 6. API Routes (Backend for Frontend) + +**Current**: Not available - all requests go directly to Django + +**Next.js Addition**: Route Handlers + +```typescript +// app/api/aggregate/route.ts +export async function GET(request: Request) { + // Aggregate multiple Django API calls + const [projects, user] = await Promise.all([ + fetch(`${DJANGO_URL}/api/v2/projects/`), + fetch(`${DJANGO_URL}/api/v2/users/me/`) + ]) + return Response.json({ projects, user }) +} +``` + +**Use Cases**: +- Aggregate multiple Django endpoints +- Add server-side caching layer +- Implement BFF patterns +- Add rate limiting or logging + +### 7. Middleware + +**Current**: Not available + +**Next.js Addition**: Edge Middleware + +```typescript +// middleware.ts +export function middleware(request: NextRequest) { + // Authentication check + const token = request.cookies.get('auth_token') + if (!token && request.nextUrl.pathname.startsWith('/projects')) { + return NextResponse.redirect(new URL('/auth/login', request.url)) + } +} +``` + +**Use Cases**: +- Authentication redirects +- Geolocation-based routing +- A/B testing +- Request logging + +### 8. Caching & Revalidation + +**Current**: React Query client-side caching only + +**Next.js Addition**: Multi-layer caching + +| Layer | Control | Use Case | +|-------|---------|----------| +| Data Cache | `fetch` options | API response caching | +| Full Route Cache | Static/dynamic | Pre-rendered pages | +| Router Cache | Automatic | Client-side navigation | + +```typescript +// Cache for 1 hour, revalidate in background +fetch(url, { next: { revalidate: 3600 } }) + +// Revalidate on demand +import { revalidatePath } from 'next/cache' +revalidatePath('/projects/[projectId]/occurrences') +``` + +--- + +## Performance Benefits + +### 1. Rendering Strategies + +| Strategy | When to Use | Antenna Use Case | +|----------|-------------|------------------| +| Static (SSG) | Content rarely changes | Terms, Code of Conduct | +| Server (SSR) | Personalized/fresh data | Project dashboards | +| Streaming | Large data sets | Occurrence lists | +| Client | Highly interactive | Identification dialogs | + +### 2. Code Splitting + +**Current**: Manual with React.lazy() + +**Next.js**: Automatic per-route code splitting + +- Each page only loads its required JavaScript +- Shared chunks extracted automatically +- Prefetching on link hover + +### 3. Font Optimization + +**Current**: Manual font loading + +**Next.js**: `next/font` +- Zero layout shift +- Automatic font subsetting +- Self-hosted (privacy) + +--- + +## Developer Experience Benefits + +### 1. TypeScript Integration +- Stricter route type safety +- Auto-generated types for params +- Server Action type inference + +### 2. Fast Refresh +- Comparable to Vite HMR +- State preservation +- Error overlay + +### 3. Built-in Dev Tools +- React DevTools integration +- Route inspector +- Build analysis (`next build --analyze`) + +### 4. Deployment +- Zero-config Vercel deployment +- Docker support +- Node.js server or static export + +--- + +## What Can Be Replaced vs Kept + +### Replace + +| Current | Next.js Replacement | +|---------|-------------------| +| React Router | App Router | +| react-helmet-async | Metadata API | +| Vite | Next.js bundler (Turbopack) | +| Manual image handling | next/image | +| Vite proxy config | rewrites in next.config.js | +| Manual route constants | File-based routing | +| Manual loading states | loading.tsx | +| Manual error boundaries | error.tsx | + +### Keep + +| Library | Reason | +|---------|--------| +| React Query | Real-time updates, optimistic mutations | +| react-hook-form | Form handling (no Next.js equivalent) | +| Tailwind CSS | Styling (works great with Next.js) | +| Radix UI | Accessible components | +| nova-ui-kit | Design tokens | +| Leaflet | Maps (no Next.js map solution) | +| Plotly | Charts | +| Axios | Can keep or switch to native fetch | +| lodash/date-fns | Utilities | + +### Evaluate + +| Library | Consideration | +|---------|--------------| +| Axios | Could switch to native fetch() for server components | +| Sentry | Use @sentry/nextjs instead | +| classnames | Could use clsx or Tailwind's cn() | + +--- + +## Feature Comparison Summary + +| Feature | Current Stack | Next.js | Improvement | +|---------|--------------|---------|-------------| +| Initial Load | Client render | Server render | 40-60% faster FCP | +| SEO | Poor (SPA) | Excellent | Crawlable content | +| Code Splitting | Manual | Automatic | Smaller bundles | +| Image Optimization | None | Built-in | Better Core Web Vitals | +| API Aggregation | None | Route Handlers | Reduce waterfalls | +| Caching | React Query | Multi-layer | Better performance | +| Loading States | Manual | Conventions | Less boilerplate | +| Error Handling | Manual | Conventions | Consistent UX | +| Type Safety | Good | Excellent | Safer routing | diff --git a/.agents/planning/03-required-changes.md b/.agents/planning/03-required-changes.md new file mode 100644 index 000000000..e02c7d077 --- /dev/null +++ b/.agents/planning/03-required-changes.md @@ -0,0 +1,565 @@ +# Everything That Needs to Change + +## Overview + +This document catalogs all changes required to migrate from the current Vite + React Router stack to Next.js App Router. Changes are categorized by scope and complexity. + +--- + +## 1. Project Structure + +### Directory Reorganization + +**Current Structure:** +``` +ui/ +├── src/ +│ ├── index.tsx # Entry point +│ ├── app.tsx # Router configuration +│ ├── pages/ # Page components +│ ├── components/ # Shared components +│ ├── design-system/ # UI primitives +│ ├── data-services/ # API hooks +│ └── utils/ # Utilities +├── vite.config.ts +├── tailwind.config.js +└── package.json +``` + +**Next.js Structure:** +``` +ui/ +├── app/ # App Router pages +│ ├── layout.tsx # Root layout +│ ├── page.tsx # Home page +│ ├── auth/ +│ │ ├── login/page.tsx +│ │ └── reset-password/page.tsx +│ ├── projects/ +│ │ ├── page.tsx +│ │ └── [projectId]/ +│ │ ├── layout.tsx +│ │ ├── page.tsx +│ │ ├── occurrences/ +│ │ │ ├── page.tsx +│ │ │ ├── loading.tsx +│ │ │ └── [id]/page.tsx +│ │ └── ... +│ └── api/ # Route handlers (optional BFF) +├── components/ # Shared components +├── design-system/ # UI primitives (unchanged) +├── data-services/ # API hooks (modified) +├── utils/ # Utilities (modified) +├── next.config.js +├── tailwind.config.js +└── package.json +``` + +### Files to Delete +- `vite.config.ts` +- `vite-env.d.ts` +- `src/index.tsx` (replaced by `app/layout.tsx`) +- `src/app.tsx` (router config moves to file system) + +### Files to Create +- `next.config.js` +- `middleware.ts` (for auth protection) +- `app/layout.tsx` (root layout with providers) +- `app/*/page.tsx` (for each route) +- `app/*/layout.tsx` (for nested layouts) +- `app/*/loading.tsx` (loading states) +- `app/*/error.tsx` (error boundaries) + +--- + +## 2. Routing Changes + +### Route Mapping + +| Current Route | Next.js Path | +|---------------|--------------| +| `/` | `app/page.tsx` | +| `/auth/login` | `app/auth/login/page.tsx` | +| `/auth/reset-password` | `app/auth/reset-password/page.tsx` | +| `/auth/reset-password-confirm` | `app/auth/reset-password-confirm/page.tsx` | +| `/projects` | `app/projects/page.tsx` | +| `/projects/:projectId` | `app/projects/[projectId]/page.tsx` | +| `/projects/:projectId/summary` | `app/projects/[projectId]/summary/page.tsx` | +| `/projects/:projectId/occurrences` | `app/projects/[projectId]/occurrences/page.tsx` | +| `/projects/:projectId/occurrences/:id` | `app/projects/[projectId]/occurrences/[id]/page.tsx` | +| `/terms-of-service` | `app/terms-of-service/page.tsx` | + +### Layout Changes + +**Current** (React Router Outlet): +```tsx +// Project layout with outlet + + + +``` + +**Next.js** (Nested Layouts): +```tsx +// app/projects/[projectId]/layout.tsx +export default function ProjectLayout({ children, params }) { + return ( + + +
{children}
+ + ) +} +``` + +### Navigation Changes + +**Current** (React Router): +```tsx +import { useNavigate, Link, useParams } from 'react-router-dom' + +const navigate = useNavigate() +navigate('/projects/123/occurrences') + + +``` + +**Next.js**: +```tsx +import { useRouter, useParams } from 'next/navigation' +import Link from 'next/link' + +const router = useRouter() +router.push('/projects/123/occurrences') + + +``` + +### Route Constants + +**Current** (`utils/constants.ts`): +```typescript +export const STRING = { + PROJECTS: '/projects', + OCCURRENCES: (projectId: string) => `/projects/${projectId}/occurrences`, +} +``` + +**Next.js** (can keep similar pattern or use type-safe routes): +```typescript +// Consider using next-safe-navigation or similar +export const routes = { + projects: () => '/projects', + occurrences: (projectId: string) => `/projects/${projectId}/occurrences`, +} as const +``` + +--- + +## 3. Data Fetching Changes + +### Server Components (New Pattern) + +**Pages with initial data fetch:** +```tsx +// app/projects/[projectId]/occurrences/page.tsx +import { cookies } from 'next/headers' + +async function getOccurrences(projectId: string) { + const token = cookies().get('auth_token')?.value + const res = await fetch(`${API_URL}/occurrences/?project=${projectId}`, { + headers: { Authorization: `Token ${token}` }, + next: { revalidate: 60 }, // Cache for 60 seconds + }) + return res.json() +} + +export default async function OccurrencesPage({ params }) { + const data = await getOccurrences(params.projectId) + return +} +``` + +### Client Components (Modified Pattern) + +**Current:** +```tsx +export const Occurrences = () => { + const { occurrences, isLoading } = useOccurrences(params) + if (isLoading) return + return +} +``` + +**Next.js:** +```tsx +'use client' + +export function OccurrencesClient({ initialData }) { + const { data } = useOccurrences({ + initialData, // Hydrate from server + refetchOnMount: false, + }) + return +} +``` + +### React Query Configuration + +**Current** (`index.tsx`): +```tsx +const queryClient = new QueryClient() + + + + +``` + +**Next.js** (`app/providers.tsx`): +```tsx +'use client' + +export function Providers({ children }) { + const [queryClient] = useState(() => new QueryClient({ + defaultOptions: { + queries: { + staleTime: 60 * 1000, + refetchOnWindowFocus: false, + }, + }, + })) + + return ( + + {children} + + ) +} +``` + +--- + +## 4. Authentication Changes + +### Token Storage + +**Current**: localStorage only + +**Next.js**: Cookies (for SSR access) + localStorage fallback + +```typescript +// middleware.ts +export function middleware(request: NextRequest) { + const token = request.cookies.get('auth_token') + + if (!token && isProtectedRoute(request.nextUrl.pathname)) { + return NextResponse.redirect(new URL('/auth/login', request.url)) + } +} + +export const config = { + matcher: ['/projects/:path*'], +} +``` + +### Auth Context Changes + +**Current:** +```tsx +const setToken = (token: string) => { + localStorage.setItem('AUTH_TOKEN', token) + setUser({ loggedIn: true, token }) +} +``` + +**Next.js:** +```tsx +const setToken = async (token: string) => { + // Set cookie for SSR + document.cookie = `auth_token=${token}; path=/; max-age=2592000` + // Also keep in localStorage for client-side access + localStorage.setItem('AUTH_TOKEN', token) + setUser({ loggedIn: true, token }) +} +``` + +--- + +## 5. Component Changes + +### 'use client' Directive + +Components that need client-side features must be marked: + +**Components requiring 'use client':** +- All components using `useState`, `useEffect`, `useContext` +- Components with event handlers (onClick, onChange, etc.) +- Components using browser APIs (localStorage, window) +- Components using React Query hooks +- Components using react-hook-form +- Interactive UI components (dialogs, dropdowns, forms) + +**Components that can be Server Components:** +- Static display components +- Layout wrappers +- Components that only receive props + +### Component Audit (Sample) + +| Component | Client/Server | Reason | +|-----------|--------------|--------| +| `Header` | Client | useState, navigation | +| `Sidebar` | Client | useState, context | +| `OccurrenceTable` | Client | useQuery, onClick | +| `OccurrenceRow` | Server | Just displays props | +| `ProjectLayout` | Server | Layout wrapper | +| `Gallery` | Client | useState, event handlers | +| `Dialog` | Client | Radix UI (interactive) | +| `Button` | Client | onClick handler | +| `StatusBadge` | Server | Static display | + +--- + +## 6. Styling Changes + +### Tailwind Configuration + +**Changes needed:** +```javascript +// tailwind.config.js +module.exports = { + content: [ + './app/**/*.{js,ts,jsx,tsx,mdx}', // Add app directory + './components/**/*.{js,ts,jsx,tsx}', + './design-system/**/*.{js,ts,jsx,tsx}', + // Remove src/pages, src/components paths + ], +} +``` + +### CSS Imports + +**Current** (`index.css`): +```css +@tailwind base; +@tailwind components; +@tailwind utilities; +``` + +**Next.js** (`app/globals.css`): +```css +@tailwind base; +@tailwind components; +@tailwind utilities; + +/* Same custom styles */ +``` + +Import in root layout: +```tsx +// app/layout.tsx +import './globals.css' +``` + +--- + +## 7. Image Handling + +### Current Image Usage + +```tsx +{occurrence.name} +``` + +### Next.js Image Component + +```tsx +import Image from 'next/image' + +{occurrence.name} +``` + +### Remote Image Configuration + +```javascript +// next.config.js +module.exports = { + images: { + remotePatterns: [ + { + protocol: 'https', + hostname: 'minio', // MinIO storage + }, + { + protocol: 'http', + hostname: 'localhost', + }, + ], + }, +} +``` + +--- + +## 8. Environment Variables + +### Rename Variables + +| Current | Next.js | +|---------|---------| +| `VITE_DOCS_URL` | `NEXT_PUBLIC_DOCS_URL` | +| `API_PROXY_TARGET` | Server-side config in `next.config.js` | + +### next.config.js Rewrites + +```javascript +// next.config.js +module.exports = { + async rewrites() { + return [ + { + source: '/api/:path*', + destination: `${process.env.API_URL || 'http://localhost:8000'}/api/:path*`, + }, + { + source: '/media/:path*', + destination: `${process.env.API_URL || 'http://localhost:8000'}/media/:path*`, + }, + ] + }, +} +``` + +--- + +## 9. Package Changes + +### Remove +```json +{ + "vite": "^4.5.3", + "@vitejs/plugin-react": "^4.2.0", + "vite-tsconfig-paths": "^4.2.1", + "vite-plugin-svgr": "^4.2.0", + "vite-plugin-eslint": "^1.8.1", + "react-router-dom": "^6.8.2", + "react-helmet-async": "^2.0.5" +} +``` + +### Add +```json +{ + "next": "^14.2.0", + "@next/bundle-analyzer": "^14.2.0" +} +``` + +### Update +```json +{ + "@sentry/react": "→ @sentry/nextjs" +} +``` + +--- + +## 10. TypeScript Changes + +### tsconfig.json + +```json +{ + "compilerOptions": { + "target": "ES2017", + "lib": ["dom", "dom.iterable", "esnext"], + "allowJs": true, + "skipLibCheck": true, + "strict": true, + "noEmit": true, + "esModuleInterop": true, + "module": "esnext", + "moduleResolution": "bundler", + "resolveJsonModule": true, + "isolatedModules": true, + "jsx": "preserve", + "incremental": true, + "plugins": [ + { "name": "next" } + ], + "paths": { + "@/*": ["./*"] + } + }, + "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"], + "exclude": ["node_modules"] +} +``` + +--- + +## 11. Testing Changes + +### Jest Configuration + +```javascript +// jest.config.js +const nextJest = require('next/jest') + +const createJestConfig = nextJest({ + dir: './', +}) + +const customJestConfig = { + setupFilesAfterEnv: ['/jest.setup.js'], + testEnvironment: 'jest-environment-jsdom', + moduleNameMapper: { + '^@/(.*)$': '/$1', + }, +} + +module.exports = createJestConfig(customJestConfig) +``` + +### Test Updates + +- Update imports for Next.js components +- Mock `next/navigation` instead of `react-router-dom` +- Use `@testing-library/react` with Next.js utilities + +--- + +## Change Summary by Effort + +### Low Effort (Configuration) +- [ ] Create next.config.js +- [ ] Update tailwind.config.js content paths +- [ ] Update tsconfig.json +- [ ] Rename environment variables +- [ ] Update package.json dependencies + +### Medium Effort (Structural) +- [ ] Create app directory structure +- [ ] Move pages to app/*/page.tsx format +- [ ] Create layout.tsx files +- [ ] Add 'use client' directives +- [ ] Update imports (Link, useRouter, etc.) + +### High Effort (Architectural) +- [ ] Implement server components for data fetching +- [ ] Update authentication to use cookies +- [ ] Create middleware for auth protection +- [ ] Modify React Query for SSR hydration +- [ ] Update all tests + +### Estimated File Changes +- **New files**: ~50 (layouts, pages, configs) +- **Modified files**: ~100 (components with imports/directives) +- **Deleted files**: ~10 (Vite config, old entry points) diff --git a/.agents/planning/04-custom-bespoke.md b/.agents/planning/04-custom-bespoke.md new file mode 100644 index 000000000..cc29e3395 --- /dev/null +++ b/.agents/planning/04-custom-bespoke.md @@ -0,0 +1,441 @@ +# What Stays Custom/Bespoke + +## Overview + +While Next.js provides many built-in features, significant portions of the Antenna application are domain-specific and will remain custom. This document identifies components and patterns that cannot be replaced by Next.js stock modules. + +--- + +## 1. Domain Models & Business Logic + +### Client-Side Model Classes + +These TypeScript classes encapsulate business logic specific to Antenna: + +``` +data-services/models/ +├── occurrence.ts # Occurrence entity with computed properties +├── detection.ts # Detection bounding box logic +├── classification.ts # ML classification results +├── identification.ts # Human identification workflow +├── deployment.ts # Monitoring station logic +├── event.ts # Temporal event grouping +├── job.ts # ML job state machine +├── pipeline.ts # ML pipeline configuration +└── project.ts # Project with member permissions +``` + +**Key Custom Logic Examples:** + +```typescript +// Occurrence model - domain-specific computed properties +class Occurrence { + get displayName(): string { /* taxonomy display logic */ } + get bestClassification(): Classification { /* ranking logic */ } + userAgreed(userId: string, taxonId?: string): boolean { /* agreement logic */ } + hasConflictingIdentifications(): boolean { /* conflict detection */ } +} + +// Job model - state machine for ML jobs +class Job { + canStart(): boolean { /* state validation */ } + canCancel(): boolean { /* running job check */ } + getProgressPercentage(): number { /* stage calculation */ } +} +``` + +**Migration**: These stay exactly as-is. No Next.js equivalent exists. + +--- + +## 2. Custom Design System + +### nova-ui-kit Integration + +The design token system from `nova-ui-kit` is deeply integrated: + +``` +design-system/ +├── components/ +│ ├── button/ # Custom button variants +│ ├── dialog/ # Modal dialogs +│ ├── table/ # Data tables with sorting +│ ├── gallery/ # Image gallery grid +│ ├── pagination/ # Custom pagination +│ ├── icon-button/ # Icon actions +│ ├── tooltip/ # Hover tooltips +│ ├── select/ # Dropdown selects +│ ├── tabs/ # Tab navigation +│ ├── accordion/ # Collapsible sections +│ └── ... (34 component types) +└── variables/ + └── variables.scss # SCSS variables +``` + +**Migration**: All 34+ design system components remain unchanged. They're framework-agnostic React components that work with Next.js. + +### Radix UI Wrappers + +Custom wrappers around Radix primitives with Antenna-specific styling: +- `Dialog` - Project dialogs with specific layouts +- `DropdownMenu` - Action menus with icons +- `Select` - Styled selects with search +- `Tooltip` - Consistent tooltip styling +- `Accordion` - Settings accordions + +--- + +## 3. ML Pipeline Visualization + +### Job Progress UI + +Complex visualization of ML pipeline stages: + +``` +pages/job-details/ +├── job-details.tsx # Main job view +├── job-stage-label/ # Stage status indicators +│ └── job-stage-label.tsx +└── job-actions/ + ├── queue-job.tsx # Start job action + ├── cancel-job.tsx # Cancel running job + └── retry-job.tsx # Retry failed job +``` + +**Custom Logic:** +- Real-time progress polling +- Stage-by-stage progress visualization +- Error display with stack traces +- Log streaming from Celery + +### Pipeline Configuration + +``` +pages/pipeline-details/ +├── pipeline-details-dialog.tsx +├── pipeline-stages.tsx # Stage configuration +└── pipeline-algorithms.tsx # Algorithm selection +``` + +--- + +## 4. Species Identification System + +### Identification Workflow + +The core ML identification workflow is entirely custom: + +``` +pages/occurrence-details/ +├── occurrence-details.tsx # Detail view with all info +├── agree/ +│ └── agree.tsx # Agreement workflow +├── suggest-id/ +│ ├── suggest-id.tsx # Taxa suggestion with autocomplete +│ └── suggest-id-popover.tsx # Suggestion dropdown +├── identification-card/ +│ ├── human-identification.tsx # Human ID display +│ └── machine-prediction.tsx # ML prediction display +├── id-quick-actions/ +│ ├── id-button.tsx # Quick ID buttons +│ └── id-quick-actions.tsx # Action bar +├── status-label/ +│ └── status-label.tsx # Verification status +└── taxonomy-info/ + └── taxonomy-info.tsx # Taxonomic hierarchy display +``` + +**Custom Features:** +- Taxa autocomplete with fuzzy matching +- Agreement/disagreement workflow +- Confidence score visualization +- Taxonomy tree navigation +- ML prediction comparison + +--- + +## 5. Geospatial Components + +### Map Integration + +Leaflet-based mapping for deployment locations: + +``` +design-system/map/ +├── map.tsx # Base map component +├── map-controls.tsx # Zoom, layers +└── map-markers.tsx # Deployment markers + +pages/deployment-details/ +└── deployment-details-form/ + └── section-location/ + ├── location-map/ + │ └── location-map.tsx # Editable location picker + └── geo-search/ + └── geo-search.tsx # Address search +``` + +**Custom Features:** +- Deployment location picker with drag-and-drop +- Geosearch integration +- Custom marker clustering for multiple deployments +- Satellite/terrain layer switching + +--- + +## 6. Image Gallery System + +### Gallery Components + +Custom gallery for browsing capture images: + +``` +components/gallery/ +├── gallery.tsx # Main gallery grid +├── gallery-item.tsx # Individual image card +├── gallery-filters.tsx # Filter controls +└── gallery-navigation.tsx # Keyboard navigation + +pages/occurrences/ +├── occurrence-gallery.tsx # Occurrence thumbnail grid +└── occurrence-navigation.tsx # Prev/next navigation + +pages/captures/ +├── capture-gallery.tsx # Source image gallery +└── capture-columns.tsx # Table/gallery toggle +``` + +**Custom Features:** +- Infinite scroll with virtualization +- Bounding box overlays on images +- Thumbnail generation handling +- Keyboard navigation (arrow keys) +- Selection mode for batch operations + +--- + +## 7. Filtering System + +### Complex Filter Components + +Domain-specific filtering for occurrences and captures: + +``` +components/filtering/ +├── filter-panel.tsx # Main filter container +├── filter-chips.tsx # Active filter display +├── filter-presets.tsx # Saved filter presets +├── filters/ +│ ├── taxon-filter.tsx # Taxonomy tree filter +│ ├── score-filter.tsx # Confidence threshold +│ ├── date-range-filter.tsx # Temporal filtering +│ ├── deployment-filter.tsx # Location filtering +│ ├── event-filter.tsx # Event grouping +│ └── status-filter.tsx # Verification status +``` + +**Custom Features:** +- Hierarchical taxa filtering with tree navigation +- Score threshold sliders +- Date range pickers with presets +- Multi-select deployment picker +- Filter persistence in URL params + +### Filter Hooks + +```typescript +// utils/useFilters.ts +export function useFilters() { + // URL-synchronized filter state + // Filter combination logic + // Clear/reset functionality +} + +// utils/useSort.ts +export function useSort() { + // Column sorting state + // Multi-column sort support +} + +// utils/usePagination.ts +export function usePagination() { + // Offset-based pagination + // Page size preferences +} +``` + +--- + +## 8. Data Export System + +### Export Configuration + +``` +pages/project/exports/ +├── exports.tsx # Export list +└── exports-columns.tsx # Export status table + +pages/export-details/ +└── export-details-dialog.tsx # Export configuration +``` + +**Custom Features:** +- Format selection (CSV, JSON, Darwin Core) +- Field mapping configuration +- Progress tracking for large exports +- Download management + +--- + +## 9. Processing Service Management + +### Service Health Monitoring + +``` +pages/processing-service-details/ +├── processing-service-details-dialog.tsx +└── processing-service-pipelines.tsx + +pages/project/processing-services/ +├── processing-services.tsx +├── processing-services-columns.tsx +├── connection-status.tsx # Health indicator +└── status-info/ + └── status-info.tsx # Detailed status +``` + +**Custom Features:** +- Real-time health check polling +- Pipeline registration workflow +- Error diagnostics display +- Service URL configuration + +--- + +## 10. Project Configuration + +### Settings Pages + +``` +pages/project/ +├── general/general.tsx # Basic settings +├── team/ # Member management (nested route) +├── default-filters/ # Default filter config +├── storage/ # S3 storage config +└── processing/ # ML processing settings +``` + +### Forms + +``` +pages/project-details/ +├── default-filters-form.tsx # Taxa inclusion/exclusion +├── processing-form.tsx # Pipeline defaults +├── pipelines-select.tsx # Pipeline multi-select +└── project-details-form.tsx # General settings +``` + +--- + +## 11. React Query Hooks (Modified but Custom) + +All domain-specific data fetching hooks remain custom: + +``` +data-services/hooks/ +├── algorithms/useAlgorithms.ts +├── captures/useCaptures.ts +├── classifications/useClassifications.ts +├── collections/useCollections.ts +├── deployments/useDeployments.ts +├── detections/useDetections.ts +├── events/useEvents.ts +├── exports/useExports.ts +├── identifications/useIdentifications.ts +├── jobs/useJobs.ts +├── members/useMembers.ts +├── occurrences/useOccurrences.ts +├── pipelines/usePipelines.ts +├── processing-services/useProcessingServices.ts +├── projects/useProjects.ts +├── sites/useSites.ts +├── storage/useStorage.ts +├── taxa/useTaxa.ts +└── users/useUsers.ts +``` + +**Modification**: These hooks will be adapted for SSR hydration but business logic remains. + +--- + +## 12. Utility Functions + +### Domain Utilities + +```typescript +// Format helpers +formatTaxonName(taxon: Taxon): string +formatConfidenceScore(score: number): string +formatTimestamp(date: Date, format: string): string + +// Validation helpers +validateDeploymentLocation(lat: number, lng: number): boolean +validateTaxonSelection(taxon: Taxon, project: Project): boolean + +// Transformation helpers +convertServerOccurrence(data: ServerOccurrence): Occurrence +buildFilterQueryString(filters: Filter[]): string +``` + +--- + +## Summary: Custom vs Stock + +### Stays 100% Custom +| Category | Components | Reason | +|----------|------------|--------| +| Domain Models | 10+ classes | Business logic | +| Design System | 34+ components | Brand/UX | +| ML Visualization | Job progress, pipelines | Domain-specific | +| Identification UI | Agree/suggest/taxonomy | Core workflow | +| Maps | Leaflet integration | No Next.js equivalent | +| Gallery | Image grid/navigation | Custom UX | +| Filtering | Taxa/score/date filters | Domain-specific | +| Export | Configuration/progress | Domain-specific | +| React Query Hooks | 20+ hooks | API integration | + +### Modified but Mostly Custom +| Category | Change | +|----------|--------| +| Page components | Add 'use client', update imports | +| Contexts | Add cookie support for SSR | +| Data hooks | Add SSR hydration support | + +### Replaced by Next.js +| Category | Replacement | +|----------|-------------| +| Router config | File-based routing | +| Route constants | Can simplify | +| Loading states | loading.tsx | +| Error boundaries | error.tsx | +| Metadata | Metadata API | +| Image tags | next/image (partial) | + +--- + +## Effort Estimate by Category + +| Category | Files | Effort | Notes | +|----------|-------|--------|-------| +| Domain Models | 10 | None | No changes needed | +| Design System | 50+ | Low | Add 'use client' where needed | +| Page Components | 21 | Medium | Convert to page.tsx format | +| Data Hooks | 22 | Medium | Add SSR support | +| Utilities | 15 | Low | Mostly unchanged | +| Contexts | 5 | Medium | Cookie + localStorage | +| Filters | 10 | Low | Unchanged | +| Maps | 5 | Low | Add 'use client' | +| Gallery | 8 | Low | Add 'use client' | + +**Total custom code preserved**: ~85% of application logic +**Total code requiring modification**: ~40% (mostly adding directives) diff --git a/.agents/planning/05-migration-steps.md b/.agents/planning/05-migration-steps.md new file mode 100644 index 000000000..d1d536c0a --- /dev/null +++ b/.agents/planning/05-migration-steps.md @@ -0,0 +1,815 @@ +# Migration Steps + +## Overview + +This document outlines a phased approach to migrating the Antenna UI from Vite + React Router to Next.js App Router. The migration is designed to be incremental, allowing the application to remain functional throughout. + +--- + +## Phase 0: Preparation + +### 0.1 Create Migration Branch +```bash +git checkout -b feature/nextjs-migration +``` + +### 0.2 Audit Current Application +- [ ] Document all routes and their data dependencies +- [ ] Identify components using browser APIs (localStorage, window) +- [ ] List all third-party libraries and check Next.js compatibility +- [ ] Note any Vite-specific features in use (import.meta, VITE_ env vars) +- [ ] Review current test coverage + +### 0.3 Set Up Development Environment +- [ ] Ensure Node.js 18.17+ is installed +- [ ] Review Docker Compose configuration for Next.js support +- [ ] Plan CI/CD changes + +### 0.4 Create Compatibility Checklist + +| Library | Status | Action Needed | +|---------|--------|---------------| +| react-query | ✅ | Works with Next.js | +| react-hook-form | ✅ | Works with Next.js | +| tailwindcss | ✅ | Works with Next.js | +| radix-ui | ✅ | Works with Next.js | +| leaflet | ⚠️ | Needs 'use client' | +| plotly | ⚠️ | Needs 'use client' | +| sentry | ⚠️ | Switch to @sentry/nextjs | +| nova-ui-kit | ✅ | Works with Next.js | + +--- + +## Phase 1: Parallel Setup + +### 1.1 Initialize Next.js Alongside Vite + +Keep existing Vite setup while adding Next.js: + +```bash +cd ui + +# Install Next.js dependencies +yarn add next@14 + +# Create Next.js config +touch next.config.js +``` + +### 1.2 Create next.config.js + +```javascript +/** @type {import('next').NextConfig} */ +const nextConfig = { + // Preserve existing paths during migration + experimental: { + // Enable if needed for gradual migration + }, + + // Proxy API requests to Django (like Vite did) + async rewrites() { + const apiUrl = process.env.API_URL || 'http://localhost:8000' + return [ + { + source: '/api/:path*', + destination: `${apiUrl}/api/:path*`, + }, + { + source: '/media/:path*', + destination: `${apiUrl}/media/:path*`, + }, + ] + }, + + // Image optimization config + images: { + remotePatterns: [ + { + protocol: 'http', + hostname: 'localhost', + }, + { + protocol: 'http', + hostname: 'minio', + }, + { + protocol: 'https', + hostname: '*.insectai.org', + }, + ], + }, + + // Transpile nova-ui-kit if needed + transpilePackages: ['nova-ui-kit'], +} + +module.exports = nextConfig +``` + +### 1.3 Update package.json Scripts + +```json +{ + "scripts": { + "dev": "vite", + "dev:next": "next dev -p 3001", + "build": "vite build", + "build:next": "next build", + "start:next": "next start", + "lint": "eslint . --ext .ts,.tsx", + "test": "jest" + } +} +``` + +### 1.4 Create App Directory Structure + +```bash +mkdir -p app +touch app/layout.tsx +touch app/page.tsx +touch app/globals.css +touch app/providers.tsx +``` + +### 1.5 Create Root Layout + +```typescript +// app/layout.tsx +import type { Metadata } from 'next' +import { Providers } from './providers' +import './globals.css' + +export const metadata: Metadata = { + title: 'Antenna - Insect Monitoring Platform', + description: 'Automated Monitoring of Insects ML Platform', +} + +export default function RootLayout({ + children, +}: { + children: React.ReactNode +}) { + return ( + + + {children} + + + ) +} +``` + +### 1.6 Create Client Providers + +```typescript +// app/providers.tsx +'use client' + +import { QueryClient, QueryClientProvider } from '@tanstack/react-query' +import { ReactQueryDevtools } from '@tanstack/react-query-devtools' +import { useState } from 'react' +import { UserContextProvider } from '@/utils/user/userContext' +import { UserInfoContextProvider } from '@/utils/user/userInfoContext' +import { UserPreferencesContextProvider } from '@/utils/userPreferences/userPreferencesContext' +import { BreadcrumbContextProvider } from '@/utils/breadcrumbContext' + +export function Providers({ children }: { children: React.ReactNode }) { + const [queryClient] = useState( + () => + new QueryClient({ + defaultOptions: { + queries: { + staleTime: 60 * 1000, + refetchOnWindowFocus: false, + }, + }, + }) + ) + + return ( + + + + + + {children} + + + + + + + ) +} +``` + +### 1.7 Move Global Styles + +```css +/* app/globals.css */ +@tailwind base; +@tailwind components; +@tailwind utilities; + +/* Copy from existing src/index.css */ +fieldset, li, ul, ol { + all: unset; +} + +a { + display: inline-block; + text-decoration: none; +} + +/* ... rest of global styles */ +``` + +--- + +## Phase 2: Core Infrastructure Migration + +### 2.1 Update TypeScript Configuration + +```json +// tsconfig.json +{ + "compilerOptions": { + "target": "ES2017", + "lib": ["dom", "dom.iterable", "esnext"], + "allowJs": true, + "skipLibCheck": true, + "strict": true, + "noEmit": true, + "esModuleInterop": true, + "module": "esnext", + "moduleResolution": "bundler", + "resolveJsonModule": true, + "isolatedModules": true, + "jsx": "preserve", + "incremental": true, + "plugins": [{ "name": "next" }], + "baseUrl": ".", + "paths": { + "@/*": ["./*"] + } + }, + "include": [ + "next-env.d.ts", + "**/*.ts", + "**/*.tsx", + ".next/types/**/*.ts" + ], + "exclude": ["node_modules"] +} +``` + +### 2.2 Update Tailwind Configuration + +```javascript +// tailwind.config.js +module.exports = { + content: [ + './app/**/*.{js,ts,jsx,tsx,mdx}', + './components/**/*.{js,ts,jsx,tsx}', + './design-system/**/*.{js,ts,jsx,tsx}', + './pages/**/*.{js,ts,jsx,tsx}', // Keep during migration + './src/**/*.{js,ts,jsx,tsx}', // Keep during migration + './node_modules/nova-ui-kit/**/*.{js,ts,jsx,tsx}', + ], + // ... rest of config unchanged +} +``` + +### 2.3 Create Authentication Middleware + +```typescript +// middleware.ts +import { NextResponse } from 'next/server' +import type { NextRequest } from 'next/server' + +export function middleware(request: NextRequest) { + const token = request.cookies.get('auth_token')?.value + const { pathname } = request.nextUrl + + // Public routes that don't require auth + const publicRoutes = [ + '/auth/login', + '/auth/reset-password', + '/auth/reset-password-confirm', + '/terms-of-service', + '/code-of-conduct', + ] + + const isPublicRoute = publicRoutes.some(route => + pathname.startsWith(route) + ) + + // Redirect to login if accessing protected route without token + if (!token && !isPublicRoute && pathname !== '/') { + return NextResponse.redirect(new URL('/auth/login', request.url)) + } + + // Redirect to projects if accessing login while authenticated + if (token && pathname === '/auth/login') { + return NextResponse.redirect(new URL('/projects', request.url)) + } + + return NextResponse.next() +} + +export const config = { + matcher: [ + '/((?!api|_next/static|_next/image|favicon.ico|media).*)', + ], +} +``` + +### 2.4 Update Auth Context for Cookies + +```typescript +// utils/user/userContext.tsx +'use client' + +import { createContext, useContext, useState, useCallback } from 'react' +import Cookies from 'js-cookie' // Add: yarn add js-cookie @types/js-cookie + +const AUTH_TOKEN_KEY = 'auth_token' + +interface UserContextType { + user: { loggedIn: boolean; token?: string } + setToken: (token: string) => void + clearToken: () => void +} + +export function UserContextProvider({ children }) { + const [user, setUser] = useState(() => { + // Check both cookie and localStorage for token + const cookieToken = Cookies.get(AUTH_TOKEN_KEY) + const localToken = typeof window !== 'undefined' + ? localStorage.getItem(AUTH_TOKEN_KEY) + : null + const token = cookieToken || localToken + return { loggedIn: !!token, token } + }) + + const setToken = useCallback((token: string) => { + // Set cookie for SSR access + Cookies.set(AUTH_TOKEN_KEY, token, { + expires: 30, // 30 days + path: '/', + sameSite: 'lax' + }) + // Also set localStorage for client-side + localStorage.setItem(AUTH_TOKEN_KEY, token) + setUser({ loggedIn: true, token }) + }, []) + + const clearToken = useCallback(() => { + Cookies.remove(AUTH_TOKEN_KEY) + localStorage.removeItem(AUTH_TOKEN_KEY) + setUser({ loggedIn: false, token: undefined }) + }, []) + + return ( + + {children} + + ) +} +``` + +--- + +## Phase 3: Route-by-Route Migration + +### Migration Order (by complexity, low to high): + +1. **Static pages** (easiest) +2. **Auth pages** (simple forms) +3. **List pages** (projects, deployments) +4. **Detail pages** (with params) +5. **Modal routes** (intercepting routes) +6. **Complex pages** (occurrences with filters) + +### 3.1 Migrate Static Pages First + +```typescript +// app/terms-of-service/page.tsx +import { Metadata } from 'next' + +export const metadata: Metadata = { + title: 'Terms of Service | Antenna', +} + +export default function TermsOfServicePage() { + return ( +
+ {/* Import and render existing component */} + +
+ ) +} +``` + +### 3.2 Migrate Auth Pages + +```typescript +// app/auth/login/page.tsx +import { Metadata } from 'next' +import { LoginForm } from '@/pages/auth/login' + +export const metadata: Metadata = { + title: 'Login | Antenna', +} + +export default function LoginPage() { + return +} +``` + +Update login component with 'use client': +```typescript +// pages/auth/login.tsx (or move to components/) +'use client' + +import { useRouter } from 'next/navigation' // Changed from react-router-dom +// ... rest of component +``` + +### 3.3 Migrate Project List + +```typescript +// app/projects/page.tsx +import { Metadata } from 'next' +import { cookies } from 'next/headers' +import { ProjectsList } from '@/components/projects/projects-list' + +export const metadata: Metadata = { + title: 'Projects | Antenna', +} + +async function getProjects() { + const token = cookies().get('auth_token')?.value + const res = await fetch(`${process.env.API_URL}/api/v2/projects/`, { + headers: { Authorization: `Token ${token}` }, + next: { revalidate: 60 }, + }) + if (!res.ok) return null + return res.json() +} + +export default async function ProjectsPage() { + const initialData = await getProjects() + return +} +``` + +### 3.4 Create Project Layout + +```typescript +// app/projects/[projectId]/layout.tsx +import { cookies } from 'next/headers' +import { ProjectLayoutClient } from '@/components/project/project-layout-client' + +async function getProject(projectId: string) { + const token = cookies().get('auth_token')?.value + const res = await fetch( + `${process.env.API_URL}/api/v2/projects/${projectId}/`, + { + headers: { Authorization: `Token ${token}` }, + next: { revalidate: 60 }, + } + ) + if (!res.ok) return null + return res.json() +} + +export default async function ProjectLayout({ + children, + params, +}: { + children: React.ReactNode + params: { projectId: string } +}) { + const project = await getProject(params.projectId) + + return ( + + {children} + + ) +} +``` + +### 3.5 Create Loading States + +```typescript +// app/projects/[projectId]/occurrences/loading.tsx +import { Skeleton } from '@/design-system/components/skeleton' + +export default function OccurrencesLoading() { + return ( +
+ +
+ {Array.from({ length: 12 }).map((_, i) => ( + + ))} +
+
+ ) +} +``` + +### 3.6 Create Error Boundaries + +```typescript +// app/projects/[projectId]/occurrences/error.tsx +'use client' + +import { useEffect } from 'react' +import { Button } from '@/design-system/components/button' + +export default function OccurrencesError({ + error, + reset, +}: { + error: Error & { digest?: string } + reset: () => void +}) { + useEffect(() => { + console.error(error) + }, [error]) + + return ( +
+

+ Something went wrong loading occurrences +

+ +
+ ) +} +``` + +### 3.7 Migrate Modal Routes (Intercepting Routes) + +For occurrence details modal that opens over the list: + +``` +app/projects/[projectId]/occurrences/ +├── page.tsx # List view +├── loading.tsx # List loading +├── [id]/ +│ └── page.tsx # Full page detail +├── @modal/ +│ └── (.)[id]/ +│ └── page.tsx # Modal detail (intercepts) +└── layout.tsx # Handles modal slot +``` + +```typescript +// app/projects/[projectId]/occurrences/layout.tsx +export default function OccurrencesLayout({ + children, + modal, +}: { + children: React.ReactNode + modal: React.ReactNode +}) { + return ( + <> + {children} + {modal} + + ) +} + +// app/projects/[projectId]/occurrences/@modal/(.)][id]/page.tsx +import { OccurrenceDetailsModal } from '@/components/occurrence-details-modal' + +export default function OccurrenceModal({ + params, +}: { + params: { id: string } +}) { + return +} +``` + +--- + +## Phase 4: Component Updates + +### 4.1 Add 'use client' Directives + +Create script to identify components needing 'use client': + +```bash +# Find components using client-side features +grep -r "useState\|useEffect\|useContext\|onClick\|onChange" \ + --include="*.tsx" src/ | cut -d: -f1 | sort -u +``` + +### 4.2 Update Navigation Imports + +```typescript +// Before (React Router) +import { Link, useNavigate, useParams, useLocation } from 'react-router-dom' + +// After (Next.js) +import Link from 'next/link' +import { useRouter, useParams, usePathname, useSearchParams } from 'next/navigation' + +// Usage changes: +// navigate('/path') → router.push('/path') +// → +// location.pathname → pathname (from usePathname) +// location.search → searchParams (from useSearchParams) +``` + +### 4.3 Update React Query Hooks for SSR + +```typescript +// data-services/hooks/occurrences/useOccurrences.ts +'use client' + +import { useQuery } from '@tanstack/react-query' + +interface UseOccurrencesOptions { + params?: FetchParams + initialData?: OccurrenceListResponse +} + +export function useOccurrences({ params, initialData }: UseOccurrencesOptions = {}) { + return useQuery({ + queryKey: ['occurrences', params], + queryFn: () => fetchOccurrences(params), + initialData, + // Don't refetch on mount if we have initial data from server + refetchOnMount: !initialData, + }) +} +``` + +--- + +## Phase 5: Testing & Validation + +### 5.1 Run Both Versions in Parallel + +```bash +# Terminal 1: Vite dev server (existing) +yarn dev + +# Terminal 2: Next.js dev server +yarn dev:next +``` + +Compare functionality between ports 3000 (Vite) and 3001 (Next.js). + +### 5.2 Create Migration Checklist + +For each route, verify: +- [ ] Page renders correctly +- [ ] Data loads (SSR or client-side) +- [ ] Navigation works (links, back button) +- [ ] Forms submit correctly +- [ ] Authentication redirects work +- [ ] Loading states display +- [ ] Errors are handled +- [ ] URL parameters work +- [ ] Filters persist +- [ ] Mobile responsive + +### 5.3 Update Tests + +```typescript +// Jest setup for Next.js +// jest.setup.js +import '@testing-library/jest-dom' + +// Mock next/navigation +jest.mock('next/navigation', () => ({ + useRouter: () => ({ + push: jest.fn(), + replace: jest.fn(), + back: jest.fn(), + }), + useParams: () => ({}), + usePathname: () => '', + useSearchParams: () => new URLSearchParams(), +})) +``` + +--- + +## Phase 6: Cleanup & Cutover + +### 6.1 Remove Vite Files + +```bash +rm vite.config.ts +rm vite-env.d.ts +rm -rf src/ # After moving all components to app/ and components/ +``` + +### 6.2 Update package.json + +```json +{ + "scripts": { + "dev": "next dev", + "build": "next build", + "start": "next start", + "lint": "next lint" + } +} +``` + +Remove Vite dependencies: +```bash +yarn remove vite @vitejs/plugin-react vite-tsconfig-paths \ + vite-plugin-svgr vite-plugin-eslint react-router-dom \ + react-helmet-async +``` + +### 6.3 Update Docker Configuration + +```dockerfile +# compose/local/ui/Dockerfile +FROM node:18-alpine + +WORKDIR /app +COPY package.json yarn.lock ./ +RUN yarn install + +COPY . . + +# For development +CMD ["yarn", "dev"] + +# For production +# RUN yarn build +# CMD ["yarn", "start"] +``` + +### 6.4 Update CI/CD + +- Update build commands +- Update test commands +- Update deployment scripts +- Configure Next.js caching + +--- + +## Timeline Estimate + +| Phase | Duration | Dependencies | +|-------|----------|--------------| +| Phase 0: Preparation | 1-2 days | None | +| Phase 1: Parallel Setup | 1-2 days | Phase 0 | +| Phase 2: Core Infrastructure | 2-3 days | Phase 1 | +| Phase 3: Route Migration | 2-3 weeks | Phase 2 | +| Phase 4: Component Updates | 1 week | Ongoing with Phase 3 | +| Phase 5: Testing | 1 week | Phase 3-4 | +| Phase 6: Cleanup | 2-3 days | Phase 5 | + +**Total estimated time**: 6-8 weeks + +--- + +## Rollback Plan + +If critical issues arise: + +1. Keep `main` branch unchanged until migration complete +2. Maintain Vite config in separate branch during migration +3. Docker Compose can switch between builds: + ```yaml + ui: + build: + context: ./ui + # target: vite # Uncomment to rollback + target: nextjs + ``` +4. Feature flag for gradual rollout if needed + +--- + +## Success Criteria + +- [ ] All routes accessible and functional +- [ ] Authentication flow works end-to-end +- [ ] All CRUD operations work +- [ ] Gallery navigation works +- [ ] Map components render +- [ ] Charts display correctly +- [ ] Export functionality works +- [ ] Job monitoring works +- [ ] No console errors +- [ ] Performance metrics improved (LCP, FCP) +- [ ] All tests passing +- [ ] Docker build succeeds +- [ ] CI/CD pipeline green diff --git a/.agents/planning/06-questions-risks.md b/.agents/planning/06-questions-risks.md new file mode 100644 index 000000000..c3cbc3740 --- /dev/null +++ b/.agents/planning/06-questions-risks.md @@ -0,0 +1,359 @@ +# Outstanding Questions, Risks, and Alternatives + +## Outstanding Questions + +### Architecture Questions + +#### Q1: Server Components vs Client Components Balance +**Question**: What percentage of pages should be Server Components? + +**Considerations**: +- Occurrences page has heavy filtering that updates URL params → mostly client +- Project dashboard with summary stats → could be server +- Gallery with real-time updates → must be client +- Settings pages → mixed + +**Research Needed**: +- Profile current page load times to establish baseline +- Identify which pages would benefit most from SSR +- Determine if SEO matters for this application (likely not - authenticated content) + +#### Q2: Authentication Strategy +**Question**: Should we use cookies, localStorage, or both for auth tokens? + +**Options**: +| Approach | Pros | Cons | +|----------|------|------| +| Cookies only | SSR access, more secure | CSRF concerns, httpOnly complexity | +| localStorage only | Simple, current approach | No SSR access, XSS vulnerable | +| Both (hybrid) | Best of both | More complexity | + +**Recommendation**: Hybrid approach with cookie for SSR + localStorage for client consistency. + +**Research Needed**: +- Review Django CSRF token handling +- Determine if httpOnly cookies work with current API setup +- Test middleware auth flow + +#### Q3: API Route Handlers (BFF) +**Question**: Should we add Next.js API routes as a Backend-for-Frontend layer? + +**Use Cases**: +- Aggregate multiple Django endpoints (reduce waterfalls) +- Add server-side caching for expensive queries +- Transform API responses before sending to client + +**Research Needed**: +- Identify slowest/most complex API calls +- Measure if aggregation would improve performance +- Consider maintenance overhead of BFF layer + +#### Q4: Image Optimization Strategy +**Question**: Should we use next/image for all images or selectively? + +**Considerations**: +- Gallery images: Many small thumbnails → next/image could help +- Detection crops: Dynamic URLs from ML pipeline → needs remote patterns +- User uploads: Variable sizes → optimization valuable + +**Research Needed**: +- Test next/image with MinIO URLs +- Measure gallery performance improvement +- Check if blur placeholders work with remote images + +#### Q5: Deployment Target +**Question**: Where will Next.js be deployed? + +**Options**: +| Target | Pros | Cons | +|--------|------|------| +| Vercel | Zero config, edge functions | Vendor lock-in, cost | +| Docker (current) | Consistent with Django | No edge, more config | +| Static export | Simplest, CDN-friendly | No SSR benefits | + +**Current**: Docker Compose with Node.js server seems most consistent. + +**Research Needed**: +- Verify Next.js standalone output works in Docker +- Test rewrites/proxying in containerized setup +- Measure cold start times + +--- + +## Risk Factors + +### High Risk + +#### R1: React Query + SSR Hydration Mismatch +**Risk**: Server-rendered data may not match client state, causing hydration errors. + +**Likelihood**: Medium +**Impact**: High (broken UI, console errors) + +**Mitigation**: +- Use `initialData` pattern correctly +- Disable `refetchOnMount` when using server data +- Ensure serialization is consistent (dates, BigInt, etc.) + +**Detection**: Hydration mismatch warnings in console during testing. + +#### R2: Authentication Race Conditions +**Risk**: Token may not be available in cookies when middleware runs. + +**Likelihood**: Medium +**Impact**: High (users redirected to login incorrectly) + +**Mitigation**: +- Ensure cookie is set before navigation +- Add loading state during auth transitions +- Test login/logout flows thoroughly + +**Detection**: Manual testing of auth flows, automated E2E tests. + +#### R3: Third-Party Library Compatibility +**Risk**: Leaflet, Plotly, or nova-ui-kit may have SSR issues. + +**Likelihood**: Medium +**Impact**: Medium (specific features broken) + +**Mitigation**: +- Mark these components with 'use client' +- Use dynamic imports with `ssr: false` if needed +- Test all map/chart features early + +**Example**: +```typescript +const Map = dynamic(() => import('@/components/map'), { ssr: false }) +``` + +**Detection**: Build errors, runtime errors on page load. + +### Medium Risk + +#### R4: Performance Regression +**Risk**: SSR overhead may slow down some operations. + +**Likelihood**: Low +**Impact**: Medium + +**Mitigation**: +- Benchmark before and after +- Use streaming for large data sets +- Implement proper caching strategies + +**Detection**: Performance monitoring, user feedback. + +#### R5: Development Velocity Slowdown +**Risk**: Team learning curve impacts delivery. + +**Likelihood**: Medium +**Impact**: Medium + +**Mitigation**: +- Phased migration allows gradual learning +- Keep Vite running in parallel initially +- Document patterns and gotchas + +**Detection**: Sprint velocity metrics, team feedback. + +#### R6: Docker Build Complexity +**Risk**: Next.js Docker builds may be more complex than Vite. + +**Likelihood**: Medium +**Impact**: Low + +**Mitigation**: +- Use Next.js standalone output mode +- Follow Vercel's Docker examples +- Test builds early and often + +**Detection**: CI/CD failures, increased build times. + +### Low Risk + +#### R7: URL/Route Compatibility +**Risk**: Some URL structures may need to change. + +**Likelihood**: Low +**Impact**: Low (redirects can handle) + +**Mitigation**: +- Map all routes before migration +- Add redirects for changed paths +- Update any hardcoded URLs in backend/docs + +#### R8: Test Suite Updates +**Risk**: Existing tests may need significant updates. + +**Likelihood**: Medium +**Impact**: Low + +**Mitigation**: +- Update Jest config for Next.js +- Mock next/navigation consistently +- Prioritize integration tests over unit tests + +--- + +## Alternatives to Next.js + +### Alternative 1: Stay with Vite + Add SSR + +**Approach**: Use vite-plugin-ssr or Vite's experimental SSR. + +**Pros**: +- Smaller change from current setup +- Vite's fast HMR preserved +- No new framework to learn + +**Cons**: +- Less mature than Next.js +- Fewer built-in features +- Smaller community/ecosystem +- More manual configuration + +**When to consider**: If team strongly prefers Vite and SSR benefits are minimal. + +### Alternative 2: Remix + +**Approach**: Migrate to Remix instead of Next.js. + +**Pros**: +- Excellent data loading patterns +- Progressive enhancement focus +- Good error handling +- React Router team's framework + +**Cons**: +- Smaller ecosystem than Next.js +- Fewer deployment options +- Less enterprise adoption +- Different mental model from current app + +**When to consider**: If data mutations are primary concern and you want progressive enhancement. + +### Alternative 3: Astro with React Islands + +**Approach**: Use Astro for static shell, React for interactive parts. + +**Pros**: +- Zero JS by default (great performance) +- Perfect for content-heavy sites +- Can use React components where needed + +**Cons**: +- Different architecture paradigm +- Overkill for data-heavy app like Antenna +- Less suitable for complex SPA interactions + +**When to consider**: If app was primarily content/documentation focused. + +### Alternative 4: Keep Vite, Optimize Current Stack + +**Approach**: Don't migrate, just optimize current SPA. + +**Improvements possible**: +- Add route-based code splitting +- Implement skeleton loading states +- Add service worker for offline/caching +- Optimize bundle size +- Add React Query prefetching + +**Pros**: +- No migration effort +- Team already familiar +- Lower risk + +**Cons**: +- Misses SSR/SEO benefits +- No server components +- Manual implementation of features Next.js provides + +**When to consider**: If SSR benefits don't justify migration effort. + +### Alternative 5: TanStack Start (Emerging) + +**Approach**: Use TanStack's new full-stack framework. + +**Pros**: +- Built by React Query team +- Familiar patterns if using TanStack +- Modern architecture + +**Cons**: +- Very new (still in development) +- Not production-ready +- Limited documentation/ecosystem + +**When to consider**: Watch for future, but not ready for production migration. + +--- + +## Decision Matrix + +| Factor | Next.js | Vite+SSR | Remix | Stay Current | +|--------|---------|----------|-------|--------------| +| SSR Support | ★★★★★ | ★★★☆☆ | ★★★★★ | ☆☆☆☆☆ | +| Ecosystem | ★★★★★ | ★★★☆☆ | ★★★★☆ | ★★★★☆ | +| Learning Curve | ★★★☆☆ | ★★★★☆ | ★★☆☆☆ | ★★★★★ | +| Migration Effort | ★★★☆☆ | ★★★★☆ | ★★☆☆☆ | ★★★★★ | +| Performance Potential | ★★★★★ | ★★★★☆ | ★★★★★ | ★★★☆☆ | +| Community Support | ★★★★★ | ★★★☆☆ | ★★★★☆ | ★★★★☆ | +| Enterprise Adoption | ★★★★★ | ★★☆☆☆ | ★★★☆☆ | ★★★★☆ | + +--- + +## Summary of Benefits + +### Performance Benefits +1. **Faster Initial Load**: Server-rendered HTML visible before JS loads +2. **Reduced Bundle Size**: Automatic code splitting per route +3. **Image Optimization**: Built-in lazy loading, WebP/AVIF, responsive images +4. **Streaming**: Large data sets can stream to client progressively +5. **Caching**: Multi-layer caching (data, full route, router) + +### Developer Experience Benefits +1. **File-based Routing**: No manual route configuration +2. **Built-in Loading/Error States**: Convention-based, consistent UX +3. **TypeScript Integration**: Better type safety for routes and params +4. **Hot Module Replacement**: Comparable to Vite +5. **Built-in Linting**: ESLint config with Next.js rules + +### SEO/Accessibility Benefits +1. **Crawlable Content**: Search engines see rendered HTML +2. **Metadata API**: Easy Open Graph and Twitter cards +3. **No Flash of Unstyled Content**: Server-rendered initial state + +### Operational Benefits +1. **Deployment Flexibility**: Vercel, Docker, static export +2. **Analytics Integration**: Built-in Web Vitals reporting +3. **Monitoring**: Better Sentry integration with @sentry/nextjs +4. **Incremental Adoption**: Can migrate gradually + +### Future-Proofing Benefits +1. **React Server Components**: Ready for RSC patterns +2. **React 19 Features**: Early access to new React features +3. **Active Development**: Vercel invests heavily in Next.js +4. **Large Ecosystem**: Extensive plugin/integration options + +--- + +## Recommendation + +**Proceed with Next.js migration** if: +- Team has bandwidth for 6-8 week migration +- Performance improvements are valuable +- Future features will benefit from SSR/Server Components +- Willing to invest in learning new patterns + +**Defer migration** if: +- Current performance is acceptable +- Team is at capacity with feature work +- SEO is not a concern (authenticated app) +- Risk tolerance is low + +**Hybrid approach**: +- Start with parallel setup (Phase 1) +- Migrate one non-critical route as POC +- Measure performance difference +- Decide whether to continue based on results diff --git a/.agents/planning/07-testing-evaluation.md b/.agents/planning/07-testing-evaluation.md new file mode 100644 index 000000000..231305a56 --- /dev/null +++ b/.agents/planning/07-testing-evaluation.md @@ -0,0 +1,765 @@ +# Testing and Evaluation Plan + +## Overview + +This document outlines a comprehensive strategy for testing and validating the Next.js migration to ensure all functionality works correctly. Testing spans automated tests, manual verification, performance benchmarks, and user acceptance criteria. + +--- + +## 1. Pre-Migration Baseline + +### 1.1 Establish Current Metrics + +Before migration, capture baseline metrics: + +**Performance Metrics** (use Chrome DevTools Lighthouse): +``` +| Metric | Current Value | Target | +|--------|---------------|--------| +| First Contentful Paint (FCP) | ___ms | < FCP | +| Largest Contentful Paint (LCP) | ___ms | < LCP | +| Time to Interactive (TTI) | ___ms | < TTI | +| Total Blocking Time (TBT) | ___ms | < TBT | +| Cumulative Layout Shift (CLS) | ___ | < CLS | +| Bundle Size (JS) | ___KB | < Size | +``` + +**Functional Baseline**: +- [ ] Document all existing features +- [ ] Screenshot each page/view +- [ ] Record current test coverage percentage +- [ ] List known bugs/issues + +### 1.2 Create Feature Inventory + +Complete checklist of all features: + +``` +## Authentication +- [ ] Login with email/password +- [ ] Logout +- [ ] Password reset request +- [ ] Password reset confirmation +- [ ] Session persistence +- [ ] Auto-redirect when not authenticated + +## Projects +- [ ] List projects +- [ ] Create project +- [ ] Edit project settings +- [ ] Delete project +- [ ] View project summary/dashboard +- [ ] Project member management + +## Deployments +- [ ] List deployments +- [ ] Create deployment +- [ ] Edit deployment +- [ ] Delete deployment +- [ ] Sync from S3 source +- [ ] View deployment details +- [ ] Map location editing + +## Events/Sessions +- [ ] List events +- [ ] View event timeline +- [ ] Filter by date range + +## Captures (Source Images) +- [ ] Gallery view +- [ ] Table view +- [ ] Image upload +- [ ] Star/unstar images +- [ ] Filter by deployment +- [ ] Filter by date +- [ ] Pagination + +## Occurrences +- [ ] Gallery view +- [ ] Table view +- [ ] Detail modal +- [ ] Filter by taxon +- [ ] Filter by score threshold +- [ ] Filter by date range +- [ ] Filter by deployment +- [ ] Sort by various fields +- [ ] Keyboard navigation +- [ ] Pagination + +## Identifications +- [ ] View ML predictions +- [ ] View human identifications +- [ ] Agree with prediction +- [ ] Suggest alternative ID +- [ ] Remove identification +- [ ] Taxa autocomplete/search + +## Jobs +- [ ] List jobs +- [ ] Create new job +- [ ] View job details +- [ ] Queue job for processing +- [ ] Cancel running job +- [ ] Retry failed job +- [ ] View job progress +- [ ] View job logs + +## Pipelines & ML +- [ ] List pipelines +- [ ] View pipeline details +- [ ] Configure project pipelines +- [ ] Test pipeline + +## Processing Services +- [ ] List services +- [ ] View service status +- [ ] Register pipelines from service +- [ ] Health check status + +## Exports +- [ ] List exports +- [ ] Create export +- [ ] Download export +- [ ] View export status + +## Collections +- [ ] List collections +- [ ] Create collection +- [ ] View collection +- [ ] Add images to collection + +## Settings +- [ ] General settings +- [ ] Team management +- [ ] Default filters +- [ ] Storage configuration +- [ ] Processing settings +- [ ] Sites management +- [ ] Devices management +``` + +--- + +## 2. Automated Testing Strategy + +### 2.1 Unit Tests + +**Configuration for Next.js:** + +```javascript +// jest.config.js +const nextJest = require('next/jest') + +const createJestConfig = nextJest({ + dir: './', +}) + +const customJestConfig = { + setupFilesAfterEnv: ['/jest.setup.js'], + testEnvironment: 'jest-environment-jsdom', + moduleNameMapper: { + '^@/(.*)$': '/$1', + }, + collectCoverageFrom: [ + 'app/**/*.{ts,tsx}', + 'components/**/*.{ts,tsx}', + 'data-services/**/*.{ts,tsx}', + '!**/*.d.ts', + '!**/node_modules/**', + ], + coverageThreshold: { + global: { + branches: 70, + functions: 70, + lines: 70, + statements: 70, + }, + }, +} + +module.exports = createJestConfig(customJestConfig) +``` + +```javascript +// jest.setup.js +import '@testing-library/jest-dom' + +// Mock next/navigation +jest.mock('next/navigation', () => ({ + useRouter: () => ({ + push: jest.fn(), + replace: jest.fn(), + back: jest.fn(), + forward: jest.fn(), + refresh: jest.fn(), + prefetch: jest.fn(), + }), + useParams: () => ({}), + usePathname: () => '/', + useSearchParams: () => new URLSearchParams(), + useSelectedLayoutSegment: () => null, + useSelectedLayoutSegments: () => [], +})) + +// Mock next/image +jest.mock('next/image', () => ({ + __esModule: true, + default: (props) => , +})) +``` + +**Test Categories:** + +| Category | Files | Priority | +|----------|-------|----------| +| Domain Models | `data-services/models/*.test.ts` | High | +| Utility Functions | `utils/*.test.ts` | High | +| Custom Hooks | `data-services/hooks/*.test.ts` | High | +| UI Components | `components/**/*.test.tsx` | Medium | +| Design System | `design-system/**/*.test.tsx` | Medium | + +### 2.2 Integration Tests + +Test data fetching and component integration: + +```typescript +// __tests__/pages/occurrences.test.tsx +import { render, screen, waitFor } from '@testing-library/react' +import { QueryClient, QueryClientProvider } from '@tanstack/react-query' +import { OccurrencesList } from '@/components/occurrences-list' + +const mockOccurrences = [ + { id: '1', determination: { name: 'Species A' }, score: 0.95 }, + { id: '2', determination: { name: 'Species B' }, score: 0.87 }, +] + +describe('Occurrences Page', () => { + it('renders occurrence list with initial data', async () => { + const queryClient = new QueryClient() + + render( + + + + ) + + await waitFor(() => { + expect(screen.getByText('Species A')).toBeInTheDocument() + expect(screen.getByText('Species B')).toBeInTheDocument() + }) + }) + + it('filters occurrences by score threshold', async () => { + // Test filter functionality + }) +}) +``` + +### 2.3 End-to-End Tests + +Use Playwright for E2E testing: + +```typescript +// e2e/auth.spec.ts +import { test, expect } from '@playwright/test' + +test.describe('Authentication', () => { + test('should login successfully', async ({ page }) => { + await page.goto('/auth/login') + + await page.fill('input[name="email"]', 'antenna@insectai.org') + await page.fill('input[name="password"]', 'localadmin') + await page.click('button[type="submit"]') + + await expect(page).toHaveURL('/projects') + await expect(page.locator('h1')).toContainText('Projects') + }) + + test('should redirect unauthenticated users to login', async ({ page }) => { + await page.goto('/projects') + await expect(page).toHaveURL('/auth/login') + }) + + test('should logout successfully', async ({ page }) => { + // Login first + await page.goto('/auth/login') + await page.fill('input[name="email"]', 'antenna@insectai.org') + await page.fill('input[name="password"]', 'localadmin') + await page.click('button[type="submit"]') + + // Then logout + await page.click('[data-testid="user-menu"]') + await page.click('[data-testid="logout-button"]') + + await expect(page).toHaveURL('/auth/login') + }) +}) +``` + +```typescript +// e2e/occurrences.spec.ts +import { test, expect } from '@playwright/test' + +test.describe('Occurrences', () => { + test.beforeEach(async ({ page }) => { + // Login before each test + await page.goto('/auth/login') + await page.fill('input[name="email"]', 'antenna@insectai.org') + await page.fill('input[name="password"]', 'localadmin') + await page.click('button[type="submit"]') + }) + + test('should display occurrence gallery', async ({ page }) => { + await page.goto('/projects/1/occurrences') + + await expect(page.locator('[data-testid="occurrence-grid"]')).toBeVisible() + await expect(page.locator('[data-testid="occurrence-card"]').first()).toBeVisible() + }) + + test('should open occurrence detail modal', async ({ page }) => { + await page.goto('/projects/1/occurrences') + + await page.locator('[data-testid="occurrence-card"]').first().click() + + await expect(page.locator('[data-testid="occurrence-modal"]')).toBeVisible() + }) + + test('should filter by taxon', async ({ page }) => { + await page.goto('/projects/1/occurrences') + + await page.click('[data-testid="filter-taxon"]') + await page.fill('[data-testid="taxon-search"]', 'Lepidoptera') + await page.click('[data-testid="taxon-option-lepidoptera"]') + + await expect(page).toHaveURL(/taxon=/) + }) +}) +``` + +### 2.4 Visual Regression Tests + +Use Playwright for screenshot comparison: + +```typescript +// e2e/visual.spec.ts +import { test, expect } from '@playwright/test' + +test.describe('Visual Regression', () => { + test('occurrences page matches snapshot', async ({ page }) => { + await page.goto('/projects/1/occurrences') + await page.waitForLoadState('networkidle') + + await expect(page).toHaveScreenshot('occurrences-page.png', { + maxDiffPixelRatio: 0.01, + }) + }) + + test('occurrence detail modal matches snapshot', async ({ page }) => { + await page.goto('/projects/1/occurrences/1') + await page.waitForLoadState('networkidle') + + await expect(page).toHaveScreenshot('occurrence-detail.png', { + maxDiffPixelRatio: 0.01, + }) + }) +}) +``` + +--- + +## 3. Manual Testing Checklist + +### 3.1 Route-by-Route Verification + +For each migrated route, verify: + +``` +## Route: /auth/login +- [ ] Page loads without errors +- [ ] Form renders correctly +- [ ] Email input accepts input +- [ ] Password input masks characters +- [ ] Submit button is clickable +- [ ] Validation errors display +- [ ] Successful login redirects to /projects +- [ ] "Forgot password" link works +- [ ] Responsive on mobile + +## Route: /projects +- [ ] Page loads without errors +- [ ] Project list displays +- [ ] "New Project" button visible (if permitted) +- [ ] Click project navigates to project page +- [ ] Pagination works +- [ ] Sort options work +- [ ] Search/filter works +- [ ] Loading state shows during fetch +- [ ] Error state shows on failure + +## Route: /projects/[projectId]/occurrences +- [ ] Page loads without errors +- [ ] Gallery view displays images +- [ ] Table view toggles correctly +- [ ] Filters panel opens/closes +- [ ] Taxon filter works +- [ ] Score filter works +- [ ] Date range filter works +- [ ] Deployment filter works +- [ ] Clear filters works +- [ ] Sorting works (each column) +- [ ] Pagination works +- [ ] Clicking occurrence opens modal +- [ ] Modal close returns to list +- [ ] URL updates with filters +- [ ] Back button preserves filters +- [ ] Keyboard navigation (arrows) works +- [ ] Loading skeleton shows +- [ ] Empty state shows when no results + +## Route: /projects/[projectId]/occurrences/[id] +- [ ] Detail view loads +- [ ] Image displays correctly +- [ ] Detection bounding boxes render +- [ ] ML predictions display +- [ ] Human identifications display +- [ ] "Agree" button works +- [ ] "Suggest ID" opens autocomplete +- [ ] Taxa autocomplete searches API +- [ ] New ID submission works +- [ ] Previous/Next navigation works +- [ ] Close button works +- [ ] Escape key closes modal + +(Continue for all routes...) +``` + +### 3.2 Cross-Browser Testing + +Test on: +- [ ] Chrome (latest) +- [ ] Firefox (latest) +- [ ] Safari (latest) +- [ ] Edge (latest) +- [ ] Chrome Mobile (Android) +- [ ] Safari Mobile (iOS) + +### 3.3 Responsive Testing + +Test at breakpoints: +- [ ] 320px (mobile portrait) +- [ ] 480px (mobile landscape) +- [ ] 768px (tablet) +- [ ] 1024px (desktop) +- [ ] 1440px (large desktop) +- [ ] 1920px (full HD) + +### 3.4 Accessibility Testing + +- [ ] Screen reader navigation (VoiceOver/NVDA) +- [ ] Keyboard-only navigation +- [ ] Color contrast ratios +- [ ] Focus indicators visible +- [ ] Alt text on images +- [ ] ARIA labels present +- [ ] Form labels associated + +--- + +## 4. Performance Testing + +### 4.1 Lighthouse Audits + +Run Lighthouse on key pages: + +```bash +# Install lighthouse CLI +npm install -g lighthouse + +# Run audit +lighthouse http://localhost:3000/projects/1/occurrences \ + --output html \ + --output-path ./lighthouse-report.html +``` + +**Pages to audit:** +- `/` (home) +- `/projects` (project list) +- `/projects/[id]/occurrences` (main gallery) +- `/projects/[id]/deployments` (deployment list) +- `/projects/[id]/jobs` (job list) + +### 4.2 Core Web Vitals + +Track these metrics: + +| Metric | Good | Needs Improvement | Poor | +|--------|------|-------------------|------| +| LCP | ≤ 2.5s | 2.5s - 4s | > 4s | +| FID | ≤ 100ms | 100ms - 300ms | > 300ms | +| CLS | ≤ 0.1 | 0.1 - 0.25 | > 0.25 | + +### 4.3 Bundle Analysis + +```bash +# Install bundle analyzer +npm install @next/bundle-analyzer + +# Add to next.config.js +const withBundleAnalyzer = require('@next/bundle-analyzer')({ + enabled: process.env.ANALYZE === 'true', +}) + +module.exports = withBundleAnalyzer(nextConfig) + +# Run analysis +ANALYZE=true npm run build +``` + +**Review:** +- [ ] No duplicate dependencies +- [ ] Large libraries code-split +- [ ] Unused exports tree-shaken +- [ ] Compare to pre-migration bundle size + +### 4.4 Server-Side Performance + +Measure server response times: + +```bash +# Time to first byte for SSR pages +curl -w "TTFB: %{time_starttransfer}s\n" -o /dev/null -s \ + http://localhost:3000/projects/1/occurrences +``` + +--- + +## 5. API Compatibility Testing + +### 5.1 Verify All API Calls + +Test each API endpoint is called correctly: + +```typescript +// Mock API and verify calls +import { rest } from 'msw' +import { setupServer } from 'msw/node' + +const server = setupServer( + rest.get('/api/v2/occurrences/', (req, res, ctx) => { + // Verify query params + expect(req.url.searchParams.get('project')).toBe('1') + expect(req.url.searchParams.get('limit')).toBe('20') + + return res(ctx.json({ results: [], count: 0 })) + }) +) + +beforeAll(() => server.listen()) +afterEach(() => server.resetHandlers()) +afterAll(() => server.close()) +``` + +### 5.2 Authentication Header Verification + +Ensure auth token is sent correctly: + +```typescript +// Intercept and verify auth header +rest.get('/api/v2/projects/', (req, res, ctx) => { + const authHeader = req.headers.get('Authorization') + expect(authHeader).toMatch(/^Token .+/) + return res(ctx.json({ results: [] })) +}) +``` + +### 5.3 Error Handling + +Test error scenarios: + +- [ ] 401 Unauthorized → Redirect to login +- [ ] 403 Forbidden → Show permission error +- [ ] 404 Not Found → Show not found page +- [ ] 500 Server Error → Show error boundary +- [ ] Network timeout → Show retry option + +--- + +## 6. Regression Testing + +### 6.1 Comparison Testing + +Run both versions side-by-side: + +```bash +# Terminal 1: Vite version +PORT=3000 yarn dev:vite + +# Terminal 2: Next.js version +PORT=3001 yarn dev:next +``` + +For each feature, compare: +- [ ] Visual appearance matches +- [ ] Behavior matches +- [ ] Performance equal or better +- [ ] No console errors + +### 6.2 Data Integrity + +Verify data displays correctly: + +- [ ] All fields render +- [ ] Numbers format correctly +- [ ] Dates display in correct timezone +- [ ] Special characters render +- [ ] Long text truncates properly +- [ ] Empty states handle gracefully + +--- + +## 7. Security Testing + +### 7.1 Authentication Security + +- [ ] Token not exposed in URLs +- [ ] Token stored securely (httpOnly if using cookies) +- [ ] Logout clears all stored tokens +- [ ] Protected routes redirect correctly +- [ ] Expired tokens handled gracefully + +### 7.2 XSS Prevention + +- [ ] User input is escaped in display +- [ ] No dangerouslySetInnerHTML without sanitization +- [ ] Content-Security-Policy headers set + +### 7.3 CSRF Protection + +- [ ] State-changing operations use CSRF tokens +- [ ] SameSite cookie attribute set + +--- + +## 8. Acceptance Criteria + +### 8.1 Must Have (P0) + +- [ ] All existing features work identically +- [ ] No data loss or corruption +- [ ] Authentication works correctly +- [ ] Performance equal or better +- [ ] No console errors in production +- [ ] All automated tests pass + +### 8.2 Should Have (P1) + +- [ ] Improved Lighthouse scores +- [ ] Reduced bundle size +- [ ] Faster initial page load +- [ ] Loading states for all async operations +- [ ] Error boundaries on all pages + +### 8.3 Nice to Have (P2) + +- [ ] Server-side rendering on key pages +- [ ] Image optimization implemented +- [ ] Streaming for large data sets +- [ ] Edge middleware for auth + +--- + +## 9. Test Environment Setup + +### 9.1 Test Database + +```bash +# Create test fixtures +docker compose run --rm django python manage.py create_demo_project + +# Ensure consistent test data +docker compose run --rm django python manage.py dumpdata \ + --natural-foreign --natural-primary \ + -e contenttypes -e auth.Permission \ + > fixtures/test_data.json +``` + +### 9.2 CI Pipeline + +```yaml +# .github/workflows/test.yml +name: Test + +on: [push, pull_request] + +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version: '18' + cache: 'yarn' + + - name: Install dependencies + run: yarn install --frozen-lockfile + + - name: Run unit tests + run: yarn test --coverage + + - name: Run E2E tests + run: yarn test:e2e + + - name: Build + run: yarn build + + - name: Upload coverage + uses: codecov/codecov-action@v3 +``` + +--- + +## 10. Sign-Off Checklist + +### 10.1 Development Sign-Off + +- [ ] All unit tests pass +- [ ] All integration tests pass +- [ ] All E2E tests pass +- [ ] No TypeScript errors +- [ ] No ESLint errors +- [ ] Code review completed +- [ ] Documentation updated + +### 10.2 QA Sign-Off + +- [ ] Manual testing complete +- [ ] Cross-browser testing complete +- [ ] Mobile testing complete +- [ ] Accessibility testing complete +- [ ] Performance testing complete +- [ ] Security testing complete + +### 10.3 Stakeholder Sign-Off + +- [ ] Demo to stakeholders +- [ ] Feedback addressed +- [ ] Production deployment approved + +--- + +## 11. Rollback Plan + +If critical issues found post-deployment: + +1. **Immediate**: Revert to previous Vite build +2. **Short-term**: Deploy hotfix to Next.js version +3. **Rollback command**: + ```bash + git revert HEAD --no-edit + git push origin main + # CI/CD redeploys previous version + ``` +4. **Monitoring**: Watch error rates for 24 hours post-rollback diff --git a/.agents/planning/08-motivations.md b/.agents/planning/08-motivations.md new file mode 100644 index 000000000..995e3281e --- /dev/null +++ b/.agents/planning/08-motivations.md @@ -0,0 +1,158 @@ +# Migration Motivations + +## Team Context + +### Team Size & Composition +- **2 full-time developers** +- **Periodic interns** (need clear patterns to onboard quickly) +- **AI agents** (benefit from consistent, predictable code structures) + +This is a small team maintaining a complex ML platform. Every architectural decision must optimize for: +1. **Onboarding speed** - New contributors (human or AI) must understand patterns quickly +2. **Maintenance burden** - Less custom code = fewer bugs to own +3. **Feature velocity** - Spend time on domain logic, not infrastructure + +--- + +## Core Motivations + +### 1. Reduce Boilerplate Code + +**Current State**: The React ecosystem requires assembling many libraries and writing glue code: +- Route definitions separate from file structure +- Custom hooks for every data fetching pattern +- Manual loading/error state handling +- Custom table sorting logic +- Form submission patterns vary by component + +**Desired State**: Convention-over-configuration where common patterns are built-in: +- File = route (no configuration) +- Data loading patterns are prescribed +- Loading/error states have standard locations +- Common UI patterns (tables, forms) have established solutions + +### 2. Stop Reinventing Established Patterns + +**Problem**: The current codebase contains custom implementations of things the open-source community has already solved: +- Custom pagination logic +- Custom filter URL synchronization +- Custom table sorting +- Custom loading state management +- Custom error boundary setup + +**Cost of Custom Solutions**: +- Bugs we own (instead of community maintaining) +- Documentation we must write +- Patterns interns/AI must learn +- Edge cases we discover over time + +**Desired State**: Use framework conventions and community-maintained solutions wherever possible. Custom code only for domain-specific logic (species identification, ML pipelines, etc.). + +### 3. Opinionated Framework Benefits + +**Why Opinions Matter for Small Teams**: + +| Aspect | Unopinionated | Opinionated | +|--------|---------------|-------------| +| Decision fatigue | High - choose everything | Low - follow conventions | +| Onboarding | "Here's how WE do it" | "Here's how everyone does it" | +| Documentation | Must write internal docs | External docs exist | +| AI assistance | Must explain patterns | AI knows standard patterns | +| Bug fixes | We maintain | Community maintains | +| Code review | Debate approaches | Standard is clear | + +**Django/DRF as Model**: The backend uses Django REST Framework, which provides: +- Serializers (data validation + transformation) +- ViewSets (CRUD with minimal code) +- Permissions (declarative access control) +- Filtering (query params to queryset) +- Pagination (standardized patterns) + +We want similar leverage on the frontend. + +### 4. Established Places for Things + +**Current Ambiguity**: +- Where does a new API hook go? (`data-services/hooks/[domain]/`) +- Where does form validation live? (in component? in hook? in utility?) +- How do we handle optimistic updates? (varies by feature) +- Where do loading states go? (inline? wrapper? context?) + +**Desired Clarity**: Framework conventions that answer: +- "Where does data loading code go?" → Loader function +- "Where does mutation code go?" → Action function +- "Where does loading UI go?" → loading.tsx / pending state +- "Where does error UI go?" → error.tsx / error boundary + +### 5. Specific Pain Points + +#### Routing +- **Current**: Manual route definitions in `app.tsx`, must keep in sync with components +- **Desired**: File-based routing where structure = routes + +#### Data Fetching + UI Sync +- **Current**: React Query hooks with varying patterns, manual cache invalidation +- **Desired**: Prescribed patterns for loading data, automatic revalidation + +#### Table Sorting +- **Current**: Custom `useSort` hook, manual implementation per table +- **Desired**: Standard table solution with sorting/filtering built-in + +#### Forms with Mixed Content +- **Current**: react-hook-form + manual file handling + custom submission logic +- **Desired**: Framework-level form handling that covers common cases + +#### Internationalization +- **Current**: Not implemented +- **Desired**: Built-in i18n support when needed + +--- + +## What We're NOT Optimizing For + +1. **Maximum Performance** - Current performance is acceptable +2. **SEO** - App is authenticated, crawlers don't matter +3. **Server-Side Rendering** - Nice to have, not the driver +4. **Cutting-Edge Features** - Stability over novelty + +--- + +## Success Criteria for Migration + +A successful migration would mean: + +1. **Interns can add features faster** by following established patterns +2. **AI agents produce better code** because patterns are standard/documented +3. **Bug count decreases** because we use community-maintained solutions +4. **Code reviews are faster** because "the right way" is clear +5. **Less custom code to maintain** overall +6. **Documentation exists externally** (framework docs, tutorials, Stack Overflow) + +--- + +## Framework Evaluation Criteria + +When evaluating Next.js, Remix, or alternatives, we prioritize: + +| Criterion | Weight | Notes | +|-----------|--------|-------| +| Convention-over-configuration | High | Reduces decisions | +| Data loading patterns | High | Core pain point | +| Form handling | High | Core pain point | +| Community size | Medium | Affects available help | +| Ecosystem maturity | Medium | Affects library availability | +| Learning resources | Medium | Affects onboarding | +| Performance | Low | Current is acceptable | +| SSR capabilities | Low | Not a driver | + +--- + +## Summary + +We're a small team that needs to maximize leverage. Every line of custom infrastructure code is: +- A line we must maintain +- A pattern we must document +- A decision we must explain to new contributors +- A potential bug we own + +The ideal framework acts like DRF does for our backend: providing sensible defaults, clear conventions, and community-maintained solutions for common patterns, so we can focus our limited engineering time on what makes Antenna unique—not on reinventing forms, tables, and data loading. diff --git a/.agents/planning/09-e2e-testing-strategy.md b/.agents/planning/09-e2e-testing-strategy.md new file mode 100644 index 000000000..fbeec3e15 --- /dev/null +++ b/.agents/planning/09-e2e-testing-strategy.md @@ -0,0 +1,605 @@ +# E2E Testing Strategy with Playwright + +## Overview + +This document outlines strategies for efficiently creating comprehensive E2E tests before or alongside the Remix migration. The goal is to establish a safety net that verifies the application works correctly, with minimal manual test-writing effort. + +--- + +## Tool Selection + +### Recommendation: Playwright + +| Feature | Playwright | Cypress | Selenium | +|---------|------------|---------|----------| +| Multi-browser | ✅ Chrome, Firefox, Safari | ⚠️ Chrome-focused | ✅ All | +| Speed | ✅ Fast (parallel) | ⚠️ Slower | ❌ Slowest | +| Auto-wait | ✅ Built-in | ✅ Built-in | ❌ Manual | +| Codegen (recording) | ✅ Excellent | ⚠️ Basic | ❌ None | +| API testing | ✅ Built-in | ✅ Plugin | ❌ Separate | +| Visual comparison | ✅ Built-in | ⚠️ Plugin | ❌ Separate | +| Trace viewer | ✅ Excellent | ⚠️ Basic | ❌ None | +| AI-friendly | ✅ Clean API | ✅ Good | ❌ Verbose | +| Modern/maintained | ✅ Microsoft | ✅ Active | ⚠️ Legacy feel | + +**Why Playwright:** +- Best codegen (recording) tool +- Excellent trace viewer for debugging +- Fast parallel execution +- Built-in visual regression +- Clean, modern API that AI can generate easily + +--- + +## Test Generation Strategies + +### Strategy 1: Playwright Codegen (Recording) + +**How it works:** +```bash +npx playwright codegen http://localhost:3000 +``` + +Opens a browser where you perform actions. Playwright records and generates test code. + +**Example output:** +```typescript +test('filter occurrences by score', async ({ page }) => { + await page.goto('http://localhost:3000/projects/1/occurrences'); + await page.getByTestId('filter-score').click(); + await page.getByText('0.8+').click(); + await expect(page).toHaveURL(/score_min=0.8/); + await expect(page.getByTestId('occurrence-card')).toHaveCount(5); +}); +``` + +**Efficiency tips:** +- Record happy paths first +- One recording session per user flow +- Clean up generated code (remove unnecessary waits) +- Add assertions manually after recording + +**Time estimate:** 5-10 min per user flow + +--- + +### Strategy 2: Analytics-Driven Prioritization + +Use real user data to identify what to test first. + +#### From Google Analytics + +**Most visited pages:** +``` +GA > Behavior > Site Content > All Pages +``` + +Prioritize E2E tests for top 10 pages by pageviews. + +**Common user flows:** +``` +GA > Behavior > Behavior Flow +``` + +Identify the most common paths users take. + +**Example priority list from analytics:** +| Rank | Page/Flow | Est. % Traffic | +|------|-----------|----------------| +| 1 | Login → Projects | 100% | +| 2 | Projects → Occurrences | 80% | +| 3 | Occurrence filtering | 70% | +| 4 | Occurrence detail view | 60% | +| 5 | Agree with ID | 40% | +| 6 | Create job | 30% | +| 7 | View job progress | 25% | +| 8 | Deployments list | 20% | + +**Action:** Export top flows, create test for each. + +#### From NewRelic + +**Transaction traces:** +``` +NewRelic > APM > Transactions > Most time consuming +``` + +Identifies which API calls are most used → which features to test. + +**Error rates by page:** +``` +NewRelic > Browser > Page views > Sort by JS errors +``` + +Prioritize testing pages with high error rates. + +**User sessions:** +``` +NewRelic > Browser > Session traces +``` + +Watch real user sessions to understand actual usage patterns. + +--- + +### Strategy 3: AI-Assisted Test Generation + +#### Option A: Claude Code Generates from Routes + +Provide Claude with: +1. Route file +2. Component code +3. API contract + +Ask it to generate Playwright tests. + +**Prompt example:** +``` +Given this Remix route file for occurrences: +[paste loader, action, component] + +Generate Playwright E2E tests covering: +1. Page loads with data +2. Filtering updates URL and reloads data +3. Pagination works +4. Error states display correctly +``` + +**Efficiency:** Claude can generate 10-20 tests/hour with good coverage. + +#### Option B: Generate from OpenAPI Schema + +The project has an OpenAPI schema (`ami-openapi-schema.yaml`). + +```bash +# Generate API contract tests +npx playwright codegen --load-storage=auth.json \ + http://localhost:8000/api/v2/docs/ +``` + +Or use tools like: +- **Optic** - API contract testing from OpenAPI +- **Dredd** - API testing from API Blueprint/OpenAPI +- **Schemathesis** - Property-based API testing + +#### Option C: Generate from TypeScript Types + +Use the existing TypeScript interfaces to generate test data: + +```typescript +// data-services/types.ts has interfaces +// Generate test fixtures automatically + +import { faker } from '@faker-js/faker' +import type { Occurrence } from './types' + +export function createMockOccurrence(): Occurrence { + return { + id: faker.string.uuid(), + determination: { + id: faker.string.uuid(), + name: faker.animal.insect(), + }, + score: faker.number.float({ min: 0, max: 1 }), + // ... + } +} +``` + +--- + +### Strategy 4: Session Recording Analysis + +#### Tools for Recording Real Sessions + +| Tool | What it Captures | Export Format | +|------|------------------|---------------| +| **FullStory** | User sessions | Video + events | +| **Hotjar** | Heatmaps + recordings | Video | +| **LogRocket** | Sessions + errors | Video + logs | +| **NewRelic Browser** | Session traces | Event timeline | +| **Sentry Session Replay** | Error sessions | Video | + +#### Converting Recordings to Tests + +1. **Watch top 10 user sessions** from analytics +2. **Note the actions** users perform +3. **Record same actions** with Playwright codegen +4. **Add assertions** for expected outcomes + +**Example workflow:** +``` +FullStory session shows: +1. User logs in +2. Clicks project "Field Study 2024" +3. Filters occurrences by "Lepidoptera" +4. Clicks first occurrence +5. Agrees with ML prediction +6. Navigates to next occurrence + +→ Record this flow in Playwright codegen +→ Add assertions for data loading +→ Save as critical-path.spec.ts +``` + +--- + +### Strategy 5: Log-Based Test Generation + +#### From Application Logs + +If logging user actions: +```python +# Django logs might show: +INFO user=5 action=filter_occurrences params={"taxon": "123", "score_min": "0.8"} +INFO user=5 action=agree_identification occurrence=456 prediction=789 +``` + +Parse logs to identify: +- Most common actions +- Common parameter combinations +- Error-prone flows + +#### From API Access Logs + +```bash +# Parse nginx/Django access logs +cat access.log | grep "POST /api/v2/identifications" | wc -l +cat access.log | grep "GET /api/v2/occurrences" | awk '{print $7}' | sort | uniq -c | sort -rn | head -20 +``` + +Identifies: +- Most-used endpoints +- Common query parameter combinations +- Which filters are actually used + +--- + +## Recommended Test Structure + +### Directory Layout +``` +e2e/ +├── fixtures/ +│ ├── auth.json # Saved auth state +│ ├── test-data.ts # Mock data generators +│ └── api-mocks.ts # MSW handlers for offline testing +├── pages/ # Page Object Models +│ ├── login.page.ts +│ ├── projects.page.ts +│ ├── occurrences.page.ts +│ └── occurrence-detail.page.ts +├── tests/ +│ ├── auth.spec.ts +│ ├── critical-path.spec.ts # Most important flows +│ ├── occurrences.spec.ts +│ ├── jobs.spec.ts +│ └── visual-regression.spec.ts +├── playwright.config.ts +└── global-setup.ts # Login once, save state +``` + +### Page Object Model Example + +```typescript +// e2e/pages/occurrences.page.ts +import { Page, Locator, expect } from '@playwright/test' + +export class OccurrencesPage { + readonly page: Page + readonly filterScore: Locator + readonly filterTaxon: Locator + readonly occurrenceCards: Locator + readonly pagination: Locator + + constructor(page: Page) { + this.page = page + this.filterScore = page.getByTestId('filter-score') + this.filterTaxon = page.getByTestId('filter-taxon') + this.occurrenceCards = page.getByTestId('occurrence-card') + this.pagination = page.getByTestId('pagination') + } + + async goto(projectId: string) { + await this.page.goto(`/projects/${projectId}/occurrences`) + } + + async filterByScore(minScore: string) { + await this.filterScore.click() + await this.page.getByText(minScore).click() + await this.page.waitForURL(/score_min=/) + } + + async filterByTaxon(taxonName: string) { + await this.filterTaxon.click() + await this.page.getByPlaceholder('Search taxa').fill(taxonName) + await this.page.getByText(taxonName).first().click() + } + + async expectOccurrenceCount(count: number) { + await expect(this.occurrenceCards).toHaveCount(count) + } + + async clickOccurrence(index: number) { + await this.occurrenceCards.nth(index).click() + } +} +``` + +### Test Example Using Page Objects + +```typescript +// e2e/tests/occurrences.spec.ts +import { test, expect } from '@playwright/test' +import { OccurrencesPage } from '../pages/occurrences.page' + +test.describe('Occurrences', () => { + let occurrencesPage: OccurrencesPage + + test.beforeEach(async ({ page }) => { + occurrencesPage = new OccurrencesPage(page) + await occurrencesPage.goto('1') + }) + + test('displays occurrence gallery', async () => { + await expect(occurrencesPage.occurrenceCards.first()).toBeVisible() + }) + + test('filters by score update URL', async ({ page }) => { + await occurrencesPage.filterByScore('0.8+') + await expect(page).toHaveURL(/score_min=0\.8/) + }) + + test('filters by taxon', async ({ page }) => { + await occurrencesPage.filterByTaxon('Lepidoptera') + await expect(page).toHaveURL(/taxon=/) + }) + + test('clicking occurrence opens detail', async ({ page }) => { + await occurrencesPage.clickOccurrence(0) + await expect(page.getByTestId('occurrence-detail')).toBeVisible() + }) +}) +``` + +--- + +## Visual Regression Testing + +### Built-in Playwright Screenshots + +```typescript +test('occurrences page visual', async ({ page }) => { + await page.goto('/projects/1/occurrences') + await page.waitForLoadState('networkidle') + + await expect(page).toHaveScreenshot('occurrences-gallery.png', { + maxDiffPixelRatio: 0.01, + }) +}) +``` + +### Before/After Migration Comparison + +```bash +# Before migration: capture baselines +npx playwright test --update-snapshots + +# After migration: compare +npx playwright test + +# Review differences +npx playwright show-report +``` + +--- + +## Test Data Strategies + +### Option A: Use Real Backend (Recommended for Migration) + +```typescript +// playwright.config.ts +export default defineConfig({ + use: { + baseURL: 'http://localhost:3000', + }, + webServer: { + command: 'npm run dev', + port: 3000, + reuseExistingServer: true, + }, +}) +``` + +Requires `docker compose up` for Django backend. + +### Option B: Mock API with MSW + +```typescript +// e2e/fixtures/api-mocks.ts +import { rest } from 'msw' + +export const handlers = [ + rest.get('/api/v2/occurrences/', (req, res, ctx) => { + return res(ctx.json({ + results: [ + { id: '1', determination: { name: 'Species A' }, score: 0.95 }, + { id: '2', determination: { name: 'Species B' }, score: 0.87 }, + ], + count: 2, + })) + }), +] +``` + +Faster but less realistic. + +### Option C: Seed Database with Fixtures + +```bash +# Before tests +docker compose run --rm django python manage.py loaddata e2e_fixtures.json + +# After tests +docker compose run --rm django python manage.py flush --no-input +``` + +--- + +## Efficient Test Generation Plan + +### Phase 1: Critical Path Tests (2 hours) + +Record and verify the most important user flows: + +| Test | Method | Time | +|------|--------|------| +| Login/logout | Codegen | 15 min | +| View projects | Codegen | 10 min | +| Filter occurrences | Codegen | 20 min | +| View occurrence detail | Codegen | 15 min | +| Agree with ID | Codegen | 15 min | +| Create job | Codegen | 20 min | +| View job progress | Codegen | 15 min | + +**Output:** 7 critical tests covering ~80% of usage + +### Phase 2: Visual Baselines (1 hour) + +Capture screenshots of all major pages: + +```typescript +const pages = [ + '/projects', + '/projects/1/summary', + '/projects/1/occurrences', + '/projects/1/occurrences/1', + '/projects/1/jobs', + '/projects/1/deployments', + '/projects/1/settings', +] + +for (const path of pages) { + test(`visual: ${path}`, async ({ page }) => { + await page.goto(path) + await page.waitForLoadState('networkidle') + await expect(page).toHaveScreenshot() + }) +} +``` + +**Output:** Visual regression suite for all pages + +### Phase 3: Edge Cases (2 hours) + +AI-generate tests for: +- Empty states (no occurrences) +- Error states (API failure) +- Pagination edge cases +- Filter combinations +- Form validation + +**Method:** Provide Claude with component code, ask for edge case tests + +### Phase 4: Analytics-Driven Expansion (Ongoing) + +Weekly: +1. Check GA for new popular flows +2. Check error logs for broken flows +3. Add tests for uncovered areas + +--- + +## CI/CD Integration + +### GitHub Actions + +```yaml +# .github/workflows/e2e.yml +name: E2E Tests + +on: + push: + branches: [main] + pull_request: + +jobs: + e2e: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Setup Node + uses: actions/setup-node@v4 + with: + node-version: '20' + + - name: Install dependencies + run: | + cd ui + npm ci + npx playwright install --with-deps + + - name: Start services + run: docker compose up -d + + - name: Wait for services + run: | + timeout 60 bash -c 'until curl -s http://localhost:8000/api/v2/ > /dev/null; do sleep 2; done' + + - name: Run E2E tests + run: cd ui && npx playwright test + + - name: Upload report + if: always() + uses: actions/upload-artifact@v4 + with: + name: playwright-report + path: ui/playwright-report/ +``` + +--- + +## Quick Start Commands + +```bash +# Install Playwright +cd ui +npm init playwright@latest + +# Record a test +npx playwright codegen http://localhost:3000 + +# Run tests +npx playwright test + +# Run with UI mode (debugging) +npx playwright test --ui + +# View report +npx playwright show-report + +# Update visual snapshots +npx playwright test --update-snapshots +``` + +--- + +## Summary: Most Efficient Approach + +| Method | Effort | Coverage | Quality | +|--------|--------|----------|---------| +| Codegen (recording) | Low | High for happy paths | Good | +| AI generation | Low | Medium-high | Good | +| Manual writing | High | Precise | Best | +| Analytics-driven | Medium | Targets real usage | Excellent | + +**Recommended mix:** +1. **Codegen** for critical paths (7-10 tests) +2. **Visual regression** for all pages (automated) +3. **AI generation** for edge cases (10-20 tests) +4. **Analytics review** to prioritize (ongoing) + +**Total initial effort:** ~5 hours for comprehensive coverage +**Ongoing effort:** ~1 hour/week to maintain and expand diff --git a/.agents/planning/remix/01-remix-benefits.md b/.agents/planning/remix/01-remix-benefits.md new file mode 100644 index 000000000..cbf06c7f5 --- /dev/null +++ b/.agents/planning/remix/01-remix-benefits.md @@ -0,0 +1,405 @@ +# What Remix Can Offer + +## Overview + +Remix is a full-stack React framework built by the React Router team. It emphasizes web standards, progressive enhancement, and has strong opinions about data flow. It's often described as "the React framework that feels like Rails/Django." + +--- + +## Philosophy Comparison + +| Aspect | Next.js | Remix | +|--------|---------|-------| +| Primary focus | Static/SSR optimization | Data flow & web standards | +| Form handling | Basic (Server Actions) | First-class (``, actions) | +| Data loading | Flexible (many patterns) | Opinionated (loaders) | +| Mutations | Server Actions (newer) | Actions (mature pattern) | +| Error handling | error.tsx convention | errorElement + boundaries | +| Mental model | "Pages with data" | "Routes as API endpoints" | +| Progressive enhancement | Optional | Core principle | + +--- + +## What Remix Provides (vs Current Stack) + +### 1. Loaders: Prescribed Data Loading + +**Current Pattern** (React Query): +```typescript +// Multiple files, manual wiring +// hooks/occurrences/useOccurrences.ts +export function useOccurrences(params) { + return useQuery({ + queryKey: ['occurrences', params], + queryFn: () => fetchOccurrences(params), + }) +} + +// pages/occurrences/occurrences.tsx +export function Occurrences() { + const { data, isLoading, error } = useOccurrences(params) + if (isLoading) return + if (error) return + return +} +``` + +**Remix Pattern** (Loaders): +```typescript +// routes/projects.$projectId.occurrences.tsx +// ONE file, prescribed pattern + +// Data loading - runs on server +export async function loader({ params, request }: LoaderFunctionArgs) { + const token = await getAuthToken(request) + const url = new URL(request.url) + const filters = Object.fromEntries(url.searchParams) + + const occurrences = await fetchOccurrences(params.projectId, filters, token) + return json({ occurrences }) +} + +// Component - receives data, never loading state +export default function Occurrences() { + const { occurrences } = useLoaderData() + return +} +``` + +**Benefits**: +- Data fetching has ONE prescribed location (loader) +- Component never sees loading state (Remix handles it) +- Type safety between loader and component +- Automatic request deduplication +- Parallel data loading for nested routes + +### 2. Actions: Prescribed Mutation Handling + +**Current Pattern** (React Query mutations): +```typescript +// hooks/identifications/useCreateIdentification.ts +export function useCreateIdentification(onSuccess) { + const queryClient = useQueryClient() + return useMutation({ + mutationFn: (data) => axios.post('/api/v2/identifications/', data), + onSuccess: () => { + queryClient.invalidateQueries(['identifications']) + queryClient.invalidateQueries(['occurrences']) + onSuccess?.() + }, + }) +} + +// In component +const { mutate, isLoading } = useCreateIdentification(() => closeModal()) + { e.preventDefault(); mutate(formData) }}> +``` + +**Remix Pattern** (Actions): +```typescript +// routes/projects.$projectId.occurrences.$id.tsx + +// Mutation handling - runs on server +export async function action({ params, request }: ActionFunctionArgs) { + const token = await getAuthToken(request) + const formData = await request.formData() + const intent = formData.get('intent') + + if (intent === 'create-identification') { + const taxonId = formData.get('taxonId') + await createIdentification(params.id, taxonId, token) + return json({ success: true }) + } + + if (intent === 'agree') { + await agreeWithPrediction(params.id, token) + return json({ success: true }) + } + + return json({ error: 'Unknown intent' }, { status: 400 }) +} + +// In component - use Remix's Form +export default function OccurrenceDetail() { + const navigation = useNavigation() + const isSubmitting = navigation.state === 'submitting' + + return ( + + + + + ) +} +``` + +**Benefits**: +- Mutations have ONE prescribed location (action) +- Form submission is standard HTML (progressive enhancement) +- Automatic revalidation after mutations +- No manual cache invalidation +- Pending states via `useNavigation()` + +### 3. Forms: First-Class Support + +**Current Pattern**: +```typescript +// Manual preventDefault, manual state, manual submission +const { register, handleSubmit, formState } = useForm() +const { mutate, isLoading } = useCreateJob() + +return ( +
mutate(data))}> + + +
+) +``` + +**Remix Pattern**: +```typescript +// Remix Form component with built-in states +import { Form, useNavigation, useActionData } from '@remix-run/react' + +export default function NewJob() { + const navigation = useNavigation() + const actionData = useActionData() + + return ( +
+ + {actionData?.errors?.name && ( + {actionData.errors.name} + )} + +
+ ) +} + +// Action handles submission and validation +export async function action({ request }: ActionFunctionArgs) { + const formData = await request.formData() + const name = formData.get('name') + + if (!name) { + return json({ errors: { name: 'Name is required' } }, { status: 400 }) + } + + const job = await createJob({ name }) + return redirect(`/jobs/${job.id}`) +} +``` + +**Benefits**: +- `
` works without JavaScript (progressive enhancement) +- Pending state via `useNavigation()` - ONE pattern +- Validation errors returned from action +- File uploads work naturally with FormData +- Can still use react-hook-form for complex client validation + +### 4. File Uploads (Mixed Content Forms) + +**Remix handles multipart forms naturally**: + +```typescript +// routes/deployments.$id.tsx +export async function action({ request }: ActionFunctionArgs) { + const formData = await request.formData() + + // Text fields + const name = formData.get('name') as string + const description = formData.get('description') as string + + // File upload + const imageFile = formData.get('image') as File + if (imageFile && imageFile.size > 0) { + const buffer = await imageFile.arrayBuffer() + await uploadToS3(buffer, imageFile.name) + } + + await updateDeployment({ name, description, imageUrl }) + return redirect('/deployments') +} + +// Component +export default function EditDeployment() { + return ( + + +