diff --git a/backend/docs/IMAGE_MIGRATION_PLAN.md b/backend/docs/IMAGE_MIGRATION_PLAN.md new file mode 100644 index 00000000..815cae1b --- /dev/null +++ b/backend/docs/IMAGE_MIGRATION_PLAN.md @@ -0,0 +1,184 @@ +# Image Migration Plan (Legacy -> Tenant-Aware v2) + +This runbook migrates existing image URLs into the v2 tenant-aware structure with generated variants and modern formats. + +--- + +## Objectives + +- No downtime and no broken images. +- Idempotent and resumable migration. +- Backfill all legacy image references (orgs, users, classrooms, and event images when available). +- Preserve auditability and rollback path. + +--- + +## Scope + +### In-scope legacy fields (current wave) + +- `User.picture` +- `Org.org_profile_image` +- `Org.org_banner_image` +- `Classroom.image` +- Event image fields are deferred to a later wave after Events-Backend alignment + +### Out-of-scope + +- Client-side hardcoded static asset paths in frontend build artifacts +- Third-party/non-bucket URLs (do not import in this migration) + +--- + +## Migration architecture + +Use a dedicated migration worker (script/job) with a durable checkpoint table/collection: + +- `image_migration_jobs` + - `tenantKey` + - `entityType` + - `entityId` + - `fieldName` + - `legacyUrl` + - `status` (`pending`, `processing`, `migrated`, `failed`, `skipped`) + - `attempts` + - `lastError` + - `newAssetRef` + - timestamps + +This makes migration restart-safe and observable. + +--- + +## Phases + +## Phase 0: Preflight and inventory + +1. Enumerate tenants from tenant config. +2. For each tenant DB, scan in-scope collections for non-empty image fields. +3. Emit inventory counts by tenant/entity/field. +4. Classify each URL: + - bucket-owned legacy URL + - already v2-formatted URL + - external URL + +Gate to continue: + +- >99% of bucket-owned references are resolvable or intentionally skipped by policy. + +--- + +## Phase 1: Build migration queue + +For each bucket-owned resolvable legacy URL create a `pending` job. + +Deduplicate by `(tenantKey, entityType, entityId, fieldName, legacyUrl)` to avoid duplicate work. + +--- + +## Phase 2: Backfill worker + +For each job: + +1. Download legacy image bytes. +2. Decode and validate (size/dimension/pixel limits). + - GIF inputs are converted to static first-frame output (animation is not preserved). +3. Select role from field mapping: + - `User.picture` -> `user_avatar` + - `Org.org_profile_image` -> `org_profile` + - `Org.org_banner_image` -> `org_banner` + - `Classroom.image` -> `classroom_photo` +4. Generate v2 variant+format outputs. +5. Upload to tenant-aware keys. +6. Update DB document: + - keep legacy field for now (compatibility) + - add new manifest/reference field +7. Mark job `migrated`. + +Failure handling: + +- Retry transient failures with exponential backoff. +- After max retries, mark `failed` and continue. + +--- + +## Phase 3: Dual-read rollout + +Update backend read serializers and frontend consumers: + +- Prefer v2 manifest reference if present. +- Fallback to legacy URL when absent. + +This allows progressive migration without user-visible regressions. + +--- + +## Phase 4: Validation and confidence gates + +Validate with automated checks: + +- Count parity: + - `migrated + skipped + failed == queued` +- Sampling: + - random N images per tenant + role render correctly +- Performance: + - thumbnail payload reduction vs legacy baseline + +Release gate recommendation: + +- <= 0.5% failed jobs and no critical tenant regressions. + +--- + +## Phase 5: Cutover writes + +After confidence window: + +1. Disable legacy-only write paths. +2. Keep dual-read for one release window. +3. New uploads only write v2 references. + +--- + +## Phase 6: Legacy cleanup + +After log-confirmed inactivity window: + +1. Delete unreferenced legacy objects in batches. +2. Keep tombstone log for deletions (object key + timestamp + job id). +3. Remove legacy URL fields after final validation (optional, can be deferred). + +Retention: keep superseded legacy objects for 30 days, then delete. + +--- + +## Rollback plan + +At any point before cleanup: + +- Flip read preference back to legacy fields. +- Pause migration workers. +- Keep v2 objects (safe side-by-side storage). + +Because legacy values are not deleted early, rollback is low-risk. + +--- + +## Recommended implementation order in this repo + +1. Replace ad hoc route uploads with shared policy-based service. +2. Add manifest/reference fields to user/org/classroom schemas (non-breaking optional). +3. Add migration queue collection and worker script. +4. Execute tenant-by-tenant migration with dashboards. +5. Cutover reads, then writes, then cleanup. + +--- + +## Decisions locked for execution + +1. Migrate bucket-owned assets only. +2. Legacy retention window is 30 days. +3. No GIF animation support; convert to static first frame when encountered. +4. Delivery defaults to private signed access. +5. Event image migration is staged after Events-Backend alignment. + diff --git a/backend/docs/IMAGE_POLICY.md b/backend/docs/IMAGE_POLICY.md new file mode 100644 index 00000000..0e59ab32 --- /dev/null +++ b/backend/docs/IMAGE_POLICY.md @@ -0,0 +1,220 @@ +# Tenant-Aware Image Policy (v2) + +## Why this policy exists + +Current behavior has three major issues: + +1. Original uploads are reused for every context (thumbnail, card, full, hero), which inflates bandwidth and slows pages. +2. S3 object organization is not tenant-scoped, which weakens multi-tenant isolation and governance. +3. Storage conventions are not centrally enforced, so routes can drift into ad hoc naming and inconsistent quality. + +This document defines the v2 image policy that standardizes ingestion, storage, delivery, and governance. + +--- + +## Policy goals + +- Make every image tenant-aware and owner-aware. +- Generate right-sized derivatives for UI contexts (thumb/card/full/hero). +- Convert to modern formats (AVIF + WebP) with JPEG fallback. +- Keep policy enforceable through shared backend code. +- Support zero-downtime migration from legacy URLs. + +--- + +## Source of truth in code + +`backend/services/imagePolicyService.js` is the policy module for: + +- tenant key validation +- deterministic key naming +- entity-role validation +- allowed variants/formats +- upload plan generation + +All image ingestion routes should call this service rather than assembling keys manually. + +--- + +## Tenant-aware object key structure + +Every image variant MUST be stored under: + +`v2/tenants/{tenantKey}/images/{entityType}/{entityId}/{role}/{yyyy}/{mm}/{assetId}/{version}/{variant}.{format}` + +Example: + +`v2/tenants/rpi/images/org/65f1.../org_banner/2026/04/2cb7.../v1/card.avif` + +Rules: + +- `tenantKey` comes from `req.school`. +- `entityType` is one of: `org`, `user`, `classroom`, `event`. +- `role` is semantic (e.g. `org_profile`, `event_cover`), not route name. +- `assetId` is immutable per upload. +- `version` increments when the same logical slot is replaced. + +This prevents collisions and supports deterministic auditing/deletion. + +--- + +## Variant + format matrix + +Generate all listed variants per uploaded role. + +### Org profile (`org_profile`) +- `thumb` 96x96 cover +- `card` 256x256 cover +- `full` 512x512 cover + +### Org banner (`org_banner`) +- `thumb` 480x200 cover +- `card` 960x360 cover +- `full` 1920x720 cover + +### User avatar (`user_avatar`) +- `thumb` 64x64 cover +- `card` 128x128 cover +- `full` 320x320 cover + +### Classroom photo (`classroom_photo`) +- `thumb` 320x180 cover +- `card` 640x360 cover +- `full` 1600x900 inside + +### Event cover (`event_cover`) +- `thumb` 320x180 cover +- `card` 640x360 cover +- `full` 1600x900 inside +- `hero` 1920x1080 inside + +### Generated formats + +Per variant, generate: + +- `avif` (preferred) +- `webp` +- `jpeg` (fallback) + +Delivery preference order: AVIF -> WebP -> JPEG. + +--- + +## Ingestion requirements + +### Input validation + +- Max upload size: 10 MB. +- Max decoded dimensions: 6000 x 6000. +- Max decoded pixels: 20 MP. +- Accepted MIME: JPEG/PNG/WebP. +- GIF uploads are not supported in v2. + +### Normalization + +- Auto-orient via EXIF. +- Strip EXIF metadata unless explicitly needed. +- Convert to sRGB. +- Enforce deterministic encoder settings by role. + +### Ownership checks + +Before writing objects: + +- Resolve target `entityType/entityId` from route. +- Authorize caller against that owner. +- Ensure role is valid for that entity type. + +If role/entity mismatch is detected, reject request with `400`. + +--- + +## Delivery model + +### Canonical references + +For each image slot, store a lightweight manifest reference: + +- `assetId` +- `version` +- `tenantKey` +- `entityType` +- `entityId` +- `role` +- available variants/formats + +### API selection + +Clients should request a semantic variant (`thumb`, `card`, `full`, `hero`) instead of raw pixel widths. + +If the browser supports AVIF/WebP, deliver those. Otherwise JPEG fallback. + +### Access control (recommended standard for this platform) + +Because these assets are primarily internal to Meridian and not intended for broad external reuse: + +- Keep S3 objects private. +- Serve through CDN with signed URLs/cookies (`private_signed` mode). +- Use short-lived signatures and immutable object keys. + +This is the standard default for multi-tenant SaaS where images are mostly app-internal. + +--- + +## Cache and lifecycle policy + +- Variant objects: `Cache-Control: public, max-age=31536000, immutable`. +- Manifest/API responses: shorter TTL (for example 60 seconds to 5 minutes). +- Enable lifecycle rules for superseded versions after retention window. + +Retention policy: + +- Keep previous version for 30 days after replacement. +- Delete superseded versions after the 30-day retention window. + +--- + +## Security and abuse controls + +- Validate MIME and decodeability (not MIME alone). +- Reject malformed/zip-bomb-like inputs. +- Block SVG upload unless sanitized pipeline exists. +- Keep strict role/entity allowlist in one policy module. +- Log each ingest with tenant, entity, role, assetId, bytes. + +--- + +## Enforcement in this codebase + +### Current gaps found + +- `imageUploadService` currently writes pass-through originals and does not generate variants. +- Several routes still build raw keys manually or use legacy upload paths. +- Multi-tenant key namespace is not consistently applied. + +### Required route upgrades + +Move all image writes to one ingestion path and remove direct `s3.upload(...)` usages in routes. + +High-priority routes: + +- `/upload-user-image` +- `/create-org` and `/edit-org` +- `/org-management/organizations/:orgId/edit` +- `/admin/rooms/:id/image` +- legacy classroom upload routes (`/upload-image/:classroomName`) should be deprecated. + +--- + +## Rollout strategy + +1. Add policy service and shared upload pipeline. +2. Update write routes to dual-write: + - new manifest/reference fields + - legacy URL fields for backward compatibility +3. Update read paths to prefer v2 manifest-derived variant URLs. +4. Backfill legacy images (see `backend/docs/IMAGE_MIGRATION_PLAN.md`). +5. Remove legacy writes and clean old objects after validation window. + +Event image migration is intentionally staged until Events-Backend alignment is complete. + diff --git a/backend/services/imagePolicyService.js b/backend/services/imagePolicyService.js new file mode 100644 index 00000000..3921f6e5 --- /dev/null +++ b/backend/services/imagePolicyService.js @@ -0,0 +1,267 @@ +const crypto = require('crypto'); + +const IMAGE_ENTITY_TYPES = Object.freeze({ + ORG: 'org', + USER: 'user', + CLASSROOM: 'classroom', + EVENT: 'event', +}); + +const IMAGE_FORMATS = Object.freeze(['avif', 'webp', 'jpeg']); +const DEFAULT_DELIVERY_FORMAT_ORDER = Object.freeze(['avif', 'webp', 'jpeg']); +const SUPPORTED_INPUT_MIME_TYPES = Object.freeze([ + 'image/jpeg', + 'image/png', + 'image/webp', +]); + +const DELIVERY_ACCESS_MODE = Object.freeze({ + default: 'private_signed', + description: 'Private S3 objects delivered through signed CDN URLs/cookies', +}); + +const POLICY_LIMITS = Object.freeze({ + maxUploadBytes: 10 * 1024 * 1024, + maxPixelCount: 20_000_000, + maxWidth: 6000, + maxHeight: 6000, +}); + +/** + * Variant presets are intentionally role-based, not route-based: + * a route chooses one role and receives deterministic outputs. + */ +const IMAGE_ROLE_PRESETS = Object.freeze({ + org_profile: { + entityType: IMAGE_ENTITY_TYPES.ORG, + variants: { + thumb: { width: 96, height: 96, fit: 'cover' }, + card: { width: 256, height: 256, fit: 'cover' }, + full: { width: 512, height: 512, fit: 'cover' }, + }, + }, + org_banner: { + entityType: IMAGE_ENTITY_TYPES.ORG, + variants: { + thumb: { width: 480, height: 200, fit: 'cover' }, + card: { width: 960, height: 360, fit: 'cover' }, + full: { width: 1920, height: 720, fit: 'cover' }, + }, + }, + user_avatar: { + entityType: IMAGE_ENTITY_TYPES.USER, + variants: { + thumb: { width: 64, height: 64, fit: 'cover' }, + card: { width: 128, height: 128, fit: 'cover' }, + full: { width: 320, height: 320, fit: 'cover' }, + }, + }, + classroom_photo: { + entityType: IMAGE_ENTITY_TYPES.CLASSROOM, + variants: { + thumb: { width: 320, height: 180, fit: 'cover' }, + card: { width: 640, height: 360, fit: 'cover' }, + full: { width: 1600, height: 900, fit: 'inside' }, + }, + }, + event_cover: { + entityType: IMAGE_ENTITY_TYPES.EVENT, + variants: { + thumb: { width: 320, height: 180, fit: 'cover' }, + card: { width: 640, height: 360, fit: 'cover' }, + full: { width: 1600, height: 900, fit: 'inside' }, + hero: { width: 1920, height: 1080, fit: 'inside' }, + }, + }, +}); + +function normalizeTenantKey(tenantKey) { + const normalized = String(tenantKey || '').trim().toLowerCase(); + if (!normalized) { + throw new Error('tenantKey is required'); + } + if (!/^[a-z0-9-]+$/.test(normalized)) { + throw new Error('tenantKey must match ^[a-z0-9-]+$'); + } + return normalized; +} + +function sanitizePathSegment(value, label) { + const normalized = String(value || '').trim().toLowerCase(); + if (!normalized) { + throw new Error(`${label} is required`); + } + if (!/^[a-z0-9-_]+$/.test(normalized)) { + throw new Error(`${label} must match ^[a-z0-9-_]+$`); + } + return normalized; +} + +function normalizeEntityId(entityId) { + const normalized = String(entityId || '').trim(); + if (!normalized) { + throw new Error('entityId is required'); + } + return normalized; +} + +function assertRolePreset(role) { + const preset = IMAGE_ROLE_PRESETS[role]; + if (!preset) { + throw new Error(`Unsupported image role: ${role}`); + } + return preset; +} + +function createAssetId() { + return crypto.randomUUID(); +} + +function isSupportedInputMimeType(mimeType) { + return SUPPORTED_INPUT_MIME_TYPES.includes(String(mimeType || '').toLowerCase()); +} + +function buildImageObjectPrefix({ + tenantKey, + entityType, + entityId, + role, + assetId, + version = 'v1', + now = new Date(), +}) { + const tenant = normalizeTenantKey(tenantKey); + const normalizedEntityType = sanitizePathSegment(entityType, 'entityType'); + const normalizedEntityId = normalizeEntityId(entityId); + const normalizedRole = sanitizePathSegment(role, 'role'); + const normalizedAssetId = sanitizePathSegment(assetId, 'assetId'); + const normalizedVersion = sanitizePathSegment(version, 'version'); + + const year = now.getUTCFullYear(); + const month = String(now.getUTCMonth() + 1).padStart(2, '0'); + + return [ + 'v2', + 'tenants', + tenant, + 'images', + normalizedEntityType, + normalizedEntityId, + normalizedRole, + String(year), + month, + normalizedAssetId, + normalizedVersion, + ].join('/'); +} + +function buildVariantObjectKey({ + tenantKey, + entityType, + entityId, + role, + assetId, + version = 'v1', + variant, + format, + now = new Date(), +}) { + const normalizedVariant = sanitizePathSegment(variant, 'variant'); + const normalizedFormat = sanitizePathSegment(format, 'format'); + const prefix = buildImageObjectPrefix({ + tenantKey, + entityType, + entityId, + role, + assetId, + version, + now, + }); + + return `${prefix}/${normalizedVariant}.${normalizedFormat}`; +} + +function buildImageUploadPlan({ + tenantKey, + entityType, + entityId, + role, + version = 'v1', + assetId = createAssetId(), + variants, + formats = IMAGE_FORMATS, + now = new Date(), +}) { + const preset = assertRolePreset(role); + const normalizedEntityType = sanitizePathSegment(entityType, 'entityType'); + if (preset.entityType !== normalizedEntityType) { + throw new Error(`Role ${role} requires entityType ${preset.entityType}`); + } + + const allVariantNames = Object.keys(preset.variants); + const selectedVariants = variants && variants.length ? variants : allVariantNames; + + selectedVariants.forEach((variantName) => { + if (!preset.variants[variantName]) { + throw new Error(`Unsupported variant ${variantName} for role ${role}`); + } + }); + + const normalizedFormats = formats.map((fmt) => sanitizePathSegment(fmt, 'format')); + normalizedFormats.forEach((fmt) => { + if (!IMAGE_FORMATS.includes(fmt)) { + throw new Error(`Unsupported format ${fmt}`); + } + }); + + const entries = []; + selectedVariants.forEach((variantName) => { + normalizedFormats.forEach((format) => { + entries.push({ + role, + variant: variantName, + format, + resize: preset.variants[variantName], + key: buildVariantObjectKey({ + tenantKey, + entityType: normalizedEntityType, + entityId, + role, + assetId, + version, + variant: variantName, + format, + now, + }), + }); + }); + }); + + return { + tenantKey: normalizeTenantKey(tenantKey), + entityType: normalizedEntityType, + entityId: normalizeEntityId(entityId), + role, + assetId, + version, + formats: normalizedFormats, + variants: selectedVariants, + entries, + }; +} + +module.exports = { + IMAGE_ENTITY_TYPES, + IMAGE_FORMATS, + SUPPORTED_INPUT_MIME_TYPES, + DEFAULT_DELIVERY_FORMAT_ORDER, + DELIVERY_ACCESS_MODE, + POLICY_LIMITS, + IMAGE_ROLE_PRESETS, + normalizeTenantKey, + buildImageObjectPrefix, + buildVariantObjectKey, + buildImageUploadPlan, + createAssetId, + isSupportedInputMimeType, +}; diff --git a/backend/services/imageUploadService.js b/backend/services/imageUploadService.js index aebd2776..4d2ecea1 100644 --- a/backend/services/imageUploadService.js +++ b/backend/services/imageUploadService.js @@ -7,7 +7,6 @@ const crypto = require('crypto'); const ALLOWED_MIME_TYPES = [ 'image/jpeg', 'image/png', - 'image/gif', 'image/webp' ]; @@ -17,7 +16,7 @@ const MAX_FILE_SIZE = 5 * 1024 * 1024; // File type validation middleware const fileFilter = (req, file, cb) => { if (!ALLOWED_MIME_TYPES.includes(file.mimetype)) { - cb(new Error('Invalid file type. Only JPEG, PNG, GIF, and WebP images are allowed.'), false); + cb(new Error('Invalid file type. Only JPEG, PNG, and WebP images are allowed.'), false); return; } cb(null, true); diff --git a/backend/tests/unit/imagePolicyService.test.js b/backend/tests/unit/imagePolicyService.test.js new file mode 100644 index 00000000..9fa0bacf --- /dev/null +++ b/backend/tests/unit/imagePolicyService.test.js @@ -0,0 +1,90 @@ +const { + IMAGE_ENTITY_TYPES, + IMAGE_ROLE_PRESETS, + SUPPORTED_INPUT_MIME_TYPES, + DELIVERY_ACCESS_MODE, + buildImageObjectPrefix, + buildVariantObjectKey, + buildImageUploadPlan, + isSupportedInputMimeType, + normalizeTenantKey, +} = require('../../services/imagePolicyService'); + +describe('imagePolicyService', () => { + test('normalizes tenant keys and rejects invalid values', () => { + expect(normalizeTenantKey('RPI')).toBe('rpi'); + expect(() => normalizeTenantKey('')).toThrow('tenantKey is required'); + expect(() => normalizeTenantKey('rpi!')).toThrow('tenantKey must match'); + }); + + test('builds deterministic tenant-aware prefix', () => { + const prefix = buildImageObjectPrefix({ + tenantKey: 'rpi', + entityType: IMAGE_ENTITY_TYPES.ORG, + entityId: '65f1a', + role: 'org_profile', + assetId: 'asset-123', + version: 'v2', + now: new Date('2026-04-01T00:00:00Z'), + }); + + expect(prefix).toBe('v2/tenants/rpi/images/org/65f1a/org_profile/2026/04/asset-123/v2'); + }); + + test('builds deterministic variant key', () => { + const key = buildVariantObjectKey({ + tenantKey: 'tvcog', + entityType: IMAGE_ENTITY_TYPES.USER, + entityId: 'abc123', + role: 'user_avatar', + assetId: 'asset-xyz', + version: 'v1', + variant: 'thumb', + format: 'webp', + now: new Date('2026-04-01T00:00:00Z'), + }); + + expect(key).toBe('v2/tenants/tvcog/images/user/abc123/user_avatar/2026/04/asset-xyz/v1/thumb.webp'); + }); + + test('builds complete upload plan with all default variants and formats', () => { + const plan = buildImageUploadPlan({ + tenantKey: 'rpi', + entityType: IMAGE_ENTITY_TYPES.ORG, + entityId: 'org1', + role: 'org_profile', + assetId: 'asset-1', + now: new Date('2026-04-01T00:00:00Z'), + }); + + const variantCount = Object.keys(IMAGE_ROLE_PRESETS.org_profile.variants).length; + expect(plan.entries).toHaveLength(variantCount * 3); + expect(plan.entries[0].key.startsWith('v2/tenants/rpi/images/org/org1/org_profile/2026/04/asset-1/v1/')).toBe(true); + }); + + test('validates role-to-entity alignment and variant allowlist', () => { + expect(() => buildImageUploadPlan({ + tenantKey: 'rpi', + entityType: IMAGE_ENTITY_TYPES.ORG, + entityId: 'org1', + role: 'user_avatar', + assetId: 'asset-2', + })).toThrow('requires entityType user'); + + expect(() => buildImageUploadPlan({ + tenantKey: 'rpi', + entityType: IMAGE_ENTITY_TYPES.ORG, + entityId: 'org1', + role: 'org_profile', + assetId: 'asset-3', + variants: ['invalid_variant'], + })).toThrow('Unsupported variant'); + }); + + test('supports only approved input mime types and private delivery mode', () => { + expect(SUPPORTED_INPUT_MIME_TYPES).toEqual(['image/jpeg', 'image/png', 'image/webp']); + expect(isSupportedInputMimeType('image/jpeg')).toBe(true); + expect(isSupportedInputMimeType('image/gif')).toBe(false); + expect(DELIVERY_ACCESS_MODE.default).toBe('private_signed'); + }); +});