From 443f83248a7c9506a66971b16ccf005859366dcd Mon Sep 17 00:00:00 2001 From: Sean Silvius Date: Sat, 11 Apr 2026 11:48:19 -0700 Subject: [PATCH 1/2] docs(packages): fix interface-shape drift in 5 adapter docs After shipping the docs in PR #76 I spot-read them and found real interface drift -- types, method signatures, and import paths that did not match the current source. These would have broken every consumer that copy-pasted the examples. The @vault-2026 technical accuracy review covered the schema sections of core-reference.md but did not cover the adapter package docs, which is where these errors sit. - EmailProvider.sendEmail: type names wrong. Doc said `sendEmail(params: SendEmailParams): Promise`. Actual is `sendEmail(params: EmailParams): Promise<{ id: string }>`. - EmailParams field table wrong: `from` is optional not required, `to` is a single string not array, `cc`/`bcc`/`headers` do not exist in EmailParams. - Documented the full EmailProvider surface (mailing lists, subscribers, campaigns, audiences), which was omitted. - Method calls wrong: doc called `mailService.compose(...)` and `mailService.replyToThread(...)`. Actual methods are `InboxEmailService.composeEmail` and `replyToThread`. - Thread metadata update: doc said `messageCount`, schema has `unreadCount`. - BlobStorage interface signatures wrong: missing `options?` on `put` and `get`, `get` return type was `string | undefined` (actual is `BlobObject | null`), and `generateKey` was missing entirely. - Added BlobObject, BlobPutOptions, and BlobGetOptions types alongside so consumers see the full shape. - InboundAdapter.handleIncoming return type wrong: said `Promise`, actual is `Promise<{ messageId, threadId }>`. - "Inbound email structure" section described a 14-field parsed form as if it were the input type. The actual InboundEmail is 4 fields: `raw: ArrayBuffer`, `from`, `to`, `headers`. The parsed fields land on the message row AFTER the adapter does parsing work. Rewrote with a clear split between input shape and parsed-form schema fields. - Classification section: `ClassificationAdapter` should be `EmailClassifier`, `summary` field does not exist on `EmailClassification` (actual fields are `tags`, `priority`). - Interface name wrong: `ClassificationAdapter` should be `EmailClassifier`. - Method signature wrong: `classify(message: ClassificationInput)` should be `classify(from: string, subject: string, body: string)`. - Result type wrong: `ClassificationResult` with `summary` field should be `EmailClassification` with `tags` and `priority` fields. - Example classifier implementation updated to the correct shape. - Worker example imported `createR2BlobStorage` and `parseInboundEmail` from the package root. Actual exports are `createR2Storage` (from `/storage` subpath) and `parseEmailHeaders` / `hashContent` (from `/parsing` subpath). Also `createR2Storage` takes `{ bucket }` config, not a raw R2Bucket. - `blobStorage.put(parsed.rawEmail)` was missing the required `key` parameter. Rewrote to use `storage.generateKey` and `storage.put(key, raw)`. - `sendEmail` example showed `to: [array]`, actual is a single string. - "Next steps" link list pointed at 5 docs at `./classification.md`, `./imap-quickstart.md`, etc. Only `./classification.md` does not exist in this package -- the rest live in other packages. Replaced with per-package npm links. - pnpm test: 609 passing across 37 files (no code change, same count as before) - pnpm typecheck: clean - pnpm lint: 0 warnings, 0 errors - oxfmt normalized the 5 modified docs Co-Authored-By: Claude Opus 4.6 (1M context) --- packages/cloudflare/docs/blob-storage.md | 27 +++++- packages/cloudflare/docs/inbound.md | 87 +++++++++++++------- packages/cloudflare/docs/quickstart.md | 42 ++++++---- packages/resend/docs/outbound.md | 96 +++++++++++++++------- packages/workers-ai/docs/classification.md | 36 ++++---- 5 files changed, 190 insertions(+), 98 deletions(-) diff --git a/packages/cloudflare/docs/blob-storage.md b/packages/cloudflare/docs/blob-storage.md index 2817913..cd7f2ce 100644 --- a/packages/cloudflare/docs/blob-storage.md +++ b/packages/cloudflare/docs/blob-storage.md @@ -14,13 +14,34 @@ Email bodies and attachments are large (kilobytes to megabytes). Storing them in ```typescript interface BlobStorage { - put(key: string, content: string | ArrayBuffer): Promise; - get(key: string): Promise; + put(key: string, content: string | ArrayBuffer, options?: BlobPutOptions): Promise; + get(key: string, options?: BlobGetOptions): Promise; delete(key: string): Promise; + generateKey(contentHash: string, extension: string): string; +} + +interface BlobObject { + text(): Promise; + arrayBuffer(): Promise; + httpMetadata?: Record; + customMetadata?: Record; +} + +interface BlobPutOptions { + httpMetadata?: Record; + customMetadata?: Record; +} + +interface BlobGetOptions { + range?: { offset: number; length: number }; } ``` -Any storage backend that supports key-value blob operations works: object storage (S3-compatible), file system, KV stores. +`get` returns a lazy `BlobObject` (not a string) so consumers choose whether to decode as text or binary via `.text()` / `.arrayBuffer()`. Partial reads are supported through `BlobGetOptions.range`, which IMAP `FETCH BODY[]` maps directly onto. + +`generateKey(contentHash, extension)` centralizes the key schema so applications do not hand-roll paths. The Cloudflare implementation builds keys of the form `emails/{year}/{month}/{contentHash}.{extension}`. + +Any storage backend that supports key-value blob operations works: object storage (S3-compatible), file system, KV stores. Ship-ready adapters in the repo: `createR2Storage` from `@rafters/mail-cloudflare/storage`. --- diff --git a/packages/cloudflare/docs/inbound.md b/packages/cloudflare/docs/inbound.md index 8c61fe4..2e7e002 100644 --- a/packages/cloudflare/docs/inbound.md +++ b/packages/cloudflare/docs/inbound.md @@ -10,39 +10,51 @@ The `InboundAdapter` interface handles incoming email. It receives a raw email, ```typescript interface InboundAdapter { - handleIncoming(email: InboundEmail): Promise; + handleIncoming(email: InboundEmail): Promise<{ + messageId: string; + threadId: string; + }>; +} + +interface InboundEmail { + raw: ArrayBuffer; // the complete RFC 5322 message as bytes + from: string; // parsed From header, for routing + to: string; // parsed To header, for mailbox resolution + headers: Record; // all headers as key-value pairs } ``` The adapter is responsible for: -1. Parsing the raw RFC 5322 email into structured fields -2. Finding or creating a thread (via In-Reply-To/References headers) +1. Parsing the raw RFC 5322 message from `raw` into structured fields (subject, date, CC, BCC, In-Reply-To, References, body parts, attachments) +2. Finding or creating a thread using the `In-Reply-To` and `References` headers 3. Storing the raw email and parsed content in blob storage 4. Creating the message record in the database -5. Updating thread metadata (messageCount, lastMessageAt, snippet) +5. Updating thread metadata (`unreadCount`, `lastMessageAt`, snippet) +6. Returning the resulting `messageId` and `threadId` for caller correlation --- -## Inbound email structure - -When an email arrives, the adapter receives: - -| Field | Source | -| ----------- | ---------------------------------- | -| from | From header (email + display name) | -| to | To header | -| cc, bcc | CC/BCC headers | -| subject | Subject header | -| messageId | Message-ID header | -| inReplyTo | In-Reply-To header | -| references | References header | -| date | Date header | -| textBody | Plain text content | -| htmlBody | HTML content | -| rawEmail | The complete RFC 5322 message | -| attachments | MIME attachments with metadata | -| headers | All headers as key-value pairs | +## Input vs parsed form + +The `InboundEmail` type received by `handleIncoming` is minimal on purpose: `raw`, `from`, `to`, and `headers`. The adapter does the parsing, not the caller. + +After parsing, the message row in the database carries the richer shape: + +| Field | Source | +| ---------------------------- | -------------------------- | +| `fromEmail`, `fromName` | From header | +| `toEmail`, `toName` | To header | +| `subject` | Subject header | +| `messageIdHeader` | Message-ID header | +| `inReplyTo` | In-Reply-To header | +| `references` | References header | +| `sentAt` | Date header | +| `receivedAt` | When ingestion happened | +| `blobKeyRaw` | Key for the raw .eml blob | +| `blobKeyHtml`, `blobKeyText` | Keys for parsed body blobs | +| `sizeBytes` | Raw blob size | +| `attachmentCount` | Count of MIME attachments | --- @@ -98,15 +110,32 @@ _dmarc.yourdomain.com TXT "v=DMARC1; p=quarantine; rua=mailto:dmarc@yourdomain ## Classification -After storage, the inbound adapter can optionally classify the message using an AI classifier. The classifier assigns: +After storage, the inbound adapter can optionally classify the message using an AI classifier. The classifier interface is `EmailClassifier` from `@rafters/mail/interfaces`: -- **category**: support, feedback, abuse, partnership, spam, billing, legal, other -- **confidence**: 0-100 score -- **summary**: one-line description of the message content +```typescript +interface EmailClassifier { + classify(from: string, subject: string, body: string): Promise; +} + +interface EmailClassification { + category: + | "support" + | "feedback" + | "abuse" + | "partnership" + | "spam" + | "billing" + | "legal" + | "other"; + confidence: number; // 0..100 + tags: string[]; // configurable pattern-matched tags + priority: "urgent" | "high" | "normal" | "low"; +} +``` -Classification results are stored on the message record. They drive folder assignment, priority, and dashboard filtering. +Classification results are stored on the message record (`aiCategory`, `aiConfidence`, priority, tags via label join). They drive folder assignment, thread priority, and dashboard filtering. -The classifier is a separate adapter (`ClassificationAdapter`). The default implementation uses zero-shot classification with a language model, but any classification strategy can be plugged in. +The default implementation in `@rafters/mail-workers-ai` uses DeBERTa-v3 zero-shot classification through Cloudflare Workers AI, but any classification strategy can be plugged in by implementing `EmailClassifier`. --- diff --git a/packages/cloudflare/docs/quickstart.md b/packages/cloudflare/docs/quickstart.md index ad99679..3a2e9f2 100644 --- a/packages/cloudflare/docs/quickstart.md +++ b/packages/cloudflare/docs/quickstart.md @@ -70,23 +70,30 @@ wrangler secret put RESEND_API_KEY ```typescript // src/index.ts import { createResendProvider } from "@rafters/mail-resend"; -import { createR2BlobStorage, parseInboundEmail } from "@rafters/mail-cloudflare"; +import { createR2Storage } from "@rafters/mail-cloudflare/storage"; +import { parseEmailHeaders, hashContent } from "@rafters/mail-cloudflare/parsing"; export default { // Handle inbound email from Cloudflare Email Routing - async email(message, env) { - const parsed = await parseInboundEmail(message); - const blobStorage = createR2BlobStorage(env.BLOB_STORAGE); + async email(message: ForwardableEmailMessage, env: Env) { + // Read the raw message bytes from the ReadableStream + const raw = await new Response(message.raw).arrayBuffer(); - // Store raw email in blob storage - const blobKey = await blobStorage.put(parsed.rawEmail); + // Parse RFC 5322 headers and hash the content for dedupe + const headers = parseEmailHeaders(Object.fromEntries(message.headers.entries())); + const contentHash = await hashContent(raw); - // Store message record in D1 + // Store the raw email in R2 via the BlobStorage adapter + const storage = createR2Storage({ bucket: env.BLOB_STORAGE }); + const blobKey = storage.generateKey(contentHash, "eml"); + await storage.put(blobKey, raw); + + // Insert message row in D1, update thread, dispatch to classifier queue // (wire up your service layer here) }, // Handle HTTP requests (webhooks, API) - async fetch(request, env) { + async fetch(request: Request, env: Env) { return new Response("Mail worker running"); }, }; @@ -151,7 +158,7 @@ Send an outbound email: const provider = createResendProvider({ apiKey: env.RESEND_API_KEY }); await provider.sendEmail({ from: "you@yourdomain.com", - to: ["recipient@example.com"], + to: "recipient@example.com", // single recipient per sendEmail call subject: "Hello from the edge", text: "Sent via @rafters/mail on Cloudflare Workers.", }); @@ -161,7 +168,10 @@ await provider.sendEmail({ ## 10. Add IMAP (optional) -To access your mailbox from Apple Mail, Thunderbird, or Outlook, add the IMAP server. See the [IMAP Quickstart](./imap-quickstart.md) for Cloudflare DO (WebSocket) or Node TCP (Fly.io) deployment options. +To access your mailbox from Apple Mail, Thunderbird, or Outlook, add the IMAP server. See the docs shipped with the IMAP runtime packages: + +- [`@rafters/mail-imap-server`](https://www.npmjs.com/package/@rafters/mail-imap-server) -- Node TCP for Fly.io / Railway / Fargate / VPS +- [`@rafters/mail-imap-cloudflare`](https://www.npmjs.com/package/@rafters/mail-imap-cloudflare) -- Durable Object + WebSocket for serverless --- @@ -178,8 +188,10 @@ To access your mailbox from Apple Mail, Thunderbird, or Outlook, add the IMAP se ## Next steps -- [Classification](./classification.md) -- auto-categorize incoming email -- [IMAP](./imap-quickstart.md) -- connect email clients -- [Newsletters](./newsletters.md) -- send to subscriber lists -- [App Passwords](./app-passwords.md) -- set up IMAP authentication -- [Deployment Guide](./imap-deployment.md) -- deploy IMAP on Fly, Railway, or Docker +Per-package docs ship with each npm package: + +- **Classification** -- see [`@rafters/mail-workers-ai`](https://www.npmjs.com/package/@rafters/mail-workers-ai) docs for auto-categorization +- **IMAP connect** -- see [`@rafters/mail-imap-server`](https://www.npmjs.com/package/@rafters/mail-imap-server) quickstart + deployment +- **IMAP auth** -- see [`@rafters/mail-imap`](https://www.npmjs.com/package/@rafters/mail-imap) `authentication.md` for the `AuthAdapter` contract +- **Newsletters** -- see [`@rafters/mail`](https://www.npmjs.com/package/@rafters/mail) `newsletters.md` for mailing lists, subscribers, campaigns +- **Inbound detail** -- see [`inbound.md`](./inbound.md) in this package diff --git a/packages/resend/docs/outbound.md b/packages/resend/docs/outbound.md index d5c6c25..0cfc411 100644 --- a/packages/resend/docs/outbound.md +++ b/packages/resend/docs/outbound.md @@ -6,76 +6,110 @@ How the system sends email. ## The email provider -Outbound email is sent through an `EmailProvider` adapter. The provider handles the HTTP API call to your email sending service. +Outbound email is sent through an `EmailProvider` adapter. The provider handles the HTTP API call to your email sending service. The interface lives in `@rafters/mail/interfaces` and covers transactional sends plus mailing list, subscriber, and campaign management. ```typescript interface EmailProvider { - sendEmail(params: SendEmailParams): Promise; + // Transactional + sendEmail(params: EmailParams): Promise<{ id: string }>; + + // Mailing lists + createMailingList(name: string): Promise; + getMailingList(id: string): Promise; + deleteMailingList(id: string): Promise; + + // Subscribers + addSubscriber(listId: string, email: string, data?: SubscriberData): Promise; + removeSubscriber(listId: string, subscriberId: string): Promise; + updateSubscriber(subscriberId: string, updates: SubscriberUpdates): Promise; + listSubscribers(listId: string): Promise; + + // Campaigns + sendCampaign(params: CampaignParams): Promise<{ id: string }>; + getCampaign(id: string): Promise<{ id: string; subject: string; sentAt: Date }>; + createCampaignDraft(params: CampaignParams): Promise<{ id: string }>; + sendCampaignDraft(campaignId: string): Promise<{ id: string }>; + getCampaignStatus(campaignId: string): Promise; + + // Audiences + listAudiences(): Promise; } ``` -Parameters: +### EmailParams (transactional sends) -| Field | Required | Purpose | -| ------- | -------- | --------------------------- | -| from | yes | Sender email address | -| to | yes | Recipient email address(es) | -| subject | yes | Email subject | -| html | no | HTML body | -| text | no | Plain text body | -| cc | no | CC recipients | -| bcc | no | BCC recipients | -| replyTo | no | Reply-To address | -| headers | no | Custom headers | +| Field | Type | Required | Purpose | +| --------- | -------------- | -------- | ----------------------------------------------------------------------- | +| `to` | string (email) | yes | Recipient email address (single recipient per send) | +| `subject` | string | yes | Email subject, minimum 1 character | +| `html` | string | no | HTML body | +| `text` | string | no | Plain text body | +| `from` | string (email) | no | Sender address. Falls back to the provider's default `from` if omitted. | +| `replyTo` | string (email) | no | Reply-To address | -At least one of `html` or `text` must be provided. +At least one of `html` or `text` should be provided. `sendEmail` returns `{ id }` where `id` is the sending provider's message identifier, used later for webhook correlation. + +Multi-recipient sends (CC, BCC, or multiple `to`) are handled outside `sendEmail`. Use the mailing list + campaign APIs for batched sends, or issue multiple `sendEmail` calls for individual delivery to several recipients. --- ## Composing and replying +Compose and reply go through the higher-level `InboxEmailService` (from `@rafters/mail/services`), which wraps the `EmailProvider` with thread management, blob storage, and RFC 5322 header generation. + +```typescript +interface InboxEmailService { + composeEmail(params: ComposeEmailParams): Promise<{ threadId: string; messageId: string }>; + replyToThread(params: ReplyToThreadParams): Promise<{ messageId: string }>; +} +``` + ### New message ```typescript -await mailService.compose({ +await inboxEmailService.composeEmail({ mailboxId: "mbx_01J...", senderId: "user_01J...", - to: ["recipient@example.com"], - subject: "Hello", - body: "Plain text content", - bodyHtml: "

HTML content

", + to: ["recipient@example.com"], // array, at least one + subject: "Hello", // max 500 chars + body: "Plain text content", // required, min 1 char + bodyHtml: "

HTML content

", // optional + cc: ["cc@example.com"], // optional + bcc: ["bcc@example.com"], // optional }); ``` The compose flow: -1. Generates a Message-ID using UUIDv7 -2. Creates a new thread (or finds existing by subject) -3. Sends via the EmailProvider -4. Stores the sent message in the database with `isOutbound: true` -5. Stores content in blob storage +1. Generates a `Message-ID` using UUIDv7 +2. Creates a new thread for the subject +3. Sends via the `EmailProvider` +4. Stores the sent message row in the database with `isOutbound: true` +5. Stores the raw email and parsed body content in blob storage ### Reply ```typescript -await mailService.replyToThread({ +await inboxEmailService.replyToThread({ threadId: "thread_01J...", mailboxId: "mbx_01J...", senderId: "user_01J...", body: "Reply text", bodyHtml: "

Reply HTML

", + cc: ["cc@example.com"], + bcc: ["bcc@example.com"], }); ``` The reply flow: 1. Loads the thread and its most recent message -2. Builds the References header from the thread's message chain -3. Sets In-Reply-To to the most recent message's Message-ID -4. Generates a new Message-ID -5. Sends via the EmailProvider +2. Builds the `References` header from the thread's message chain +3. Sets `In-Reply-To` to the most recent message's `Message-ID` +4. Generates a new `Message-ID` +5. Sends via the `EmailProvider` 6. Stores the reply in the same thread with `isOutbound: true` -7. Updates thread metadata (messageCount, lastMessageAt, snippet) +7. Updates thread metadata (`unreadCount`, `lastMessageAt`, snippet) --- diff --git a/packages/workers-ai/docs/classification.md b/packages/workers-ai/docs/classification.md index 46e1a5a..c195de0 100644 --- a/packages/workers-ai/docs/classification.md +++ b/packages/workers-ai/docs/classification.md @@ -4,28 +4,23 @@ Automatic categorization of incoming email using AI or rule-based classifiers. --- -## The classification adapter +## The classifier interface ```typescript -interface ClassificationAdapter { - classify(message: ClassificationInput): Promise; +interface EmailClassifier { + classify(from: string, subject: string, body: string): Promise; } -interface ClassificationInput { - subject: string; - from: string; - textBody: string; - htmlBody?: string; -} - -interface ClassificationResult { - category: AiCategory; - confidence: number; // 0-100 - summary: string; // one-line description +interface EmailClassification { + category: AiCategory; // one of: support, feedback, abuse, partnership, + // spam, billing, legal, other + confidence: number; // 0..100 + tags: string[]; // pattern-matched tags (configurable) + priority: ThreadPriority; // urgent, high, normal, low } ``` -The adapter is called after a message is stored. Classification results are written to the message record. +The classifier is called after a message is stored. The result is written to the message record's `aiCategory`, `aiConfidence`, and related fields, and to the thread's `priority`. Tags are applied as labels via `inboxMessageLabel`. --- @@ -92,20 +87,21 @@ Spam detection can combine multiple signals: the AI classifier's spam category, The simplest classifier uses zero-shot classification with a language model: ```typescript -const classifier: ClassificationAdapter = { - async classify(input) { +const classifier: EmailClassifier = { + async classify(from, subject, body) { const result = await model.run({ - text: `${input.subject}\n\n${input.textBody}`, + text: `${subject}\n\n${body}`, labels: ["support", "feedback", "billing", "partnership", "abuse", "legal", "spam", "other"], }); return { category: result.label as AiCategory, confidence: Math.round(result.score * 100), - summary: await model.summarize(input.textBody, { maxLength: 100 }), + tags: [], // fill in via pattern matching if desired + priority: "normal", // derive from category or subject signals }; }, }; ``` -You can also implement rule-based classification (regex patterns on subject/sender), or a hybrid that uses rules first and falls back to AI for uncertain cases. +You can also implement rule-based classification (regex patterns on subject/sender), or a hybrid that uses rules first and falls back to AI for uncertain cases. The shipped `@rafters/mail-workers-ai` classifier uses this hybrid approach with configurable `tagPatterns`. From 09a9af76b79b58e257b1709aa1908045806d746c Mon Sep 17 00:00:00 2001 From: Sean Silvius Date: Sat, 11 Apr 2026 12:06:04 -0700 Subject: [PATCH 2/2] docs(core): fix accuracy drift in threading, migrations, newsletters Three more core docs had significant accuracy drift that would have misled consumers reading them on npm. These were NOT covered by the prior @vault-2026 schema review (which focused on core-reference.md). ## packages/core/docs/threading.md - The doc described a threading engine that matches messages to threads using In-Reply-To / References / subject fallback. That engine does not exist in the core threading module -- threading.ts only exports generateMessageId, buildReferences, and generateSnippet as header-building blocks. Thread matching is the inbound adapter's responsibility and has a TODO in threading.ts pointing to #13. - The doc claimed "subject matching within the same mailbox (configurable)". Grep for it -- no such feature exists. Subject matching is intentionally NOT part of the design because subject lines collide across unrelated conversations. - The doc claimed "Deleting a thread soft-deletes all its messages". Checked the schema: inbox_message.threadId has ON DELETE CASCADE, which is a hard-delete cascade, not soft-delete. Soft-deleting a thread (setting deletedAt) does NOT cascade to messages. Clarified. - The doc's thread model table was missing columns that exist in the schema: startedAt, updatedAt, archivedAt. Added. ## packages/core/docs/migrations.md - The doc said "13 tables across two domains" and listed 3 newsletter tables as if they were part of migrationSQL. Actually migrationSQL only contains 10 inbox tables. The 3 newsletter tables (platform_audience, platform_subscriber, broadcast_audit) are Drizzle schema exports but are NOT in migrationSQL and are not written to by any shipped service. - The newsletter table names were wrong: doc said mailing_list, subscriber, campaign. Actual names are platform_audience, platform_subscriber, broadcast_audit. - The doc said system folders are "initialized automatically" when a mailbox is created. They are not. The consumer has to explicitly call FolderService.initSystemFolders(mailboxId). - The "Conventions" section said "every table has deletedAt". Two label join tables (inbox_message_label, inbox_thread_label) do not have deletedAt because their delete semantic is direct row removal. ## packages/core/docs/newsletters.md - The entire schema section described a completely different design than what exists in newsletter.ts. The doc had tables mailing_list, subscriber, campaign with fields like status, metadata, scheduledAt, htmlBody -- none of which exist. Actual tables are platform_audience (providerListId, name, description, slug), platform_subscriber (userId, audienceId, providerSubscriberId, subscribedAt), and broadcast_audit (providerCampaignId, subject, contentHash, sentBy, audienceName, recipientCount, sentAt). - The doc described a "status lifecycle" (subscribed -> unsubscribed / bounced / complained) that does not exist because there is no status column on platform_subscriber. - The doc described a campaign lifecycle (draft -> scheduled -> sending -> sent) that does not exist because there is no status column on broadcast_audit (which is an audit log, not a campaign record). - Rewrote to accurately describe: (1) the EmailProvider surface as the authoritative API for mailing lists / subscribers / campaigns, (2) the 3 platform tables as OPTIONAL local mirror / audit tables the consumer may include in their own migrations, (3) that the provider (Resend) owns the authoritative data and the framework does not duplicate it locally by default. ## Verification - pnpm test: 609 passing - pnpm typecheck: clean - pnpm lint: 0 warnings, 0 errors - pnpm format:check: clean on modified files (.claude/settings.json is untracked session state, not part of this PR) Co-Authored-By: Claude Opus 4.6 (1M context) --- packages/core/docs/migrations.md | 80 ++++++++------- packages/core/docs/newsletters.md | 163 +++++++++++++++--------------- packages/core/docs/threading.md | 84 ++++++++++----- 3 files changed, 178 insertions(+), 149 deletions(-) diff --git a/packages/core/docs/migrations.md b/packages/core/docs/migrations.md index 18e36c9..e96d67a 100644 --- a/packages/core/docs/migrations.md +++ b/packages/core/docs/migrations.md @@ -14,47 +14,49 @@ This separation means the framework works with any SQLite-compatible database wi ## Schema -The core schema has 13 tables across two domains: - -### Inbox tables (10) - -| Table | Purpose | -| ------------------- | --------------------------------------------------------------- | -| mailbox | Email addresses that send and receive | -| inbox_folder | Folders per mailbox (inbox, sent, drafts, spam, trash, archive) | -| inbox_label | Labels for categorization (system and custom) | -| inbox_thread | Conversation threads | -| inbox_message | Individual email messages | -| inbox_message_label | Message-to-label associations | -| inbox_thread_label | Thread-to-label associations | -| inbox_attachment | File attachments | -| thread_assignment | Team member assignments to threads | -| thread_note | Internal notes on threads | - -### Newsletter tables (3) - -| Table | Purpose | -| ------------ | -------------------------------- | -| mailing_list | Named subscriber lists | -| subscriber | People on mailing lists | -| campaign | Broadcast messages sent to lists | +The core schema currently ships **10 inbox tables** in the exported migration SQL. An additional 3 newsletter tables are defined in the Drizzle schema but are not yet part of the migration SQL and are not written to by any shipped service -- they are reserved for future platform-side broadcast tracking and can be ignored at install time. + +### Inbox tables (10) -- in `migrationSQL` + +| Table | Purpose | +| --------------------- | --------------------------------------------------------------- | +| `mailbox` | Email addresses that send and receive | +| `inbox_folder` | Folders per mailbox (inbox, sent, drafts, spam, trash, archive) | +| `inbox_label` | Labels for categorization (system and custom) | +| `inbox_thread` | Conversation threads | +| `inbox_message` | Individual email messages | +| `inbox_message_label` | Message-to-label associations | +| `inbox_thread_label` | Thread-to-label associations | +| `inbox_attachment` | File attachments | +| `thread_assignment` | Team member assignments to threads | +| `thread_note` | Internal notes on threads | + +### Newsletter tables (3) -- schema-only, NOT in `migrationSQL` + +| Table | Purpose | +| --------------------- | --------------------------------------------------- | +| `platform_audience` | Platform-side mirror of the provider's mailing list | +| `platform_subscriber` | Platform-side mirror of a subscriber on an audience | +| `broadcast_audit` | Audit trail for campaign sends | + +These are available as Drizzle schema exports (`platformAudience`, `platformSubscriber`, `broadcastAudit` from `@rafters/mail/schema`) if you want to include them in your own migrations, but the shipped `migrationSQL` string does not create them. The `EmailProvider` mailing list / subscriber / campaign methods talk to the provider's API (Resend, etc.), not to these tables. --- ## System folders -When a mailbox is created, these folders are initialized automatically: +System folders are not created automatically at migration time. Apply the migration, create a mailbox, then call `FolderService.initSystemFolders(mailboxId)` to populate the following folders for that mailbox: -| Folder | Slug | Purpose | -| ------- | ------- | ---------------------------------- | -| Inbox | inbox | Default landing for incoming email | -| Sent | sent | Outbound messages | -| Drafts | drafts | Unsent compositions | -| Spam | spam | Spam-classified messages | -| Trash | trash | Soft-deleted messages | -| Archive | archive | Archived threads | +| Folder | Slug | Purpose | +| ------- | --------- | ---------------------------------- | +| Inbox | `inbox` | Default landing for incoming email | +| Sent | `sent` | Outbound messages | +| Drafts | `drafts` | Unsent compositions | +| Spam | `spam` | Spam-classified messages | +| Trash | `trash` | Soft-deleted messages | +| Archive | `archive` | Archived threads | -System folders cannot be deleted. Custom folders can be created by the user. +System folders have `isSystem = true` and cannot be deleted via `FolderService.delete`. Custom folders can be created with `FolderService.create`. --- @@ -62,11 +64,11 @@ System folders cannot be deleted. Custom folders can be created by the user. All tables follow these conventions: -- **IDs**: UUIDv7 via default function. Text primary keys. Timestamp-ordered. -- **Timestamps**: integer milliseconds. -- **Soft delete**: every table has deletedAt. Null means active. -- **JSON columns**: SQLite text with JSON mode. Parsed at read time. -- **User references**: plain text columns (ownerId, assigneeId, etc.). No foreign keys to external auth tables. The auth adapter resolves identity at runtime. +- **IDs**: UUIDv7 via default function. Text primary keys. Timestamp-ordered, so sort-by-id is chronological. +- **Timestamps**: `integer` with `mode: 'timestamp_ms'`, default `unixepoch('subsecond') * 1000`. Millisecond precision. +- **Soft delete**: all tables have `deletedAt` except the two label join tables (`inbox_message_label`, `inbox_thread_label`), where deletion is a direct row removal because there is nothing semantic to "soft delete" -- a label is either applied or not. +- **JSON columns**: SQLite `text` with `mode: 'json'`. Stored as serialized JSON strings, parsed at read time. +- **User references**: plain `text` columns (`ownerId`, `assigneeId`, `authorId`, `appliedBy`, etc.). No foreign keys to external auth tables. The `AuthAdapter` resolves identity at runtime. --- diff --git a/packages/core/docs/newsletters.md b/packages/core/docs/newsletters.md index c201b9c..3e95759 100644 --- a/packages/core/docs/newsletters.md +++ b/packages/core/docs/newsletters.md @@ -1,118 +1,113 @@ # Newsletters and Broadcasts -Sending email to audiences. Mailing lists, subscribers, campaigns. +Sending email to many recipients via mailing lists, subscribers, and campaigns. --- ## Vocabulary -@rafters/mail uses specific terms: +`@rafters/mail` uses generic platform terms in its interface surface: -| Term | Not this | Meaning | -| ------------ | --------- | -------------------------------- | -| Mailing list | Audience | A named list of subscribers | -| Subscriber | Contact | Someone on a mailing list | -| Campaign | Broadcast | A message sent to a mailing list | +| Interface name | Vendor synonyms | Meaning | +| -------------- | ----------------------------------- | --------------------------------- | +| `MailingList` | Audience (Resend), List (Mailchimp) | A named collection of subscribers | +| `Subscriber` | Contact, Audience Member | Someone on a mailing list | +| `Campaign` | Broadcast | A message sent to a mailing list | -Vendor terms (audience, contact, broadcast) appear only inside adapter implementations. +Vendor terms (`audience`, `broadcast`) appear inside adapter implementations and in the platform-side mirror tables, but they do not leak into the interface surface consumers call. --- -## Mailing lists +## The EmailProvider surface -A mailing list is a collection of subscribers who opted in to receive email. +All mailing list, subscriber, and campaign operations go through the `EmailProvider` interface. The provider is the authoritative source of truth for list membership and delivery. `@rafters/mail` does not duplicate that data locally unless the consumer opts into platform-side tracking. -| Field | Purpose | -| ----------- | ---------------------------------- | -| name | Display name ("Weekly Newsletter") | -| description | What subscribers signed up for | -| mailboxId | Which mailbox sends campaigns | - -Lists are per-mailbox. The support mailbox and the marketing mailbox have separate lists. - ---- - -## Subscribers - -| Field | Purpose | -| -------------- | --------------------------------------------- | -| email | Subscriber email address | -| name | Display name (optional) | -| status | subscribed, unsubscribed, bounced, complained | -| subscribedAt | When they opted in | -| unsubscribedAt | When they opted out | -| metadata | Custom fields (JSON) | +```typescript +interface EmailProvider { + // Mailing lists + createMailingList(name: string): Promise; + getMailingList(id: string): Promise; + deleteMailingList(id: string): Promise; + + // Subscribers + addSubscriber(listId: string, email: string, data?: SubscriberData): Promise; + removeSubscriber(listId: string, subscriberId: string): Promise; + updateSubscriber(subscriberId: string, updates: SubscriberUpdates): Promise; + listSubscribers(listId: string): Promise; + + // Campaigns + sendCampaign(params: CampaignParams): Promise<{ id: string }>; + getCampaign(id: string): Promise<{ id: string; subject: string; sentAt: Date }>; + createCampaignDraft(params: CampaignParams): Promise<{ id: string }>; + sendCampaignDraft(campaignId: string): Promise<{ id: string }>; + getCampaignStatus(campaignId: string): Promise; + + // Audiences (read-only list of available mailing lists) + listAudiences(): Promise; +} +``` -### Status lifecycle +### MailingList, Subscriber, Campaign shapes -``` -subscribed -> unsubscribed (user opts out) -subscribed -> bounced (delivery failure) -subscribed -> complained (marked as spam) +```typescript +interface MailingList { + id: string; + name: string; + createdAt: Date; +} + +interface Subscriber { + id: string; + email: string; + firstName?: string; + lastName?: string; + unsubscribed: boolean; +} + +interface CampaignParams { + listId: string; + subject: string; // 1..200 chars + html: string; // min 1 char + text?: string; + from: string; // sender email + replyTo?: string; + name?: string; +} + +interface CampaignStatus { + id: string; + status: "draft" | "queued" | "sending" | "sent" | "cancelled"; + subject: string; + sentAt: Date | null; +} ``` -Once a subscriber is `unsubscribed`, `bounced`, or `complained`, they do not receive future campaigns. Re-subscription requires explicit opt-in. +`sendCampaign` publishes a campaign immediately. `createCampaignDraft` + `sendCampaignDraft` is the two-step flow for drafts that might be edited before send. Polling `getCampaignStatus` is how you observe the send through `queued -> sending -> sent`. --- -## Campaigns - -A campaign is a message sent to all active subscribers on a mailing list. +## Platform-side mirror tables (optional) -| Field | Purpose | -| ------------- | ------------------------------------------------ | -| mailingListId | Target list | -| subject | Email subject | -| htmlBody | HTML content (rendered from template) | -| textBody | Plain text fallback | -| status | draft, scheduled, sending, sent, failed | -| scheduledAt | When to send (optional, for scheduled campaigns) | -| sentAt | When sending completed | +The `@rafters/mail` schema ships three newsletter-related tables in `packages/core/src/schema/newsletter.ts`: -### Campaign lifecycle +| Table | Purpose | +| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `platform_audience` | Local mirror of a mailing list, keyed by `providerListId`. Stores `name`, `description`, and a URL-safe `slug`. Use when you want to show audiences in your own UI without re-fetching from the provider. | +| `platform_subscriber` | Association between one of your app users (`userId`) and one audience (`audienceId`), plus the `providerSubscriberId` for provider correlation. Use when you need to answer "which of my users is on which list" without round-tripping the provider. | +| `broadcast_audit` | Audit log of campaign sends: `providerCampaignId`, `subject`, `contentHash`, `sentBy` (user id), `audienceName`, `recipientCount`, `sentAt`. Use for compliance, "who sent what to whom" queries, and content-hash deduplication across sends. | -``` -draft -> scheduled -> sending -> sent -draft -> sending -> sent -draft -> sending -> failed -``` - -### Sending - -Campaigns are sent individually to each subscriber (not BCC). Each send is a separate API call to the email provider. This allows per-recipient tracking (delivery, opens, clicks) and personalization. - -Rate limiting is handled by the email provider adapter. The campaign runner respects the provider's concurrency limits. +**These tables are not in the exported `migrationSQL` string.** They exist as Drizzle schema definitions you can include in your own migrations if you want platform-side tracking. If you only use the `EmailProvider` interface and trust the provider's own audience/subscriber storage, you do not need these tables at all. --- ## Templates -Campaign content is typically rendered from a template: +Campaign HTML is usually rendered from a React Email template via the `TemplateRenderer` interface. See the `@rafters/mail-react-email` package for the shipped renderer and baseline templates, and its `docs/templates.md` for writing your own. -```typescript -const html = await renderer.render(NewsletterEmail, { - title: "Weekly Update", - content: markdownContent, - unsubscribeUrl: `https://yourdomain.com/unsubscribe?id=${subscriber.id}`, -}); -``` - -Every campaign email must include an unsubscribe link. The template adapter handles this requirement. +Every campaign email should include an unsubscribe link as required by CAN-SPAM and similar regulations. The provider handles the actual unsubscribe flow when a recipient clicks the link; the template is responsible for emitting a valid URL that the provider routes back to the subscriber record. --- -## Analytics - -Campaign analytics are derived from webhook events: - -| Metric | Source | -| ------------ | ------------------------------------------- | -| Sent | Total send attempts | -| Delivered | Delivery confirmations from provider | -| Bounced | Permanent delivery failures | -| Opened | Open tracking events (if tracking enabled) | -| Clicked | Click tracking events (if tracking enabled) | -| Unsubscribed | Unsubscribe actions from this campaign | -| Complained | Spam complaints from this campaign | +## Delivery and analytics -Analytics are per-campaign. Aggregate metrics (subscriber growth, engagement rates) are computed from campaign history. +Delivery events (delivered, bounced, complained, opened, clicked) come from the provider's webhook. `@rafters/mail-resend` ships a webhook handler at `@rafters/mail-resend/webhooks` that validates the signature and invokes user-provided callbacks for each event type. Correlate events back to campaigns via the `providerCampaignId` stored in `broadcast_audit`, or query the provider directly with `getCampaignStatus`. diff --git a/packages/core/docs/threading.md b/packages/core/docs/threading.md index 1ff65cc..fe2cd7b 100644 --- a/packages/core/docs/threading.md +++ b/packages/core/docs/threading.md @@ -1,41 +1,71 @@ # Threading -How @rafters/mail builds and maintains email threads. +How `@rafters/mail` builds and maintains email threads. --- -## How threads work +## RFC 5322 headers -Every email carries headers that identify its place in a conversation: +Every email carries three headers that identify its place in a conversation: -- **Message-ID**: a globally unique identifier assigned when the email is sent -- **In-Reply-To**: the Message-ID of the email being replied to -- **References**: the full chain of Message-IDs in the conversation, oldest to newest +- **`Message-ID`**: a globally unique identifier assigned when the email is sent +- **`In-Reply-To`**: the `Message-ID` of the email being replied to +- **`References`**: the full chain of `Message-ID` values in the conversation, oldest to newest -When a new message arrives, the threading engine uses these headers to find or create a thread: +`@rafters/mail` ships three building blocks for working with these headers from the `@rafters/mail/threading` subpath export: -1. Check `In-Reply-To` -- if it matches an existing message, join that thread -2. Check `References` -- walk the chain looking for any known message -3. Fall back to subject matching within the same mailbox (configurable) -4. If no match, create a new thread +```typescript +import { generateMessageId, buildReferences, generateSnippet } from "@rafters/mail/threading"; + +const id = generateMessageId("yourdomain.com"); +// -> "<019d7d...@yourdomain.com>" + +const refs = buildReferences(" ", ""); +// -> " " (already present, not re-appended) + +const snippet = generateSnippet("Hello world\n\nLong body text...", 200); +// -> "Hello world Long body text..." (first 200 chars) +``` + +`buildReferences` caps the chain at 50 entries to prevent unbounded growth; the most recent 50 are kept and the oldest are dropped. This matches the RFC 5322 recommendation for trimming long reference chains. --- -## Thread model +## Thread matching is the inbound adapter's job + +The `@rafters/mail/threading` module does not match messages to threads. Matching is the responsibility of the code that ingests inbound email -- typically an implementation of `InboundAdapter` -- because it requires database queries that the threading module cannot do on its own. + +The expected matching strategy, in priority order: -A thread is a container for related messages: +1. **`In-Reply-To`** -- if the incoming header matches a `messageIdHeader` on an existing `inbox_message` row, join that message's thread. +2. **`References`** -- walk the chain; any hit on an existing `messageIdHeader` joins that thread. +3. **Fresh thread** -- if no header match, create a new thread. -| Field | Purpose | -| ------------- | ------------------------------------ | -| subject | The conversation subject | -| snippet | Preview of the most recent message | -| participants | All email addresses involved | -| messageCount | Total messages in the thread | -| unreadCount | Messages not yet marked as read | -| status | open, pending, resolved, closed | -| priority | low, normal, high, urgent | -| folderId | Current folder (nullable) | -| lastMessageAt | Timestamp of the most recent message | +Subject-based matching is intentionally NOT part of this flow. Subject lines collide across unrelated conversations; header-based matching is the only reliable path. + +The shipped `InboxEmailService.composeEmail` always creates a fresh thread. Replies go through `replyToThread`, which takes an explicit `threadId` -- the caller is expected to know the target thread. Inbound matching logic is not yet shipped in a runtime adapter. + +--- + +## Thread model + +A thread is a container for related messages in `inbox_thread`: + +| Field | Type | Purpose | +| --------------- | -------------------------- | --------------------------------------- | +| `subject` | text (NOT NULL) | The conversation subject | +| `snippet` | text (nullable) | Preview of the most recent message | +| `participants` | json string[] (nullable) | All email addresses involved | +| `messageCount` | integer (default 1) | Total messages in the thread | +| `unreadCount` | integer (default 1) | Messages not yet marked as read | +| `status` | text (default "open") | `open`, `pending`, `resolved`, `closed` | +| `priority` | text (default "normal") | `low`, `normal`, `high`, `urgent` | +| `folderId` | text (nullable) | Current folder. `ON DELETE SET NULL`. | +| `startedAt` | timestamp (NOT NULL) | When the thread was created | +| `lastMessageAt` | timestamp (NOT NULL) | Timestamp of the most recent message | +| `updatedAt` | timestamp (NOT NULL, auto) | Last time any thread field changed | +| `archivedAt` | timestamp (nullable) | When archived, if applicable | +| `deletedAt` | timestamp (nullable) | Soft-delete marker | Threads track their own read state via `unreadCount` rather than a single boolean. A thread with 10 messages where 3 are unread shows `unreadCount: 3`. @@ -70,9 +100,11 @@ UUIDv7 is timestamp-ordered, which means Message-IDs naturally sort by creation ## Thread assignment -Messages are assigned to threads at ingest time. The assignment is permanent -- a message belongs to one thread for its lifetime. If the threading engine cannot find a match, it creates a new single-message thread. +Messages are assigned to threads at ingest time. The assignment is permanent -- a message belongs to one thread for its lifetime. If the inbound adapter cannot find a match via `In-Reply-To` / `References`, it creates a new single-message thread. + +Moving a thread between folders is a logical operation that updates `inbox_thread.folderId`; messages stay in place. -Moving a thread between folders moves all its messages. Deleting a thread soft-deletes all its messages. +**On deletion:** `inbox_message.threadId` has `ON DELETE CASCADE`, so **hard-deleting** a thread row also hard-deletes all its messages. **Soft-deleting** a thread (setting `deletedAt`) does NOT cascade to messages -- soft deletes are manual per-row. If you want to hide a thread and all its messages together, either hard-delete (irreversible) or soft-delete the thread AND each message explicitly. ---