feat: implement image upload in chat input and image handling#501
Open
feat: implement image upload in chat input and image handling#501
Conversation
- Add message_image table to both SQLite and PostgreSQL schemas - Add mediaType and imageId columns to message_part table - Create image queries for saving and retrieving images - Create image serving route at /i/:imageId - Update agent handler to process image uploads - Update chat-message-part-mappings for file type parts - Resolve image URLs to data URLs for model messages in agent service - Extend AgentRequestUserMessageSchema with optional images array Co-authored-by: Christophe Blefari <christophe.blefari@gmail.com>
- Create useImageUpload hook with file picker, paste, drag-and-drop support - Create ChatInputImagePreview component for thumbnail previews - Integrate image upload in ChatInputBase with drag zone and file input - Add 'Upload image' option to the plus menu - Update SendMessageArgs and queue types to support images - Update use-agent to pass images in prepareSendMessagesRequest - Display images in UserMessageBubble - Add getMessageImages helper to lib/ai Co-authored-by: Christophe Blefari <christophe.blefari@gmail.com>
Co-authored-by: Christophe Blefari <christophe.blefari@gmail.com>
Co-authored-by: Christophe Blefari <christophe.blefari@gmail.com>
Contributor
🚀 Preview Deployment
Preview will be automatically removed when this PR is closed. |
…ately
- Pass images as FileUIPart[] via sendMessage({text, files}) instead of a separate ref
- Extract images from message file parts in prepareSendMessagesRequest
- Add extractImagesFromMessage helper to lib/ai
- Images now appear in the chat bubble immediately after sending
Co-authored-by: Christophe Blefari <christophe.blefari@gmail.com>
Image and chart routes need to be proxied to the backend server during development, otherwise the frontend can't access them. Co-authored-by: Christophe Blefari <christophe.blefari@gmail.com>
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
9 issues found across 20 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="apps/frontend/src/hooks/use-image-upload.ts">
<violation number="1" location="apps/frontend/src/hooks/use-image-upload.ts:51">
P2: Limit the loop to only files that can actually be kept; currently extra files are still decoded and then dropped by `.slice(0, MAX_IMAGES)`.</violation>
</file>
<file name="apps/backend/src/handlers/agent.ts">
<violation number="1" location="apps/backend/src/handlers/agent.ts:33">
P1: Images are saved before access control checks, allowing unauthorized requests to persist image data even when chat access is forbidden.</violation>
</file>
<file name="apps/backend/src/db/sqlite-schema.ts">
<violation number="1" location="apps/backend/src/db/sqlite-schema.ts:279">
P2: This schema can orphan uploaded image blobs after message deletion, causing unbounded storage growth.</violation>
</file>
<file name="apps/backend/src/routes/image.ts">
<violation number="1" location="apps/backend/src/routes/image.ts:22">
P1: Do not trust stored `mediaType` when setting `Content-Type`; restrict to `image/*` (or force a safe fallback) to prevent serving active content from `/i/:imageId`.</violation>
</file>
<file name="apps/backend/src/utils/chat-message-part-mappings.ts">
<violation number="1" location="apps/backend/src/utils/chat-message-part-mappings.ts:60">
P2: Avoid returning `file` UI parts with empty URLs. If `imageId` is missing, skip the part instead of emitting `url: ''`.</violation>
</file>
<file name="apps/frontend/src/components/chat-input.tsx">
<violation number="1" location="apps/frontend/src/components/chat-input.tsx:148">
P1: Document-level paste handling intercepts image paste outside this chat input and can trigger multiple handlers when inline chat inputs are mounted. Scope paste handling to this component's DOM subtree.</violation>
<violation number="2" location="apps/frontend/src/components/chat-input.tsx:177">
P2: The hardcoded fallback text is applied in a shared input component, so image-only submits in inline edit mode can overwrite message text with `Describe this image`.</violation>
</file>
<file name="apps/backend/src/types/chat.ts">
<violation number="1" location="apps/backend/src/types/chat.ts:119">
P1: Validate `mediaType` and `data` in `AgentRequestImageSchema`; accepting any string allows non-image content to be persisted and served with attacker-controlled `Content-Type`.</violation>
</file>
<file name="apps/backend/src/db/pg-schema.ts">
<violation number="1" location="apps/backend/src/db/pg-schema.ts:263">
P2: `message_part` lacks a constraint requiring `mediaType` and `imageId` when `type = 'file'`, allowing invalid file parts to be stored and later rendered with missing image data.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
| ); | ||
| } | ||
|
|
||
| const imageParts = await saveAndBuildImageParts(message.images); |
Contributor
There was a problem hiding this comment.
P1: Images are saved before access control checks, allowing unauthorized requests to persist image data even when chat access is forbidden.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At apps/backend/src/handlers/agent.ts, line 33:
<comment>Images are saved before access control checks, allowing unauthorized requests to persist image data even when chat access is forbidden.</comment>
<file context>
@@ -28,16 +30,24 @@ export const handleAgentRoute = async (opts: HandleAgentMessageInput): Promise<H
);
}
+ const imageParts = await saveAndBuildImageParts(message.images);
+
let chatId = opts.chatId;
</file context>
|
|
||
| const buffer = Buffer.from(image.data, 'base64'); | ||
| return reply | ||
| .header('Content-Type', image.mediaType) |
Contributor
There was a problem hiding this comment.
P1: Do not trust stored mediaType when setting Content-Type; restrict to image/* (or force a safe fallback) to prevent serving active content from /i/:imageId.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At apps/backend/src/routes/image.ts, line 22:
<comment>Do not trust stored `mediaType` when setting `Content-Type`; restrict to `image/*` (or force a safe fallback) to prevent serving active content from `/i/:imageId`.</comment>
<file context>
@@ -0,0 +1,26 @@
+
+ const buffer = Buffer.from(image.data, 'base64');
+ return reply
+ .header('Content-Type', image.mediaType)
+ .header('Cache-Control', 'public, max-age=31536000, immutable')
+ .send(buffer);
</file context>
| }, [imageUpload.addFiles]); // eslint-disable-line | ||
|
|
||
| useEffect(() => { | ||
| const handler = (e: ClipboardEvent) => imageUpload.handlePaste(e); |
Contributor
There was a problem hiding this comment.
P1: Document-level paste handling intercepts image paste outside this chat input and can trigger multiple handlers when inline chat inputs are mounted. Scope paste handling to this component's DOM subtree.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At apps/frontend/src/components/chat-input.tsx, line 148:
<comment>Document-level paste handling intercepts image paste outside this chat input and can trigger multiple handlers when inline chat inputs are mounted. Scope paste handling to this component's DOM subtree.</comment>
<file context>
@@ -87,9 +90,66 @@ function ChatInputBase({
+ }, [imageUpload.addFiles]); // eslint-disable-line
+
+ useEffect(() => {
+ const handler = (e: ClipboardEvent) => imageUpload.handlePaste(e);
+ document.addEventListener('paste', handler);
+ return () => document.removeEventListener('paste', handler);
</file context>
Suggested change
| const handler = (e: ClipboardEvent) => imageUpload.handlePaste(e); | |
| const handler = (e: ClipboardEvent) => { | |
| if (dropZoneRef.current?.contains(e.target as Node)) { | |
| imageUpload.handlePaste(e); | |
| } | |
| }; |
Comment on lines
+119
to
+120
| mediaType: z.string(), | ||
| data: z.string(), |
Contributor
There was a problem hiding this comment.
P1: Validate mediaType and data in AgentRequestImageSchema; accepting any string allows non-image content to be persisted and served with attacker-controlled Content-Type.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At apps/backend/src/types/chat.ts, line 119:
<comment>Validate `mediaType` and `data` in `AgentRequestImageSchema`; accepting any string allows non-image content to be persisted and served with attacker-controlled `Content-Type`.</comment>
<file context>
@@ -115,9 +115,16 @@ export const MentionSchema = z.object({
});
+export const AgentRequestImageSchema = z.object({
+ mediaType: z.string(),
+ data: z.string(),
+});
</file context>
Suggested change
| mediaType: z.string(), | |
| data: z.string(), | |
| mediaType: z.enum(['image/png', 'image/jpeg', 'image/gif', 'image/webp']), | |
| data: z.string().min(1).regex(/^[A-Za-z0-9+/]+={0,2}$/), |
| } | ||
|
|
||
| const newImages: UploadedImage[] = []; | ||
| for (const file of fileArray) { |
Contributor
There was a problem hiding this comment.
P2: Limit the loop to only files that can actually be kept; currently extra files are still decoded and then dropped by .slice(0, MAX_IMAGES).
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At apps/frontend/src/hooks/use-image-upload.ts, line 51:
<comment>Limit the loop to only files that can actually be kept; currently extra files are still decoded and then dropped by `.slice(0, MAX_IMAGES)`.</comment>
<file context>
@@ -0,0 +1,131 @@
+ }
+
+ const newImages: UploadedImage[] = [];
+ for (const file of fileArray) {
+ const dataUrl = await readFileAsDataUrl(file);
+ newImages.push({
</file context>
|
|
||
| // file/image columns | ||
| mediaType: text('media_type'), | ||
| imageId: text('image_id').references(() => messageImage.id, { onDelete: 'set null' }), |
Contributor
There was a problem hiding this comment.
P2: This schema can orphan uploaded image blobs after message deletion, causing unbounded storage growth.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At apps/backend/src/db/sqlite-schema.ts, line 279:
<comment>This schema can orphan uploaded image blobs after message deletion, causing unbounded storage growth.</comment>
<file context>
@@ -273,6 +273,10 @@ export const messagePart = sqliteTable(
+
+ // file/image columns
+ mediaType: text('media_type'),
+ imageId: text('image_id').references(() => messageImage.id, { onDelete: 'set null' }),
},
(t) => [
</file context>
| reasoningText: part.text, | ||
| providerMetadata: part.providerMetadata, | ||
| }; | ||
| case 'file': |
Contributor
There was a problem hiding this comment.
P2: Avoid returning file UI parts with empty URLs. If imageId is missing, skip the part instead of emitting url: ''.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At apps/backend/src/utils/chat-message-part-mappings.ts, line 60:
<comment>Avoid returning `file` UI parts with empty URLs. If `imageId` is missing, skip the part instead of emitting `url: ''`.</comment>
<file context>
@@ -56,6 +57,14 @@ export const convertUIPartToDBPart = (
reasoningText: part.text,
providerMetadata: part.providerMetadata,
};
+ case 'file':
+ return {
+ messageId,
</file context>
| imageUpload.clearImages(); | ||
|
|
||
| await onSubmitMessage({ | ||
| text: trimmedInput || (images.length > 0 ? 'Describe this image' : ''), |
Contributor
There was a problem hiding this comment.
P2: The hardcoded fallback text is applied in a shared input component, so image-only submits in inline edit mode can overwrite message text with Describe this image.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At apps/frontend/src/components/chat-input.tsx, line 177:
<comment>The hardcoded fallback text is applied in a shared input component, so image-only submits in inline edit mode can overwrite message text with `Describe this image`.</comment>
<file context>
@@ -99,17 +159,26 @@ function ChatInputBase({
+ imageUpload.clearImages();
+
+ await onSubmitMessage({
+ text: trimmedInput || (images.length > 0 ? 'Describe this image' : ''),
+ images: images.length > 0 ? images : undefined,
+ });
</file context>
| providerMetadata: jsonb('provider_metadata').$type<ProviderMetadata>(), | ||
|
|
||
| // file/image columns | ||
| mediaType: text('media_type'), |
Contributor
There was a problem hiding this comment.
P2: message_part lacks a constraint requiring mediaType and imageId when type = 'file', allowing invalid file parts to be stored and later rendered with missing image data.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At apps/backend/src/db/pg-schema.ts, line 263:
<comment>`message_part` lacks a constraint requiring `mediaType` and `imageId` when `type = 'file'`, allowing invalid file parts to be stored and later rendered with missing image data.</comment>
<file context>
@@ -258,6 +258,10 @@ export const messagePart = pgTable(
providerMetadata: jsonb('provider_metadata').$type<ProviderMetadata>(),
+
+ // file/image columns
+ mediaType: text('media_type'),
+ imageId: text('image_id').references(() => messageImage.id, { onDelete: 'set null' }),
},
</file context>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds full image upload support to the chat input, including upload, storage, display, and passing images to the LLM for multimodal conversations.
Backend
message_imagetable (both SQLite and PostgreSQL) for storing uploaded image data as base64. AddedmediaTypeandimageIdcolumns tomessage_partforfiletype parts./i/:imageIdserving stored images with immutable caching headers.AgentRequestUserMessageSchemato accept an optionalimagesarray ({ mediaType, data }).filetype UI parts with server URLs.convertUIPartToDBPart/convertDBPartToUIParthandlefiletype parts, mapping between image IDs and URLs.convertToModelMessages, resolves server image URLs (/i/{id}) to data URLs so the model provider receives actual image content.Frontend
FileUIPart[]via the AI SDK'ssendMessage({ text, files }), then extracts them inprepareSendMessagesRequestfor the backend. Images display immediately in the chat bubble via data URLs, and persist via server URLs after reload.UserMessageBubbleviagetMessageImageshelper./iand/cproxy routes to dev server config.Types
SendMessageArgs,QueuedMessage, andNewQueuedMessageto include optionalimages.getMessageImagesandextractImagesFromMessageutilities.