feat: Add Apple Intelligence on-device formatting#92
feat: Add Apple Intelligence on-device formatting#92kentaro wants to merge 7 commits intoamicalhq:mainfrom
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds macOS Foundation Models ("Apple Intelligence") support: new native RPCs and Swift FoundationModelService, types/schemas, a formatting provider, model sync endpoints and service logic, macOS-only UI for availability/sync, and tests for formatter, model service, and schemas. Changes
Sequence Diagram(s)sequenceDiagram
participant UI as React UI (AppleIntelligenceProvider)
participant Router as tRPC Router (models.ts)
participant Service as ModelService
participant Bridge as NativeBridge
participant Swift as Swift Backend (FoundationModelService)
participant DB as Database
UI->>Router: checkAppleIntelligenceAvailability()
Router->>Bridge: checkFoundationModelAvailability()
Bridge->>Swift: checkAvailability()
Swift-->>Bridge: {available, reason?}
Bridge-->>Router: result
Router-->>UI: {available, reason}
UI->>Router: syncAppleIntelligenceModel()
Router->>Service: syncAppleIntelligenceModel(nativeBridge)
alt available
Service->>Bridge: checkFoundationModelAvailability()
Bridge->>Swift: checkAvailability()
Swift-->>Bridge: {available: true}
Service->>DB: upsertModel(appleIntelligence metadata)
DB-->>Service: success
Service-->>Router: {available: true}
Router-->>UI: toast: synced
else unavailable
Service->>Bridge: checkFoundationModelAvailability()
Bridge->>Swift: checkAvailability()
Swift-->>Bridge: {available: false, reason}
Service->>DB: removeModel(appleIntelligence)
DB-->>Service: success
Service-->>Router: {available: false, reason}
Router-->>UI: toast: notAvailable
end
sequenceDiagram
participant Transcription as TranscriptionService
participant Formatter as AppleIntelligenceFormatter
participant Bridge as NativeBridge
participant Swift as Swift Backend (FoundationModelService)
Transcription->>Transcription: detect provider = AppleIntelligence
alt NativeBridge available
Transcription->>Formatter: format({systemPrompt, userPrompt,...})
Formatter->>Bridge: generateWithFoundationModel(params)
Bridge->>Swift: generate(params)
Swift-->>Bridge: {content}
Bridge-->>Formatter: result
Formatter->>Formatter: parse <formatted_text> tags or fallback
Formatter-->>Transcription: formatted text
Transcription->>Transcription: update transcript, metrics
else NativeBridge unavailable
Transcription->>Transcription: log warning, skip formatting
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Fix all issues with AI agents
In
`@apps/desktop/src/renderer/main/pages/settings/ai-models/components/apple-intelligence-provider.tsx`:
- Around line 15-22: The hooks
(api.models.checkAppleIntelligenceAvailability.useQuery, api.useUtils,
api.models.syncAppleIntelligenceModel.useMutation) are being conditionally
skipped by the early return on isMac; move the platform guard (the if (!isMac)
return null) below all hook calls so hooks run unconditionally and in the same
order, and update the availabilityQuery call to pass an options object with
enabled: isMac so the query only executes on macOS (e.g., availabilityQuery =
api.models.checkAppleIntelligenceAvailability.useQuery(undefined, { enabled:
isMac })); keep other hooks as-is but ensure no other returns occur before the
hooks are declared.
In
`@packages/native-helpers/swift-helper/Sources/SwiftHelper/services/FoundationModelService.swift`:
- Around line 7-37: The generate method uses the wrong LanguageModelSession
initializer and omits passing the system prompt into the session/response;
change LanguageModelSession(instructions: ...) to LanguageModelSession(model:
model) (e.g., let model = SystemLanguageModel.default; let session =
LanguageModelSession(model: model)) and then ensure the systemPrompt
(params.systemPrompt) is provided to the session/respond call per the
FoundationModels API (either by adding the prompt as an initial message/context
to the session or by using the documented respond(...) parameters such as
schema:, generating:, or options: to include system context); update the call to
session.respond(...) to match the official signature and include the
systemPrompt where required.
- Around line 26-37: In generate(params:
GenerateWithFoundationModelParamsSchema) build a GenerationOptions using
params.temperature and params.maxTokens and pass it into
LanguageModelSession.respond(to:options:) instead of the no-options overload;
update the code around LanguageModelSession(instructions:) and the respond call
to create GenerationOptions(temperature: params.temperature,
maximumResponseTokens: params.maxTokens) and call session.respond(to:
params.userPrompt, options: options) so caller-provided temperature/maxTokens
are applied.
🧹 Nitpick comments (5)
packages/native-helpers/swift-helper/Sources/SwiftHelper/RpcHandler.swift (1)
240-277: Consider using[weak self]in theTaskclosure for consistency.Other async handlers in this file (Lines 38, 46, 100) use
[weak self]in their closures. TheTaskon Line 257 capturesselfstrongly. While this is unlikely to cause issues given the process lifecycle, using[weak self]would be consistent with the rest of the file.Proposed diff
// Run async Foundation Model call on a background queue let requestId = request.id - Task { + Task { [weak self] in + guard let self = self else { return } do { let result = try await self.foundationModelService.generate(params: generateParams) self.sendResult(id: requestId, result: result)packages/types/scripts/generate-json-schemas.ts (1)
127-146: RemoveCheckFoundationModelAvailabilityParamsSchemafrom the Swift model generation since the method doesn't accept params.The
checkFoundationModelAvailabilitycase inRpcHandler.swiftignores params entirely and directly callsfoundationModelService.checkAvailability()with no arguments. UnlikegenerateWithFoundationModel, which decodes and usesGenerateWithFoundationModelParamsSchema, this method doesn't decode or pass params to the service. The optional empty object schema is therefore unnecessary in the quicktype input and can be removed from the command ingenerate-swift-models.ts.apps/desktop/src/pipeline/providers/formatting/apple-intelligence-formatter.ts (1)
35-56: Consider trimming the extracted text from XML tags.The regex captures content between
<formatted_text>tags including any leading/trailing whitespace or newlines that the model might insert (e.g.,<formatted_text>\nHello world\n</formatted_text>). The empty check on Line 49 uses.trim(), but the returned value on Line 56 does not trim. This could introduce unwanted leading/trailing whitespace in the formatted output.Optional: Trim extracted text
- const formattedText = match ? match[1] : result.content; + const formattedText = match ? match[1].trim() : result.content.trim();apps/desktop/src/services/model-service.ts (1)
826-885: Clean implementation with proper platform guarding and error handling.The method correctly:
- Short-circuits on non-macOS platforms
- Delegates availability check to the native bridge
- Upserts or removes the model based on availability
- Provides graceful error handling returning a structured result
One minor observation: the provider name
"AppleIntelligence"and model id"apple-intelligence"are used as magic strings here and likely referenced elsewhere (formatter, tRPC routes, UI). Consider extracting these to shared constants to prevent drift, but this is not urgent.apps/desktop/src/services/transcription-service.ts (1)
498-574: Consider extracting the repeated provider-instantiation pattern.All three formatting branches (OpenRouter, Ollama, AppleIntelligence) follow an identical structure: check precondition → log → instantiate provider → call
formatWithProvider→ update result. This is a pre-existing pattern, but with a third provider it's becoming worth extracting into a helper (e.g., a factory or a map of provider constructors) to reduce the growing if/else-if chain and make adding future providers less error-prone.
...esktop/src/renderer/main/pages/settings/ai-models/components/apple-intelligence-provider.tsx
Show resolved
Hide resolved
| class FoundationModelService { | ||
|
|
||
| func checkAvailability() -> CheckFoundationModelAvailabilityResultSchema { | ||
| #if canImport(FoundationModels) | ||
| if #available(macOS 26, *) { | ||
| let model = SystemLanguageModel.default | ||
| switch model.availability { | ||
| case .available: | ||
| return CheckFoundationModelAvailabilityResultSchema(available: true, reason: nil) | ||
| case .unavailable(let reason): | ||
| return CheckFoundationModelAvailabilityResultSchema(available: false, reason: String(describing: reason)) | ||
| @unknown default: | ||
| return CheckFoundationModelAvailabilityResultSchema(available: false, reason: "unknown") | ||
| } | ||
| } | ||
| #endif | ||
| return CheckFoundationModelAvailabilityResultSchema(available: false, reason: "deviceNotEligible") | ||
| } | ||
|
|
||
| func generate(params: GenerateWithFoundationModelParamsSchema) async throws -> GenerateWithFoundationModelResultSchema { | ||
| #if canImport(FoundationModels) | ||
| if #available(macOS 26, *) { | ||
| let instructions = params.systemPrompt | ||
| let session = LanguageModelSession(instructions: instructions) | ||
| let response = try await session.respond(to: params.userPrompt) | ||
| return GenerateWithFoundationModelResultSchema(content: response.content) | ||
| } | ||
| #endif | ||
| throw NSError(domain: "FoundationModelService", code: -1, | ||
| userInfo: [NSLocalizedDescriptionKey: "Foundation Models not available on this device"]) | ||
| } |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
Apple FoundationModels framework LanguageModelSession macOS 26 Swift API documentation
💡 Result:
Apple’s FoundationModels framework (Swift) docs you’re looking for are in Apple Developer Documentation under the FoundationModels module, with LanguageModelSession as the main entry point for running the on-device LLM.
Key Apple API reference pages (Swift)
https://developer.apple.com/documentation/foundationmodels/systemlanguagemodel
https://developer.apple.com/documentation/foundationmodels/languagemodelsession
https://developer.apple.com/documentation/foundationmodels/languagemodelsession/response
https://developer.apple.com/documentation/foundationmodels/languagemodelsession/respond(to:generating:includeSchemaInPrompt:options:)
https://developer.apple.com/documentation/foundationmodels/languagemodelsession/respond(to:schema:includeSchemaInPrompt:options:)
Sources: Apple Developer Documentation pages for SystemLanguageModel, LanguageModelSession, LanguageModelSession.Response, and LanguageModelSession.respond(...). [1][2][3][4][5]
Minimal Swift usage (session-based)
import FoundationModels
let model = SystemLanguageModel.default
let session = LanguageModelSession(model: model)
let response = try await session.respond(to: "Write a 1-sentence summary of macOS 26.")
print(response.content)(Types/methods documented on the pages above.) [1][2][4]
macOS 26 note (availability / simulator)
Apple states Foundation Models framework is available on iOS 26, iPadOS 26, and macOS 26. [6]
If you’re testing in the simulator, you may need the host Mac to be running macOS 26 (common error: “Host is not running macOS 26.”). [7]
Sources
[1] Apple Developer Documentation — SystemLanguageModel: (developer.apple.com)
[2] Apple Developer Documentation — LanguageModelSession (reference listing via related Apple doc pages): (developer.apple.com)
[3] Apple Developer Documentation — LanguageModelSession.Response: (developer.apple.com)
[4] Apple Developer Documentation — respond(to:generating:includeSchemaInPrompt:options:): (developer.apple.com)
[5] Apple Developer Documentation — respond(to:schema:includeSchemaInPrompt:options:): (developer.apple.com)
[6] Apple Newsroom — Foundation Models framework availability on iOS 26/iPadOS 26/macOS 26: (apple.com)
[7] Stack Overflow — simulator/macOS 26 host requirement error: (stackoverflow.com)
Citations:
- 1: https://developer.apple.com/documentation/foundationmodels/systemlanguagemodel?utm_source=openai
- 2: https://developer.apple.com/documentation/foundationmodels/languagemodelsession/respond%28to%3Aschema%3Aincludeschemainprompt%3Aoptions%3Aisolation%3A%29?changes=_10_5&utm_source=openai
- 3: https://developer.apple.com/documentation/foundationmodels/languagemodelsession/response?utm_source=openai
- 4: https://developer.apple.com/documentation/foundationmodels/languagemodelsession/respond%28to%3Agenerating%3Aincludeschemainprompt%3Aoptions%3A%29?utm_source=openai
- 5: https://developer.apple.com/documentation/foundationmodels/languagemodelsession/respond%28to%3Aschema%3Aincludeschemainprompt%3Aoptions%3A%29?utm_source=openai
- 6: https://www.apple.com/newsroom/2025/09/apples-foundation-models-framework-unlocks-new-intelligent-app-experiences/?utm_source=openai
- 7: https://stackoverflow.com/questions/79665051/no-llms-on-ios-if-you-dont-have-macos-26-tahoe/79724238?utm_source=openai
LanguageModelSession constructor signature does not match official Apple documentation.
According to Apple's FoundationModels API reference, LanguageModelSession is initialized with a model parameter, not instructions. The correct constructor is LanguageModelSession(model:), but the code uses LanguageModelSession(instructions:). The system prompt handling is not documented in the official API; verify how to pass instructions/prompts to the session, as the current approach will not work as written.
Also confirm that session.respond(to:) should include the system prompt context. The documented API shows additional parameters like schema:, generating:, and options: that may be necessary for proper functionality.
🤖 Prompt for AI Agents
In
`@packages/native-helpers/swift-helper/Sources/SwiftHelper/services/FoundationModelService.swift`
around lines 7 - 37, The generate method uses the wrong LanguageModelSession
initializer and omits passing the system prompt into the session/response;
change LanguageModelSession(instructions: ...) to LanguageModelSession(model:
model) (e.g., let model = SystemLanguageModel.default; let session =
LanguageModelSession(model: model)) and then ensure the systemPrompt
(params.systemPrompt) is provided to the session/respond call per the
FoundationModels API (either by adding the prompt as an initial message/context
to the session or by using the documented respond(...) parameters such as
schema:, generating:, or options: to include system context); update the call to
session.respond(...) to match the official signature and include the
systemPrompt where required.
There was a problem hiding this comment.
LanguageModelSession(instructions:) is a valid initializer in the macOS 26 FoundationModels API. Verified working at runtime:
[FoundationModel] Raw response type: Response<String>
[FoundationModel] Raw response description: Response<String>(
userPrompt: "...",
duration: 1.12,
content: "<formatted_text>\n私の好きな食べ物は以下の3つです:\n- ラーメン\n- カレー\n- 親子丼\n</formatted_text>"
)
The Apple documentation only lists LanguageModelSession(model:), but the instructions: parameter does exist and functions as the system prompt. The CodeRabbit web search results appear to be incomplete.
The other two issues (Hooks violation and GenerationOptions) have been addressed in 5b9a549.
There was a problem hiding this comment.
@kentaro ご指摘とランタイムログによる検証をありがとうございます！LanguageModelSession(instructions:) が有効なイニシャライザであることを確認いただき感謝します。
公式ドキュメントに記載されていない API が実際に存在し機能している点、貴重な情報です。実際の動作ログで instructions: パラメータがシステムプロンプトとして正しく機能していることが確認できました。
また、他の2点（Hooks 違反と GenerationOptions）への対応もありがとうございました（5b9a549）。
packages/native-helpers/swift-helper/Sources/SwiftHelper/services/FoundationModelService.swift
Show resolved
Hide resolved
|
@kentaro Thanks for this. |
Integrate macOS 26 Foundation Models as a formatting provider for transcribed text. Apple Intelligence runs entirely on-device with no API key required, providing text formatting with proper punctuation, capitalization, and structure. Key changes: - Add FoundationModelService in Swift helper with availability check and text generation - Add AppleIntelligenceFormatter following the same pattern as OpenRouter/Ollama formatters - Add RPC methods: checkFoundationModelAvailability, generateWithFoundationModel - Add Apple Intelligence provider UI in settings with availability detection - Add model sync via tRPC routes - Wire up provider routing in TranscriptionService - Remove misleading "If the text is empty" instruction from formatter prompt that caused on-device models to return empty formatted_text tags - Graceful degradation on non-macOS and older macOS via platform checks at every layer Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix React Hooks violation: move platform guard below all hook calls and add `enabled: isMac` to prevent query on non-Mac platforms - Pass temperature/maxTokens to Foundation Model via GenerationOptions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Zod z.number() maps to Swift Double via quicktype, but GenerationOptions.maximumResponseTokens expects Int. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
kentaro
force-pushed
the
feat/apple-intelligence-formatting
branch
from
29b9885 to
bbaaf11
Compare
|
@kentaro is the formatting working for you with Apple Intelligence? For me, the responses are very disappointing, to say the least. |
|
Speak as follows:
It has been formatted with Markdown syntax like this:
However, it seems Typeless can format without explicit instruction, but I wasn't able to implement that far. 
Details are as follows. |
|
Maybe it was a language issue on my end. I will retry and let you know. |
…ersationally The Foundation Model was treating raw transcription text as a conversational query and responding with "I'm sorry, but I can't assist with that request" instead of formatting it. Prefix the user prompt with "Format the following transcribed text:" so the model understands the task. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Thanks for reporting this! The root cause is that the on-device Foundation Model is small and was treating raw transcription text as a conversational query — so instead of formatting it, the model was responding to it ("I'm sorry, but I can't assist with that request."). I've pushed a fix (9390d6d) that wraps the user prompt with It was working for me because I use Japanese as the input language, which is less likely to be interpreted as a conversational query by the model. English text like "hello" or short phrases were being treated as user questions. Could you try again with the latest changes and let me know if it improves? |
Always use amical-notes formatting rules (Markdown with smart structure detection) for Apple Intelligence on-device formatting. This ensures consistent output with bullet points, headers, etc. based on content structure, without the user needing to explicitly request list formatting. Added overrideAppType option to constructFormatterPrompt for future extensibility to other formatters. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Update: I've added basic implicit formatting for Apple Intelligence (a097068, 426f88f, df11fe6). The on-device model now uses the The |
On-device Foundation Models need concrete examples to reliably detect implicit structure in speech (e.g., comma-separated items should become bullet lists, sequential steps should become numbered lists). Added three examples covering lists, steps, and simple sentences. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Output is pasted into text fields that don't render Markdown, so **bold** and *italic* markers appear as literal characters. Keep only list and header formatting. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
3 participants
Summary
<formatted_text>tagsChanges
FoundationModelServicewith availability check and text generation viaLanguageModelSessionAppleIntelligenceFormatteraligned with OpenRouter/Ollama pattern + empty result fallbackcheckFoundationModelAvailability,generateWithFoundationModelmethods"If the text is empty, return <formatted_text></formatted_text>"instruction that misled small on-device models#if canImport,@available, TypeScript platform guard)Test plan
AppleIntelligenceFormatter(10 tests passing)transcription-service.ts:468)process.platformguard (TS),#if canImport(FoundationModels)(compile-time),#available(macOS 26, *)(runtime)transcriptions.test.tsunrelated to this PR)Summary by CodeRabbit
New Features
Chores
Tests