Feature/add prompt caching to bedrock client#659
Open
kallilsouza wants to merge 2 commits intomainfrom
Open
Conversation
…hods - Added a new PromptMessage dataclass to encapsulate message text and caching behavior. - Updated the generate_text method in BaseAIPlatformClient to accept a list of PromptMessage objects instead of a single string. - Modified HistorySummaryService and BedrockClient to utilize the new PromptMessage structure for improved request formatting.
…ntegration classes - Changed the type hint for prompt_msgs in BaseAIPlatformClient and BedrockClient from list to List[PromptMessage] for better type safety and clarity. - This aligns with the recent introduction of the PromptMessage dataclass.
AlanJaeger
approved these changes
Aug 29, 2025
elitonzky
approved these changes
Sep 2, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Introduce PromptMessage dataclass and update AI integration methods
Why
This is done to use AWS Bedrock's prompt caching for faster model inference
Notes
The cache will only be applied if the prompt text hits the minimum number of tokens per cache checkpoint for the model used.