Skip to content

Fix Image.frombytes() mode param usage #1365

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

akhilam512
Copy link

currently we're passing frame.format which typically contains file format strings like 'PNG' or 'JPEG' as the mode parameter to PIL.Image.frombytes()

This function actually expects image modes like 'RGB', 'RGBA', etc

I standardized all Image.frombytes() calls to use 'RGB' mode instead, fixed this in six files:

  • src/pipecat/transports/base_output.py
  • src/pipecat/services/gemini_multimodal_live/events.py
  • src/pipecat/services/moondream.py
  • src/pipecat/services/google/google.py
  • src/pipecat/services/anthropic.py
  • src/pipecat/processors/aggregators/openai_llm_context.py

For most of the places where it's used, 'RGB' as the image mode works perfectly since it is immediately convert to JPEG anyway. However in base_output.py, I went with 'RGB' for consistency, though I wonder if preserving other modes is useful? Not sure if a png with an alpha channel can be passed here for example, and we might prefer RGBA in that case (to account for that, we might need to add a dedicated class member for storing image mode in ImageRawFrame perhaps)

@akhilam512
Copy link
Author

Hmm sorry this isn't a complete solution still, I was a bit too eager to send the PR. Testing this now I get:

unexpected exception: not enough image data

I think this is because as its name suggests frombytes() expects raw bytes but we're still sending a compressed bytes via a png/jpg file. So frombytes() is probably not the right usage here at all, instead using something like Image.open(io.BytesIO(image)) and then saving the resultant image seems to work

@markbackman
Copy link
Contributor

@akhilam512 mind if I turn this into a draft until it's ready for review?

@markbackman markbackman marked this pull request as draft March 18, 2025 13:05
@markbackman
Copy link
Contributor

@akhilam512 is this something you're still working on or can I close it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants