Skip to content

Substitute image names in prompts for image collections #521

@lchoquel

Description

@lchoquel

Before submitting

  • I'm using the latest released version of the library
  • I've searched open issues and found no duplicate

Describe the bug, tell us what went wrong

To include an image input into an LLM prompt, you reference it in the prompt. And then the image gets sent to the LLM and the in the prompt text, we substitute the image name by a image number as in Image-3.
This is broken when the images come from a collection of images like a ListContent or a simple list[] nested in a structured object.
We must find a way to avoid including their names as it can be misleading for the LLM.

Reproduction snippet

Run integration test with inference enabled:
make tip TEST=test_analyze_image_collection

Library version

0.17.3

Python version

3.11

Operating system

No response

Stack trace / error output

Additional context & screenshots

No response

Would you like to help fix this issue?

Not at this time

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions