Skip to content

[New] LLM corrupted image detection#240

Merged
douglasdotc merged 25 commits intomainfrom
corrupt-image-detect
Apr 24, 2026
Merged

[New] LLM corrupted image detection#240
douglasdotc merged 25 commits intomainfrom
corrupt-image-detect

Conversation

@douglasdotc
Copy link
Copy Markdown
Collaborator

@douglasdotc douglasdotc commented Apr 21, 2026

Description

This PR adds a new keyword to detect corrupted images.

Resolved issues

ZAP-1424

Documentation

Added automatically

Tests

  1. Start Ollama Qwen model following: https://canonical-yarf.readthedocs-hosted.com/latest/how-to/using-the-llm-client/
  2. Start Mir server.
  3. Start YARF Interactive:
$ yarf --platform Mir


INFO: *** Welcome to the YARF interactive console. ***
INFO: You can use this console to execute Robot Framework keywords interactively.
INFO: The value of ${CURDIR} is CWD, you can change it using the `Set Variable` keyword.
INFO: You can press RIGHT_ARROW to auto-complete the keyword.
INFO: You can press CRTL + SPACE to view supported keywords on a prefix.
>>>>> Enter interactive shell
iRobot can interpret single or multiple keyword calls,
as well as FOR, IF, WHILE, TRY
and resource file syntax like *** Keywords*** or *** Variables ***.

Type "help" for more information.
> Configure Llm Client   model=qwen3-vl:2b-instruct
> Check For Visual Corruption                                                                                                                                   # ΔT: 0.001s
< {'corrupted': True, 'description': 'The image is entirely black, which is a typical indication of a corrupted or missing image.'}
> Check For Visual Corruption                                                                                                                               # ΔT: 171.163s
< {'corrupted': False, 'description': 'The image shows a file manager window displaying various folders and files on a dark theme, with a proper layout and no signs of corruption or noise.'}

Signed-off-by: Douglas Chiang <douglasdothnc@gmail.com>
Signed-off-by: Douglas Chiang <douglasdothnc@gmail.com>
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new Robot Framework keyword to use the configured multimodal LLM to assess whether the current screenshot (or a provided image) appears corrupted, and updates the generated library reference docs accordingly.

Changes:

  • Add Detect Corrupted Image keyword to LlmClient (screenshot capture + LLM prompt + JSON parsing).
  • Add helper to fetch other RF library instances via BuiltIn().get_library_instance.
  • Regenerate reference docs for the new keyword.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 7 comments.

File Description
yarf/rf_libraries/libraries/llm_client/LlmClient.py Adds the new corrupted-image detection keyword and related helper/validation logic.
docs/reference/rf_libraries/libraries/llm_client/library-LlmClient.md Documents the newly added keyword in the generated RF library reference.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread yarf/rf_libraries/libraries/llm_client/LlmClient.py Outdated
Comment thread yarf/rf_libraries/libraries/llm_client/LlmClient.py
Comment thread yarf/rf_libraries/libraries/llm_client/LlmClient.py Outdated
Comment thread yarf/rf_libraries/libraries/llm_client/LlmClient.py Outdated
Comment thread yarf/rf_libraries/libraries/llm_client/LlmClient.py Outdated
Comment on lines +157 to +163
@keyword
async def detect_corrupted_image(
self,
image: Image.Image | None = None,
custom_prompt: str | None = None,
) -> dict[str, Any]:
"""
douglasdotc and others added 4 commits April 21, 2026 13:11
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Douglas Chiang <douglasdothnc@gmail.com>
Signed-off-by: Douglas Chiang <douglasdothnc@gmail.com>
Signed-off-by: Douglas Chiang <douglasdothnc@gmail.com>
Copy link
Copy Markdown
Collaborator

@p-gentili p-gentili left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool stuff! Some comments below, mostly to focus more on UX and stability.

Comment thread docs/reference/rf_libraries/libraries/llm_client/library-LlmClient.md Outdated
Comment thread docs/reference/rf_libraries/libraries/llm_client/library-LlmClient.md Outdated
Comment thread yarf/rf_libraries/libraries/llm_client/LlmClient.py Outdated
Comment thread yarf/rf_libraries/libraries/llm_client/LlmClient.py Outdated
Comment thread yarf/rf_libraries/libraries/llm_client/LlmClient.py Outdated
Comment thread yarf/rf_libraries/libraries/llm_client/LlmClient.py Outdated
Co-authored-by: Paolo Gentili <paolo.gentili@canonical.com>
Comment thread yarf/rf_libraries/libraries/llm_client/LlmClient.py Outdated
Signed-off-by: Douglas Chiang <douglasdothnc@gmail.com>
Signed-off-by: Douglas Chiang <douglasdothnc@gmail.com>
Comment thread yarf/rf_libraries/libraries/llm_client/LlmClient.py Outdated
Comment thread yarf/rf_libraries/libraries/llm_client/LlmClient.py
douglasdotc and others added 3 commits April 22, 2026 10:49
Co-authored-by: Paolo Gentili <paolo.gentili@canonical.com>
Signed-off-by: Douglas Chiang <douglasdothnc@gmail.com>
Signed-off-by: Douglas Chiang <douglasdothnc@gmail.com>
Comment thread yarf/errors/yarf_errors.py Outdated
Comment thread yarf/rf_libraries/libraries/llm_client/LlmClient.py Outdated
Comment thread docs/reference/rf_libraries/libraries/llm_client/library-LlmClient.md Outdated
Comment thread yarf/rf_libraries/libraries/llm_client/LlmClient.py
Signed-off-by: Douglas Chiang <douglasdothnc@gmail.com>
Signed-off-by: Douglas Chiang <douglasdothnc@gmail.com>
Signed-off-by: Douglas Chiang <douglasdothnc@gmail.com>
Signed-off-by: Douglas Chiang <douglasdothnc@gmail.com>
Copy link
Copy Markdown
Collaborator

@p-gentili p-gentili left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found some issues in the code itself

Comment thread yarf/rf_libraries/libraries/llm_client/LlmClient.py Outdated
Comment thread yarf/rf_libraries/libraries/llm_client/LlmClient.py Outdated
Comment thread yarf/rf_libraries/libraries/llm_client/LlmClient.py Outdated
Comment thread yarf/rf_libraries/libraries/llm_client/LlmClient.py Outdated
douglasdotc and others added 5 commits April 24, 2026 10:18
Co-authored-by: Paolo Gentili <paolo.gentili@canonical.com>
Co-authored-by: Paolo Gentili <github@login.pgentili.com>
Signed-off-by: Douglas Chiang <douglasdothnc@gmail.com>
Signed-off-by: Douglas Chiang <douglasdothnc@gmail.com>
Signed-off-by: Douglas Chiang <douglasdothnc@gmail.com>
Comment thread yarf/errors/yarf_errors.py Outdated
@p-gentili p-gentili dismissed their stale review April 24, 2026 09:53

LGTM - leaving the testing to Fernando

douglasdotc and others added 2 commits April 24, 2026 10:54
Co-authored-by: Paolo Gentili <paolo.gentili@canonical.com>
Signed-off-by: Douglas Chiang <douglasdothnc@gmail.com>
Comment thread yarf/rf_libraries/libraries/llm_client/LlmClient.py
douglasdotc and others added 2 commits April 24, 2026 13:04
Co-authored-by: Fernando Bravo <39527354+fernando79513@users.noreply.github.com>
Signed-off-by: Douglas Chiang <douglasdothnc@gmail.com>
Copy link
Copy Markdown
Contributor

@fernando79513 fernando79513 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM +1!

Copy link
Copy Markdown
Collaborator

@p-gentili p-gentili left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested with this beauty (a problem with mesa apparently) running Qwen3.5 4B

Image

@douglasdotc douglasdotc merged commit d74a88a into main Apr 24, 2026
6 checks passed
@douglasdotc douglasdotc deleted the corrupt-image-detect branch April 24, 2026 15:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants