Add integration test for NVR edge cases

[vaibhavdaren]

Add test to verify file artifact provider correctly handles NVR edge cases.
This is continued from discussion on the PR for file provider: #4510 (comment)

Related: #4546

Pull Request Checklist

• write the documentation
• extend the test coverage
• update the specification

[gemini-code-assist]

Summary of Changes

Hello @vaibhavdaren, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a new integration test to validate the robustness of the file artifact provider, ensuring it accurately handles various NVR scenarios, particularly when dealing with the installation of multiple RPM packages. This enhancement improves the reliability of artifact provisioning within the system.

Highlights

• New Integration Test: An integration test has been added to verify the file artifact provider's behavior with NVR (Name-Version-Release) edge cases.
• NVR Edge Case Handling: The test specifically checks the correct installation and origin verification of multiple RPM packages (jq and nano) when provided via the file artifact provider from a local directory.
• Test Setup: The test uses dnf download to acquire RPMs and then runs tmt with the file artifact provider to ensure packages are correctly installed and identified within the test environment.

Changelog

• tests/prepare/artifact/nvr-edge-cases/data/.fmf/version
  • Added a version file for the test data.
• tests/prepare/artifact/nvr-edge-cases/data/main.fmf
  • Added a new FMF plan defining the discover, provision, and execute steps for the NVR edge case test.
  • Specified jq and nano as required packages for the test execution.
• tests/prepare/artifact/nvr-edge-cases/data/test.sh
  • Added a shell script to verify the installation of jq and nano.
  • Included checks to confirm that jq and nano originated from the tmt-artifact-shared repository.
• tests/prepare/artifact/nvr-edge-cases/main.fmf
  • Added the main FMF file describing the NVR edge case test, including a summary and detailed description.
• tests/prepare/artifact/nvr-edge-cases/test.sh
  • Added the primary test script using beakerlib to set up the test environment.
  • Implemented steps to download jq and nano RPMs using dnf into a temporary directory.
  • Executed tmt run with the file artifact provider pointing to the downloaded RPMs to simulate the NVR edge case.

Activity

• The pull request was created by vaibhavdaren.
• New test files have been added to cover NVR edge cases for the file artifact provider.
• The changes are related to issue Enhance artifact install plugin to verify RPM packages using hashes for same-NVR scenarios #4546.
• The author has included a checklist for documentation, test coverage, and specification updates, which are currently pending.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request adds a valuable integration test for the file artifact provider, covering edge cases with NVRs by testing its ability to handle a directory containing multiple RPMs. The overall implementation is solid. I've included one suggestion to improve the maintainability of the test script by reducing code duplication.

[vaibhavdaren]

/packit test

[vaibhavdaren]

/packit build

[vaibhavdaren]

Note🥇: Isolated the failing test ^_^ in testing-farm:fedora-rawhide-x86_64:full
https://artifacts.dev.testing-farm.io/f4e74ca8-7d1a-43f6-961c-e62e586f157b

[vaibhavdaren]

/packit retest-failed

[LecrisUT]

Ok, did some local testing to see how this behaves, indeed the priority takes precedence over all this. @vaibhavdaren @AthreyVinay Do you have a document to put the test cases and behavior?

[vaibhavdaren]

/packit test

[vaibhavdaren]

/packit build

[vaibhavdaren]

Ok, did some local testing to see how this behaves, indeed the priority takes precedence over all this. @vaibhavdaren @AthreyVinay Do you have a document to put the test cases and behavior?

We dont have a file, I think we can use #4546 to document our findings.

[vaibhavdaren]

When dependency are present in the shared artifact dir the tests pass on rawhide: https://artifacts.dev.testing-farm.io/361bd67e-9794-4863-8d91-f2152506bc83

[LecrisUT]

When dependency are present in the shared artifact dir the tests pass on rawhide: https://artifacts.dev.testing-farm.io/361bd67e-9794-4863-8d91-f2152506bc83

Can you elaborate? Basically the fact that jq built F43 failed is expected, but the fact that it fallbacked to system is unexpected and we must prevent that in the final design, i.e. the installation must fail with that exact packages requested. If you make it install jq-1.8.1-2.fc44.x86_64 instead of jq it will fail as we expect it to. I have put some more details of observations in the google docs, but I will move it to #4546, let's move the discussion there.

[vaibhavdaren]

but the fact that it fallbacked to system is unexpected and we must prevent that in the final design.
Yes, We are on the same page.

Can you elaborate? Basically the fact that jq built F43 failed is expected, but the fact that it fallbacked to system is unexpected and we must prevent that in the final design.

Yes sure..

Basically the fact that jq built F43 failed is expected

1. First question: Understanding why? [ In Progress]
2. It passes when we have the dependencies in shared-repo as well: See /tests/prepare/artifact/nvr-edge-cases/package-with-dependency

[vaibhavdaren]

/packit build