Regression tests

[obo]

Please create a directory called tests and put either a bash script there (e.g. run-all.sh), or a Makefile to test SLTev.
The testing should:

• rely only on SLTev repo, i.e. sample-data for now (extended tests with elitr-testset should be added later)
• should test the calculation of MT-only, ASR-only and SLT (three runs of SLTev)
• should diff the outputs and expected outputs and exit with non-zero status if the results are not exactly the same
• ideally be so quick that you could include it in a pre-commit hook

[mohammad2928]

Hi,
I will do it as soon as possible.

summary of files in the "SLTev/tests"
...
1- run-all.sh (or maybe a Makefile)
2- the output of three MT, ASR, SLT evaluations.
...

[mzilinec]

Hello, we definitely should not use a makefile or a bash script to implement tests, if possible.
The correct way to do this is using a standardized test framework for Python, I suggest pytest.
The tests can be easily integrated into Github and the results will be visible for every commit in this repository.
In addition, we can set up rules so that commits failing tests can't be released, merged into master etc.

[mohammad2928]

Hi,
Thanks Matus. I agree and I will try with pytest.

[srdecny]

I can help with setting up the CI (automated testing on every pull request) via Github Actions.

[srdecny]

And while we're at it, I heavily recommend using PRs to get new code to master, instead of directly pushing. You can automatically close issues with a PR, have a chance to revisit the code and it makes the overall history much neater (since you can look up the issue that the code is supposed to solve).

[mohammad2928]

I appreciate any help and idea.
Thanks.

[srdecny]

I see there's an example index file for showcasing the SLTev evaluation process. What about writing a script that will simply evaluate that index file? I don't think the exact results matter that much, we care about the evaluation not crashing. With the test script finished, we can setup a Github Action to run the script every time something is pushed to the main branch, so we can quickly notice if something went wrong, because the Action will fail.

[srdecny]

We can use a Dockerfile to install the dependencies, and then execute the test script inside that Dockerfile, either at build-time (when the Dockerfile is being built), or at runtime (inside the Docker container)

[mohammad2928]

Yes, I agree. But I think it is not complete because SLTev has multiple tasks, I don't have any experience with git actions.
Let's start and implement a simple demo of it and we will complete it step by step.

[srdecny]

Sure, so the first step is to write a Python/bash script that will run all of the tasks on a fixed set of input files. The script can assume all of the dependencies are alread installed. Once that's done, we'll figure how to integrate it into Github :)

[obo]

I agree with Vojtech. There is one more thing to create: the expected output file. The script (e.g. called minitest.sh) should:

1. run SLTev on this index file
2. store the output to minitest.out
3. diff minitest.out with minitest.expected
4. die if the outputs differ.

Obviously, when we fix some bugs in score calculations and the outputs will change, the commit should also update the .expected file.