Enforce sample numbering (controversial?)

[Matistjati]

Closes #25.

First, a minor sanity check: check that every .ans file in sample has a corresponding .in file.

For the controversial part: Kattis is now exposing more details about testcases in their UI.

Here, it seems like the participant was 3 test cases off from AC. However, they are actually only one testcase off: the two testcases at the end are the sample, because i named them 1.in and 2.in. I think that this is somewhat common, and that the fix is to number samples.

Because testdata_tools uses sample both as the "input" and "output" folder for samples, we don't really have any choice but to rename the samples.

Thus, this is not backwards-compatible, in the sense that you if you use the new gen.sh on old problems, you will be forced to do some cleanup.

Sample run:

matistjati@DESKTOP-BJV27M6:~/po/swedish-olympiad-2026/online/absolutbio/data$ rm -rf sample && git restore sample && ./generator.sh
ERROR: Some samples did not start with three digits followed by -
ERROR: Please change your samples to use the following the sample commands:
sample 001-1
sample 002-2
sample 003-3
sample 004-4
matistjati@DESKTOP-BJV27M6:~/po/swedish-olympiad-2026/online/absolutbio/data$

[Matistjati]

What do you think? Pinging some people who actively use problemtools: @hairez @Tagl . Any others may also chime in.

[Matistjati]

For reference, Kattis has implemented the correct behavior for Legacy: https://www.kattis.com/problem-package-format/spec/legacy.html#test-data-groups
Test cases and groups will be used in lexicographical order on file base name

But this will be fixed in 2025-09. https://www.kattis.com/problem-package-format/spec/2025-09.html#test-cases
Here, base name is defined to be the relative path from the data directory to the test case input file, without extensions. This is the name of the test case.

I'm considering closing this.

[simonlindholm]

IIRC we use 1.in, 2.in, ... because it is required by some other tool, maybe problem2pdf? We could solve this by using a different name for the symlink than for the actual file.

[Matistjati]

IIRC we use 1.in, 2.in, ... because it is required by some other tool, maybe problem2pdf? We could solve this by using a different name for the symlink than for the actual file.

Can't believe I missed that. That's a great solution!

Also, problemtools already warns for ans files without .in, so that part isn't needed either (ERROR No matching input file for answer '/home/matistjati/po/swedish-olympiad-2026/online/absolutbio/data/sample/x.ans')

[Matistjati]

I'm not sure that that's the solution we want either. If we do this, we lose the property that two files should have the same content iff they have the same name. For example, this triggers Problemtool's warning:
WARNING Identical input files: '['data/sample/1.in', 'data/secret/test/002-test.in']'
I guess this could be fixed by making the check symlink-aware, but I just don't think that this change is worth it, considering that it will automatically be fixed in the new problem format version.

[simonlindholm]

I thought that check was already symlink-aware.

            hashes = collections.defaultdict(list)
            for root, dirs, files in os.walk(self._datadir):
                for filename in files:
                    filepath = os.path.join(root, filename)
                    if filepath.endswith('.in') and not os.path.islink(filepath):
                        ...
                        hashes[filehash].append(os.path.relpath(filepath, self._problem.probdir))
            for _, files in hashes.items():
                if len(files) > 1:
                    self.warning(f"Identical input files: '{str(files)}'")

[Matistjati]

...right, I didn't even my own error message...

What do you think, should we make this change?

[simonlindholm]

Sure, why not.

[jsannemo]

Just make sure that compiling a full problemset with the tex template class works, as Simon mentioned.

…

On Mon, 22 Dec 2025, 15:17 Simon Lindholm, ***@***.***> wrote: *simonlindholm* left a comment (Kodsport/testdata_tools#30) <#30 (comment)> Sure, why not. — Reply to this email directly, view it on GitHub <#30 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHIMXPMZ26HK6JT3N7X4ZT4C74Q5AVCNFSM6AAAAACPXSD5BCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMOBSGI3TGNRUGM> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>