GBEA TODOs

[ttusar]

EDIT: Keeping here only the TODOs we finished already, a list of new/current ones can be found below

• Tea: fix issue socket_server_start(): Accept failed: Too many open files
• Tea: add back the "rw" to suite names
• Tea: Top Trumps has only discrete variables!
• Tea or Vanessa: automatically run the correct server for the chosen suite

[ttusar]

It looks as if the problem is fixed now. See the gbea branch.

[CIGbalance]

python3 ~/coco/do.py run-socket-python & sleep 3s && python3 example_experiment.py toy-socket 10

[ttusar]

Some notes of the conversation with @nikohansen:

• How to handle GBEA's code and data: The ideal would be to place the experiments code into COCO's master branch and only download the GBEA-related data (& possibly code if it is packaged as a docker container) only if/when needed.
• We can add the scripts to automate the addition of new real-world suites (and supporting their post-processing) into the tools.
• We can assume that a suite of problems either contains only problems with known optimal values or only problems with unknown optimal values. In any case, they should be logged into the same file (no need for a new file). Tea will check how this is currently implemented (is this handled separately for each problem?)
• Tea will check whether the post-processing also requires tdat files to be there (not really important, just to know).
• The function values of a suite should be "normalized" (not exactly, but should have a similar range). The suite should make sure that they are similar.
• The runlength-based targets can be used to automatically choose the relevant target values for a suite. This might require to create some data first.
• We can add some additional information to the dat file (it shouldn't represent a problem for post-processing).
• We can increase the evaluation number by a multiple without introducing problems, but might need to take care of corner cases (at the beginning!).
• Handling noise in case the true value is unknown is a bit tricky. The other case (the true value is known) should be straightforward to support (log both as given by the header), but hasn't been done yet.
• Recommendations should also be "easy" to support (see observers/loggers should take recommendations into account numbbo/coco#169).
• Automate the socket tests: you might not need to run the terminal, just the process; add a shutdown command that shuts down the socket server.
• In order to use these suites for a competition, one would need to choose some dimensions (the middle ones) and a not too small budget and then compute the area under the ECDFs (there is some code for this in the future cocopp). This number could be averaged over dimensions.

[nikohansen]

The function values of a suite should be "normalized" (not exactly, but should have a similar range). The suite should make sure that they are similar.

We may make an exception for real world suites and decide to log the raw data and only make some normalization afterwards, keeping in mind that the logged values are then uniform in the raw data only.

[CIGbalance]

High priority:

• Both: fix remaining issues (top trumps on Linux - Permission error on Linux #46 and mario gan on Windows - Working directory problem in Mario GAN #43)
• Vanessa: specify value ranges for all existing GBEA suites (nadir + ideal)
• Tea: The function values of a suite should be "normalized" (not exactly, but should have a similar range). The suite should make sure that they are similar.
• Vanessa: create a distribution for mario gan as is done for top trumps
• Tea: make the post-processing work

For competitions:

• Tea: update COCO documentation
• Both: evaluation scripts

Low priority:

• Both: figure out a way to dynamically set the number of simulations (would need to make additional suites where the number of simulations differs)
• Both: check that the suites can be safely parallelized
• Both: customization scripts for real-world problems - for experiments and post-processing (put them in the tools folder)
• Tea: We should support some kind of versioning of the evaluators that would be visible also from the logs in COCO

Done:

• Vanessa: MarioGAN.jar output not formatted as expected, got 0.08589678344726563 (experiment in C) - is this still a thing?
• Vanessa: calling python3 directly from MarioGAN.jar fails on windows -> find a way to do an alias
• Vanessa: sends Tea any random search "raw data" results with the rw-logger
• Vanessa: set problem types according to function groups (see coco_problem_set_type calls)
• Tea: Recommendations should also be "easy" to support (see observers/loggers should take recommendations into account numbbo/coco#169).

[ttusar]

Unfortunately, the "output not formatted as expected" still appears, I will create an issue for it