How to know which model is the best and what is its accuracy?

[meriemferdjouni]

I have trained 200 models using trainer.try_models() but now I need to know which model performed the best and its accuracy?

Anyone knows how to get the details such as: what evaluation score was used? what is the best model? and how can we use it later for prediction?

Thank you.

[anaptoro]

Im facing the same issue!

[maartenjv]

Some information on this was added in #39

[anaptoro]

Some information on this was added in #39

Thanks, #39 addresses the prediction issue but Im still not able to see how can I see the model's results gathered, in order to compare them and then pick the best one for my problem.... @maartenjv have you managed how to open it using tensorboard?

[maartenjv]

Opening the save_model with Tensorboard is possible, just use e.g.: 'tensorboard --logdir exp_folder\tuner-1\5'. But its not very informative as it does not nicely show the main graph in a summerized manner (as in model.summary()). You can get the accuracies etc. from there if you want.

Also, you can do something like this to see what building blocks out of the spec are used in the final model. But is does not show clearly the sequence or relation between blocks.

trainer = single_trainer.SingleTrainer(
    data=csv_data.Provider(
        label_index=0,
        logits_dimension=2,
        record_defaults=[0.0,0.0,0.0,0.0,0.0],
        filename="./train.csv"),
    spec= constants.DEFAULT_DNN)

loaded = tf.saved_model.load(exp_folder + '/tuner-1/18/saved_model/1632837857')

for idx, v in enumerate(loaded.variables):
    for s in trainer._spec.blocks_to_use:
        if s in v.name:
            print(v.name)
            print(v.shape)