Documentation on predictions

[clairelaudy]

Hi,
Would it be possible to have a little bit more documentation on the way one must call BioPathNet for prediction task ?

1. What is expected in the test file ?
   For instance, in test_vis.txt file, does each line correspond to an edge that one wants the network to predict ?
   If so, why aren't they the same in the result file ? I see a lot of relations with different levels of probability in the result, but I don't understand how they relate to the relations that are in the input test_vis.txt file.

2. Is it possible to disable the prediction of reverse egdes with an option in the config file?

3. Is there a documentation of how the negatives are handled?

Thanks

[emyyue]

Hi,

thanks for your question!

1. test files and REVERSE relations

• For test_vis.txt: these are the edges that will be predicted and visualized in a local graph. The main point is that there will be TWO files! One that is the FORWARD direction (as specified in the test_vis.txt) and another is the REVERSE direction (not specified but implicitly performed as well: for pathway12 GO:annotation gene1 (found in test_vis.txt) these will be pathway12 r gene1 and gene1 r-1 pathway12

• For test_pred.txt: this is going to be similar, as REVERSE directions are also computed. But in addition, one against all predictions will be generated. So:
  • pathway12 FORWARD (r) against all nodes of the same type of gene1
  • gene1 REVERSE (r-1) against all nodes of the same type of pathway12

REVERSE directed edges are always implicitly created and also used for message passing. For the calculation of the performance metrics, the probabilities or ranks for both directions are averaged. These are then the numbers (hits@k, mrr) you see in the log.txt file when running script/run.py.

• coming back to how harmonize the visualizations of both FORWARD and REVERSE direction: We have no clear answer. Both directions sometimes reveal something interesting, and for the performance metrics, the probability/ranks are averaged. But we didn't add them into an average graph. Usually, we choose the FORWARD direction for final visualization.

2. disable predictions of REVERSE edges: they are important, as they are averaged. For visualization, perhaps ignore them.

3. Which negatives do you mean? During training? Or prediction?

Best

[emyyue]

Please also find our internal README

#################################################
###################### RUN ######################
#################################################

# to run BioPathNet: make sure config file has the right data and output directory
# --gpus null for on CPU
# testing is done on "test.txt"

python script/run.py -s 1024 -c config/knowledge_graph/mock/mockdata_run.yaml --gpus [0] 

python script/run.py -s 1024 -c config/knowledge_graph/mock/mockdata_rotate.yaml --gpus [0]
python script/run.py -s 1024 -c config/knowledge_graph/mock/mockdata_transe.yaml --gpus [0] 
python script/run.py -s 1024 -c config/knowledge_graph/mock/mockdata_distmult.yaml --gpus [0] 
python script/run.py -s 1024 -c config/knowledge_graph/mock/mockdata_rgcn.yaml --gpus [0] 



##################################################
###################### PRED ######################
##################################################

# to do predictions given a checkpoint (model_epoch_8.pth)
# important predict.py will load test file named "test_pred.txt"
#### best way to generate this is to have all unique (head, relation, tail),
#### as predictions are done over p(all_t of node_type(t) | h, r), AS WELL AS p(all_h of node_type(h)| given t, r-1)
# important config file needs to have the flag "remove_pos: no"
#### else there will be no predictions of the specific triplets in test_pred.txt

python script/predict.py -c config/knowledge_graph/mock/mockdata_vis.yaml --gpus [0] --checkpoint dir/to/checkpoint/model_epoch_8.pth


#################################################
###################### VIS ######################
#################################################

# to visualize given a checkpoint (model_epoch_8.pth) and specific triplets to visualize in "test_vis.txt"
# same config file as in predictions can be used
# test_vis.txt needs to contain the ones to visualize
# also define a node_color_dict: which node type which color

## for graph
python script/visualize_graph.py -c config/knowledge_graph/mock/mockdata_vis.yaml --gpus [0] --checkpoint dir/to/checkpoint/model_epoch_8.pth

## for detail to each path
python script/visualize.py -c config/knowledge_graph/mock/mockdata_vis.yaml --gpus [0] --checkpoint dir/to/checkpoint/model_epoch_8.pth




#################################################
################## http-server ##################
#################################################
### visualize graph by following these instructions:
Instructions to run source stackoverflow: (https://stackoverflow.com/questions/38497334/how-to-run-html-file-on-localhost):
You can run your file in http-server.

1> Have Node.js installed in your system.

2> In CMD, run the command npm install http-server -g

3> Navigate to the specific path of your file folder in CMD and run the command http-server

4> Go to your browser and type localhost:8080. Your Application should run there.

[clairelaudy]

Hi,
Thanks for your answer.
I managed to run the learning and prediction scripts on some data that I have, but I had to set 'strict_negative' to 'no' in the config file. Otherwise I had an IndexError in the biopathnet/task.py file.
This is why I am asking if you have documentation on how to handle the negative examples, or the different options in the config file.

[emyyue]

The flag 'strict_negative' controls how the negatives are sampled. These are used in the supervised scheme (binary classification) of maximizing the score for true links and minimizing the score for links that are not present.

If it is set to yes, then only true negatives are sampled (which usually makes the most sense). If it is set to no, then also positives are sampled.

I am not quite sure what would trigger the error, can you paste it here?

[clairelaudy]

The command line is:
python script/run.py -s 1024 -c config/mockdata_run.yaml --gpus null

The config and data files arre the following ones:

brg.txt

entity_names.txt

entity_types.txt

learning.txt

validation.txt

test.txt

mockdata_run.yaml

The error stack is the following one:

15:38:26   Epoch 0 begin
Traceback (most recent call last):
  File "/Users/claudy/work/projects/BioPathNet/script/run.py", line 102, in <module>
    train_and_validate(cfg, solver)
  File "/Users/claudy/work/projects/BioPathNet/script/run.py", line 59, in train_and_validate
    solver.train(**kwargs)
  File "/opt/miniconda3/envs/biopathnet/lib/python3.10/site-packages/torchdrug/core/engine.py", line 161, in train
    loss, metric = model(batch)
  File "/opt/miniconda3/envs/biopathnet/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/miniconda3/envs/biopathnet/lib/python3.10/site-packages/torchdrug/tasks/reasoning.py", line 88, in forward
    pred = self.predict(batch, all_loss, metric)
  File "/Users/claudy/work/projects/BioPathNet/biopathnet/task.py", line 326, in predict
    neg_index = self._strict_negative(pos_h_index, pos_t_index, pos_r_index)
  File "/opt/miniconda3/envs/biopathnet/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/Users/claudy/work/projects/BioPathNet/biopathnet/task.py", line 439, in _strict_negative
    neg_h_index = functional.variadic_sample(neg_h_candidate, num_h_candidate, self.num_negative)
  File "/opt/miniconda3/envs/biopathnet/lib/python3.10/site-packages/torchdrug/layers/functional/functional.py", line 432, in variadic_sample
    sample = input[index]
IndexError: index 9 is out of bounds for dimension 0 with size 9