Get intermediate output without changing Schnetpack

[guacke]

[1] https://discuss.pytorch.org/t/how-can-i-extract-intermediate-layer-output-from-loaded-cnn-model/77301
[2] https://forums.fast.ai/t/pytorch-best-way-to-get-at-intermediate-layers-in-vgg-and-resnet/5707

[amerelsamman]

Hello,

It seems these methods require me to re-instantiate the module which outputs a new variable each time, that is it is untrained forward hook through the network. It does not get the trained value though.

I could be misunderstanding. But I have something working now but each time I run it I get a new tensor. I looked online (youtube actually :P) and it seems that this is the way this thing goes. It reinstatiates the model that you call and it is thus not a trained variable, its weights will be different each time.

I am doing more research :)

[amerelsamman]

here is my code for it:

#Import the model

from schnetpack.representation import SchNet

#define the activation hook for an arbitrary layer name

def get_activation(name):
def hook(model, input, output):
activation[name] = output.detach()
return hook

#instatiate the model and then use the hook for 'embeddding' layer as the name

model = SchNet()
model.embedding.register_forward_hook(get_activation('embedding'))
output = model(inputs)
print(activation['embedding'])

perhaps I am close I just have to change something.

[guacke]

Could you add your code to an iPython notebook and commit it to this repo? Make sure that all required data is available to run the notebook (except the qm9.db, for which we will have to rely on a functioning Globus endpoint).

[amerelsamman]

Done! I have put it in notebooks. It seems to be instatiating the model but not my trained version

By the way, this seems to be an important and complicated area of pytorch with many ways to do it as I searched online. There has to be a way I just have to find it.

[nbilliet]

In order to extract the output of a specific layer the following code template should be used

model = ModelClass()

layer_output=None

def layer_hook_fn(self, inp_tensor, out_tensor):
   global layer_output
   layer_output=out_tensor

model.layer_name.register_forward_hook(layer_hook_fn)

The layer output will then be store in the layer_output variable and will be replaced on succesive forward passes

[amerelsamman]

This is how I have used the template so far (also you can see it in notebooks once I commit):

from schnetpack.representation import SchNet

model = SchNet()
layer_output=None

def layer_hook_fn(self, inp_tensor, out_tensor):
global layer_output
layer_output=out_tensor

model.embedding.register_forward_hook(layer_hook_fn)

print(layer_output)

*** With how I applied the template, when I run the code. layer_output just gives "none"

Two things confuse me:

1. How do I use the global layer_output part. I tried many things, but last thing was to try putting "embedding," or "x"

2. Say I use it correctly, how do I then print the output.

Thank you so much. This stuff really helps expand my understanding on how these algorithms are built and work in the first place.

[amerelsamman]

NOTE spaces are provided in the code appropriately but not in the above comment

[nbilliet]

Registering the function as the forward hook causes the use of model(inp_tensor) to direct the layer output that the function is hooked to to stream to the variable layer_output. So when you call the layer_output variable after a forward pass it will contain the output tensor of the desired layer

[amerelsamman]

yes I see what you mean. That is really helpful

However it seems I am not there yet because my code outputs "none." do you know why this could be?

What is to go in the "global layer_output" part. Am I supposed to replace it?

Essentially, it seems there is a gap, how do I tell the program, "okay I want you to look for 'embedding' layer in this module"

Thanks so much. Please feel free to take your time. I already really appreciate your tremendous help and hope to pay it back someday (if that even becomes possible once day :P)

I have tried replacing with "embedding" or "x" but none of that worked

Yes it seems you mean you want me to replace "layer_output" with what I want. But I tried this and it does not seem to work. Let me give it another shot and play around more

[nbilliet]

To utilize your example

class SchNet(torch.nn.Module):
   def __init__(args):
      super(SchNet, self).__init__()
      self.embedding = nn.Embedding()
      ...

model = SchNet()

If I then wish to extract the embedding layers output during a forward pass i register a hook to it

embedding_output=None

def embedding_hook(self, inp_tensor, out_tensor):
   # Global allows us to utilize embedding_output outside the current function scope
   global embedding_output 
   embedding_output=out_tensor

model.embedding.register_forward_hook(embedding_hook)

Now that we have hooked this layer the variable embedding_output will be changed when we call the forward function from our model class.

How you have currently implemented it in your notebook

model = SchNet()
layer_output=None

def layer_hook_fn(self, inp_tensor, out_tensor):
    global layer_output
    layer_output=out_tensor
    
model.embedding.register_forward_hook(layer_hook_fn)

print(layer_output)

It is to be expected that the layer_output variable will return None as you have not called the forward function. Remember that the hook function is linked to the forward function and will only be applied when we apply the forward function. As such the variable layer_output will only change when model(batch_inp_tensors) is called

[amerelsamman]

I have tried replacing layer with embedding as you did before. I even copied your code now to try again. But that still returns none. Tried restarting the kernal, same problem.

Do you have to change anything to the source code? Or copy it entirely rather than import it as I did

Sorry, I mean how do you call the forward function? your code copied on its own should work no?

[nbilliet]

Whenever you feed a network some input the forward function is called

[amerelsamman]

aha! yes I know what you mean I put model(input). This worked but it resulted in same problem as before. NOT none but now it instatiates the model each time. It is not a trained layer hmmm but I know there must be a way to get at the trained layer.....

That is the embedding layer is different each time