Proper way of using CNN embedding networks if there's more than 1 channel

[jcgraciosa]

Hello everyone,

I am working on applying SBI to some solid Earth science problems.
My simulator generates pressure, $p$, and velocity fields, $v_x$ and $v_y$, and I am sampling from several locations to produce observations with dimensions (C = 3, H, W).
In the tutorial here, the simulator produces a single channel 32 x 32 image which is then flattened and eventually fed into the append_simulations.
My question is: how do I properly "pack" the observables in the case of multiple channels?
Should I flatten it and pack it as (Channel 0 H x W, Channel 1 H x W, Channel 2 H x W)?

Thank you so much for the help! :)

[michaeldeistler]

Hi there,

I don't think there's a clear answer for this---different approaches might work well. I am not an expert in computer vision, but I think the standard way to process multiple channels is to process each channel individually with a CNN (in your case probably a separate CNN for each channel) and to then combine them in the end (i.e. by flattening after the CNNs and then processing the flattened output with a multi-layer-perceptron). Such an embedding network is not available as a "pre-configured" embedding net in sbi, so you would have to implement it as in the tutorial that you linked.

Hope this helps!
Michael

[jcgraciosa]

Hello Michael,

Thank you so much for promptly answering the question.
That was helpful, but I have a couple of additional questions:

1. I think this embedding network [here] (

   sbi/sbi/neural_nets/embedding_nets/cnn.py

   Line 70 in 0ea6f63

   def __init__(

   ) can also handle multiple channels, or am I wrong?
2. For append_simulations(), is there a restriction on the dimensions of theta and x? I.e. do they have to be 2D tensors with each row corresponding to a simulation run?

Thank you in advance!
JC

[psteinb]

Hey @jcgraciosa, so first of,

append_simulations will throw an error if it receives x tensors with any shape other than (B, D) (B being the batch size, and D being the dimension that is left). The CNN from the sbi package that you mention can handle multiple channels in the input, but I'd say that it is not hard to write-up your own CNN if you don't like it.

Note also, you have multiple options to go for the embeddings.

1. Either you train a CNN (as mentioned above) along with the density estimator or
2. you pretrain a autoencoder-style (a network which has a bottleneck) network before and only hand-in the encoder to the density estimator.

The latter provides you with a somewhat better handle if the encoder is doing a good job as you can check the reconstruction quality on your embeddings independently.
In either case, you'd treat multiple input channels alike in the CNN or encoder by either merge them at some point through convs layers or dedicated 1x1 conv layers.

Cheers,
P

[michaeldeistler]

1. Ah, yes, you are right, it can handle multiple channels.
2. For NPE: theta must be of shape B, T, where B is the batch dimension (i.e. number of simulations) and T is the number of parameters. x must be of shape B, *data_shape, where *data_shape can have multiple dimensions itself (i.e. be an image). If *data_shape is not an integer (as in your case), then you must provide an embedding net that reduces *data_shape into an integer. As such, the @psteinb's comment append_simulations will throw an error if it receives x tensors with any shape other than (B, D) is only correct without embedinng net.