Relaxing constant sigma assumption

[kevinykuo]

If we wanted to make this bit

NeuralProcesses/NP_architecture2.R

Lines 68 to 71 in 5119ac0

	# for the toy example, assume y* ~ N(mu, sigma) with fixed sigma
	sigma_star <- tf$constant(noise_sd, dtype = tf$float32)
	list(mu = mu_star, sigma = sigma_star)

more general, what would be the correct way to do it? Would we try to estimate it from the n_draws draws of each of the y* predictions?

[kasparmartens]

You could make noise_sd a parameter and try to learn it (for numerical stability, you probably want to lower- and upper-bound it).

[kevinykuo]

Thanks for the reply! Do you mean e.g. outputting another quantity connected to hidden?

NeuralProcesses/NP_architecture2.R

Lines 59 to 66 in 5119ac0

	hidden <- input %>%
	tf$layers$dense(dim_g_hidden, tf$nn$relu, name = "decoder_layer1", reuse = tf$AUTO_REUSE)
	# mu will be of the shape [N_star, n_draws]
	mu_star <- hidden %>%
	tf$layers$dense(1L, name = "decoder_layer2", reuse = tf$AUTO_REUSE) %>%
	tf$squeeze(axis = 2L) %>%
	tf$transpose()

That seems straightforward but I wasn't sure how to justify it since the decoder looked like it should only predict the target y's and we needed to obtain the variance elsewhere. But I guess we would be taking the samples of z in the input so that randomness is accounted for.

[kasparmartens]

Depends what kind of noise model you want to assume. The most natural one would probably be the one which assumes constant noise. E.g. in the GP-regression model, typical choice for p(y|f, x) would be Normal distribution with mean f(x) and variance \sigma^2, i.e. the latter would not depend on input x. In this case, \sigma^2 would be a single variable (not parameterised by a network).

If we are interested in scenarios where noise level varies with x, then we could indeed consider parameterising \sigma^2 along the lines as you described.