Replacing No-data channels with dummy dc timeseries

[saleem-muhammed]

A recent addition to the pipeline enforced the data from channels that are zero (or don't have data for the analysis segments) to be replaced with unity at all the samples, as a dc time-series will not have any correlation with the DARM. This was done at the level of reading the frames using the .read() function within the timeseries.py.

• Check if this is indeed working or is the right way to do it.
• If the above doesn't work, think of replacing the channels that don't;t have data so that at the time of inference, they will not be read at all.

[alecgunny]

Can you elaborate by what you mean by

unit at all the frequency bins

Is this for the strain channel at criterion-evaluation time, or for the witness channels? Do we ever calculate or use the spectra of the witness channels?

[saleem-muhammed]

Not all the witness channels necessarily have data all the time. in other words, w(t) data have a different duty cycle than h(t) data. The frame reader, by default, puts zero at all samples (sorry I said frequency bin earlier, that was wrong!) which will make the predictions Nans. So, I decided to enforce them to be 1 instead of zero as a simple fix.

[alecgunny]

Hmmm that's strange, is it the outputs of DeepClean that are NaN when one of the witness channels is all 0, or that somehow the spectrum comes out NaN or 0 in the denominator making the ASD loss function NaN? Or is it that when a channel all has any fixed value (including 0), the standard deviation comes out to 0 and so it gets NaN'd during the normalization step in preprocessing?

[saleem-muhammed]

Good point. The standard deviation could well be the culprit. Need to check this. The reason why I did not tend to remove them completely was that even if some channels are zero in the training, they may not be so in the cleaning segment, so identifying zeros and removing them will mess up everything. Also it will need keeping track of what channels were removed while training so that we make sure those channels are not being used in cleaning

[alecgunny]

even if some channels are zero in the training, they may not be so in the cleaning segment, so identifying zeros and removing them will mess up everything

This is true, but if the network was trained on this channel as basically noise (no correlation with the strain channel), it will have no way to make sense of real data at inference time, so it'll still just be noise, so you're better off just excluding it altogether right?

The exception to this would be if you're doing retraining from a checkpoint that was trained on real data, in which case keeping it as all 0s might actually be desirable since (I think, I need to double check the math on this) it will make the gradients with respect to the first convolution kernel applied to this channel to be 0, so this kernel won't be updated on bad data.

[alecgunny]

Yeah so you can confirm this by doing something like:

import torch

# create a dummy input tensor and set one of
# the channels to 0
x = torch.randn((1, 8, 32))
x[:, 4] = 0

# do a couple convolutions and create a
# dummy loss function on their output.
# all we care about is how the gradients
# flow back to the first convolutional layer
conv1 = torch.nn.Conv1d(8, 4, 8)
conv2 = torch.nn.Conv1d(4, 1, 4)
y = conv2(conv1(x))
loss = y.mean()

# make sure that the gradients with respect
# to the channel of the first layer's convolutional
# kernel that operated on the 0s channel is 0
loss.backward()
assert (conv1.weight.grad[:, 4].numpy() == 0).all()

However, the gradient of the loss wrt to that channel in the input tensor won't be 0, so the other channels of the convolutional kernel (and in fact all the weights in the network) will be trained conditioning on that 0 data. Since it's not really an option to add or remove channels on retraining, since that will fundamentally alter the architecture of the network, the question is do we train the network on:

• The all 0s channel, using all 0s at inference time for when a channel becomes defective. This has the advantage of not updating the weights of the corresponding convolutional filter and giving the network some way of identifying "bad data" at inference time, but has the disadvantage that in fitting this bad data, the network might sacrifice performance on good data, which might start coming in at inference time if this channel comes back online.
• The 0s channel filled with their mean from the previous training, similarly filling with this mean at inference time if a channel is all 0s (or should we use some flag for this since it might be possible that a channel is just legimitely all 0s)? This has the advantage that it might help mitigate the loss of performance on "good" data by using a value that has been legitimately observed (though it's not immediately obvious to me to what extent), but suffers from the drawback that we'd have to either accept that the corresponding convolutional filter is going to get updated or come up with a gradient masking scheme to avoid this. It also precludes a way from indicating to the network at inference time that a channel has gone bad (it's just seeing the mean, which it's used to seeing).

Either way I think in practice if we're retraining, we'll just need to detect if a channel's values are all 0 and if so

• avoid attempting to normalize this channel
• avoid updating the preprocessing statistics used to normalize the channel at inference time

To keep the client pipeline from having to keep up with all of this logic, it probably makes sense at this point to move the normalization and denormalization steps to the server by implementing them as torch models which act before and after the model itself (but between the snapshot and aggregation models). I'll create a new issue for this