This is a project in the subject "Advanced Deep Learning for Computer Vision".
The goal is the prediction of rainy areas from radar images with deep generative networks.
Weather data source: Link
(Note: this server is continuously updated and contains the files for measurements of the past 48 hours. Older data is deleted.
See details at: Link (first file, bottom of page 4))
The files contain rain measurements converted to unitless pixel intensities (0-255). To get back the values in mm/h, use the following formulas: 1 (page 10), 2.
The radar maps are recorded with 5 minutes frequency. The maps are uniformly masked and cover a 900km x 900km area over Germany and some adjacent areas. Spatial resolution is 1km * 1km per pixel. The data is stored in binary files and once downloaded, it can be loaded with the src.get_data() method. We use the radar maps to randomly crop 64 X 64 pixel frame sequences and validate them to prevent having masked parts, empty and overly saturated frames. This dataset is then used as input for the networks.
The rough idea of our approach is depicted below.
The generator is a U-net and it gets a sequence of consecutive frames and outputs the next frame at timestamp t+1. The first (spatial) discriminator gets the whole sequence as input with the prediction or ground truth appended to it. The second (temporal) discriminator uses optical flow frames precalculated with the continuity equation for each pixel neighborhood (Lucas-Kanade method). In order to warp the last frame of the input into a "physical" estimation of the next frame, the second order discretized advection (warp) equation is solved, with the optical flow images used as the velocity field:
This is then appended to the prediction/ground truth and fed into the discriminator as input. The two discriminators only differ in their inputs but have identical architecture.