Welcome!
Here you can find the scripts that I wrote for my MSc thesis in Applied Sciences & Engineering: Applied Computer Science at Vrije Universiteit Brussel. I performed this MSc thesis under supervision of Prof. Dr. Wim Thiery and with help of my daily supervisor Inne Vanderkelen. The files have references to files/folders stored on Hydra, the HPC cluster of VUB and ULB. The manuscript of my MSc thesis can be found here.
In this project, we predict nightly thunderstorms over Lake Victoria from a satellite-derived product using machine and deep learning. As a proxy for thunderstorm activity we obtained the 'Overshooting Top' (OT) dataset designed by NASA's Langley Research Center, which is derived from the SEVIRI satellite product.
We predict the 1% most storm-intense nights with a hit rate of 85% and a false alarm rate of 15%. Further research will (i) expand on the deep learning application for these predictions, (ii) implement spatial predictions and (iii) use the SEVIRI images as input for our model (while keeping the OT dataset as prediction label).
CreateTuples.py
This script loops through the OT dataset and generates day-night tuples, which will be used for this project.CreateShards.py
This script loops through the day-night tuples and combines them into shards, which we can efficiently load into RAM via Webdataset.GetVariable.py
This script loops through the day-night tuples and extracts only one variable from them, and then stores them in netCDF4 files.DataLoader.py
This script divides the data into training, validation and test sets and creates Dataloader objects, which can efficiently loop through these subsets.Classifier.py
This script specifies how training, validation and testing should be performed, when weights need to be updated (only during training) and determines the logging.CNN.py
This script provides an easy way to generate Convolutional Neural Networks (CNNs) in an object-oriented manner.Trainer.py
This script determines the Trainer, which handles communication between the Dataloader, CNN and Classifier objects.If you are interested in this topic, feel free to read (parts of) my MSc thesis.
Have a nice day!