Dataset

Dataset and split list:

Name	Setting
Blockworld	Dynamic
	Static
	Textured
MNIST World	Dynamic partially observed
	Static partially observed
	Dynamic fully observed
	Dynamic fully observed, no self motion

Download a specific subset from huggingface using the download script, see there for more options. For instance, to download just the blockworld tex validation split, run the following:

bash ./download_datasets.sh --dataset blockworld --configs tex --splits validation --download-workers 16 --extract-workers 16

Adding a New Video Dataset

To add a new video dataset, structure the dataset directory as follows:

data/
├── {dataset_name}/
│   ├── training/
│   │   ├── video_xxx.mp4
│   │   ├── ...
│   ├── validation/
│   │   ├── video_xxx.mp4
│   │   ├── ...
│   ├── metadata/
│   │   ├── training.pt
│   │   ├── validation.pt
│   │   ├── ...

Next, implement the dataset class in datasets/video/{dataset_name}.py, inheriting BaseAdvancedVideoDataset, then override the necessary methods. Additionally, define the dataset configuration in configurations/dataset/{dataset_name}.yaml. While having all three splits (training, validation, test) is not required, each split folder can be structured arbitrarily as long as the metadata files correctly reference the video files.

Please see the example in blockworld.py for more guidance. If training DFoT or DFoT-SSM, you may need to train a new VAE as well; instructions for that are below. Fill in _ALL_SPLITS, and the metadata construction function in order to use it. You may also choose not to use the metadata structure.