Computational Depth-of-Field

This repository contains a ready-to-use framework for computing depth maps and applying these maps for many purposes

HOW TO RUN THE APPLICATION (LINUX/MACOS):

1. Open terminal at project directory
2. Install the required python packages with command pip install -r requirements.txt
3. (Only on Linux) Run command sudo apt install libopencv-dev to install opencv (needed for ken-burns mp4 video creation)
4. (Only on MacOs) Run command brew install opencv to install opencv (needed for ken-burns mp4 video creation)
5. Run command curl -o AdelaiDepth/LeReS/Minist_Test/res101.pth https://cloudstor.aarnet.edu.au/plus/s/lTIJF4vrvHCAI31/download to download weights of the pretrained RGB -> Depth CNN
6. Run command source build.sh to build the project
7. Run command python3 app.py to run the application

IMPLEMENTED FEATURES:

Basic features:

1. Load an RGB image from disk: click on the button "Load Image" and select the image you want to open from file system. It will be displayed in the interface. (see implementation details)
2. Allow users to scribble depth annotations in UI: once an image has been loaded, click on the button "Start Drawing". From now on, you can scribble annotations with your mouse on the displayed image (see implementation details) . If you wish to change the color of annotations, just click on the "Color" button and select the new color (see implementation details). If you wish to change the thickness of annotations, use the "brush" bar provided. Once you have finished to scribble on the image, click on "Save Image" to save the annotated image and "Save Mask" to save the mask which specifies which pixels are annotated (see implementation details).
3. Diffuse annotations across the image using Poisson image editing: once you have saved the annotated image and relative mask, click on the button "Poisson". You will be asked to select a mask from file system, a correspondent annotated image, and some parameters (number of iterations and beta). Then, the result will be automatically saved in the outputs folder. The Poisson-based implemented algorithm is the Anisotropic Diffusion, which diffuses depth annotations across the image taking edges into account. (see poisson/anisotropic diffusion implementation details)
4. Allow users to select focus depth and aperture size: you can select a focus point and a relative aperture size after clicking on the "Bilateral" button. (see implementation details)
5. Simulate depth-of-field using a spatially varying cross-bilater filter: click on the "Bilateral" button. Right after that, click on a desired focus point on the image. Then, you will be allowed to select an aperture size and a depth-map (previously generated, for instance, with poisson image editing button). The result will be automatically saved in the project/outputs folder. (see cross-bilater filter implementation details)
6. Save and display the result: The result is saved in the project/outputs folder. Click on the "Load Image" button again and select the saved result from the folder to display it in the interface.

Extended features:

1. Use a pretrained RGB -> Depth CNN to supplement the depth: click on the button "NN estimation" to call the pretrained CNN on the currently loaded image. The result will be saved in the outputs folder. (see implementation details)
2. Find a user-friendly way to combine predicted depth-map and user scribbles: Once you have created your own depth-map with poisson image editing and scribbles, you can create a merged depth map with the result of the pretrained CNN clicking on the button "Fuse Maps" (see implementation details). You will be asked to select two depth-maps and corrispective weights. The result will be saved in a folder of your choice.
3. Implement Ken-Burns effect with depth-based parallax: click on the button "Ken Burns". You will be asked to select a depth-map, the length of the video, number of fps, and a type of effect (forward or back and forth). Then, the process will start and the result will be saved in the outputs folder. (see Forward warping implementation details) (see Ken-Burns effect implementation details)

Custom features

1. Edge enhancement using a pretrained RGB -> edge map for better anisotropic diffusion: click on the button "Edge Enhance" to create a new image with enhanced edges (see implementation details). This is very useful to obtain a better anisotropic diffusion result if the original image has edges that are not marked enough.

TEST DATA AND EXAMPLE RESULTS:

You can find some test data in the folder project/images and relative results for all the steps in the folder project/results.