The multi-camera calibration module is designed to calibrate multiple cameras and generate the transformation matrices required to create a top-down view of a scene. By leveraging OpenCV and ChArUco boards, this module ensures precise stitching of images and enables accurate distance measurements.
Originally developed for the Hanze team’s participation in the Self Driving Challenge 2024, this project has since been generalized for broader applications.
- How does it work?
- Contributing
- License
- References
To calibrate the cameras, we need to capture images of the ChArUco board from all cameras. It is extremely important that the ChArUco board is clearly visible in all images. A ChArUco board is a chessboard with ArUco markers on it. The ArUco markers allow us to tell which corner is which, making it extremely useful for stitching images together.
# Import OpenCV
import cv2
# Initialize the cameras.
cam_left = cv2.VideoCapture(0)
cam_center = cv2.VideoCapture(1)
cam_right = cv2.VideoCapture(2)
# Get the frames of each camera.
ret, frame_left = cam_left.read()
ret, frame_center = cam_center.read()
ret, frame_right = cam_right.read()
Next up is locating the corners of the ChArUco board in all images. OpenCV has a built-in module for dealing with ArUco markers, called cv2.aruco. We can use this module to find the corners of the ChArUco board in each image.
The next step is to find the homography between the center camera and the other cameras. A homography is a transformation matrix that maps points from one image to another. We can use the homography to make the other cameras have the same perspective as the center camera. This is used to make one big image from all the cameras, giving us more details of the road.
To stitch the images together, we need to find the offsets between the leftmost camera and the other cameras. This is required to ensure that when we combine them, they blend seamlessly and look like one big image.
The bottom part of the stitched image contains the kart and parts of the road that are not relevant. To declutter the image and speed up processing, we can safely crop the bottom part of the image.
In order to avoid the Pac-Man Effect, we need to find the vanishing point of the road and crop away everything above it. We can do this by finding the intersection of the ChArUco board's lines.
Note
Pac-Man Effect: The belief that someone attempting to go over the edge of the flat Earth would teleport to the other side.
Table 1: On the left is an image illustrating the calculation of the vanishing point, while on the right is the outcome after cropping.
| Before Cropping | After Cropping |
|---|---|
 |
 |
To get a top-down view of the road, we need to find the homography between the real-world grid of the ChArUco board and the ChArUco board that we see in the image. A top-down view is useful for path planning, calculating distances, and the curvature of the road.
Table 2: On the left is an image showing the homography between ChArUco board points in the image and the real world. On the right is the resulting top-down view.
| Before Transform | After Transform |
|---|---|
 |
 |
If the ChArUco board is not perfectly parallel to the kart, the top-down image will be slightly rotated (or heavily, depending on how good you are at placing the ChArUco board). We can correct this by finding the angle of the center camera and rotating the top-down image. We can simply modify the homography matrix to rotate the image.
The final step is to calculate the region of interest. The top-down image is very large and contains a lot of unnecessary information. We don't need the entire image; instead, we only want to look ahead a certain amount of meters and a certain width.
We know the real-world size of a square on the ChArUco board, which gives us the pixel-to-meter ratio. We can modify the homography matrix to only show a certain region, e.g., 10 meters ahead and 5 meters on each side.
Contributions, bug reports, and feature requests are welcome! If you'd like to contribute, please follow these steps:
- Fork the repository.
- Create a new branch for your changes.
- Make your changes and commit them.
- Push your changes to your fork.
- Submit a pull request with a description of your changes.
Even if you're not a developer, you can still support the project in other ways. If you find this project useful, consider showing your appreciation by donating to my Ko-fi page.
This project is licensed under the MIT License - see the LICENSE file for details.
- Medium: Multi-View Image Stitching based on the pre-calibrated camera homographies
- OpenCV: ArUco Marker Detection
- OpenCV: Camera Calibration and 3D Reconstruction
- OpenCV: Camera Calibration using ChArUco Boards
- OpenCV: Documentation
- StackOverflow: Bird's eye view perspective transformation from camera calibration opencv python
- Wikipedia: Rotation Matrix
- Wikipedia: Vanishing Point