Robot Lab - Stereo Vision
I spent most of the class working with stereo vision algorithms in OpenCV. This page describes the work done, along with various notes.
While any pair of decent webcams should give decent results, we used the fairly low priced Minoru 3D webcam. The biggest advantage is the compact casing providing a consistent distance between the two cameras. With a modern Linux distribution (tested in Ubuntu 11.10, Ubuntu 12.04, and Arch Linux), these cameras simply show up as two video devices (e.g. /dev/video0, /dev/video1).
The Minoru cameras (at least in Linux) support two video capture modes:
- 320 x 240 at 30 FPS
- 640 x 480 at 15 FPS
OpenCV (as of version 2.4.1) contains three stereo vision algorithms StereoBM, StereoSGBM, and StereoVar. Most of my time was spent writing a GUI for experimenting with these algorithms. Since they each take a large amount of parameters, I felt that real-time feedback was important.
You can find my code on GitHub, and a short video showing its use. It currently only supports StereoBM and StereoSGBM, since StereoVar does not have a Python wrapper. I have started work in manually creating a wrapper for StereoVar, but it is currently incomplete.
As the video shows I was not able to get very good results with the two algorithms tested. I'm convinced that I'm either doing something wrong in calibration (I don't think this is the case), or that I am using bad parameters (more likely). I'm not sure what else to currently other than explore the parameters more. Hopefully the third algorithm (StereoVar) would work better.
I have code to export the depth information as a point cloud to either ROS or MeshLab, which is not integrated into the code yet. This seems less useful unless I'm able to get better results out of the algorithms.
Another thing I am still working on is trying to wrap the StereoVar algorithm in Python to include in the GUI. I have basic code written using Cython, but it is still incomplete.
- OpenCV does not currently (as of version 2.4.1) support changing the frame rate from live video devices. Therefore, you must record from the Minoru cameras at 320x240.
- To synchronize frames from two cameras in OpenCV, use the "grab" function, followed by the "retrieve" function, instead of "read".
- You cannot use the OpenCV highgui function imshow (or any function dealing with GUI stuff), while the gtk3 module is loaded. You will get symbol conflicts.
- Drawing OpenCV images (stored in Python as an NumPy ndarray) to a Cairo surface (used by GTK to draw widgets), requires the image to have an alpha channel. You can insert an opaque alpha channel to a numpy image with:
image_with_alpha = numpy.insert(image, 3, 255, 2)
- To use threads with PyGObject, first call before main: