Stereo Imaging

==Stereo Imaging== (Under Construction)

Namespace

Emgu.CV.CameraCalibration

References

EMGU Reference
EMGU Camera Calibration OpenCV

Downloads

[Source Code V1.0]

Example

The following example shows the use of the stereo calibration function within EMGU to produce a matched stereo camera pair for 3D re-construction. This library calibrates the FOV of two cameras and uses their intrinsics to calculate the alignment between the paired images. The following example shows the the use of the CameraCalibration class in 3D re-construction of a pair of images from web cameras.

Software To be uploaded

Ideal Settings

Pre-Requisites

The code provided should run straight out of the Emgu.Example folder (V2.4.2), extract it to this location. If the code fails to execute re-reference the EMGU libraries and include the required opencv dlls in the bin directory. Note that the project is set to build to the output path "..\..\..\bin\" you may wish to change this if you don't extract to the EMGU.Example folder.

Two cameras are required for this application. They don't have to be the same (although this is suggested), if not additional frame adjustments may be required to produce frames of the same size. Matched camera pairs will produce the best results as the will have similar CCD size and lenses.

To run 3D re-construction calibration of the cameras must be done first you will require a chess board to do this. There is one available here (,pdf) (or png here), print this out and place it onto a flat surface. For more information on simple camera calibration see here.

EMGU Coding Level: The rated level for this example is Advanced. This is not designed as a beginners tutorial and general knowledge of the EMGU video capture example is expected. While comments are provided in the code you may find some basic detail missed.

The Code

The code provided in this sample is basic there is little or no error checking. Support is available through the Forums but please try and examine the code before saying it doesn't work for you. The code is not optimised instead is better formatted to provided an understanding of the stages involved.

The Code: Variables

Setting of the chessboard size is achieved using width and heigh variables try playing with these and with a larger chessboard to see what effects they have on calibration. The Bgr array line_colour_array is used to draw the detected chessboard corners on the image.

const int width = 9;//9 //width of chessboard no. squares in width - 1
const int height = 6;//6 // heght of chess board no. squares in heigth - 1
Size patternSize = new Size(width, height); //size of chess board to be detected
Bgr[] line_colour_array = new Bgr[width * height]; // just for displaying coloured lines of detected chessboard

These buffers store the relevant calibration information acquired from FindChessboardCorners(), they are set to 100 by default using the static int buffer_length. If you are running a slower system reduce this before executing the program.

MCvPoint3D32f[][] corners_object_Points = new MCvPoint3D32f[buffer_length][]; //stores the calculated size for the chessboard
PointF[][] corners_points_Left = new PointF[buffer_length][];//stores the calculated points from chessboard detection Camera 1
PointF[][] corners_points_Right = new PointF[buffer_length][];//stores the calculated points from chessboard detection Camera 2

The intrinsic parameters used to map the pair of images together are stored in IntrinsicCam1 and IntrinsicCam2 . Depending on the calibration method used CALIB_TYPE some parameters of these intrinsics will need setting up before being used (see here.

IntrinsicCameraParameters IntrinsicCam1 = new IntrinsicCameraParameters();
IntrinsicCameraParameters IntrinsicCam2 = new IntrinsicCameraParameters();
ExtrinsicCameraParameters EX_Param;

To apply the StereoCalibrate a Q map is generated using CvInvoke.cvStereoRectify(). The output from his algorithm is stored in the following variables. We only use the Q matrix in producing the disparity map.

Matrix<double> fundamental; //fundemental output matrix for StereoCalibrate
Matrix<double> essential; //essential output matrix for StereoCalibrate
Rectangle Rec1 = new Rectangle(); //Rectangle Calibrated in camera 1
Rectangle Rec2 = new Rectangle(); //Rectangle Caliubrated in camera 2
Matrix<double> Q = new Matrix<double>(4, 4); //This is what were interested in the disparity-to-depth mapping matrix
Matrix<double> R1 = new Matrix<double>(3, 3); //rectification transforms (rotation matrices) for Camera 1.
Matrix<double> R2 = new Matrix<double>(3, 3); //rectification transforms (rotation matrices) for Camera 1.
Matrix<double> P1 = new Matrix<double>(3, 4); //projection matrices in the new (rectified) coordinate systems for Camera 1.
Matrix<double> P2 = new Matrix<double>(3, 4); //projection matrices in the new (rectified) coordinate systems for Camera 2.
private MCvPoint3D32f[] _points; //Computer3DPointsFromStereoPair

The Code: Methods

There are numerous methods used in the code only the most relevant will be discussed. This is an advanced coding example, additional comments are available in the code.

public Form1()

Here the Bgr buffer is filled to display the detected chessboard corners, the same array is used to draw both arrays on there associative image. An example of how looking up camera

Methods Available

Used

• CalibrateCamera(Point3D<Single>[][], Point2D<Single>[][], MCvSize, IntrinsicCameraParameters, CALIB_TYPE, ExtrinsicCameraParameters[])
• FindChessboardCorners(Image<Gray, Byte>, MCvSize, CALIB_CB_TYPE, Point2D<Single>[])
• StereoCalibrate(Point3D<Single>[][], Point2D<Single>[][], Point2D<Single>[][], IntrinsicCameraParameters, IntrinsicCameraParameters, MCvSize, CALIB_TYPE, MCvTermCriteria, ExtrinsicCameraParameters, Matrix<Single>, Matrix<Single>)

Unused

• DrawChessboardCorners(Image<Gray, Byte>, MCvSize, Point2D<Single>[], Boolean)
• FindExtrinsicCameraParams2(Point3D<Single>[], Point2D<Single>[], IntrinsicCameraParameters)
• FindHomography(Matrix<Single>, Matrix<Single>)
• FindHomography(Matrix<Single>, Matrix<Single>, Double)
• FindHomography(Matrix<Single>, Matrix<Single>, HOMOGRAPHY_METHOD, Double)
• FindHomography(Point2D<Single>[], Point2D<Single>[], HOMOGRAPHY_METHOD, Double)
• ProjectPoints2(Point3D<Single>[], ExtrinsicCameraParameters, IntrinsicCameraParameters, Matrix<Single>[])
• Undistort2<C, D>(Image<C, D>, IntrinsicCameraParameters)

Extra Methods

• StereoSGBM(int minDisparity, int numDisparities, int SADWindowSize, int P1, int P2, int disp12MaxDiff, int preFilterCap, int uniquenessRatio, int speckleWindowSize, int speckleRange, bool fullDP);
• MCvPoint3D32f[] ReprojectImageTo3D(Image<Gray, short> disparity, Matrix<double> Q);
• cvStereoRectify(IntPtr cameraMatrix1, IntPtr cameraMatrix2, IntPtr distCoeffs1, IntPtr distCoeffs2, MCvSize imageSize, IntPtr R, IntPtr T, IntPtr R1, IntPtr R2, IntPtr P1, IntPtr P2, IntPtr Q, STEREO_RECTIFY_TYPE flags)

Bugs

1. There is a refresh problem when displaying the disparity map image in some cases. This is dues to the demand on processing the main images.