Working with Images

Emgu CV 3.x

Open CV 3.0 has been changed, the C interface that use IplImage has been slowly phased out and the C++ interface that utilize Mat is recommended in this release. In 3.0 release, Emgu CV has adapted to use the Mat class as a result. The Image<,> class is still available in this release for backward compatibility reason. If you are interest in using Image<,> class, you can checkout the Emgu CV 2.x version of this tutorial.

Creating Image

To Create a 3 channel image of 400x200, you can use the following code.

Mat img = new Mat(200, 400, DepthType.Cv8U, 3);

An empty Mat can also be created by calling

Mat img = new Mat();

which should be used when allocating a Mat to store the results of CvInvoke image processing routines. e.g.

Mat invert = new Mat(); 
CvInvoke.BitwiseNot(inputImage, invert);

Unlike the Image<,> class, where you will need to pre-allocate memory with the correct size before passing it as an IOutputArray, when an empty Mat is passed as an IOutputArray, Open CV will automatically allocate memory for the Mat. You can also load an image from file using the CvInvoke.Imread function:

Mat img = CvInvoke.Imread("myimage.jpg", CvEnum.ImreadModes.AnyColor);

Accessing the pixels from Mat

Unlike the Image<,> class, where memory are pre-allocated and fixed, the memory of Mat can be automatically re-allocated by Open CV function calls. We cannot pre-allocate managed memory and assume the same memory are used through the life time of the Mat object. As a result, Mat class do not contains a Data property like the Image<,> class, where the pixels can be access through a managed array. To access the data of the Mat, there are a few possible choices.

The easy way and safe way that cost an additional memory copy

The first option is to copy the Mat to an Image<,> object using the Mat.ToImage function. e.g.

Image<Bgr, Byte> img = mat.ToImage<Bgr, Byte>();

The pixel data can then be accessed using the Image<,>.Data property.

You can also convert the Mat to an Matrix<> object. Assuming the Mat contains 8-bit data,

Matrix<Byte> matrix = new Matrix<Byte>(mat.Rows, mat.Cols, mat.NumberOfChannels);
mat.CopyTo(matrix);

Note that you should create Matrix<> with a matching type to the Mat object. If the Mat contains 32-bit floating point value, you should replace Matrix<Byte> in the above code with Matrix<float>. The pixel data can then be accessed using the Matrix<>.Data property.

The fastest way with no memory copy required. Be caution!!!

The second option is a little bit tricky, but will provide the best performance. This will usually require you to know the size of the Mat object before it is created. So you can allocate managed data array, and create the Mat object by forcing it to use the pinned managed memory. e.g.

//load your 3 channel bgr image here
Mat m1 = ...; 

//3 channel bgr image data, if it is single channel, the size should be m1.Width * m1.Height
byte[] data = new byte[m1.Width * m1.Height * 3]; 

GCHandle handle = GCHandle.Alloc(data, GCHandleType.Pinned);
using (Mat m2 = new Mat(m1.Size, DepthType.Cv8U, 3, handle.AddrOfPinnedObject(), m1.Width * 3))
    CvInvoke.BitwiseNot(m1, m2);
handle.Free();

At this point the data array contains the pixel data of the inverted image. Note that if the Mat m2 was allocated with the wrong size, data[] array will contains all 0s, and no exception will be thrown. So be really careful when performing the above operations.

Emgu CV 2.x

Depth and Color as Generic Parameter

An Image is defined by its generic parameters: color and depth. To create a 8bit unsigned Grayscale image, in Emgu CV it is done by calling

Image<Gray, Byte> image = new Image<Gray, Byte>( width, height);

Not only this syntax make you aware the color and the depth of the image, it also restrict the way you use functions and capture errors in compile time. For example, the SetValue(TColor color, Image<Gray, Byte> mask) function in Image<TColor, TDepth> class (version >= 1.2.2.0) will only accept colors of the same type, and mask has to be an 8-bit unsigned grayscale image. Any attempts to use a 16-bit floating point or non-grayscale image as a mask will results a compile time error!

Creating Image

Although it is possible to create image by calling CvInvoke.cvCreateImage, it is suggested to construct a Image<TColor, TDepth> object instead. There are several advantages using the managed Image<TColor, TDepth> class

• Memory is automatically released by the garbage collector
• Image<TColor, TDepth> class can be examined by Debugger Visualizer
• Image<TColor, TDepth> class contains advanced method that is not available on OpenCV, for example, generic operation on image pixels, conversion to Bitmap etc.

Image Color

The first generic parameter of the Image class specific the color of the image type. For example

Image<Gray, ...> img1;

indicates that img1 is a single channel grayscale image.

Color Types supported in Emgu CV 1.3.0.0 includes:

• Gray
• Bgr (Blue Green Red)
• Bgra (Blue Green Red Alpha)
• Hsv (Hue Saturation Value)
• Hls (Hue Lightness Saturation)
• Lab (CIE L*a*b*)
• Luv (CIE L*u*v*)
• Xyz (CIE XYZ.Rec 709 with D65 white point)
• Ycc (YCrCb JPEG)

Image Depth

Image Depth is specified using the second generic parameter Depth. The types of depth supported in Emgu CV 1.4.0.0 include

• Byte
• SByte
• Single (float)
• Double
• UInt16
• Int16
• Int32 (int)

Creating a new image

To create an 480x320 image of Bgr color and 8-bit unsigned depth. The code in C# would be

 Image<Bgr, Byte> img1 = new Image<Bgr, Byte>(480, 320);

If you wants to specify the background value of the image, let's say in Blue. The code in C# would be

 Image<Bgr, Byte> img1 = new Image<Bgr, Byte>(480, 320, new Bgr(255, 0, 0));

Reading image from file

Creating image from file is also simple. If the image file is "MyImage.jpg", in C# it is

 Image<Bgr, Byte> img1 = new Image<Bgr, Byte>("MyImage.jpg");

Creating image from Bitmap

It is also possible to create an Image<TColor, TDepth> from a .Net Bitmap object. The code in C# would be

 Image<Bgr, Byte> img = new Image<Bgr, Byte>(bmp); //where bmp is a Bitmap

Automatic Garbage Collection

The Image<TColor, TDepth> class automatically take care of the memory management and garbage collection.

Once the garbage collector decided that there is no more reference to the Image<TColor, TDepth> object, it will call the Disposed method, which release the unmanaged IplImage structure.

The time of when garbage collector decides to dispose the image is not guaranteed. When working with large image, it is recommend to call the Dispose() method to explicitly release the object. Alternatively, use the using keyword in C# to limit the scope of the image

using (Image<Gray, Single> image = new Image<Gray, Single>(1000, 800))
{
   ... //do something here in the image
} //The image will be disposed here and memory freed

Getting or Setting Pixels

The safe (slow) way

• Suppose you are working on an Image<Bgr, Byte>. You can obtain the pixel on the y-th row and x-th column by calling

Bgr color = img[y, x];

• Setting the pixel on the y-th row and x-th column is also simple

img[y,x] = color;

The fast way

The Image pixels values are stored in the Data property, a 3D array. Use this property if you need to iterate through the pixel values of the image.

For example, in an grayscale image (Image<Gray, Byte>), instead of doing this:

Gray byCurrent = imageGray[x, y];

You would do this:

Byte byCurrent = imageGray.Data[x, y, 0];

Methods

Naming Convention

• Method XYZ in Image<TColor, TDepth> class corresponds to the OpenCV function cvXYZ. For example, Image<TColor, TDepth>.Not() function corresponds to cvNot function with the resulting image being returned.
• Method _XYZ is usually the same as Method XYZ except that the operation is performed inplace rather than returning a value. For example, Image<TColor, TDepth>._Not() function performs the bit-wise inversion inplace.

Operators Overload

The operators + - * / has been overloaded (version > 1.2.2.0) such that it is perfectly legal to write codes like:

Image<Gray, Byte> image3 = (image1 + image2 - 2.0) * 0.5;

Generic Operation

One of the advantage of using Emgu CV is the ability to perform generic operations.

It's best if I demonstrate this with an example. Suppose we have an grayscale image of bytes

 Image<Gray, Byte> img1 = new Image<Gray, Byte>(400, 300, new Gray(30));

To invert all the pixels in this image we can call the Not function

 Image<Gray, Byte> img2 = img1.Not();

As an alternative, we can also use the generic method Convert available from the Image<TColor, TDepth> class

 Image<Gray, Byte> img3 = img1.Convert<Byte>( delegate(Byte b) { return (Byte) (255-b); } );

The resulting image img2 and img3 contains the same value for each pixel.

At first glance it wouldn't seems to be a big gain when using generic operations. In fact, since OpenCV already has an implementation of the Not function and performance-wise it is better than the generic version of the equivalent Convert function call. However, there comes to cases when generic functions provide the flexibility with only minor performance penalty.

Let's say you have an Image<Gray, Byte> img1 with pixels set. You wants to create a single channel floating point image of the same size, where each pixel of the new image, correspond to the old image, described with the following delegate

 delegate(Byte b) { return (Single) Math.cos( b * b / 255.0); }

This operation can be completed as follows in Emgu CV

 Image<Gray, Single> img4 = img1.Convert<Single>( delegate(Byte b) { return (Single) Math.cos( b * b / 255.0); }  );

The syntax is simple and meaningful. On the other hand, this operation in OpenCV is hard to perform since equivalent function such as Math.cos is not available.

Drawing Objects on Image

The Draw( ) method in Image< Color, Depth> can be used to draw different types of objects, including fonts, lines, circles, rectangles, boxes, ellipses as well as contours. Use the documentation and intellisense as a guideline to discover the many functionality of the Draw function.

Color and Depth Conversion

Converting an Image<TColor, TDepth> between different colors and depths are simple. For example, if you have Image<Bgr, Byte> img1 and you wants to convert it to a grayscale image of Single, all you need to do is

 Image<Gray, Single> img2 = img1.Convert<Gray, Single>();

Displaying Image

Using ImageBox

Emgu CV recommends the use of ImageBox control for display purpose, for the following reasons

• ImageBox is a high performance control for displaying image. Whenever possible, it displays a Bitmap that shares memory with the Image object, therefore no memory copy is needed (very fast).
• The user will be able to examine the image pixel values, video frame rates, color types when the image is being displayed.
• It is convenient to perform simple image operations with just a few mouse clicks.

Converting to Bitmap

The Image class has a ToBitmap() function that return a Bitmap object, which can easily be displayed on a PictureBox control using Windows Form.

XML Serialization

Why do I care?

One of the future of Emgu CV is that Image<TColor, TDepth> can be XML serializated. You might ask why we need to serialization an Image. The answer is simple, we wants to use it in a web service!

Since the Image<TColor, TDepth> class implements ISerializable, when you work in WCF (Windows Communication Fundation), you are free to use Image<TColor, TDepth> type as parameters or return value of a web service.

This will be ideal, for example, if you are building a cluster of computers to recognize different groups of object and have a central computer to coordinate the tasks. I will also be useful if your wants to implement remote monitoring software that constantly query image from a remote server, which use the Capture class in Emgu CV to capture images from camera.

Conversion to XML

You can use the following code to convert an Image<Bgr, Byte> image to XmlDocument:

StringBuilder sb = new StringBuilder();
(new XmlSerializer(typeof(Image<Bgr, Byte>))).Serialize(new StringWriter(sb), o);
XmlDocument xDoc = new XmlDocument();
xDoc.LoadXml(sb.ToString());

Conversion from XML

You can use the following code to convert a XmlDocument xDoc to Image<Bgr,Byte>

Image<Bgr, Byte> image = (Image<Bgr, Byte>) 
(new XmlSerializer(typeof(Image<Bgr, Byte>))).Deserialize(new XmlNodeReader(xDoc));