Video Files
Capture: Video Files
Namespace
Emgu.CV
Emgu.CV.Capture
References
EMGU Capture Reference
EMGU VideWriter Reference
OpenCV Use Reference
Downloads
Example
The following example shows the use of the Capture function within EMGU. The function of this library is to allow video streaming for web camera type devices and video files. This example will shows an example of reading and viewing video files, and recoding video files from a capture device using the VideWriter class. A video capture example is available here for the acquisition from devices.
Software
Pre-Requisites
The code provided should run straight out of the Emgu.Example folder (V2.4.2), extract it to this location. If the code fails to execute re-reference the EMGU libraries and include the required opencv dlls in the bin directory. Note that the project is set to build to the output path "..\..\..\bin\" you may wish to change this if you don't extract to the EMGU.Example folder.
The video record function will require a webcam or alternative capture device. It will acquire from the Os default device. Some additional video codecs may be required if they are not already installed on your system.
EMGU Coding Level: While the coding is not advanced the rated level for this example is Intermediate. This is not designed as a full on tutorial and general knowledge of the EMGU is expected. While the coding is basic the are several methods involved that may be of putting the newcomers of EMGU.
The Code
The code provided in this sample is basic there is little or no error checking. Support is available through the Forums but please try and examine the code before saying it doesn't work for you. The code is not optimised instead is better formatted to provided an understanding of the stages involved.
As the example demonstrates two methods, both viewing and recording video, some of the methods have two operations within them. They are controlled by setting VideoMethod enum called CurrentState. The methods are discussed individually in the order they appear in the code bellow.
The Code: Variables
An Open and Save file dialogue are globally declared, this is just so the file name for either saving or reading of the video can be accessed by other threads if needed.
OpenFileDialog OF = new OpenFileDialog();
SaveFileDialog SF = new SaveFileDialog();
The current method of viewing/recording videos is checked using the following variables. The two boolean variables playstate and recordstate are used for changing the button image and action depending on the program operation they record the play/pause/run state of the recording and viewing process. The enumerator CurrentState is used to classify the current state either 'Viewing' or 'Recording', this allows the program to operate according to reading or writing video files.
//current video mode and state
bool playstate = false;
bool recordstate = false;
VideoMethod CurrentState = VideoMethod.Viewing; //default state
public enum VideoMethod
{
Viewing,
Recording
};
The following variables and Class initialiser variables (VW,SW, and _Capture) deal with the capture device/video, storing its variables, and for writing video files. The _Capture variable is used in two ways depending on the program operation. If viewing videos it is initialised using the Capture(string 'video file name') method. If recording a video it is initialised using the default constructor Capture() setting up the variable to capture from the default video acquisition device installed on the system. To enable selection of device see the Capture: Camera example for reference.
The VideoWriter variable VW is used to set up a video writer when recording a video from the acquisition device. Caution must be used when using the class as even when correctly disposing of the object video index buffers can still become corrupt. Using uncompressed video formats and the re-encoding the video later is the most stable method of use. Other variables are used for storing and displaying of video information.
//Capture Setting and variables
Capture _Capture;
double FrameRate = 0;
double TotalFrames = 0;
VideoWriter VW;
Stopwatch SW;
int Frame_width;
int Frame_Height;
int FrameCount;
The Code: Methods
The Form1() method initialises the forms controls as default and no extra code is included here. TheProcessFrame() method has two sections, separated with an two if statements looking at the CurrentState of the program, either viewing or recording videos.
Viewing videos (VideoMethod.Viewing) is a fairly simple in it's application firstly it shows the frame and then updates the forms controls, labels and trackbar information. This is all done using thread safe calls to methods that include delegate operations if the controls need invoking. As the process frame is called from an independent thread set up in the Capture variable an control invoke is required to prevent cross thread interference. Then the ProcessFrame() thread is told to sleep for a specific period of time. This allows the video to be displayed at a correct frame rate. If this is removed the video will play as fast as the computer can process each frame. In this example delay is dependant on the frame rate that the video was encoded with, at least what the codec says it was. This is not always truthful and sometimes the video will play faster than what is expected. Additional check can be introduced by checking the total time of the video and the total number of frames and setting a delay according to that. These variables are available through the GetCaptureProperty() method call. Although to get the final time stamp you will have to set the Capture variable to the final frame before reading the time stamp and then reset it to the starting frame. More information on the GetCaptureProperty() method call is available here.
Finally a check is done to see if the video has played all the way through, this is a relatively easy comparison of checking the current frame and seeing if it's the same number as the total number of frames. If the video has played all the way through an automated button method call is made to stop the video and the video is rewound to the beginning by setting the frame number using the SetCaptureProperty() method call.
if (CurrentState == VideoMethod.Viewing)
{
try
{
//Show image
DisplayImage(_Capture.RetrieveBgrFrame().ToBitmap());
//Show time stamp
double time_index = _Capture.GetCaptureProperty(Emgu.CV.CvEnum.CAP_PROP.CV_CAP_PROP_POS_MSEC);
UpdateTextBox("Time: " + TimeSpan.FromMilliseconds(time_index).ToString(), Time_Label);
//show frame number
double framenumber = _Capture.GetCaptureProperty(Emgu.CV.CvEnum.CAP_PROP.CV_CAP_PROP_POS_FRAMES);
UpdateTextBox("Frame: " + framenumber.ToString(), Frame_lbl);
//update trackbar
UpdateVideo_CNTRL(framenumber);
/*Note: We can increase or decrease this delay to fastforward of slow down the display rate
if we want a re-wind function we would have to use _Capture.SetCaptureProperty(Emgu.CV.CvEnum.CAP_PROP.CV_CAP_PROP_POS_FRAMES, FrameNumber*);
//and call the process frame to update the picturebox ProcessFrame(null, null);. This is more complicated.*/
//Wait to display correct framerate
Thread.Sleep((int)(1000.0 / FrameRate)); //This may result in fast playback if the codec does not tell the truth
//Lets check to see if we have reached the end of the video
//If we have lets stop the capture and video as in pause button was pressed
//and reset the video back to start
if (framenumber == TotalFrames)
{
//pause button update
play_pause_BTN_MouseUp(null, null);
framenumber = 0;
UpdateVideo_CNTRL(framenumber);
_Capture.SetCaptureProperty(Emgu.CV.CvEnum.CAP_PROP.CV_CAP_PROP_POS_FRAMES, framenumber);
//call the process frame to update the picturebox
ProcessFrame(null, null);
}
}
catch
{
}
}
Recording videos (VideoMethod.Recording) is a much smaller section of code as less information is displayed and VideoWriter is managed according to the press of the play_pause_BTN button. The captured frame is always displayed to the user, upon pressing the play_pause_BTN (now set up as a record stop button) the frames are passed to the VideoWriter. The video writer is disposed when the play_pause_BTN is pressed again this tries to provided a successful closing event for the video files so it is written correctly. This is not essential at this stage and the VideoWriter can be written to intermittently say every 3rd frame etc. but it must be disposed correctly using another event call such as the form closing event. Further video compression can be obtained by re-sizing the frame before passing it to the VideoWriter for recording to the video file.
if (CurrentState == VideoMethod.Recording)
{
Image<Bgr, Byte> frame = _Capture.RetrieveBgrFrame(); //capture to a Image variable so we can use it for writing to the VideoWriter
DisplayImage(_Capture.RetrieveBgrFrame().ToBitmap()); //Show the image
//if we wanted to compresse the image to a smaller size to save space on our video we could use
//frame.Resize(100,100, Emgu.CV.CvEnum.INTER.CV_INTER_LINEAR)
//But the VideoWriter must be set up with the correct size
if (recordstate && VW.Ptr != IntPtr.Zero)
{
VW.WriteFrame(frame); //If we are recording and videowriter is avaliable add the image to the videowriter
//Update frame number
FrameCount++;
UpdateTextBox("Frame: " + FrameCount.ToString(), Frame_lbl);
//Show time stamp or there abouts
UpdateTextBox("Time: " + TimeSpan.FromMilliseconds(SW.ElapsedMilliseconds).ToString(), Time_Label);
}
}
The Play/Pause Record/Stop button is made from a panel for ease. As with the ProcessFrame() method there are two sections, one for viewing videos and another for recording them. All of this is only done if a the _Capture is set up. For VideoMethod.Viewing (viewing video) the actions are simple, change the boolean flag operator so any other method now the state of the viewing process (Play/Pause). Start of Pause the capture variable and update the forms controls, in this case thread safe enabling/dis-enabling of the trackbar and the changing of the buttons image. The trackbar is disabled during playing of the video as the program is much more buggy when moving it while a capture event is running. While it should be possible achieving such operation is beyond the scope of this simple application.
if (CurrentState == VideoMethod.Viewing)
{
playstate = !playstate; //change playstate to the opposite
/*Update Play panel image*/
if (playstate)
{
play_pause_BTN.BackgroundImage = VideoCapture.Properties.Resources.Pause;
UpdateVideo_CNTRL(false); //disable this as it's not safe when running
//this may work in legacy call method and be cause by a cross threading issue
_Capture.Start();
}
else
{
play_pause_BTN.BackgroundImage = VideoCapture.Properties.Resources.Play;
_Capture.Pause();
UpdateVideo_CNTRL(true);
}
}
For VideoMethod.Recording (recording video) mode things are a little more involved. As when video viewing a boolean operator is used to update other methods on the current state in recording of the acquired video. If the record process is starting the frame count is reset to 0 and a stopwatch started, the stopwatch won't give us exact timestamps of the video but it will allow the user to see roughly how long they've been recording. This can be important since video files are big and they can happily populate your hard drive until you run out of space. The program also checks to see if the VideoWriter has been disposed, as the VideoWriter is disposed when a video recording has stopped to allow proper writing of the video file. If it has been disposed the user is informed and asked to select a new video file name. This is done by re-calling the rocord video menu selection method recordVideoToolStripMenuItem_Click() and passing null parameters.
If the recording is being stopped the VideoWriter is disposed in an attempt to finalise the building of the frame index and write the video file correctly. This is buggy and sometimes the frame index may become corrupt. Programs like VLC media player will cope with an issue like this where windows media player will usually spit its dummy out. Other than this the buttons image is changed and the stopwatch is stopped. The _Capture is allowed to continue in order to allows the user to see the video frames before recording again.
else if (CurrentState == VideoMethod.Recording)
{
recordstate = !recordstate; //change playstate to the opposite
/*Update Play panel image*/
if (recordstate)
{
//Set up image/varibales
play_pause_BTN.BackgroundImage = VideoCapture.Properties.Resources.Stop;
FrameCount = 0;
SW = new Stopwatch();
//check to see if we have disposed of the video before
if (VW.Ptr == IntPtr.Zero)
{
//explain to the user what's happening
MessageBox.Show("VideoWriter has been finilised, please re-initalise a video file");
//lets re-call the recordVideoToolStripMenuItem_Click to save on programing
recordVideoToolStripMenuItem_Click(null, null);
}
SW.Start();
}
else
{
//Stop recording and dispose of the VideoWriter this will finialise the video
//Some codecs don't dispose correctley use uncompressed to stop this error
//VLC video player will play videos where the index has been corrupted. http://www.videolan.org/vlc/index.html
VW.Dispose();
//set image/variable
play_pause_BTN.BackgroundImage = VideoCapture.Properties.Resources.Record;
SW.Stop();
}
}
The Video_CNTRL_MouseCaptureChanged() event is called when the user has moved the trackbar on the form. This is prevented when recording or playing videos by disabling the trackbar. Only if the user is viewing videos and the video is not playing then this control is enable for the user. The event call does a few checks to see if the _Capture variable has been set up and see if it's running. Despite the checking to see if the _Capture is running this is never the case if it is allowed then the an attempt to pause the capture and adjust the frame position is made. This is buggy since there is cross threading issues. In the programs current state only the else statement will be actuated. This sets the position of the frame in the video and calls the ProcessFrame() method in order to show the current frame to the user.
private void Video_CNTRL_MouseCaptureChanged(object sender, EventArgs e)
{
if (_Capture != null)
{
//we don't use this when running since it has an unstable call and wil cause a crash
if (_Capture.GrabProcessState == System.Threading.ThreadState.Running)
{
_Capture.Pause();
while (_Capture.GrabProcessState == System.Threading.ThreadState.Running) ;//do nothing wait for stop
_Capture.SetCaptureProperty(Emgu.CV.CvEnum.CAP_PROP.CV_CAP_PROP_POS_FRAMES, Video_CNTRL.Value);
_Capture.Start();
}
else
{
_Capture.SetCaptureProperty(Emgu.CV.CvEnum.CAP_PROP.CV_CAP_PROP_POS_FRAMES, Video_CNTRL.Value);
//call the process frame to update the picturebox
ProcessFrame(null, null);
}
}
}
The method called when an open video file menu option is selected openVideoToolStripMenuItem_Click looks long but it's actions are quite simple. After showing an OpenFileDialog the program will check to see if the _Capture variable has been initialised, if so it will dispose of the existing one. It displays the video file name in the main forms title and sets the mode of the program. It then sets up the _Capture variable to work from the video file selected and sets up an event to show the video. Information is then gathered from the video using the GetCaptureProperty() method, FrameRate and TotalFrames are self explanatory , but codec_double can lead to some confusion. The code of the video is a 'fourcc' (fourcc ref) code consisting of 4 'bytes' or 4 'characters' representing the encoding type of the video. A double is 32 bits which when divided by a bytes length(8 bits) gives you 4 bytes. The comments give both a step by step conversion process and a all in one call placing them into one line of code. This makes up a bulk of the method with the rest of the code simple updating various form controls and the image of the button.
if (_Capture != null)
{
if (_Capture.GrabProcessState == System.Threading.ThreadState.Running) _Capture.Stop(); //Stop urrent capture if running
_Capture.Dispose();//dispose of current capture
}
try
{
this.Text = "Viewing Video: " + OF.FileName; //display the viewing method and location
//set the current video state
CurrentState = VideoMethod.Viewing;
//set up new capture and get video information
_Capture = new Capture(OF.FileName);
_Capture.ImageGrabbed += ProcessFrame; //attache event call to process frames
//Get information about the video file
FrameRate = _Capture.GetCaptureProperty(Emgu.CV.CvEnum.CAP_PROP.CV_CAP_PROP_FPS);
TotalFrames = _Capture.GetCaptureProperty(Emgu.CV.CvEnum.CAP_PROP.CV_CAP_PROP_FRAME_COUNT);
//The four_cc returns a double so we must convert it
double codec_double = _Capture.GetCaptureProperty(Emgu.CV.CvEnum.CAP_PROP.CV_CAP_PROP_FOURCC);
//for more on fourcc video discriptors
/* http://www.fourcc.org/codecs.php */
//step by step
//UInt32 Udouble = Convert.ToUInt32(codec_double);
//byte[] bytes = BitConverter.GetBytes(Udouble);
//char[] char_array = System.Text.Encoding.UTF8.GetString(bytes).ToCharArray();
//string s = new string(char_array);
//or in one
string s = new string(System.Text.Encoding.UTF8.GetString(BitConverter.GetBytes(Convert.ToUInt32(codec_double))).ToCharArray());
Codec_lbl.Text = "Codec: " + s;
//set up the trackerbar
UpdateVideo_CNTRL(true); //re-enable incase it is disabled by record video
Video_CNTRL.Minimum = 0;
Video_CNTRL.Maximum = (int)TotalFrames;
//set up the button and images
play_pause_BTN.BackgroundImage = VideoCapture.Properties.Resources.Play;
playstate = false;
}
catch (NullReferenceException excpt)
{
MessageBox.Show(excpt.Message);
}
The method called when a record video file menu option is selected recordVideoToolStripMenuItem_Click is, like when the Play/Pause Record/Stop button is pressed, a bit more involved. While doing the same checks as the open video method of disposing of the _Capture variable if it exists and creating a new one a VideoWriter must also be produced. To do this the frame width produced by the camera is acquired using the GetCaptureProperty() method. These can be manually set however the frame passed to the VideoWriter using the VideoWriter.WriteFrame() must be the same size. This can be acieved using the Image.Resize()>/code> method. The frame rate can not be acquired by the GetCaptureProperty() method and must be provided directly. This is why some videos will not play at the correct speed as the value is taken literally by the codec most video programs will do additional checks to ensure this by looking at the number of frames and the length of the video. For compression codecs a fourcc code can be provided but it additional checks must be made to ensure the codec is installed on the OS. In the example a -1 reference is passed telling OpenCV to open a dialogue for user input in the codec selection.
if (_Capture != null)
{
if (_Capture.GrabProcessState == System.Threading.ThreadState.Running) _Capture.Stop(); //Stop urrent capture if running
_Capture.Dispose();//dispose of current capture
}
try
{
//record the save location
this.Text = "Saving Video: " + SF.FileName; //display the save method and location
//set the current video state
CurrentState = VideoMethod.Recording;
//set up new capture
_Capture = new Capture(); //Use the default device
_Capture.ImageGrabbed += ProcessFrame; //attach event call to process frames
//get/set the capture video information
Frame_width = (int)_Capture.GetCaptureProperty(Emgu.CV.CvEnum.CAP_PROP.CV_CAP_PROP_FRAME_WIDTH);
Frame_Height = (int)_Capture.GetCaptureProperty(Emgu.CV.CvEnum.CAP_PROP.CV_CAP_PROP_FRAME_HEIGHT);
FrameRate = 15; //Set the framerate manually as a camera would retun 0 if we use GetCaptureProperty()
//Set up a video writer component
/* ---USE----
/* VideoWriter(string fileName, int compressionCode, int fps, int width, int height, bool isColor)
*
* Compression code.
* Usually computed using CvInvoke.CV_FOURCC. On windows use -1 to open a codec selection dialog.
* On Linux, use CvInvoke.CV_FOURCC('I', 'Y', 'U', 'V') for default codec for the specific file name.
*
* Compression code.
* -1: allows the user to choose the codec from a dialog at runtime
* 0: creates an uncompressed AVI file (the filename must have a .avi extension)
*
* isColor.
* true if this is a color video, false otherwise
*/
VW = new VideoWriter(@SF.FileName, -1, (int)FrameRate, Frame_width, Frame_Height, true);
//set up the trackerbar
UpdateVideo_CNTRL(false);//disable the trackbar
//set up the button and images
play_pause_BTN.BackgroundImage = VideoCapture.Properties.Resources.Record;
recordstate = false;
//Start aquring from the webcam
_Capture.Start();
}
catch (NullReferenceException excpt)
{
MessageBox.Show(excpt.Message);
}
The rest of the methods at the end following the /*ThreadSafe Operations*/ are simply thread safe methods to update form controls from other threads. This is done through calling a method delegate when a form requires an invocation as the thread accessing it is not it's parent. This stops multiple threads writing to controls at the same time and prevents cross thread interference. Following those is the form closing events which ensure that the _Capture variable is disposed of and the acquisition device is released if still active.
Methods Available
Used
- Capture()
- Capture(String)
- Dispose()
- RetrieveBgrFrame()
- GetCaptureProperty(CAP_PROP)
- SetCaptureProperty(CAP_PROP, Double)
Unused
- Capture()
- Start()
- Stop()
- Wait()
- RetrieveGrayFrame()
- RetrieveBgrFrame(Int32)
- RetrieveGrayFrame(Int32)
- QueryFrame() Legacy Method
- QueryGrayFrame() Legacy Method
- QuerySmallFrame() Legacy Method
- DuplexQueryFrame() WCF dependant
- DuplexQuerySmallFrame() WCF dependant
- Equals(Object)
- GetHashCode()
- GetType()
- MemberwiseClone()
- ToString()
Bugs
- Due to the video files being read in a thread accessing the
SetCaptureProperty() can cause issues. This has been controlled in the released version however is likely to become the cause of random program crashes if a user removes/adds some of the functionality.