How do I capture and Process each and every frame of an image using CImg library?
<p>I'm working on a project based on real time image processing using CImg Library in Raspberrypi.</p>

<p>I need to capture images at higher frame rates (say atleast 30 fps), when I use the inbuilt Raspicam commands such as</p>

<pre><code>sudo raspistill -o -img_%d.jpg -tl 5 -t 1000  -a 512
</code></pre>

<p>/* -tl : time lapse duration in msec
    -t  : total time duration (1000 msec = 1 sec)
    -a : displays frame numbers 
*/</p>

<p>with this command though it shows 34 frames per second,I could only capture maximum of 4 frames/images (and rest of the frames are skipped) </p>

<pre><code>sudo raspistill -o -img_%d.jpg -tl 5 -tl 1000 -q 5 -md 7 -w 640 -h 480 -a 512
</code></pre>

<p>and From this above command I could capture at a maximum of 7-8 images per second but by reducing the resolution and quality of the images.</p>

<p>But I don't want to compromise on the quality of an image since I will be capturing an image, processing it immediately and will be deleting an image to save memory. </p>

<p>Later I tried using V4L2(Video for Linux) drivers to make use of the best performance of a camera, but in the internet, tutorials regarding V4l2 and cimg are quite scarce, I couldn't find one.</p>

<p>I have been using the following commands</p>

<pre><code># Capture a JPEG image
 v4l2-ctl --set-fmt-video=width=2592,height=1944,pixelformat=3
 v4l2-ctl --stream-mmap=3 --stream-count=1 –stream-to=somefile.jpg
</code></pre>

<p>(source : <a href="http://www.geeetech.com/wiki/index.php/Raspberry_Pi_Camera_Module" rel="nofollow">http://www.geeetech.com/wiki/index.php/Raspberry_Pi_Camera_Module</a>)</p>

<p>but I couldn't get enough information about those parameters such as (stream-mmap &amp; stream-count) what does it exactly, and how does these commands help me in capturing 30 frames/images per second ?</p>

<p><strong>CONDITIONS:</strong></p>

<ol>
<li><p>Most importantly I don't want to use OPENCV, MATLAB or any other image processing softwares, since my image processing task is very simple (I.e detection of led light blink) also my objective is to have a light weight tool to perform these operations at the cost of higher performance.</p></li>
<li><p>And also my programming code should be in either C or C++ but not in python or Java (since processing speed matters !)</p></li>
<li><p>Please make a note that,my aim is not to record a video but to capture as many frames as possible and to process each and individual images. </p></li>
</ol>

<p>For using in Cimg I searched over few docs from a reference manual, but I couldn't understand it clearly how to use it for my purpose.</p>

<p>The class cimg_library::CImgList represents lists of cimg_library::CImg images. It can be used for instance to store different frames of an image sequence.
(source : <a href="http://cimg.eu/reference/group__cimg__overview.html" rel="nofollow">http://cimg.eu/reference/group__cimg__overview.html</a> )</p>

<ul>
<li>I found the following examples, But i'm not quite sure whether it suits my task</li>
</ul>

<p>Load a list from a YUV image sequence file. </p>

<pre><code>CImg&lt;T&gt;&amp; load_yuv 
(
const char *const 
filename, 

const unsigned int 
size_x, 

const unsigned int 
size_y, 

const unsigned int 
first_frame = 0, 

const unsigned int 
last_frame = ~0U, 

const unsigned int 
step_frame = 1, 

const bool 
yuv2rgb = true 
</code></pre>

<p>Parameters
filename
Filename to read data from. 
size_x
Width of the images. 
size_y
Height of the images. 
first_frame
Index of first image frame to read. 
last_frame
Index of last image frame to read. 
step_frame
Step applied between each frame. 
yuv2rgb
Apply YUV to RGB transformation during reading. </p>

<p>But here, I need rgb values from an image frames directly without compression.</p>

<p>Now I have the following code in OpenCv which performs my task, but I request you to help me in implementing the same using CImg libraries (which is in C++) or any other light weight libraries or something with v4l2</p>

<pre><code>#include &lt;iostream&gt;
#include &lt;opencv2/opencv.hpp&gt;

using namespace std;
using namespace cv;

int main (){
    VideoCapture capture (0); //Since you have your device at /dev/video0

    /* You can edit the capture properties with "capture.set (property, value);" or in the driver with "v4l2-ctl --set-ctrl=auto_exposure=1"*/

    waitKey (200); //Wait 200 ms to ensure the device is open

    Mat frame; // create Matrix where the new frame will be stored
    if (capture.isOpened()){
        while (true){
            capture &gt;&gt; frame; //Put the new image in the Matrix

            imshow ("Image", frame); //function to show the image in the screen
        }
    }
}
</code></pre>

<ul>
<li>I'm a beginner to the Programming and Raspberry pi, please excuse if there are any mistakes in the above problem statements.</li>
</ul>

<p>"With some of your recommendations, I slighthly modified the raspicam c++ api code and combined with CIMG image processing functionality "</p>

<pre><code> #include "CImg.h"
    #include &lt;iostream&gt;
    #include &lt;cstdlib&gt;
    #include &lt;fstream&gt;
    #include &lt;sstream&gt;
    #include &lt;sys/timeb.h&gt;
    #include "raspicam.h"
    using namespace std;
    using namespace cimg_library;
     bool doTestSpeedOnly=false;
    size_t nFramesCaptured=100;
//parse command line
//returns the index of a command line param in argv. If not found, return -1

    int findParam ( string param,int argc,char **argv ) {
    int idx=-1;
    for ( int i=0; i&lt;argc &amp;&amp; idx==-1; i++ )
        if ( string ( argv[i] ) ==param ) idx=i;
    return idx;

}


//parse command line
//returns the value of a command line param. If not found, defvalue is returned
float getParamVal ( string param,int argc,char **argv,float defvalue=-1 ) {
    int idx=-1;
    for ( int i=0; i&lt;argc &amp;&amp; idx==-1; i++ )
        if ( string ( argv[i] ) ==param ) idx=i;

    if ( idx==-1 ) return defvalue;
    else return atof ( argv[  idx+1] );
}




raspicam::RASPICAM_EXPOSURE getExposureFromString ( string str ) {
    if ( str=="OFF" ) return raspicam::RASPICAM_EXPOSURE_OFF;
    if ( str=="AUTO" ) return raspicam::RASPICAM_EXPOSURE_AUTO;
    if ( str=="NIGHT" ) return raspicam::RASPICAM_EXPOSURE_NIGHT;
    if ( str=="NIGHTPREVIEW" ) return raspicam::RASPICAM_EXPOSURE_NIGHTPREVIEW;
    if ( str=="BACKLIGHT" ) return raspicam::RASPICAM_EXPOSURE_BACKLIGHT;
    if ( str=="SPOTLIGHT" ) return raspicam::RASPICAM_EXPOSURE_SPOTLIGHT;
    if ( str=="SPORTS" ) return raspicam::RASPICAM_EXPOSURE_SPORTS;
    if ( str=="SNOW" ) return raspicam::RASPICAM_EXPOSURE_SNOW;
    if ( str=="BEACH" ) return raspicam::RASPICAM_EXPOSURE_BEACH;
    if ( str=="VERYLONG" ) return raspicam::RASPICAM_EXPOSURE_VERYLONG;
    if ( str=="FIXEDFPS" ) return raspicam::RASPICAM_EXPOSURE_FIXEDFPS;
    if ( str=="ANTISHAKE" ) return raspicam::RASPICAM_EXPOSURE_ANTISHAKE;
    if ( str=="FIREWORKS" ) return raspicam::RASPICAM_EXPOSURE_FIREWORKS;
    return raspicam::RASPICAM_EXPOSURE_AUTO;
}


    raspicam::RASPICAM_AWB getAwbFromString ( string str ) {
    if ( str=="OFF" ) return raspicam::RASPICAM_AWB_OFF;
    if ( str=="AUTO" ) return raspicam::RASPICAM_AWB_AUTO;
    if ( str=="SUNLIGHT" ) return raspicam::RASPICAM_AWB_SUNLIGHT;
    if ( str=="CLOUDY" ) return raspicam::RASPICAM_AWB_CLOUDY;
    if ( str=="SHADE" ) return raspicam::RASPICAM_AWB_SHADE;
    if ( str=="TUNGSTEN" ) return raspicam::RASPICAM_AWB_TUNGSTEN;
    if ( str=="FLUORESCENT" ) return raspicam::RASPICAM_AWB_FLUORESCENT;
    if ( str=="INCANDESCENT" ) return raspicam::RASPICAM_AWB_INCANDESCENT;
    if ( str=="FLASH" ) return raspicam::RASPICAM_AWB_FLASH;
    if ( str=="HORIZON" ) return raspicam::RASPICAM_AWB_HORIZON;
    return raspicam::RASPICAM_AWB_AUTO;
    }


    void processCommandLine ( int argc,char **argv,raspicam::RaspiCam &amp;Camera ) {
    Camera.setWidth ( getParamVal ( "-w",argc,argv,640 ) );
    Camera.setHeight ( getParamVal ( "-h",argc,argv,480 ) );
    Camera.setBrightness ( getParamVal ( "-br",argc,argv,50 ) );
    Camera.setSharpness ( getParamVal ( "-sh",argc,argv,0 ) );
    Camera.setContrast ( getParamVal ( "-co",argc,argv,0 ) );
    Camera.setSaturation ( getParamVal ( "-sa",argc,argv,0 ) );
    Camera.setShutterSpeed( getParamVal ( "-ss",argc,argv,0 ) );
    Camera.setISO ( getParamVal ( "-iso",argc,argv ,400 ) );
   if ( findParam ( "-vs",argc,argv ) !=-1 )
        Camera.setVideoStabilization ( true );
    Camera.setExposureCompensation ( getParamVal ( "-ec",argc,argv ,0 ) );

    if ( findParam ( "-gr",argc,argv ) !=-1 )
      Camera.setFormat(raspicam::RASPICAM_FORMAT_GRAY);
    if ( findParam ( "-yuv",argc,argv ) !=-1 ) 
      Camera.setFormat(raspicam::RASPICAM_FORMAT_YUV420);
    if ( findParam ( "-test_speed",argc,argv ) !=-1 )
        doTestSpeedOnly=true;
    int idx;
    if ( ( idx=findParam ( "-ex",argc,argv ) ) !=-1 )
        Camera.setExposure ( getExposureFromString ( argv[idx+1] ) );
    if ( ( idx=findParam ( "-awb",argc,argv ) ) !=-1 )
        Camera.setAWB( getAwbFromString ( argv[idx+1] ) );

    nFramesCaptured=getParamVal("-nframes",argc,argv,100);
    Camera.setAWB_RB(getParamVal("-awb_b",argc,argv ,1), getParamVal("-awb_g",argc,argv ,1));

    }


    //timer functions
    #include &lt;sys/time.h&gt;
    #include &lt;unistd.h&gt;
    class Timer{
    private:
    struct timeval _start, _end;

    public:
      Timer(){}
    void start(){
        gettimeofday(&amp;_start, NULL);
    }
    void end(){
        gettimeofday(&amp;_end, NULL);
    }
    double getSecs(){
    return double(((_end.tv_sec  - _start.tv_sec) * 1000 + (_end.tv_usec - _start.tv_usec)/1000.0) + 0.5)/1000.;
    }

    }; 

    void saveImage ( string filepath,unsigned char *data,raspicam::RaspiCam &amp;Camera ) {
    std::ofstream outFile ( filepath.c_str(),std::ios::binary );
    if ( Camera.getFormat()==raspicam::RASPICAM_FORMAT_BGR ||  Camera.getFormat()==raspicam::RASPICAM_FORMAT_RGB ) {
        outFile&lt;&lt;"P6\n";
    } else if ( Camera.getFormat()==raspicam::RASPICAM_FORMAT_GRAY ) {
        outFile&lt;&lt;"P5\n";
    } else if ( Camera.getFormat()==raspicam::RASPICAM_FORMAT_YUV420 ) { //made up format
        outFile&lt;&lt;"P7\n";
    }
    outFile&lt;&lt;Camera.getWidth() &lt;&lt;" "&lt;&lt;Camera.getHeight() &lt;&lt;" 255\n";
    outFile.write ( ( char* ) data,Camera.getImageBufferSize() );
    }


    int main ( int argc,char **argv ) {

    int a=1,b=0,c;
    int x=444,y=129; //pixel coordinates
    raspicam::RaspiCam Camera;
    processCommandLine ( argc,argv,Camera );
    cout&lt;&lt;"Connecting to camera"&lt;&lt;endl;

    if ( !Camera.open() ) {
        cerr&lt;&lt;"Error opening camera"&lt;&lt;endl;
        return -1;
       }
     //   cout&lt;&lt;"Connected to camera ="&lt;&lt;Camera.getId() &lt;&lt;" bufs="&lt;&lt;Camera.getImageBufferSize( )&lt;&lt;endl;
    unsigned char *data=new unsigned char[  Camera.getImageBufferSize( )];
    Timer timer;


       // cout&lt;&lt;"Capturing...."&lt;&lt;endl;
       // size_t i=0;
    timer.start();


    for (int i=0;i&lt;=nFramesCaptured;i++)
        {
        Camera.grab();
        Camera.retrieve ( data );
                std::stringstream fn;
                fn&lt;&lt;"image.jpg";
                saveImage ( fn.str(),data,Camera );
    //  cerr&lt;&lt;"Saving "&lt;&lt;fn.str()&lt;&lt;endl;
    CImg&lt;float&gt; Img("/run/shm/image.jpg");
         //Img.display("Window Title");

    // 9 PIXELS MATRIX GRAYSCALE VALUES 
    float pixvalR1 = Img(x-1,y-1);

    float pixvalR2 = Img(x,y-1);

    float pixvalR3 = Img(x+1,y-1);

    float pixvalR4 = Img(x-1,y);

    float pixvalR5 = Img(x,y);

    float pixvalR6 = Img(x+1,y);

    float pixvalR7 = Img(x-1,y+1);

    float pixvalR8 = Img(x,y+1);

    float pixvalR9 = Img(x+1,y+1);

    // std::cout&lt;&lt;"coordinate value :"&lt;&lt;pixvalR5 &lt;&lt; endl;


    // MEAN VALUES OF RGB PIXELS
    float light = (pixvalR1+pixvalR2+pixvalR3+pixvalR4+pixvalR5+pixvalR6+pixvalR7+pixvalR8+pixvalR9)/9 ;

    // DISPLAYING MEAN RGB VALUES OF 9 PIXELS
    // std::cout&lt;&lt;"Lightness value :"&lt;&lt;light &lt;&lt; endl;


    // THRESHOLDING CONDITION
     c = (light &gt; 130 ) ? a : b; 

    // cout&lt;&lt;"Data is " &lt;&lt; c &lt;&lt;endl;

    ofstream fout("c.txt", ios::app);
    fout&lt;&lt;c;
    fout.close();


    }   

    timer.end();
       cerr&lt;&lt; timer.getSecs()&lt;&lt; " seconds for "&lt;&lt; nFramesCaptured &lt;&lt; "  frames : FPS " &lt;&lt; ( ( float ) ( nFramesCaptured ) / timer.getSecs() ) &lt;&lt;endl;

    Camera.release();

    std::cin.ignore();


    }
</code></pre>

<ul>
<li>from this code, I would like to know how can we get the data directly from camera.retrieve(data), without storing it as an image file and to access the data from an image buffer, to process the image and delete it further.</li>
</ul>

<p>As per the recommendations of Mark Setchell, which i made a slight changes in the code and i'm getting good results, but, Is there any way to improve the processing performance to get higher Frame rate ? with this code i'm able to get at a maximum of 10 FPS.</p>

<pre><code>#include &lt;ctime&gt;
#include &lt;fstream&gt;
#include &lt;iostream&gt;
#include &lt;thread&gt;
#include &lt;mutex&gt;
#include &lt;raspicam/raspicam.h&gt;

// Don't want any X11 display by CImg
#define cimg_display 0

#include &lt;CImg.h&gt;

using namespace cimg_library;
using namespace std;

#define NFRAMES     1000
#define NTHREADS    2
#define WIDTH       640
#define HEIGHT      480

// Commands/status for the worker threads
#define WAIT    0
#define GO      1
#define GOING   2
#define EXIT    3
#define EXITED  4
volatile int command[NTHREADS];

// Serialize access to cout
std::mutex cout_mutex;

// CImg initialisation
// Create a 1280x960 greyscale (Y channel of YUV) image
// Create a globally-accessible CImg for main and workers to access
CImg&lt;unsigned char&gt; img(WIDTH,HEIGHT,1,1,128);

////////////////////////////////////////////////////////////////////////////////
// worker thread - There will 2 or more of these running in parallel with the
//                 main thread. Do any image processing in here.
////////////////////////////////////////////////////////////////////////////////
void worker (int id) {

   // If you need a "results" image of type CImg, create it here before entering
   // ... the main processing loop below - you don't want to do malloc()s in the
   // ... high-speed loop
   // CImg results...

   int wakeups=0;

   // Create a white for annotating
   unsigned char white[] = { 255,255,255 };

   while(true){
      // Busy wait with 500us sleep - at worst we only miss 50us of processing time per frame
      while((command[id]!=GO)&amp;&amp;(command[id]!=EXIT)){
         std::this_thread::sleep_for(std::chrono::microseconds(500));
      }
      if(command[id]==EXIT){command[id]=EXITED;break;}
      wakeups++;

      // Process frame of data - access CImg structure here
      command[id]=GOING;

      // You need to add your processing in HERE - everything from
      // ... 9 PIXELS MATRIX GRAYSCALE VALUES to
      // ... THRESHOLDING CONDITION
    int a=1,b=0,c;
    int x=330,y=84;

// CImg&lt;float&gt; Img("/run/shm/result.png");
float pixvalR1 = img(x-1,y-1);

float pixvalR2 = img(x,y-1);

float pixvalR3 = img(x+1,y-1);

float pixvalR4 = img(x-1,y);

float pixvalR5 = img(x,y);

float pixvalR6 = img(x+1,y);

float pixvalR7 = img(x-1,y+1);

float pixvalR8 = img(x,y+1);

float pixvalR9 = img(x+1,y+1);


// MEAN VALUES OF RGB PIXELS
float light = (pixvalR1+pixvalR2+pixvalR3+pixvalR4+pixvalR5+pixvalR6+pixvalR7+pixvalR8+pixvalR9)/9 ;

// DISPLAYING MEAN RGB VALUES OF 9 PIXELS
// std::cout&lt;&lt;"Lightness value :"&lt;&lt;light &lt;&lt; endl;


// THRESHOLDING CONDITION
 c = (light &gt; 130 ) ? a : b; 

// cout&lt;&lt;"Data is " &lt;&lt; c &lt;&lt;endl;

ofstream fout("c.txt", ios::app);
fout&lt;&lt;c;
fout.close();
      // Pretend to do some processing.
      // You need to delete the following "sleep_for" and "if(id==0...){...}"
     // std::this_thread::sleep_for(std::chrono::milliseconds(2));


    /*  if((id==0)&amp;&amp;(wakeups==NFRAMES)){
        //  Annotate final image and save as PNG
          img.draw_text(100,100,"Hello World",white);
         img.save_png("result.png");
      } */
   }

   cout_mutex.lock();
   std::cout &lt;&lt; "Thread[" &lt;&lt; id &lt;&lt; "]: Received " &lt;&lt; wakeups &lt;&lt; " wakeups" &lt;&lt; std::endl;
   cout_mutex.unlock();
}

//timer functions
#include &lt;sys/time.h&gt;
#include &lt;unistd.h&gt;
class Timer{
    private:
    struct timeval _start, _end;

public:
  Timer(){}
    void start(){
        gettimeofday(&amp;_start, NULL);
    }
    void end(){
        gettimeofday(&amp;_end, NULL);
    }
    double getSecs(){
    return double(((_end.tv_sec  - _start.tv_sec) * 1000 + (_end.tv_usec - _start.tv_usec)/1000.0) + 0.5)/1000.;
    }

}; 

int main ( int argc,char **argv ) {

Timer timer;
   raspicam::RaspiCam Camera;
   // Allowable values: RASPICAM_FORMAT_GRAY,RASPICAM_FORMAT_RGB,RASPICAM_FORMAT_BGR,RASPICAM_FORMAT_YUV420
   Camera.setFormat(raspicam::RASPICAM_FORMAT_YUV420);

   // Allowable widths: 320, 640, 1280
   // Allowable heights: 240, 480, 960
   // setCaptureSize(width,height)
   Camera.setCaptureSize(WIDTH,HEIGHT);

   std::cout &lt;&lt; "Main: Starting"  &lt;&lt; std::endl;
   std::cout &lt;&lt; "Main: NTHREADS:" &lt;&lt; NTHREADS &lt;&lt; std::endl;
   std::cout &lt;&lt; "Main: NFRAMES:"  &lt;&lt; NFRAMES  &lt;&lt; std::endl;
   std::cout &lt;&lt; "Main: Width: "   &lt;&lt; Camera.getWidth()  &lt;&lt; std::endl;
   std::cout &lt;&lt; "Main: Height: "  &lt;&lt; Camera.getHeight() &lt;&lt; std::endl;

   // Spawn worker threads - making sure they are initially in WAIT state
   std::thread threads[NTHREADS];
   for(int i=0; i&lt;NTHREADS; ++i){
      command[i]=WAIT;
      threads[i] = std::thread(worker,i);
   }

   // Open camera
   cout&lt;&lt;"Opening Camera..."&lt;&lt;endl;
   if ( !Camera.open()) {cerr&lt;&lt;"Error opening camera"&lt;&lt;endl;return -1;}

   // Wait until camera stabilizes
   std::cout&lt;&lt;"Sleeping for 3 secs"&lt;&lt;endl;
   std::this_thread::sleep_for(std::chrono::seconds(3));
 timer.start();
   for(int frame=0;frame&lt;NFRAMES;frame++){
      // Capture frame
      Camera.grab();

      // Copy just the Y component to our mono CImg
      std::memcpy(img._data,Camera.getImageBufferData(),WIDTH*HEIGHT);

      // Notify worker threads that data is ready for processing
      for(int i=0; i&lt;NTHREADS; ++i){
         command[i]=GO;
      }
   }
timer.end();
cerr&lt;&lt; timer.getSecs()&lt;&lt; " seconds for "&lt;&lt; NFRAMES &lt;&lt; "  frames : FPS " &lt;&lt; ( ( float ) ( NFRAMES ) / timer.getSecs() ) &lt;&lt; endl;
   // Let workers process final frame, then tell to exit
 //  std::this_thread::sleep_for(std::chrono::milliseconds(50));

   // Notify worker threads to exit
   for(int i=0; i&lt;NTHREADS; ++i){
      command[i]=EXIT;
   }

   // Wait for all threads to finish
   for(auto&amp; th : threads) th.join();
}
</code></pre>

<p>COMPILED COMMAND FOR EXECUTION OF THE CODE :</p>

<pre><code>g++ -std=c++11 /home/pi/raspicam/src/raspicimgthread.cpp -o threadraspicimg -I. -I/usr/local/include -L /opt/vc/lib -L /usr/local/lib -lraspicam -lmmal -lmmal_core -lmmal_util -O2 -L/usr/X11R6/lib -lm -lpthread -lX11

**RESULTS :**
Main: Starting
Main: NTHREADS:2
Main: NFRAMES:1000
Main: Width: 640
Main: Height: 480
Opening Camera...
Sleeping for 3 secs
99.9194 seconds for 1000  frames : FPS 10.0081
Thread[1]: Received 1000 wakeups
Thread[0]: Received 1000 wakeups

real    1m43.198s
user    0m2.060s
sys     0m5.850s
</code></pre>

<p>And one more query is that, when i used normal Raspicam c++ API code to perform the same tasks (the code which i mentioned previous to this) i got almost same results with very slight enhancement in the performance (ofcourse my frame rate is increased from 9.4 FPS to 10 FPS).</p>

<p>But in the code 1:</p>

<p>I have been saving images in a ram disk for processing and then i'm deleting.
I haven't used any threads for parallel processing.</p>

<p>in the code 2 :</p>

<p>We are not saving any images in the disk and directly processing it from the buffer. And we are also using threads to improve the processing speed.</p>

<p>unfortunately, though we made some changes in the code 2 from the code 1, I'm not able to get desired results (which is to be performed at 30 FPS)</p>

<p>Awaiting your favorable suggestions and any help is really appreciated.</p>

<p>Thanks in advance</p>

<p>Best Regards
BLV Lohith Kumar</p>