EzDev.org

opencvsharp

opencvsharp - .NET Framework wrapper for OpenCV


Multiple results in OpenCVSharp3 MatchTemplate

I am trying to find image occurences inside an image. I have written the following code to get a single match using OpenCVSharp3:

Mat src = OpenCvSharp.Extensions.BitmapConverter.ToMat(Resources.all);
Mat template = OpenCvSharp.Extensions.BitmapConverter.ToMat(Resources.img);
Mat result = src.MatchTemplate(template, TemplateMatchModes.CCoeffNormed);

double minVal, maxVal;
OpenCvSharp.Point minLoc, maxLoc;
result.MinMaxLoc(out minVal, out maxVal, out minLoc, out maxLoc);
Console.WriteLine("maxLoc: {0}, maxVal: {1}", maxLoc, maxVal);

How can I get more matches based on a threshold value?

Thank you!


Source: (StackOverflow)

Convert Point2f[] to Mat[] OpenCV

I am trying to convert a Point2f[] to a Mat[] using the OpenCvSharp library and have no idea where to start. Some help would be greatly appreciated.

rect = cv2.minAreaRect(c)
box = np.int0(cv2.cv.BoxPoints(rect))

This is the code i am trying to translate into OpenCvSharp Detect barcodes in python

This is the first time i have used OpenCvSharp so i can do everything in the linked page except the lines i have include.


Source: (StackOverflow)

OpenCV (EMGUCV wrapper) integration in Unity

As you know OpenCV is very useful library that let you do amazing and powerful things in Computer vision. So I passed a good time to figure out how to use it in Unity3d, I had many problems, and searching in the Net, I have found several suggestions but not one worked for me.

  • I’m using a Unity Pro 4.0
  • This version of Emgu CV (emgucv-windows-universal-gpu 2.4.9.1847)
  • My target for unity project is: windows and not web player

Source: (StackOverflow)

What is the difference between OpenCV.NET, OpenCVSharp and EmguCV?

What is the difference between OpenCV.NET, OpenCVSharp and EmguCV?

They are derived from OpenCV.

So, what is the difference in their design, implementation and application philosophies?


Source: (StackOverflow)

Memory Leak Issue. Eye-tracking in Unity with OpenCVSharp

I've been working on this project for a few months now, where I'm trying to integrate eye-tracking into Unity using OpenCVSharp. I've managed to get everything working, including the actual tracking of the pupil etc, however I've got a memory leak. Basically after 20-30seconds of the program running it freezes and the console errors saying "Unable to allocate (insert number here) bits". After looking at the memory usage during running of the program, you can see its use steadily climb until it maxes then crashes.

Now I've spent quite a while trying to fix the issue, and read a lot of help posts about releasing images/storage etc correctly. Despite the fact I'm doing this, it doesn't appear to be releasing them correctly. I tried using the garbage collector to force it to reclaim the memory however that didn't seem to work either. Am I just doing something fundamentally wrong with the images and how I reclaim them? Or is having the creation of new images each frame (even though I'm releasing them) causing the problem.

Any help would be greatly appreciated. Here's the code below, you can ignore a lot of the stuff within the update function as its to do with the actual tracking section and calibration. I realise the code is pretty messy, sorry about that! The main section to worry about is EyeDetection().

using UnityEngine;
using System.Collections;
using System;
using System.IO;
using OpenCvSharp;
using OpenCvSharp.Blob;
//using System.Xml;
//using System.Threading;
//using AForge;

//using OpenCvSharp.Extensions;
//using System.Windows.Media;
//using System.Windows.Media.Imaging;



public class CaptureScript2 : MonoBehaviour
{
    //public GameObject planeObj;
    public WebCamTexture webcamTexture;     //Texture retrieved from the webcam
    //public Texture2D texImage;              //Texture to apply to plane
    public string deviceName;

    private int devId = 1;
    private int imWidth = 800;             //camera width
    private int imHeight = 600;             //camera height
    private string errorMsg = "No errors found!";
    private static IplImage camImage;                   //Ipl image of the converted webcam texture
    //private static IplImage yuv;
    //private static IplImage dst;
    private CvCapture cap;                  //Current camera capture
    //private IplImage eyeLeft;
    //private IplImage eyeRight;
    //private IplImage eyeLeftFinal;
    //private IplImage eyeRightFinal;
    private double leftEyeX;
    private double leftEyeY;
    private double rightEyeX;
    private double rightEyeY;
    private int calibState;
    private double LTRCPx;
    private double LTLCPx;
    private double LBLCPy;
    private double LTLCPy;
    private double RTRCPx;
    private double RTLCPx;
    private double RBLCPy;
    private double RTLCPy;
    private double gazeWidth;
    private double gazeHeight;
    private double gazeScaleX;
    private double gazeScaleY;

    public static CvMemStorage storageFace;
    public static CvMemStorage storage;

    public static double gazePosX;
    public static double gazePosY;

    private bool printed = true;
    //private CvRect r;
    //private IplImage smallImg;

    CvColor[] colors = new CvColor[]
    {
        new CvColor(0,0,255),
        new CvColor(0,128,255),
        new CvColor(0,255,255),
        new CvColor(0,255,0),
        new CvColor(255,128,0),
        new CvColor(255,255,0),
        new CvColor(255,0,0),
        new CvColor(255,0,255),
    };

    //scale for small image
    const double Scale = 1.25;
    const double scaleEye = 10.0;
    const double ScaleFactor = 2.5;
    //must show 2 eyes on the screen
    const int MinNeighbors = 2;
    const int MinNeighborsFace = 1;


    // Use this for initialization
    void Start ()
    {


        //Webcam initialisation
        WebCamDevice[] devices = WebCamTexture.devices;
        Debug.Log ("num:" + devices.Length);

        for (int i=0; i<devices.Length; i++) 
        {
            print (devices [i].name);
            if (devices [i].name.CompareTo (deviceName) == 1) 
            {
                devId = i;
            }
        }

        if (devId >= 0) 
        {
            //mainImage = new IplImage (imWidth, imHeight, BitDepth.U8, 3);


        }

        //create capture from current device
        cap = Cv.CreateCameraCapture(devId);
        //set properties of the capture
        Cv.SetCaptureProperty(cap, CaptureProperty.FrameWidth, imWidth);
        Cv.SetCaptureProperty(cap, CaptureProperty.FrameHeight, imHeight);
        //create window to display capture
        //Cv.NamedWindow("Eye tracking", WindowMode.AutoSize);
        Cv.NamedWindow ("EyeLeft", WindowMode.AutoSize);
        Cv.NamedWindow ("EyeRight", WindowMode.AutoSize);
        Cv.NamedWindow ("Face", WindowMode.AutoSize);

        calibState = 1;


    }


    void Update ()
    {
        if(Input.GetKeyDown(KeyCode.Space) && calibState < 3)
        {
            calibState++;
        }

        if(Input.GetMouseButtonDown(0) && calibState < 4)
        {
            printed = false;
            calibState++;

            Cv.DestroyAllWindows();
            Cv.ReleaseCapture(cap);

            cap = Cv.CreateCameraCapture(devId);
        }
        //if device is connected
        if (devId >= 0)
        {   
            //cap = Cv.CreateCameraCapture(devId);
            //Cv.Release
            //retrieve the current frame from camera
            camImage = Cv.QueryFrame(cap);
            //detect eyes and apply circles
            //
            EyeDetection();

            Cv.ReleaseImage(camImage);
            //PupilTracking();



            switch(calibState)
            {
            case 1:
                LTRCPx = leftEyeX;
                RTRCPx = rightEyeX;

                break;

            case 2:

                LTLCPx = leftEyeX;
                LTLCPy = leftEyeY;
                RTLCPx = rightEyeX;
                RTLCPy = rightEyeY;

                break;
            case 3:

                LBLCPy = leftEyeY;// + rightEyeY) /2 ;
                RBLCPy = rightEyeY;


                break;

            case 4:

                //gazeWidth = (((LTRCPx - LTLCPx) + (RTRCPx - RTLCPx)) / 2) * -1;
                //gazeHeight = ((LBLCPy - LTLCPy) + (RBLCPy - RTLCPy)) /2;
                gazeWidth = LTLCPx -LTRCPx;
                gazeHeight = LBLCPy - LTLCPy;

                gazeScaleX = (Screen.width/gazeWidth);
                gazeScaleY = Screen.height/gazeHeight;

                gazePosX = gazeScaleX *(leftEyeX - LTRCPx);
                gazePosY = gazeScaleY *(leftEyeY - LTLCPy);

                break;
            }


            //Cv.ReleaseCapture(cap);

        } 
        else 
        {
            Debug.Log ("Can't find camera!");
        }

        //print (calibState);
        if(printed == false)
        {
            print ("Gaze pos x = " + gazePosX);
            print ("Gaze pos Y = " + gazePosY);
            print ("Scale x = " + gazeScaleX);
            print ("Scale y = " + gazeScaleY);
            print ("Gaze width = " + gazeWidth);
            print ("Gaze Height = " + gazeHeight);
            print ("left eye x = " + leftEyeX);
            print ("left eye Y = " + leftEyeY);
            print ("calib state = " + calibState);

            printed = true;
        }



        //Cv.ShowImage("Eye tracking", mainImage);
        //Cv.ShowImage ("EyeLeft", grayEyeLeft);
        //Cv.ShowImage ("EyeRight", grayEyeRight);

    }



    void EyeDetection()
    {
        IplImage mainImage = new IplImage (imWidth, imHeight, BitDepth.U8, 3);

        IplImage smallImg = new IplImage(mainImage.Width, mainImage.Height ,BitDepth.U8, 1);
        Cv.Resize (camImage, mainImage, Interpolation.Linear);

        IplImage gray = new IplImage(mainImage.Size, BitDepth.U8, 1);

        Cv.CvtColor (mainImage, gray, ColorConversion.BgrToGray);
        Cv.Resize(gray, smallImg, Interpolation.Linear);
        Cv.EqualizeHist(smallImg, smallImg);
        Cv.ReleaseImage (gray);


            //IplImage hack = Cv.LoadImage("\\Users\\User\\Desktop\\Honours Projects\\Project10\\Project\\Assets\\bug.jpeg");
            //Cv.Erode (hack, hack);
            //Cv.ReleaseImage (hack);

            //uint sizeStore = 2877212;
        CvHaarClassifierCascade cascadeFace = CvHaarClassifierCascade.FromFile("\\Users\\User\\Documents\\opencv\\sources\\data\\haarcascades\\haarcascade_frontalface_alt2.xml");

        CvMemStorage storageFace = new CvMemStorage();
        storageFace.Clear ();

        CvSeq<CvAvgComp> faces = Cv.HaarDetectObjects(smallImg, cascadeFace, storageFace, ScaleFactor, MinNeighborsFace, 0, new CvSize(30,30));

        for(int j = 0; j < faces.Total; j++)
        {
            CvRect face = faces[j].Value.Rect;

            CvHaarClassifierCascade cascadeEye = CvHaarClassifierCascade.FromFile ("\\Users\\User\\Documents\\opencv\\sources\\data\\haarcascades\\haarcascade_eye.xml");

            IplImage faceImg = new IplImage(face.Width, face.Height, BitDepth.U8, 1);
            IplImage faceImgColour = new IplImage(face.Width, face.Height, BitDepth.U8, 3);

            CvMemStorage storage = new CvMemStorage();
            storage.Clear ();

            Cv.SetImageROI(smallImg, face);
            Cv.Copy (smallImg, faceImg);
            Cv.ResetImageROI(smallImg);

            Cv.SetImageROI(mainImage, face);
            Cv.Copy (mainImage, faceImgColour);
            Cv.ResetImageROI(mainImage);


            Cv.ShowImage ("Face", faceImgColour);


            CvSeq<CvAvgComp> eyes = Cv.HaarDetectObjects(faceImg, cascadeEye, storage, ScaleFactor, MinNeighbors, 0, new CvSize(30, 30));
            for(int i = 0; i < eyes.Total; i++)
            {
                CvRect r = eyes[i].Value.Rect;


                Cv.SetImageROI(faceImgColour, r);

                if(i == 1)
                {
                    IplImage eyeLeft = new IplImage(new CvSize(r.Width, r.Height), BitDepth.U8, 3);

                    Cv.Copy(faceImgColour, eyeLeft);

                    IplImage yuv = new IplImage(eyeLeft.Size, BitDepth.U8, 3);
                    IplImage dst = new IplImage(eyeLeft.Size, BitDepth.U8, 3);
                    IplImage grayEyeLeft = new IplImage(eyeLeft.Size, BitDepth.U8, 1);
                    IplImage eyeLeftFinal = new IplImage(Cv.Round(grayEyeLeft.Width * scaleEye), Cv.Round(grayEyeLeft.Height * scaleEye), BitDepth.U8, 1);
                    Cv.CvtColor(eyeLeft, yuv, ColorConversion.BgrToCrCb);
                    Cv.Not(yuv, dst);
                    Cv.CvtColor(dst,eyeLeft,ColorConversion.CrCbToBgr);
                    Cv.CvtColor(eyeLeft, grayEyeLeft, ColorConversion.BgrToGray);

                    Cv.Resize (grayEyeLeft, eyeLeftFinal, Interpolation.Linear);
                    Cv.Threshold(eyeLeftFinal, eyeLeftFinal, 230, 230, ThresholdType.Binary);
                    CvBlobs b1 = new CvBlobs(eyeLeftFinal);
                    if(b1.Count > 0)
                    {
                        leftEyeX = b1.LargestBlob().Centroid.X;
                        leftEyeY = b1.LargestBlob().Centroid.Y;
                    }

                    Cv.ShowImage ("EyeLeft", eyeLeftFinal);

                    Cv.ReleaseImage (yuv);
                    Cv.ReleaseImage (dst);
                    Cv.ReleaseImage (grayEyeLeft);
                    Cv.ReleaseImage (eyeLeftFinal);
                    b1.Clear();

                    Cv.ReleaseImage (eyeLeft);


                }
                if(i == 0)
                {
                    IplImage eyeRight = new IplImage(new CvSize(r.Width, r.Height), BitDepth.U8, 3);

                    Cv.Copy(faceImgColour, eyeRight);

                    IplImage yuv2 = new IplImage(eyeRight.Size, BitDepth.U8, 3);
                    IplImage dst2 = new IplImage(eyeRight.Size, BitDepth.U8, 3);
                    IplImage grayEyeRight = new IplImage(eyeRight.Size, BitDepth.U8, 1);
                    IplImage eyeRightFinal = new IplImage(Cv.Round(grayEyeRight.Width * scaleEye), Cv.Round(grayEyeRight.Height * scaleEye), BitDepth.U8, 1);
                    Cv.CvtColor(eyeRight, yuv2, ColorConversion.BgrToCrCb);
                    Cv.Not(yuv2, dst2);
                    Cv.CvtColor(dst2,eyeRight,ColorConversion.CrCbToBgr);
                    Cv.CvtColor(eyeRight, grayEyeRight, ColorConversion.BgrToGray);

                    Cv.Resize (grayEyeRight, eyeRightFinal, Interpolation.Linear);
                    Cv.Threshold(eyeRightFinal, eyeRightFinal, 230, 230, ThresholdType.Binary);
                    CvBlobs b2 = new CvBlobs(eyeRightFinal);

                    if(b2.Count > 0)
                    {
                        rightEyeX = b2.LargestBlob().Centroid.X;
                        rightEyeY = b2.LargestBlob().Centroid.Y;
                    }

                    Cv.ShowImage ("EyeRight", eyeRightFinal);

                    Cv.ReleaseImage (yuv2);
                    Cv.ReleaseImage (dst2);
                    Cv.ReleaseImage (grayEyeRight);
                    Cv.ReleaseImage (eyeRightFinal);
                    b2.Clear ();

                    Cv.ReleaseImage (eyeRight);

                }

                Cv.ResetImageROI(faceImgColour);
            }

            //Cv.ShowImage("Eye tracking", mainImage);

            Cv.ReleaseImage (faceImg);
            Cv.ReleaseImage (faceImgColour);
            Cv.ReleaseMemStorage(storage);
            Cv.ReleaseHaarClassifierCascade(cascadeEye);


        }

        Cv.ReleaseMemStorage(storageFace);
        Cv.ReleaseHaarClassifierCascade(cascadeFace);


        //PupilTracking ();
        Cv.ReleaseImage(smallImg);
        Cv.ReleaseImage (mainImage);
        GC.Collect();

    }

    void OnGUI ()
    {
            GUI.Label (new Rect (200, 200, 100, 90), errorMsg);
    }

    void OnDestroy()
    {
        Cv.DestroyAllWindows();
        Cv.ReleaseCapture(cap);
    }

Source: (StackOverflow)

OpenCV: Projection matrices from Fundamental matrix and SolvePnPRansac are totally different

As part of my master thesis I am exploring Structure From Motion. After reading parts of the H&Z book, following online tutorials and reading through many SO posts I have some useful results, but I have also some problems. I'm using OpenCVSharp wrapper. All images are taken with the same camera.

What I have now:


First I calculate initial 3d points coordinates. I do this with these steps:

  1. Calculate Farneback's dense optical flow.
  2. Find the Fundamental matrix using Cv2.FindFundamentalMat with RANSAC
  3. Get Essential matrix using camera intrinsics (at this point I use pre-determined intrinsics) and decompose it:

    Mat essential = camera_matrix.T() * fundamentalMatrix * camera_matrix;
    
    SVD decomp = new SVD(essential, OpenCvSharp.SVDFlag.ModifyA);
    
    Mat diag = new Mat(3, 3, MatType.CV_64FC1, new double[] {
        1.0D, 0.0D, 0.0D,
        0.0D, 1.0D, 0.0D,
        0.0D, 0.0D, 0.0D
    });
    
    Mat Er = decomp.U * diag * decomp.Vt;
    
    SVD svd = new SVD(Er, OpenCvSharp.SVDFlag.ModifyA);
    
    Mat W = new Mat(3, 3, MatType.CV_64FC1, new double[] {
        0.0D, -1.0D, 0.0D,
        1.0D, 0.0D, 0.0D,
        0.0D, 0.0D, 1.0D
    });
    
    Mat Winv = new Mat(3, 3, MatType.CV_64FC1, new double[] {
        0.0D, 1.0D, 0.0D,
        -1.0D, 0.0D, 0.0D,
        0.0D, 0.0D, 1.0D
    });
    
    Mat R1 = svd.U * W * svd.Vt;
    Mat T1 = svd.U.Col[2];
    Mat R2 = svd.U * Winv * svd.Vt;
    Mat T2 = -svd.U.Col[2];
    
    Mat[] Ps = new Mat[4];
    
    for (int i = 0; i < 4; i++)
        Ps[i] = new Mat(3, 4, MatType.CV_64FC1);
    
    Cv2.HConcat(R1, T1, Ps[0]);
    Cv2.HConcat(R1, T2, Ps[1]);
    Cv2.HConcat(R2, T1, Ps[2]);
    Cv2.HConcat(R2, T2, Ps[3]);
    
  4. Then I check which projection matrix has the most points in front of both cameras by triangulating the points and then multiplying them by projection matrices (I tried both Cv2.TriangulatePoints and H&Z version with similar results) and checking for positive Z values (after converting from homogenous values):

    P * point3D
    
  5. At this point I should have more or less correct 3D points. The 3D visualization looks quite correct.

Then I calculate SolvePNP for every new frame by using again the dense optical flow and with the previous projection matrix known I calculate next 3D points and add them to the model. Again 3D visualization looks more or less correct (no bundle adjustment at this point).

Since I need to use SolvePNP for every new frame I started by checking it with the one calculated for the first 2 images with the fundamental matrix. Theoretically the projection matrix should be the same or almost the same as the one calculated with the initial algorithm - I use the initial 3D points and the corresponding 2D points in the second image. But it's not the same.

Here is the one calculated by decomposing the fundamental matrix:

0,955678480016302 -0,0278536127242155 0,293091827064387 -0,148461857222772 
-0,0710609269521247 0,944258717203142 0,321443338158658 -0,166586733489084 
0,285707870900394 0,328023857736121 -0,900428432059693 0,974786098164824 

And here is the one I got from the SolvePnPRansac:

0,998124823499476 -0,0269266503551759 -0,0549708305812315 -0,0483615883381834 
0,0522887223187244 0,8419572918112 0,537004476968512 -2,0699592377647 
0,0318233598542908 -0,538871853288516 0,841786433426546 28,7686946357429

Both of them look like correct projection matrices, but they are different.

For those patient people who read the whole post I have 3 questions:

1. Why are these matrices different? I know the reconstruction is up to scale, but since I have an arbitrary scale assigned in the first place the SolvePNP should keep that scale.
2. I noticed one strange thing - the translation in the first matrix seems to be exactly the same no matter what images I use.
3. Is the overal algorithm correct, or am I doing something wrong? Do I miss some important step?

If more code is required let me know and I will edit the question.

Thank You!


Source: (StackOverflow)

OpenCV: Concurrent GetSubImage to extract a portion of an image

I've implemented a GetSubImage function in order to extract a portion of an image. I use SetROI, Copy, and ResetROI. But this not works when Parallel Tasks are using the same image because of the SetROI is not parallelizable.

Any other way to extract portion of an image that can run concurrent?


Source: (StackOverflow)

Eye detection using OpenCVSharp in Unity (fps issues)

I'm currently working on a project involving integrating OpenCVSharp into Unity, to allow eye tracking within a game environment. I've managed to get OpenCVSharp integrated into the Unity editor and currently have eye-detection (not tracking) working within a game. It can find your eyes within a webcam image, then display where its currently detected them on a texture, which I display within the scene.

However its causing a HUGE fps drop, mainly because every frame its converting a webcam texture into an IPLimage so that OpenCV can handle it. It then has to convert it back to a 2Dtexture to be displayed within the scene, after its done all the eye detection. So understandably its too much for the CPU to handle. (As far as I can tell its only using 1 core on my CPU).

Is there a way to do all the eye detection without converting the texture to an IPLimage? Or any other way to fix the fps drop. Some things that I've tried include:

  • Limiting the frames that it updates on. However this just causes it to run smoothly, then stutter horribly on the frame that it has to update.

  • Looking at threading, but as far as I'm aware Unity doesn't allow it. As far as I can tell its only using 1 core on my CPU which seems a bit silly. If there was a way to change this it could fix the issue?

  • Tried different resolutions on the camera, however the resolution that the game can actually run smoothly at, is too small for the eye's to actually be detected, let alone tracked.

I've included the code below, of if you would prefer to look at it in a code editor here is a link to the C# File. Any suggestions or help would be greatly appreciated!

For reference I used code from here (eye detection using opencvsharp).

using UnityEngine;
using System.Collections;
using System;
using System.IO;
using OpenCvSharp;
//using System.Xml;

//using OpenCvSharp.Extensions;
//using System.Windows.Media;
//using System.Windows.Media.Imaging;



public class CaptureScript : MonoBehaviour
{
    public GameObject planeObj;
    public WebCamTexture webcamTexture;     //Texture retrieved from the webcam
    public Texture2D texImage;              //Texture to apply to plane
    public string deviceName;

    private int devId = 1;
    private int imWidth = 640;              //camera width
    private int imHeight = 360;             //camera height
    private string errorMsg = "No errors found!";
    static IplImage matrix;                 //Ipl image of the converted webcam texture

    CvColor[] colors = new CvColor[]
    {
        new CvColor(0,0,255),
        new CvColor(0,128,255),
        new CvColor(0,255,255),
        new CvColor(0,255,0),
        new CvColor(255,128,0),
        new CvColor(255,255,0),
        new CvColor(255,0,0),
        new CvColor(255,0,255),
    };

    const double Scale = 1.25;
    const double ScaleFactor = 2.5;
    const int MinNeighbors = 2;


// Use this for initialization
    void Start ()
    {
            //Webcam initialisation
            WebCamDevice[] devices = WebCamTexture.devices;
            Debug.Log ("num:" + devices.Length);

            for (int i=0; i<devices.Length; i++) {
                    print (devices [i].name);
                    if (devices [i].name.CompareTo (deviceName) == 1) {
                            devId = i;
                    }
            }

            if (devId >= 0) {
                    planeObj = GameObject.Find ("Plane");
                    texImage = new Texture2D (imWidth, imHeight, TextureFormat.RGB24, false);
                    webcamTexture = new WebCamTexture (devices [devId].name, imWidth, imHeight, 30);
                    webcamTexture.Play ();

                    matrix = new IplImage (imWidth, imHeight, BitDepth.U8, 3);
            }


    }

    void Update ()
    {
        if (devId >= 0)
        {
                //Convert webcam texture to iplimage
                Texture2DtoIplImage();

            /*DO IMAGE MANIPULATION HERE*/

            //do eye detection on iplimage
            EyeDetection();


            /*END IMAGE MANIPULATION*/

            if (webcamTexture.didUpdateThisFrame) 
            {
                //convert iplimage to texture
                IplImageToTexture2D();
            }

        } 
        else 
        {
            Debug.Log ("Can't find camera!");
        }

    }

    void EyeDetection()
    {

        using(IplImage smallImg = new IplImage(new CvSize(Cv.Round (imWidth/Scale), Cv.Round(imHeight/Scale)),BitDepth.U8, 1))
        {
            using(IplImage gray = new IplImage(matrix.Size, BitDepth.U8, 1))
            {
                Cv.CvtColor (matrix, gray, ColorConversion.BgrToGray);
                Cv.Resize(gray, smallImg, Interpolation.Linear);
                Cv.EqualizeHist(smallImg, smallImg);
            }


            using(CvHaarClassifierCascade cascade = CvHaarClassifierCascade.FromFile (@"C:\Users\User\Documents\opencv\sources\data\haarcascades\haarcascade_eye.xml"))
            using(CvMemStorage storage = new CvMemStorage())
            {
                storage.Clear ();
                CvSeq<CvAvgComp> eyes = Cv.HaarDetectObjects(smallImg, cascade, storage, ScaleFactor, MinNeighbors, 0, new CvSize(30, 30));
                for(int i = 0; i < eyes.Total; i++)
                {
                    CvRect r = eyes[i].Value.Rect;
                    CvPoint center = new CvPoint{ X = Cv.Round ((r.X + r.Width * 0.5) * Scale), Y = Cv.Round((r.Y + r.Height * 0.5) * Scale) };
                    int radius = Cv.Round((r.Width + r.Height) * 0.25 * Scale);
                    matrix.Circle (center, radius, colors[i % 8], 3, LineType.AntiAlias, 0);
                }
            }

        }
    }

    void OnGUI ()
    {
            GUI.Label (new Rect (200, 200, 100, 90), errorMsg);
    }

    void IplImageToTexture2D ()
    {
            int jBackwards = imHeight;

            for (int i = 0; i < imHeight; i++) {
                    for (int j = 0; j < imWidth; j++) {
                            float b = (float)matrix [i, j].Val0;
                            float g = (float)matrix [i, j].Val1;
                            float r = (float)matrix [i, j].Val2;
                            Color color = new Color (r / 255.0f, g / 255.0f, b / 255.0f);


                            jBackwards = imHeight - i - 1; // notice it is jBackward and i
                            texImage.SetPixel (j, jBackwards, color);
                    }
            }
            texImage.Apply ();
            planeObj.renderer.material.mainTexture = texImage;

    }

    void Texture2DtoIplImage ()
    {
            int jBackwards = imHeight;

            for (int v=0; v<imHeight; ++v) {
                    for (int u=0; u<imWidth; ++u) {

                            CvScalar col = new CvScalar ();
                            col.Val0 = (double)webcamTexture.GetPixel (u, v).b * 255;
                            col.Val1 = (double)webcamTexture.GetPixel (u, v).g * 255;
                            col.Val2 = (double)webcamTexture.GetPixel (u, v).r * 255;

                            jBackwards = imHeight - v - 1;

                            matrix.Set2D (jBackwards, u, col);
                            //matrix [jBackwards, u] = col;
                    }
            }
    }
}

Source: (StackOverflow)

Correct way to extract Translation from Essential Matrix through SVD

I calibrated my camera and found the intrinsic parameters(K). Also I have calculated the Fundamental Matrix (F).

Now E= K_T* F * K . So far so good.

Now we pass the Essential Matrix(E) to the SVD to use the decomposition values (U,W,V) to extract the Rotation and Translation:

 essentialMatrix = K.Transpose().Mul(fund).Mul(K);
 CvInvoke.cvSVD(essentialMatrix, wMatrix, uMatrix, vMatrix, Emgu.CV.CvEnum.SVD_TYPE.CV_SVD_DEFAULT);

** Question) At this point, two methods have been proposed, and it has confused me which one really give out the right answer- specifically for Translation:

At first method enter link description here the author suggests to compute the R,T as following:

enter image description here

But in Second method [http://isit.u-clermont1.fr/~ab/Classes/DIKU-3DCV2/Handouts/Lecture16.pdf] the author provides another formula for T which is +U , -U as shown below:

enter image description here

I am implementing this on C# .Net using openCv libraries. Anybody knows which Translation Formula is the right one?


Source: (StackOverflow)

How can I convert Mat to Bitmap using OpenCVSharp?

First, I tried this,

    public static Bitmap MatToBitmap(Mat mat)
    {
        return OpenCvSharp.Extensions.BitmapConverter.ToBitmap(mat);
    }

enter image description here

So, then, I tried this,

    public static Bitmap MatToBitmap(Mat mat)
    {
        mat.ConvertTo(mat, MatType.CV_8U);
        return OpenCvSharp.Extensions.BitmapConverter.ToBitmap(mat);
    }

The image is completely black,

enter image description here

    public static Bitmap ConvertMatToBitmap(Mat matToConvert) 
    {            
        return new Bitmap(matToConvert.Cols, matToConvert.Rows, 4*matToConvert.Rows, System.Drawing.Imaging.PixelFormat.Format8bppIndexed, matToConvert.Data);
    }

This doesn't work either.

enter image description here


Source: (StackOverflow)

How to use new C++ style API in OpenCVSharp code?

I'm using OpenCVSharp but currently have half of my code in the C and the other half in the C++ API, I'm trying to port it all to the C++ version to avoid the deprecated API as well as to avoid loading images twice (share Mat instead of having one Mat and one CvMat per image)

Here's the code I had that works :

            CvMat distortion = new CvMat(8, 1, MatrixType.F64C1);
            distortion[0, 0] = camera.CameraConfig.k1;
            distortion[1, 0] = camera.CameraConfig.k2;
            distortion[2, 0] = camera.CameraConfig.p1;
            distortion[3, 0] = camera.CameraConfig.p2;
            distortion[4, 0] = camera.CameraConfig.k3;
            distortion[5, 0] = 0;
            distortion[6, 0] = 0;
            distortion[7, 0] = 0;

            CvMat intrinsic = new CvMat(3, 3, MatrixType.F32C1);
            intrinsic[0, 0] = camera.CameraConfig.fx;
            intrinsic[0, 1] = camera.CameraConfig.skew;
            intrinsic[0, 2] = camera.CameraConfig.cx;
            intrinsic[1, 0] = 0;
            intrinsic[1, 1] = camera.CameraConfig.fy;
            intrinsic[1, 2] = camera.CameraConfig.cy;
            intrinsic[2, 0] = 0;
            intrinsic[2, 1] = 0;
            intrinsic[2, 2] = 1;
            Cv.Undistort2(camera.SourceImage, newSourceImage,intrinsic,distortion);

And the code (that seemed like an obvious port as I typed it) that doesn't work (I end up with a single colored image of a color that is present in the scene) :

            Mat distortion = new Mat(8, 1, MatType.CV_64FC1);
            distortion.Set(0, 0, camera.CameraConfig.k1);
            distortion.Set(1, 0, camera.CameraConfig.k2);
            distortion.Set(2, 0, camera.CameraConfig.p1);
            distortion.Set(3, 0, camera.CameraConfig.p2);
            distortion.Set(4, 0, camera.CameraConfig.k3);
            distortion.Set(5, 0, 0);
            distortion.Set(6, 0, 0);
            distortion.Set(7, 0, 0);

            Mat intrinsic = new Mat(3, 3, MatType.CV_32FC1);
            intrinsic.Set(0, 0, camera.CameraConfig.fx);
            intrinsic.Set(0, 1, camera.CameraConfig.skew);
            intrinsic.Set(0, 2, camera.CameraConfig.cx);
            intrinsic.Set(1, 0, 0);
            intrinsic.Set(1, 1, camera.CameraConfig.fy);
            intrinsic.Set(1, 2, camera.CameraConfig.cy);
            intrinsic.Set(2, 0, 0);
            intrinsic.Set(2, 1, 0);
            intrinsic.Set(2, 2, 1);
            var newSourceImage = camera.SourceImage.Undistort(intrinsic, distortion);

Am I wrongly setting the values? Is the porting not that straightforward?


Source: (StackOverflow)

Kalman prediction & correction same as start values

Im implementing a kalman filter that receives real measurements from a camshift head tracking after previously detecting the face with an Haar Cascade. I initialise the state pre and state post variables from the kalman filter with the head position from the Haar Cascade, and call kalman predict and correct while doing the camshift to get some smoothing. The problem is that the predicted and corrected values are always the start values from the haar cascade. Should i update the state pre or state post variables while doing camshift?

private CvKalman Kf ;
public CvMat measurement = new CvMat(2,1, MatrixType.F32C1);
public int frameCounter = 0;
public float[] A = {1,0,1,0, 0,1,0,1, 0,0,1,0, 0,0,0,1};
public float[] H = {1,0,0,0, 0,1,0,0};
public float[] Q = {0.0001f,0,0,0, 0,0.0001f,0,0, 0,0,0.0001f,0, 0,0,0,0.0001f};
public float[] R = {0.2845f,0.0045f,0.0045f,0.0455f};
public float[] P = {100,0,0,0, 0,100,0,0, 0,0,100,0, 0,0,0,100};

initkalman is called once while doing the haar cascade, and tracking window is the initial head position.

void initKalman(CvRect trackinWindow){
    Kf = new CvKalman (4, 2, 0);
    Marshal.Copy (A, 0, Kf.TransitionMatrix.Data, A.Length);
    Marshal.Copy (H, 0, Kf.MeasurementMatrix.Data, H.Length);
    Marshal.Copy (Q, 0, Kf.ProcessNoiseCov.Data, Q.Length);
    Marshal.Copy (R, 0, Kf.MeasurementNoiseCov.Data, R.Length);
    Marshal.Copy (P, 0, Kf.ErrorCovPost.Data, P.Length);
    measurement.mSet (0, 0, trackingWindow.X);
    measurement.mSet (1, 0, trackingWindow.Y);

    Kf.StatePost.mSet(0,0,trackingWindow.X);
    Kf.StatePost.mSet(1,0,trackingWindow.Y);
    Kf.StatePost.mSet(2, 0, 0);
    Kf.StatePost.mSet(3, 0, 0);
}

I call the processKalman function in each camshift iteration, being now tracking window the actual head position

    CvPoint processKalman(CvRect trackingwindow)
{

    CvMat prediction = Cv.KalmanPredict(Kf);

    CvPoint predictionPoint;
    predictionPoint.X = (int)prediction.DataArraySingle [0];
    predictionPoint.Y = (int)prediction.DataArraySingle [1];

    Debug.Log (predictionPoint.X);

    measurement.mSet (0, 0, trackingWindow.X);
    measurement.mSet (1, 0, trackingWindow.Y);

    CvMat estimated = Cv.KalmanCorrect(Kf,measurement);

    CvPoint auxCP;

    auxCP.X = (int)estimated.DataArraySingle [0];
    auxCP.Y = (int)estimated.DataArraySingle [1];
    return auxCP;

}

This is not working and always returning only the initial head position.The guy from this excellent blog is changing the state post with the actual measurement before calling the predict function, but the only thing that changes for me by doing so is that the predicted and corrected values now are identical with the camshift head position for each frame.


Source: (StackOverflow)

OpenCVSharp AccessViolationException in FindCirclesGrid

I'm using OpenCVSharp to run some calibration tests but i can't seem to get FindCirclesGrid to work, i'm getting a really unexpected AccessViolationException when calling FindCirclesGrid.

I'm not sure what i could be doing wrong as the first two line are pretty much exactly as in samples,centers is not initialized as it is an output parameter and everything that is passed down to OpenCV seems to be initialized in OpenCVSharp's wrapper function.

void test()
{
    Mat im = Cv2.ImRead(@"path_to_my_file.jpg");
    Size patternsize = new Size(11, 4);
    Point2f[] centers;
    var f = Cv2.FindCirclesGrid(im, patternsize, out centers, FindCirclesGridFlag.AsymmetricGrid);
}

I'm using the latest OpenCVSharp straight from nuget

Edit1 : i forgot to mention this in the question but i have already tried adding the foillowing after FindCirclesGrid to ensure objects weren't wrongly collected before they should, this changed nothing. Also the bug happens just the same in debug and release.

 Console.Writeline(im.ToString());
 Console.Writeline(patternsize.ToString());
 Console.Writeline(centers.ToString());
 Console.Writeline(f.ToString());

Source: (StackOverflow)

Document detection on scanned image OpenCV

I use OpenCV for image pre-processing. I need cut only scanned photo, whithout white area. I use algoritm:

  • image_canny <- apply canny edge detector to this channel
  • for threshold in bunch_of_increasing_thresholds:
  • image_thresholds[threshold] <- apply threshold to this channel for each contour found in {image_canny} U image_thresholds:
  • Approximate contour with polygons
  • if the approximation has four corners and the angles are close to 90 degrees.for find rectangle object on scanned image. But this example not work, if i put my picture in corner of scanner!

Can anybody advise, how i can find this photo on scanned image? any examples, methods?


Source: (StackOverflow)

How can I get all the points on a line in OpenCvSharp?

There is a line from Point(x1, y1) to Point(x2, y2). And I wanna get all the points on that line.

I know, OpenCv has LineIterator. But I couldn't find how can I use it on C#?

There is sample usage in C++ (For giving an idea):

LineIterator it(img, pt1, pt2, 8);
for(int i = 0; i < it.count; i++, ++it)
{
    Point pt= it.pos(); 
    //Draw Some stuff using that Point pt
}

Thanks in advance.


Source: (StackOverflow)