OpenCV different approach on detecting go board

Question

i am working on an Android app that will recognize a GO board and create a SGF file of it.

i made a version that is able to detect a board and warp the perspective to make it square ( code and example image below) unfortunately it gets a bit harder when adding stones.(image below)

Important things about a average go board:

• round black and white stones
• black lines on the board
• board color ranges from white to light brown and sometimes with a wood grain
• stones are placed on intersections of two lines

correct me if i am wrong but i think my current approach is not a good one. Has somebody a general idea on how i can separate the stones and lines from the rest of the picture?

My code:

    Mat input = inputFrame.rgba(); //original image
    Mat gray = new Mat();          //grayscale image

    //convert image to grayscale
    Imgproc.cvtColor( input, gray, Imgproc.COLOR_RGB2GRAY);

    //try to improve histogram (more contrast)
    equalizeHist(gray, gray);

    //blur image
    Size s = new Size(5,5);
    GaussianBlur(gray, gray, s, 0);

    //apply adaptive treshold 
    adaptiveThreshold( gray, gray, 255, Imgproc.ADAPTIVE_THRESH_GAUSSIAN_C, Imgproc.THRESH_BINARY,11,2);

    //adding secondary treshold, removes a lot of noise
    threshold(gray, gray, 0, 255, Imgproc.THRESH_BINARY + Imgproc.THRESH_OTSU);

Some images:

_{(source: eightytwo.axc.nl)}

_{(source: eightytwo.axc.nl)}

EDIT: 05-03-2016

Yay! managed to detect lines stones and color correctly. precondition the picture has to be only the board itself, without any other background visible.
I use houghLinesP (60lines) and houghCircles (17circles), duration on my phone(1th gen Moto G) about 5 seconds.
Detecting board and warp it turns out to be quite a challenge when it has to be working under different angles and lightning conditions.. still working on that

Suggestions for different approaches are still welcome!!

_{(source: eightytwo.axc.nl)}

EDIT: 15-03-2016

i found a nice way to get line intersects with cross type morphological transformations, works amazing when the picture is taken directly above the board unfortunately not while at an angle (see below)
_{(source: eightytwo.axc.nl)}

In my last update i showed line and stone detection with a picture taken from directly above since then i have been working on detecting the board and warping it in a way that my line and stone detection becomes useful.

harris corner detection
I struggled to get the right parameter settings and i am still not sure if they are optimal, can't find much information on how to optimize image before using harris corners. right now it detects to many corners to be useful. though it feels like it could work. (upper line with pictures in example)

    Mat corners = new Mat();
    Imgproc.cornerHarris(image, corners, 5, 3, 0.03);

    Mat mask = new Mat(corners.size(), CvType.CV_8U, new Scalar(1));
    Core.MinMaxLocResult maxVal = Core.minMaxLoc(corners);

    Core.inRange(corners, new Scalar(maxVal.maxVal * 0.01), new Scalar(maxVal.maxVal), mask);

cross type morphological transformations
works great when picture is taken directly from above, used from an angle or with a rotated board does not work (middle line with pictures in example)

    Imgproc.GaussianBlur(image, image, new Size(5, 5), 0);
    Imgproc.adaptiveThreshold(image, image, 255, Imgproc.ADAPTIVE_THRESH_GAUSSIAN_C, Imgproc.THRESH_BINARY_INV, 11, 2);

    int morph_elem = 1;     //0: Rect - 1: Cross - 2: Ellipse
    int morph_size = 5;

    int morph_operator = 0; //0: Opening - 1: Closing \n 2: Gradient - 3: Top Hat \n 4: Black Hat
    Mat element = getStructuringElement( morph_elem, new Size(2 * morph_size + 1, 2 * morph_size + 1), new Point( morph_size, morph_size ));
    morphologyEx(image, image, morph_operator + 2, element);

contour and houghlines
if there are no stones on the outer boardline and light conditions not to harsh it works pretty well. contours are only part of the board quite often(lower line with pictures in example)

    Imgproc.GaussianBlur(image, image, new Size(5, 5), 0);
    Imgproc.adaptiveThreshold(image, image, 255, Imgproc.ADAPTIVE_THRESH_GAUSSIAN_C, Imgproc.THRESH_BINARY_INV, 11, 2);

    Mat hierarchy = new Mat();
    MatOfPoint biggest     = null;
    int contourId          = 0;
    double biggestArea     = 0;

    double minSize = 2000;
    List<MatOfPoint> contours = new ArrayList<>();

    findContours(InvertedImage, contours, hierarchy, Imgproc.RETR_EXTERNAL, Imgproc.CHAIN_APPROX_SIMPLE);

    //find biggest
    for( int x = 0; x < contours.size() ; x++ ){

        double area = Imgproc.contourArea(contours.get(x));

        if( area > minSize && area > biggestArea ){

            biggestArea = area;
            biggest     = contours.get(x);
            contourId   = x;
        }
    }

providing the right picture all three the methods work but not good enough to be reliable. any thoughts on parameters, image pre-processing, different approaches or anything that might improve the detection are welcome=)

link to picture

EDIT: 31-03-2016

detecting lines and stones is pretty much solved so i will close this question. created a new one for detecting and warping accurately.

anybody interested in my progress: this is my GOSU Snap Alpha channel don't expect to much of it right now!

EDIT: 16-10-2016

Update: i saw some people are still following this question. I tested some more stuff and started using Tensorflow, my neural network looks promising, you can have a look at it here. A lot of work has to be done still, my current image dataset is awful and right now i am working on getting a big dataset.

the app works best using a square board with thick lines and decent lightning.

Answer 1

Assuming that you don't want to "force" your end user to take a cleanest pictures (like using an overlay like some of the QR code scanner for example)

Perhaps you could use some morphological transformations with differents kernels :

• Opening and closing with a rectangular kernel for the lines
• Opening and closing with an ellipse kernel to get the stones (it should be possible to invert the image at some point to get back the white or the black one)

Take a look at http://docs.opencv.org/2.4/doc/tutorials/imgproc/opening_closing_hats/opening_closing_hats.html (sorry this one is in C++ but I think this is almost the same in Java)

I had try these operations to remove a grid from a Sudoku to avoid noise in cell extraction and it worked like a charm.

Let me know of these informations were usefull for you (this is for sure a very interesting case)

Answer 2

I'm working on same program. I avoid finding lines at all. First use perspective transform to get the board into a square as you have done. Find the edges of the 19x19 grid. Then assuming the board is 19x19 you can just compute the position of the lines. This works well for me. Then you compute the closest intersection of the center of the stone to determine which row and col line the stone is on. Works pretty well for me. Only probably is calibrating program for different lighting conditions and different color stones and boards.