Find Image Content and Draw Rectangle Around It

Question

About

I’m using WinForms. In my form, I have a picturebox. The picturebox size mode is set to zoom. I use the picturebox to view TIF images. The TIF images are grayscale (Black and White ONLY).

What My App Does

My application finds the first black pixel and the last black pixel in the document and draws a red rectangle around it. In hopes that it would draw the rectangle around the content of the image.

The Problem

Sometimes the TIF documents have spots/dots around the content of the image. This throws my application off. It doesn't know where the content of the image begins and ends. How can I find the content of the TIF documents and draw a rectangle around it if the document has spots/dots?

About the Document

• Black and white only (1 bit depth)
• TIF document
• Document always has letters and numbers
• The letters and numbers are always bigger than the spots/dots
• There can be multiple spots all over the image even in the content
• The background is always white
• The content is always black
• The spots are also black

---

Download Test Image Links:

• File Dropper: http://www.filedropper.com/test-tifs

• Rapid Share: https://ufile.io/2qiir

---

What I Found

Upon my research, I found AForge.Imaging library which has many imaging filters that may potentially help me achieve my goal. I'm thinking about removing the spots/dots using the median filter or use the other filters to achieve the desired result.

What I Tried

I tried applying the median filter from AForge library to get rid of the spots but that only got rid of some of the spots. I had to repeat replying the filter multiple times to get rid of MOST of the spots to find the content and it still had a hard time finding the content. That method didn't work too well for me.

Link to AForge Filters: http://www.aforgenet.com/framework/docs/html/cdf93487-0659-e371-fed9-3b216efb6954.htm

---

Code

    private void btn_Draw_Click(object sender, EventArgs e)
    {
        // Wrap the creation of the OpenFileDialog instance in a using statement,
        // rather than manually calling the Dispose method to ensure proper disposal
        using (OpenFileDialog dlg = new OpenFileDialog())
        {
            if (dlg.ShowDialog() == DialogResult.OK)
            {
                pictureBox1.Image = new Bitmap(dlg.FileName);

                int xMax = pictureBox1.Image.Width;
                int yMax = pictureBox1.Image.Height;

                startX = Int32.MaxValue;
                startY = Int32.MaxValue;
                endX = Int32.MinValue;
                endY = Int32.MinValue;


                using (Bitmap bmp = new Bitmap(pictureBox1.Image))
                {
                    for (var y = 0;  y < yMax; y+=3)
                    {
                        for (var x = 0; x < xMax; x+=3)
                        {
                            Color col = bmp.GetPixel(x, y);
                            if(col.ToArgb() == Color.Black.ToArgb())
                            {
                                // Finds first black pixel
                                if (x < startX)
                                    startX = x;
                                if(y < startY)
                                    startY = y;

                                // Finds last black pixel
                                if (x > endX)
                                    endX = x;
                                if (y > endY)
                                    endY = y;
                            }
                        }
                    }


                    int picWidth = pictureBox1.Size.Width;
                    int picHeight = pictureBox1.Size.Height;


                    float imageRatio = xMax / (float)yMax; // image W:H ratio
                    float containerRatio = picWidth / (float)picHeight; // container W:H ratio

                    if (imageRatio >= containerRatio)
                    {
                        // horizontal image
                        float scaleFactor = picWidth / (float)xMax;
                        float scaledHeight = yMax * scaleFactor;
                        // calculate gap between top of container and top of image
                        float filler = Math.Abs(picHeight - scaledHeight) / 2;
                        //float filler = 0;

                        startX = (int)(startX * scaleFactor);
                        endX = (int)(endX * scaleFactor);
                        startY = (int)((startY) * scaleFactor + filler);
                        endY = (int)((endY) * scaleFactor + filler);
                    }
                    else
                    {
                        // vertical image
                        float scaleFactor = picHeight / (float)yMax;
                        float scaledWidth = xMax * scaleFactor;
                        float filler = Math.Abs(picWidth - scaledWidth) / 2;
                        startX = (int)((startX) * scaleFactor + filler);
                        endX = (int)((endX) * scaleFactor + filler);
                        startY = (int)(startY * scaleFactor);
                        endY = (int)(endY * scaleFactor);
                    }

                    //var scaleX = picWidth / (float)xMax;
                    //var scaleY = picHeight / (float)yMax;

                    //startX = (int)Math.Round(startX * scaleX);
                    //startY = (int)Math.Round(startY * scaleY);
                    //endX = (int)Math.Round(endX * scaleX);
                    //endY = (int)Math.Round(endY * scaleY);

                }
            }
        }
    }

    private bool _once = true;
    private void pictureBox1_Paint(object sender, PaintEventArgs e)
    {
        if (_once)
        {
            //Rectangle ee = new Rectangle(35, 183, 405, 157);
            Rectangle ee = new Rectangle(startX, startY, endX - startX, endY - startY);
            System.Diagnostics.Debug.WriteLine(startX + ", " + startY + ", " + (endX - startX) + ", " + (endY - startY));
            using (Pen pen = new Pen(Color.Red, 2))
            {
                e.Graphics.DrawRectangle(pen, ee);
            }
            //_once = false;
        }
    }

Tif document that DOES NOT HAVE any spots and dots around content

Tif document that HAS spots and dots around content

Example Image 1:

---

Example Image 2

---

:

Example Image 3

---

Answer 1

A solution could be to find areas of black pixels. When a black pixel is found, check the colour of the neighbouring pixels and create an area of black pixels. When the area is big enough, it can be considered as content. The pseudo code below illustrates this. But it is a very resource intensive solution and should at least be optimized.

    private List<List<Point>> areas = new List<List<Point>>();

    public void PopulateAreas()
    {
        //Find all the areas
        for (var y = 0; y < yMax; y += 3)
        {
            for (var x = 0; x < xMax; x += 3)
            {
                Color col = bmp.GetPixel(x, y);
                if (col.ToArgb() == Color.Black.ToArgb())
                {
                    //Found a black pixel, check surrounding area
                    var area = new List<Point>();
                    area.Add(new Point(x, y));
                    areas.Add(area);
                    AppendSurroundingPixelsToArea(area, x, y);
                }
            }
        }

        var startX = Int32.MaxValue;
        var startY = Int32.MaxValue;
        var endX = Int32.MinValue;
        var endY = Int32.MinValue;
        //Loop through list of areas. 
        foreach (var area in areas)
        {
            //Minimum size of area
            if (area.Count > 5)
            {
                var minx = area.Min(p => p.X);
                if (area.Min(p => p.X) < startX)
                    startX = minx;

                //Do the same for the others...
            }
        }
    }

    public void AppendSurroundingPixelsToArea(List<Point> area, int startX, int startY)
    {
        for(var x = startX - 1; x <= startX + 1; x++)
        for (var y = startY - 1; y <= startY + 1; y++)
        {
            if ((x != 0 || y != 0) && IsBlackPixel(bmp, x, y))
            {
                //Add to the area
                if (PointDoesNotExistInArea(area, x, y))
                {
                    area.Add(new Point(x, y));
                    AppendSurroundingPixelsToArea(area, x, y);
                }
            }
        }
    }

Answer 2

I have solution for this,

And I suggest "kodak imaging professional". This is the viewer to display multi-page tiff files. with many features like: Annotation, invert color, rotate image... etc., and these are inbuilt functionalities.

Answer 3

Following experiment seems to meet all your requirements.

I put following controls onto Form1

A MenuStrip: Docking=Top, with 2 MenuItems - one to open a file, second to run an algorithm

A progressbar: Docking=Top, to watch performance of loading and algorithm

A panel with Docking=Fill and AutoScroll=true

A picture into the panel, Point(0,0), the rest by default. SizeMode=Normal.

Update

using System;
using System.Collections.Generic;
using System.Drawing;
using System.Windows.Forms;

namespace WindowsFormsApplication1
{
    public partial class Form1 : Form
    {
        public Form1()
        {
            InitializeComponent();
        }

        // Opens an image file.
        private void openToolStripMenuItem_Click(object sender, EventArgs e)
        {
            OpenFileDialog dlg = new OpenFileDialog();
            if (dlg.ShowDialog() == System.Windows.Forms.DialogResult.OK)
            {
                this.image = Image.FromFile(dlg.FileName) as Bitmap;
                this.pictureBox1.Image = image;
                this.pictureBox1.Invalidate();
            }
        }

        Bitmap image;

        // finds top, left, right and bottom bounds of the content in TIFF file.
        // 
        private void findBoundsToolStripMenuItem_Click(object sender, EventArgs e)
        {
            int contentSize = 70;

            this.left = 0;
            this.top = 0;
            this.right = this.pictureBox1.Width - 1;
            this.bottom = this.pictureBox1.Height - 1;

            int h = image.Height;
            int w = image.Width;
            this.progressBar1.Value = 0;
            this.progressBar1.Maximum = 4;

            for (int y = 0; y < h; y++)
            {
                for (int x = 0; x < w; x++)
                {

                    if (this.image.GetPixel(x, y).ToArgb() == Black)
                    {
                        int size = this.image.GetBlackRegionSize(x, y);
                        if (this.image.GetBlackRegionSize(x, y) > contentSize)
                        {
                            this.top = y;
                            goto label10;
                        }
                    }

                }
            }

        label10:
            this.progressBar1.Increment(1);
            for (int y = h - 1; y >= 0; y--)
            {
                for (int x = 0; x < w; x++)
                {

                    if (this.image.GetPixel(x, y).ToArgb() == Black)
                    {
                        if (this.image.GetBlackRegionSize(x, y) > contentSize)
                        {
                            this.bottom = y;
                            goto label11;
                        }
                    }
                }
            }

        label11:
            this.progressBar1.Increment(1);
            for (int x = 0; x < w; x++)
            {
                for (int y = 0; y < h; y++)
                {

                    if (this.image.GetPixel(x, y).ToArgb() == Black)
                    {
                        if (this.image.GetBlackRegionSize(x, y) > contentSize)
                        {
                            this.left = x;
                            goto label12;
                        }
                    }
                }
            }

        label12:
            this.progressBar1.Increment(1);
            for (int x = w - 1; x >= 0; x--)
            {
                for (int y = 0; y < h; y++)
                {

                    if (this.image.GetPixel(x, y).ToArgb() == Black)
                    {
                        if (this.image.GetBlackRegionSize(x, y) > contentSize)
                        {
                            this.right = x;
                            goto label13;
                        }
                    }
                }
            }

        label13:
            this.progressBar1.Increment(1);

            this.pictureBox1.Invalidate();
        }

        internal static readonly int Black = Color.Black.ToArgb();
        internal static readonly int White = Color.White.ToArgb();

        int top;
        int bottom;
        int left;
        int right;

        private void pictureBox1_Paint(object sender, PaintEventArgs e)
        {
            if (pictureBox1.Image == null)
            {
                return;
            }

            int xMax = pictureBox1.Image.Width;
            int yMax = pictureBox1.Image.Height;

            int startX = this.left;
            int startY = this.top;
            int endX = this.right;
            int endY = this.bottom;


            int picWidth = pictureBox1.Size.Width;
            int picHeight = pictureBox1.Size.Height;


            float imageRatio = xMax / (float)yMax; // image W:H ratio
            float containerRatio = picWidth / (float)picHeight; // container W:H ratio

            if (imageRatio >= containerRatio)
            {
                // horizontal image
                float scaleFactor = picWidth / (float)xMax;
                float scaledHeight = yMax * scaleFactor;
                // calculate gap between top of container and top of image
                float filler = Math.Abs(picHeight - scaledHeight) / 2;
                //float filler = 0;

                startX = (int)(startX * scaleFactor);
                endX = (int)(endX * scaleFactor);
                startY = (int)((startY) * scaleFactor + filler);
                endY = (int)((endY) * scaleFactor + filler);
            }
            else
            {
                // vertical image
                float scaleFactor = picHeight / (float)yMax;
                float scaledWidth = xMax * scaleFactor;
                float filler = Math.Abs(picWidth - scaledWidth) / 2;
                startX = (int)((startX) * scaleFactor + filler);
                endX = (int)((endX) * scaleFactor + filler);
                startY = (int)(startY * scaleFactor);
                endY = (int)(endY * scaleFactor);
            }

            //if (_once)
            //Rectangle ee = new Rectangle(35, 183, 405, 157);
            Rectangle ee = new Rectangle(startX, startY, endX - startX, endY - startY);
            System.Diagnostics.Debug.WriteLine(startX + ", " + startY + ", " + (endX - startX) + ", " + (endY - startY));
            using (Pen pen = new Pen(Color.Red, 2))
            {
                e.Graphics.DrawRectangle(pen, ee);
            }
            //_once = false;
        }
    }

    static class BitmapHelper
    {
        internal static int GetBlackRegionSize(this Bitmap image, int x, int y)
        {
            int size = 0;
            GetRegionSize(image, new List<Point>(), x, y, 0, ref size);
            return size;
        }

        // this constant prevents StackOverFlow exception.
        // also it has effect on performance.
        // It's value must be greater than the value of contentSize defined in findBoundsToolStripMenuItem_Click(object sender, EventArgs e) method.
        const int MAXLEVEL = 100;

        static void GetRegionSize(this Bitmap image, List<Point> list, int x, int y, int level, ref int size)
        {
            if (x >= image.Width || x < 0 || y >= image.Height || y < 0 || list.Contains(x, y) || image.GetPixel(x, y).ToArgb() != Form1.Black || level > MAXLEVEL)
            {
                return;
            }

            if (size < level)
            {
                size = level;
            }

            list.Add(new Point(x, y));

            image.GetRegionSize(list, x, y - 1, level + 1, ref size);
            image.GetRegionSize(list, x, y + 1, level + 1, ref size);
            image.GetRegionSize(list, x - 1, y, level + 1, ref size);
            image.GetRegionSize(list, x + 1, y, level + 1, ref size);
        }

        static bool Contains(this List<Point> list, int x, int y)
        {
            foreach (Point point in list)
            {
                if (point.X == x && point.Y == y)
                {
                    return true;
                }
            }
            return false;
        }
    }

}

"this.pictureBox1.Size = image.Size;" has been removed. Paint event handler's code changed. PictureBox size mode can be set to Zoom now.

Update 2

I tried to simplify code and increase performance.

using System;
using System.Collections.Generic;
using System.Drawing;
using System.Windows.Forms;

namespace WindowsFormsApplication3
{
    public partial class Form1 : Form
    {
        public Form1()
        {
            InitializeComponent();
            this.pictureBox1.Paint += new PaintEventHandler(this.pictureBox1_Paint);
        }

        // Opens an image file
        // and runs "FindBounds()" method
        private void openToolStripMenuItem_Click(object sender, EventArgs e)
        {
            OpenFileDialog dlg = new OpenFileDialog();
            if (dlg.ShowDialog() == System.Windows.Forms.DialogResult.OK)
            {
                this.image = Image.FromFile(dlg.FileName) as Bitmap;
                FindBounds();
                this.pictureBox1.Image = image;
                this.pictureBox1.Invalidate();
            }
        }

        Bitmap image;

        // Possible maximum side of a spot or a dot in the image
        int maxSpotOrDotSide = 7;

        // Finds top, left, right and bottom bounds of the content in TIFF file.
        private void FindBounds()
        {
            // Possible maximum area of a spot or a dot in the image
            int maxSpotOrDotArea = maxSpotOrDotSide * maxSpotOrDotSide;

            this.left = 0;
            this.top = 0;
            this.right = this.pictureBox1.Width - 1;
            this.bottom = this.pictureBox1.Height - 1;

            int h = image.Height;
            int w = image.Width;
            int num = w * h;

            // Incrementers. I tested with greater values
            // like "x = 2", "x = 5" and it increased performance.
            // But we must be carefull as this may cause skipping content.
            int dx = 1; // Incrementer for "x"
            int dy = 1; // Incrementer for "y"

            // Initialization of "progressBar1"
            this.progressBar1.Value = 0;
            this.progressBar1.Maximum = num;

            // Content of the image
            BlackContent imageContent = null;

            // Here we will scan pixels of the image
            // starting from top left corner and 
            // finishing at bottom right
            for (int y = 0; y < h; y += dx)
            {
                for (int x = 0; x < w; x += dy)
                {
                    int val = y * w + x;
                    this.progressBar1.Value = val > num ? num : val;
                    // This block skips scanning imageContent
                    // thus should increase performance.
                    if (imageContent != null && imageContent.Contains(x, y))
                    {
                        x = imageContent.Right;
                        continue;
                    }

                    // Interesting things begin to happen
                    // after we detect the first Black pixel
                    if (this.image.GetPixel(x, y).ToArgb() == Black)
                    {
                        BlackContent content = new BlackContent(x, y);
                        // Start Flood-Fill algorithm
                        content.FloodFill(this.image);
                        if (content.Area > maxSpotOrDotArea)
                        {
                            if (imageContent == null)
                            {
                                imageContent = content;
                            }
                            imageContent.Include(content.Right, content.Bottom);
                            imageContent.Include(content.Left, content.Top);
                        }
                        else
                        {
                            // Here it's better we increase values of the incrementers.
                            // Depending on size of spots/dots.
                            // It should increase performance.
                            if (dx < content.Width) dx = content.Width;
                            if (dy < content.Height) dy = content.Height;
                        }
                    }

                }
            }

            // Everything is done.
            this.progressBar1.Value = this.progressBar1.Maximum;

            // If image content has been detected 
            // then we save the information
            if (imageContent != null)
            {
                this.left = imageContent.Left;
                this.top = imageContent.Top;
                this.right = imageContent.Right;
                this.bottom = imageContent.Bottom;
            }

            this.pictureBox1.Invalidate();
        }

        internal static readonly int Black = Color.Black.ToArgb();
        internal static readonly int White = Color.White.ToArgb();

        int top;
        int bottom;
        int left;
        int right;

        private void pictureBox1_Paint(object sender, PaintEventArgs e)
        {
            if (pictureBox1.Image == null)
            {
                return;
            }

            int xMax = pictureBox1.Image.Width;
            int yMax = pictureBox1.Image.Height;

            int startX = this.left;
            int startY = this.top;
            int endX = this.right;
            int endY = this.bottom;


            int picWidth = pictureBox1.Size.Width;
            int picHeight = pictureBox1.Size.Height;


            float imageRatio = xMax / (float)yMax; // image W:H ratio
            float containerRatio = picWidth / (float)picHeight; // container W:H ratio

            if (imageRatio >= containerRatio)
            {
                // horizontal image
                float scaleFactor = picWidth / (float)xMax;
                float scaledHeight = yMax * scaleFactor;
                // calculate gap between top of container and top of image
                float filler = Math.Abs(picHeight - scaledHeight) / 2;
                //float filler = 0;

                startX = (int)(startX * scaleFactor);
                endX = (int)(endX * scaleFactor);
                startY = (int)((startY) * scaleFactor + filler);
                endY = (int)((endY) * scaleFactor + filler);
            }
            else
            {
                // vertical image
                float scaleFactor = picHeight / (float)yMax;
                float scaledWidth = xMax * scaleFactor;
                float filler = Math.Abs(picWidth - scaledWidth) / 2;
                startX = (int)((startX) * scaleFactor + filler);
                endX = (int)((endX) * scaleFactor + filler);
                startY = (int)(startY * scaleFactor);
                endY = (int)(endY * scaleFactor);
            }

            //if (_once)
            //Rectangle ee = new Rectangle(35, 183, 405, 157);
            Rectangle ee = new Rectangle(startX, startY, endX - startX, endY - startY);
            System.Diagnostics.Debug.WriteLine(startX + ", " + startY + ", " + (endX - startX) + ", " + (endY - startY));
            using (Pen pen = new Pen(Color.Red, 2))
            {
                e.Graphics.DrawRectangle(pen, ee);
            }
            //_once = false;
        }
    }

    // This class is similar to System.Drawing.Region class
    // except that its only rectangular. 
    // Because all we need is to draw a rectagnle
    // around the image this property must 
    // make it faster, at least I hope so.

    class BlackContent
    {
        internal void FloodFill(Bitmap image)
        {
            FloodFillPrivate(image, this.left + 1, this.top, 0);
        }

        // Legendary Flood-Fill algorithm.
        // Quite often it ends up with StackOverFlow exception.
        // But this class and its rectangularity property
        // must prevent this disaster.
        // In my experiments I didn't encounter incidents.
        void FloodFillPrivate(Bitmap image, int x, int y, int level)
        {
            if (x >= image.Width || x < 0 || y >= image.Height || y < 0 || this.Contains(x, y) || image.GetPixel(x, y).ToArgb() != Form1.Black)
            {
                return;
            }

            this.Include(x, y);

            FloodFillPrivate(image, x, y - 1, level + 1);
            FloodFillPrivate(image, x, y + 1, level + 1);
            FloodFillPrivate(image, x - 1, y, level + 1);
            FloodFillPrivate(image, x + 1, y, level + 1);
        }

        internal BlackContent(int x, int y)
        {
            this.left = x;
            this.right = x;
            this.top = y;
            this.bottom = y;
        }

        internal void Include(int x, int y)
        {
            if (x < this.left)
            {
                this.left = x;
            }
            if (this.right < x)
            {
                this.right = x;
            }
            if (this.bottom < y)
            {
                this.bottom = y;
            }
            if (y < this.top)
            {
                this.top = y;
            }
        }

        internal bool Contains(int x, int y)
        {
            return !(x < this.left || x > this.right || y < this.top || y > this.bottom);
        }

        int left;
        internal int Left { get { return this.left; } }
        int top;
        internal int Top { get { return this.top; } }
        int right;
        internal int Right { get { return this.right; } }
        int bottom;
        internal int Bottom { get { return this.bottom; } }

        internal int Area
        {
            get
            {
                return Width * Height;
            }
        }

        internal int Width
        {
            get
            {
                return (this.right - this.left + 1);
            }
        }

        internal int Height
        {
            get
            {
                return (this.bottom - this.top + 1);
            }
        }
    }

}

I watched the performance with ProgressBar. This one's quite faster. I must also mention that your images are too big.