How can HMMs be used for handwriting recognition?

Question

The problem is a bit different than traditional handwriting recognition. I have a dataset that are thousands of the following. For one drawn character, I have several sequential (x, y) coordinates where the pen was pressed down. So, this is a sequential (temporal) problem.

I want to be able to classify handwritten characters based on this data, and would love to implement HMMs for learning purposes. But, is this the right approach? How can they be used to do this?

Answer 1

This problem is actually a mix of two problems:

1. recognizing one character from your data
2. recognizing a word from a (noisy) sequence of characters

A HMM is used for finding the most likely sequence of a finite number of discrete states out of noisy measurements. This is exactly problem 2, since noisy measurements of discrete states a-z,0-9 follow eachother in a sequence.

For problem 1, a HMM is useless because you aren't interested in the underlying sequence. What you want is to augment your handwritten digit with information on how you wrote it.

Personally, I would start by implementing regular state-of-the-art handwriting recognition which already is very good (with convolutional neural networks or deep learning). After that, you can add information about how it was written, for example clockwise/counterclockwise.

Answer 2

I think HMM can be used in both problems mentioned by @jens. I'm working on online handwriting too, and HMM is used in many articles. The simplest approach is like this:

1. Select a feature.
2. If selected feature is continuous convert it to discrete.
3. Choose HMM parameters: Topology and # of states.
4. Train character models using HMM. one model for each class.
5. Test using test set.

for each item:

1. the simplest feature is angle of vector which connects consecutive points. You can use more complicated features like angles of vectors obtained by Douglas & Peucker algorithm.
2. the simplest way for discretization is using Freeman codes, but clustering algorithms like k-means and GMM can be used too.
3. HMM topologies: Ergodic, Left-Right, Bakis and Linear. # of states can be obtained by trial & error. HMM parameters can be variable for each model. # of observations is determined by discretization. observation samples can be have variable length.
4. I recommend Kevin Murphy HMM toolbox.
5. Good luck.