UWP Live Handwriting Recognition

I’ve been working on an inking application that I want to use to possibly replace my notebook. One of the key concepts behind my app is to recognise the handwriting live (as the user is writing) in the background of the application. So here’s a quick tutorial on how to transfer handwriting to text without the need for the user to press a button:

Using the StrokesCollected Event

An InkCanvas control is associated with a single instance of an InkPresenter object. Within the InkPresenter class there is an event called “StrokesCollected”.

StrokesCollected is triggered when one or more ink strokes are processed (wet to dry) by the application thread.

In order to recognise the handwriting live, we need to be able to call our recognition method as the strokes are made by the user, through handling the StrokesCollected event we can achieve this. The logic being:

If the StrokesCollectted event has been triggered (meaning the user has written something onto our canvas and it has been processed by the thread), then collect the strokes and pass them in to be recognised.

First we have to register the event, I did this within the MainPage constructor in the MainPage class.


//Register the StrokesCollected event

InkCanvas.InkPresenter.StrokesCollected += InkPresenter_StrokesCollected;

You then create a method to handle the event once it has been called; in this example my method is called “InkPresenter_StrokesCollected”.


 private void InkPresenter_StrokesCollected(InkPresenter sender, InkStrokesCollectedEventArgs args)
 {
    RecognizeText();
 }

Because I want to reuse the code to recognise text, I have a separate method to avoid writing the same code twice:


private async void RecognizeText()
{
    IReadOnlyList currentStrokes = InkCanvas.InkPresenter.StrokeContainer.GetStrokes();

    if (currentStrokes.Count > 0)
    {
        RecogizerName.IsEnabled = false;

        var recognitionResults = await inkRecognizerContainer.RecognizeAsync(
            InkCanvas.InkPresenter.StrokeContainer,
            InkRecognitionTarget.All);

        if (recognitionResults.Count > 0)
        {
            // Display recognition result
            string str = "";
            foreach (var r in recognitionResults)
            {
                str = r.GetTextCandidates()[0];
            }
            Status.Text = str;
        }
        else
        {
            Status.Text = "No text recognized.";
        }

        RecogizerName.IsEnabled = true;
    }
    else
    {
        Status.Text = "Must first write something.";
    }
}

That’s it! Now every time the user makes a stroke, the application will start to recognise what they have written without them having to press a button. If you know a better way of achieving this, please let me know!

You can see the full code for my ongoing project here: https://github.com/liliankasem/Jots/