[MPC Series] Inking: What is it and how do I implement it?

Using pen and paper is a natural activity. We’ve learnt how to use this form of input from the age of 3, granted at that age it was just scribbling with crayons and it was most likely on walls, not paper (I’m guilty here). But still, using pen and paper comes naturally to us which is why it is a key element of More Personal Computing.

What is More Personal Computing

MPC, in short it’s all about context. It’s about using the right input, at the right time for the right job – let’s call this ‘The 3Rs’. It’s also about using the input that feels most natural for the situation that you are in, and putting the user at the centre of it all.

For example: You’re in the car driving to a new location. You’re lost and need directions but you can’t stop the car, and you can’t take your eyes off the road. At this point the best and most natural thing to do would be to just ask “How do I get to x” and receive an update on your SatNav and voiced directions. In this situation you would want to use speech over any other form of interaction and the device you are using should allow you to use speech to interact with it – this is more personal computing.

Inking

A lot of devices now let you use pens to interact with them. We have the Apple Pencil, the Surface Pen, and other forms of stylus but they are hardly taken advantage of. There are a lot of applications where pen input would be more natural for the user, yet there is no option for the user to use pen input. For developers, there are so many SDKs that make this possible and this blog will show an example of an application that uses Inking in UWP.

But today I’m going to focus on how you can take advantage and use the Inking APIs available in Windows 10 UWP apps.

Jots

I started out by building a multi-touch notepad application. There are already plenty of note taking applications out there, but they are all bit over complicated so I set out to build something as simple as possible. So what is Jots? A simple UWP application that you can use to take notes on a never-ending canvas, which also takes everything you write and parses it to text.

CODE!

Before I even start, I feel that I have to apologise for the use of American English in the APIs: you will definitely see me use the word recognise when talk about the code, then later use recognize in the code – I’m sorry – I can’t change this!

Creating an “unlimited” canvas
There really isn’t a way to make a canvas completely unlimited, but you can make it big enough that it feels unlimited by wrapping the ink canvas inside a scroll viewer and making the height and width a stupidly large number. First in Main.xaml, I’ll create a new InkCanvas and wrap it in a ScrollViewer:

<ScrollViewer x:Name="scrollView" HorizontalScrollMode="Enabled" HorizontalScrollBarVisibility="Hidden"  VerticalScrollMode="Enabled" VerticalScrollBarVisibility="Hidden" ZoomMode="Enabled" >
          <InkCanvas x:Name="inkCanvas" Width="600000" Height="600000" />
</ScrollViewer>

Now that we have a canvas, we have to set it up in the code behind (Main.xaml.cs). By default (without this code) Pen is the only input activated on the InkCanvas. The code below is only code you need to make your canvas take in other forms of input. However because I want a multi-touch canvas that you can write on and manipulate, I have commented out the touch input because I want touch to be used to manipulate the canvas, not to write with.

You will need to import “Windows.UI.Core” here.

//Allow Mouse and Pen inputs to the ink canvas (default is pen only).
//Uncomment the "touch" part to enable it as an input for the ink canvas
inkCanvas.InkPresenter.InputDeviceTypes =
    CoreInputDeviceTypes.Mouse |
    //CoreInputDeviceTypes.Touch |
    CoreInputDeviceTypes.Pen;

That’s it. Now you have a “never ending”, multi-touch note taking application.

But there’s so much more we can do, so why stop here?

Ink Recogniser
In the Universal Windows Platform (UWP), there is an API available that allows you to take ink strokes and convert them to text – this is using the InkRecognizer class which can be used by importing “Windows.UI.Input.Inking“. Before going forward, we need to change the UI of our application so we can fit the next few features on-screen. First we need to define a set of rows for the grid so that we can split the grid up and use different rows for different UI controls.

<!-- Split the main grid into 3 different rows, with the middle row being the biggest and taking up all the "left over" space -->
<Grid.RowDefinitions>
    <RowDefinition Height="Auto" />
    <RowDefinition Height="*" />
    <RowDefinition Height="Auto" />
</Grid.RowDefinitions>

<!-- The first row of the grid will be used for a menu at the top -->
<Grid Grid.Row="0" Background="White">
    <StackPanel Grid.Column="0" Orientation="Horizontal" Background="White">
        <!-- We will use this later for buttons -->
    </StackPanel>
</Grid>

<!-- This middle row will be used for the InkCanvas -->
<Grid Grid.Row="1" Grid.ColumnSpan="2">
    <!-- Moved all of the previously written xaml into this middle grid -->
    <ScrollViewer x:Name="scrollView" HorizontalScrollMode="Enabled" HorizontalScrollBarVisibility="Hidden"  VerticalScrollMode="Enabled" VerticalScrollBarVisibility="Hidden" ZoomMode="Enabled" >
        <InkCanvas x:Name="inkCanvas" Width="600000" Height="600000" />
    </ScrollViewer>
</Grid>

<!-- This last row will be used to display the InkRecogniser text results -->
<Grid Grid.Row="2" Margin="1" Background="White">
    <StackPanel Orientation="Horizontal" HorizontalAlignment="Center">
        <TextBlock x:Name="Status" Foreground="Red" Margin="20,20,20,20" FontSize="20" />
    </StackPanel>
</Grid>

Now that the UI is sorted and we have a TextBlock we can use to output the results of the InkRecognizer, we can now create the recogniser in the code. For this feature, we need to import “Windows.UI.Input.Inking“.

Then we create an InkRecognizerContainer and then initialize it within the Main method;

//Outside the main method
InkRecognizerContainer inkRecognizerContainer = null;

//Inside the main method
inkRecognizerContainer = new InkRecognizerContainer();

Then we can use the container to recognise strokes and convert them to text, I did this by creating an async method so that I can reuse the code and call the recogniser every time the user finishes writing something. This can be done through handling the InkPresenter_StrokesCollected event which we first need to register inside the main method:

//Register the StrokeCollected event so that we can recognise the text after each stroke
inkCanvas.InkPresenter.StrokesCollected += InkPresenter_StrokesCollected;

Then create the method to contain all of the ink recognition code and a method to handle the StrokesCollected event:

private async void RecognizeText()
{
    IReadOnlyList<InkStroke> currentStrokes = inkCanvas.InkPresenter.StrokeContainer.GetStrokes();

    if (currentStrokes.Count > 0)
    {
        var recognitionResults = await inkRecognizerContainer.RecognizeAsync(
            inkCanvas.InkPresenter.StrokeContainer,
            InkRecognitionTarget.All);

        if (recognitionResults.Count > 0)
        {
            // Display recognition result
            string str = "";
            foreach (var r in recognitionResults)
            {
                str = r.GetTextCandidates()[0];
            }
            Status.Text = str;
        }
        else
        {
            Status.Text = "No text recognized.";
        }
    }
    else
    {
        Status.Text = "Must first write something.";
    }
}

private void InkPresenter_StrokesCollected(InkPresenter sender, InkStrokesCollectedEventArgs args)
{
    RecognizeText();
}

So what the code above is doing is getting all of the strokes written on the InkCanvas. If strokes are found, we call the InkRecognizerContainer’s RecognizeAsync method in order to get the results (this gives us the text from the InkCanvas!). We then use a simple foreach statement to put the results into a string – this is then used to set the text of the TextBlock “Status” in the UI so users can see the result.

Again very simple right??

Let’s go even further!

Clear, Save, and Load
There is no point in having a note taking application that doesn’t save your notes! So I’ve added three buttons at the top of my application, through yet another StackPanel (I put this code in side the first grid row that was defined previously in this tutorial).

<AppBarButton x:Name="SaveBtn" Icon="Save" Label="Save" Click="OnSaveAsync" />
<AppBarButton x:Name="LoadBtn" Icon="OpenFile" Label="Load" Click="OnLoadAsync" />
<AppBarButton x:Name="Clear" Icon="Refresh" Label="Clear" Click="OnClearClicked" />

Now we can write a bit of code to implement what happens when the above buttons are clicked. For the Save and Load features, we first need to import “Windows.Storage” and “Windows.Storage.Streams

//When the clear button is clicked: clear all of the strokes on the InkCanvas
private void OnClearClicked(object sender, RoutedEventArgs e)
{
    inkCanvas.InkPresenter.StrokeContainer.Clear();
}

//When the load button is clicked: open the windows file picker and wait for the user to select a file via the UI;
//read the file and save it as a stream; then load the file into the InkCanvas
private async void OnLoadAsync(object sender, RoutedEventArgs e)
{
    var openPicker = new Windows.Storage.Pickers.FileOpenPicker();

    openPicker.SuggestedStartLocation = Windows.Storage.Pickers.PickerLocationId.PicturesLibrary;
    openPicker.FileTypeFilter.Add(".gif");
    openPicker.FileTypeFilter.Add(".isf");

    StorageFile file = await openPicker.PickSingleFileAsync();

    if (null != file)
    {
        using (var stream = await file.OpenSequentialReadAsync())
        {
            await inkCanvas.InkPresenter.StrokeContainer.LoadAsync(stream);
        }
    }
}

//When the save button is clicked: open the windows file picker and wait for the user to name and save the file via the UI;
//open the file as a stream; save the InkCanvas strokes into the stream (file)
private async void OnSaveAsync(object sender, RoutedEventArgs e)
{
    //Using if statement so we don't save an empty canvas
    if (inkCanvas.InkPresenter.StrokeContainer.GetStrokes().Count > 0)
    {
        var savePicker = new Windows.Storage.Pickers.FileSavePicker();
        savePicker.SuggestedStartLocation = Windows.Storage.Pickers.PickerLocationId.PicturesLibrary;
        savePicker.FileTypeChoices.Add("Gif with embedded ISF", new System.Collections.Generic.List<string> { ".gif" });

        StorageFile file = await savePicker.PickSaveFileAsync();

        if (null != file)
        {
            using (IRandomAccessStream stream = await file.OpenAsync(FileAccessMode.ReadWrite))
            {
                await inkCanvas.InkPresenter.StrokeContainer.SaveAsync(stream);
            }
        }
    }
}

Okay that’s good. We can now save our notes… and load them back in… And clear the InkCanvas… Now, wouldn’t be a lot better, easier even, to save our notes directly to somewhere meaningful – save to OneNote maybe? I’m not sure about you, but that sounds perfect to me! So how do I do this?

I’m sorry to say you’ll have to wait for part 2. I know, I know, I was just getting to the good part but I haven’t actually implemented this yet, but the moment I do I will definitely post this here. ETA: 1-2 weeks!