Using CNNs to recognize Handwritten Math

23 Jun 2026 · Stone Liu

Convolutional Neural Networks

In a previous entry, I built an entire neural network from scratch to recognize digits in the MNIST dataset. This time I want to dive into the exciting world of Convolutional Neural Networks or CNNs.

CNNs are essentially the same thing as the typical artifical neural networks that I have built but are much better suited for 2-dimensional data such as images. Apparently there is an entire labeled dataset called Math Writing for this!

Architecture of a Convolutional Neural Network

A CNN is comprised of three types of layers. These are the convolutional layers, the pooling layers, and the fully connected layers. The input to the CNN typically has

dimensions, regarding width, height, and depth. These I take it to be somewhat vague in terms of what they reference but for example a typical photograph has channels which reflect the RGB Red, Green, and Blue pixel values of a colored image.

1. Convolutional Layer can be thought of relating local areas of source image to the neurons of the output layer.

   • But what is a Convolution? It’s essentially a generalized way to combine two arbitrary functions together. Yeah, not the most helpful explaination, so I want to dig further into this with you!

     • Adding Two Random Variables If we visualize all the possible combinations of rolling two sided die, we get that since each die has possible outcomes, we can pair each number to get .

     • I love the visualization he creates for a convolution. Say we have two arrays of numbers each. The convolution represented as

       can be visualized as follows:

       We first reverse the corresponding elements of like so. Then we shift one block at a time.

       So overall . To double check, numpy actually has a convolution function built in!

       np.convolve((7,8,1), (9,1,7))
       => [63 79 66 57 7]

     This is essentially what the concept of the kernel is doing in the convolutional layers. We are dragging a lense across a set of pixels and extracting some sort of feature we want out of it. Kinda like a moving average.

2. Pooling Layer downsamples on the input which in turn reduces the number of parameters being passed into the activations.
3. Fully Connected Layers will produce class scores (if for classification) from the activations, similar to the standard feedforward neural networks.

Building a Classifier

The task of recognizing handwritten math is a complicated task, since mathematical notation can devolve into more than what is this sequence of characters, I’m going to start out with fine tuning a current pre-trained model called Document Image Transformer. Lets examine and read what this type of model does.

Proof of Concept

To give this a go, I want to first pretrain a convolutional neural network on the Math Writing dataset. The idea is to first rasterize the inkml files into two dimensional grayscale images to be passed as input to our neural network. This is essentially a bigger version of the MNIST classifier that I built previously. Only this time our input is and our output contains hundreds of potential classes to mark the symbols.

Okay so first lets convert all the inkml files in the dataset into two lists, one holds the raw grayscale values for each character input, and the other holds the ground truth label.

def inkml_to_image_and_label(data_path: Path):
    imgs, labels = [], []
    for sym in iter_symbols(data_path):
        if sym.strokes and sym.label:
            imgs.append(rasterize(sym.strokes, size=IMG_SIZE, line_width=STROKE_SIZE))
            labels.append(sym.label)
    return imgs, labels

I have gone for image sizes of and a stroke size of pixels wide.