While we can’t visualize the high-dimensional representations directly, we can visualize them using dimensionality reduction. To be concrete, let’s use one hidden layer with a hundred sigmoid neurons. The input representation, MNIST, is a collection of 784-dimensional vectors! And, even for a very simple network, we’ll have a high-dimensional hidden representation. The hidden representations we learn are also of very high dimensionality.įor example, suppose we are trying to classify MNIST. The strength of neural networks is classifying high-dimensional data, like computer vision data, which often has tens or hundreds of thousands of dimensions. Unfortunately, neural networks are usually not low-dimensional. (There’s quite a bit we can learn from low-dimensional neural networks, as explored in my post Neural Networks, Manifolds, and Topology.) If we have a question about what it is doing, we can just look. Low-dimensional neural networks are really easy to reason about because we can just look at their representations, and at how one representation transforms into another. The middle “hidden” layer’s representation is a warped, easier to classify, version of the raw data. 1 The input layer’s representation is the raw data. We call the versions of the data corresponding to different layers representations. Given a point on one of the curves, our network should predict which curve it came from. In fact, low-dimensional neural networks – networks which have only two or three neurons in each layer – are quite easy to understand.Ĭonsider the following dataset, consisting of two curves on the plane. Not all neural networks are hard to understand. The combination of neural networks and dimensionality reduction turns out to be a very interesting tool for visualizing high-dimensional data – a much more powerful tool than dimensionality reduction on its own.Īs we dig into this, we’ll observe what I believe to be an important connection between neural networks, visualization, and user interface. Understanding neural networks is just scratching the surface, however, because understanding the network is fundamentally tied to understanding the data it operates on. I think that dimensionality reduction, thoughtfully applied, can give us a lot of traction on understanding neural networks.

Despite this, it remains challenging to understand what, exactly, these networks are doing.

They’ve also brought promising results to many other areas, including language understanding and machine translation. I think these techniques form a set of basic building blocks to try and understand machine learning, and specifically to understand the internal operations of deep neural networks.ĭeep neural networks are an approach to machine learning that has revolutionized computer vision and speech recognition in the last few years, blowing the previous state of the art results out of the water. Trying to visualize high dimensional data is, by itself, very interesting, but my real goal is something else.

In a previous post, we explored techniques for visualizing high-dimensional data. Data visualization, machine learning, word embeddings, neural networks, deep learning, user interface, wikipedia

Visualized learning