Max Pechyonkin

Why Swift May Be the Next Big Thing in Deep Learning

Mon, 27 May 2019 00:00:00 +0000

Note: this article is also available on Medium.

Introduction

If you are into programming, when you hear Swift, you will probably think about app development for iOS or MacOS. If you’re into deep learning, then you must have heard about Swift for Tensorflow (abbreviated as S4TF). Then, you can ask yourself: “Why would Google create a version of TensorFlow for Swift? There are already versions for Python and C++; why add another language?” In this post, I will try to answer this question and outline the reasons why you should carefully follow S4TF as well as the Swift language itself. The goal of this post is not to give very detailed explanations but to provide a general overview with plenty of links so that you can go and dig deeper if you get interested.

🧠 Swift Has Strong Support Behind It

Swift was created by Chris Lattner when he was working at Apple. Now, Chris Lattner works at Google Brain, one of the best artificial intelligence research teams in the world. The fact that the creator of the Swift language now works at a lab researching deep learning should tell you that this is a serious project.

Some time ago, people at Google realized that even though Python is an excellent language, it has many limitations that are hard to overcome. A new language was needed for TensorFlow, and after long deliberation, Swift was chosen as a candidate. I will not go into details here, but there is a document that describes drawbacks of Python and what other languages were considered and how eventually it was narrowed down to Swift.

💪 Swift for TensorFlow is Much More Than Just a Library

Swift for TensorFlow is not just TF for another language. It is essentially another branch (in git terms) of the Swift language itself. This means that S4TF is not a library; it is a language in its own right, with features built into it that support all functionality needed for TensorFlow. For example, S4TF has very powerful automatic differentiation system in it, which is one of the foundations of deep learning needed for calculating gradients. Contrast this with Python, where automatic differentiation is not a core component of the language. Some of the features developed initially as part of S4TF were later integrated into the Swift language itself.

⚡️ Swift is Fast

When I first learned that Swift runs as fast as C code, I was astonished. I knew that C was highly optimized and allowed to achieve very high speed, but that comes at the cost of micro-managing memory, which leads to C being not memory safe). Besides, C is not a language that is very easy to learn.

Now, Swift runs as fast as C in numerical computation, and it doesn’t have memory safety issues, and it is much easier to learn. LLVM compiler behind Swift is very powerful and has very efficient optimizations that will ensure your code will run very fast.

📦 You Can Use Python, C and C++ Code in Swift

Since Swift for machine learning is at a very early stage of its life, this means there are not many machine learning libraries for Swift. You shouldn’t worry about it too much, because Swift has amazing Python interoperability. You simply import any Python library in Swift, and it just works. Similarly, you can import C and C++ libraries into Swift (for C++, you need to make sure that header files are written in plain C, without C++ features).

To summarize, if you need specific functionality, but it is not implemented in Swift yet, you can import corresponding Python, C or C++ package. Impressive!

⚙️ Swift Can Go Very Low-Level

If you ever used TensorFlow, most likely you did it through a Python package. Underneath the hood, Python version of TensorFlow library has C code underneath. So when you call any function in TensorFlow, at some level you hit some C code. This means there is a limit of how low you can go trying to inspect the source code. For example, if you want to see how convolutions are implemented, you won’t be able to see Python code for that, because that’s implemented in C.

In Swift, that is different. Chris Lattner called Swift “syntactic sugar for LLVM [assembly language]”. This means that in essence, Swift sits very close to the hardware, and there are no other layers of code written in C in between. This also means that the Swift code is very fast, as was described above. It all leads to you as developer being able to inspect code from a very high level to a very low level, without the need to go into C.

📈 What’s Next

Swift is only one part of the innovation in deep learning happening at Google. There is another component that is very closely related: MLIR, which stands for Multi-Level Intermediate Representation. MLIR will be Google’s unifying compiler infrastructure, allowing to write code in Swift (or any other supported language) and compile it to any supported hardware. Currently, there are a plethora of compilers for different target hardware, but MLIR will change that, allowing not only for code reusability but also for writing custom low-level components of the compiler. It will also allow researchers to apply machine learning to optimize low-level algorithms:

While MLIR acts as a compiler for ML, we also see it enabling the use of machine learning techniques within compilers as well! This is particularly important as engineers developing numerical libraries do not scale at the same rate as the diversification of ML models or hardware.

Imagine being able to use deep learning to help optimize low-level memory tiling algorithms on data (a task similar to what Halide is trying to accomplish). Moreover, this is only the beginning and other creative applications of machine learning in compilers away!

Summary

If you are into deep learning, then Swift is a language you should probably start learning. It brings many advantages as compared to Python. Google is investing heavily into making Swift a key component of its TensorFlow ML infrastructure, and it is very likely Swift will become the language of deep learning. So, starting to get involved with Swift early will give you first-mover advantage.

Links for Further Exploration

Key Deep Learning Architectures - ZFNet

Mon, 28 Jan 2019 00:00:00 +0000

ZFNet [2013, paper by Zeiler et al.]

Main ideas

Deconvolution, transfer learning (supervised pretraining is the term used in the paper), learning rate annealing with momentum

Why it is important

This architecture won the ImageNet competition in 2013, achieving the error rate of 14.8% (compared to 15.4% in the previous year).

Besides, this paper introduced a way to visualize learned convolutional features. Before this work, convolutional neural networks were pretty much black boxes, whose internal workings were unknown. This paper offered essential insights into how convolutional neural networks are learning internal representations. To achieve this goal, the authors introduce a way to map learned features into input pixel space by using a specially designed Deconvolutional Network.

Brief description

Deconvolution

Before I get into the application of deconvolutions in this paper, first I would like to note that you should not confuse deconvolutions in deep learning with deconvolutions in signal processing, they are very different. In deep learning, deconvolutions are also called transposed convolution and partially strided convolutions. Below, I explain why the term transposed is used.

Convolution allows going from a specified input dimension to some output dimension. Note that the shape of the output dimension is defined by parameters of a given convolution: input size, kernel size, strides, and padding. Deconvolution (transposed convolution) allows to go in the opposite direction: from the output dimension go back to the input dimension. Why would such operation be necessary? There are two reasons:

deconvolution is used during backpropagation of error through the convolutional layer
deconvolution is used for upscaling of input in specific deep learning applications such as superresolution and hourglass networks, to name a few.

Backpropagation for the convolutional layer is a deconvolution operation applied to the incoming gradient of the convolutional layer. If you look at convolution operation as the matrix multiplication of input by a convolution matrix $C$ (which is defined by the convolution kernel weights), then backpropagating the error is equivalent to multiplying the incoming gradient by the transpose of the convolution matrix $C$. This is why deconvolution is also called transposed convolution. Note that multiplying by the transposed convolution matrix, $C^T$, is equivalent to deconvolution with filters rotated by 180 degrees.

As for using deconvolutions in upscaling, you perform deconvolution but learn new filters, so that convolutional filters from preceding layers don’t need to be reused.

Deconvolution Network for Learned Features Visualizations

Visualizations produced are reconstructed patterns from an input image coming from the validation set that cause high activations in a given feature map.

To produce input images that result in maximum activations, a separate Deconvnet is attached to the output of each layer whose features we want to visualize. Each Deconvnet itself is a neural network whose input is activations of a particular layer, and the output is an image depicting pixels that are responsible for maximum activation of a convolutional kernel of our interest for a given convolutional layer. In the paper, Deconvnet is described as a sequence of transposed convolutions, de-pooling and a special modified ReLU. The sequence of operations is the reverse of the steps that were used in the original neural network to produce a particular layer’s output. For example, let’s imagine we are interested in visualizing conv3 layer of some neural net. The computation below is performed to get to the output of the layer conv3:

input image -> conv1 -> relu -> conv2 -> relu -> pool -> conv3

Deconvnet attached to the output, will perform operations in reverse:

de-conv3 -> de-pool -> relu* -> de-conv2 -> relu* -> de-conv1 -> output image

The output of the Deconvnet has the same dimensions as the input image in the convnet we are visualizing.

Note that de-conv stands for deconvolution, this operation is also sometimes called transposed convolution. It uses transposed convolutional kernels of the original network that we want to visualize.

De-pooling operation (de-pool) is expanding dimensionality of data by remembering the pooled positions in the forward pass of the network we want to visualize.

Modified ReLU (relu*) only passes forward positive activation, this is equivalent to backpropagating only positive gradients, which is different from backpropagating through a regular ReLU. This is why the authors call it modified.

The paper is not very clear about why all these steps are necessary. After some research, I found another way to look at the Deconvnet: it is just a modified backpropagation step of the original convnet we are visualizing. Let’s get back to the example from above:

input image -> conv1 -> relu -> conv2 -> relu -> pool -> conv3

Backward pass for this computation looks like this:

backprop conv3 -> backprop pool -> backprop conv2 -> backprop relu -> backprop conv1 -> gradients of the input image

Now, backprop through a convolutional layer is just a deconvolution with the kernel rotated by 180 degrees (transposed convolution), so we have backprop conv = de-conv. Backprop through pooling is the same as de-pool as defined in the paper. Lastly, instead of using standard backpropagation through ReLU, the authors use modified ReLU, that only passes the positive activations. So, the backprop step now looks like this:

de-conv3 -> de-pool -> relu* -> de-conv2 -> relu* -> de-conv1 -> output image

To conclude, the Deconvnet is a slightly modified backward pass through a convnet that you are trying to visualize whose output is an image that produces the maximum activation of a selected convolutional kernel for a selected convolutional layer. That image lets you understand what kind of feature this kernel learned to recognize.

Understanding Features Visualizations

Examples of visualized features for layers 1 and 2. Note the size of activations projections and image patches are tiny for layer 1 and larger for layer 2. This has to do with different receptive fields of neurons in different layers. More on that below.

Examples of visualized features for layer3. Note the size of activations projections and image patches are somewhat small, but larger than in layers 1 and 2, and more complex.

The visualizations above are fascinating as they offer insights into the internal representation of knowledge learned by the network. It is essential to understand what precisely do they tell us. The images in the left side of the figures are projections from the highest activating neuron in a particular feature map to the pixel space of the input utilizing modified backpropagation outlined above. The images in the right side of the figure are parts of input images that produce a given projection on the left. In other words, visualization of activations is produced for each image separately.

Another critical thing to note is that the size of each of these projections is equal to each neuron’s receptive field upon the input image. Note that neurons in layers closer to the image have a smaller receptive field on the input image, and the further you go away from the image, the larger the receptive field becomes. I made a toy example in Excel that demonstrates that idea in figures below.

In this toy example, each neuron in the first convolutional layer looks at a 3 by 3 pixels patch of the input image.

When you go to the next convolutional layer, each neuron looks look at a 5 by 5 pixels patch of the input image. Conclusion: the farther the layer is from the input, the larger the receptive field.

Now that you understand receptive fields recall that each convolution kernel slides across the input space and for each position it produces 1 activation. As a result, we have a 2D feature map for this kernel, consisting of numbers representing neuron activations. We can look and find the highest activation, and then we can trace back receptive fields through the network and find at which patch of the input image this particular highly activated neuron is looking. These patches are on the right side of the panel of each layer.

Examples of visualized features for layers 4 and 5. Note the size of activations projections and image patches are larger than those of layer 3. The complexity of detected entities is also higher than in previous layers.

Lastly, it is worth mentioning that the higher you go in the network (further away from the input), the more complex representations become. They increase complexity by combining more straightforward features from layers below. For example, if layer 2 detects circles, then layer 5 uses those detected circles when detecting god faces because dog eyes look like circles.

Architecture

ZFNet architecture. Source: original paper.

Overall, the architecture of the network is an optimized version of the last year’s winner - AlexNet (you can read the detailed review here). The authors spent some time to find out the bottlenecks of AlexNet and removing them, achieving superior performance.

(a): 1st layer ZFNET features without feature scale clipping. (b): 1st layer features from AlexNet. Note that there are a lot of dead features - ones where the network did not learn any patterns. (c): 1st layer features for ZFNet. Note that are only a few dead features. (d): 2nd layer features from AlexNet. The grid-like patterns are so-called aliasing artifacts. They appear when receptive fields of convolutional neurons overlap and neighboring neurons learn similar structure. (e): 2nd layer features for ZFNet. Note that there are no aliasing artifacts. Source: original paper.

In particular, they reduced filter size in the first convolutional layer from 11x11 to 7x7, which resulted in fewer dead features learned in the first layer (see the image below for an example of that). A dead feature is a situation where a convolutional kernel fails to learn any significant representation. Visually it looks like a monotonic single-color image, where all the values are close to each other.

In addition to changing the filter size, the authors of FZNet have doubled the number of filters in all convolutional layers and the number of neurons in the fully connected layers as compared to the AlexNet. In the AlexNet, there were 48-128-192-192-128-2048-2048 kernels/neurons, and in the ZFNet all these doubled to 96-256-384-384-256-4096-4096. This modification allowed the network to increase the complexity of internal representations and as a result, decrease the error rate from 15.4% for last year’s winner, to 14.8% to become the winner in 2013.

To summarize, it was not a revolutionary change, but rather evolutionary, building upon the ideas of the previous year’s winner.

Additional readings

Intuitively Understanding Convolutions for Deep Learning - pay attention to “Convolutions are still linear transforms” section, where a convolution is represented as a matrix-matrix multiplication (using so-called Toeplitz matrix)
Stanford’s CS231n Winter 2017 Lecture 12
A guide to convolutional arithmetic (with animations)
Demystifying Convolution in Popular Deep Learning Framework — Caffe - low-level operations explained

Return to all architectures

Deep Learning Vision for Non-Vision Tasks

Mon, 21 Jan 2019 00:00:00 +0000

Introduction

Note: this post was also published in Towards Data Science.

In recent years, deep learning has revolutionized computer vision. And thanks to transfer learning and amazing learning resources, anyone can start getting state of the art results within days and even hours, by using a pre-trained model and adapting it to your domain. As deep learning is becoming commoditized, what is needed is its creative application to different domains.

Today, deep learning in computer vision has largely solved visual object classification, object detection, and recognition. In these areas, deep neural networks outperform human performance.

Even if your data is not visual, you can still leverage the power of these vision deep learning models, mostly CNNs. To do that, you have to transform your data from the non-vision domain into images and then use one of the models trained on images with your data. You will be surprised how powerful this approach is!

In this post, I will present 3 cases where companies used deep learning creatively, applying vision deep learning models to non-vision domains. In each of these cases, a non-computer vision problem was transformed and stated in such a way as to leverage the power of a deep learning model suitable for image classification.

Case 1: Oil Industry

Beam pumps are often used in the oil industry to extract oil and gas from under the ground. They are powered by an engine connected to a walking beam. The walking beam transfers rotational motion of the engine to the vertical reciprocating motion of the sucker rod that acts as a pump and transfers oil to the surface.

A walking beam pump, also known as pumpjack. Source.

As any complex mechanical system, beam pumps are prone to failures. To help with diagnostics, a dynamometer is attached to the sucker for the purpose of measuring the load on the rod. After measuring it is then plotted to produce a dynamometer pump card that shows the load across parts of the rotation cycle of the engine.

An example dynamometer card. Source.

When something goes wrong in the beam pump, dynamometer cards will change their shape. Often times an expert technician will be invited to examine the card and make a judgment call about which part of the pump is malfunctioning and what is needed to be done to fix it. This process is time-consuming and requires very narrow expertise to be solved efficiently.

On the other hand, this process looks like it could be automated, this is why classical machine learning systems were tried but did not achieve good results, around 60% accuracy.

One of the companies that applied deep learning to this domain is Baker Hughes¹. In their case, dynamometer cards were converted to images and then used as inputs to an Imagenet-pretrained model. Results were very impressive - accuracy went up from 60% to 93% by just taking a pretrained model and finetuning it with new data. After further optimizations of model training, they were able to achieve an accuracy of 97%.

An example of a system deployed by Baker Hughes. On the left, you can see the input image, and on the right is a real-time classification of failure mode. The system runs on a portable device, and classification time is shown in the lower right corner. Source.

Not only did it beat previous classical machine learning based methods, the company now could be more efficient by not needing beam pump technicians to spend time trying to diagnose a problem. They could come and start fixing mechanical failures immediately.

To learn more, you can also read a paper that discusses a similar approach².

Case 2: Online Fraud Detection

Computer users have unique patterns and habits when they use a computer. The way you use your mouse when you browse a website or type at a keyboard when composing an email is unique.

In this particular case, Splunk solved a problem³ of classifying users by using the way they use a computer mouse. If your system can uniquely identify users based on mouse usage patterns, then this can be used in fraud detection. Imagine the following situation: fraudsters steal someone’s login and password and then use them to log in and make a purchase at an online store. The way they use computer mouse is unique to them and the system will easily detect this anomaly and prevent fraudulent transactions from taking place, and also notify the real account owner.

Using a special Javascript code, all mouse activity can be collected. The software records mouse activity every 5-10 milliseconds. As result, data for each user can be 5000-10000 data points per user per page. The data represents two challenges: the first one is that this is a lot of data for each user and the second is that each user’s data set will contain a different number of data points, which is not very convenient because usually, sequences of different lengths require more sophisticated deep learning architectures.

The solution was to convert each user’s mouse activity on each web page into a single image. In each image, mouse movements are represented by a line whose color encodes mouse speed and left and right clicks are represented by green and red circles. This way of processing initial data solves both problems: first of all, all images are of the same size, and secondly, now image-based deep learning models can be used with this data.

In each image, mouse movements are represented by a line whose color encodes mouse speed and left and right clicks are represented by green and red circles. Source.

Splunk used TensorFlow + Keras to build a deep learning system for classifying users. They performed 2 experiments:

Group classification of users of a financial services website - regular customers vs. non-customers while accessing similar pages. A relatively small training dataset of 2000 images. After training a modified architecture based on VGG16 for only 2 minutes, the system was able to recognize these two classes with above 80% accuracy.
Individual classification of users. The task is for a given user make a prediction whether it is this user or an impersonator. A very small training dataset of only 360 images. Based on VGG16 but modified to take account of the small dataset and reduce overfitting (probably dropout and batch normalization). After 3 minutes of training achieved an accuracy of about 78%, which is very impressive considering the very challenging nature of the task.

To read more, please refer to the full article describing the system and experiments.

Case 3: Acoustic Detection of Whales

In this example, Google used convolutional neural networks to analyze audio recordings and detect humpback whales in them⁴. This can be useful for research purposes, such as to track individual whale movements, properties of songs, the number of whales etc. It is not the purpose that is interesting, but how data was processed to be used with a convolutional neural network, which needs images.

The way to convert audio data to an image is by using spectrograms. Spectrograms are visual representations of frequency-based features of audio data.

An example of a spectrogram of a male voice saying "nineteenth century". Source.

After converting audio data to spectrograms, Google researchers used a ResNet-50 architecture for training the model. They were able to achieve the following performance:

90% precision: 90% of all audio clips classified as whale songs are classified
90% recall: given an audio recording of a whale song, there is 90% chance it will be labeled as such.

This result is very impressive and will definitely help whale researches.

Let’s switch focus from whales to what you can do when working with your audio data. When creating a spectrogram, you can select frequencies to be used, and that will depend on the type of audio data that you have. You will want different frequencies for human speech, humpback whale songs, or industrial equipment recordings because in all these cases most important information is contained in different frequency bands. You will have to use your domain knowledge to select that parameter. For example, if you are working with human speech data, then your first choice should be a mel-frequency cepstrum spectrogram.

There are good packages to work with audio. Librosa is a free audio-analysis Python library that can produce spectrograms using CPU. If you are developing in TensorFlow and want to do spectrogram computation on the GPU, that is also possible.

Please refer to the original Google AI blog article to learn more about how Google worked with humpback whale data.

Summary

To summarize, the general approach outlined in this post follows two steps. First, find a way to convert your data into images and second, use a pretrained convolutional network or train one from scratch. The first step is harder then the second, this is where you have to be creative and think if the data you have can be converted to images. I hope that the examples I provided can be useful for solving your problem. If you have other examples or questions, please write them in the comments below.

References

Key Deep Learning Architectures - LeNet-5

Tue, 02 Oct 2018 00:00:00 +0000

LeNet-5 architecture. Source.

Architecture

LeNet-5 [1998, paper by LeCun et al.]

Main Ideas

convolution, local receptive fields, shared weights, spacial subsampling

Why it is Important

LeNet-5 was used on large scale to automatically classify hand-written digits on bank cheques in the United States. This network is a convolutional neural network (CNN). CNNs are the foundation of modern state-of-the art deep learning-based computer vision. These networks are built upon 3 main ideas: local receptive fields, shared weights and spacial subsampling. Local receptive fields with shared weights are the essence of the convolutional layer and most architectures described below use convolutional layers in one form or another.

Another reason why LeNet is an important architecture is that before it was invented, character recognition had been done mostly by using feature engineering by hand, followed by a machine learning model to learn to classify hand engineered features. LeNet made hand engineering features redundant, because the network learns the best internal representation from raw images automatically.

Brief Description

By modern standards, LeNet-5 is a very simple network. It only has 7 layers, among which there are 3 convolutional layers (C1, C3 and C5), 2 sub-sampling (pooling) layers (S2 and S4), and 1 fully connected layer (F6), that are followed by the output layer. Convolutional layers use 5 by 5 convolutions with stride 1. Sub-sampling layers are 2 by 2 average pooling layers. Tanh sigmoid activations are used throughout the network. There are several interesting architectural choices that were made in LeNet-5 that are not very common in the modern era of deep learning.

First, individual convolutional kernels in the layer C3 do not use all of the features produced by the layer S2, which is very unusual by today’s standard. One reason for that is to made the network less computationally demanding. The other reason was to make convolutional kernels learn different patterns. This makes perfect sense: if different kernels receive different inputs, they will learn different patterns.

Second, the output layer uses 10 Euclidean Radial Basis Function neurons that compute L2 distance between the input vector of dimension 84 and manually predefined weights vectors of the same dimension. The number 84 comes from the fact that essentially the weights represent a 7x12 binary mask, one for each digit. This forces network to transform input image into an internal representation that will make outputs of layer F6 as close as possible to hand-coded weights of the 10 neurons of the output layer.

LeNet-5 was able to achieve error rate below 1% on the MNIST data set, which was very close to the state of the art at the time (produced by a boosted ensemble of three LeNet-4 networks).

Additional Readings

Understanding Neural Networks Using Excel
Convolutional Neural Networks (CNNs / ConvNets)
Paper: “A guide to convolution arithmetic for deep learning”
Visual explanation of convolution kernels (which are also used also in image processing)
Convolution animations GIFs
Probability concepts explained: Maximum likelihood estimation

Return to all architectures

Note: this post was originally published on Medium.

Key Deep Learning Architectures for Visual Object Recognition

Tue, 02 Oct 2018 00:00:00 +0000

This series provides an overview of some of the prominent neural network architectures. Reading through this guide and all supplemental materials should help you develop an understanding of the modern neural network architectures and the main ideas behind them.

Before starting, you should have some familiarity with the basics of neural networks, backpropagation algorithm, and gradient descent. To learn these, I recommend two amazing courses:

“Practical Deep Learning For Coders” from fast.ai
“CS231n: Convolutional Neural Networks for Visual Recognition” from Stanford

List of architectures in this guide

Sorted in chronological order.

LeNet (1998)
AlexNet (2012)
ZFNet (2013)

Coming soon:

GoogLeNet (2014)
Inception (2014)
VGG (2014)
InceptionV2, InceptionV3 (2015)
ResNet (2015)
InceptionV4 (2016)
DenseNet (2016)
Xception (2016)
MobileNet (2017)
NASNet (2017)
SE-ResNet (2017)
MobileNetV2 (2018)

Key Deep Learning Architectures - AlexNet

Tue, 02 Oct 2018 00:00:00 +0000

Architecture

AlexNet [2012, paper by Krizhevsky et al.]

Main Ideas

ReLU nonlinearity, training on multiple GPUs, local response normalization, overlapping pooling, data augmentation, dropout

Why it is Important

AlexNet won the ImageNet competition in 2012 by a large margin. It was the biggest network at the time. The network demonstrated the potential of training large neural networks quickly on massive datasets using widely available gaming GPUs; before that neural networks had been trained mainly on CPUs. AlexNet also used novel ReLU activation, data augmentation, dropout and local response normalization. All of these allowed to achieve state-of-the art performance in object recognition in 2012.

Brief Description

ReLU Nonlinearity

The benefits of ReLU (excerpt from the paper).

ReLU is a so-called non-saturating activation. This means that gradient will never be close to zero for a positive activation and as result, the training will be faster.

By contrast, sigmoid activations are saturating, which makes gradient close to zero for large absolute values of activations. Very small gradient will make the network train slower or even stop, because the step size during gradient descent’s weight update will be small or zero (so-called vanishing gradient problem).

By employing ReLU, training speed of the network was six times faster as compared to classical sigmoid activations that had been popular before ReLU. Today, ReLU is the default choice of activation function.

Local Response Normalization

Local response normalization formula from the paper. Color labeling is mine.

After layers C1 and C2, activities of neurons were normalized according to the formula above. What this did is scaled the activities down by taking into account 5 neuron activities at preceding and following feature channels at the same spatial position.

An example of local response normalization made in Excel by me.

These activities were squared and used together with parameters $n$, $k$, $\alpha$ and $\beta$ to scale down each neuron’s activity. Authors argue that this created “competition for big activities amongst neuron outputs computed using different kernels”. This approach reduced top-1 error by 1%. In the table above you can see an example of neuron activations scaled down by using this approach. Also note that the values of $n$, $k$, $\alpha$ and $\beta$ were selected using cross-validation.

Overlapping Pooling

Overlapping pooling of the kind used by AlexNet. Source.

AlexNet used max pooling of size 3 and stride 2. This means that the largest values were pooled from 3x3 regions, centers of these regions being 2 pixels apart from each other vertically and horizontally. Overlapping pooling reduced tendency to overfit and also reduced test error rates by 0.4% and 0.3% (for top-1 and top-5 error correspondingly).

Data Augmentation

Data augmentation is a regularization strategy (a way to prevent overfitting). AlexNet uses two data augmentation approaches.

The first takes random crops of input images, as well as rotations and flips and uses them as inputs to the network during training. This allows to vastly increase the size of the data; the authors mention the increase by the factor of 2048. Another benefit is the fact that augmentation is performed on the fly on CPU while the GPUs train previous batch of data. In other words, this type of augmentation is essentially computationally free, and also does not require to store augmented images on disk.

The second data augmentation strategy is so-called PCA color augmentation. First, PCA on all pixels of ImageNet training data set is performed (a pixel is treated as a 3-dimensional vector for this purpose). As result, we get a 3x3 covariance matrix, as well as 3 eigenvectors and 3 eigenvalues. During training, a random intensity factor based on PCA components is added to each color channel of an image, which is equivalent to changing intensity and color of illumination. This scheme reduces top-1 error rate by over 1% which is a significant reduction.

Test Time Data Augmentation

The authors do not explicitly mention this as contribution of their paper, but they still employed this strategy. During test time, 5 crops of original test image (4 corners and center) are taken as well as their horizontal flips. Then predictions are made on these 10 images. Predictions are averaged to make the final prediction. This approach is called test time augmentation (TTA). Generally, it does not need to be only corners, center and flips, any suitable augmentation will work. This improves testing performance and is a very useful tool for deep learning practitioners.

Dropout

AlexNet used 0.5 dropout during training. This means that during forward pass, 50% of all activations of the network were set to zero and also did not participate in backpropagation. During testing, all neurons were active and were not dropped. Dropout reduces “complex co-adaptations” of neurons, preventing them to depend heavily on other neurons being present. Dropout is a very efficient regularization technique that makes the network learn more robust internal representations, significantly reducing overfitting.

Architecture

AlexNet architecture from paper. Color labeling is mine.

Architecture itself is relatively simple. There are 8 trainable layers: 5 convolutional and 3 fully connected. ReLU activations are used for all layers, except for the output layer, where softmax activation is used. Local response normalization is used only after layers C1 and C2 (before activation). Overlapping max pooling is used after layers C1, C2 and C5. Dropout was only used after layers F1 and F2.

Due to the fact that the network resided on 2 GPUs, it had to be split in 2 parts that communicated only partially. Note that layers C2, C4 and C5 only received as inputs outputs of preceding layers that resided on the same GPU. Communication between GPUs only happened at layer C3 as well as F1, F2 and the output layer.

The network was trained using stochastic gradient descent with momentum and learning rate decay. In addition, during training, learning rate was decreased manually by the factor of 10 whenever validation error rate stopped improving.

Additional Readings

Paper: Rectified Linear Units Improve Restricted Boltzmann Machines
Paper: Dropout: A Simple Way to Prevent Neural Networks from Overfitting
Quora: Why are GPUs well-suited to deep learning?
Why are GPUs necessary for training Deep Learning models?
Data Augmentation: How to use Deep Learning when you have Limited Data.
Color intensity data augmentation: Fancy PCA (Data Augmentation) with Scikit-Image
PCA Color Augmentation
Since PCA in the paper is done on the whole entirety of ImageNet data set (or maybe subsample, but that is not mentioned), the data most probably will not fit in memory. In that case, incremental PCA may be used that performs PCA in batches. This thread is also useful in explaining how to do partial PCA without loading the whole data in memory.
Test time augmentation

Return to all architectures

Note: this post was originally published on Medium.

Stochastic Weight Averaging — a New Way to Get State of the Art Results in Deep Learning

Sat, 28 Apr 2018 00:00:00 +0000

In this article, I will discuss two interesting recent papers that provide an easy way to improve performance of any given neural network by using a smart way to ensemble. They are:

“Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs” by Garipov et. al
“Averaging Weights Leads to Wider Optima and Better Generalization” by Izmailov et. al

Additional prerequisite reading that will make context of this post much more easy to understand:

“Improving the way we work with learning rate” by Vitaly Bushaev

Traditional Ensembling of Neural Networks

Traditional ensembling combines several different models and makes them predict on the same input. Then some way of averaging is used to determine the final prediction of the ensemble. It can be simple voting, an average or even another model that learns to predict correct value or label based on the inputs of models in the ensemble. Ridge regression is one particular way of combining several predictions which is used by Kaggle-winning machine learning practitioners.

Snapshot Ensemble is created by saving a model each time the learning rate cycle is at the end. Then the saved models are used together during prediction. Source.

When applied in deep learning, ensembling can be used to combine predictions of several neural networks to produce one final prediction. Usually it is a good idea to use neural networks of different architectures in an ensemble, because they will likely make mistakes on different training samples and therefore the benefit of ensembling will be larger.

However, you can also ensemble models with the same architecture and it will give surprisingly good results. One very cool trick exploiting this approach was proposed in the snapshot ensembling paper. The authors take weights snapshot while training the same network and then after training create an ensemble of nets with the same architecture but different weights. This allows to improve test performance, and it is a very cheap way too because you just train one model once, just saving weights from time to time.

Snapshot ensemble uses cyclical learning rates with annealing. Source.

You can refer to this awesome post for more details. If you aren’t yet using cyclical learning rates, then you definitely should, as it becomes the standard state-of-the art training technique that is very simple, not computationally heavy and provides significant gains at almost no additional cost.

All of the examples above are ensembles in the model space, because they combine several models and then use models’ predictions to produce the final prediction.

In the paper that I am discussing in this post, however, the authors propose to use a novel ensembling in the weights space. This method produces an ensemble by combining weights of the same network at different stages of training and then uses this model with combined weights to make predictions. There are 2 benefits from this approach:

when combining weights, we still get one model at the end, which speeds up predictions
it turns out, this method beats the current state-of-the art snapshot ensembling

Let’s see how it works. But first we need to understand some important facts about loss surfaces and generalizable solutions.

Solutions in the Weight Space

The first important insight is that a trained network is a point in multidimensional weight space. For a given architecture, each distinct combination of network weights produces a separate model. Since there are infinitely many combinations of weights for any given architecture, there will be infinitely many solutions. The goal of training of a neural network is to find a particular solution (point in the weight space) that will provide low value of the loss function both on training and testing data sets.

During training, by changing weights, training algorithm changes the network and travel in the weight space. Gradient descent algorithm travels on a loss plane in this space where plane elevation is given by the value of the loss function.

Narrow and Wide Optima

It is very hard to visualize and understand the geometry of multidimensional weight space. At the same time, it is very important to understand it because stochastic gradient descent essentially traverses a loss surface in this highly multidimensional space during training and tries to find a good solution — a “point” on the loss surface where loss value is low. It is known that such surfaces have many local optima. But it turns out that not all of them are equally good.

Hinton: “To deal with hyper-planes in a 14-dimensional space, visualize a 3-D space and say “fourteen” to yourself very loudly. Everyone does it.” (source)

One metric that can distinguish a good solution from a bad one is its flatness. The idea being that training data set and testing data set will produce similar but not exactly the same loss surfaces. You can imagine that a test surface will be shifted a bit relative to the train surface. For a narrow solution, during test time, a point that gave low loss can have a large loss because of this shift. This means that this “narrow” solution did not generalize well — training loss is low, while testing loss is large. On the other hand, for a “wide” and flat solution, this shift will lead to training and testing loss being close to each other.

Narrow and wide optima. Flat minimum will produce similar loss during training and testing. Narrow loss, however, will give very different results during training and testing. In other words, wide minimum is more generalizable than narrow. Source.

I explained the difference between narrow and wide solutions because the new method which is the focus of this post leads to nice and wide solutions.

Snapshot Ensembling

Initially, SGD will make a big jump in the weight space. Then, as the learning rate gets smaller due to cosine annealing, SGD will converge to some local solution and the algorithm will take a “snapshot” of the model by adding it to the ensemble. Then the rate is reset to high value again and SGD takes a large jump again before converging to some different local solution.

Cycle length in the snapshot ensembling approach is 20 to 40 epochs. The idea of long learning rate cycles is to be able to find sufficiently different models in the weight space. If the models are too similar, then predictions of the separate networks in the ensemble will be too close and the benefit of ensembling will be negligible.

Snapshot ensembling works really well and improves model performance, but Fast Geometric Ensembling works even better.

Fast Geometric Ensembling (FGE)

Fast geometric ensembling is very similar to snapshot ensembling, but is has some distinguishing features. It uses linear piecewise cyclical learning rate schedule instead of cosine. Secondly, the cycle length in FGE is much shorter — only 2 to 4 epochs per cycle. At first intuition, the short cycle is wrong because the models at the end of each cycle will be close to each other and therefore ensembling them will not give any benefits. However, as the authors discovered, because there exist connected paths of low loss between sufficiently different models, it is possible to travel along those paths in small steps and the models encountered along will be different enough to allow ensembling them with good results. Thus, FGE shows improvement compared to snapshot ensembles and it takes smaller steps to find the model (which makes it faster to train).

LEFT: Traditional intuition is that good local minima are separated by regions of high loss. This is true if we travel along the lines connecting local minima. MIDDLE and RIGHT: However, there exist paths between local minima, such that loss stays low on these paths. FGE takes snapshots along these paths and creates an ensemble out of the snapshots. Source.

To benefit from both snapshot ensembling or FGE, one needs to store multiple models and then make predictions for all of them before averaging for the final prediction. Thus, for additional performance of the ensemble, one needs to pay with higher amount of computation. So there is no free lunch there. Or is there? This is where the new paper with stochastic weight averaging comes in.

Stochastic Weight Averaging (SWA)

Stochastic weight averaging closely approximates fast geometric ensembling but at a fraction of computational loss. SWA can be applied to any architecture and data set and shows good result in all of them. The paper suggests that SWA leads to wider minima, the benefits of which I discussed above. SWA is not an ensemble in its classical understanding. At the end of training you get one model, but it’s performance beats snapshot ensembles and approaches FGE.

LEFT: $W_1$, $W_2$ and $W_3$ represent 3 independently trained networks, $W_{swa}$ is the average of them. MIDDLE: $W_{swa}$ provides superior performance on the test set as compared to SGD. RIGHT: Note that even though $W_{swa}$ shows worse loss during training, it generalizes better. Source.

Intuition for SWA comes from empirical observation that local minima at the end of each learning rate cycle tend to accumulate at the border of areas on loss surface where loss value is low (points $W_1$, $W_2$ and $W_3$ are at the border of the red area of low loss in the left panel of figure above). By taking the average of several such points, it is possible to achieve a wide, generalizable solution with even lower loss ($W_{SWA}$ in the left panel of the figure above).

Here is how it works. Instead of an ensemble of many models, you only need two models:

the first model that stores the running average of model weights ($W_{SWA}$ in the formula). This will be the final model after the end of the training which will be used for predictions.
the second model ($w$ in the formula) that will be traversing the weight space, exploring it by using a cyclical learning rate schedule.

At the end of each learning rate cycle, the current weights of the second model ($w$) will be used to update the weight of the running average model ($w_{SWA}$) by taking weighted mean between the old running average weights and the new set of weights from the second model:

\[w_{SWA} \leftarrow \frac{w_{SWA} \cdot n_{models} + w}{n_{models} + 1}\]

By following this approach, you only need to train one model, and store only two models in memory during training. For prediction, you only need the running average model and predicting on it is much faster than using ensemble described above, where you use many models to predict and then average results.

Implementations

Authors of the paper provide their own implementation in PyTorch.

Also, SWA is implemented in the awesome fast.ai library that everyone should be using. And if you haven’t yet seen their course, then follow the links.

Understanding Hinton’s Capsule Networks. Part 4. CapsNet Architecture.

Wed, 21 Feb 2018 00:00:00 +0000

Part of Understanding Hinton’s Capsule Networks Series:

Part 1: Intuition
Part 2: How Capsules Work
Part 3: Dynamic Routing Between Capsules
Part 4: CapsNet Architecture (you are reading it now)

Introduction

In this part, I will walk through the architecture of the CapsNet. I will also offer my shot at calculating the number of trainable parameters in the CapsNet. My resulting number is around 8.2 million of trainable parameters which is different from the 11.36 officially referred to in the paper. The paper itself is not very detailed and hence it leaves some open questions about specifics of the network implementation that are as of today still unanswered because the authors did not provide their code. Nonetheless, I still think that counting parameters in a network is a good exercise for purely learning purposes as it allows one to practice understanding of all building blocks of a particular architecture.

The CapsNet has 2 parts: encoder and decoder. The first 3 layers are encoder, and the second 3 are decoder:

Layer 1. Convolutional layer
Layer 2. PrimaryCaps layer
Layer 3. DigitCaps layer
Layer 4. Fully connected #1
Layer 5. Fully connected #2
Layer 6. Fully connected #3

Part I. Encoder.

CapsNet encoder architecture. Source: original paper.

Encoder part of the network takes as input a 28 by 28 MNIST digit image and learns to encode it into a 16-dimensional vector of instantiation parameters (as explained in the previous posts of this series), this is where the capsules do their job. The output of the network during prediction is a 10-dimensional vectors of lengths of DigitCaps’ outputs. The decoder has 3 layers: two of them are convolutional and the last one is fully connected.

Layer 1. Convolutional layer

Input: 28x28 image (one color channel).
Output: 20x20x256 tensor.
Number of parameters: 20992.

Convolutional layer’s job is to detect basic features in the 2D image. In the CapsNet, the convolutional layer has 256 kernels with size of 9x9x1 and stride 1, followed by ReLU activation. If you don’t know what this means, here are some awesome resources that will allow you to quickly pick up key ideas behind convolutions. To calculate the number of parameters, we need to also remember that each kernel in a convolutional layer has 1 bias term. Hence this layer has (9x9 + 1)x256 = 20992 trainable parameters in total.

Layer 2. PrimaryCaps layer

Input: 20x20x256 tensor.
Output: 6x6x8x32 tensor.
Number of parameters: 5308672.

This layer has 32 primary capsules whose job is to take basic features detected by the convolutional layer and produce combinations of the features. The layer has 32 “primary capsules” that are very similar to convolutional layer in their nature. Each capsule applies eight 9x9x256 convolutional kernels (with stride 2) to the 20x20x256 input volume and therefore produces 6x6x8 output tensor. Since there are 32 such capsules, the output volume has shape of 6x6x8x32. Doing calculation similar to the one in the previous layer, we get 5308672 trainable parameters in this layer.

Layer 3. DigitCaps layer

Input: 6x6x8x32 tensor.
Output: 16x10 matrix.
Number of parameters: 1497600.

This layer has 10 digit capsules, one for each digit. Each capsule takes as input a 6x6x8x32 tensor. You can think of it as 6x6x32 8-dimensional vectors, which is 1152 input vectors in total. As per the inner workings of the capsule (as described here), each of these input vectors gets their own 8x16 weight matrix that maps 8-dimensional input space to the 16-dimensional capsule output space. So, there are 1152 matrices for each capsule, and also 1152 c coefficients and 1152 b coefficients used in the dynamic routing. Multiplying: 1152 x 8 x 16 + 1152 + 1152, we get 149760 trainable parameters per capsule, then we multiply by 10 to get the final number of parameters for this layer.

The loss function

The loss function might look complicated at first sight, but it really is not. It is very similar to the SVM loss function. In order to understand the main idea about how it works, recall that the output of the DigitCaps layer is 10 sixteen-dimensional vectors. During training, for each training example, one loss value will be calculated for each of the 10 vectors according to the formula below and then the 10 values will be added together to calculate the final loss. Because we are talking about supervised learning, each training example will have the correct label, in this case it will be a ten-dimensional one-hot encoded vector with 9 zeros and 1 one at the correct position. In the loss function formula, the correct label determines the value of $T_c$: it is 1 if the correct label corresponds with the digit of this particular DigitCap and 0 otherwise.

Color coded loss function equation. Source: author, based on original paper.

Suppose the correct label is 1, this means the first DigitCap is responsible for encoding the presence of the digit 1. For this DigitCap’s loss function $T_c$ will be one and for all remaining nine DigitCaps $T_c$ will be 0. When $T_c$ is 1 then the first term of the loss function is calculated and the second becomes zero. For our example, in order to calculate the first DigitCap’s loss we take the output vector of this DigitCap and subtract it from $m+$, which is fixed at 0.9. Then we keep the resulting value only in the case when it is greater than zero and square it. Otherwise, return 0. In other words, the loss will be zero if the correct DigitCap predicts the correct label with greater than 0.9 probability, and it will be non-zero if the probability is less than 0.9.

Loss function value for correct and incorrect DigitCap. Note that the red graph is “squashed” vertically compared to the green one. This is due to the lambda multiplier from the formula. Source: author.

For DigitCaps who do not match with the correct label, $T_c$ will be zero and therefore the second term will be evaluated (corresponding to $(1 — T_c)$ part). In this case we can see that the loss will be zero if the mismatching DigitCap predicts an incorrect label with probability less than 0.1 and non-zero if it predicts an incorrect label with probability more than 0.1.

Finally, in the formula lambda coefficient is included for numerical stability during training (its value is fixed at 0.5). The two terms in the formula have squares because this loss function has $L_2$ norm and the authors apparently consider this norm to work better.

Part II. Decoder.

CapsNet decoder architecture. Source: original paper.

Decoder takes a 16-dimensional vector from the correct DigitCap and learns to decode it into an image of a digit (note that it only uses the correct DigitCap vector during training and ignores the incorrect ones). Decoder is used as a regularizer, it takes the output of the correct DigitCap as input and learns to recreate an 28 by 28 pixels image, with the loss function being Euclidean distance between the reconstructed image and the input image. Decoder forces capsules to learn features that are useful for reconstructing the original image. The closer the reconstructed image to the input image, the better. Examples of reconstructed images can be seen in the image below.

Top row: original images. Bottom row: reconstructed images. Source: original paper.

Layer 4. Fully connected #1

Input: 16x10.
Output: 512.
Number of parameters: 82432.

Each output of the lower level gets weighted and directed into each neuron of the fully connected layer as input. Each neuron also has a bias term. For this layer there are 16x10 inputs that are all directed to each of the 512 neurons of this layer. Therefore, there are (16x10 + 1)x512 trainable parameters.

For the following two layers calculation is the same: number of parameters = (number of inputs + bias) x number of neurons in the layer. This is why there is no explanation for fully connected layers 2 and 3.

Layer 5. Fully connected #2

Input: 512.
Output: 1024.
Number of parameters: 525312.

Layer 6. Fully connected #3

Input: 1024.
Output: 784 (which after reshaping gives back a 28x28 decoded image).
Number of parameters: 803600.

Total number of parameters in the network: 8238608.

Conclusion

This wraps up the series on the CapsNet. There are many very good resources around the internet. If you would like to learn more on this fascinating topic, please have a look at this awesome compilation of links about CapsNets.

Understanding Hinton’s Capsule Networks. Part 3. Dynamic Routing Between Capsules.

Wed, 29 Nov 2017 00:00:00 +0000

Part of Understanding Hinton’s Capsule Networks Series:

Part 1: Intuition
Part 2: How Capsules Work
Part 3: Dynamic Routing Between Capsules (you are reading it now)
Part 4: CapsNet Architecture

Introduction

This is the third post in the series about a new type of neural network, based on capsules, called CapsNet. I already talked about the intuition behind it, as well as what is a capsule and how it works. In this post, I will talk about the novel dynamic routing algorithm that allows to train capsule networks.

One of the earlier figures explaining capsules and routing between them. Source.

As I showed in Part II, a capsule $i$ in a lower-level layer needs to decide how to send its output vector to higher-level capsules $j$. It makes this decision by changing scalar weight $c_{ij}$ that will multiply its output vector and then be treated as input to a higher-level capsule. Notation-wise, $c_{ij}$ represents the weight that multiplies output vector from lower-level capsule $i$ and goes as input to a higher level capsule $j$.

Things to know about weights $c_{ij}$:

Each weight is a non-negative scalar
For each lower level capsule $i$, the sum of all weights $c_{ij}$ equals to 1
For each lower level capsule $i$, the number of weights equals to the number of higher-level capsules
These weights are determined by the iterative dynamic routing algorithm

The first two facts allow us to interpret weights in probabilistic terms. Recall that the length a capsule’s output vector is interpreted as probability of existence of the feature that this capsule has been trained to detect. Orientation of the output vector is the parametrized state of the feature. So, in a sense, for each lower level capsule $i$, its weights $c_{ij}$ define a probability distribution of its output belonging to each higher level capsule $j$.

Recall: computations inside of a capsule as described in Part II of the series. Source: author.

Dynamic Routing Between Capsules

So, what exactly happens during dynamic routing? Let’s have a look at the description of the algorithm as published in the paper. But before we dive into the algorithm step by step, I want you to keep in your mind the main intuition behind the algorithm:

Lower level capsule will send its input to the higher level capsule that “agrees” with its input. This is the essence of the dynamic routing algorithm.

Now that we have this in mind, let’s go through the algorithm line by line.

Dynamic routing algorithm, as published in the original paper.

The first line says that this procedure takes all capsules in a lower level $l$ and their outputs $\hat{u}$, as well as the number of routing iterations $r$. The very last line tells you that the algorithm will produce the output of a higher level capsule $v_j$. Essentially, this algorithm tells us how to calculate forward pass of the network.

In the second line you will notice that there is a new coefficient $b_{ij}$ that we haven’t seen before. This coefficient is simply a temporary value that will be iteratively updated and, after the procedure is over, its value will be stored in $c_{ij}$. At start of training the value of $b_{ij}$ is initialized at zero.

Line 3 says that the steps in 4–7 will be repeated $r$ times (the number of routing iterations).

Step in line 4 calculates the value of vector $c_i$ which is all routing weights for a lower level capsule $i$. This is done for all lower level capsules. Why softmax? Softmax will make sure that each weight $c_{ij}$ is a non-negative number and their sum equals to one. Essentially, softmax enforces probabilistic nature of coefficients $c_{ij}$ that I described above.

At the first iteration, the value of all coefficients $c_{ij}$ will be equal, because on line two all $b_{ij}$ are set to zero. For example, if we have 3 lower level capsules and 2 higher level capsules, then all $c_{ij}$ will be equal to 0.5. The state of all $c_{ij}$ being equal at initialization of the algorithm represents the state of maximum confusion and uncertainty: lower level capsules have no idea which higher level capsules will best fit their output. Of course, as the process is repeated these uniform distributions will change.

After all weights $c_{ij}$ were calculated for all lower level capsules, we can move on to line 5, where we look at higher level capsules. This step calculates a linear combination of input vectors, weighted by routing coefficients $c_{ij}$, determined in the previous step. Intuitively, this means scaling down input vectors and adding them together, which produces output vector $s_j$. This is done for all higher level capsules.

Next, in line 6 vectors from last step are passed through the squash nonlinearity, that makes sure the direction of the vector is preserved, but its length is enforced to be no more than 1. This step produces the output vector $v_j$ for all higher level capsules.

To summarize what we have so far: steps 4–6 simply calculate the output of higher level capsules. Step on line 7 is where the weight update happens. This step captures the essence of the routing algorithm. This steps looks at each higher level capsule $j$ and then examines each input and updates the corresponding weight $b_{ij}$ according to the formula. The formula says that the new weight value equals to the old value plus the dot product of current output of capsule $j$ and the input to this capsule from a lower level capsule $i$. The dot product looks at similarity between input to the capsule and output from the capsule. Also, remember from above, the lower level capsule will sent its output to the higher level capsule whose output is similar. This similarity is captured by the dot product. After this step, the algorithm starts over from step 3 and repeats the process $r$ times.

Dot product is an operation that takes 2 vectors and outputs a scalar. There are several scenarios possible for the two vectors of given lengths but different relative orientations: (a) largest positive possible values; (b) positive dot product; (c) zero dot product; (d) negative dot product; (e) largest possible negative dot product. You can think of the dot product as a measure of similarity in the context of CapsNets. Source: author.

After $r$ times, all outputs for higher level capsules were calculated and routing weights have been established. The forward pass can continue to the next level of network.

Intuitive Example of Weight Update Step

Two higher level capsules with their outputs represented by purple vectors, and inputs represented by black and orange vectors. Lower level capsule with orange output will decrease the weight for higher level capsule 1 (left side) and increase the weight for higher level capsule 2 (right side). Source: author.

In the figure above, imagine that there are two higher level capsules, their output is represented by purple vectors $v_1$ and $v_2$ calculated as described in previous section. The orange vector represents input from one of the lower level capsules and the black vectors represent all the remaining inputs from other lower level capsules.

We see that in the left part the purple output $v_1$ and the orange input $\hat{u}$ point in the opposite directions. In other words, they are not similar. This means their dot product will be a negative number and as result routing coefficient $c_{11}$ will decrease. In the right part, the purple output $v_2$ and the orange input $\hat{v}$ point in the same direction. They are similar. Therefore, the routing coefficient $c_{12}$ will increase. This procedure is repeated for all higher level capsules and for all inputs of each capsule. The result of this is a set of routing coefficients that best matches outputs from lower level capsules with outputs of higher level capsules.

How Many Routing Iterations to Use?

The paper examined a range of values for both MNIST and CIFAR data sets. Author’s conclusion is two-fold:

More iterations tends to overfit the data
It is recommended to use 3 routing iterations in practice

Conclusion

In this article, I explained the dynamic routing algorithm by agreement that allows to train the CapsNet. The most important idea is that similarity between input and output is measured as dot product between input and output of a capsule and then routing coefficient is updated correspondingly. Best practice is to use 3 routing iterations.

In the next post, I will walk you through CapsNet architecture, where we will put together all pieces of the puzzle that we learned so far.

Understanding Hinton’s Capsule Networks. Part 2. How Capsules Work.

Wed, 15 Nov 2017 00:00:00 +0000

Part of Understanding Hinton’s Capsule Networks Series:

Part 1: Intuition
Part 2: How Capsules Work (you are reading it now)
Part 3: Dynamic Routing Between Capsules
Part 4: CapsNet Architecture

Introduction

In Part 1 of this series on capsule networks, I talked about the basic intuition and motivation behind the novel architecture. In this part, I will describe, what capsule is and how it works internally as well as intuition behind it. In the next part I will focus mostly on the dynamic routing algorithm.

What is a Capsule?

In order to answer this question, I think it is a good idea to refer to the first paper where capsules were introduced — “Transforming Autoencoders” by Hinton et al. The part that is important to understanding of capsules is provided below:

“Instead of aiming for viewpoint invariance in the activities of “neurons” that use a single scalar output to summarize the activities of a local pool of replicated feature detectors, artificial neural networks should use local “capsules” that perform some quite complicated internal computations on their inputs and then encapsulate the results of these computations into a small vector of highly informative outputs. Each capsule learns to recognize an implicitly defined visual entity over a limited domain of viewing conditions and deformations and it outputs both the probability that the entity is present within its limited domain and a set of “instantiation parameters” that may include the precise pose, lighting and deformation of the visual entity relative to an implicitly defined canonical version of that entity. When the capsule is working properly, the probability of the visual entity being present is locally invariant — it does not change as the entity moves over the manifold of possible appearances within the limited domain covered by the capsule. The instantiation parameters, however, are “equivariant” — as the viewing conditions change and the entity moves over the appearance manifold, the instantiation parameters change by a corresponding amount because they are representing the intrinsic coordinates of the entity on the appearance manifold.”

The paragraph above is very dense, and it took me a while to figure out what it means, sentence by sentence. Below is my version of the above paragraph, as I understand it:

Artificial neurons output a single scalar. In addition, CNNs use convolutional layers that, for each kernel, replicate that same kernel’s weights across the entire input volume and then output a 2D matrix, where each number is the output of that kernel’s convolution with a portion of the input volume. So we can look at that 2D matrix as output of replicated feature detector. Then all kernel’s 2D matrices are stacked on top of each other to produce output of a convolutional layer.

Not only can the CapsNet recognize digits, it can also generate them from internal representations. Source.

Then, we try to achieve viewpoint invariance in the activities of neurons. We do this by the means of max pooling that consecutively looks at regions in the above described 2D matrix and selects the largest number in each region. As result, we get what we wanted — invariance of activities. Invariance means that by changing the input a little, the output still stays the same. And activity is just the output signal of a neuron. In other words, when in the input image we shift the object that we want to detect by a little bit, networks activities (outputs of neurons) will not change because of max pooling and the network will still detect the object.

The above described mechanism is not very good, because max pooling loses valuable information and also does not encode relative spatial relationships between features. We should use capsules instead, because they will encapsulate all important information about the state of the features they are detecting in a form of a vector (as opposed to a scalar that a neuron outputs).

Capsules encapsulate all important information about the state of the feature they are detecting in vector form.

Capsules encode probability of detection of a feature as the length of their output vector. And the state of the detected feature is encoded as the direction in which that vector points to (“instantiation parameters”). So when detected feature moves around the image or its state somehow changes, the probability still stays the same (length of vector does not change), but its orientation changes.

Imagine that a capsule detects a face in the image and outputs a 3D vector of length $0.99$. Then we start moving the face across the image. The vector will rotate in its space, representing the changing state of the detected face, but its length will remain fixed, because the capsule is still sure it has detected a face. This is what Hinton refers to as activities equivariance: neuronal activities will change when an object “moves over the manifold of possible appearances” in the picture. At the same time, the probabilities of detection remain constant, which is the form of invariance that we should aim at, and not the type offered by CNNs with max pooling.

How does a capsule work?

Let us compare capsules with artificial neurons. Table below summarizes the differences between the capsule and the neuron:

Important differences between capsules and neurons. Source: author, inspired by the talk on CapsNets given by naturomics.

Recall, that a neuron receives input scalars from other neurons, then multiplies them by scalar weights and sums. This sum is then passed to one of the many possible nonlinear activation functions, that take the input scalar and output a scalar according to the function. That scalar will be the output of the neuron that will go as input to other neurons. The summary of this process can be seen on the table and diagram below on the right side. In essence, artificial neuron can be described by 3 steps:

scalar weighting of input scalars
sum of weighted input scalars
scalar-to-scalar nonlinearity

Left: capsule diagram; right: artificial neuron. Source: author, inspired by the talk on CapsNets given by naturomics.

On the other hand, the capsule has vector forms of the above 3 steps in addition to the new step, affine transform of input:

matrix multiplication of input vectors
scalar weighting of input vectors
sum of weighted input vectors
vector-to-vector nonlinearity

Let’s have a better look at the 4 computational steps happening inside the capsule.

1. Matrix Multiplication of Input Vectors

Input vectors that our capsule receives ($u_1$, $u_2$ and $u_3$ in the diagram) come from 3 other capsules in the layer below. Lengths of these vectors encode probabilities that lower-level capsules detected their corresponding objects and directions of the vectors encode some internal state of the detected objects. Let us assume that lower level capsules detect eyes, mouth and nose respectively and out capsule detects face.

These vectors then are multiplied by corresponding weight matrices $W$ that encode important spatial and other relationships between lower level features (eyes, mouth and nose) and higher level feature (face). For example, matrix $W_{2j}$ may encode relationship between nose and face: face is centered around its nose, its size is 10 times the size of the nose and its orientation in space corresponds to orientation of the nose, because they all lie on the same plane. Similar intuitions can be drawn for matrices $W_{1j}$ and $W_{3j}$. After multiplication by these matrices, what we get is the predicted position of the higher level feature. In other words, $\hat{u}_1$ represents where the face should be according to the detected position of the eyes, $\hat{u}_2$ represents where the face should be according to the detected position of the mouth and $\hat{u}_3$ represents where the face should be according to the detected position of the nose.

At this point your intuition should go as follows: if these 3 predictions of lower level features point at the same position and state of the face, then it must be a face there.

Predictions for face location of nose, mouth and eyes capsules closely match: there must be a face there. Source: author, based on original image.

2. Scalar Weighting of Input Vectors

At the first glance, this step seems very familiar to the one where artificial neuron weights its inputs before adding them up. In the neuron case, these weights are learned during backpropagation, but in the case of the capsule, they are determined using “dynamic routing”, which is a novel way to determine where each capsule’s output goes. I will dedicate a separate post to this algorithm and only offer some intuition here.

Lower level capsule will send its input to the higher level capsule that “agrees” with its input. This is the essence of the dynamic routing algorithm. Source.

In the image above, we have one lower level capsule that needs to “decide” to which higher level capsule it will send its output. It will make its decision by adjusting the weights $C$ that will multiply this capsule’s output before sending it to either left or right higher-level capsules $J$ and $K$.

Now, the higher level capsules already received many input vectors from other lower-level capsules. All these inputs are represented by red and blue points. Where these points cluster together, this means that predictions of lower level capsules are close to each other. This is why, for the sake of example, there is a cluster of red points in both capsules $J$ and $K$.

So, where should our lower-level capsule send its output: to capsule $J$ or to capsule $K$? The answer to this question is the essence of the dynamic routing algorithm. The output of the lower capsule, when multiplied by corresponding matrix $W$, lands far from the red cluster of “correct” predictions in capsule $J$. On the other hand, it will land very close to “true” predictions red cluster in the right capsule $K$. Lower level capsule has a mechanism of measuring which upper level capsule better accommodates its results and will automatically adjust its weight in such a way that weight $C$ corresponding to capsule $K$ will be high, and weight $C$ corresponding to capsule $J$ will be low.

3. Sum of Weighted Input Vectors

This step is similar to the regular artificial neuron and represents combination of inputs. I don’t think there is anything special about this step (except it is sum of vectors and not sum of scalars). We therefore can move on to the next step.

4. “Squash”: Novel Vector-to-Vector Nonlinearity

Another innovation that CapsNet introduce is the novel nonlinear activation function that takes a vector, and then “squashes” it to have length of no more than 1, but does not change its direction.

Squashing nonlinearity scales input vector without changing its direction.

The right side of equation (blue rectangle) scales the input vector so that it will have unit length and the left side (red rectangle) performs additional scaling. Remember that the output vector length can be interpreted as probability of a given feature being detected by the capsule.

Graph of the novel nonlinearity in its scalar form. In real application the function operates on vectors. Source: author.

On the left is the squashing function applied to a 1D vector, which is a scalar. I included it to demonstrate the interesting nonlinear shape of the function.

It only makes sense to visualize one dimensional case; in real application it will take vector and output a vector, which would be hard to visualize.

Conclusion

In this part we talked about what the capsule is, what kind of computation it performs as well as intuition behind it. We see that the design of the capsule builds up upon the design of artificial neuron, but expands it to the vector form to allow for more powerful representational capabilities. It also introduces matrix weights to encode important hierarchical relationships between features of different layers. The result succeeds to achieve the goal of the designer: neuronal activity equivariance with respect to changes in inputs and invariance in probabilities of feature detection.

Summary of the internal workings of the capsule. Note that there is no bias because it is already included in the W matrix that can accommodate it and other, more complex transforms and relationships. Source: author.

The only parts that remain to conclude the series on the CapsNet are the dynamic routing between capsules algorithm as well as the detailed walkthrough of the architecture of this novel network. These will be discussed in the following posts.

Max Pechyonkin

Why Swift May Be the Next Big Thing in Deep Learning

Introduction

🧠 Swift Has Strong Support Behind It

💪 Swift for TensorFlow is Much More Than Just a Library

⚡️ Swift is Fast

📦 You Can Use Python, C and C++ Code in Swift

⚙️ Swift Can Go Very Low-Level

📈 What’s Next

Summary

Links for Further Exploration

Key Deep Learning Architectures - ZFNet

Main ideas

Why it is important

Brief description

Deconvolution

Deconvolution Network for Learned Features Visualizations

Understanding Features Visualizations

Architecture

Additional readings

Return to all architectures

Deep Learning Vision for Non-Vision Tasks

Introduction

Case 1: Oil Industry

Case 2: Online Fraud Detection

Case 3: Acoustic Detection of Whales

Summary

References

Key Deep Learning Architectures - LeNet-5

Architecture

Main Ideas

Why it is Important

Brief Description

Additional Readings

Return to all architectures

Key Deep Learning Architectures for Visual Object Recognition

List of architectures in this guide

Key Deep Learning Architectures - AlexNet

Architecture

Main Ideas

Why it is Important

Brief Description

ReLU Nonlinearity

Local Response Normalization

Overlapping Pooling

Data Augmentation

Test Time Data Augmentation

Dropout

Architecture

Additional Readings

Return to all architectures

Stochastic Weight Averaging — a New Way to Get State of the Art Results in Deep Learning

Traditional Ensembling of Neural Networks

Solutions in the Weight Space

Narrow and Wide Optima

Snapshot Ensembling

Fast Geometric Ensembling (FGE)

Stochastic Weight Averaging (SWA)

Implementations

Thanks for reading! If you enjoyed this post, subscribe to updates and I will let you know when a new awesome post is published!

Understanding Hinton’s Capsule Networks. Part 4. CapsNet Architecture.

Part of Understanding Hinton’s Capsule Networks Series:

Introduction

Part I. Encoder.

Layer 1. Convolutional layer

Layer 2. PrimaryCaps layer

Layer 3. DigitCaps layer

The loss function

Part II. Decoder.

Layer 4. Fully connected #1

Layer 5. Fully connected #2

Layer 6. Fully connected #3

Conclusion

Understanding Hinton’s Capsule Networks. Part 3. Dynamic Routing Between Capsules.

Part of Understanding Hinton’s Capsule Networks Series:

Introduction

Dynamic Routing Between Capsules

Intuitive Example of Weight Update Step

How Many Routing Iterations to Use?

Conclusion