Convolutional Neural Networks (CNNs)

Table of Contents

1. Module 1: The building blocks of CNNs

• Course Introduction
• Why do we need CNNs?
• Local connectivity and settings sharing
• What are convolution layers?
• Pooling layers and activation functions
• Real applications of CNNs

2. Module 2: Training CNNs to detect patterns

• How do CNNs build visual understanding?
• From features to classification
• The training process explained
• Factors influencing convergence

3. The evolution of CNN architectures (17m 21s)

• The birth of CNNs with LeNet
• AlexNet and the deep learning revolution
• VGG and the power of depth
• ResNet and skip connections
• Beyond CNN: what now?

1. The building blocks of CNNs

1.1 Course Introduction

Imagine you are trying to teach a computer to recognize a cat. You could describe rules: pointy ears, whiskers, fur. But what about a cat curled up in a ball, or photographed from behind, or partially hidden behind a dog? The truth is that you can’t write enough rules to cover all possibilities. And yet, in the time it took me to say that sentence, a convolutional neural network could have correctly identified thousands of images of cats it had never seen before. This is the power of CNNs, and this is what we will understand here.

If you have worked with neural networks before, you may have used CNNs as out-of-the-box tools. You import a pre-trained model, feed it images, and get predictions as output — and it works, until it doesn’t. Your model might suddenly start performing poorly, and you will have no idea why. You then realize that you have treated your model like a black box. This course is about opening that box.

We’ll start by understanding why CNNs exist in the first place — what limitations of traditional neural networks were they designed to address? Next, we’ll break down the building blocks: convolution layers, pooling operations, and how these components work together to process spatial information efficiently. From here, we’ll explore how CNNs actually learn. You’ll see how a network progresses from detecting simple edges to recognizing complex objects, and we’ll look at techniques for visualizing what’s happening inside these networks at each step. Finally, we’ll trace the evolution of CNN architectures from LeNet in the 1990s, through AlexNet, VGG, and ResNet, understanding the innovations each has made and why they matter to your work today.

1.2 Why do we need CNNs?

To understand why convolutional neural networks exist, we first need to understand what happens when we try to process images with a standard, fully-connected neural network.

The parameter explosion problem

Let’s say you have a simple grayscale image, 28 x 28 pixels, like the handwritten digits of the famous MNIST dataset, for example. Each pixel is an input, resulting in 784 input neurons. Connect them to a hidden layer of, say, 512 neurons, and you already have around 400,000 parameters. It’s manageable.

But now consider a more realistic image. A modest 224 x 224 pixel color photograph has three color channels — red, green and blue. This represents over 150,000 input values. Connect that to the same hidden layer of 512 neurons, and you’re looking at about 77 million parameters in the first layer alone — and real-world networks have many layers.

This explosion of parameters creates serious problems:

1. Computational cost: Training becomes slow and memory intensive.
2. Overfitting: With so many parameters to adjust and relatively limited training data, the network memorizes training images rather than learning generalizable patterns.

The problem of pixel independence

But here’s the deeper problem: a fully connected network treats each pixel independently — it has no notion that pixels close to each other are related, that a pattern in one corner of an image means the same thing as that pattern in the center. We need an architecture that respects the spatial structure of images.

Additionally, if you train a fully connected network to recognize a dog in the middle of an image, it will not automatically recognize that same dog if it appears in the corner — it would have to learn the concept of “dog” separately for each possible position. This is extremely inefficient.

The CNN solution

We therefore need an architecture that:

• Respects the spatial structure of images
• Can learn a pattern once and detect it everywhere

And that’s why CNNs are our solution. They solve both problems through two key principles: *local connectivity and *parameter sharing.

1.3 Local connectivity and settings sharing

Local Connectivity (Local Connectivity)

Instead of connecting each neuron to each input pixel, a CNN connects each neuron to only a small region of the input — a local neighborhood of pixels. This region is called the receptive field.

If you think about it, this actually reflects human vision. When you look at a photograph, you don’t process each pixel simultaneously with respect to every other pixel — you perceive local features: an edge here, a texture there, a shape in this region. Your visual system builds understanding from these local observations. CNNs work the same way.

Let’s make this concrete. Imagine an image of 28 x 28 pixels. In a fully connected network, a single neuron in the first hidden layer would have 784 connections — one for each pixel. But with local connectivity, that neuron might only connect to a 5 x 5 region, forming just 25 connections instead of 784. Multiply this reduction across an entire layer and the savings become substantial.

Parameter Sharing (Parameter Sharing)

Local connectivity alone does not solve our second problem. If we learn a separate set of weights for each local region, we still cannot recognize the same pattern in different locations — this is where parameter sharing comes in.

In a CNN, we don’t just limit connections to local regions — we use the same weights for each region. A single set of weights slides across the entire image, looking for the same pattern everywhere. This set of sliding weights is called a filter or kernel.

Suppose you want to detect vertical edges — the pattern that defines a vertical edge looks the same whether it is in the upper left corner or the lower right. So why learn separate edge detectors for each location? Instead, we learn a single edge detector and apply it everywhere.

These two principles together — local connectivity and parameter sharing — are the foundation of every convolutional neural network, from the simplest to the most sophisticated.

1.4 What are convolution layers?

We established that CNNs use local connectivity and parameter sharing to process images efficiently. But how does the network actually detect patterns? This is the job of the convolution layer.

The filter (filter / kernel)

The core of a convolution layer is something called a filter or kernel. A filter is just a small grid of numbers, typically 3 x 3, 5 x 5, or 7 x 7. These numbers are weights, and they are what the network learns during training.

The convolution operation

Here’s how it works. Imagine a 5 x 5 grayscale image and a 3 x 3 filter. We place the filter in the upper left corner, covering a 3 x 3 region of the image. Then we multiply each filter weight by the corresponding pixel value, add all nine products together, and this sum becomes an output value. Then we slide the filter one position to the right and repeat. We continue to slide across and down, until we have covered the entire image.

Notice what’s happening here:

• Filter only connects to a small local region at a time → local connectivity
• Same filter weights are used at each position → parameter sharing

The feature map

The grid of output values ​​we get from the dragging operation is called a feature map. A single filter produces a single feature map, but convolution layers typically use many filters — 32, 64, or even hundreds. Each filter will have different weights, so each one performs a different calculation when sliding over the image. The result is a stack of feature maps, one per filter, all output from a single convolution layer.

Stride and padding

Two parameters control how the filter moves:

• Stride: This is the distance the filter moves in each step. A stride of 1 means we move one pixel at a time, touching every position. A stride of 2 means we skip every other position, producing smaller output.

• Padding: When the filter is on the edge of the image, it cannot slide further without going out of bounds — this means the output ends up smaller than the input. A 5 x 5 image with a 3 x 3 filter and no padding produces an output of 3 x 3. Padding adds additional pixels, usually zeros, around the edge of the input, allowing the filter to cover the edge regions and preserve the original dimensions.

Color images (multi-channel)

Color images have three channels — red, green and blue. When convolving on a multi-channel input, the filter extends across all channels. A 3 x 3 filter on a color image is actually 3 x 3 x 3 = 27 weights in total. The convolution multiplies across all channels and sums everything into a single output value per position.

This is the convolution layer. It takes an input, drags filters on it, and produces feature maps as output.

1.5 Pooling layers and activation functions

At this point we have convolution layers producing feature maps. But a CNN isn’t just convolution after convolution — there are two other components that play essential roles: pooling layers and activation functions.

Pooling layers

After a convolution layer, we often have feature maps that are still quite large spatially. Pooling is a way to reduce these, essentially reducing the height and width while retaining the important information.

There are many types of pooling, but the most common is max pooling. Max pooling works like this:

1. We define a small window, usually 2 x 2, with a stride usually also 2.
2. We drag this window onto the feature map.
3. At each position we simply take the maximum value in this window.
4. This maximum becomes the output for this region.

A max pooling of 2 x 2 with stride 2 halves the spatial dimensions. A 28 x 28 feature map becomes 14 x 14. The depth (the number of channels) remains the same.

Why do this? There are mainly three reasons:

1. Computational reduction: Fewer values ​​to process means fewer calculations, faster training and less memory.

2. Translation invariance: If a feature moves a pixel or two in the input, the maximum in that region will likely be the same. This makes the network less sensitive to small changes in position.

3. Larger receptive field: By gradually reducing spatial dimensions, pooling helps later layers see large portions of the original image. A 3x3 filter at the end of the network, after heavily pooled feature maps, effectively covers a much larger region of the original input than a 3x3 filter in the first layer.

Activation functions

Each convolution layer performs a weighted sum — multiplying the inputs by the weights and adding them together. But the problem is that weighted sums are linear operations. The images, however, are not linear. The relationship between pixels and what they represent — whether it’s a face, a car, or a handwritten number — is highly complex and non-linear. To capture this complexity, we need to introduce non-linearity into the network. This is what activation functions do.

The most common choice today is ReLU (Rectified Linear Unit). It’s remarkably simple:

• If the value is positive, we keep it.
• If the value is negative, we set it to zero.

That’s all. But this simple operation, applied after each convolution layer, gives the network the ability to learn complex non-linear patterns. Without activation functions, a CNN would just be an elaborate linear filter — with them, it becomes capable of representing almost any visual pattern.

ReLU(x) = max(0, x)

The typical flow of a CNN

To summarize, the typical flow through a CNN looks like this:

Convolution → Activation (ReLU) → Pooling

If you repeat this sequence several times, you have the core structure of a convolutional neural network.

1.6 Real-world applications of CNNs

We’ve covered the building blocks — convolution layers, pooling, and activation functions. But what can you actually do with these components? Where are CNNs making an impact in the real world?

Image classification

The obvious answer is of course image classification. Given a photo, it tells you what’s in it. Is it a cat or a dog? Is it a stop sign or a give way sign? This is the task that put CNNs on the map, and it remains a core application.

Medical imaging

Radiologists spend years learning how to spot tumors, fractures and abnormalities in X-rays, CT scans and MRIs. CNNs can be trained to help with this, flagging potential problems, prioritizing urgent cases, even detecting things that human eyes might miss. In dermatology, CNNs have been trained to identify skin cancers from photographs with accuracy comparable to experienced specialists.

Autonomous vehicles

A self-driving car must understand its surroundings in real time — identify pedestrians, other vehicles, lane markings, traffic signs and obstacles. This isn’t a single classification problem — it’s object detection and segmentation happening continually frame after frame, with lives depending on accuracy.

Satellite and aerial imagery

CNNs can analyze vast amounts of aerial imagery to monitor deforestation, track urban development, assess crop health, or estimate damage after natural disasters.

Document analysis (OCR)

CNNs power Optical Character Recognition (OCR) systems that can read handwritten text, extract information from forms, process invoices and scan historical documents.

Quality control in manufacturing

CNNs inspect products on assembly lines. They can detect things like a tiny crack in a component, a misaligned part, and even a surface defect at astonishing speeds.

Face recognition

CNNs can identify individuals from photos and videos with remarkable accuracy. This has legitimate uses in security and authentication, but also, of course, raises serious privacy concerns that the company is still resolving.

Agriculture

Camera-equipped drones fly over fields, while CNNs analyze images to detect plant diseases, assess irrigation needs, estimate yields and identify weeds for targeted treatment.

2. Training CNNs to detect patterns

2.1 How do CNNs build visual understanding?

So far, we’ve seen the individual components — convolution layers, activation functions, and pooling. But knowing the parts doesn’t explain how a CNN actually understands what it’s looking at. How does a network go from raw pixel values ​​to recognizing a face, a car or a handwritten number?

The answer lies in how these layers stack together. Each layer builds on what the previous layer detected, creating increasingly complex and abstract representations. This is called hierarchical feature learning, and it is fundamental to how CNNs work.

The feature hierarchy

Layer 1 — Raw Pixels: The first convolution layer sees the raw pixels. It learns to detect the simplest possible patterns — things like edges, lines, etc. A filter could activate strongly on a vertical edge. Another feature could address horizontal edges. Even though these are primitive features, they are fundamental to everything else.

Layer 2 — Simple textures: The second layer no longer sees the pixels — it sees the feature maps from the first layer and combines them. For example, a particular arrangement of edges could form a simple texture. The second layer learns to detect these combinations.

Layers 3-4 — Complex Shapes: The third and fourth layers start detecting more complex shapes — things like curves, circles, grids, etc. These layers combine the corners and simple textures of the previous layers into richer structures.

Deep Layers — Object Parts: Deeper layers respond to things like eyes, wheels, windows, leaves, etc. — identifiable parts of objects. They were never explicitly told what an eye is. They found that this particular combination of shapes and textures, built from edges detected many layers upstream, is useful for the task at hand.

Deepest Layers — Whole Objects: Deepest layers combine these parts into whole objects. An eye, plus another eye, plus a nose, plus a mouth, arranged in a particular spatial configuration — it is a face.

No one programs these feature detectors, and no one tells the network to look for edges in layer 1 and eyes in layer 5. The network discovers this hierarchy on its own through training.

The importance of depth

The depth of the network is very important. Shallow networks can only construct simple hierarchies of features. They could detect edges and some basic combinations, but they cannot represent the complex abstractions needed to recognize objects in varying poses, lighting conditions, and contexts. Deeper networks can build richer hierarchies, which means more layers of abstraction between raw pixels and final classification.

2.2 From features to classification

By the time an image has gone through multiple layers of convolution and pooling, we have a rich set of feature maps representing high-level concepts. But feature maps are not a classification — so how do we get from features to a final prediction?

The feature extraction phase

To the left of the process, we have our input — an image. As it passes through the network, we see a stack of feature maps produced. Notice how with each successive layer, the spatial dimension shrinks, but the stacks become deeper. This is the feature extraction phase:

• Early layers detect edges around the face and the contour of the sofa.
• Deeper layers recognize higher-level features — things like facial features, furniture shapes, flower patterns, etc.

The classification phase (flattening + fully-connected)

These final feature maps are *flattened. We take a three-dimensional block of information and stretch it into a one-dimensional vector. We’re basically saying: here are all the features that were detected — now let’s figure out what they mean together.

This flattened vector is passed to the fully-connected layers, which learn to map feature combinations to specific classes.

At the end we get our predictions, for example:

clock:    0.01%
bouquet:  0.04%
girl:     0.94%

These percentages come from applying a softmax function to the raw network output scores. The network tells us it is very confident this image should be classified as “girl.”

The two distinct phases of a CNN

A CNN therefore has two distinct phases:

1. Feature extraction: Where the convolution and pooling layers transform the image into hierarchical representations.
2. Classification: Where these representations are flattened and passed through fully connected layers, which produce a final prediction.

2.3 The training process explained

We talked about everything a CNN can do — extract features, build hierarchies, and finally classify. But when a network is first created, it can’t do any of that. The weights of the filters are random, the weights of the network layers are also random. So how does a CNN go from random weights to accurate predictions? This is done through training.

Training is the process of adjusting the weights of the network so that its predictions match reality. For example, we show him a picture of a cat. He makes a prediction, he is told the correct answer, and he adjusts his weights to do better next time. Repeat this thousands or millions of times, and the network learns.

Component 1: Data

The first thing we need is data — and we need a lot of it. A training dataset will consist of images associated with labels (labels). There will be a picture of something — a cat, a dog, or a car — and the network will be taught the labels for these examples.

Component 2: The loss function (loss function)

We need a way to measure how much the network is wrong — that’s what the loss function does. A loss function measures the difference between the network’s predictions and the actual target values. This is a crucial component because the goal of training a neural network is to minimize this loss.

• When the network predicts “cat” with 90% confidence, and the image is indeed a cat → the loss will be low.
• When it predicts “cat” with 90% confidence and the image is actually a dog → the loss will be high.

The most common loss function for classification is cross-entropy loss, which penalizes confident incorrect predictions more severely than those where the model is not so confident.

Component 3: Gradient descent (gradient descent)

It will be helpful to visualize the loss as a landscape — a terrain of hills and valleys through all possible weight configurations. High points are bad (the network makes a lot of mistakes there), low points are good. Gradient descent is how we navigate this landscape.

At any point, we can calculate the gradient — the direction of steepest rise. If we go in the opposite direction, we go down, that is, we go down to a lower loss. The gradient tells us two things for each weight:

1. In which direction to adjust it
2. To what extent does this weight contribute to the error

Weights that cause large errors receive large adjustments. Weights that already work well receive small adjustments.

The typical flow is:

1. Calculate the gradient
2. Take a step down
3. Calculate the new gradient
4. Take another step
5. And so on

Each step is an update of the weights. On many steps, we move down to a configuration where the network makes good predictions.

Backpropagation (backpropagation)

How do we calculate these gradients? This is done using backpropagation. Backpropagation is just the chain rule of calculation applied systematically across the network.

We start at the output (loss) and work backwards, calculating how much each weight contributed to that loss. The output layer weights directly affect the loss, so their gradients are simple. But what about a filter in the first convolution layer? Its effect on loss is indirect — it affects feature maps, which affect later layers, which ultimately affect prediction, which affects loss. Backpropagation traces this chain of effects backwards, calculating the responsibility of each weight in the final error.

The great thing is that this works no matter how deep the network is. Dozens of layers, millions of weights — backpropagation will calculate a gradient for each.

The training loop

To summarize, the training loop looks like this:

1. Prendre un batch d'images
2. Les passer en avant à travers le réseau (forward pass)
3. Calculer la perte
4. Rétropropager pour obtenir les gradients (backward pass)
5. Mettre à jour les poids
6. Répéter

Each iteration pushes the weights towards better performance. Over thousands of iterations, random weights become edge detectors, texture recognizers, and object classifiers.

2.4 Factors influencing convergence

As we have seen, training a CNN means running the training loop over and over again. But how well this process works depends heavily on the choices we make before training even begins.

Factor 1: The learning rate (learning rate)

The learning rate controls the step size we take with each weight update.

• Too small: The training process will be very slow. The network will improve, but at a glacial pace. You might need days or weeks to reach acceptable performance.
• Too big: We exceed the target. Instead of gently descending to a lower loss, we bounce back erratically, possibly getting worse rather than better.

What we normally do is start with a moderate value and reduce it over time — taking large steps at first when we are far from a good solution, then smaller steps later when we get closer to the desired performance. This is called learning rate scheduling.

Factor 2: The batch size (batch size)

We rarely update weights after each individual image. Instead, we process a batch of images, average their gradients, then update. The batch size dictates how many images we include in each batch.

• Small batches: More frequent updates, but noisier gradients. Because each update is based on limited data, the direction might not be entirely right.
• Large batches: More stable gradients, but fewer updates per pass through the dataset. There are also hardware considerations — larger batches use more memory, so you’re often limited by what fits on your GPU.

Typical batch sizes range from 16 to 256.

Factor 3: Weight initialization (weight initialization)

When training starts, we need to put the weights on something. We can’t use all zeros — if every weight is the same, every neuron learns the same thing, and the network can’t differentiate. We need randomness to break the symmetry, but not just any randomness:

• If initial weights are too large: Activations explode as they propagate through layers, and gradients become unstable.
• If initial weights are too small: Activations shrink toward zero and gradients vanish — deep layers learn almost nothing.

Modern initialization schemes like Xavier and He initialization set the scale of random weights based on the number of inputs of each layer. The goal is to keep activations and gradients within a reasonable range as they pass through the network.

The interaction between factors

These three factors — learning rate, batch size, and initialization — interact with each other and with the network architecture. A learning rate that works well with one batch size might fail with another. An initialization scheme designed for one activation function might not be suitable for another.

This is why training deep networks involves experimentation. We have principles and heuristics, but finding the right combination often requires trial and error, monitoring training curves, and adjusting based on what we observe.

3. The evolution of CNN architectures

3.1 The birth of CNNs with LeNet

Everything we’ve discussed so far — convolution layers, pooling, hierarchical feature learning, training with backpropagation — none of it emerged overnight. There is a story behind this, and that story begins with a network called LeNet.

The historical origin

In the late 1980s and early 1990s, Yann LeCun and his colleagues at Bell Labs were working on a practical problem — they were trying to read handwritten numbers on checks and mail. It wasn’t purely academic: banks and postal services processed millions of documents daily, and automating number recognition had real business value.

LeCun’s insight was to combine three ideas:

1. Local connectivity
2. Shared weights
3. Trainable feature extraction

The result was LeNet-5, released in 1998, although earlier versions existed throughout the 1990s.

Architecture of LeNet-5

By modern standards, this is tiny:

• Input: 32 x 32 pixel grayscale image
• 2 convolution layers
• 2 pooling layers
• 3 layers fully connected
• Approximately 60,000 parameters in total

You could train it on a laptop today in seconds. But the network architecture established patterns that we still use today.

Input (32x32) → Conv1 → Pool1 → Conv2 → Pool2 → FC1 → FC2 → FC3 → Output

Business Impact of LeNet

LeNet has been commercially deployed to read zip codes and process checks. For years, it handled a significant portion of check processing in the United States.

Why didn’t CNN dominate immediately?

Why did it take until 2012 for deep learning to explode? There are a few reasons:

1. The datasets were small: LeNet was trained on tens of thousands of images of numbers, not millions of diverse photographs.
2. Computing power was limited: Training larger networks was prohibitively slow.
3. Competition: Techniques like *Support Vector Machines (SVM) achieved competitive results on many tasks without the complexity of neural networks.

LeNet proved that CNNs could work, but proving that they could dominate across vision tasks required bigger data, faster hardware, and deeper architectures. This breakthrough was still more than a decade away.

3.2 AlexNet and the deep learning revolution

For more than a decade after LeNet, neural networks fell out of fashion. CNNs were seen as useful for niche applications like number recognition, but not as a general solution for computer vision. Then came 2012.

The context: ImageNet Challenge

Every year, researchers participated in the ImageNet Large Scale Visual Recognition Challenge — a competition to classify images into one of 1,000 categories (dogs, cars, furniture, food, plants, etc.) — essentially, a wide variety of everyday objects. The dataset contained over a million training images.

For years, the best system used handcrafted features combined with traditional classifiers. Error rates were around 25 to 26%. Progress was incremental — a fraction of a percent improvement was considered significant.

AlexNet’s breakthrough

In 2012, a team from the University of Toronto submitted a CNN called AlexNet. He achieved a top-5 error rate of 15.3% — almost 11 percentage points better than second place. This wasn’t an incremental improvement, it was a huge leap.

The 5 key innovations of AlexNet

1. Depth: AlexNet had 8 layers — 5 convolutional and 3 fully connected. This is not deep by today’s standards, but it was substantially deeper than LeNet. More layers meant more levels of abstraction and more ability to learn complex features.

2. Scale: AlexNet had approximately 60 million parameters trained on 1.2 million images. The team used two GPUs working in parallel, just to make training feasible — which was a very novel approach at the time.

3. ReLU: Previous networks often used Sigmoid or Tanh activation functions, which saturate and slow down training. AlexNet used ReLU (Rectified Linear Units) throughout the network.

4. Dropout: To combat overfitting with so many parameters, AlexNet randomly zeroed half of the neurons in fully connected layers during training. This forced the network to learn redundant representations, making it more robust.

5. Data Augmentation: The team artificially expanded the training dataset by cropping, flipping, and color-adjusting the images. This helped the network generalize rather than memorize specific training examples.

None of these ideas were entirely new, but AlexNet combined them effectively on an unprecedented scale.

The immediate impact

The impact was immediate. Within a year, almost all competitive submissions to ImageNet used CNNs. Researchers who had dismissed neural networks took note. Investments have poured into deep learning. GPU manufacturers realized they had a new market. The modern era of AI had begun.

3.3 VGG and the power of depth

AlexNet proved that deeper networks could achieve remarkable results, but this raised a question: how should we design these deeper architectures? Were there any principles that needed to be followed when designing a CNN?

In 2014, researchers from the Visual Geometry Group at Oxford proposed an answer. Their network, known as VGGNet, was built on a simple philosophy: use small filters and go deep.

Architecture of VGG

The architecture is very monotonous:

• 3 x 3 convolution blocks, followed by max pooling
• Repeated with increasing channel depth: 64 → 128 → 256 → 512 filters
• VGG used 3 x 3 filters everywhere, in each convolution layer

Why are smaller filters better?

Think about what happens when you stack two 3 x 3 convolution layers. The second layer sees a 3 x 3 region of the first layer’s output, but each of these positions in the first layer was calculated from a 3 x 3 region of the input — so the second layer actually sees a 5 x 5 region of the original input. Stack three 3 x 3 layers, and you cover a 7 x 7 region.

Parameter comparison:

• A unique 7 x 7 filter: 49 weight
• Three filters 3 x 3 stacked: 27 weights in total

You cover the same spatial extent with almost half the parameters. And the benefits don’t stop there — between these three little layers, you have activation functions. Each introduces non-linearity, so three 3 x 3 layers are not only more parameter efficient — they are also more expressive.

VGG variants

VGG had several variants, the most famous being:

• VGG-16: 16 layers, 138 million parameters
• VGG-19: 19 layers

VGG demonstrated that you didn’t need clever architectural tricks — you needed depth, and you needed to apply simple principles consistently.

3.4 ResNet and skip connections

VGG showed that deeper networks perform better. So the natural question was: how far can we go?

The problem of degradation

Researchers have built networks with 20 layers, 30 layers, and even 50 layers. Unfortunately, they ran into problems. Logically, you would think the problem was overfitting — that’s what you might expect with more parameters. But the problem was even worse: deeper networks performed worse on the training data itself.

A 56-layer network had a higher training error than a 20-layer network. Adding layers made things worse, not the other way around. It was extremely confusing. In theory, a deeper network should do at least as well as a shallower one.

The cause: the optimization problem

The issue found was related to optimization. The gradients that flowed backward through dozens of layers were either:

• *vanish: shrink so much that early layers learned almost nothing
• Explode (explode): become unstable

Even with good initialization and valid activations, training very deep networks was unreliable.

The solution: skip connections (residual connections)

In 2015, researchers at Microsoft introduced an elegantly simple solution: skip connections.

The idea is this: instead of asking each layer to directly learn the desired output, ask it to learn the difference between its input and the desired output. Then add the entry at the end.

Sortie = F(x) + x

Where F(x) is the transformation learned by the convolution layers, and x is the original input going directly to bypass.

If we take a block of two or three convolution layers, normally the input passes through these layers and produces an output. With the skip connection, we also take the original input and add it directly to the output, bypassing the layers entirely. Now the layers don’t need to learn a full transformation — they only need to learn the residual (what should change from input to output).

If the optimal transformation is close to identity, the layers can simply learn to produce zeros, and the skip connection passes the input unchanged. This is why these networks are called *Residual Networks or ResNets.

Why do skip connections work?

Skip connections mainly help with gradient flow. During backpropagation, gradients flow directly through the skip connection, bypassing the convolution layers entirely. Even though the gradients across layers fade, the gradients across the jump path survive. This means that the layers can learn without the signal degrading to anything.

Results

The results were dramatic. ResNet successfully trained networks with 50, 101, even 152 layers. The 152-layer ResNet won the 2015 ImageNet competition with a top-5 error rate of 3.6% — better than average human performance on this task.

Think about this progression:

In less than two decades, networks have grown 30 times deeper.

Skip connections have become one of the most influential architectural innovations in deep learning. They appear not only in CNNs, but throughout deep learning — Transformers, the architecture behind modern language models, use a similar mechanism.

3.5 Beyond CNNs: what now?

From LeNet reading handwritten digits to ResNet outperforming human performance on ImageNet, we have traveled a remarkable arc. But the story doesn’t end there.

Transfer Learning

One of the most important practical impacts of these powerful architectures is transfer learning.

What is transfer learning? It is the process of leveraging the knowledge gained from building a model for a particular task, and then reusing that knowledge as a starting point for a different task.

Training a deep CNN from scratch requires millions of images and significant computing power. But a network trained on ImageNet has already learned to detect edges, textures, shapes and parts of objects. These features are useful for almost all visual tasks.

In practice:

1. Take a pre-trained ResNet on ImageNet
2. Remove the final classification layer
3. Add a new layer tailored to your task (perhaps classifying medical images, identifying plant species, or detecting manufacturing defects)
4. Fine-tune (fine-tune) the network to your specific dataset

Because convolution layers already know how to extract meaningful features, you need much less data to get good results. Tasks that would be impossible with a few hundred images become feasible.

The emergence of Transformers

Even as CNNs matured, researchers explored fundamentally different approaches. In 2017, a new architecture emerged in natural language processing: Transformer. Instead of processing sequences step by step, Transformers used a mechanism called self-attention to connect each element to every other element simultaneously. They have proven to be remarkably effective for language tasks.

Vision Transformer (ViT)

Researchers have asked the question: could attention work for images too? In 2020, the Vision Transformer (ViT) answered this question.

Instead of convolving filters on an image, ViT:

1. Divides the image into patches, e.g. 16 x 16 pixels each
2. Treat each patch as a token
3. Process these tokens with the Transformer architecture

No convolutions at all, no built-in assumptions about local connectivity or spatial structure — just attention. With enough data, Vision Transformers matched or surpassed CNNs in image classification. This was surprising — CNNs had been designed specifically for images, incorporating useful biases about spatial relationships. The Transformers made no such assumptions, and yet they learned what they needed to.

The current landscape

Today, the landscape is mixed:

• CNN: Remain dominant in many applications, especially where efficiency matters or data is limited. Their inductive biases help them learn efficiently from smaller datasets.
• Vision Transformers: Excellent when data is abundant and computational resources available.
• Hybrid Architectures: Combine convolution layers with attention mechanisms, trying to get the best of both worlds.

Conclusion

The field may continue to evolve, but everything we’ve covered in this course — feature hierarchies, training dynamics, architectural innovations — remains fundamental. Whether you’re working with a classic ResNet, a Vision Transformer, or whatever comes next, these concepts will help you understand what’s happening and why.

4. Summary of key concepts

Glossary of important terms

Timeline of major CNN architectures

Typical architecture of a CNN

Input Image
    ↓
[Conv 3x3] → [ReLU] → [MaxPool 2x2]
    ↓
[Conv 3x3] → [ReLU] → [MaxPool 2x2]
    ↓
[Conv 3x3] → [ReLU] → [MaxPool 2x2]
    ↓
[Flatten]
    ↓
[Fully Connected] → [ReLU]
    ↓
[Fully Connected] → [ReLU]
    ↓
[Fully Connected] → [Softmax]
    ↓
Output (class probabilities)

Skip Connection ResNet (residual block)

Input x
  ├─────────────────────────────────┐
  ↓                                 │
[Conv 3x3] → [BN] → [ReLU]        │
  ↓                                 │
[Conv 3x3] → [BN]                  │
  ↓                                 │
[+] ←────────────────────────────── ┘
  ↓
[ReLU]
  ↓
Output F(x) + x

Training: Convolutional Neural Networks (CNNs) — Pratheerth Padman

Search Terms

convolutional · neural · networks · cnns · deep · machine · data · science · cnn · skip · alexnet · architecture · classification · component · connections · factor · feature · layers · vgg · activation · architectures · connectivity · convolution · depth