CNN发展总结

正在学习李沐老师的d2l课程，梳理一下提到的几个标志性的CNN的逻辑以及发展的逻辑。

1 LeNet

LeNet是最早的卷积神经网络，引入了卷积层、池化层等等。其架构如下：

那时的汇聚层、激活函数还是AvergePool和Sigmoid/Tanh，并非后来实践表明表现更好的ReLU和MaxPooling。

2 AlexNet

与LeNet思路类似，在imageNet比赛中取得了不错的成绩，证明了更深更大的模型可以有非常好的效果

1. 更深的模型层数
2. 卷积通道数目巨大，卷积窗口更大
3. 激活函数从Sigmoid/Tanh改成ReLU
4. 通过Dropout全连接层控制模型复杂度
5. 使用大量增强数据：翻转、裁切、变色

其架构如下：

3 使用块的网络（VGG）

在AlexNet的实现中，我们可以意识到两件事：

1. 更深更大的神经网络确实可以有更好的效果
2. 有很多层的组合（例如卷积层+池化层）反复出现

因此为了方便构建模块化地构建更大的网络，我们可以将一系列“层”定义一个“块”，以“块”为基本单位之一构建网络。

4 网络中的网络（NiN）

VGG把神经网络的规模做得非常巨大，但是此时又暴露出了内存/显存占用巨大的问题——尤其是卷积层之后的第一个全连接层，因输入输出的维度都巨大，参数的数量也巨大。

为了改善存储占用的问题，NiN网络中使用了使用NiN块与GAP层。

• NiN块中使用了1$\times$1的卷积层，相当于通道维度的全连接层，不仅可以减少参数量，而且可以减少过拟合
• Global Average Pooling （GAP）层被用来代替全连接层，减少了存储的参数的数量

4 含并⾏连结的⽹络（GoogLeNet）

有了NiN之后，我们有了控制网络总参数量的工具。由此，我们可以构建更深、更复杂的网络。

GoogLeNet的想法很自然：

• 在我们构建了足够强大的线形网络之后，我们也可以构建一些并行的有分支的网络
• 在纠结要用哪一种卷积参数的时候，我们可以选择全都要

GoogLeNet也借鉴了NiN网络：

• 使用了1$\times$1的卷积网络降维通道
• 使用GAP层，代替全连接层
• 在深度的维度上拼接，实现了并行输出的整合

首先定义了一个Inception块

其中白底的仅仅是用于调整深度，深紫底的才是实际提取特征的层。

由此可以构建出GoogLeNet

#

5 残差网络（ResNet）

• 超深神经网络存在两大挑战：
  1. 参数量庞大，计算和存储开销大。
  2. 深度增加导致梯度消失/梯度爆炸，训练困难。

前者在前面的模型中已经解决，后者ResNet可以解决。

ResNet的核心思想如下：

1. 更复杂的网络应包含原有的更简单网络，保证性能至少不劣于浅层网络。
2. 引入残差函数 $F(x) = f(x) - x$，将学习目标改写为：$f(x) = F(x) + x$

残差函数的作用主要想概括为以下两点：

1. 保证“性能不劣于浅层网络”

   • 当 $F(x) = 0$ 时，网络输出即退化为恒等映射 $y = x$
   • 网络可以在恒等映射附近开始学习，使训练更容易
   • 如果$F(x)$对于结果不好，那$F(x)$拿不到梯度，还是在0附近

2. 梯度传递优势

   • 前向传播时，恒等映射提供了直接捷径
   • 反向传播时，梯度可以沿 $+x$ 这条捷径无衰减地传回，缓解梯度消失问题

ResNet最大的挑战是：相加的时候捷径的形状不同。

1. 通道数量不同

   通过增加$1\times1$卷积层来控制通道数

2. 维度不同

   通过卷积核为1，有stride的卷积层来控制维度

ResNet基本结构的图示如下：

6 稠密连接网络（DenseNet）

DenseNet 继承了ResNet的思想，将其进一步发展。

如果说ResNet是泰勒展开的一阶近似，那DenseNet就是高阶近似。DenseNet讲更多的过往的层传到现在的结果。

DenseNet主要由两种块构成：

• 稠密块

  大量旁路，为了将这么多结果整合在一起，DenseNet像GoogLeNet一样，在通道维度把结果拼接起来。

• 过渡块

  经过稠密块后通道数巨大，因此使用$1\times1$卷积层减小通道数，使用平均汇聚层减小维数。

I’m currently taking D2L course taught by Mu Li, and I’d like to summarize the logic behind several iconic CNNs and the overall development trajectory they follow.

1 LeNet

LeNet is one of the earliest convolutional neural networks, introducing concepts such as convolutional layers and pooling layers. Its architecture is as follows:

At that time, the pooling layer and activation functions used were AveragePool and Sigmoid/Tanh, rather than ReLU and MaxPooling which proved to perform better in later practice.

2 AlexNet

Similar in concept to LeNet, AlexNet achieved impressive results in the ImageNet competition, demonstrating that deeper and larger models can deliver excellent performance.

1. A greater number of layers in the model
2. A large number of convolutional channels and larger convolutional windows
3. The activation function was changed from Sigmoid/Tanh to ReLU
4. Dropout was applied to fully connected layers to control model complexity
5. A large amount of augmented data was used: flipping, cropping, color variation

Its architecture is as follows:

3 Networks with Blocks (VGG)

From the implementation of AlexNet, we can draw two conclusions:

1. Deeper and larger neural networks indeed yield better results.
2. Many combinations of layers (such as convolutional layers + pooling layers) repeat frequently.

Therefore, to facilitate the modular construction of larger networks, we can define a series of “layers” as a “block”, and use “blocks” as one of the basic units to build networks.

4 Network-in-Network (NiN)

VGG scaled up neural networks to an extremely large size, but this also exposed the problem of high memory/video memory consumption—especially the first fully connected layer after the convolutional layers. Due to the large input and output dimensions, the number of parameters was also massive.

To address the issue of storage consumption, the NiN network uses NiN blocks and GAP layers.

• The NiN block uses 1×1 convolutional layers, which act as fully connected layers in the channel dimension. This not only reduces the number of parameters but also mitigates overfitting.
• The Global Average Pooling (GAP) layer is used to replace fully connected layers, further reducing the number of stored parameters.

5 Networks with Parallel Connections (GoogLeNet)

With NiN, we gained tools to control the total number of parameters in a network. Thus, we could build deeper and more complex networks.

The idea behind GoogLeNet is quite intuitive:

• After building a sufficiently powerful linear network, we can also construct parallel sub-networks with branches.
• When unsure which set of convolutional parameters to choose, we can simply use all of them.

GoogLeNet also drew inspiration from the NiN network:

• It uses 1×1 convolutional networks to reduce the number of channels.
• It uses GAP layers to replace fully connected layers.
• It concatenates outputs along the depth dimension to integrate parallel outputs.

First, an Inception block is defined:

The white-background parts are only used to adjust the depth; the dark purple-background parts are the actual feature extraction layers.

Based on this, GoogLeNet can be constructed:

#

6 Residual Network (ResNet)

• Ultra-deep neural networks face two major challenges:
  1. A massive number of parameters, leading to high computational and storage costs.
  2. Increasing depth causes gradient vanishing/exploding, making training difficult.

The first problem has been addressed in previous models, and ResNet solves the second one.

The core idea of ResNet is as follows:

1. A more complex network should include the original simpler network, ensuring its performance is at least no worse than that of the shallower network.
2. A residual function $F(x) = f(x) - x$ is introduced, rewriting the learning objective as: $f(x) = F(x) + x$.

The role of the residual function can be summarized into two main points:

1. Ensuring “performance is no worse than that of a shallower network”
   • When $F(x) = 0$, the network output degrades to an identity mapping $y = x$.
   • The network can start learning around the identity mapping, making training easier.
   • If $F(x)$ performs poorly, it receives no gradient and remains near 0.
2. Advantages in gradient propagation
   • During forward propagation, the identity mapping provides a direct shortcut.
   • During backpropagation, gradients can travel back along this $+x$ shortcut without attenuation, alleviating the gradient vanishing problem.

The biggest challenge in ResNet is that the shapes of the shortcut paths differ when adding them together.

1. Different numbers of channels
   • A 1×1 convolutional layer is used to adjust the number of channels.
2. Different dimensions
   • A convolutional layer with a kernel size of 1 and a non-unity stride is used to adjust the dimensions.

The diagram of the basic ResNet structure is as follows:

7 Densely Connected Network (DenseNet)

DenseNet builds upon the ideas of ResNet and develops them further.

If ResNet is a first-order approximation of a Taylor expansion, then DenseNet is a higher-order approximation. DenseNet incorporates the outputs of many previous layers into the current layer.

DenseNet consists mainly of two types of blocks:

• Dense block
  • It has numerous bypass connections. To integrate so many outputs, DenseNet, like GoogLeNet, concatenates the results along the channel dimension.
• Transition block
  • After a dense block, the number of channels becomes very large. Therefore, a 1×1 convolutional layer is used to reduce the number of channels, and an average pooling layer is used to decrease the spatial dimensions.