ConvNeXt: In Search of the Last Convolutional Layer

ViTs are precise but not so efficient and CNNs are efficient but not so precise. Let’s create a precise and efficient neural network. Publicado originalmente en Level Up Coding.

Created by the author with Dall-E 3

The “Roaring 20s” of visual recognition began with the introduction of Vision Transformers (ViTs), which quickly superseded ConvNets as the state-of-the-art image classification model.A vanilla ViT, on the other hand, faces difficulties when applied to general computer vision tasks such as object detection and semantic segmentation. It is the hierarchical Transformers (e.g., Swin Transformers) that reintroduced several ConvNet priors, making Transformers practically viable as a generic vision backbone and demonstrating remarkable performance on a wide variety of vision tasks [1].

The 2010s were marked by the progress of deep learning. An epoch characterized by the renaissance of Convolutional Neural Networks.

The invention of back-propagation-trained ConvNets dates back to the 1980s, presented in the academic paper: Handwritten Digit Recognition with a Back-Propagation Network.

Source: Orginal paper.

Although the algorithm was introduced in 1989, it was not until late 2012 that its true potential for visual feature learning was revealed in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC).

The AlexNet architecture, developed by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton revolutionized the field. Outperforming traditional algorithms by achieving a top-5 error rate of 15.3%, which was significantly lower than the 26.2% of its closest competitor.

The field has since evolved at a rapid speed, and representative ConvNets architectures have been developed:

Best deep CNN architectures and their principles: from AlexNet to EfficientNet | AI Summer How convolutional neural networks work? What are the principles behind designing one CNN architecture? How did we go…

But why CNNs have become so popular?

From the ConvNeXt paper:

ConvNets have several built-in inductive biases that make them well-suited to a wide variety of computer vision applications. The most important one is translation equivariance, which is a desirable property for tasks like object detection. ConvNets are also inherently efficient due to the fact that when used in a sliding-window manner, the computations are shared [1].

Inductive bias

Inductive bias refers to the assumptions a learning algorithm uses to make predictions about unseen data. These biases guide the learning process and help the model generalize better to new information.

Photo by Loic Leray on Unsplash.

CNNs have the following inductive biases:

Locality: ConvNets assume that local regions in an image are more relevant for understanding the image’s content. This is why they use small filters (kernels) that convolve over the image. These filters capture local patterns like edges, textures, and colors, which are fundamental building blocks of higher-level features.
Translation Equivariance: If an object in an image moves, its representation in the feature map moves in the same way. This enables CNNs to detect objects regardless of their position.
Hierarchical Structure: Convolutional Networks are designed to learn features hierarchically. The first layers retrieve low-level features (like edges), and as you dive deeper into the structure they capture more complex and abstract characteristics.
Shared Weights and Spatial Invariance: The same filter is applied across the image, reducing the amount of parameters and making the network more efficient. The assumption here is that useful features (edges, textures, or patterns) can appear in different parts of an image.

A fAIry tale of the Inductive Bias Do we need inductive bias? How simple models can reach the performance of complex models

In recap: ConvNets capture local patterns through convolutional filters, can recognize objects regardless of their position thanks to translation equivariance and spatial invariance, learn features hierarchically (from simple to complex), and employ weight sharing for efficiency.

Translation equivariance | Source.

The rise of Vision Transfomers (ViTs)

Around the 2010s, neural network design for Natural Language Processing (NLP) took a very different path from computer vision, as a new generation of models known as transformers replaced completely recurrent neural networks (RNNs) and its variants**.**

Photo by Jack Anstey on Unsplash.

Despite the differences between language and vision domains, the two fields surprisingly converged in 2020.

A new variant of traditional transformers was adapted for analyzing images. The Vision Transformer.

Completely altering the computer vision landscape, ViTs were able to outperform CNNs in many tasks due to their ability to capture global context, scalability with large datasets, and flexibility in handling different input sizes.

Types of Vision Transformer architectures | Source.

How do Vision Transformers work?

To assess the importance of various parts of an image, vision transformers use self-attention mechanisms allowing them to recognize complex relationships.

In areas where an understanding of the bigger picture is important, this global perspective will be helpful as opposed to the local CNN focus.

In addition, ViTs are capable of adapting to varying resolutions and can be used more effectively in images with a higher resolution.

Vision Transformer (ViT) representation | Source

ViTs were introduced in the famous paper: AN IMAGE IS WORTH 16X16 WORDS

Vision Transformers [ViT]: A very basic introduction A Simple and basic understanding of how transformers can be used in images

Integration of ViTs and CNNs (Hierarchical transformers)

The biggest problem within ViTs is its global attention design, which has a quadratic complexity to the input size (as the size of the input image increases, the number of calculations grows exponentially).

This might not be a challenge for ImageNet classification tasks but becomes a notable problem with higher-resolution inputs.

Hierarchical Transformers were designed to bridge this gap by employing a hybrid approach.

The sliding window strategy (attention within local windows).

You can think of it as a kernel that travels through the image, but instead of performing convolution operations, it applies self-attention mechanisms in each step.

Swin Transformer | Source.

The Swin Transformer is a representative milestone in developing hierarchical ViTs. It demonstrated that transformers can achieve state-of-the-art performance across many computer vision tasks and not only image classification.

Swin Transformer’s success and rapid adoption also revealed one thing:

The essence of convolution is not becoming irrelevant; rather, it remains much desired and has never faded.

ConvNeXt Design

The design process of ConvNeXt was driven by one key question:

How do design decisions in Transformers impact ConvNets performance?

The strategy is defined in the original paper in the following way:

We provide a trajectory going from a ResNet to a ConvNet that bears a resemblance to Transformers. We consider two model sizes in terms of FLOPs, one is the ResNet-50 / Swin-T regime with FLOPs around 4.5×10⁹ and the other is the ResNet-200 / Swin-B regime which has FLOPs around 15.0 × 10⁹ [1].

So basically, the strategy was to modify a traditional ResNet architecture by integrating it with certain ViTs (e.g. Swin) characteristics.

Why ResNets and Swin Transformers?

ResNet-50 ResNet has been a foundational model in computer vision, marking a significant advancement with its Residual Connections.

The choice to start with a standard ResNet model, such as ResNet-50 allowed the authors to explore how modernizing a traditional and well-understood architecture could lead to improvements akin to more recent Transformer-based models.

ResNet-50 Architecture | Source.

Swin Transformer The Swin transformer, representing a modern approach in the ViTs field, was chosen as a reference due to its similarity in size and performance to ResNet-50.

A Comprehensive Guide to Microsoft’s Swin Transformer In-depth Explanation and Animations

The Swin-T, a small version of the Swin Transformer, has similar GFLOPS to ResNet-50 but offers higher accuracy, making it an ideal model for comparison.

Swin Transformer review | Source.

What are FLOPs? FLOPs (Floating Point Operations per Second) measure the number of floating-point calculations a computer can perform in one second. A floating-point operation is any mathematical calculation like addition, subtraction, multiplication, or division.

High FLOPs indicate a computationally intensive model, which might require more processing power and energy.

GFLOPs (Giga Floating Point Operations per Second) is a more specific term that implies a specific scale (billions of GFLOPs). It is commonly used to compare the performance of different algorithms at a higher scale.

(e.g. When we say a model has 3.7 GFLOPs, it means the model’s computations involve 3.7 billion operations each second it’s running).

Model development

The chart below represents the whole process with all the transformations applied to the original ResNet-50.

In each step, the accuracy and the GFLOPs of the model change. After the final phase, there is a comparison with the Swin-T model.

(The results shown are from the ResNet-50 for simplicity reasons. Check Appendix C of the original paper for the results of the ResNet-200*)*.

Transformations applied to the original ResNet | Source.

1.- Macro design

1.1.- Stage ratio

The term “ratio” is used to describe how computation tasks are divided among a neural network.

In ResNet’s design, the computation distribution is largely empirical [2]. That is to say, rather than theoretical models, the development of the architecture was based on experimental data and observation.

Traditional ResNets have a ratio of (3:4:6:3) [2].

On the other hand, transformers have a more defined ratio for distributing computational resources, like 1:1:3:1 in Swin-T [3], where each stage is balanced except for the third, which is heavier.

Following the design, we adjust the number of blocks in each stage from (3, 4, 6, 3) in ResNet-50 to (3, 3, 9, 3). This improves the model accuracy from 78.8% to 79.4% [1].

Change in the ratio (3, 4, 6, 3) → (3, 3, 9, 3) | Source — edited by the author.

1.2.- Stem cell

The “stem” in a neural network refers to the initial layers that first process the input image.

In ResNets, the initial layer is usually a convolutional layer, which reduces the image size while extracting initial features.

In contrast, Vision Transformers begin with a “patchify” step, downsampling the image by dividing it into smaller patches.

“Patchify” layer representation | Source.

The ConvNeXt model adopts this approach by using a patchify layer which was implemented using a 4×4, stride 4 convolutional layer.

This change simplifies the input processing and slightly improves the model’s accuracy (79.4% → 79.5%), suggesting that a streamlined initial processing stage can be effective, despite its simplicity compared to the original ResNet stem design [1].

2.- ResNeXt-ify

ResNeXt is a neural network architecture that extends the principles of residual networks by incorporating the concept of ‘grouped convolutions’.

The key idea in ResNeXt is to have a set of parallel convolutions within each block of the network. This design allows the model to increase its width and capacity while keeping the computational complexity manageable [4].

Grouped convolution | Source.

ConvNeXt incorporates this idea in its architecture with depthwise convolutions, a special case of grouped convolution where:

Number of groups = Number of channels

Types of grouped convolutions (depthwise convs. were popularized by MobileNet and Xception) | Source.

Following the strategy proposed in ResNeXt, we increase the network width to the same number of channels as Swin-T’s (from 64 to 96). This brings the network performance to 80.5% with increased FLOPs (5.3G) [1].

Understanding Depthwise Separable Convolutions and the efficiency of MobileNets Explanation of MobileNets and Depthwise Separable Convolutions

3.- Inverted Bottleneck

In a standard bottleneck, the number of channels of the input is first reduced and then expanded within the network block.

An inverted bottleneck, on the other hand, starts with a narrower input, expands it to a wider internal dimension for intermediate processing, and then compresses it back to a narrower output.

Inverted bottleneck | Source.

Transformers often use an inverted bottleneck design where the hidden dimension inside a block (specifically the MLP — Multilayer Perceptron block) is significantly larger than the input dimension.

This design has some parallels in ConvNets, particularly in architectures like MobileNetV2, where it has become a common feature to increase the internal dimensions of a network layer before compressing it back down [5].

Despite the increased FLOPs for the depthwise convolution layer, this change reduces the whole network FLOPs to 4.6G, due to the significant FLOPs reduction in the downsampling residual blocks’ shortcut 1×1 conv layer. Interestingly, this results in slightly improved performance (80.5% → 80.6%) [1].

4.- Large Kernel

This is one of the most interesting parts, to understand it better let´s make a recap about the evolution of kernels in computer vision.

A quick summary of kernels in CNNs and ViTs

Photo by Andrew Neel on Unsplash.

Initially, large kernels were common, allowing networks to capture a wide range of features from input images.

However, with the invention of VGGNet, a higher number of multiple layers with small (3×3) kernels, proved to be more effective and efficient in practice [6].

However, with the rise of Vision Transformers (ViTs), larger kernels were reintroduced in the vision landscape, with window sizes of at least (7×7).

This facilitated models to capture global contextual information, akin to the broad receptive field of transformers’ self-attention mechanisms.

ConvNeXt kernel design

Kernel size transformations | Source.

As in previous steps, ConvNeXt opted for the ViT feature (a large kernel).

Researchers experimented with various kernel sizes, and it was observed that the benefit of larger kernel sizes reaches a saturation point at 7×7. But a better performance than smaller kernel sizes such as 5.

But what is the first step in the diagram (move up the depthwise conv. layer) ?

In the context of CNNs, ‘moving up the depthwise conv layer’ means altering the order in which the layers are applied to the input data.

In a standard architecture, depthwise convolutional layers might appear deeper within the network, after other types of layers. However, we are repositioning the depthwise conv. layer to occur earlier in the process.

This design change is inspired by the structure of Transformers, where: the Multihead Self-Attention (MSA) block is placed prior to the MLP layers [1].

By moving the depthwise convolutional layer up, the computationally complex operations (large-kernel convolutions) work on a lower number of channels. And the more efficient operations, like the 1x1 convolutions, handle the increased channels which is where more computational work is required.

This rearrangement aims to improve the efficiency of the network by reducing the overall computational load (GFLOPs).

5.- Micro Design

Micro design transformations |Source.

Replacing ReLU with GELU: While ReLU is common in ConvNets, GELU — a smoother variant — is often used in transformers. Incorporating GELU into ConvNets didn’t change the accuracy, but reduced GLOPS by 0.1.

ReLU and GELU activation functions | Created by the author.

Reducing Activation Functions: Transformers use fewer activations compared to ResNets. Mimicking this by reducing GELU activations improved performance (from 80.6 to 81.3).
Fewer Normalization Layers: Similarly, using fewer BatchNorm layers, like in Transformers, enhanced performance by 0.1.

Swin T vs ResNet vs ConvNeXt block | Source.

Here we remove two BatchNorm (BN) layers, leaving only one BN layer before the conv 1 × 1 layers. This further boosts the performance to 81.4%, already surpassing Swin-T’s result [1].

Substituting BN with LN: Using Layer Normalization instead of Batch Normalization, which is also a common practice in transformers, was beneficial as well, slightly improving the accuracy by 0.1.

Batch and Layer Normalization | Source.

Separate Downsampling Layers:

In ResNet, the spatial downsampling is achieved by the residual block at the start of each stage.
In Swin Transformers, a separate downsampling layer is added between stages.

Adopting swin transformer’s downsampling approach instead of ResNet method surprisingly led to significant accuracy gains (from 81.5 to 82).

Testing the model

Photo by Tobias Tullius on Unsplash

As important as training is the testing part. We need to ensure that the model performs well, with a method of evaluation, another way we won´t be able to improve it.

But how can we do this?

Typically these models are tested on a series of benchmarks which are giant datasets or frameworks where different capabilities of the aglorithm can be evaluated.

1.- ImageNet Classification ConvNeXt-T achieved 82.1% top-1 accuracy surpassing Swin Transformers. The biggest version (ConvNeXt-L) also had impressive results with 85.5%, being one of the best models in this benchmark.

Top-1 accuracy on ImageNet-1k trained models | Source — edited by the author.

The three architectures at the top of the table (RegNet and EffNet versions) have the best combination of accuracy and computational requirements, among the ones listed.

However it’s important to visualize the global picture, normally models are evaluated with small and large datasets, to see how they scale.

As discussed before, one of the advantages of ConvNeXt is its capability to adapt to higher volumes of data, due to the fact of having transformer features in its architecture.

Let´s see how the model performs in the image-net 22k dataset:

Top-1 accuracy on ImageNet-22k pre-trained models | Source — edited by the author.

The results were quite interesting, in this occasion ConvNeXt achieved the highest accuracy (87.8%) with ConvNeXt-XL. However, the parameters and the FLOPS are still quite high compared with other architectures such as EffNet V2-XL.

However, other versions of ConvNeXt had a better trade-off in accuracy and FLOPS. A clear example is the ConvNeXt-L which had the second-best accuracy in the benchmark (87.5%) but with a significantly lower number of parameters (198M) and a FLOP of 101.0G which is more or less at the level of EffNetV2-XL (94.0G).

The small version (ConvNeXt-T) also scaled quite well and provided an impressive balance between accuracy and FLOPs (82.9% / 4.5G).

2.- COCO Object Detection

The COCO (Common Objects in Context) dataset is used for evaluating the performance of models on tasks like object detection, segmentation, and captioning.

Object detection consists of identifying and finding the position of objects in a set of images.

Image classification, object localization, and semantic segmentation | Source

Before analyzing the benchmark is important to understand the following metrics:

AP_bbox: Average Precision for bounding box detection at different IoU (Intersection over Union) thresholds. AP_50 and AP_75 represent the precision at 50% and 75% IoU thresholds, respectively. Higher AP values are better, indicating more accurate detection.
AP_mask: Similar to AP_bbox, but for segmentation masks.

Coco object detection and segmentation results | Source — edited by the author.

ConvNeXt-T has slightly lower FLOPs (714G) and higher FPS (13.5) than Swin-T, indicating it might be a more efficient model, while also maintaining comparable accuracy as indicated by the AP scores.

3.- Efficiency

Inference throughput comparisons on an A100 GPU | Source — edited by the author.

This benchmark is the strongest point of the model, it outperformed all Swin Transformer versions. This can be seen in the throughput column which indicates the images per second that the model can process.

In general, ConvNeXt throughput surpasses the Swin transformer by around 40%. Indicating a clear improvement of the efficiency, while not only maintaining but also improving the accuracy.

This is precisely the main objective of ConvNeXt: