Exploring the Evolution of Vision Transformers: From ViT to Cutting-Edge Architectures

The Vision Transformer (ViT) has revolutionized the field of computer vision, drawing inspiration from the successes of transformers in natural language processing (NLP). Since its inception, the ViT has sparked a wave of research that explores various orthogonal directions, enhancing its capabilities and applications. This article delves into the advancements following the initial ViT submission, addressing key questions about adapting ViTs for specific tasks, the best architectures, training techniques, and the ongoing debate between supervised and self-supervised pre-training.

Understanding the Vision Transformer

At its core, the Vision Transformer employs self-attention mechanisms, akin to those used in NLP, but applies them to image patches instead of word embeddings. This shift allows the model to capture long-range dependencies in images, making it a powerful tool for various computer vision tasks. However, the journey of ViTs has been marked by numerous innovations and adaptations that have expanded their utility and efficiency.

Important Note

Before diving deeper, it’s essential to familiarize yourself with the foundational concepts of self-attention, the original ViT, and transformers in general. For a comprehensive understanding, consider reviewing previous posts on these topics.

DeiT: Training ViT on a Reasonable Scale

One of the significant advancements in the ViT landscape is the introduction of Data-efficient Image Transformers (DeiT). This approach demonstrated that ViTs could be effectively trained on ImageNet without relying on external datasets. The key innovation here is the use of knowledge distillation, where a smaller model (the student) learns to mimic the outputs of a larger, pre-trained model (the teacher).

Knowledge Distillation

Knowledge distillation is a technique where a new model is trained to replicate the output of an ensemble of models, effectively compressing the knowledge of multiple models into a single, more efficient one. This method not only improves performance but also reduces inference time, making it suitable for deployment on embedded devices.

Self-Distillation

Interestingly, self-distillation allows a single model to act as both teacher and student. By training the model to match its own outputs, self-distillation can enhance performance without the need for multiple models. This approach has shown promising results, although the underlying reasons for its effectiveness remain largely unexplained.

Hard-Label Distillation

In the DeiT framework, a learnable global token, known as the distillation token, is concatenated with the patch embeddings. This token is derived from a trained CNN backbone, allowing the model to leverage the strengths of CNNs while training on ImageNet. The loss function used in this approach combines the cross-entropy losses from both the true labels and the teacher’s outputs, enabling the model to learn from both sources effectively.

Pyramid Vision Transformer (PVT)

To address the computational challenges associated with self-attention, the Pyramid Vision Transformer (PVT) introduced a variant called Spatial-Reduction Attention (SRA). This method reduces the spatial dimensions of the keys and values, significantly improving efficiency. PVT has been successfully applied to tasks such as object detection and semantic segmentation, where high-resolution images are crucial.

PVT-v2 Enhancements

The subsequent iteration, PVT-v2, introduced overlapping patch embeddings, convolutional feedforward networks, and linear-complexity self-attention layers. These enhancements allow for better local continuity in image representations and improved processing of varying image resolutions.

Swin Transformer: Hierarchical Vision Transformer

The Swin Transformer builds upon the concept of locality by implementing local self-attention within non-overlapping windows. This hierarchical approach allows for efficient scaling and representation learning, making it suitable for a wide range of vision tasks. The model progressively merges windows, creating a global representation while maintaining computational efficiency.

Self-Supervised Training on Vision Transformers: DINO

Facebook AI Research’s DINO framework has demonstrated the potential of self-supervised learning in training Vision Transformers. By employing a unique self-distillation approach, DINO achieves impressive results without the need for labeled data. The model learns robust representations that can be utilized effectively in downstream tasks, showcasing the power of self-supervised learning in the ViT domain.

Scaling Vision Transformers

Scaling is a critical aspect of deep learning, and recent studies have shown that larger models can achieve remarkable performance on tasks like few-shot learning. By training a ViT model with 2 billion parameters, researchers achieved a top-1 accuracy of 90.45% on ImageNet. This highlights the importance of model size and data availability in enhancing representation quality.

Replacing Self-Attention: Independent Token and Channel Mixing Methods

Recent research has explored alternatives to self-attention, focusing on independent token and channel mixing methods. Architectures like MLP-Mixer and XCiT propose novel ways to mix information across patches and channels, challenging the dominance of self-attention in ViT architectures.

MLP-Mixer

The MLP-Mixer architecture employs two MLP layers to mix features across spatial locations and patches, demonstrating that effective representation learning can be achieved without traditional self-attention mechanisms.

XCiT

XCiT introduces a cross-covariance attention function that operates along the feature dimension of tokens, offering a fresh perspective on information mixing in ViTs.

Multiscale Vision Transformers (MViT)

Inspired by CNN architectures, Multiscale Vision Transformers leverage hierarchical feature representations by gradually expanding channel capacity while reducing spatial resolution. This approach allows for the effective capture of both low-level and high-dimensional features, enhancing the model’s performance across various tasks.

Application of ViTs in Specific Domains

Video Classification: TimeSformer

The TimeSformer architecture adapts ViTs for video recognition by applying spatial and temporal attention mechanisms. This dual attention approach enables the model to capture both spatial and temporal correlations, significantly improving performance in video classification tasks.

Semantic Segmentation: SegFormer

SegFormer, developed by NVIDIA, utilizes a hierarchical transformer encoder to output multiscale features for semantic segmentation. By avoiding positional encodings and employing a simple MLP decoder, SegFormer achieves impressive results in dense prediction tasks.

Medical Imaging: UNETR

The UNETR model adapts ViTs for 3D medical image segmentation, demonstrating that transformers can effectively capture global multi-scale information. By integrating skip connections, UNETR enhances segmentation performance across various medical imaging tasks.

Conclusion

The evolution of Vision Transformers has opened up numerous avenues for research and application in computer vision. From knowledge distillation techniques to innovative architectures like PVT, Swin, and SegFormer, the landscape continues to evolve rapidly. As researchers explore new methods and applications, the potential for ViTs to push the boundaries of image recognition remains vast.

If you found this exploration of Vision Transformers insightful, consider supporting our work by sharing this article or making a small donation. Your support helps us continue contributing to the open-source and open-access ML/AI community.

Stay motivated and curious!

Cited as:

@article{adaloglou2021transformer,
title = "Transformers in Computer Vision",
author = "Adaloglou, Nikolas",
journal = "https://theaisummer.com/",
year = "2021",
howpublished = {https://github.com/The-AI-Summer/transformers-computer-vision},
}

References

• Deep Learning in Production Book 📖: Learn how to build, train, deploy, scale, and maintain deep learning models. Understand ML infrastructure and MLOps using hands-on examples. Learn more.

Disclosure: Some links may be affiliate links, and at no additional cost to you, we may earn a commission if you decide to make a purchase after clicking through.