Vision transformers are artificial intelligence models that analyze images by learning how different parts of an image relate to one another. Instead of relying primarily on specialized filters that scan local regions, as traditional convolutional neural networks do, vision transformers use an attention mechanism to determine which visual elements are most relevant to understanding the image. This allows them to combine information from distant regions, recognize relationships between objects, and build increasingly sophisticated representations of visual scenes.
The central idea is to treat an image as a sequence of smaller visual units, much as a language transformer treats a sentence as a sequence of words or word fragments. By comparing these units and adjusting how much attention each gives to the others, a vision transformer can learn patterns that help it identify objects, classify scenes, locate features, and support more complex visual tasks.
Understanding how these models work requires looking at how images are converted into tokens, how attention connects those tokens, and how the resulting representations become useful for visual recognition.
How vision transformers turn images into sequences
An image is a grid of pixels. Each pixel contains numerical values representing color or intensity, but individual pixels rarely provide enough information to identify an object. Recognizing a bicycle, for example, requires interpreting relationships among its wheels, frame, handlebars, and other components.
A vision transformer begins by dividing an image into smaller, usually square regions called patches. These patches serve a role similar to tokens in a language model. Rather than processing every pixel as an independent element, the model works with a more manageable sequence of image regions.
In the original Vision Transformer architecture, commonly abbreviated ViT, an image is divided into patches of a fixed size. Each patch is converted into a vector: an ordered collection of numbers that encodes its pixel values. A learned linear transformation then maps that vector into a representation of a chosen dimension. This representation is called a patch embedding.
For example, an image measuring 224 by 224 pixels divided into patches measuring 16 by 16 pixels produces 196 patches. Each patch becomes one token in the sequence. The model can then process these 196 tokens together, rather than treating the image as a single undifferentiated array.
The patch size determines the resolution of this initial representation. Smaller patches preserve finer spatial detail but produce longer sequences. Larger patches reduce the number of tokens and can make processing more efficient, although they may discard useful local information. The appropriate choice depends on the task, computational budget, and desired level of visual detail.
Turning patches into vectors does not automatically tell the model where those patches came from. A sequence of patch embeddings must also preserve information about spatial arrangement. Otherwise, the model could have difficulty distinguishing an arrangement in which a wheel sits beneath a bicycle frame from one in which the same visual components appear in unrelated locations.
Vision transformers therefore incorporate positional information, commonly through positional embeddings or other positional representations. This information allows the model to account for where each patch belongs in the original image. The precise method varies across architectures, but the purpose is the same: to combine visual content with information about spatial relationships.
Once the image has been divided, embedded, and associated with positional information, its patch representations can enter the transformer’s main computational machinery.
What attention means in a vision transformer
Attention is a mathematical mechanism for determining how strongly one representation should use information from other representations. In a vision transformer, it enables each image patch to incorporate information from other patches, including regions that may be far away in the original image.
Consider a photograph of a dog standing on grass. A patch containing part of the dog’s ear might be difficult to interpret in isolation. Information from nearby patches could help identify the ear’s shape, while information from more distant patches could provide evidence that the ear belongs to a dog whose head appears elsewhere in the image. Attention gives the model a way to combine such evidence.
The mechanism relies on three learned representations: queries, keys, and values. These names come from the mathematical formulation of attention, and each has a specific role.
A query represents the kind of information a patch is seeking from the other patches. A key represents features that help determine whether a patch is relevant to a particular query. A value contains the information that can be passed from that patch to other parts of the sequence.
The model calculates a compatibility score between a query and the keys of other tokens. Higher scores generally indicate that the corresponding tokens are more relevant to the query under the model’s learned representation. These scores are converted into normalized weights, usually with a softmax function, so that the weights can be used to combine the value vectors.
The result is a weighted combination of information from the sequence. A patch can draw heavily on some regions and less on others, with the pattern of weights depending on the input and the learned parameters.
In standard scaled dot-product attention, the calculation is expressed as:
Attention(Q,K,V)=softmax(QK⊤dk)V\operatorname{Attention}(Q,K,V) = \operatorname{softmax} \left( \frac{QK^\top}{\sqrt{d_k}} \right)VAttention(Q,K,V)=softmax(dkQK⊤)V
Here, QQQ, KKK, and VVV are the matrices of queries, keys, and values, while dkd_kdk is the dimension of each key vector. The scaling factor helps keep the dot products at a suitable numerical scale before the softmax operation.
The important point is that attention does not simply select one object or one patch. It computes a context-dependent mixture of information from many tokens. The resulting representation of a patch can therefore reflect both its own visual content and its relationships with the rest of the image.
Attention weights are not necessarily direct measures of human-interpretable importance. They describe how the model distributes information within a particular computation. Understanding why a model makes a particular prediction generally requires more than inspecting one attention map.
How self-attention connects distant parts of an image
The form of attention most commonly used in vision transformers is self-attention. It is called self-attention because the queries, keys, and values are derived from the same sequence of image tokens.
Every token can compare its query with the keys of other tokens and use the resulting weights to aggregate their values. In standard global self-attention, this allows each patch to access information from every other patch in a single attention layer.
This capability distinguishes transformers from many conventional convolutional neural networks. A convolutional layer applies learned filters to local neighborhoods. It is particularly effective at detecting nearby patterns such as edges, corners, textures, and simple shapes. Deeper convolutional layers can gradually combine these patterns into representations of larger structures, expanding the region of the image that influences each feature.
A global self-attention layer, by contrast, can directly connect distant regions. A patch representing one end of an object can exchange information with a patch representing another end, even when many patches lie between them in the sequence.
Imagine an image of a person holding an umbrella. A patch containing part of the umbrella’s handle may be easier to interpret when considered alongside patches representing the canopy and the person’s hand. Self-attention allows these regions to exchange information without requiring a long stack of local operations to establish a connection.
This does not mean that a vision transformer automatically understands every object or relationship. Its ability to use distant information depends on what it learns during training, the quality of the image representations, and the architecture’s capacity. Global access provides a computational opportunity, not a guarantee of correct interpretation.
Attention is also not the only way a model can integrate information across an image. Convolutional networks, hierarchical vision transformers, and other architectures can all build broad contextual representations, but they do so through different combinations of operations.
Why transformers use multiple attention heads
A vision transformer typically uses multi-head self-attention rather than a single attention operation. Multiple heads allow the model to learn different patterns of relationships between image tokens at the same time.
Each attention head has its own learned projections for queries, keys, and values. It computes attention independently, producing a representation that reflects its own learned way of combining information. The outputs of the heads are then combined and transformed into the representation passed to the next part of the network.
Different heads can become sensitive to different kinds of visual relationships. One may respond strongly to similarities in appearance, another to spatial configurations, and another to relationships between object parts and their surroundings. These are illustrative possibilities, not fixed roles assigned to individual heads. The patterns learned by particular heads can overlap, vary across models, and be difficult to interpret consistently.
The advantage is flexibility. An image contains many simultaneous kinds of structure: textures, boundaries, shapes, repeated patterns, and relationships among objects. A single attention distribution may not capture all these relationships equally well. Multiple heads give the model several learned ways to combine visual information.
The heads are not independent miniature vision systems. Their outputs are integrated into a shared network, and their contributions are shaped by training. The overall representation reflects their combined computations rather than a simple collection of separately identifiable visual skills.
How a transformer builds increasingly useful visual features
Attention is one part of a larger architecture. A vision transformer typically contains a sequence of transformer blocks, each of which transforms the token representations.
Within a standard block, a self-attention operation is followed by a feed-forward neural network. The feed-forward network applies learned transformations to each token representation, introducing additional nonlinear processing. Residual connections help preserve information as it passes through the block, while layer normalization helps stabilize the computations.
The attention operation and feed-forward network serve complementary purposes. Attention mixes information across tokens, allowing a patch’s representation to incorporate context from other regions. The feed-forward network transforms the resulting representation, enabling the model to learn more complex features from the combined information.
As tokens pass through successive blocks, their representations can become progressively more context-sensitive. Early layers may preserve substantial information about local appearance, while deeper layers may encode more abstract patterns and relationships. This progression is a useful general description, not a strict division of labor: the information represented at each layer depends on the model, its training, and the images it processes.
For example, an image-classification model might initially encode color transitions and local textures. Through repeated transformations, its representations may become useful for identifying parts of objects and combining evidence about the overall scene. The final prediction then draws on information that has been processed across many layers.
Some vision transformers include a special learned token, often called the class token, which is added to the patch sequence. This token participates in self-attention and can gather information from the image patches as the network processes the sequence. A classification head uses its final representation to predict the image’s category.
Other architectures use different strategies. They may aggregate patch features through pooling, make predictions at the patch level, or produce spatially organized outputs for tasks such as segmentation. The class token is therefore a common design choice, not a requirement of all vision transformers.
How vision transformers learn to recognize images
The architecture specifies how information flows through the model, but it does not determine what the model knows. That knowledge is acquired through training, which adjusts the model’s parameters based on examples and a learning objective.
In supervised image classification, a model receives training images paired with labels such as cat, car, or tree. It produces predictions, compares them with the correct labels through a loss function, and uses backpropagation to calculate how its parameters contributed to the error. An optimization algorithm then adjusts those parameters to reduce the loss over repeated training examples.
During this process, the model learns patch embeddings, attention projections, feed-forward transformations, and other parameters that make its representations useful for the task. It does not need a human to specify which patches should attend to a dog’s face or which patterns define a bicycle. Instead, the training process encourages internal computations that help produce accurate predictions.
However, labels alone do not guarantee that a model learns the intended visual concepts. If a training dataset contains misleading correlations, a model may rely on background features or other incidental patterns. For instance, it might associate an animal category with a particular environment rather than learning robust evidence from the animal’s appearance.
Vision transformers can also be trained through self-supervised methods, in which learning signals are derived from the images themselves rather than relying exclusively on manually assigned labels. Some methods ask models to predict missing or altered portions of an image. Others train representations to capture relationships among different views or transformations of the same content.
These approaches can help models learn useful visual representations from large collections of unlabeled images. A model trained this way may subsequently be fine-tuned for a specific task using labeled examples. The effectiveness of any training strategy depends on the data, objective, architecture, and intended application.
The resulting model is not simply a database of memorized images. It learns parameterized transformations that can generalize to unfamiliar examples, although the reliability of that generalization varies and must be evaluated.
Where vision transformers are used
Image classification is one of the clearest applications of vision transformers. Given an image, a classifier produces scores for possible categories. The model’s final representation summarizes visual evidence in a form that a classification head can use to distinguish among those categories.
Many computer vision tasks require more detailed outputs than a single label. Object detection involves identifying objects and estimating their locations, often through bounding boxes. Image segmentation assigns labels to individual pixels or regions, distinguishing, for example, a person from the surrounding street. Vision transformer architectures can support these tasks by producing spatial representations and combining them with task-specific prediction components.
Some models process an image at multiple scales or use hierarchical representations in which tokens correspond to progressively larger regions. These designs can be useful when a task depends on both fine detail and broad scene context. They also help manage the computational demands of processing high-resolution images.
Vision transformers are also important in multimodal systems that work with both images and text. A model may learn to map visual representations and text representations into a compatible space, allowing it to compare an image with descriptions, retrieve relevant images, or support question answering about visual content. Other systems combine a vision encoder with a language model so that visual information can contribute to text generation.
In these systems, attention may operate within an image, within a text sequence, or across representations from different modalities, depending on the architecture. The general principle remains the same: learned relationships determine how information from different elements is combined.
Applications include document analysis, industrial inspection, scientific imaging, robotics, and medical image analysis. In each case, the model’s usefulness depends on more than its architecture. Image quality, training data, task-specific validation, computational requirements, and the consequences of errors all matter. In high-stakes settings, model predictions may need expert review and careful assessment before they can support decisions.
Why attention has computational costs
Global self-attention offers a powerful way to connect image regions, but it becomes expensive as the number of patches increases.
If an image produces NNN tokens, the attention operation typically computes relationships between every pair of tokens. The resulting attention matrix contains N2N^2N2 pairwise scores per attention head. Consequently, the main computational and memory costs associated with standard attention grow approximately quadratically with sequence length, although the total cost of a transformer also includes other operations.
This growth matters because the number of patches rises rapidly when image resolution increases or patch size decreases. Doubling the number of tokens can increase the size of the attention matrix by roughly a factor of four. An image that requires modest resources at one resolution may therefore become much more expensive to process at a substantially higher resolution.
Researchers and engineers use several strategies to address this limitation. Windowed attention restricts attention to local groups of patches, reducing the number of relationships computed at once. Some hierarchical models gradually combine local information into coarser representations. Other approaches reduce the number of tokens, approximate attention computations, or use implementations that lower memory requirements.
These methods involve trade-offs. Restricting attention can improve efficiency but may limit direct communication between distant regions. Hierarchical processing can reduce sequence length but may discard details during downsampling. More efficient attention algorithms can reduce practical costs, yet they do not necessarily eliminate every computational constraint.
Vision transformers therefore involve architectural choices about how much context to preserve, how finely to represent an image, and how much computation is practical for the task.
How vision transformers compare with convolutional neural networks
Convolutional neural networks, or CNNs, have long been a central architecture in computer vision. They use filters that operate over local image neighborhoods and share the same learned parameters across different spatial locations. This design gives them a strong built-in preference for local patterns and makes efficient use of the regular structure of images.
A standard vision transformer has a different starting point. It converts image patches into tokens and uses attention to learn relationships among those tokens. Rather than relying entirely on fixed local neighborhoods for information exchange, global self-attention can connect distant patches directly.
Neither approach is universally superior. CNNs can be highly efficient, and their architectural assumptions often suit visual data well. Vision transformers offer flexible global interactions and can learn powerful representations, particularly when supported by suitable training data and computational resources. Hybrid architectures combine convolutional operations and transformer blocks to draw on the strengths of both.
Training conditions are important in this comparison. Some vision transformers benefit substantially from large-scale pretraining, while carefully designed models can perform well with more limited data. CNNs also benefit from extensive training and pretraining, and their performance depends on architecture, optimization, and the task being evaluated. There is no single rule that determines which family will perform best in every setting.
The choice should reflect the problem being solved, the available data, the need for local detail or broad context, the computational budget, and the reliability requirements of the application.
What vision transformers can and cannot tell us about an image
Vision transformers demonstrate that a model can learn useful visual representations by combining information across a sequence of image regions. But their success at prediction should not be confused with human-like understanding.
A model may correctly classify an image without representing every object or relationship in a way that corresponds to human concepts. It can exploit statistical regularities that work well on familiar data but fail when conditions change. Altered lighting, unusual viewpoints, unfamiliar environments, image corruption, or differences between training and deployment data can all affect performance.
Attention maps can offer clues about how a model distributes information among image patches. They can reveal which tokens receive relatively high or low attention weights in a particular layer and head. Yet a high attention weight does not automatically mean that a patch is the main reason for the model’s final decision. The prediction depends on a chain of transformations, including value projections, feed-forward layers, residual connections, and later attention operations.
Evaluating a model’s reliability therefore requires more than examining its attention patterns. Researchers test performance on held-out examples, assess robustness to changes in input conditions, investigate errors, and use interpretability methods suited to the questions they want to answer. The appropriate evaluation depends on whether the model is intended to classify ordinary photographs, inspect manufactured components, interpret scientific imagery, or assist with a safety-critical task.
Another important limitation is that attention does not eliminate the need for useful visual representations. If the patch embeddings fail to preserve important details, later layers may not recover them reliably. Similarly, positional information, training objectives, and architectural choices all influence which relationships the model can learn.
The enduring contribution of vision transformers is their approach to visual information processing. By representing an image as a collection of tokens and using attention to exchange information among them, they provide a flexible way to learn relationships across space. Their capabilities arise from the interaction of this mechanism with learned representations, training data, and the rest of the network—not from attention alone.
Understanding that distinction helps explain both why vision transformers have become important in computer vision and why their performance must still be judged by careful evidence rather than by the apparent sophistication of their internal computations.