AI Glossary
Self-Attention
A type of attention mechanism that allows a model to examine relationships between different parts of the same input.
Overview
As AI systems became more advanced, researchers realized that models needed better ways to understand relationships within a piece of information.
Consider the sentence:
“The student submitted the assignment because she finished it early.”
Humans immediately understand that “she” refers to the student. For an AI system, identifying that relationship requires understanding how words relate to one another.
This is where self-attention becomes important.
Self-attention allows a model to examine different parts of the same input and determine which pieces of information are most relevant to one another. Instead of processing words independently, the model learns relationships across the entire sequence.
A helpful way to think about self-attention is reading comprehension.
When humans read a paragraph, they constantly connect ideas, references, and context. Self-attention helps AI models perform a similar process mathematically.
This capability became one of the key innovations behind transformer models, which now power many modern AI systems including large language models.
Understanding self-attention helps explain why modern AI systems are significantly better at understanding context than many earlier approaches.
Why It Matters
Self-attention helps AI models understand relationships between different parts of an input.
Real-World Example
When a chatbot generates a response, self-attention helps it understand how words and phrases within a conversation relate to one another.