Skip to main content
Practice

Understanding Sentences from Multiple Perspectives with Multi-Head Attention

Multi-Head Attention is a mechanism that runs multiple Self-Attention operations in parallel to better capture the different relationships in a sentence.

While a single Self-Attention layer can capture key word relationships, understanding complex contexts from only one perspective is often limited.

To solve this, multiple parallel Self-Attention layers were introduced, enabling the model to interpret the sentence from several viewpoints.


Example of Multi-Head Attention​

Let's consider the sentence "The student is sitting at the desk reading a book."

Multi-Head Attention understands this sentence from various perspectives, such as:

  • Attention 1: Focusing on the relationship between student and sitting

  • Attention 2: Focusing on the relationship between book and reading

Applying these multiple perspectives simultaneously helps Multi-Head Attention build a richer understanding of the sentence’s meaning.


How Does Multi-Head Attention Work?​

  1. The input sentence is duplicated and passed into multiple Self-Attention structures.

  2. Each structure independently uses different weights to calculate relationships between the words.

  3. The output from all structures is then combined.

  4. Finally, this consolidated information is used to represent the sentence's meaning.

This process allows the model to incorporate diverse relational insights at once, producing a more accurate overall sentence representation.


Multi-Head Attention is a key component that enables Transformer models to understand sentences with greater precision.

In the next lesson, we'll apply what we've learned so far to solve a simple quiz.

Want to learn more?

Join CodeFriends Plus membership or enroll in a course to start your journey.