Think Together · Start with a Short Story
The girl reads, “The cup hit the plate, and it broke,” and wonders what “it” means. How can a model connect information from different positions?
Lay Out the Process
- Current positionNeeds relevant information
- Compare representationsCalculate relationship weights
- Combine with weightsGather context
- Continue processingForm a new representation
The attention mechanism combines information from different positions using calculated weights.
It is a little like referring to related words while reading, but in reality it is a numerical operation, not a person's gaze.
This mechanism lets a model use clues from positions both near and far away. If a sentence is ambiguous, the model may still be unable to identify the right reference.
Now You Try
If “it broke” has too little context, must the model identify one definite reference?
Ready? Open this to see the explanation
No. It should not pretend the answer is certain. It can point out the ambiguity or ask for more information.
One Idea to Take Away
The attention mechanism combines information from different positions using calculated weights.
Try covering the diagram above and explaining each step to someone in your family. If one step is hard to explain, return to that card and look again.