The attention mechanism computes a set of weights over the tokens in the input (or context) sequence for each step of processing, reflecting how relevant each input token is to the computation currently being performed — for instance, how relevant each word in a source sentence is to correctly translating a given target word, or how relevant each prior token is to predicting the next one in an autoregressive model. Mechanically, this is computed via query, key, and value projections: a query (representing the current focus) is compared against keys (representing each input token) to produce attention scores, which are normalized (typically via softmax) into weights and used to compute a weighted sum over the corresponding values — allowing the model to dynamically focus more on relevant tokens and less on irrelevant ones, rather than treating all input tokens with equal importance.
Option D describes positional encoding's role (covered directly in an earlier question in this set) — capturing token order — which is a distinct mechanism from attention; attention operates on token *content and relevance*, while positional encoding separately supplies *order* information as an input feature, since self-attention itself is permutation-invariant without it. Option A misdirects the weighting toward the output sequence specifically, when attention weights are computed primarily over the input/context tokens being attended to. Option C describes the decoding/generation procedure (autoregressive sampling), not attention's mechanism.
[Reference: Core Machine Learning and AI Knowledge domain — transformer attention mechanism vs. positional encoding., ]
Contribute your Thoughts:
Chosen Answer:
This is a voting comment (?). You can switch to a simple comment. It is better to Upvote an existing comment if you don't have anything to add.
Submit