Attention Mechanism

Attention Pooling

Definition

1. Dataset: \(m\) tuples of keys and values

   \[\mathcal{D} \stackrel{\textrm{def}}{=} \{(\mathbf{k}_1, \mathbf{v}_1), \ldots (\mathbf{k}_m, \mathbf{v}_m)\}\]

2. Attention Pooling: query \(q\) that operate on (\(k\), \(v\)) pairs

   \[\textrm{Attention}(\mathbf{q}, \mathcal{D}) \stackrel{\textrm{def}}{=} \sum_{i=1}^m \alpha(\mathbf{q}, \mathbf{k}_i) \mathbf{v}_i,\]

Attention Pooling by Similarity

Kernel Regression

1. Non-parametric Model: Captures complex, non-linear relationships.

2. Focus on Similarity: Kernel shape and bandwidth control how it adapts to local data structure.

3. Kernel Function: Less sensitive to outliers than parametric models.

Attention Scoring Function

Sequence-to-sequence Model

The Bahdanau Attention Mechanism

Multi-Head Attention

CNNs, RNNs, and Self-Attention

The Transformer Architecture

The Vision Transformer Architecture

Large-Scale Pretraining with Transformers

1. Encoder-Only

   Pretraining BERT

   

   Fine-Tuning BERT

   

2. Encoder–Decoder

   Pretraining T5

   

   Fine-Tuning T5

   

3. Decoder-Only

   GPT-2

   

   GPT-3