Compute sinusoidal positional encoding values for a given position and dimension.
Sinusoidal positional encoding injects position information into transformer inputs without learned parameters. Even dimensions use sin, odd dimensions use cos, with wavelengths forming a geometric progression from 2π to 10000×2π, allowing the model to attend to relative positions.
Positional Encoding
PE(pos, 2i) = sin(pos / 10000^(2i/d_model)); PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))
Sinusoidal encodings generalize to longer sequences than seen during training and don't add trainable parameters. However, modern models (GPT, BERT) often use learned positional embeddings or RoPE instead.
10000 creates a range of wavelengths so that nearby positions have similar encodings while distant positions are easily distinguishable. It was chosen empirically in the original Transformer paper.