Positional Encoding Calculator
Compute sinusoidal positional encoding values for a given position and dimension.
Inputs
PE sin (even dim)
-0.544021
PE cos (odd dim)
-0.839072
Angle
10.000000
Step by step
Angle = pos / 10000^(2i/d_model)
10 / 10000^(0/512)
= 10.000000
PE(pos, 2i) = sin(angle)
sin(10.0000)
= -0.544021
PE(pos, 2i+1) = cos(angle)
cos(10.0000)
= -0.839072
How it works
Sinusoidal positional encoding injects position information into transformer inputs without learned parameters. Even dimensions use sin, odd dimensions use cos, with wavelengths forming a geometric progression from 2π to 10000×2π, allowing the model to attend to relative positions.
Formula
Positional Encoding
PE(pos, 2i) = sin(pos / 10000^(2i/d_model)); PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))
- pos
- Token position in the sequence
- i
- Dimension index
Frequently Asked Questions
Why use sin and cos instead of learned embeddings?
Sinusoidal encodings generalize to longer sequences than seen during training and don't add trainable parameters. However, modern models (GPT, BERT) often use learned positional embeddings or RoPE instead.
Why the 10000 base?
10000 creates a range of wavelengths so that nearby positions have similar encodings while distant positions are easily distinguishable. It was chosen empirically in the original Transformer paper.
You might also need
- Embedding Layer Size CalculatorCommonly used together
- Multi-head Attention Parameters CalculatorCommonly used together
- Transformer Parameters CalculatorCommonly used together
- ConvTranspose Output CalculatorAlso in Neural Networks
- Trainable Parameters CalculatorAlso in Neural Networks
- CNN Output Shape CalculatorAlso in Neural Networks