Embedding Dimension Calculator
Estimate an appropriate embedding dimension size for a given vocabulary or dataset.
Inputs
Recommended Dimension
239
Nearest Power of 2
256
Step by step
Heuristic: vocab^0.25 × 16
50000^0.25 × 16
= 239
Recommended (clamped 64-4096)
clamp(239, 64, 4096)
= 239
Nearest power of 2 (GPU-friendly)
2^ceil(log2(239))
= 256
How it works
Embedding dimension should be large enough to capture semantic relationships but not so large that it wastes compute and memory. A common heuristic is dimension ≈ vocab^0.25 × constant. Standard choices are 256, 384, 768, or 1536 for most applications.
Formula
Dimension Heuristic
dim ≈ vocab_size^0.25 × 16 (clamped to practical range)
- vocab_size
- Number of unique items to embed
Frequently Asked Questions
Why use powers of 2?
GPU hardware is optimized for dimensions that are powers of 2 or multiples of 64/128. Non-aligned dimensions waste memory bandwidth and CUDA core utilization.
What dimensions do popular models use?
OpenAI text-embedding-3-small uses 1536, Cohere uses 1024, and sentence-transformers models commonly use 384 or 768 dimensions.