Calculate the dot product of two vectors used in embeddings and attention scores.
The dot product of two vectors is the sum of element-wise products. It equals ||A|| × ||B|| × cos(θ) where θ is the angle between them. It is fundamental in neural network computations, attention mechanisms, and similarity measures.
Dot Product
a · b = sum(a_i * b_i)
A positive dot product means the vectors point in roughly the same direction (angle < 90°), zero means they are orthogonal, and negative means they point in opposite directions (angle > 90°).
Every neuron computes a dot product between its weight vector and the input vector, followed by a bias addition and activation function. Attention mechanisms also rely heavily on dot products between query and key vectors.