Calculate the number of trainable parameters in a recurrent neural network layer.
A vanilla RNN layer has two weight matrices: W_ih (input to hidden, size input×hidden) and W_hh (hidden to hidden, size hidden×hidden), plus two bias vectors of size hidden. Total = input×hidden + hidden² + 2×hidden.
RNN Parameters
params = input_size × hidden_size + hidden_size^2 + 2 × hidden_size
LSTMs have 4× the parameters of a vanilla RNN because they have four gate matrices (input, forget, cell, output) instead of one.
The hidden-to-hidden matrix is h×h, which grows quadratically. For large hidden sizes (512+), this dominates total parameters.