Calculate the parameter count and memory footprint of an embedding layer.
An embedding layer is a lookup table of shape (vocabulary_size × embedding_dimension). Each token in the vocabulary has its own dense vector. For GPT-2's 50,257 vocab with 768-dim embeddings, that's 38.6M parameters taking ~73 MB in FP16.
Embedding Size
params = vocab_size × embedding_dim; memory = params × bytes_per_param
Yes — embedding vectors are learned during training and encode semantic meaning. Similar words end up with similar embedding vectors (word2vec property).
Large vocabularies (32K-100K+ tokens) combined with high embedding dimensions (4096+) mean embedding layers can have hundreds of millions of parameters — a significant fraction of smaller models.