Calculate the memory footprint required to hold a given LLM context length.
Context memory scales linearly with sequence length. For each token in context, the model must store activations and KV cache entries across all layers. Doubling context length approximately doubles inference memory requirements, which is why long-context models need significantly more VRAM.
Context Memory
memory = 3 × context_length × hidden_dim × num_layers × bytes_per_element
Each token in context requires KV cache entries stored across all attention layers. With 32 layers and 4096-dim hidden state, each token adds ~512KB of KV cache in FP16.
Use quantized KV cache (INT8), sliding window attention, GQA (grouped query attention), or sparse attention patterns to reduce memory per token.