Calculate how much of a sequence will be truncated to fit a maximum input length.
When an input sequence exceeds a model's maximum context length, it must be truncated — typically by cutting tokens from the end (or sometimes the middle or start, depending on the strategy) — which discards information. This calculator computes the truncated length as min(original, max), the number of tokens lost, and the percentage of the original sequence discarded, helping you gauge how much information loss to expect for long documents.
tokens_lost = max(0, sequence_length - max_length)
Convention varies by framework — many NLP pipelines truncate from the end by default, but some tasks (e.g. summarization of long documents) benefit from truncating the middle or keeping both the start and end.
It depends on the task — for classification tasks where key information appears early, even high loss percentages may not hurt performance; for tasks needing full-document context, any truncation risks missing critical information.
Consider chunking the document into multiple overlapping windows and aggregating predictions, or using a model with a longer context window if truncation loss is unacceptable.