Calculate BERTScore, a contextual embedding-based metric for text similarity evaluation.
BERTScore evaluates text generation quality using contextual embeddings from BERT. It computes token-level cosine similarity between candidate and reference embeddings, then aggregates using greedy matching to produce precision, recall, and F1 scores that capture semantic similarity beyond surface-level n-gram matching.
BERTScore F1
F1 = 2 * P_BERT * R_BERT / (P_BERT + R_BERT)
BERTScore captures semantic equivalence (paraphrases, synonyms) that surface-level n-gram metrics miss entirely. 'The cat sat on the mat' and 'A feline rested on the rug' have low BLEU but high BERTScore.
The default is often RoBERTa-large, but any contextual embedding model can be used. The choice affects absolute scores but relative rankings tend to be consistent across model choices.