What is evaluation bias?

Answer

When the common datasets used for evaluating research / tools themselves exhibit representation bias. When the benchmark that is widely used has bias, it is replicated at scale.

Relevant part of lecture

supplementary material