Arguments against a standardised benchmark suite

From GPBenchmarks
Jump to: navigation, search

Defining a standardised benchmark suite for GP has both advantages and disadvantages. Here we collect some of the arguments against the idea.

Benchmarks are not an end in themselves

This paper discusses the current state of the machine learning community, how it has perhaps lost its focus on problem-solving and instead focused solely on small statistical improvements to benchmark datasets. New researchers learn how to make small improvements on existing problems, but don't get much practise in formulating new problems.

Machine Learning that Matters, Wagstaff, KL.

Another paper from the ML point of view: C. Drummond and N. Japkowicz, Warning: statistical benchmarking is addictive. Kicking the habit in machine learning. Journal of Experimental & Theoretical Artificial Intelligence, 22(1):67–80, 2010. Again, the argument is that performance on standardised benchmarks can become an end in itself.

Personal tools