Using Heuristic-Search Based Engines for Estimating Human Skill at Chess

Abstract

Establishing heuristic-search based chess programs as appropriate tools for estimating human skill levels at chess may seem impossible due to the following issues: the programs’ evaluations and decisions tend to change with the depth of search and with the program used. In this research, we provide an analysis of the differences between heuristic-search based programs in estimating chess skill. We used four different chess programs to perform analyses of large data sets of recorded human decisions, and obtained very similar rankings of skill-based performances of selected chess players using any of these programs at various levels of search. A conclusion is that, given two chess players, all the programs unanimously rank one player to be clearly stronger than the other, or all the programs assess their strengths to be similar. We also repeated our earlier analysis with the program CRAFTY of the human World Chess Champions with currently one of the strongest chess programs, RYBKA 3², and obtained qualitatively very similar results as with CRAFTY. This speaks in favour of computer heuristic search being adequate for estimating skill levels of chess players, despite the above stated issues.