Engine Similarity
Engine Similarity,
a loosely defined relation between (two) chess engines concerning their positional playing style, therefor mostly dependent on their evaluation features and weights.
One approach to assess engine similarity is to count how often these engines agree with the same move after a shallow search
over a set of carefully selected, quiet test positions with apparently several possible best moves - another, in k-best mode, how often they propose the same ranking of k moves.
A relative high similarity measure between two engines [2] could be a symptom of using similar evaluation ideas, features and weights, along with automated tuning (or trainig), or even code copying aka cloning.
Similarity testing is intended as first automated "screening" to trigger further investigations in case moves are too similar.
Contents
Similarity Testers
In December 2010, Don Dailey published SIM03 to test the similary with a set of UCI compliant engines with 8238 in-build positions - still freely available from the Komodo site [3]. A pair of different chess engines could then be compared for similarity by matching up and counting the number of positions for which each engine chose the same move. SIMEX is a more user friendly successor by Ed Schröder [4] using external EPD-files. Overall similarity of multiple engines may be visualized as hierarchical clustering dendrogram [5].
See also
- Category:Clone
- Category:Derivative
- ICGA Investigations
- Engine Testing
- Match Statistics
- Playing Strength
Publications
- Ingo Althöfer (1997). On the k-best Mode in Computer Chess: Measuring the Similarity of Move Proposals. ICCA Journal, Vol. 20, No. 3
- Mark Levene, Judit Bar-Ilan (2005). Comparing Move Choices of Chess Search Engines. ICGA Journal, Vol. 28, No. 2, pdf
- Paolo Ciancarini, Gian Piero Favini (2009). Detecting clones in game-playing software. Entertainment Computing, Vol. 1, No. 1
- Don Dailey, Adam Hair, Mark Watkins (2014). Move Similarity Analysis in Chess Programs. Entertainment Computing, Vol. 5, No. 3, preprint as pdf [6]
Forum Posts
1999
- Chess program similarity experiment by Bruce Moreland, CCC, January 24, 1999
- This test is not scientific! by Don Dailey, CCC, January 26, 1999
- Re: This test is not scientific! by Don Dailey, CCC, January 26, 1999
- Comparing Bionic-Impakt vs Bionic by Albrecht Heeffer, CCC, January 27, 1999 » Bionic Impakt, Bionic
2000 ...
- question to vas on similarity of rybka 1.0 to fruit by duncan, Rybka Forum, August 25, 2008
2010 ...
- Similarity Detector Available by Don Dailey, CCC, December 26, 2010
- Cluster analysis of similarity test by Miguel A. Ballicora, CCC, December 31, 2010
- Similarity testing for source code by Mark Lefler, CCC, February 16, 2011
- Pairwise Analysis of Chess Engine Move Selections by Adam Hair, CCC, April 17, 2011
- Fritz and Naum shown as Rybka/Strelka clones by Kai Laskos, CCC (Engine Origins), October 17, 2011
- Pairwise Analysis of Chess Engine Move Selections Revisited by Adam Hair, CCC, March 04, 2012
- Similarity tool myth - debunked by Don Dailey, CCC, August 23, 2012
- Fritz 11 by Rebel, OpenChess Forum, December 10, 2013
2015 ...
- Similarity test by Martin Sedlak, CCC, January 24, 2015 » Cheng
- Similarity test for Capivara > version 0.0.8 available? by Günther Simon, CCC, May 01, 2016 » Capivara
- New Similarity Dendrogram by Kai Laskos, CCC, December 03, 2016 (Images broken links)
- Experiment 11 - Similarity between the top engines back then and now by Ed Schröder, CCC, March 14, 2019
- How to measure overall similarity by Ferdinand Mosca, CCC, April 02, 2019
- Why all Lc0 runs result in such similarity of quiet moves selection? by Kai Laskos, CCC, May 01, 2019
- Similarity tester - 2nd generation - BETA by Ed Schröder, CCC, August 09, 2019
- What the heck happens here? by Kai Laskos, CCC, August 20, 2019
- SIMEX 2.1 by Ed Schröder, CCC, September 01, 2019
- Similarity Report 2019 by Ed Schröder, CCC (Engine Origins), September 23, 2019
2020 ...
- Maybe not the best diversity of strongest chess engines under development by Kai Laskos, CCC, November 14, 2020 » NNUE
- NNUE Research Project by Ed Schröder, CCC, March 10, 2021 » NNUE
- Simex including NNUE by jjoshua2, CCC, March 11, 2021 » NNUE
External Links
Chess Engines
- Komodo Chess Engine - Official Site with Similary tester version 03 download
- Similary tester 2nd generation (SIMEX) by Ed Schröder
- Similarity Report 2019 by Ed Schröder
- nnue | Home of the Dutch Rebel by Ed Schröder » NNUE
- GitHub - fsmosca/Similarity-Dendrogram: Plot dendrogram on chess engines similarity by Ferdinand Mosca
Misc
- Similarity from Wikipedia
- Similarity (geometry) from Wikipedia
- Lexical similarity from Wikipedia
- Semantic similarity from Wikipedia
- Structural similarity from Wikipedia
- Similarity measure from Wikipedia
- Dendrogram from Wikipedia
- Hierarchical clustering from Wikipedia
References
- ↑ A phylogenetic tree of living things, based on RNA data and proposed by Carl Woese, showing the separation of bacteria, archaea, and eukaryotes. Trees constructed with other genes are generally similar, although they may place some early-branching groups very differently, thanks to long branch attraction. The exact relationships of the three domains are still being debated, as is the position of the root of the tree. It has also been suggested that due to lateral gene transfer, a tree may not be the best representation of the genetic relationships of all organisms. For instance some genetic evidence suggests that eukaryotes evolved from the union of some bacteria and archaea (one becoming an organelle and the other the main cell). Autor: Eric Gaba, September 2006, Wikimedia Commons
- ↑ i.e. > 65% in SIM03
- ↑ Komodo Chess Engine - Official Site with Similary tester version 03 download
- ↑ SIMEX 2.1 provided by Ed Schröder, CCC, September 01, 2019
- ↑ How to measure overall similarity by Ferdinand Mosca, CCC, April 02, 2019
- ↑ Pairwise Analysis of Chess Engine Move Selections by Adam Hair, CCC, April 17, 2011
