Changes

Jump to: navigation, search

Match Statistics

8,794 bytes added, 10:58, 4 February 2022
no edit summary
return ''
</pre>
 
Beside the above SPRT implementation using pentanomial frequencies and a simulation tool in [[Python]] <ref>[http://www.talkchess.com/forum3/viewtopic.php?f=7&t=72962&start=6 Re: Stockfish Reverts 5 Recent Patches] by [[Michel Van den Bergh]], [[CCC]], February 02, 2020</ref> <ref>[https://github.com/vdbergh/pentanomial GitHub - vdbergh/pentanomial: SPRT for pentanomial frequencies and simulation tools] by [[Michel Van den Bergh]]</ref>, [[Michel Van den Bergh]] wrote a much faster multi-threaded [[C]] version <ref>[http://www.talkchess.com/forum3/viewtopic.php?f=7&t=72962&start=7 Re: Stockfish Reverts 5 Recent Patches] by [[Michel Van den Bergh]], [[CCC]], February 02, 2020</ref> <ref>[https://github.com/vdbergh/simul GitHub - vdbergh/simul: A multi-threaded pentanomial simulator] by [[Michel Van den Bergh]]</ref>.
 
<span id="TournamentManager"></span>
=Tournaments=
* [[Depth#DiminishingReturns|Depth | Diminishing Returns]]
* [[Draw]]
* [[Engine Similarity]]
* [[LOS Table]]
* [[Pawn Advantage, Win Percentage, and Elo]]
=Publications=
==1920 ...==
* [[Mathematician#LLThurstone|L. L. Thurstone]] ('''1927'''). ''[https://psycnet.apa.org/record/1928-00527-001 A law of comparative judgement]''. [https://en.wikipedia.org/wiki/Psychological_Review Psychological Review], Vol. 34, No. 4 <ref>[https://en.wikipedia.org/wiki/Law_of_comparative_judgment Law of comparative judgment - Wikipedia]</ref>
* [[Ernst Zermelo]] ('''1929'''). ''Die Berechnung der Turnier-Ergebnisse als ein Maximumproblem der Wahrscheinlichkeitsrechnung''. [http://gdz.sub.uni-goettingen.de/dms/load/img/?IDDOC=82727 pdf] (German)
==1930 ...==
* [[Mathematician#ACAitken|Alexander Aitken]] ('''1935'''). ''[https://www.cambridge.org/core/journals/proceedings-of-the-royal-society-of-edinburgh/article/ivon-least-squares-and-linear-combination-of-observations/7106C26F19F2EBF75BCEE7FA285780B9 On Least Squares and Linear Combinations of Observations]''. Proceedings of the [https://en.wikipedia.org/wiki/Royal_Society_of_Edinburgh Royal Society of Edinburgh]
==1940 ...==
* [[Mathematician#AWald|Abraham Wald]] ('''1945'''). ''Sequential Tests of Statistical Hypotheses''. [https://en.wikipedia.org/wiki/Annals_of_Mathematical_Statistics Annals of Mathematical Statistics], Vol. 16, No. 2, [https://en.wikipedia.org/wiki/Digital_object_identifier doi]: [http://projecteuclid.org/euclid.aoms/1177731118 10.1214/aoms/1177731118]
* [[Mathematician#AWald|Abraham Wald]] ('''1947'''). ''Sequential Analysis''. [https://en.wikipedia.org/wiki/John_Wiley_%26_Sons John Wiley and Sons], [http://www.abebooks.com/book-search/title/sequential-analysis/author/abraham-wald/ AbeBooks]
==1950 ...==* [[Mathematician#Mosteller|Frederick Mosteller]] ('''1951'''). ''[https://psycnet.apa.org/record/1951-07176-001 Remarks on the method of paired comparisons: I. The least squares solution assuming equal standard deviations and equal correlations]''. [https://en.wikipedia.org/wiki/Psychometrika Psychometrika], Vol. 16, No. 1* [[Mathematician#RABradley|Ralph A. Bradley]], [[Mathematician#METerry|Milton E. Terry]] ('''1952'''). ''[http://biomet.oxfordjournals.org/content/39/3-4/324.citation Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons]''. [https://en.wikipedia.org/wiki/Biometrika Biometrika], Vol. 39, Nos. 3/4, [https://en.wikipedia.org/wiki/Digital_object_identifier doi]: 10.2307/2334029, [http://www.jstor.org/stable/2334029?seq=1#page_scan_tab_contents JSTOR 2334029]
==1960 ...==
* [[Mathematician#WAGlenn|William A. Glenn]], [[Mathematician#HADavid|Herbert A. David]] ('''1960'''). ''[https://www.jstor.org/stable/2527957?seq=1#page_scan_tab_contents Ties in Paired-Comparison Experiments Using a Modified Thurstone-Mosteller Model]''. [https://en.wikipedia.org/wiki/Biometrics_(journal) Biometrics], Vol. 16, No. 1* [[Mathematician#FNDavid|Florence Nightingale David]] ('''1962'''). ''[http://books.google.com/books/about/Games_Gods_and_Gambling.html?id=8ddP8zNx9nQC&redir_esc=y Games, Gods & Gambling: A History of Probability and Statistical Ideas]''. [https://en.wikipedia.org/wiki/Dover_Publications Dover Publications]* [[Mathematician#PVRao|P. V. Rao]], ISBN[[Mathematician#LLKupper|L. L. Kupper]] ('''1967'''). ''[https://www.tandfonline.com/doi/abs/10.1080/01621459.1967.10482901 Ties in Paired-13Comparison Experiments: 978A Generalization of the Bradley-0486400235Terry Model]''. [https://en.wikipedia.org/wiki/Journal_of_the_American_Statistical_Association Journal of the American Statistical Association], Vol. 62, No. 317==1970 ...==* [https://www.semanticscholar.org/author/Roger-R.-Davidson/32819150 Roger R. Davidson] ('''1970'''). ''[https://www.jstor.org/stable/2283595 On Extending the Bradley-Terry Model to Accommodate Ties in Paired Comparison Experiments]''. [https://en.wikipedia.org/wiki/Journal_of_the_American_Statistical_Association Journal of the American Statistical Association], Vol. 64, No. 329* <span id="Bloss"></span>[http://what-when-how.com/earth-scientists/bloss-f-donald-earth-scientist/ F. Donald Bloss] ('''1973'''). ''[https://www.amazon.de/Rate-Your-Chess-F-Donald-Bloss/dp/0442008295 Rate your own Chess]''. Van Nostrand Reinhold Inc.
* [[Tony Marsland]], [[Paul Rushton]] ('''1973'''). ''[http://dl.acm.org/citation.cfm?id=805703 Mechanisms for Comparing Chess Programs].'' [[ACM 1973|ACM Annual Conference]], [http://webdocs.cs.ualberta.ca/~tony/OldPapers/Marsland-Rushton-ACM73 pdf]
* [[James Gillogly]] ('''1978'''). ''Performance Analysis of the Technology Chess Program''. Ph.D. Thesis. Tech. Report CMU-CS-78-189, [[Carnegie Mellon University]], [http://reports-archive.adm.cs.cmu.edu/anon/anon/usr/ftp/scan/CMU-CS-77-gillogly.pdf CMU-CS-77 pdf] » [[Tech]]
* [[David Cahlander]] ('''1979'''). ''Strength of a Chess Playing Computer''. [[ICGA Journal#2_1|ICCA Newsletter, Vol. 2, No. 1]]
* [[Jack Good]] ('''1979'''). ''On the Grading of Chess Players''. [[Personal Computing#3_3|Personal Computing, Vol. 3, No. 3]], pp. 47
* <span id="Ratliff"></span>[[Gary L. Ratliff]] ('''1979'''). ''Practical Rating Program''. [[Personal Computing#3_9|Personal Computing, Vol. 3, No. 9]], pp. 62 » [[#Bloss|Bloss]]
* [[Frieder Schwenkel]] ('''1979'''). ''Berating the ratings system''. [[Personal Computing#3_11|Personal Computing, Vol. 3, No. 11]], pp. 77 » [[#Ratliff|Ratliff]], [[#Bloss|Bloss]]
* [[John Shaposka]] ('''1979'''). ''"JS" Takes the Bloss Test''. [[Personal Computing#3_12|Personal Computing, Vol. 3, No. 12]], pp. 75 » [[#Ratliff|Ratliff]], [[#Bloss|Bloss]]
==1980 ...==
* Floyd R. Kirk ('''1980'''). ''Bloss Flunks Test''. [[Personal Computing#4_8|Personal Computing, Vol. 4, No. 8]], pp. 72 » [[#Ratliff|Ratliff]], [[#Bloss|Bloss]]
* [[John F. White]] ('''1981'''). ''[http://yourcomputeronline.wordpress.com/2010/12/10/survey-chess-games/ Survey-Chess Games]''. [[Your Computer]], [http://yourcomputeronline.wordpress.com/2010/10/31/augustseptember-1981-contents-and-editorial/ August/September 1981] <ref>[https://en.wikipedia.org/wiki/The_Master_Game The Master Game from Wikipedia]</ref>
* [[Ken Thompson]] ('''1982'''). ''Computer Chess Strength''. [[Advances in Computer Chess 3]]
==1990 ...==
* [[Hans Berliner]], [[Gordon Goetsch]], [[Murray Campbell]], [[Carl Ebeling]] ('''1990'''). ''Measuring the Performance Potential of Chess Programs.'' [https://en.wikipedia.org/wiki/Artificial_Intelligence_%28journal%29 Artificial Intelligence], Vol. 43, No. 1
* [http://www.ics.uci.edu/~sternh/ [Mathematician#HSStern|Hal Stern]] ('''1990'''). ''Are all Linear Paired Comparison Models Equivalent''. [http://www.dtic.mil/dtic/tr/fulltext/u2/a236856.pdf pdf]
* [[Eric Hallsworth]] ('''1990'''). ''Speed, Processors and Ratings''. [[Selective Search|Computer Chess News Sheet]] 25, pp 6, [http://www.chesscomputeruk.com/SS_25.pdf pdf] hosted by [[Mike Watters]]
* [[Hans Berliner]], [[Danny Kopec]], [[Ed Northam]] ('''1991'''). ''A taxonomy of concepts for evaluating chess strength: examples from two difficult categories''. [[Advances in Computer Chess 6]], [http://www.sci.brooklyn.cuny.edu/%7Ekopec/Publications/Publications/O_20_C.pdf pdf]
* [[Steve Maughan]] ('''1992'''). ''Are You Sure It's Better?'' [[Selective Search]] 40, pp. 21, [http://www.chesscomputeruk.com/SS_40.pdf pdf] hosted by [[Mike Watters]]
* [[Warren D. Smith]] ('''1993'''). ''Rating Systems for Gameplayers, and Learning''. [http://scorevoting.net/WarrenSmithPages/homepage/ratingspap.ps ps]
* [[Mark E. Glickman]] ('''1993'''). ''Paired Comparison Models with Time Varying Parameters''. Ph.D. thesis, [[Harvard University]], advisor [[Mathematician#HSStern|Hal Stern]], [http://www.glicko.net/research/thesis.pdf pdf]
* [[Mark E. Glickman]] ('''1995'''). ''A Comprehensive Guide To Chess Ratings''. [http://www.glicko.net/research/acjpaper.pdf pdf]
* [[Mark E. Glickman]], [[Christopher Chabris]] ('''1996'''). ''Using Chess Ratings as Data in Psychological Research''. [https://pdfs.semanticscholar.org/1ffd/3432f56476f0047426b37f7f433f5a6575b0.pdf pdf]
* [[Robert Hyatt]], [[Monroe Newborn]] ('''1997'''). ''CRAFTY Goes Deep''. [[ICGA Journal#20_2|ICCA Journal, Vol. 20, No. 2]] » [[Crafty]]
* [[Mark E. Glickman]], [[Mathematician#ACJones|Albyn C. Jones]] ('''1999'''). ''Rating the Chess Rating System''. [http://www.glicko.net/research/chance.pdf pdf]
==2000 ...==
* [[Ernst A. Heinz]] ('''2000'''). ''[http://link.springer.com/chapter/10.1007/3-540-45579-5_18 New Self-Play Results in Computer Chess]''. [[CG 2000]]
* [[Kenneth W. Regan]], [[Bartłomiej Macieja]], [[Guy Haworth|Guy McCrossan Haworth]] ('''2011'''). ''[http://centaur.reading.ac.uk/23800/ Understanding Distributions of Chess Performances]''. [[Advances in Computer Games 13]], [http://www.cse.buffalo.edu/~regan/papers/pdf/RMH11.pdf pdf]
* [[Trevor Fenner]], [[Mark Levene]], [[Mathematician#GLoizou|George Loizou]] ('''2011'''). ''A Discrete Evolutionary Model for Chess Players' Ratings''. [https://arxiv.org/abs/1103.1530 arXiv:1103.1530]
* [[Rémi Coulom]] ('''2012'''). ''Paired Comparisons with Ties: Modeling Game Outcomes in Chess''. [http://www.grappa.univ-lille3.fr/~coulom/ChessOutcomes.pdf pdf preprint] <ref>[http://www.talkchess.com/forum/viewtopic.php?topic_view=threads&p=471004&t=44180 Re: EloStat, Bayeselo and Ordo] by [[Rémi Coulom]], [[CCC]], June 25, 2012</ref>
* [[Diogo R. Ferreira]] ('''2012'''). ''Determining the Strength of Chess Players based on actual Play''. [[ICGA Journal#35_1|ICGA Journal, Vol. 35, No. 1]]
* [[Daniel Shawul]], [[Rémi Coulom]] ('''2013'''). ''Paired Comparisons with Ties: Modeling Game Outcomes in Chess''. <ref>[http://www.talkchess.com/forum/viewtopic.php?topic_view=threads&p=471004&t=44180 Re: EloStat, Bayeselo and Ordo] by [[Rémi Coulom]], [[CCC]], June 25, 2012</ref> <ref>[http://www.talkchess.com/forum3/viewtopic.php?f=7&t=72087&start=3 Re: Understanding and Pushing the Limits of the Elo Rating Algorithm] by [[Daniel Shawul]], [[CCC]], October 15, 2019</ref> * [[Diogo R. Ferreira]] ('''2013'''). ''The Impact of the Search Depth on Chess Playing Strength''. [[ICGA Journal#36_2|ICGA Journal, Vol. 36, No. 2]]» [[Depth]], [[Depth#DiminishingReturns|Diminishing Returns]], [[Playing Strength]], [[Houdini]] <ref>[https://www.hiarcs.net/forums/viewtopic.php?t=10004 Ply versus ELO] by Greg, [[Computer Chess Forums|HIARCS Forum]], May 30, 2020 » [[Diogo R. Ferreira#Impact|Diogo R. Ferreira - Impact of the Search Depth ...]]</ref>
* [[Miguel A. Ballicora]] ('''2014'''). ''ORDO v0.9.6 Ratings for chess and other games''. September 2014, [https://docs.google.com/viewer?a=v&pid=sites&srcid=ZGVmYXVsdGRvbWFpbnxnYXZpb3RhY2hlc3NlbmdpbmV8Z3g6NmQ0NmNhNGM4YjA3YTc5ZQ pdf] » [[Ordo]] <ref>[https://sites.google.com/site/gaviotachessengine/ordo Ordo] by [[Miguel A. Ballicora]]</ref>
* [[Don Dailey]], [[Adam Hair]], [[Mark Watkins]] ('''2014'''). ''[http://www.sciencedirect.com/science/article/pii/S1875952113000177 Move Similarity Analysis in Chess Programs]''. [http://www.journals.elsevier.com/entertainment-computing/ Entertainment Computing], Vol. 5, No. 3, [http://magma.maths.usyd.edu.au/~watkins/papers/DHW.pdf preprint as pdf] <ref>[http://www.top-5000talkchess.nlcom/cloneforum/viewtopic.htm A php?t=38772 Pairwise Comparison Analysis of Chess Engine Move Selections] by [[Adam Hair]], hosted by [[Ed Schroder|Ed SchröderCCC]], April 17, 2011</ref>
* [[Kenneth W. Regan]], [[Tamal T. Biswas]], [[Jason Zhou]] ('''2014'''). ''Human and Computer Preferences at Chess''. [http://www.cse.buffalo.edu/~regan/papers/pdf/RBZ14aaai.pdf pdf]
* [[Erik Varend]] ('''2014'''). ''Quality of play in chess and methods for measuring''. [http://www.chessanalysis.ee/Quality%20of%20play%20in%20chess%20and%20methods%20for%20measuring.pdf pdf] <ref>[http://www.talkchess.com/forum/viewtopic.php?t=54571 Questions regarding rating systems of humans and engines] by [[Erik Varend]], [[CCC]], December 06, 2014</ref> <ref>[http://www.talkchess.com/forum/viewtopic.php?t=60721 chess statistics scientific article] by Nuno Sousa, [[CCC]], July 06, 2016</ref>
* [[Mathematician#XiaoouLi|Xiaoou Li]], [[Mathematician#JingchenLiu|Jingchen Liu]], [[Mathematician#ZhiliangYing|Zhiliang Ying]] ('''2014'''). ''[https://www.tandfonline.com/doi/abs/10.1080/07474946.2014.961861 Generalized Sequential Probability Ratio Test for Separate Families of Hypotheses]''. [https://www.tandfonline.com/loi/lsqa20?open=36&year=2017&repitition=0 Sequential Analysis], Vol. 33, No. 4, [http://stat.columbia.edu/~jcliu/paper/GSPRT_SQA3.pdf pdf]
==2015 ...==
* [[Tamal T. Biswas]], [[Kenneth W. Regan]] ('''2015'''). ''Quantifying Depth and Complexity of Thinking and Knowledge''. [http://www.icaart.org/EuropeanProjectSpace.aspx?y=2015 ICAART 2015], [http://www.cse.buffalo.edu/~regan/papers/pdf/BiReICAART15CR.pdf pdf]
* [[Jean-Marc Alliot]] ('''2017'''). ''Who is the Master''? [[ICGA Journal#39_1|ICGA Journal, Vol. 39, No. 1]], [http://www.alliot.fr/CHESS/draft-icga-39-1.pdf draft as pdf] » [[Stockfish]], [[Jean-Marc Alliot#WhoistheMaster|Who is the Master?]]
* [[Mathematician#IAdler|Ilan Adler]], [[Mathematician#YangCao|Yang Cao]], [[Richard Karp]], [[Mathematician#EAPekoz|Erol A. Peköz]], [[Mathematician#SMRoss|Sheldon M. Ross]] ('''2017'''). ''[https://pubsonline.informs.org/doi/10.1287/opre.2017.1657 Random Knockout Tournaments]''. [https://en.wikipedia.org/wiki/Operations_Research_(journal) Operations Research], Vol. 65, No. 6, [https://arxiv.org/abs/1612.04448 arXiv:1612.04448]
* [[Michel Van den Bergh]] ('''2017'''). ''A Practical Introduction to the GSPRT''. [http://hardy.uhasselt.be/Toga/GSPRT_approximation.pdf pdf]
* [[Leszek Szczecinski]], [[Aymen Djebbi]] ('''2019'''). ''Understanding and Pushing the Limits of the Elo Rating Algorithm''. [https://arxiv.org/abs/1910.06081 arXiv:1910.06081] <ref>[http://www.talkchess.com/forum3/viewtopic.php?f=7&t=72087 Understanding and Pushing the Limits of the Elo Rating Algorithm] by [[Michel Van den Bergh]], [[CCC]], October 15, 2019</ref>
* [[Michel Van den Bergh]] ('''2019'''). ''The Generalized Maximum Likelihood Ratio for the Expectation Value of a Distribution''. [http://hardy.uhasselt.be/Fishtest/support_MLE_multinomial.pdf pdf]
=Forum & Blog Postings=
* [http://www.talkchess.com/forum/viewtopic.php?t=13800 a beat b,b beat c,c beat a question] by [[Uri Blass]], [[CCC]], May 16, 2007
* [http://www.talkchess.com/forum/viewtopic.php?t=16545 how to do a proper statistical test] by [[Rein Halbersma]], [[CCC]], September 19, 2007
* [http://www.talkchess.com/forum3/viewtopic.php?f=2&t=24590 Comparing two version of the same engine] by [[Fermin Serrano]], [[CCC]], October 26, 2008
* [http://www.talkchess.com/forum3/viewtopic.php?f=2&t=25461 Debate: testing at fast time controls] by [[Fermin Serrano]], [[CCC]], December 15, 2008
* [http://www.talkchess.com/forum/viewtopic.php?t=27516 Elo Calcuation] by [[Edmund Moshammer]], [[CCC]], April 19, 2009
* [http://www.talkchess.com/forum/viewtopic.php?t=30624 Likelihood of superiority] by [[Marco Costalba]], [[CCC]], November 15, 2009
* [http://www.talkchess.com/forum/viewtopic.php?t=36592 Do You really need 1000s of games for testing?] by [[Jouni Uski]], [[CCC]], November 04, 2010
* [http://www.talkchess.com/forum/viewtopic.php?t=36979 GUI idea: Testing until certainty] by [[Albert Silver]], [[CCC]], December 07, 2010
* [http://www.talkchess.com/forum/viewtopic.php?t=37056 SPRT and Engine testing] by [[Adam Hair]], [[CCC]], December 13, 2010 » [[Match Statistics#SPRT|SPRT]]
'''2011'''
* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=38698 Testing very small changes ( <= 5 ELO points of gain)] by [[Fermin Serrano]], [[CCC]], April 08, 2011
* [http://www.talkchess.com/forum/viewtopic.php?t=38772 Pairwise Analysis of Chess Engine Move Selections] by [[Adam Hair]], [[CCC]], April 17, 2011
* [http://www.talkchess.com/forum/viewtopic.php?t=39511 Ply vs ELO] by Andriy Dzyben, [[CCC]], June 28, 2011
* [http://www.talkchess.com/forum/viewtopic.php?t=40193 One billion random games] by [[Steven Edwards]], [[CCC]], August 27, 2011
* [http://www.talkchess.com/forum/viewtopic.php?t=41341 Increase in Elo ..Question For The Experts] by [[Steve Blincoe|Steve B]], [[CCC]], December 05, 2011
'''2012'''
* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=42032 Are all test the same?] by [[Fermin Serrano]], [[CCC]], January 17, 2012
* [http://www.talkchess.com/forum/viewtopic.php?t=42729 Advantage for White; Bayeselo (to Rémi Coulom)] by [[Edmund Moshammer]], [[CCC]], March 03, 2012
* [http://www.talkchess.com/forum/viewtopic.php?t=42737 Pairwise Analysis of Chess Engine Move Selections Revisited] by [[Adam Hair]], [[CCC]], March 04, 2012
* [http://www.talkchess.com/forum/viewtopic.php?t=44219 Human Elo ratings: averages and standard deviations] by [[Jesús Muñoz]], [[CCC]], March 18, 2012 <ref>[http://en.chessbase.com/post/arpad-elo-and-the-elo-rating-system Arpad Elo and the Elo Rating System] by [[Dan Ross]], [[ChessBase|ChessBase News]], December 16, 2007</ref>
* [http://www.talkchess.com/forum/viewtopic.php?t=42998 Elo uncertainties calculator] by [[Jesús Muñoz]], [[CCC]], March 24, 2012
* [http://www.talkchess.com/forum/viewtopic.php?t=43134 Elo versus speed] by [[Peter Österlund]], [[CCC]], April 02, 2012
* [http://www.talkchess.com/forum/viewtopic.php?t=43323 Pawn Advantage, Win Percentage, and Elo] by [[Adam Hair]], [[CCC]], April 15, 2012 » [[Pawn Advantage, Win Percentage, and Elo]]
* [http://www.talkchess.com/forum/viewtopic.php?t=43598 Elo Increase per Doubling] by [[Adam Hair]], [[CCC]], May 07, 2012
* [http://www.talkchess.com/forum/viewtopic.php?t=44003 Rybka odds matches and the strength of engines] by [[Kai Laskos]], [[CCC]], June 09, 2012 » [[Rybka]]
* [http://www.talkchess.com/forum/viewtopic.php?t=44147 A new way to compare chess programs] by [[Larry Kaufman]], [[CCC]], June 21, 2012 » [[Komodo]]
* [http://www.talkchess.com/forum/viewtopic.php?t=46572 A word for casual testers] by [[Don Dailey]], [[CCC]], December 25, 2012
'''2013'''
* [http://www.talkchess.com/forum/viewtopic.php?t=46759 A poor man's testing environment] by [[Ed Schroder|Ed Schröder]], [[CCC]], January 04, 2013 <ref>[http://www.top-5000.nl/tuning.htm Testing a chess engine from the ground up] from [http://www.top-5000.nl/ Home of the Dutch Rebel] by [[Ed Schroder|Ed Schröder]]</ref> » [[Engine Testing]]
* [http://www.talkchess.com/forum/viewtopic.php?t=46786 Noise in ELO estimators: a quantitative approach] by [[Marco Costalba]], [[CCC]], January 06, 2013
* [http://www.talkchess.com/forum/viewtopic.php?t=47086 Updated Dendrogram] by [[Kai Laskos]], [[CCC]], February 02, 2013
* [http://www.talkchess.com/forum/viewtopic.php?t=47885 Fishtest Distributed Testing Framework] by [[Marco Costalba]], [[CCC]], May 01, 2013
* [http://www.talkchess.com/forum/viewtopic.php?t=48649 The influence of the length of openings] by [[Kai Laskos]], [[CCC]], July 14, 2013
* [http://www.talkchess.com/forum/viewtopic.php?t=48733 Scaling at 2x nodes (or doubling time control)] by [[Kai Laskos]], [[CCC]], July 23, 2013 » [[Match Statistics#DoublingTC|Doubling TC]], [[Depth#DiminishingReturns|Diminishing Returns]], [[Playing Strength]], [[Houdini]]
* [http://www.talkchess.com/forum/viewtopic.php?t=48863 Type I error in LOS based early stopping rule] by [[Kai Laskos]], [[CCC]], August 06, 2013 <ref>[https://en.wikipedia.org/wiki/Type_I_and_type_II_errors Type I and type II errors from Wikipedia]</ref>
* [http://www.talkchess.com/forum/viewtopic.php?t=48864 How much elo is pondering worth] by [[Michel Van den Bergh]], [[CCC]], August 07, 2013 » [[Pondering]]
* [http://www.talkchess.com/forum/viewtopic.php?t=49248 Contempt and the ELO model] by [[Michel Van den Bergh]], [[CCC]], September 05, 2013 » [[Contempt Factor]]
* [http://www.talkchess.com/forum/viewtopic.php?t=49393 1 draw=1 win + 1 loss (always!)] by [[Michel Van den Bergh]], [[CCC]], September 19, 2013
* [http://www.talkchess.com/forum/viewtopic.php?t=49584 SPRT and narrowing of (elo1 - elo0) difference] by [[Jesús Muñoz]], [[CCC]], October 05, 2013 » [[Match Statistics#SPRT|SPRT]]* [http://www.talkchess.com/forum/viewtopic.php?t=49727 sprt and margin of error] by [[Larry Kaufman]], [[CCC]], October 15, 2013 » [[Match Statistics#SPRT|SPRT]]
* [http://www.open-chess.org/viewtopic.php?f=5&t=2477 How (not) to use SPRT ?] by [[Mark Watkins|BB+]], [[Computer Chess Forums|OpenChess Forum]], October 19, 2013
* [http://www.talkchess.com/forum/viewtopic.php?t=50266 Houdini, much weaker engines, and Arpad Elo] by [[Kai Laskos]], [[CCC]], November 29, 2013 » [[Houdini]], [[Pawn Advantage, Win Percentage, and Elo]] <ref>[https://en.wikipedia.org/wiki/Arpad_Elo Arpad Elo - Wikipedia]</ref>
* [https://chesscomputer.tumblr.com/post/98632536555/using-the-stockfish-position-evaluation-score-to/embed Using the Stockfish position evaluation score to predict victory probability] by unavoidablegrain, [https://en.wikipedia.org/wiki/Tumblr Tumblr], September 28, 2014 » [[Pawn Advantage, Win Percentage, and Elo]], [[Stockfish]]
* [http://www.talkchess.com/forum/viewtopic.php?t=53891 Elo estimation using quasi-Monte Carlo integration] by Branko Radovanovic, [[CCC]], September 30, 2014
* [http://www.talkchess.com/forum/viewtopic.php?t=54331 SPRT question] by [[Robert Hyatt]], [[CCC]], November 13, 2014 » [[Match Statistics#SPRT|SPRT]]* [http://www.talkchess.com/forum/viewtopic.php?t=54359 Usage sprt / cutechess-cli] by [[Michael Hoffmann]], [[CCC]], November 16, 2014 » [[Cutechess-cli]], [[Match Statistics#SPRT|SPRT]]
==2015 ...==
* [http://www.talkchess.com/forum/viewtopic.php?t=55130 2-SPRT] by [[Michel Van den Bergh]], [[CCC]], January 28, 2015 » [[Match Statistics#SPRT|SPRT]]
* [http://www.talkchess.com/forum/viewtopic.php?t=55893 Script for computing SPRT probabilities] by [[Michel Van den Bergh]], [[CCC]], April 05, 2015
* [http://www.talkchess.com/forum/viewtopic.php?t=56067 Maximum ELO gain per test game played?] by Forrest Hoch, [[CCC]], April 20, 2015
* [http://www.talkchess.com/forum/viewtopic.php?t=56095 Getting SPRT right] by [[Alexandru Mosoi]], [[CCC]], April 22, 2015 » [[Match Statistics#SPRT|SPRT]]* [http://www.talkchess.com/forum/viewtopic.php?t=56358 SPRT questions] by [[Uri Blass]], [[CCC]], May 15, 2015 » [[Match Statistics#SPRT|SPRT]]* [http://www.talkchess.com/forum/viewtopic.php?t=56426 Adam Hair's article on Pairwise comparison of engines] by [[Charles Roberson]], [[CCC]], May 19, 2015 <ref>[http://www.top-5000.nl/clone.htm A Pairwise Comparison of Chess Engine Move Selections] by [[Adam Hair]], hosted by [[Ed Schroder|Ed Schröder]]</ref>
* [http://www.talkchess.com/forum/viewtopic.php?t=57223 computing elo of multiple chess engines] by [[Alexandru Mosoi]], [[CCC]], August 09, 2015
* [http://www.talkchess.com/forum/viewtopic.php?t=57270 Some musings about search] by [[Ed Schroder|Ed Schröder]], [[CCC]], August 14, 2015 » [[Automated Tuning]], [[Search]]
* [http://www.talkchess.com/forum/viewtopic.php?t=57437 Bullet vs regular time control, say 40/4m CCRL/CEGT] by [[Ed Schroder|Ed Schröder]], [[CCC]], August 29, 2015
* [http://www.talkchess.com/forum/viewtopic.php?t=57465 The SPRT without draw model, elo model or whatever...] by [[Michel Van den Bergh]], [[CCC]], September 01, 2015 » [[Match Statistics#SPRT|SPRT]]
: [http://talkchess.com/forum/viewtopic.php?t=57465&start=19 Re: The SPRT without draw model, elo model or whatever..] by [[Michel Van den Bergh]], [[CCC]], August 18, 2016
* [http://www.talkchess.com/forum/viewtopic.php?t=57482 Name for elo without draws?] by [[Marcel van Kervinck]], [[CCC]], September 02, 2015
* [http://www.talkchess.com/forum/viewtopic.php?t=61636 Differences between top engines related to "style"] by [[Kai Laskos]], October 07, 2016
* [http://www.talkchess.com/forum/viewtopic.php?t=61781 SPRT when not used for self testing] by [[Andrew Grant]], [[CCC]], October 21, 2016
* [http://www.talkchess.com/forum/viewtopic.php?t=61784 Doubling of time control] by [[Andreas Strangmüller]], [[CCC]], October 21, 2016 » [[Match Statistics#DoublingTC|Doubling TC]], [[Depth#DiminishingReturns|Diminishing Returns]], [[Playing Strength]], [[Komodo]]* [http://www.talkchess.com/forum/viewtopic.php?t=62146 Stockfish 8 - Double time control vs. 2 threads] by [[Andreas Strangmüller]], [[CCC]], November 15, 2016 » [[Match Statistics#DoublingTC|Doubling TC]], [[Depth#DiminishingReturns|Diminishing Returns]], [[Playing Strength]], [[Stockfish]]
* [https://rjlipton.wordpress.com/2016/11/30/when-data-serves-turkey/ When Data Serves Turkey] by [[Kenneth W. Regan|Ken Regan]], [https://rjlipton.wordpress.com/ Gödel's Lost Letter and P=NP], November 30, 2016
* [https://rjlipton.wordpress.com/2016/12/08/magnus-and-the-turkey-grinder/ Magnus and the Turkey Grinder] by [[Kenneth W. Regan|Ken Regan]], [https://rjlipton.wordpress.com/ Gödel's Lost Letter and P=NP], December 08, 2016 » [[Pawn Advantage, Win Percentage, and Elo]] <ref>[https://en.wikipedia.org/wiki/World_Chess_Championship_2016 World Chess Championship 2016 from Wikipedia]</ref>
* [http://www.talkchess.com/forum/viewtopic.php?t=62438 Statistical Interpretation] by [[Dennis Sceviour]], [[CCC]], December 10, 2016
* [http://www.talkchess.com/forum/viewtopic.php?t=62510 Absolute ELO scale] by [[Nicu Ionita]], [[CCC]], December 17, 2016
* [http://www.talkchess.com/forum/viewtopic.php?t=62598 A question about SPRT] by [[Andrew Grant]], [[CCC]], December 25, 2016 » [[Match Statistics#SPRT|SPRT]]
* [http://www.talkchess.com/forum/viewtopic.php?t=62622 Diminishing returns and hyperthreading] by [[Kai Laskos]], [[CCC]], December 27, 2016 » [[Depth#DiminishingReturns|Diminishing Returns]], [[Playing Strength]], [[Thread]]
'''2017'''
* [http://www.talkchess.com/forum/viewtopic.php?t=62868 Progress in 30 years by four intervals of 7-8 years] by [[Kai Laskos]], [[CCC]], January 19, 2017 » [[Playing Strength]]
* [http://www.talkchess.com/forum/viewtopic.php?t=62922 sprt tourney manager] by [[Richard Delorme]], [[CCC]], January 24, 2017 » [[Amoeba#TournamentManager|Amoeba Tournament Manager]], [[Match Statistics#SPRT|SPRT]]
* [http://www.talkchess.com/forum/viewtopic.php?t=63327 Binomial distribution for chess statistics] by [[Lyudmil Antonov]], [[CCC]], March 03, 2017
* [http://www.talkchess.com/forum/viewtopic.php?t=63355 Higher than expected by me efficiency of Ponder ON] by [[Kai Laskos]], [[CCC]], March 06, 2017 » [[Pondering]]
* [http://www.talkchess.com/forum/viewtopic.php?t=63875 Wilo rating properties from FGRL rating lists] by [[Kai Laskos]], [[CCC]], May 01, 2017 » [[FGRL]]
* [http://www.talkchess.com/forum/viewtopic.php?t=63888 MATCH sanity] by [[Ed Schroder]], [[CCC]], May 03, 2017 » [[Portable Game Notation]]
: [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=63888&start=2 Re: MATCH sanity] by [[Salvatore Giannotti]], [[CCC]], May 03, 2017
* [http://www.talkchess.com/forum/viewtopic.php?t=63903 Symmetric multiprocessing (SMP) scaling - SF8 and K10.4] by [[Andreas Strangmüller]], [[CCC]], May 05, 2017 » [[Lazy SMP]], [[Komodo]], [[Stockfish]]
* [http://www.talkchess.com/forum/viewtopic.php?t=63955 Symmetric multiprocessing (SMP) scaling - K10.4 Contempt=0] by [[Andreas Strangmüller]], [[CCC]], May 11, 2017 » [[SMP]], [[Komodo]], [[Contempt Factor]]
* [http://www.talkchess.com/forum/viewtopic.php?t=64683 Invariance with time control of rating schemes] by [[Kai Laskos]], [[CCC]], July 22, 2017 <ref>[http://hardy.uhasselt.be/Toga/normalized_elo.pdf Normalized Elo] (pdf) by [[Michel Van den Bergh]]</ref>
* [http://www.talkchess.com/forum/viewtopic.php?t=64719 Ways to avoid "Draw Death" in Computer Chess] by [[Kai Laskos]], [[CCC]], July 25, 2017
* [http://www.talkchess.com/forum/viewtopic.php?t=64824&start=2 SMP NPS measurements] by [[Peter Österlund]], [[CCC]], August 06, 2017 » [[Lazy SMP]], [[Parallel Search]], [[Nodes per secondSecond]]
: [http://www.talkchess.com/forum/viewtopic.php?t=64824&start=3 ELO measurements] by [[Peter Österlund]], [[CCC]], August 06, 2017 » [[Playing Strength]]
* [http://www.talkchess.com/forum/viewtopic.php?t=65061 What is a Match?] by [[Henk van den Belt]], [[CCC]], September 01, 2017
* [http://www.talkchess.com/forum/viewtopic.php?t=66000 ELO progression measured by year] by [[Ed Schroder]], [[CCC]], December 13, 2017
'''2018'''
* [https://groups.google.com/d/msg/fishcooking/nqgLNUfjkok/gfMr7amXCAAJ Wrong use of SPRT] by [[Uri Blass]], [[Computer Chess Forums|FishCooking]], February 09, 2018 » [[Contempt Factor]], [[Match Statistics#SPRT|SPRT]]
* [http://www.talkchess.com/forum/viewtopic.php?t=66775 Feed bayeselo with pure game results without PGN] by [[Sergei Markoff|Sergei S. Markoff]], [[CCC]], March 08, 2018
* [http://www.talkchess.com/forum/viewtopic.php?t=66793 Elo measurement of contempt in SF in self-play] by [[Michel Van den Bergh]], [[CCC]], March 10, 2018 » [[Contempt Factor|Contempt]], [[Playing Strength]], [[Stockfish]]
* [http://www.talkchess.com/forum3/viewtopic.php?f=2&t=69672 Schizophrenic rating model for Leela] by [[Kai Laskos]], [[CCC]], January 21, 2019 » [[Leela Chess Zero]]
* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=71419 best way to determine elos of a group] by [[Daniel Shawul]], [[CCC]], July 30, 2019
==2020 ...==
* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=72962 Stockfish Reverts 5 Recent Patches] by Deberger, [[CCC]], February 01, 2020 » [[Stockfish]]
: [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=72962&start=6 Re: Stockfish Reverts 5 Recent Patches] by [[Michel Van den Bergh]], [[CCC]], February 02, 2020
* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=73419 repackaging of Fishtest's SPRT calculation code] by [[Jon Dart]], [[CCC]], March 20, 2020 » [[#SPRT|SPRT]]
* [http://www.talkchess.com/forum3/viewtopic.php?f=2&t=74037 Stockfish_dev is probably stronger than Sargon 1978 v1.00] by [[Kai Laskos]], [[CCC]], May 29, 2020 » [[Stockfish]], [[Sargon]]
* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=74319 Throwing out draws to calculate Elo] by [[Dann Corbit]], [[CCC]], June 29, 2020
* [http://www.talkchess.com/forum3/viewtopic.php?f=2&t=75006 Jackknife estimate of the variance of engine results for Arasan test suite] by [[Kai Laskos]], [[CCC]], September 05, 2020 » [[#Statistical Analysis|Statistical Analysis]]
'''2021'''
* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=76261 What K factor should be used if two players are in different K factor brackets?] by [[Tamás Kuzmics]], [[CCC]], January 09, 2021 <ref>[https://en.wikipedia.org/wiki/Elo_rating_system#Most_accurate_K-factor Most accurate K-factor - Elo rating system from Wikipedia]</ref> <ref>[https://ratings.fide.com/calculator_rtd.phtml FIDE Chess Rating calculators: Chess Rating change calculator]</ref>
* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=76370 Cutechess-cli and SPRT] by [[Tom King]], [[CCC]], January 19, 2021 » [[Cutechess-cli]], [[#SPRT|SPRT]]
* [http://www.talkchess.com/forum3/viewtopic.php?f=2&t=76382 correspondence chess in the age of NNUE] by [[Larry Kaufman]], [[CCC]], January 21, 2021 » [[NNUE]]
* [http://www.talkchess.com/forum3/viewtopic.php?f=2&t=77544 what is the rating of engines when you do not count draws?] by [[Uri Blass]], [[CCC]], June 23, 2021
'''2022'''
* [https://www.talkchess.com/forum3/viewtopic.php?f=7&t=79280 How do you know you improved ?] by Philippe Chevalier, [[CCC]], February 03, 2022
=External Links=
* [http://www.top-5000.nl/tuning.htm Testing a chess engine from the ground up] from [http://www.top-5000.nl/ Home of the Dutch Rebel] by [[Ed Schroder|Ed Schröder]] <ref>[http://www.talkchess.com/forum/viewtopic.php?t=46759 A poor man's testing environment] by [[Ed Schroder|Ed Schröder]], [[CCC]], January 04, 2013</ref> » [[Engine Testing]]
* [http://www.top-5000.nl/match.htm MATCH - eng-eng utility] by [[Ed Schroder|Ed Schröder]]
* [http://walkofmind.com/programming/chess/mat_stats.html Statistics of material imbalances in chess games] by [[Alessandro Scotti]] » [[Material]]
==Rating Systems==
* [https://en.wikipedia.org/wiki/Chessmetrics Chessmetrics from Wikipedia]
* [https://en.wikipedia.org/wiki/Elo_rating_system Elo rating system from Wikipedia]
* [http://en.chessbase.com/post/arpad-elo-and-the-elo-rating-system Arpad Elo and the Elo Rating System] by [[Dan Ross]], [[ChessBase|ChessBase News]], December 16, 2007
* [https://www.kaggle.com/c/chess Chess ratings - Elo versus the Rest of the World] | [https://en.wikipedia.org/wiki/Kaggle Kaggle]
* [http://wismuth.com/elo/calculator.html Elo Win Probability Calculator] by [[Mathematician#FLabelle|François Labelle]]
* [http://www.husvankempen.de/nunn/rating/tablejoseph.htm LOS Table] by [[Joseph Ciarrochi]] from [[CEGT]] <ref>[https://www.stmintz.com/ccc/index.php?id=484357 table for detecting significant difference between two engines] by [[Joseph Ciarrochi]], [[CCC]], February 03, 2006</ref>
* [https://en.wikipedia.org/wiki/Margin_of_error Margin of error from Wikipedia]
* [https://en.wikipedia.org/wiki/Confidence_interval Confidence interval from Wikipedia]
* [https://en.wikipedia.org/wiki/Statistical_hypothesis_testing Statistical hypothesis testing from Wikipedia]
* [https://en.wikipedia.org/wiki/Sequential_probability_ratio_test Sequential probability ratio test from Wikipedia]
* [https://en.wikipedia.org/wiki/Null_hypothesis Null hypothesis from Wikipedia]
* [https://en.wikipedia.org/wiki/Alternative_hypothesis Alternative hypothesis from Wikipedia]
* [https://en.wikipedia.org/wiki/Type_I_and_type_II_errors Type I and type II errors from Wikipedia]
* [https://www.khanacademy.org/math/probability/statistics-inferential/hypothesis-testing/v/type-1-errors Type 1 Errors | Hypothesis testing with one sample] | [https://en.wikipedia.org/wiki/Khan_Academy Khan Academy]
==Hypothesis Testing==
* [https://en.wikipedia.org/wiki/Statistical_hypothesis_testing Statistical hypothesis testing from Wikipedia]
* [https://en.wikipedia.org/wiki/Likelihood-ratio_test Likelihood-ratio test from Wikipedia]
* [https://en.wikipedia.org/wiki/Score_test Score test from Wikipedia]
* [https://en.wikipedia.org/wiki/Wald_test Wald test from Wikipedia]
==Sequential Analysis==
* [https://en.wikipedia.org/wiki/Sequential_analysis Sequential analysis from Wikipedia]
* [https://en.wikipedia.org/wiki/Sequential_probability_ratio_test Sequential probability ratio test from Wikipedia]
* [https://github.com/vdbergh/pentanomial GitHub - vdbergh/pentanomial: SPRT for pentanomial frequencies and simulation tools] by [[Michel Van den Bergh]]
* [https://github.com/vdbergh/simul GitHub - vdbergh/simul: A multi-threaded pentanomial simulator] by [[Michel Van den Bergh]]
==Data Visualization==
* [http://www.top-5000.nl/clone.htm A Pairwise Comparison of Chess Engine Move Selections] by [[Adam Hair]], hosted by [[Ed Schroder|Ed Schröder]] <ref>[https://en.wikipedia.org/wiki/UPGMA UPGMA from Wikipedia]</ref> <ref>[http://www.southampton.ac.uk/~re1u06/teaching/upgma/ UPGMA Worked Example] by [http://www.southampton.ac.uk/biosci/about/staff/re1u06.page? Richard Edwards]</ref> <ref>[http://www.talkchess.com/forum/viewtopic.php?t=56426 Adam Hair's article on Pairwise comparison of engines] by [[Charles Roberson]], [[CCC]], May 19, 2015</ref> <ref>[[Don Dailey]], [[Adam Hair]], [[Mark Watkins]] ('''2014'''). ''[http://www.sciencedirect.com/science/article/pii/S1875952113000177 Move Similarity Analysis in Chess Programs]''. [http://www.journals.elsevier.com/entertainment-computing/ Entertainment Computing], Vol. 5, No. 3, [http://magma.maths.usyd.edu.au/~watkins/papers/DHW.pdf preprint as pdf]</ref>
* [https://blog.ebemunk.com/a-visual-look-at-2-million-chess-games/ A Visual Look at 2 Million Chess Games - Thinking Through the Party] by [[Buğra Fırat]], March 02, 2016 <ref>[http://www.talkchess.com/forum/viewtopic.php?t=65610 A Visual Look at 2 Million Chess Games] by Brahim Hamadicharef, [[CCC]], November 02, 2017</ref>
* [https://github.com/ebemunk/chess-dataviz GitHub - ebemunk/chess-dataviz: chess visualization library written for d3.js] by [[Buğra Fırat]] » [[JavaScript]]
: [https://en.wikipedia.org/wiki/ARMS_Charity_Concerts ARMS Charity Concert], [https://en.wikipedia.org/wiki/Madison_Square_Garden Madison Square Garden], [http://forums.ledzeppelin.com/index.php?/topic/394-arms-benefit-concerts-in-nyc-dec-1983/ December 08, 1983]
: {{#evu:https://www.youtube.com/watch?v=IdVJw-b3HHE|alignment=left|valignment=top}}
* [[:Category:Björk|Björk]] - [https://en.wikipedia.org/wiki/Possibly_Maybe Possibly Maybe] (1996), [https://en.wikipedia.org/wiki/YouTube YouTube] Video
: {{#evu:https://www.youtube.com/watch?v=iyqKy5P1Y0Q|alignment=left|valignment=top}}
=References=
[[Category:Jan Hammer]]
[[Category:Simon Phillips]]
[[Category:Björk]]

Navigation menu