23,480

edits
# Changes

no edit summary

<span id="Regression"></span>

=Regression=

[[FILE:Linear regression.svg|border|right|thumb|300px|[https://en.wikipedia.org/wiki/Linear_regression Linear Regression] <ref>Random data points and their [https://en.wikipedia.org/wiki/Linear_regression linear regression]. [https://commons.wikimedia.org/wiki/File:Linear_regression.svg Created] with [https://en.wikipedia.org/wiki/Sage_%28mathematics_software%29 Sage] by Sewaqu, November 5, 2010, [https://en.wikipedia.org/wiki/Wikimedia_Commons Wikimedia Commons]</ref> ]]

[https://en.wikipedia.org/wiki/Regression_analysis Regression analysis] is a [https://en.wikipedia.org/wiki/Statistics statistical process] with a substantial overlap with machine learning to [https://en.wikipedia.org/wiki/Prediction predict] the value of an [https://en.wikipedia.org/wiki/Dependent_and_independent_variables Y variable] (output), given known value pairs of the X and Y variables. While [https://en.wikipedia.org/wiki/Linear_regression linear regression] deals with continuous outputs, [https://en.wikipedia.org/wiki/Logistic_regression logistic regression] covers binary or discrete output, such as win/loss, or win/draw/loss. Parameter estimation in regression analysis can be formulated as the [https://en.wikipedia.org/wiki/Mathematical_optimization minimization] of a [https://en.wikipedia.org/wiki/Loss_function cost or loss function] over a [https://en.wikipedia.org/wiki/Training_set training set] <ref>[https://en.wikipedia.org/wiki/Loss_function#Use_in_statistics Loss function - Use in statistics - Wkipedia]</ref>, such as [https://en.wikipedia.org/wiki/Mean_squared_error mean squared error] or [https://en.wikipedia.org/wiki/Cross_entropy#Cross-entropy_error_function_and_logistic_regression cross-entropy error function] for [https://en.wikipedia.org/wiki/Binary_classification binary classification] <ref>"Using [https://en.wikipedia.org/wiki/Cross_entropy#Cross-entropy_error_function_and_logistic_regression cross-entropy error function] instead of [https://en.wikipedia.org/wiki/Mean_squared_error sum of squares] leads to faster training and improved generalization", from [https://en.wikipedia.org/wiki/Sargur_Srihari Sargur Srihari], [http://www.cedar.buffalo.edu/~srihari/CSE574/Chap5/Chap5.2-Training.pdf Neural Network Training] (pdf)</ref>. The minimization is implemented by [[Iteration|iterative]] optimization [[Algorithms|algorithms]] or [https://en.wikipedia.org/wiki/Metaheuristic metaheuristics] such as [https://en.wikipedia.org/wiki/Iterated_local_search Iterated local search], [https://en.wikipedia.org/wiki/Gauss%E2%80%93Newton_algorithm Gauss–Newton algorithm], or [https://en.wikipedia.org/wiki/Conjugate_gradient_method conjugate gradient method].

<span id="LinearRegression"></span>

==Linear Regression==

<span id="LogisticRegression"></span>

==Logistic Regression==

==Instances==

==1980 ...==

* [[Thomas Nitsche]] ('''1982'''). ''A Learning Chess Program.'' [[Advances in Computer Chess 3]]

* [[Donald H. Mitchell]] ('''1984'''). ''Using Features to Evaluate Positions in Experts' and Novices' Othello Games''. ~~Masters ~~Master thesis, Department of Psychology, [[Northwestern University]], Evanston, IL

==1985 ...==

* [[Tony Marsland]] ('''1985'''). ''Evaluation-Function Factors''. [[ICGA Journal#8_2|ICCA Journal, Vol. 8, No. 2]], [http://webdocs.cs.ualberta.ca/~tony/OldPapers/evaluation.pdf pdf]

* [[Peter Mysliwietz]] ('''1994'''). ''Konstruktion und Optimierung von Bewertungsfunktionen beim Schach.'' Ph.D. thesis (German)

==1995 ...==

* [[Michael Buro]] ('''1995'''). ''[~~http~~https://www.jair.org/~~papers~~index.php/jair/article/~~paper179.html ~~view/10146 Statistical Feature Combination for the Evaluation of Game Positions]''. [https://en.wikipedia.org/wiki/Journal_of_Artificial_Intelligence_Research JAIR], Vol. 3

* [[Chris McConnell]] ('''1995'''). ''Tuning Evaluation Functions for Search''. [http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=9B2A0CCA8B1AFB594A879799D974111A?doi=10.1.1.53.9742&rep=rep1&type=pdf pdf]

* [[Chris McConnell]] ('''1995'''). ''Tuning Evaluation Functions for Search'' (Talk), [http://www.cs.cmu.edu/afs/cs.cmu.edu/user/ccm/www/talks/tune.ps ps]

* [[Borko Bošković]], [[Sašo Greiner]], [[Janez Brest]], [[Aleš Zamuda]], [[Viljem Žumer]] ('''2008'''). ''[https://link.springer.com/chapter/10.1007%2F978-3-540-68830-3_12 An Adaptive Differential Evolution Algorithm with Opposition-Based Mechanisms, Applied to the Tuning of a Chess Program]''. [https://link.springer.com/book/10.1007/978-3-540-68830-3 Advances in Differential Evolution], [https://en.wikipedia.org/wiki/Springer_Science%2BBusiness_Media Springer]

'''2009'''

* [[Joel Veness]], [[David Silver]], [[William Uther]], [[Alan Blair]] ('''2009'''). ''[http://papers.nips.cc/paper/3722-bootstrapping-from-game-tree-search Bootstrapping from Game Tree Search]''. [http://nips.cc/ Neural Information Processing Systems (NIPS), 2009], [http://~~books~~jveness.~~nips~~info/publications/nips2009%20-%20bootstrapping%20from%20game%20tree%20search.~~cc~~pdf pdf] » [[Meep]] <ref>[http:/~~papers~~/~~files~~www.talkchess.com/~~nips22~~forum/~~NIPS2009_0508~~viewtopic.~~pdf pdf~~php?start=0&t=31667 A paper about parameter tuning] by [[Rémi Coulom]], [[CCC]], January 12, 2010</ref>

* [[Eli David|Omid David]], [[Jaap van den Herik]], [[Moshe Koppel]], [[Nathan S. Netanyahu]] ('''2009'''). ''Simulating Human Grandmasters: Evolution and Coevolution of Evaluation Functions''. [http://www.sigevo.org/gecco-2009/ GECCO '09], [https://arxiv.org/abs/1711.06840 arXiv:1711.0684]

* [[Eli David|Omid David]] ('''2009'''). ''Genetic Algorithms Based Learning for Evolving Intelligent Organisms''. Ph.D. Thesis.

* [[Hung-Jui Chang]], [[Jr-Chang Chen]], [[Gang-Yu Fan]], [[Chih-Wen Hsueh]], [[Tsan-sheng Hsu]] ('''2018'''). ''Using Chinese dark chess endgame databases to validate and fine-tune game evaluation functions''. [[ICGA Journal#40_2|ICGA Journal, Vol. 40, No. 2]] » [[Chinese Dark Chess]], [[Endgame Tablebases]]

* [[Wen-Jie Tseng]], [[Jr-Chang Chen]], [[I-Chen Wu]], [[Tinghan Wei]] ('''2018'''). ''Comparison Training for Computer Chinese Chess''. [https://arxiv.org/abs/1801.07411 arXiv:1801.07411] <ref>[http://www.talkchess.com/forum3/viewtopic.php?f=7&t=52861&start=7 Re: multi-dimensional piece/square tables] by Tony P., [[CCC]], January 28, 2020 » [[Piece-Square Tables]]</ref>

* [[Jeremy Rapin]], [[Olivier Teytaud]] ('''2018'''). ''Nevergrad - A gradient-free optimization platform''. [https://github.com/facebookresearch/nevergrad GitHub - facebookresearch/nevergrad: A Python toolbox for performing gradient-free optimization]

==2020 ...==

* [[Andrew Grant]] ('''2020'''). ''Evaluation & Tuning in Chess Engines''. [https://github.com/AndyGrant/Ethereal/blob/master/Tuning.pdf pdf] <ref>[http://www.talkchess.com/forum3/viewtopic.php?f=7&t=74877 Evaluation & Tuning in Chess Engines] by [[Andrew Grant]], [[CCC]], August 24, 2020</ref>

=Forum Posts=

* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=68753 methods for tuning coefficients] by [[Stuart Cracraft]], [[CCC]], October 28, 2018

* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=69035 Particle Swarm Optimization Code] by [[Erik Madsen]], [[CCC]], November 24, 2018 » [[MadChess]]

* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=69207 Gradient Descent Introduction] by [[Michael Hoffmann|Desperado]], [[CCC]], December 09, 2018

'''2019'''

* [http://www.talkchess.com/forum3/viewtopic.php?f=2&t=69532 Automated tuning... finally... (Topple v0.3.0)] by [[Vincent Tang]], [[CCC]], January 08, 2019 » [[Topple]]

* [http://www.talkchess.com/forum3/viewtopic.php?f=2&t=75104 Yet another parameter tuner using optuna framework] by [[Ferdinand Mosca]], [[CCC]], September 14, 2020

: [http://www.talkchess.com/forum3/viewtopic.php?f=2&t=75104&start=15 Re: Yet another parameter tuner using optuna framework] by [[Karlson Pfannschmidt]], [[CCC]], September 16, 2020 <ref>[https://optunity.readthedocs.io/en/latest/user/solvers/TPE.html Tree-structured Parzen Estimator — Optunity 1.1.0 documentation]</ref>

* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=75234 evaluation tuning - where to start?] by [[Maksim Korzh]], [[CCC]], September 27, 2020

* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=75267 How to calculate piece weights with logistic regression?] by [[Maksim Korzh]], [[CCC]], October 01, 2020 » [[Automated Tuning#Regression|Regression]], [[Point Value by Regression Analysis]], [[Point Value]]

* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=75411 Unsupervised reinforcement tuning from zero] by Madeleine Birchfield, [[CCC]], October 16, 2020 » [[Reinforcement Learning]]

* [http://www.talkchess.com/forum3/viewtopic.php?f=2&t=76024 Laskas parameter optimizer] by [[Ferdinand Mosca]], [[CCC]], December 09, 2020

'''2021'''

* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=76227 How to calc the derivative for gradient descent?] by Brian Neal, [[CCC]], January 04, 2021

* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=76238 Help with Texel's tuning] by [[Maksim Korzh]], [[CCC]], January 05, 2021 » [[Texel's Tuning Method]]

* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=76265 Tapered Evaluation and MSE (Texel Tuning)] by [[Michael Hoffmann]], [[CCC]], January 10, 2021 » [[Texel's Tuning Method]], [[Tapered Eval]]

* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=76288 Training data] by [[Michael Hoffmann]], [[CCC]], January 12, 2021

* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=76292 Why using the game result instead of evaluation scores] by [[Michael Hoffmann]], [[CCC]], January 12, 2021

* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=76294 Using Mini-Batch for tunig] by [[Michael Hoffmann]], [[CCC]], January 12, 2021

* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=76380 Texel tuning variant] by [[Ferdinand Mosca]], [[CCC]], January 21, 2021

* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=76385 Parameter Tuning Algorithm] by [[Michael Hoffmann]], [[CCC]], January 21, 2021

=External Links=

* [https://en.wikipedia.org/wiki/Tikhonov_regularization Tikhonov regularization (Ridge regression) from Wikipedia]

==Code==

* ~~<span id="Rockstar"></span>~~[https://github.com/~~lantonov~~facebookresearch/nevergrad GitHub - facebookresearch/~~Rockstar Rockstar~~nevergrad: ~~Implementation of ROCK~~A Python toolbox for performing gradient-free optimization]* ~~algorithm (Gaussian kernel regression + natural gradient descent) for optimisation | ~~[https://github.com/fsmosca/Optuna-Game-Parameter-Tuner GitHub- fsmosca/Optuna-Game-Parameter-Tuner: A game search and evaluation parameter tuner using optuna framework] by [[~~Lyudmil Antonov]] and [[Joona Kiiski~~Ferdinand Mosca]] » [[~~Automated Tuning#ROCK|ROCK*~~Deuterium]] <ref>[http://www.talkchess.com/~~forum~~forum3/viewtopic.php?f=2&t=~~65045 ROCK* black-box optimizer for chess~~75104 Yet another parameter tuner using optuna framework] by [[~~Jon Dart~~Ferdinand Mosca]], [[CCC]], ~~August 31~~September 14, ~~2017~~2020</ref>* [https://github.com/~~zamar~~fsmosca/~~spsa SPSA Tuner for Stockfish Chess Engine | ~~Lakas GitHub- fsmosca/Lakas: Game parameter optimizer using nevergrad framework] by [[~~Joona Kiiski~~Ferdinand Mosca]] ~~» ~~<ref>[http://www.talkchess.com/forum3/viewtopic.php?f=2&t=76024 Laskas parameter optimizer] by [[~~Stockfish~~Ferdinand Mosca]], [[~~Stockfish's Tuning Method~~CCC]]~~* [https:~~, December 09, 2020</~~/github.com/scikit-optimize/scikit-optimize GitHub - scikit-optimize/scikit-optimize: Sequential model-based optimization with a `scipy.optimize` interface]~~ref>

* [https://github.com/kiudee/bayes-skopt GitHub - kiudee/bayes-skopt: A fully Bayesian implementation of sequential model-based optimization] by [[Karlson Pfannschmidt]] » [[Fat Fritz]] <ref>[https://en.chessbase.com/post/fat-fritz-update-and-fat-fritz-jr Fat Fritz 1.1 update and a small gift] by [[Albert Silver]]. [[ChessBase|ChessBase News]], March 05, 2020</ref>

* [https://github.com/kiudee/chess-tuning-tools GitHub - kiudee/chess-tuning-tools] by [[Karlson Pfannschmidt]] » [[Leela Chess Zero]]

* [https://github.com/krasserm/bayesian-machine-learning GitHub - krasserm/bayesian-machine-learning: Notebooks about Bayesian methods for machine learning] by [https://krasserm.github.io/ Martin Krasser] <ref>[http://www.talkchess.com/forum3/viewtopic.php?f=7&t=74209 Great input about Bayesian optimization of noisy function methods] by [[Vivien Clauzon]], [[CCC]], June 16, 2020</ref>

* [https://github.com/thomasahle/noisy-bayesian-optimization GitHub - thomasahle/noisy-bayesian-optimization: Bayesian Optimization for very Noisy functions] by [[Thomas Dybdahl Ahle]] » [[FastChess]]* [https://github.com/~~fsmosca~~scikit-optimize/~~Optuna~~scikit-~~Game~~optimize GitHub -~~Parameter~~scikit-~~Tuner GitHub - fsmosca~~optimize/~~Optuna~~scikit-~~Game~~optimize: Sequential model-~~Parameter-Tuner~~based optimization with a `scipy.optimize` interface]* <span id="Rockstar"></span>[https://github.com/lantonov/Rockstar Rockstar: ~~A game search and evaluation parameter tuner using optuna framework~~Implementation of ROCK* algorithm (Gaussian kernel regression + natural gradient descent) for optimisation | GitHub] by [[~~Ferdinand Mosca~~Lyudmil Antonov]] and [[Joona Kiiski]] » [[~~Deuterium~~Automated Tuning#ROCK|ROCK*]] <ref>[http://www.talkchess.com/~~forum3~~forum/viewtopic.php?~~f=2&~~t=~~75104 Yet another parameter tuner using optuna framework~~65045 ROCK* black-box optimizer for chess] by [[~~Ferdinand Mosca~~Jon Dart]], [[CCC]], ~~September 14~~August 31, ~~2020~~2017</ref>* [https://github.com/zamar/spsa SPSA Tuner for Stockfish Chess Engine | GitHub] by [[Joona Kiiski]] » [[Stockfish]], [[Stockfish's Tuning Method]]

==Misc==

* [[:Category:The Next Step Quintet|The Next Step Quintet]] feat. [http://www.tivonpennicott.com/ Tivon Pennicott] - [http://www.discogs.com/Next-Step-Quintet-The-Next-Step-Quintet/release/4970720 Regression], [https://el-gr.facebook.com/KerameioBar KerameioBar] [https://en.wikipedia.org/wiki/Athens Athens], [https://en.wikipedia.org/wiki/Greece Greece], September 2014, [https://en.wikipedia.org/wiki/YouTube YouTube] Video