25,161
edits
Changes
Created page with "'''Home * Learning * Book Learning''' FILE:France in XXI Century. School.jpg|border|right|thumb| Book Learning <ref>France in 2000 year (XXI century). Fu..."
'''[[Main Page|Home]] * [[Learning]] * Book Learning'''
[[FILE:France in XXI Century. School.jpg|border|right|thumb| Book Learning <ref>France in 2000 year (XXI century). Future school. France, paper card, A reproduction of the early 20th century, scan, [https://en.wikipedia.org/wiki/Learning Learning from Wikipedia]</ref> ]]
'''Book Learning''',<br/>
is a technique of unsupervised [[Reinforcement Learning|reinforcement learning]] motivated by basic requirements for successfully playing a sequence of games, such as avoiding losing games in the same way from certain lines from the [[Opening Book|opening book]], or to recognize its own strong points and guide the engine into positions it does well <ref>[http://www.top-5000.nl/authors/rebel/hints.htm Learning Methods] by [[Ed Schroder|Ed Schröder]]</ref>.
Unsupervised reinforcement learning is characterized by [[Trial and Error|trial and error]] and feeding back reinforcement signals, in this context to modify move probabilities or to add new book moves. The signals are either based on the final outcome of the game, called result-driven learning, and/or by [[Search|search-driven]] learning, i.e. from the [[Score|score]] and trend of first few searches after leaving the book to determine how the program likes the position <ref>[[Robert Hyatt]] ('''1999'''). ''[http://www.craftychess.com/hyatt/learning.html Book Learning - a Methodology to Tune an Opening Book Automatically]''. [[ICGA Journal#22_1|ICCA Journal, Vol. 22, No. 1]]</ref>.
=Chess Engines=
==Crafty==
[[Robert Hyatt]] on Book learning in [[Crafty]] <ref>[https://www.stmintz.com/ccc/index.php?id=68381 Re: Book learning] by [[Robert Hyatt]], [[CCC]], September 12, 1999</ref>:
* <code>Crafty does two kinds of 'book learning'. Result learning sound just like what you are doing. If it loses a game, it won't play the last book move it played ever again. And this gets backed up several moves back as well to get it out of bad lines quickly.</code>
* <code>Normal book learning uses the first ten non-book moves (search results) to determine the 'trend' of the game. This trend score is then used to adjust the book move that was played in this game...</code>
Both are on by default... and further <ref>[https://www.stmintz.com/ccc/index.php?id=340060 Re: questions about book learning] by [[Robert Hyatt]], [[CCC]], January 03, 2004</ref>:
# <code>Crafty remembers the evaluations for the first 10 moves out of book, after each search has been completed. It uses these evaluations to detect a "trend". IE is the evaluation good and getting better? Is it bad and getting worse? Is it good but dropping (ie it grabbed a gambit pawn and is beginning to see that it was bad) or is it bad but getting better (IE it offered a gambit, the opponent took it, and the score is going up). It factors all of that together and marks the book line as good or bad.</code>
# <code>Crafty takes the result of a game when it loses, and updates the book line so that moves tried near the end of the line simply don't get played, and alternatives near the front of the book line get tried next.</code>
# <code>There is more to it than that, and you can look at the crafty.doc file <ref>[http://www.craftychess.com/hyatt/craftydoc.html Crafty Documentation]</ref> to at least see what I am doing in more detail. It is '''very''' effective. </code>
==MChess==
[[Marty Hirsch]] in [[Advances in Computer Games 9]] on machine learning in [[MChess|MChess Pro]], excerpt <ref>[[Marty Hirsch]] ('''2001'''). ''Machine Learning in MChess Professional'' - [[Advances in Computer Games 9]], pp. 133-142, 135</ref>
Some sets of heuristics are employed to decide which moves to add to the opening book, and which to delete from it. Moves are deleted when the score just out of book is not too high, and the score later in the game is worse. Moves are added when the score just out of book is not too low, and the score later in the game is satisfactory. A further pre-condition for moves that may be added to the opening book is that the increase in score has been mainly monotonic. If this condition is not met, a future game may enable better moves to be found due to the general learning. In other words, only the best moves the program is able to discover (at a given level) are added to the opening book. Different thresholds are applied for the storage and deletion of moves for White and Black.
...
Some programs now support a different type of book learning in which no moves are added to the book, but instead, unsuccessful moves are given a lower probability of selection. This approach narrows the book over the time and may optimize the results against a specific opponent, at the expense of the results against other opponents. The book learning described in this paper maintains the opening book variety and improves the book against multiple opponents.
==RomiChess==
As explained by [[Michael Sherwin]], [[RomiChess]] applies [https://en.wikipedia.org/wiki/Monkey_see,_monkey_do Monkey see Monkey do] [https://en.wikipedia.org/wiki/Mimicry mimicry] to learn book lines <ref>[http://www.talkchess.com/forum/viewtopic.php?t=37062&start=15 Re: Positional learning] by [[Michael Sherwin]], [[CCC]], December 18, 2010</ref> :
Romi remembers and incorporates winning lines regardless of which side played the moves into the [[Opening Book|opening book]] and can play them back instantly up to 180 [[Ply|ply]] if the stats for that line remain good.
=See also=
* [[Brainfish]]
* [[Match Statistics]]
* [[Opening Book]]
* [[Persistent Hash Table]]
* [[Polyglot]]
* [[Reinforcement Learning]]
* [[Trial and Error]]
=Selected Publications=
* [[Arthur Samuel]] ('''1959'''). ''[http://domino.watson.ibm.com/tchjr/journalindex.nsf/600cc5649e2871db852568150060213c/39a870213169f45685256bfa00683d74!OpenDocument Some Studies in Machine Learning Using the Game of Checkers]''. IBM Journal July 1959
* [[Arthur Samuel]] ('''1967'''). ''Some Studies in Machine Learning. Using the Game of Checkers. II-Recent Progress''. [http://researcher.watson.ibm.com/researcher/files/us-beygel/samuel-checkers.pdf pdf]
* [[Michael Byrne]] ('''1996'''). ''M-Chess Pro 5.0 Book Learning Feature''. [[Computer Chess Reports]], Vol. 5, Nos. 3+4, p. 85 » [[MChess]]
* [[Robert Hyatt]] ('''1999'''). ''[http://www.craftychess.com/hyatt/learning.html Book Learning - a Methodology to Tune an Opening Book Automatically]''. [[ICGA Journal#22_1|ICCA Journal, Vol. 22, No. 1]]
* [[Michael Buro]] ('''1999'''). ''Toward Opening Book Learning.'' [[ICGA Journal#22_2|ICCA Journal, Vol. 22, No. 2]], [https://skatgame.net/mburo/ps/book.pdf pdf] <ref>[http://www.talkchess.com/forum/viewtopic.php?t=50680 Anybody tried Logistello's book learning for chess?] by [[Rémi Coulom]], [[CCC]], December 29, 2013</ref>
* [[Johannes Fürnkranz]] ('''2000''') ''Machine Learning in Games: A Survey''. Austrian Research Institute for Artificial Intelligence, Technical Report OEFAI-TR-2000-3, [http://www.ofai.at/cgi-bin/get-tr?download=1&paper=oefai-tr-2000-31.pdf pdf] - Chapter 2, Book Learning
* [[Marty Hirsch]] ('''2001'''). ''Machine Learning in MChess Professional'' - [[Advances in Computer Games 9]]
* [[Marek Strejczek]] ('''2004'''). ''Some aspects of chess programming''. [[Technical University of Łódź]] , Faculty of Electrical and Electronic Engineering, Department of Computer Science, [http://nesik.republika.pl/download//SomeAspectsOfChessProgramming.zip zipped pdf], 4.2 Experiments with book learning
* [[Thomas Widjaja]] ('''2004'''). ''Knowledge Engineering und Lernen in Spielen - Opening Book Learning''. [http://www.ke.tu-darmstadt.de/lehre/archiv/ss04/se-spiele/OpeningBookLearning.pdf slides as pdf] (German)
* [[Chrilly Donninger]], [[Ulf Lorenz]] ('''2006'''). ''[http://www.springerlink.com/content/b01701g6kx317574/ Innovative Opening-Book Handling]''. [[Advances in Computer Games 11]], [http://www2.cs.uni-paderborn.de/cs/ag-monien/PERSONAL/FLULO/publications/icga_open_springer.pdf pdf]
=Forum Posts=
==1998 ...==
* [https://www.stmintz.com/ccc/index.php?id=17861 Book learning and rating bias] by [[Don Dailey]], [[CCC]], May 01, 1998
* [https://www.stmintz.com/ccc/index.php?id=25754 BookLearning Under the Microscope!!!] by Robert Henry Durrett, [[CCC]], August 31, 1998
* [https://www.stmintz.com/ccc/index.php?id=37968 Book learning?] by [[Werner Inmann]], [[CCC]], December 31, 1998
* [https://www.stmintz.com/ccc/index.php?id=68359 Book learning] by [[James Robertson]], [[CCC]], September 12, 1999
==2000 ...==
* [https://www.stmintz.com/ccc/index.php?id=135774 Opening book learning] by Gordon Rattray, [[CCC]], October 31, 2000
* [https://www.stmintz.com/ccc/index.php?id=147500 What information to store in book learning?] by [[Christian Söderström]], [[CCC]], January 02, 2001
* [http://www.open-aurec.com/wbforum/viewtopic.php?f=18&t=37030 question about opening book and learning] by [[Uri Blass]], [[Computer Chess Forums|Winboard Forum]], April 26, 2002
* [https://www.stmintz.com/ccc/index.php?id=226258 question about book and learning] by [[Uri Blass]], [[CCC]], April 26, 2002
* [https://www.stmintz.com/ccc/index.php?id=294486 Shredder and Engine Book Learning] by Stephen Ham, [[CCC]], April 24, 2003 » [[Shredder]]
* [http://www.talkchess.com/forum/viewtopic.php?t=21754 A simple book learning method] by [[Alvaro Cardoso]], [[CCC]], June 12, 2008 » [[Persistent Hash Table]]
==2010 ...==
* [http://talkchess.com/forum/viewtopic.php?t=50662 Computer based Opening theory] by Lucas Braesch, [[CCC]], December 28, 2013
* [http://talkchess.com/forum/viewtopic.php?t=50662&start=16 Re: Computer based Opening theory] by [[Marcel van Kervinck]], [[CCC]], December 28, 2013
* [http://www.talkchess.com/forum/viewtopic.php?t=50680 Anybody tried Logistello's book learning for chess?] by [[Rémi Coulom]], [[CCC]], December 29, 2013
* [http://www.talkchess.com/forum/viewtopic.php?t=56313 Position learning and opening books] by Forrest Hoch, [[CCC]], May 11, 2015
* [http://www.talkchess.com/forum/viewtopic.php?t=61210 Polyglot book learning] by [[Günther Simon]], [[CCC]], August 24, 2016 » [[Polyglot]]
* [http://www.talkchess.com/forum/viewtopic.php?t=62639 Cumulative building of a shared search tree] by [[Bojun Guo]], [[CCC]], December 28, 2016 » [[Chinese Chess]], [[Opening Book]], [[Persistent Hash Table]]
=External Links=
* [http://www.top-5000.nl/authors/rebel/hints.htm Learning Methods] by [[Ed Schroder|Ed Schröder]]
=References=
<references />
'''[[Learning|Up one Level]]'''
[[FILE:France in XXI Century. School.jpg|border|right|thumb| Book Learning <ref>France in 2000 year (XXI century). Future school. France, paper card, A reproduction of the early 20th century, scan, [https://en.wikipedia.org/wiki/Learning Learning from Wikipedia]</ref> ]]
'''Book Learning''',<br/>
is a technique of unsupervised [[Reinforcement Learning|reinforcement learning]] motivated by basic requirements for successfully playing a sequence of games, such as avoiding losing games in the same way from certain lines from the [[Opening Book|opening book]], or to recognize its own strong points and guide the engine into positions it does well <ref>[http://www.top-5000.nl/authors/rebel/hints.htm Learning Methods] by [[Ed Schroder|Ed Schröder]]</ref>.
Unsupervised reinforcement learning is characterized by [[Trial and Error|trial and error]] and feeding back reinforcement signals, in this context to modify move probabilities or to add new book moves. The signals are either based on the final outcome of the game, called result-driven learning, and/or by [[Search|search-driven]] learning, i.e. from the [[Score|score]] and trend of first few searches after leaving the book to determine how the program likes the position <ref>[[Robert Hyatt]] ('''1999'''). ''[http://www.craftychess.com/hyatt/learning.html Book Learning - a Methodology to Tune an Opening Book Automatically]''. [[ICGA Journal#22_1|ICCA Journal, Vol. 22, No. 1]]</ref>.
=Chess Engines=
==Crafty==
[[Robert Hyatt]] on Book learning in [[Crafty]] <ref>[https://www.stmintz.com/ccc/index.php?id=68381 Re: Book learning] by [[Robert Hyatt]], [[CCC]], September 12, 1999</ref>:
* <code>Crafty does two kinds of 'book learning'. Result learning sound just like what you are doing. If it loses a game, it won't play the last book move it played ever again. And this gets backed up several moves back as well to get it out of bad lines quickly.</code>
* <code>Normal book learning uses the first ten non-book moves (search results) to determine the 'trend' of the game. This trend score is then used to adjust the book move that was played in this game...</code>
Both are on by default... and further <ref>[https://www.stmintz.com/ccc/index.php?id=340060 Re: questions about book learning] by [[Robert Hyatt]], [[CCC]], January 03, 2004</ref>:
# <code>Crafty remembers the evaluations for the first 10 moves out of book, after each search has been completed. It uses these evaluations to detect a "trend". IE is the evaluation good and getting better? Is it bad and getting worse? Is it good but dropping (ie it grabbed a gambit pawn and is beginning to see that it was bad) or is it bad but getting better (IE it offered a gambit, the opponent took it, and the score is going up). It factors all of that together and marks the book line as good or bad.</code>
# <code>Crafty takes the result of a game when it loses, and updates the book line so that moves tried near the end of the line simply don't get played, and alternatives near the front of the book line get tried next.</code>
# <code>There is more to it than that, and you can look at the crafty.doc file <ref>[http://www.craftychess.com/hyatt/craftydoc.html Crafty Documentation]</ref> to at least see what I am doing in more detail. It is '''very''' effective. </code>
==MChess==
[[Marty Hirsch]] in [[Advances in Computer Games 9]] on machine learning in [[MChess|MChess Pro]], excerpt <ref>[[Marty Hirsch]] ('''2001'''). ''Machine Learning in MChess Professional'' - [[Advances in Computer Games 9]], pp. 133-142, 135</ref>
Some sets of heuristics are employed to decide which moves to add to the opening book, and which to delete from it. Moves are deleted when the score just out of book is not too high, and the score later in the game is worse. Moves are added when the score just out of book is not too low, and the score later in the game is satisfactory. A further pre-condition for moves that may be added to the opening book is that the increase in score has been mainly monotonic. If this condition is not met, a future game may enable better moves to be found due to the general learning. In other words, only the best moves the program is able to discover (at a given level) are added to the opening book. Different thresholds are applied for the storage and deletion of moves for White and Black.
...
Some programs now support a different type of book learning in which no moves are added to the book, but instead, unsuccessful moves are given a lower probability of selection. This approach narrows the book over the time and may optimize the results against a specific opponent, at the expense of the results against other opponents. The book learning described in this paper maintains the opening book variety and improves the book against multiple opponents.
==RomiChess==
As explained by [[Michael Sherwin]], [[RomiChess]] applies [https://en.wikipedia.org/wiki/Monkey_see,_monkey_do Monkey see Monkey do] [https://en.wikipedia.org/wiki/Mimicry mimicry] to learn book lines <ref>[http://www.talkchess.com/forum/viewtopic.php?t=37062&start=15 Re: Positional learning] by [[Michael Sherwin]], [[CCC]], December 18, 2010</ref> :
Romi remembers and incorporates winning lines regardless of which side played the moves into the [[Opening Book|opening book]] and can play them back instantly up to 180 [[Ply|ply]] if the stats for that line remain good.
=See also=
* [[Brainfish]]
* [[Match Statistics]]
* [[Opening Book]]
* [[Persistent Hash Table]]
* [[Polyglot]]
* [[Reinforcement Learning]]
* [[Trial and Error]]
=Selected Publications=
* [[Arthur Samuel]] ('''1959'''). ''[http://domino.watson.ibm.com/tchjr/journalindex.nsf/600cc5649e2871db852568150060213c/39a870213169f45685256bfa00683d74!OpenDocument Some Studies in Machine Learning Using the Game of Checkers]''. IBM Journal July 1959
* [[Arthur Samuel]] ('''1967'''). ''Some Studies in Machine Learning. Using the Game of Checkers. II-Recent Progress''. [http://researcher.watson.ibm.com/researcher/files/us-beygel/samuel-checkers.pdf pdf]
* [[Michael Byrne]] ('''1996'''). ''M-Chess Pro 5.0 Book Learning Feature''. [[Computer Chess Reports]], Vol. 5, Nos. 3+4, p. 85 » [[MChess]]
* [[Robert Hyatt]] ('''1999'''). ''[http://www.craftychess.com/hyatt/learning.html Book Learning - a Methodology to Tune an Opening Book Automatically]''. [[ICGA Journal#22_1|ICCA Journal, Vol. 22, No. 1]]
* [[Michael Buro]] ('''1999'''). ''Toward Opening Book Learning.'' [[ICGA Journal#22_2|ICCA Journal, Vol. 22, No. 2]], [https://skatgame.net/mburo/ps/book.pdf pdf] <ref>[http://www.talkchess.com/forum/viewtopic.php?t=50680 Anybody tried Logistello's book learning for chess?] by [[Rémi Coulom]], [[CCC]], December 29, 2013</ref>
* [[Johannes Fürnkranz]] ('''2000''') ''Machine Learning in Games: A Survey''. Austrian Research Institute for Artificial Intelligence, Technical Report OEFAI-TR-2000-3, [http://www.ofai.at/cgi-bin/get-tr?download=1&paper=oefai-tr-2000-31.pdf pdf] - Chapter 2, Book Learning
* [[Marty Hirsch]] ('''2001'''). ''Machine Learning in MChess Professional'' - [[Advances in Computer Games 9]]
* [[Marek Strejczek]] ('''2004'''). ''Some aspects of chess programming''. [[Technical University of Łódź]] , Faculty of Electrical and Electronic Engineering, Department of Computer Science, [http://nesik.republika.pl/download//SomeAspectsOfChessProgramming.zip zipped pdf], 4.2 Experiments with book learning
* [[Thomas Widjaja]] ('''2004'''). ''Knowledge Engineering und Lernen in Spielen - Opening Book Learning''. [http://www.ke.tu-darmstadt.de/lehre/archiv/ss04/se-spiele/OpeningBookLearning.pdf slides as pdf] (German)
* [[Chrilly Donninger]], [[Ulf Lorenz]] ('''2006'''). ''[http://www.springerlink.com/content/b01701g6kx317574/ Innovative Opening-Book Handling]''. [[Advances in Computer Games 11]], [http://www2.cs.uni-paderborn.de/cs/ag-monien/PERSONAL/FLULO/publications/icga_open_springer.pdf pdf]
=Forum Posts=
==1998 ...==
* [https://www.stmintz.com/ccc/index.php?id=17861 Book learning and rating bias] by [[Don Dailey]], [[CCC]], May 01, 1998
* [https://www.stmintz.com/ccc/index.php?id=25754 BookLearning Under the Microscope!!!] by Robert Henry Durrett, [[CCC]], August 31, 1998
* [https://www.stmintz.com/ccc/index.php?id=37968 Book learning?] by [[Werner Inmann]], [[CCC]], December 31, 1998
* [https://www.stmintz.com/ccc/index.php?id=68359 Book learning] by [[James Robertson]], [[CCC]], September 12, 1999
==2000 ...==
* [https://www.stmintz.com/ccc/index.php?id=135774 Opening book learning] by Gordon Rattray, [[CCC]], October 31, 2000
* [https://www.stmintz.com/ccc/index.php?id=147500 What information to store in book learning?] by [[Christian Söderström]], [[CCC]], January 02, 2001
* [http://www.open-aurec.com/wbforum/viewtopic.php?f=18&t=37030 question about opening book and learning] by [[Uri Blass]], [[Computer Chess Forums|Winboard Forum]], April 26, 2002
* [https://www.stmintz.com/ccc/index.php?id=226258 question about book and learning] by [[Uri Blass]], [[CCC]], April 26, 2002
* [https://www.stmintz.com/ccc/index.php?id=294486 Shredder and Engine Book Learning] by Stephen Ham, [[CCC]], April 24, 2003 » [[Shredder]]
* [http://www.talkchess.com/forum/viewtopic.php?t=21754 A simple book learning method] by [[Alvaro Cardoso]], [[CCC]], June 12, 2008 » [[Persistent Hash Table]]
==2010 ...==
* [http://talkchess.com/forum/viewtopic.php?t=50662 Computer based Opening theory] by Lucas Braesch, [[CCC]], December 28, 2013
* [http://talkchess.com/forum/viewtopic.php?t=50662&start=16 Re: Computer based Opening theory] by [[Marcel van Kervinck]], [[CCC]], December 28, 2013
* [http://www.talkchess.com/forum/viewtopic.php?t=50680 Anybody tried Logistello's book learning for chess?] by [[Rémi Coulom]], [[CCC]], December 29, 2013
* [http://www.talkchess.com/forum/viewtopic.php?t=56313 Position learning and opening books] by Forrest Hoch, [[CCC]], May 11, 2015
* [http://www.talkchess.com/forum/viewtopic.php?t=61210 Polyglot book learning] by [[Günther Simon]], [[CCC]], August 24, 2016 » [[Polyglot]]
* [http://www.talkchess.com/forum/viewtopic.php?t=62639 Cumulative building of a shared search tree] by [[Bojun Guo]], [[CCC]], December 28, 2016 » [[Chinese Chess]], [[Opening Book]], [[Persistent Hash Table]]
=External Links=
* [http://www.top-5000.nl/authors/rebel/hints.htm Learning Methods] by [[Ed Schroder|Ed Schröder]]
=References=
<references />
'''[[Learning|Up one Level]]'''
