Difference between revisions of "Endgame Tablebases"

From Chessprogramming wiki
Jump to: navigation, search
(Quotes)
 
(26 intermediate revisions by 2 users not shown)
Line 6: Line 6:
 
precalculated [[Endgame|endgame]] [https://en.wikipedia.org/wiki/Table_%28database%29 tables] generated by an exhaustive [[Retrograde Analysis|retrograde analysis]]. During its [[Chess Game|game play]] and/or [[Search|search]], [[Interior Node Recognizer|recognizing]] a specific [[Material|material composition]], a chess program can probe, or in principle compute these tables to determine the outcome of positions definitively and act as an [[Oracle|oracle]] providing the optimal moves. The tables are usually divided into separate [https://en.wikipedia.org/wiki/Computer_file files] for each material configuration and [[Side to move|side to move]]. What all different [[Endgame Tablebases#Formats|formats]] of tablebases have in common is that every possible piece placement has its own, unique index number which represents the position where the information about the position is located in the file. Positions with non-null castling rights could be represented in an EGTB but typically are not.
 
precalculated [[Endgame|endgame]] [https://en.wikipedia.org/wiki/Table_%28database%29 tables] generated by an exhaustive [[Retrograde Analysis|retrograde analysis]]. During its [[Chess Game|game play]] and/or [[Search|search]], [[Interior Node Recognizer|recognizing]] a specific [[Material|material composition]], a chess program can probe, or in principle compute these tables to determine the outcome of positions definitively and act as an [[Oracle|oracle]] providing the optimal moves. The tables are usually divided into separate [https://en.wikipedia.org/wiki/Computer_file files] for each material configuration and [[Side to move|side to move]]. What all different [[Endgame Tablebases#Formats|formats]] of tablebases have in common is that every possible piece placement has its own, unique index number which represents the position where the information about the position is located in the file. Positions with non-null castling rights could be represented in an EGTB but typically are not.
  
The term '''Tablebase''' opposed to endgame database was coined by [[Steven Edwards]] by means of a data vector and a single file access for both sides to move, when he introduced his [[Edwards' Tablebases]], addressing several shortcomings of [[Ken Thompson|Ken Thompson's]] earlier [[Thompson's Databases|database]] of positions with the weaker and hence potentially losing side to move only <ref>[[Ernst A. Heinz]] ('''1999'''). ''Endgame Databases and Efficient Index Schemes for Chess.'' [[ICGA Journal#22_1|ICCA Journal, Vol. 22, No. 1]], [http://people.csail.mit.edu/heinz/ps/edb_index.ps.gz ps]</ref> .  
+
The term '''Tablebase''' opposed to endgame database was coined by [[Steven Edwards]] by means of a data vector and single file access for both sides to move, when he introduced his [[Edwards' Tablebases]], addressing several shortcomings of [[Ken Thompson|Ken Thompson's]] earlier [[Thompson's Databases|database]] of positions with the weaker and hence potentially losing side to move only <ref>[[Ernst A. Heinz]] ('''1999'''). ''Endgame Databases and Efficient Index Schemes for Chess.'' [[ICGA Journal#22_1|ICCA Journal, Vol. 22, No. 1]], [http://people.csail.mit.edu/heinz/ps/edb_index.ps.gz ps]</ref> .  
  
 
=Purposes=
 
=Purposes=
When a [[Chess Game|chess game]] goes to the [[Endgame|endgame]] phase, it may complete soon with the result as a draw or a win/loss. Both human/chess engines can finish the endgame by continuing searching, applying some endgame rules. The game can finish in this way but it is not in a perfect way: it takes much time to make move, use more moves than necessary. Sometimes the winning side may lose the change since the rule 50 moves.
+
When a [[Chess Game|chess game]] goes to the [[Endgame|endgame]] phase, it may complete soon with the result as a draw or a win/loss. Both human/chess engines can finish the endgame by continuing searching, and applying some endgame rules. Even the game can finish in this way but it is not in a perfect way: it takes time to compute moves and may use more moves than necessary. Sometimes the winning side may lose the change since the rule 50 moves and/or makes blunders. Both human and chess engines may miss the chances badly if they don’t know or not good enough about the endgame they are playing.
  
EGTB can help to solve this period. It is a kind of dictionary of all endgame positions which can answer directly or indirectly for a given position:
+
An EGTB can help to solve this period. It is a kind of dictionary of all endgame positions that can answer immediately for a given position:
* it is a draw or a win/loss
+
* If it is a draw or a win/loss
* how far/how many move from matting/being mated
+
* How far/how many move from matting/being mated
* what is the best move
+
* What is the best move
  
 
The main advantages of using EGTBs:  
 
The main advantages of using EGTBs:  
* It does not search but retrieves data thus it can move instantly
+
* Engines don’t have to search but retrieve data thus they can move instantly
* The move is the perfect one, leading the shortest line to win
+
* Moves are the perfect ones, leading the shortest line to win
 +
* Engines don’t need to have the knowledge and code about those endgames
  
 
The main disadvantages of using EGTBs:
 
The main disadvantages of using EGTBs:
* Data too large for downloading and storing
+
* Their data typically are too large for downloading and storing
* If use when searching, it may slow down the search
+
* They may slow down the search
* Require a complicated code for probing
+
* Require complicated codes for probing
  
 
=Data=
 
=Data=
An EGTB is a set of endgames. Each endgame is a set of positions (they must have the same material). Each position associates with an integer number which informs how far that position is from mating/being mated or converting (depends on the type of its metrics). All numbers of an endgame are simply organized as two arrays of integers (one array for one side/color). In other words, each item in those arrays is an integer and its index on an array is mapped to a unique position.
+
An EGTB is a set of endgames. Each endgame is a set of records about positions’ information. All involved positions in an endgame must have the same material. Typically each position's record is associated with an integer number which informs how far that position is from mating/being mated or converting (depends on the type of its metrics).  
  
Based on values those integer numbers can answer directly two purposes of the EGTB: for a given position it is a draw or a win/loss position and how far it is from mating/being mated or converting. From those numbers, the best move of a position can be indirectly calculated as below part mentions. Theoretically, we can add more information to each item. However, due to large involving positions, any additional information may make the whole EGTB becomes significantly larger. None of the popular EGTBs store any extra information for each item.
+
Based on values of those integer numbers it can answer directly two questions of the EGTB: 1) for a given position it is a draw or a win/loss position and 2) how far it is from mating/being mated or converting. To answer the 3rd question, the best move of a position can be indirectly calculated: generate all legal moves, make them, probe all new positions, compare probing scores for the highest one and the associated move is the best one.
  
Those arrays of an endgame can be stored in files. Typically each array is stored in one file. However, sometimes they can be combined into one file or device into more files to be easy to copy or manage. Those files may be in the form of none or compressed. If they are compressed, they are divided and compressed by small blocks thus they don't need to decompress all to extract data of just a few positions.  
+
Theoretically, we can add more information to each position’s record. However, due to the large number of involved positions, any additional information may make the whole EGTB become significantly larger. Thus so far none of the popular EGTBs store any extra information and their records actually contain those integers only.
  
 +
All records of an endgame are simply organized as two arrays (one array for one side/colour). In other words, each item in those arrays is an integer and its index on an array is mapped to a unique position. For a given chess position, we will calculate its index and then access its data record/integer via that index without searching.
 +
 +
==Endgame files==
 +
Those arrays of an endgame can be stored in files. Typically each array is stored in one file. However, sometimes they can be combined into one larger file or split into more files to easier copy or manage. Those files may be in compressed or uncompressed forms. If they are compressed, they are typically divided and compressed by small blocks thus they don't need to load and decompress all data to extract just a position.
 +
 +
==In memory==
 +
Some chess engines create themselves EGTBs on-flight and keep all data in memory without saving/loading to files. Due to the limit by the initial time they can generate and use very small EGTBs only.
 +
 +
==Sizes==
 
The size of an endgame depends on:
 
The size of an endgame depends on:
 
* The number of involving pieces: more pieces, the bigger size
 
* The number of involving pieces: more pieces, the bigger size
Line 49: Line 59:
 
==DTC==  
 
==DTC==  
 
'''Depth to Conversion'''
 
'''Depth to Conversion'''
'Conversion' is the changing of the force on the board, by capture and/or Pawn-conversion. For each position that is represented, 'DTC' indicates the theoretical value, and the number of winner's moves to the goal of 'Won Conversion' if won/lost - assuming that the winner is minimizing and the loser is maximizing DTC. DTC <span style="font-family: Symbol;">£ </span>DTM.
+
'Conversion' is the changing of the force on the board, by capture and/or Pawn-conversion. For each position that is represented, 'DTC' indicates the theoretical value, and the number of winners moves to the goal of 'Won Conversion' if won/lost - assuming that the winner is minimizing and the loser is maximizing DTC. DTC <span style="font-family: Symbol;">£ </span>DTM.
  
 
==DTZ==  
 
==DTZ==  
 
'''Depth to Zeroing (of the ply count)'''
 
'''Depth to Zeroing (of the ply count)'''
The ply-count is zeroed if and only if a Pawn is pushed or a capture is made. For each position that is represented, 'DTZ' indicates the theoretical value, and the number of winner's moves to the goal of 'Win and ply-count zeroed' if won/lost - assuming that the winner is minimizing and the loser is maximizing DTZ. DTZ <span style="font-family: Symbol;">£ </span>DTC <span style="font-family: Symbol;">£ </span>DTM.
+
The ply-count is zeroed if and only if a Pawn is pushed or a capture is made. For each position that is represented, 'DTZ' indicates the theoretical value, and the number of winners moves to the goal of 'Win and ply-count zeroed' if won/lost - assuming that the winner is minimizing and the loser is maximizing DTZ. DTZ <span style="font-family: Symbol;">£ </span>DTC <span style="font-family: Symbol;">£ </span>DTM.
  
 
==DTZ50==  
 
==DTZ50==  
Line 59: Line 69:
 
The DTC, DTM and DTZ metrics do not consider the current FIDE [[Fifty-move Rule|50-move rule]] whereby a game in which no [[Irreversible Moves|irreversible moves]] have been played for 100 plies can be claimed as a draw by the player 'on move'. In this metric, in addition to the DTC/DTM/DTZ ''draws'', positions requiring more than 50 winner's moves to zero the ply-count are defined as ''draws''. DTZ <span style="font-family: Symbol;">£ </span>DTZ50 if DTZ50 <span style="font-family: Symbol; font-size: 14.66px;">¹ </span>''draw.'' 'DTZ50' is one of a set of 'DTZk' rules: a ''k''-''move rule'' effectively applies if there are only ''k'' moves left before a draw-claim might be made. ''k1'' <span style="font-family: Symbol;">£ </span>''k2'' implies ''DTZk2'' <span style="font-family: Symbol;">£ </span>''DTZk1 if DTZk1'' <span style="font-family: Symbol; font-size: 14.66px;">¹ </span>draw.
 
The DTC, DTM and DTZ metrics do not consider the current FIDE [[Fifty-move Rule|50-move rule]] whereby a game in which no [[Irreversible Moves|irreversible moves]] have been played for 100 plies can be claimed as a draw by the player 'on move'. In this metric, in addition to the DTC/DTM/DTZ ''draws'', positions requiring more than 50 winner's moves to zero the ply-count are defined as ''draws''. DTZ <span style="font-family: Symbol;">£ </span>DTZ50 if DTZ50 <span style="font-family: Symbol; font-size: 14.66px;">¹ </span>''draw.'' 'DTZ50' is one of a set of 'DTZk' rules: a ''k''-''move rule'' effectively applies if there are only ''k'' moves left before a draw-claim might be made. ''k1'' <span style="font-family: Symbol;">£ </span>''k2'' implies ''DTZk2'' <span style="font-family: Symbol;">£ </span>''DTZk1 if DTZk1'' <span style="font-family: Symbol; font-size: 14.66px;">¹ </span>draw.
  
Note that, because depth is measured in winner's moves rather than plies, this metric does '''not''' define as ''draw'' a position in which 50 winner's moves and 51 loser's moves are required to zero the ply-count, despite the fact that the loser would claim a draw after 100 plies had been played in that endgame.
+
Note that, because depth is measured in winner's moves rather than plies, this metric does '''not''' define as ''draw'' a position in which 50 winner's moves and 51 loser's moves are required to zero the ply count, despite the fact that the loser would claim a draw after 100 plies had been played in that endgame.
  
 
==DTR==  
 
==DTR==  
 
'''Depth by the (''k''-move) Rule'''
 
'''Depth by the (''k''-move) Rule'''
'The Rule' referred to here is a notional set of ''k''-move rules. For each position that is represented, 'DTR' indicates the theoretical value and, for won/lost positions'','' the least ''k'' for which DTZ''k'' is not ''draw'' - assuming that the winner is minimising and the loser is maximising DTR. DTZ <span style="font-family: Symbol;">£ </span>DTR. No DTR EGTs have been created to date <ref>[http://kirill-kryukov.com/chess/discussion-board/viewtopic.php?f=6&t=3934 DTR is no good!] by [[Harm Geert Muller]], [[Computer Chess Forums|CCRL Forum]], September 24, 2008</ref> .
+
'The Rule' referred to here is a notional set of ''k''-move rules. For each position that is represented, 'DTR' indicates the theoretical value and, for won/lost positions'','' the least ''k'' for which DTZ''k'' is not ''draw'' - assuming that the winner is minimising and the loser is maximising DTR. DTZ <span style="font-family: Symbol;">£ </span>DTR. No DTR EGTs have been created to date <ref>[http://kirill-kryukov.com/chess/discussion-board/viewtopic.php?f=6&t=3934 DTR is no good!] by [[Harm Geert Muller]], [[Computer Chess Forums|CCRL Forum]], September 24, 2008</ref>.
  
 
==DTZR==  
 
==DTZR==  
'''Depth to Zeroing move, give that DTR is being minimaxed'''
+
'''Depth to Zeroing move, given that DTR is being minimaxed'''
If DTR = ''draw'', then DTZR = ''draw''. Otherwise, Otherwise DTRZR indicates the theoretical value, and the number of winner's moves to the zeroing of the ply count, given that the two sides are minimaxing first DTR and then DTZR. DTZR would be used in conjunction with DTR to progress a deep win in the context of a ''k''-move rule because DTR can remain unchanged over a sequence of moves. DTZR <span style="font-family: Symbol;">£ </span>DTR but it is not always the case that DTZ <span style="font-family: Symbol;">£ </span>DTZR as may seem intuitively obvious: the loser may zero the move-count to maximise DTR.
+
If DTR = ''draw'', then DTZR = ''draw''. Otherwise, DTRZR indicates the theoretical value, and the number of winners moves to the zeroing of the ply count, given that the two sides are minimaxing first DTR and then DTZR. DTZR would be used in conjunction with DTR to progress a deep win in the context of a ''k''-move rule because DTR can remain unchanged over a sequence of moves. DTZR <span style="font-family: Symbol;">£ </span>DTR but it is not always the case that DTZ <span style="font-family: Symbol;">£ </span>DTZR as may seem intuitively obvious: the loser may zero the move-count to maximise DTR.
  
 
==Summary==  
 
==Summary==  
To summarize the different metrics of tablebases, each metric has its strengths and weaknesses. [[Nalimov Tablebases|Nalimov's DTM EGTs]] are widely and commercially available, and used by many chess engines or chess GUIs. DTC EGTs are more easily computed and stored as they involve smaller depth-ranges and are more compressible. For endgames with less or equal than 5 pieces on the board, KNNKP comes closest to requiring more than one [[Byte]] per position to store DTM: maxDTM = 115 (1-0) and 73 (0-1) necessitating 190 values in the EGT. For KPPKP, maxDTM = 127 (1-0) but only 42 (0-1). But for certain 6 piece endgames, one Byte is not enough: maxDTM = 262 for a KRNKNN position. The DTC and DTZ metrics increasingly postpone the need for two Bytes per position in an uncompressed EGT, but KRBNKQN has maxDTC = maxDTZ = 517 (0-1), both wtm and btm.
+
To summarize the different metrics of tablebases, each metric has its strengths and weaknesses. [[Nalimov Tablebases|Nalimov's DTM EGTs]] are widely and commercially available, and used by many chess engines or chess GUIs. DTC EGTs are more easily computed and stored as they involve smaller depth ranges and are more compressible. For endgames with less or equal to 5 pieces on the board, KNNKP comes closest to requiring more than one [[Byte]] per position to store DTM: maxDTM = 115 (1-0) and 73 (0-1) necessitating 190 values in the EGT. For KPPKP, maxDTM = 127 (1-0) but only 42 (0-1). But for certain 6-piece endgames, one Byte is not enough: maxDTM = 262 for a KRNKNN position. The DTC and DTZ metrics increasingly postpone the need for two Bytes per position in an uncompressed EGT, but KRBNKQN has maxDTC = maxDTZ = 517 (0-1), both wtm and btm.
  
In extremis, he unmoderated use of DTM, DTC or DTZ EGTs will let slip a winnable position in the context of the 50-move rule ... hence the marginal need for the DTR and DTZR metrics, coupled with the ability to recompute DTR/DTZR EGTs in the context of a limited move-budget.
+
In extremis, the unmoderated use of DTM, DTC or DTZ EGTs will let slip a winnable position in the context of the 50-move rule ... hence the marginal need for the DTR and DTZR metrics, coupled with the ability to recompute DTR/DTZR EGTs in the context of a limited move-budget.
 
<span id="Bitbases"></span>
 
<span id="Bitbases"></span>
 
=Bitbases=  
 
=Bitbases=  
Line 80: Line 90:
  
 
=Indexing=  
 
=Indexing=  
Every position is assigned to a unique index to specify the location of the information stored about it. The main purpose of indexing is to locate and read information (from databases) such as distance to mate for a given position based on its index.  
+
Every position is assigned to a unique index to specify the location of the information stored about it. The main purpose of indexing is to locate and read the information (from databases) such as distance to mate for a given position based on its index.  
  
There are two main aims when generating the index number. The easier the algorithm to generate the index from any given position, the faster will be the probing of the tablebase as well as the generation of it. But the second aim is to get a relatively small range of index-numbers, to keep the size of the file as small as possible.
+
There are two main aims when generating the index number. The easier the algorithm to generate the index from any given position, the faster will be the probing of the tablebase as well as the generation of it. But the second aim is to get a relatively small range of index numbers, to keep the size of the file as small as possible.
  
The most straight forward way would be to for every piece on the board use 6 bit (1-64 squares)
+
The most straightforward way would be to for every piece on the board use 6 bit (1-64 squares)
 
<pre>
 
<pre>
 
index = position_white_King + position_black_King * 64
 
index = position_white_King + position_black_King * 64
Line 93: Line 103:
 
</pre>
 
</pre>
  
Almost all work on this field is to reduce index spaces and reduce data sizes (e.g., by compressing). The above indexing can be optimised in several ways.
+
Almost all work in this field is to reduce index spaces and reduce data sizes (e.g., by compressing). The above indexing can be optimised in several ways.
  
 
Remove redundant symmetrical indexes:
 
Remove redundant symmetrical indexes:
* A chess board can be rotated and mirrored (if pawns are on the board only a vertical reflection is possible).
+
* A chessboard can be rotated and mirrored (if pawns are on the board only a vertical reflection is possible).
 
* For pieces of the same type, different arrangements can be left away.
 
* For pieces of the same type, different arrangements can be left away.
 
Remove illegal indexes:
 
Remove illegal indexes:
 
* Pieces may not stand on the same square.
 
* Pieces may not stand on the same square.
* The side to move may not give check.
+
* The side to move may not give a check.
 
* The distance between the two Kings must be greater than one.
 
* The distance between the two Kings must be greater than one.
  
 
=Retrieving=
 
=Retrieving=
 
==Probe==
 
==Probe==
When a chess program (e.g., a chess engine or a chess GUI) works with an EGTB, it needs to retrieve data (integer numbers) for some given chess positions. The process named probe and has below steps:
+
When a chess program (e.g., a chess engine or a chess GUI) works with an EGTB, it needs to retrieve data (integer numbers) for some given chess positions. The process is named probe and has below steps:
* convert a given chess position into an index via indexing algorithms
+
* Convert a given chess position into an index via indexing algorithms
* access data (typically organized as an integer array) using the index to retrieve the integer of that chess position
+
* Access data (typically organised as arrays of integers) using the index to retrieve the integer of that chess position
  
 
If data of that endgame is not ready in memory, usually the probe process has to do some extra work:
 
If data of that endgame is not ready in memory, usually the probe process has to do some extra work:
* From data index calculate to block index
+
* From the data index calculate the block index
 
* Read data of that block from storage into memory
 
* Read data of that block from storage into memory
* Uncompress the block (if it is compressed)
+
* Uncompress the block data (if it is compressed)
* Convert the endgame index into block index and retrieve the right data
+
* Convert the endgame index into a block index and retrieve the needed data
  
 
==Search==
 
==Search==
After probing the chess engine may have the result as an integer number for the queried position. Depending on the metric, the number may be a distance to mate or distance to conversion, the position is a draw or a win or loss for the querying side. Sometimes that value (state of draw/win/loss) may be enough for searching. However, sometimes that state is not enough and it needs to know which is the best move to make. The solution is to generate all legal moves from that position, make one by one, probe that EGTB for values of each new position. After all, compare the values of all children's positions to find which one is the best to make.
+
After probing the chess engine may have the result as an integer number for the queried position. Depending on the metric, the number may be a distance to mate or distance to conversion, the position is a draw or a win or loss for the querying side. Sometimes that value (state of draw/win/loss) may be enough for searching. However, sometimes that state is not enough and it needs to know which is the best move to make. The solution is to generate all legal moves from that position, make them one by one, and probe that EGTB for the values of each new position. After all, compare the values of all children's positions to find which one is the best to make.
  
 
==Speed==
 
==Speed==
Typically the probe process requires some computing and it may access storage which is usually so slow, compared with one stored in memory. All may make it be a slow one. An engine that probes EGTB when searching may make the search be slower. The good point is that the engine can stop searching on the branch probed the EGTB.
+
Typically the probe process requires some computing and it may access storage and decompress which is usually so slow. An engine that probes EGTB when searching may make the whole search be slower. Sometimes the benefit from probing EGTBs may not be enough to cover the loss of slower search. That’s why the probing should be planned and implemented carefully. EGTB files may have to be stored in fast storage and/or using some huge systems’ caches. The good point is that the engine can stop searching on the branch probed by the EGTB.
 
 
  
 
=Brief Info=  
 
=Brief Info=  
Line 183: Line 192:
 
|}
 
|}
 
<span id="Formats"></span>
 
<span id="Formats"></span>
 +
=7-man EGTBs=
 +
==2005==
 +
In 2005, [[Yakov Konoval]] started to collaborate with [[Marc Bourzutschky]] on constructing 7-man EGTBs. Yakov wrote the generator and Marc a verification program and some utilities for extracting the data. In October 2005, Yakov Konoval and Marc Bourzutschky announced that a position in the ending of a KRRNkrr requires 290 moves to convert to a simpler winning endgame <ref>[https://www.stmintz.com/ccc/index.php?id=456178 Subject: KRRNKRR win in 290: a new record] by [[Marc Bourzutschky]] from [[CCC]], October 16, 2005</ref>. The old record was 243 moves from a position in a rook and knight versus two knights endgame, discovered by [[Lewis Stiller]] in 1991 <ref>[https://en.wikipedia.org/wiki/Chess_endgame Chess endgame from Wikipedia]</ref>.
 +
 +
==2006==
 +
In March 2006, the wizards of 7-men endgames, Marc Bourzutschky and Yakov Konoval found 330 moves win in KQBNkqb <ref>[http://www.xs4all.nl/~timkr/chess2/diary_16.htm 311. 31 March 2006: White plays and wins in 330 moves] by [https://en.wikipedia.org/wiki/Tim_Krabb%C3%A9 Tim Krabbé]</ref> and in May 2006 a 517 moves win in KQNkrbn <ref>[http://www.xs4all.nl/~timkr/chess2/diary_16.htm OPEN CHESS DIARY 316. 26 May 2006: A 517-move win] by [https://en.wikipedia.org/wiki/Tim_Krabb%C3%A9 Tim Krabbé]</ref>.
 +
 +
==2012==
 +
In July 2012, [[Vladimir Makhnychev]] and [[Victor Zakharov]] from [[Moscow State University]] completed 4+3 DTM-tablebases (525 endings including KPPPKPP). The next set of 5+2 DTM-tablebases was completed in August 2012 <ref>[https://en.wikipedia.org/wiki/Endgame_tablebase#Background Endgame tablebase from Wikipedia - Background]</ref>. The tablebases are named [[Lomonosov Tablebases]], using the [[Lomonosov Supercomputer]].
 +
 +
==2013==
 +
[[ChessOK|Convekta Ltd.]] announced the release of [[Lomonosov Tablebases]]. Until December 31, 2013, the tablebases will be available online through [[ChessOK]] [[ChessOK#Products|products]] <ref>[http://chessok.com/?page_id=27966 Lomonosov Endgame Tablebases] - [[ChessOK]]</ref>.
 +
 +
==2018==
 +
Starting in March 2018, '''seven piece''' [[Syzygy Bases]], originally created for up to six pieces by [[Ronald de Man]] in 2013, were generated by [[Bojun Guo]], finished in August 2018 <ref>[http://talkchess.com/forum3/viewtopic.php?f=7&t=66797&start=472 Re: 7-men Syzygy attempt] by [[Bojun Guo]], [[CCC]], August 19, 2018 </ref>.
 +
 +
 +
=Quotes=
 +
[[Evert]] explained pros and cons of some popular EGTBs, March 07, 2018 <ref>[https://www.talkchess.com/forum/viewtopic.php?p=753451#p753451 Re: The history of Syzygy tablebases] by [[Evert]], [[CCC]], March 07, 2018</ref>:
 +
Scorpio and (iirc) Shredder are bitbases: they store whether a position is won for the side to move or not. Typically that is enough during the search, as long as the root position is not a tablebase position. In that case you need a way to make progress, and the bitbases don't really help. However, in practice chess programs can win many endgames without tablebases.
 +
Nalimov and Gaviota store distance to mate (DTM) for the side to move (or not-won if the position is not won), but they ignore draws by the 50-move rule. Nalimov has 6 men, but comes with an obnoxious custom license. Gaviota probing code is open source under a liberal license, but it only has up to 5 men. Compression is (iirc) much better than Nalimov.
 +
Lomonosov is DTM up to 7 pieces, but not available for free (and impractically large).
 +
Syzygy bases store the number of moves needed to win a position under the 50-move rule (DTZ50). You win by either mating the opponent, making a winning pawn push or by converting to another won ending. The generator and probing code are open source.
 +
For practical purposes, if you pick just one, Syzygy is probably the best pick. In a pinch, Syzygy+Gaviota might be better, since they will give you the shortest DTM in positions that are won under the 50-move rule. Syzygy alone doesn't do that. If that is better or not is a matter of taste.
 +
Nalimov should be a no-go because of the license and Lomonosov is simply not an option.
 +
Pure bitbases are probably the least useful these days (Syzygy and I think Gaviota actually already include bitbases for the search).
 +
 
=Formats=  
 
=Formats=  
 
* [[Chenard#EGDB|Chenard's EGDB]]
 
* [[Chenard#EGDB|Chenard's EGDB]]
 
* [[Edwards' Tablebases]]
 
* [[Edwards' Tablebases]]
 
* [[Gaviota Tablebases]]
 
* [[Gaviota Tablebases]]
 +
* [[Felicity Tablebases]]
 
* [[Lomonosov Tablebases]]
 
* [[Lomonosov Tablebases]]
 
* [[Nalimov Tablebases]]
 
* [[Nalimov Tablebases]]
Line 192: Line 229:
 
* [[Syzygy Bases]]
 
* [[Syzygy Bases]]
 
* [[Thompson's Databases]]
 
* [[Thompson's Databases]]
 +
 
=Generators=  
 
=Generators=  
 
* [[FinalGen]]
 
* [[FinalGen]]
Line 198: Line 236:
 
* [[Retro]]
 
* [[Retro]]
 
<span id="7-men"></span>
 
<span id="7-men"></span>
=7-man EGTBs=
 
==2005==
 
In 2005, [[Yakov Konoval]] started to collaborate with [[Marc Bourzutschky]] on constructing 7-man EGTBs. Yakov wrote the generator and Marc a verification program and some utilities for extracting the data. In October 2005, Yakov Konoval and Marc Bourzutschky announced that a position in the ending of a KRRNkrr requires 290 moves to convert to a simpler winning endgame <ref>[https://www.stmintz.com/ccc/index.php?id=456178 Subject: KRRNKRR win in 290: a new record] by [[Marc Bourzutschky]] from [[CCC]], October 16, 2005</ref> . The old record was 243 moves from a position in a rook and knight versus two knights endgame, discovered by [[Lewis Stiller]] in 1991 <ref>[https://en.wikipedia.org/wiki/Chess_endgame Chess endgame from Wikipedia]</ref> .
 
 
==2006==
 
In March 2006, the wizards of 7-men endgames, Marc Bourzutschky and Yakov Konoval found a 330 moves win in KQBNkqb <ref>[http://www.xs4all.nl/~timkr/chess2/diary_16.htm 311. 31 March 2006: White plays and wins in 330 moves] by [https://en.wikipedia.org/wiki/Tim_Krabb%C3%A9 Tim Krabbé]</ref> and in May 2006 a 517 moves win in KQNkrbn <ref>[http://www.xs4all.nl/~timkr/chess2/diary_16.htm OPEN CHESS DIARY 316. 26 May 2006: A 517-move win] by [https://en.wikipedia.org/wiki/Tim_Krabb%C3%A9 Tim Krabbé]</ref> .
 
 
==2012==
 
In July 2012, [[Vladimir Makhnychev]] and [[Victor Zakharov]] from [[Moscow State University]] completed 4+3 DTM-tablebases (525 endings including KPPPKPP). The next set of 5+2 DTM-tablebases was completed during August 2012 <ref>[https://en.wikipedia.org/wiki/Endgame_tablebase#Background Endgame tablebase from Wikipedia - Background]</ref> . The tablebases are named [[Lomonosov Tablebases]], using the [[Lomonosov Supercomputer]].
 
 
==2013==
 
[[ChessOK|Convekta Ltd.]] announced the release of [[Lomonosov Tablebases]]. Until December 31, 2013, the tablebases will be available online through [[ChessOK]] [[ChessOK#Products|products]] <ref>[http://chessok.com/?page_id=27966 Lomonosov Endgame Tablebases] - [[ChessOK]]</ref>.
 
 
==2018==
 
Starting in March 2018, '''seven piece''' [[Syzygy Bases]], originally created for up to six pieces by [[Ronald de Man]] in 2013, were generated by [[Bojun Guo]], finished in August 2018 <ref>[http://talkchess.com/forum3/viewtopic.php?f=7&t=66797&start=472 Re: 7-men Syzygy attempt] by [[Bojun Guo]], [[CCC]], August 19, 2018 </ref>.
 
  
 
=See also=  
 
=See also=  
Line 319: Line 342:
 
* [[Ernst A. Heinz]] ('''1999'''). ''Endgame Databases and Efficient Index Schemes for Chess.'' [[ICGA Journal#22_1|ICCA Journal, Vol. 22, No. 1]], [http://people.csail.mit.edu/heinz/ps/edb_index.ps.gz ps]
 
* [[Ernst A. Heinz]] ('''1999'''). ''Endgame Databases and Efficient Index Schemes for Chess.'' [[ICGA Journal#22_1|ICCA Journal, Vol. 22, No. 1]], [http://people.csail.mit.edu/heinz/ps/edb_index.ps.gz ps]
 
* [[Ernst A. Heinz]] ('''1999'''). ''Knowledgeable Encoding and Querying of Endgame Databases.'' [[ICGA Journal#22_2|ICCA Journal, Vol. 22, No. 2]], [http://people.csail.mit.edu/heinz/ps/know_edb.ps.gz ps]
 
* [[Ernst A. Heinz]] ('''1999'''). ''Knowledgeable Encoding and Querying of Endgame Databases.'' [[ICGA Journal#22_2|ICCA Journal, Vol. 22, No. 2]], [http://people.csail.mit.edu/heinz/ps/know_edb.ps.gz ps]
* [[Christoph Wirth]], [[Jürg Nievergelt]] ('''1999'''). ''Exhaustive and Heuristic Retrograde Analysis of the KPPKP Endgame.'' [[ICGA Journal#22_2|ICCA Journal, Vol. 22, No. 2]]
+
* [[Christoph Wirth]], [[Jürg Nievergelt]] ('''1999'''). ''Exhaustive and Heuristic Retrograde Analysis of the KPPKP Endgame.'' [[ICGA Journal#22_2|ICCA Journal, Vol. 22, No. 2]] <ref>[http://kirill-kryukov.com/chess/discussion-board/viewtopic.php?f=6&t=2923 Chris Wirth's endgame tablebases] by [[Denis Mendoza|Denis P. Mendoza]], [[Computer Chess Forums|CCRL Endgame Tablebases]], December 13, 2007 » [[Chessterfield]]</ref>
 
* [[Eugene Nalimov]], [[Christoph Wirth]], [[Guy Haworth]] ('''1999'''). ''[http://centaur.reading.ac.uk/4564/ KQQKQQ and the Kasparov-World Game]''. [[ICGA Journal#22_4|ICCA Journal, Vol. 22, No. 4]]
 
* [[Eugene Nalimov]], [[Christoph Wirth]], [[Guy Haworth]] ('''1999'''). ''[http://centaur.reading.ac.uk/4564/ KQQKQQ and the Kasparov-World Game]''. [[ICGA Journal#22_4|ICCA Journal, Vol. 22, No. 4]]
 
* [[Ulrich Thiemonds]] ('''1999'''). ''Ein regelbasiertes Spielprogramm für Schachendspiele''. [https://en.wikipedia.org/wiki/University_of_Bonn University of Bonn], Diplom thesis,  [https://www.idb.uni-bonn.de/publications/da/da_thiemonds_1999.pdf pdf] (German)  
 
* [[Ulrich Thiemonds]] ('''1999'''). ''Ein regelbasiertes Spielprogramm für Schachendspiele''. [https://en.wikipedia.org/wiki/University_of_Bonn University of Bonn], Diplom thesis,  [https://www.idb.uni-bonn.de/publications/da/da_thiemonds_1999.pdf pdf] (German)  
Line 423: Line 446:
 
* [[Karsten Müller]], [[Guy Haworth]] ('''2019'''). ''FinalGen revisited: new discoveries''. [[ICGA Journal#41_1|ICGA Journal, Vol. 41, No. 1]] » [[FinalGen]]
 
* [[Karsten Müller]], [[Guy Haworth]] ('''2019'''). ''FinalGen revisited: new discoveries''. [[ICGA Journal#41_1|ICGA Journal, Vol. 41, No. 1]] » [[FinalGen]]
 
* [[Karsten Müller]], [[Guy Haworth]] ('''2019'''). ''Stalemate and ‘DTS’ depth to stalemate endgame tables''. [[ICGA Journal#41_4|ICGA Journal, Vol. 41, No. 4]]  » [[Stalemate]]
 
* [[Karsten Müller]], [[Guy Haworth]] ('''2019'''). ''Stalemate and ‘DTS’ depth to stalemate endgame tables''. [[ICGA Journal#41_4|ICGA Journal, Vol. 41, No. 4]]  » [[Stalemate]]
 +
==2020 ...==
 +
* [[Guy Haworth]] ('''2020'''). ''CEN: Thomas Ströhlein’s Endgame Tables, a 50th Anniversary''. [[ICGA Journal#42_23|ICGA Journal, Vol. 42, Nos. 2-3]]
 +
* [[Rejwana Haque]], [[Ting Han Wei]], [[Martin Müller]] ('''2021'''). ''On the Road to Perfection? Evaluating Leela Chess Zero Against Endgame Tablebases''. [[Advances in Computer Games 17]]
 +
* [[Dave Gomboc]], [[Christian R. Shelton]] ('''2021'''). ''Chess endgame compression via logic minimization''. [[Advances in Computer Games 17]]
  
 
=Forum Posts=  
 
=Forum Posts=  
Line 454: Line 481:
 
* [http://www.open-aurec.com/wbforum/viewtopic.php?f=4&t=4307 EGTB compression] by [[Stef Luijten]], [[Computer Chess Forums|Winboard Forum]], February 10, 2006
 
* [http://www.open-aurec.com/wbforum/viewtopic.php?f=4&t=4307 EGTB compression] by [[Stef Luijten]], [[Computer Chess Forums|Winboard Forum]], February 10, 2006
 
* [http://kirill-kryukov.com/chess/discussion-board/viewtopic.php?f=6&t=2759 utility for testing tablebases integrity] by [[Victor Zakharov]], [[Computer Chess Forums|CCRL Discussion Board]], October 24, 2007
 
* [http://kirill-kryukov.com/chess/discussion-board/viewtopic.php?f=6&t=2759 utility for testing tablebases integrity] by [[Victor Zakharov]], [[Computer Chess Forums|CCRL Discussion Board]], October 24, 2007
 +
* [http://kirill-kryukov.com/chess/discussion-board/viewtopic.php?f=6&t=2923 Chris Wirth's endgame tablebases] by [[Denis Mendoza|Denis P. Mendoza]], [[Computer Chess Forums|CCRL Endgame Tablebases]], December 13, 2007 » [[Chessterfield]]
 
* [http://www.talkchess.com/forum/viewtopic.php?t=19178 EGTB compression (H. G. Muller tablebase...)] by [[Dann Corbit]], [[CCC]], January 25, 2008
 
* [http://www.talkchess.com/forum/viewtopic.php?t=19178 EGTB compression (H. G. Muller tablebase...)] by [[Dann Corbit]], [[CCC]], January 25, 2008
 
* [http://kirill-kryukov.com/chess/discussion-board/viewtopic.php?f=6&t=3220 Endgame Metrics] by [[Joshua Shriver]], [[Computer Chess Forums|CCRL Discussion Board]], March 04, 2008  
 
* [http://kirill-kryukov.com/chess/discussion-board/viewtopic.php?f=6&t=3220 Endgame Metrics] by [[Joshua Shriver]], [[Computer Chess Forums|CCRL Discussion Board]], March 04, 2008  
Line 461: Line 489:
 
* [http://www.talkchess.com/forum/viewtopic.php?t=25546 Creating EGTBs] by [[Carey Bloodworth|Carey]], [[CCC]], December 21, 2008
 
* [http://www.talkchess.com/forum/viewtopic.php?t=25546 Creating EGTBs] by [[Carey Bloodworth|Carey]], [[CCC]], December 21, 2008
 
* [http://www.talkchess.com/forum/viewtopic.php?t=26184 Other endgame tablebase generators] by [[Denis Mendoza|Denis P. Mendoza]], [[CCC]], January 23, 2009
 
* [http://www.talkchess.com/forum/viewtopic.php?t=26184 Other endgame tablebase generators] by [[Denis Mendoza|Denis P. Mendoza]], [[CCC]], January 23, 2009
 +
* [http://kirill-kryukov.com/chess/discussion-board/viewtopic.php?f=6&t=4219 Other endgame tablebase generators] by [[Denis Mendoza|Denis P. Mendoza]], [[Computer Chess Forums|CCRL Endgame Tablebases]], January 28, 2009
 
==2010 ...==  
 
==2010 ...==  
 
* [http://kirill-kryukov.com/chess/discussion-board/viewtopic.php?f=6&t=5163 KK partitioning+very cheap compression fit in 8 GB (7-man)] by [[Urban Koistinen]], [[Computer Chess Forums|CCRL Discussion Board]], May 30, 2010
 
* [http://kirill-kryukov.com/chess/discussion-board/viewtopic.php?f=6&t=5163 KK partitioning+very cheap compression fit in 8 GB (7-man)] by [[Urban Koistinen]], [[Computer Chess Forums|CCRL Discussion Board]], May 30, 2010
Line 530: Line 559:
 
* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=73481 Using Freeware AI and Dynamically Generated Endgame Tablebases] by Steve Schooler, [[CCC]], March 27, 2020
 
* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=73481 Using Freeware AI and Dynamically Generated Endgame Tablebases] by Steve Schooler, [[CCC]], March 27, 2020
 
* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=73598 Almost perfect DTM tablebase] by [[Dann Corbit]], [[CCC]], April 08, 2020
 
* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=73598 Almost perfect DTM tablebase] by [[Dann Corbit]], [[CCC]], April 08, 2020
 +
* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=76089 Asynchronous tablebase lookups] by [[Steinar H. Gunderson|Sesse]], [[CCC]], December 17, 2020 <ref>[https://kernel.dk/io_uring.pdf?source=techstories.org io_uring.pdf]</ref>
 +
'''2021'''
 +
* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=76659 Stalemate Tablebases] by Dries De Clercq, [[CCC]], February 21, 2021 » [[Stalemate]]
 +
* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=77249 knbk DTM] by [[Martin Sedlak]], [[CCC]], May 05, 2021 » [[KBNK Endgame]], [[#DTM|DTM]]
 +
: [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=77249&start=4 Re: knbk DTM] by [[Harm Geert Muller]], [[CCC]], May 05, 2021 » [[Marcel van Kervinck|Marcel van Kervinck's]] pretty fast kbnk generator
 +
* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=77899 tablebase neural nets] by [[Robert Pope]], [[CCC]], August 07, 2021 » [[Neural Networks]]
 +
'''2022'''
 +
* [https://talkchess.com/forum3/viewtopic.php?f=7&t=79938 Definite occurance ranking of 7-Man EGTB] by [[Daniel Infuehr]], [[CCC]], May 24, 2022
 +
* [https://talkchess.com/forum3/viewtopic.php?f=7&t=80522 Fathom, munmap issue] by [[Pawel Osikowski]], [[CCC]], August 19, 2022
 +
* [https://talkchess.com/forum3/viewtopic.php?f=2&t=80608 Are tablebases useless for Stockfish15?] by [[Jouni]], [[CCC]], September 02, 2022
 +
* [https://talkchess.com/forum3/viewtopic.php?f=7&t=80696 endgame table generation] by [[Dave Gomboc]], [[CCC]], September 17, 2022
 +
 +
'''2023'''
 +
* [https://talkchess.com/forum3/viewtopic.php?f=7&t=82115 very strange behavior with syzygy tables] by [[Carbec]], [[CCC]], June 02, 2023
 +
* [https://talkchess.com/forum3/viewtopic.php?f=7&t=82613 EGTB encoding] by [[Edmund]], [[CCC]], September 18, 2023
 +
* [https://talkchess.com/forum3/viewtopic.php?f=2&t=82937 Need a tip for 6-man syzygy ...] by [[Frank Quisinsky]], [[CCC]], December 02, 2023
 +
 +
'''2024'''
 +
* [https://banksiagui.com/forums/viewtopic.php?p=650#p650 Progress reports] by [[Pham Hong Nguyen|Nguyen Pham]], April 16, 2024
 +
* [https://talkchess.com/viewtopic.php?t=83676 Felicity EGTB generators and probers] by [[Pham Hong Nguyen|Nguyen Pham]], [[CCC]], April 20, 2024
 +
  
 
=External Links=  
 
=External Links=  
Line 567: Line 617:
  
 
==Online Lookup==  
 
==Online Lookup==  
* [https://www.chessdb.cn/queryc_en/ Chess Cloud Database Query Interface] by [[noobpwnftw]] [[Syzygy Bases|Syzygy]] DTZ50 probing
+
* [https://www.chessdb.cn/queryc_en/ Chess Cloud Database Query Interface] by [[Bojun Guo|noobpwnftw]] [[Syzygy Bases|Syzygy]] DTZ50 probing
 
* [http://tb7.chessok.com/ Lomonosov - online 7-man tablebases]
 
* [http://tb7.chessok.com/ Lomonosov - online 7-man tablebases]
 
* [http://chessok.com/?page_id=361 Endgame Nalimov Tablebases Online] by [[ChessOK]]
 
* [http://chessok.com/?page_id=361 Endgame Nalimov Tablebases Online] by [[ChessOK]]

Latest revision as of 09:55, 17 July 2024

Home * Knowledge * Endgame Tablebases

Samuel Bak - Persistence [1]

Endgame Tablebases, (EGTBs or EGTs)
precalculated endgame tables generated by an exhaustive retrograde analysis. During its game play and/or search, recognizing a specific material composition, a chess program can probe, or in principle compute these tables to determine the outcome of positions definitively and act as an oracle providing the optimal moves. The tables are usually divided into separate files for each material configuration and side to move. What all different formats of tablebases have in common is that every possible piece placement has its own, unique index number which represents the position where the information about the position is located in the file. Positions with non-null castling rights could be represented in an EGTB but typically are not.

The term Tablebase opposed to endgame database was coined by Steven Edwards by means of a data vector and single file access for both sides to move, when he introduced his Edwards' Tablebases, addressing several shortcomings of Ken Thompson's earlier database of positions with the weaker and hence potentially losing side to move only [2] .

Purposes

When a chess game goes to the endgame phase, it may complete soon with the result as a draw or a win/loss. Both human/chess engines can finish the endgame by continuing searching, and applying some endgame rules. Even the game can finish in this way but it is not in a perfect way: it takes time to compute moves and may use more moves than necessary. Sometimes the winning side may lose the change since the rule 50 moves and/or makes blunders. Both human and chess engines may miss the chances badly if they don’t know or not good enough about the endgame they are playing.

An EGTB can help to solve this period. It is a kind of dictionary of all endgame positions that can answer immediately for a given position:

  • If it is a draw or a win/loss
  • How far/how many move from matting/being mated
  • What is the best move

The main advantages of using EGTBs:

  • Engines don’t have to search but retrieve data thus they can move instantly
  • Moves are the perfect ones, leading the shortest line to win
  • Engines don’t need to have the knowledge and code about those endgames

The main disadvantages of using EGTBs:

  • Their data typically are too large for downloading and storing
  • They may slow down the search
  • Require complicated codes for probing

Data

An EGTB is a set of endgames. Each endgame is a set of records about positions’ information. All involved positions in an endgame must have the same material. Typically each position's record is associated with an integer number which informs how far that position is from mating/being mated or converting (depends on the type of its metrics).

Based on values of those integer numbers it can answer directly two questions of the EGTB: 1) for a given position it is a draw or a win/loss position and 2) how far it is from mating/being mated or converting. To answer the 3rd question, the best move of a position can be indirectly calculated: generate all legal moves, make them, probe all new positions, compare probing scores for the highest one and the associated move is the best one.

Theoretically, we can add more information to each position’s record. However, due to the large number of involved positions, any additional information may make the whole EGTB become significantly larger. Thus so far none of the popular EGTBs store any extra information and their records actually contain those integers only.

All records of an endgame are simply organized as two arrays (one array for one side/colour). In other words, each item in those arrays is an integer and its index on an array is mapped to a unique position. For a given chess position, we will calculate its index and then access its data record/integer via that index without searching.

Endgame files

Those arrays of an endgame can be stored in files. Typically each array is stored in one file. However, sometimes they can be combined into one larger file or split into more files to easier copy or manage. Those files may be in compressed or uncompressed forms. If they are compressed, they are typically divided and compressed by small blocks thus they don't need to load and decompress all data to extract just a position.

In memory

Some chess engines create themselves EGTBs on-flight and keep all data in memory without saving/loading to files. Due to the limit by the initial time they can generate and use very small EGTBs only.

Sizes

The size of an endgame depends on:

  • The number of involving pieces: more pieces, the bigger size
  • Type of pieces: for standard chess, more Pawns is the smaller size
  • Type of metric
  • The largest value: the integer number should be large enough to cover the largest value. Different endgames will have different largest values. Thus some endgames need only one byte per item but some need 2 bytes
  • Indexing: The algorithms to map between an array item and a position
  • Compress ratio

So far the size of an EGTB is the most important factor of success.

Metrics

DTM

Depth to Mate 'Mate' is the ultimate goal and ends the game. For each position that is represented, 'DTM' indicates the theoretical value, and the number of winner's moves to 'Mate' if won/lost - assuming that the winner is minimizing and the loser is maximizing DTM.

DTC

Depth to Conversion 'Conversion' is the changing of the force on the board, by capture and/or Pawn-conversion. For each position that is represented, 'DTC' indicates the theoretical value, and the number of winners moves to the goal of 'Won Conversion' if won/lost - assuming that the winner is minimizing and the loser is maximizing DTC. DTC £ DTM.

DTZ

Depth to Zeroing (of the ply count) The ply-count is zeroed if and only if a Pawn is pushed or a capture is made. For each position that is represented, 'DTZ' indicates the theoretical value, and the number of winners moves to the goal of 'Win and ply-count zeroed' if won/lost - assuming that the winner is minimizing and the loser is maximizing DTZ. DTZ £ DTC £ DTM.

DTZ50

Depth to Zeroing (of the ply count) in the context of the 50-Move Rule The DTC, DTM and DTZ metrics do not consider the current FIDE 50-move rule whereby a game in which no irreversible moves have been played for 100 plies can be claimed as a draw by the player 'on move'. In this metric, in addition to the DTC/DTM/DTZ draws, positions requiring more than 50 winner's moves to zero the ply-count are defined as draws. DTZ £ DTZ50 if DTZ50 ¹ draw. 'DTZ50' is one of a set of 'DTZk' rules: a k-move rule effectively applies if there are only k moves left before a draw-claim might be made. k1 £ k2 implies DTZk2 £ DTZk1 if DTZk1 ¹ draw.

Note that, because depth is measured in winner's moves rather than plies, this metric does not define as draw a position in which 50 winner's moves and 51 loser's moves are required to zero the ply count, despite the fact that the loser would claim a draw after 100 plies had been played in that endgame.

DTR

Depth by the (k-move) Rule 'The Rule' referred to here is a notional set of k-move rules. For each position that is represented, 'DTR' indicates the theoretical value and, for won/lost positions, the least k for which DTZk is not draw - assuming that the winner is minimising and the loser is maximising DTR. DTZ £ DTR. No DTR EGTs have been created to date [3].

DTZR

Depth to Zeroing move, given that DTR is being minimaxed If DTR = draw, then DTZR = draw. Otherwise, DTRZR indicates the theoretical value, and the number of winners moves to the zeroing of the ply count, given that the two sides are minimaxing first DTR and then DTZR. DTZR would be used in conjunction with DTR to progress a deep win in the context of a k-move rule because DTR can remain unchanged over a sequence of moves. DTZR £ DTR but it is not always the case that DTZ £ DTZR as may seem intuitively obvious: the loser may zero the move-count to maximise DTR.

Summary

To summarize the different metrics of tablebases, each metric has its strengths and weaknesses. Nalimov's DTM EGTs are widely and commercially available, and used by many chess engines or chess GUIs. DTC EGTs are more easily computed and stored as they involve smaller depth ranges and are more compressible. For endgames with less or equal to 5 pieces on the board, KNNKP comes closest to requiring more than one Byte per position to store DTM: maxDTM = 115 (1-0) and 73 (0-1) necessitating 190 values in the EGT. For KPPKP, maxDTM = 127 (1-0) but only 42 (0-1). But for certain 6-piece endgames, one Byte is not enough: maxDTM = 262 for a KRNKNN position. The DTC and DTZ metrics increasingly postpone the need for two Bytes per position in an uncompressed EGT, but KRBNKQN has maxDTC = maxDTZ = 517 (0-1), both wtm and btm.

In extremis, the unmoderated use of DTM, DTC or DTZ EGTs will let slip a winnable position in the context of the 50-move rule ... hence the marginal need for the DTR and DTZR metrics, coupled with the ability to recompute DTR/DTZR EGTs in the context of a limited move-budget.

Bitbases

see main article Endgame Bitbases

Inside the search much denser tables with value {won, draw, loss, invalid} or boolean {win, not_win} information are sufficient with respect to bounds. Denser tables are more efficiently used, given that memory and cache are limited resources. Boolean Win/Not_Win Bitbases require only one bit per position (and if necessary, further enquiry if draw is to be distinguished from loss with respect to bounds). Value Bitbases require two bits per position.

Indexing

Every position is assigned to a unique index to specify the location of the information stored about it. The main purpose of indexing is to locate and read the information (from databases) such as distance to mate for a given position based on its index.

There are two main aims when generating the index number. The easier the algorithm to generate the index from any given position, the faster will be the probing of the tablebase as well as the generation of it. But the second aim is to get a relatively small range of index numbers, to keep the size of the file as small as possible.

The most straightforward way would be to for every piece on the board use 6 bit (1-64 squares)

index = position_white_King + position_black_King * 64

for each piece {
    index = index * 64 + position
}

Almost all work in this field is to reduce index spaces and reduce data sizes (e.g., by compressing). The above indexing can be optimised in several ways.

Remove redundant symmetrical indexes:

  • A chessboard can be rotated and mirrored (if pawns are on the board only a vertical reflection is possible).
  • For pieces of the same type, different arrangements can be left away.

Remove illegal indexes:

  • Pieces may not stand on the same square.
  • The side to move may not give a check.
  • The distance between the two Kings must be greater than one.

Retrieving

Probe

When a chess program (e.g., a chess engine or a chess GUI) works with an EGTB, it needs to retrieve data (integer numbers) for some given chess positions. The process is named probe and has below steps:

  • Convert a given chess position into an index via indexing algorithms
  • Access data (typically organised as arrays of integers) using the index to retrieve the integer of that chess position

If data of that endgame is not ready in memory, usually the probe process has to do some extra work:

  • From the data index calculate the block index
  • Read data of that block from storage into memory
  • Uncompress the block data (if it is compressed)
  • Convert the endgame index into a block index and retrieve the needed data

Search

After probing the chess engine may have the result as an integer number for the queried position. Depending on the metric, the number may be a distance to mate or distance to conversion, the position is a draw or a win or loss for the querying side. Sometimes that value (state of draw/win/loss) may be enough for searching. However, sometimes that state is not enough and it needs to know which is the best move to make. The solution is to generate all legal moves from that position, make them one by one, and probe that EGTB for the values of each new position. After all, compare the values of all children's positions to find which one is the best to make.

Speed

Typically the probe process requires some computing and it may access storage and decompress which is usually so slow. An engine that probes EGTB when searching may make the whole search be slower. Sometimes the benefit from probing EGTBs may not be enough to cover the loss of slower search. That’s why the probing should be planned and implemented carefully. EGTB files may have to be stored in fast storage and/or using some huge systems’ caches. The good point is that the engine can stop searching on the branch probed by the EGTB.

Brief Info

of some popular Endgame tablebases:

Format Metric 1st published 5 men 6 men 7 men
Thompson DTC 1991 2.5 GiB (not completed) - -
Edwards DTM 1994 56 GiB (estimated) - -
Nalimov DTM 1998 7.1 GiB 1.2 TiB -
Scorpio WDL 2005 214 MiB 48.1 GiB -
Gaviota DTM 2008 6.5 GiB - -
Lomonosov DTM 2012 - - 140 TiB
Syzygy WDL + DTZ50 2013 / 2018 939 MiB 150.2 GB 17 TB

7-man EGTBs

2005

In 2005, Yakov Konoval started to collaborate with Marc Bourzutschky on constructing 7-man EGTBs. Yakov wrote the generator and Marc a verification program and some utilities for extracting the data. In October 2005, Yakov Konoval and Marc Bourzutschky announced that a position in the ending of a KRRNkrr requires 290 moves to convert to a simpler winning endgame [4]. The old record was 243 moves from a position in a rook and knight versus two knights endgame, discovered by Lewis Stiller in 1991 [5].

2006

In March 2006, the wizards of 7-men endgames, Marc Bourzutschky and Yakov Konoval found 330 moves win in KQBNkqb [6] and in May 2006 a 517 moves win in KQNkrbn [7].

2012

In July 2012, Vladimir Makhnychev and Victor Zakharov from Moscow State University completed 4+3 DTM-tablebases (525 endings including KPPPKPP). The next set of 5+2 DTM-tablebases was completed in August 2012 [8]. The tablebases are named Lomonosov Tablebases, using the Lomonosov Supercomputer.

2013

Convekta Ltd. announced the release of Lomonosov Tablebases. Until December 31, 2013, the tablebases will be available online through ChessOK products [9].

2018

Starting in March 2018, seven piece Syzygy Bases, originally created for up to six pieces by Ronald de Man in 2013, were generated by Bojun Guo, finished in August 2018 [10].


Quotes

Evert explained pros and cons of some popular EGTBs, March 07, 2018 [11]:

Scorpio and (iirc) Shredder are bitbases: they store whether a position is won for the side to move or not. Typically that is enough during the search, as long as the root position is not a tablebase position. In that case you need a way to make progress, and the bitbases don't really help. However, in practice chess programs can win many endgames without tablebases.
Nalimov and Gaviota store distance to mate (DTM) for the side to move (or not-won if the position is not won), but they ignore draws by the 50-move rule. Nalimov has 6 men, but comes with an obnoxious custom license. Gaviota probing code is open source under a liberal license, but it only has up to 5 men. Compression is (iirc) much better than Nalimov.
Lomonosov is DTM up to 7 pieces, but not available for free (and impractically large).
Syzygy bases store the number of moves needed to win a position under the 50-move rule (DTZ50). You win by either mating the opponent, making a winning pawn push or by converting to another won ending. The generator and probing code are open source.
For practical purposes, if you pick just one, Syzygy is probably the best pick. In a pinch, Syzygy+Gaviota might be better, since they will give you the shortest DTM in positions that are won under the 50-move rule. Syzygy alone doesn't do that. If that is better or not is a matter of taste.
Nalimov should be a no-go because of the license and Lomonosov is simply not an option.
Pure bitbases are probably the least useful these days (Syzygy and I think Gaviota actually already include bitbases for the search).

Formats

Generators

See also

Selected Publications

1965 ...

1970 ...

1975 ...

1985 ...

1986

1987

1988

1989

1990 ...

1991

1992

1993

1994

1995 ...

1996

1997

1999

2000 ...

2001

Guy Haworth (2001). Discarding Like Pieces. 6th Computer Olympiad Workshop
Guy Haworth (2001). Depth by The Rule. 6th Computer Olympiad Workshop
Guy Haworth (2001). 3-5-Man Chess Data. 6th Computer Olympiad Workshop
Guy Haworth (2001). 3-5-Man Mutual Zugzwangs in Chess. 6th Computer Olympiad Workshop

2002

2003

2004

2005 ...

2006

2008

2009

2010 ...

2011

2013

2014

2015 ...

2016

2017

2018

2019

2020 ...

Forum Posts

1995 ...

overlapping tablebase lookup by Jay Scott, CCC, February 09, 1999

2000 ...

Wu/Beal predates Koistinen by Guy Haworth, CCC, December 04, 2001

2005 ...

Re: Endgame Metrics by John Kominek, CCRL Discussion Board, March 12, 2008

2010 ...

2011

2012

2013

2014

2015 ...

2016

Re: Nalimov EGTB problem related to DTM? by Ronald de Man, CCC, February 14, 2016
EPD to test EGTB probe code by Sergei S. Markoff, CCC, May 24, 2016

2017

2018

2019

2020 ...

2021

Re: knbk DTM by Harm Geert Muller, CCC, May 05, 2021 » Marcel van Kervinck's pretty fast kbnk generator

2022

2023

2024


External Links

Compression Schemes for Gaviota Tablebases

GitHub

ChessBase

ChessOK

Online Lookup

Download

Statistics

References

  1. Chess in the Art of Samuel Bak, Center for Holocaust & Genocide Studies, University of Minnesota
  2. Ernst A. Heinz (1999). Endgame Databases and Efficient Index Schemes for Chess. ICCA Journal, Vol. 22, No. 1, ps
  3. DTR is no good! by Harm Geert Muller, CCRL Forum, September 24, 2008
  4. Subject: KRRNKRR win in 290: a new record by Marc Bourzutschky from CCC, October 16, 2005
  5. Chess endgame from Wikipedia
  6. 311. 31 March 2006: White plays and wins in 330 moves by Tim Krabbé
  7. OPEN CHESS DIARY 316. 26 May 2006: A 517-move win by Tim Krabbé
  8. Endgame tablebase from Wikipedia - Background
  9. Lomonosov Endgame Tablebases - ChessOK
  10. Re: 7-men Syzygy attempt by Bojun Guo, CCC, August 19, 2018
  11. Re: The history of Syzygy tablebases by Evert, CCC, March 07, 2018
  12. Subject: brute-force vs. intuition in math & chess by Bill Dubuque, August 1996
  13. Jesper Torp Kristensen (2005). Generation and compression of endgame tables in chess with fast random access using OBDDs. Master thesis, supervisor Peter Bro Miltersen, Aarhus University
  14. Knowledge discovery from Wikipedia
  15. Chris Wirth's endgame tablebases by Denis P. Mendoza, CCRL Endgame Tablebases, December 13, 2007 » Chessterfield
  16. Kasparov versus the World from Wikipedia
  17. Generating egtbs ICGAJ by Tony Werten, CCC, December 04, 2001, with reference to Computing endgames with few men by Urban Koistinen
  18. Wu / Beal retrograde analisys algorithm by Alvaro Jose Povoa Cardoso, Winboard Forum, March 10, 2007
  19. OBDD - Ordered Binary Decision Diagram from Wikipedia
  20. Plakoto from Wikipedia
  21. Endgame tablebases with the fifty-move rule by Galen Huntington
  22. tablebase compression / academic integrity by Ronald de Man, CCC, May 19, 2016
  23. Computing endgames with few men by Urban Koistinen
  24. Ren Wu, Don Beal (2001). Fast, Memory-efficient Retrograde Algorithms. ICGA Journal, Vol. 24, No. 3
  25. Computing endgames with few men by Urban Koistinen
  26. Easy mate by Frank Phillips, CCC, September 13, 2003 » Checkmate
  27. A018214 - OEIS | Alkane (or paraffin) numbers
  28. Lomonosov Endgame Tablebases - ChessOK
  29. kervinck/pfkpk · GitHub
  30. io_uring.pdf
  31. Gaviota tablebases, probing code v4 (UPDATE) by Miguel A. Ballicora, CCC, March 11, 2011
  32. EGTB: Better algorithm by Urban Koistinen, CCC, April 07, 2001
  33. Generating egtbs ICGAJ by Tony Werten, CCC, December 04, 2001
  34. Ren Wu, Don Beal (2001). Fast, Memory-efficient Retrograde Algorithms. ICGA Journal, Vol. 24, No. 3
  35. New 6-piece tablebases by Ronald de Man, CCC, April 01, 2013
  36. Scorpio 6men EGBB Now available by Joshua Shriver, CCC, January 14, 2014
  37. Yet another KPK endgame table generator: pfkpk by Marcel van Kervinck, CCC, September 05, 2015

Up one level