Difference between revisions of "Syzygy Bases"

From Chessprogramming wiki
Jump to: navigation, search
(Created page with "'''Home * Knowledge * Endgame Tablebases * Syzygy Bases''' FILE:Three Planets Dance Over La Silla.jpg|border|right|thumb|Syzygy - Three Planets Dance...")
 
(2020 ...)
 
(116 intermediate revisions by 4 users not shown)
Line 4: Line 4:
  
 
'''Syzygy Bases''', <ref>[http://www.talkchess.com/forum/viewtopic.php?t=47681&start=45 Re: New 6-piece tablebases] by [[Ronald de Man]], [[CCC]], April 10, 2013</ref><br/>
 
'''Syzygy Bases''', <ref>[http://www.talkchess.com/forum/viewtopic.php?t=47681&start=45 Re: New 6-piece tablebases] by [[Ronald de Man]], [[CCC]], April 10, 2013</ref><br/>
a compact six piece endgame database developed by [[Ronald de Man]], published April 01, 2013, but not as [https://en.wikipedia.org/wiki/April_Fools%27_Day April Fool]. Syzygy Bases consist of two sets of files, '''WDL''' files storing win/draw/loss information considering the [[Fifty-move Rule|fifty-move rule]] for access during [[Search|search]], and '''DTZ''' files with [[Endgame Tablebases#DTZ50|distance-to-zero]] information for access at the [[Root|root]]. The generator is released under the [[Free Software Foundation#GPL|GNU General Public License Version 2]], the [[Thread|thread safe]] probing code is released without restrictions <ref>[http://www.talkchess.com/forum/viewtopic.php?t=47681 New 6-piece tablebases] by [[Ronald de Man]], [[CCC]], April 01, 2013</ref> . As of February 2015, all 3-5 and most important 6-men Syzygy Bases are commercially available on 4 [https://en.wikipedia.org/wiki/DVD DVDs] by [[ChessBase]] as ''Endgame Turbo 4'' for their products [[Fritz|Deep Fritz 14]], [[Komodo#8|Komodo Chess 8]], [[Houdini|Houdini 4]] or [[ChessBase (Database)|ChessBase 12/13]] <ref>[http://shop.chessbase.com/en/products/endspiel_turbo_4 Endgame Turbo 4] by [[ChessBase]]</ref> <ref>[http://en.chessbase.com/post/syzygy-tablebases-newest-fastest-smallest Syzygy Tablebases: newest, fastest, smallest] by [[Albert Silver]], [[ChessBase|ChessBase News]], February 08, 2015</ref> .  
+
a compact six piece endgame database developed by [[Ronald de Man]], published on April 01, 2013. Since August 2018, '''seven piece''' Syzygy Bases are available after an effort by [[Bojun Guo]] started in March 2018 <ref>[http://talkchess.com/forum3/viewtopic.php?f=7&t=66797&start=472 Re: 7-men Syzygy attempt] by [[Bojun Guo]], [[CCC]], August 19, 2018 </ref>. The generator is released under the [[Free Software Foundation#GPL|GNU General Public License Version 2]], the [[Thread|thread safe]] probing code is released without restrictions <ref>[http://www.talkchess.com/forum/viewtopic.php?t=47681 New 6-piece tablebases] by [[Ronald de Man]], [[CCC]], April 01, 2013</ref> .
  
=File Sizes=  
+
=Data=
 +
==File Types==
 +
Syzygy Bases consist of two sets of files, '''WDL''' files (extension .rtbw) storing win/draw/loss information considering the [[Fifty-move Rule|fifty-move rule]] for access during [[Search|search]], and '''DTZ''' files (extension .rtbz) with [[Endgame Tablebases#DTZ50|distance-to-zero]] information for access at the [[Root|root]]. WDL has full data for two sides but DTZ50 omitted data of one side to save space. Each endgame has a pair of those types.
 +
 
 +
==File Sizes==  
 
{| class="wikitable"
 
{| class="wikitable"
 
|-
 
|-
Line 12: Line 16:
 
! WDL  
 
! WDL  
 
! DTZ  
 
! DTZ  
 +
! Total
 
|-
 
|-
! 5  
+
! 3-5  
| style="text-align:right;" | 378 MiB  
+
| style="text-align:right;" | 378.1 [https://en.wikipedia.org/wiki/Mebibyte MiB]
| style="text-align:right;" | 561 MiB  
+
| style="text-align:right;" | 561.9 MiB
 +
| style="text-align:right;" | 939.0 MiB
 
|-
 
|-
 
! 6  
 
! 6  
| style="text-align:right;" | 68.3 GiB  
+
| style="text-align:right;" | 67.8 [https://en.wikipedia.org/wiki/Gibibyte GiB]
| style="text-align:right;" | 81.9 GiB  
+
| style="text-align:right;" | 81.4 GiB
 +
| style="text-align:right;" | 149.2 GiB
 +
|-
 +
! 7
 +
| style="text-align:right;" | 8.5 [https://en.wikipedia.org/wiki/Tebibyte TiB]
 +
| style="text-align:right;" | 8.3 TiB
 +
| style="text-align:right;" | 16.7 TiB
 
|}
 
|}
  
The additional use of [https://en.wikipedia.org/wiki/Haswell_%28microarchitecture%29 Haswell's] [[BMI2]] [[BMI2#PDEP|Parallel Bits Deposit (PDEP)]] and [[BMI2#PEXT|Parallel Bits Extract (PEXT)]] instructions, or their slower serial implementations, allows further size reduction <ref>[http://www.talkchess.com/forum/viewtopic.php?t=48220&start=1 Re: PEXT Bitboards] by [[Ronald de Man]], [[CCC]], June 07, 2013</ref> .
+
==Comparision==
 +
Syzygy EGTB is significantly smaller than any existent [[Endgame Tablebases#DTM|DTM]] EGTB. It is 7 times as small as Gaviota for 5 men, 8 times as small as Nalimov for 6 men, 8 times as small as Lomonosov for 7 men. However, when all DTM EGTBs have full data of two sides, Syzygy EGTB omits data of one side for DTZ data to save space. [[Ronald de Man]] estimated if keep them all, the Syzygy's 6 men size may increase 158 GB, become 307 GB in total, double on size, be 4 times as small as Nalimov 6 men.
 +
 
 +
=Generation=
 +
==Up to 6-man==
 +
On the first release (Apr 01, 2013) the generator was ready to generate all endgames up to 6 men. It is multithreading and processes completely in RAM. To generate all 6 men, it requires a system with at least 32 GB of RAM and may run in 5 days (the period was measured with a computer 6-core i3930K @ 4.2Ghz, 64 GB).
  
Generation of 7-men tablebases would require about 1 TB of RAM. Generation time will be about 64 x per table, which means around 175 x time total <ref>[http://kirill-kryukov.com/chess/discussion-board/viewtopic.php?f=6&t=7396#p83338  Re: Syzygy tb generator for windows] by [[Ronald de Man|syzygy]], [[Computer Chess Forums|CCRL Discussion Board]], June 01, 2014</ref> . Ronald de Man isn't interested in creation of 7-men Syzygy <ref>[http://kirill-kryukov.com/chess/discussion-board/viewtopic.php?f=6&t=7618#p83771 Re: 7-piece syzygy] by [[Ronald de Man|syzygy]], [[Computer Chess Forums|CCRL Discussion Board]], July 03, 2014</ref> .
+
==7-man==
 +
Ronald de Man wasn't initially interested in the creation of 7-men Syzygy Based <ref>[http://kirill-kryukov.com/chess/discussion-board/viewtopic.php?f=6&t=7618#p83771 Re: 7-piece syzygy] by [[Ronald de Man|syzygy]], [[Computer Chess Forums|CCRL Discussion Board]], July 03, 2014</ref> since the generation would require about 1 TB of RAM, too expensive at that time. Generation time would be about 64 x per table, which means around 175 x time total <ref>[http://kirill-kryukov.com/chess/discussion-board/viewtopic.php?f=6&t=7396#p83338  Re: Syzygy tb generator for windows] by [[Ronald de Man|syzygy]], [[Computer Chess Forums|CCRL Discussion Board]], June 01, 2014</ref>.His original generator could not create them. But in 2018, he supported [[Bojun Guo]] in his 5-month attempt to generated them <ref>[http://www.talkchess.com/forum/viewtopic.php?t=66797 7-men Syzygy attempt] by [[Bojun Guo]], [[CCC]], March 10, 2018</ref>. His hardware was estimated at over US$ 90K. In August of 2018 their creation was completed <ref>[https://syzygy-tables.info/ Powered by Ronald de Man's Syzygy endgame tablebases, 7-piece tables generated by Bojun Guo and a public API hosted by lichess.org], August 19, 2018</ref><ref>[http://tablebase.lichess.ovh/tables/standard/7/ Index of /tables/standard/7/ on lichess], August 19, 2018</ref>.
  
=During the Search=  
+
==8-man==
 +
After the completion of 7-man, many people start being curious about the feasibleness of building 8-man. Ronald de Man made an estimation that task requires computers with 64 TB RAM and 2000 TB hard disks<ref>[http://talkchess.com/forum3/viewtopic.php?f=2&t=74185&sid=016570559fa97e785404dd65020c0ad6&start=20#p848245 Re: What is the best way to obtain the 7-piece tablebases?] by [[Ronald de Man|syzygy]], [[CCC]], June 22, 2020</ref> (costed about $640K and $40K respectively in 2020 <ref>[http://talkchess.com/forum3/viewtopic.php?t=74185&start=30#p848394 Re: What is the best way to obtain the 7-piece tablebases?] by [[Dann Corbit]], [[CCC]], June 23, 2020</ref>).
 +
 
 +
==Checksums==
 +
Syzygy endgame files may contain 128-bit checksum keys at the end of those files. It also has its own code for checksums (based on Google's cityhash library).
 +
 
 +
=Search=
 +
==During the Search==  
 
During the [[Search|search]], with the WDL tables stored on [https://en.wikipedia.org/wiki/Solid-state_drive SSD] <ref>[http://www.talkchess.com/forum/viewtopic.php?t=47931&start=2 Re: SSD and the use of Tablebases] by [[Ronald de Man]], [[CCC]], May 08, 2013</ref> , it is possible to probe the tables at all [[Depth|depths]] without much slowdown. They have been tested in Ronald de Man's engine [[Sjaak]] (playing on [[FICS]] as TrojanKnight(C)) a couple of months quite successfully, don't probing in [[Quiescence Search|quiescence search]].
 
During the [[Search|search]], with the WDL tables stored on [https://en.wikipedia.org/wiki/Solid-state_drive SSD] <ref>[http://www.talkchess.com/forum/viewtopic.php?t=47931&start=2 Re: SSD and the use of Tablebases] by [[Ronald de Man]], [[CCC]], May 08, 2013</ref> , it is possible to probe the tables at all [[Depth|depths]] without much slowdown. They have been tested in Ronald de Man's engine [[Sjaak]] (playing on [[FICS]] as TrojanKnight(C)) a couple of months quite successfully, don't probing in [[Quiescence Search|quiescence search]].
  
=At the Root=  
+
==At the Root==  
At the [[Root|root]], since pure DTZ50-optimal play (i.e. minimaxing the number of moves to the next [[Captures|capture]] or [[Pawn Push|pawn move]] by either side) can be very unnatural, it might be desirable to let the engine search on the winning moves until it becomes clear that insufficient progress is being made and only then switch to DTZ-optimal play (e.g. by detecting [[Repetitions|repetitions]] and monitoring the [[Halfmove Clock|halfmove clock]]) <ref>[http://www.talkchess.com/forum/viewtopic.php?t=47681&start=8 Re: New 6-piece tablebases] by [[Ronald de Man]], [[CCC]], April 05, 2013</ref> .
+
At the [[Root|root]], since pure DTZ50-optimal play (i.e. minimaxing the number of moves to the next [[Captures|capture]] or [[Pawn Push|pawn move]] by either side) can be very unnatural, it might be desirable to let the engine search on the winning moves until it becomes clear that insufficient progress is being made and only then switch to DTZ-optimal play (e.g. by detecting [[Repetitions|repetitions]] and monitoring the [[Halfmove Clock|halfmove clock]]) <ref>[http://www.talkchess.com/forum/viewtopic.php?t=47681&start=8 Re: New 6-piece tablebases] by [[Ronald de Man]], [[CCC]], April 05, 2013</ref>.
  
=Quotes=  
+
=Pros & Cons=
 +
==Pros==
 +
* Small sizes. It is about 8 times as small as the second-best EGTBs. Having small sizes is the main success key of Syzygy Bases since it is much easier to create, store, provide, download  than other EGTBs
 +
* Free and more popular (than other EGTBs) to find on the Internet
 +
* Support [[Endgame Tablebases#DTZ50|DTZ50]] metric. That metric can help engines to have better results than [[Endgame Tablebases#DTM|DTM]] which is supported widely by other EGTBs
 +
 
 +
==Cons==
 +
* Hard to integrate with chess engines. [[Ronald de Man]] has not provided probing code as an independent library but as a part of [[Stockfish]] chess engines. The probe code has integrated too deeply with that chess engine code and it requires a lot of effort to de-integrate, modify and integrate with other chess projects
 +
* Hard to understand and contribute to the project. ETGB itself is a hard topic. Syzygy EGTB has also integrated with many advanced techniques/tricks. It is written in old-style C language. All make it become very hard to understand and/or modify to improve or for other purposes
 +
* [[Endgame Tablebases#DTZ50|DTZ50]]  metric may lead the engines to win in much longer and ugly ways, compared with [[Endgame Tablebases#DTM|DTM]] one
 +
 
 +
=Data publish=
 +
[[Ronald de Man]] has provided only open source code for generators/probers but not endgame files themselves. Using his tools some people have generated endgames and published them via DVD or online.
 +
 
 +
==DVD==
 +
As of February 2015, all 3-5 and most important 6-men Syzygy Bases are commercially available on 4 [https://en.wikipedia.org/wiki/DVD DVDs] by [[ChessBase]] as ''Endgame Turbo 4'' for their products [[Fritz|Deep Fritz 14]], [[Komodo#8|Komodo Chess 8]], [[Houdini|Houdini 4]] or [[ChessBase (Database)|ChessBase 12/13]] <ref>[http://shop.chessbase.com/en/products/endspiel_turbo_4 Endgame Turbo 4] by [[ChessBase]]</ref> <ref>[http://en.chessbase.com/post/syzygy-tablebases-newest-fastest-smallest Syzygy Tablebases: newest, fastest, smallest] by [[Albert Silver]], [[ChessBase|ChessBase News]], February 08, 2015</ref>.
 +
 
 +
==Free Download==
 +
There are some free FTP servers for downloading such as [[Bojun Guo]] and [[Lichess]] servers.
 +
 
 +
===3-6 men===
 +
The sizes of those men are small enough to download and store on modern computers. Users should download them in full sets (3, 4, 5, 6 men).
 +
 
 +
===7 men===
 +
All 7 men files' size is over 16.7 TiB, over storages of typical modern computers. They also require a long time to download too. Thus some users choose to download one or a few endgames only, based on their statistics of use in endgames. Bellow is the top 20 of those endgames by their order. The first one, KRPPvKRP, has a significantly higher frequency of use than the others and should be always downloaded:
 +
 
 +
{| class="wikitable"
 +
|-
 +
! Rank
 +
! Name
 +
! Rank
 +
! Name
 +
! Rank
 +
! Name
 +
! Rank
 +
! Name
 +
|-
 +
! 1
 +
| style="text-align:right;" | KRPPvKRP
 +
! 6
 +
| style="text-align:right;" | KNPPvKNP
 +
! 11
 +
| style="text-align:right;" | KBPPvKNP
 +
! 16
 +
| style="text-align:right;" | KRBPPvKR
 +
|-
 +
! 2
 +
| style="text-align:right;" | KBPPvKBP
 +
! 7
 +
| style="text-align:right;" | KNPPvKBP
 +
! 12
 +
| style="text-align:right;" | KRPPvKRB
 +
! 17
 +
| style="text-align:right;" | KBPPvKPP
 +
|-
 +
! 3
 +
| style="text-align:right;" | KPPPvKPP
 +
! 8
 +
| style="text-align:right;" | KRBPvKRP
 +
! 13
 +
| style="text-align:right;" | KRPPvKPP
 +
! 18
 +
| style="text-align:right;" | KRPPPvKP
 +
|-
 +
! 4
 +
| style="text-align:right;" | KRPPPvKR
 +
! 9
 +
| style="text-align:right;" | KQPPvKPP
 +
! 14
 +
| style="text-align:right;" | KBPPvKRP
 +
! 19
 +
| style="text-align:right;" | KRBPvKRB
 +
|-
 +
! 5
 +
| style="text-align:right;" | KQPPvKQP
 +
! 10
 +
| style="text-align:right;" | KQPPPvKP
 +
! 15
 +
| style="text-align:right;" | KRNPvKRP
 +
! 20
 +
| style="text-align:right;" | KRPPvKRN
 +
|}
 +
 
 +
=Probe Code and Tools=
 +
==Stockfish==
 +
[[Ronald de Man]] did not provide the probe code as an independent library. Instead, he published it firstly as an already integrated code for [[Stockfish]] chess engines. It is c++ code and it has been rewritten, updated several times by [[Stockfish]] team.
 +
 
 +
==Fathom==
 +
'''Fathom''' is a stand-alone Syzygy based probing tool and [https://en.wikipedia.org/wiki/Application_programming_interface API] by [[Basil Falcinelli]], introduced in November 2015 along with his [[Gull|Gull 3]] release <ref>[http://www.talkchess.com/forum/viewtopic.php?t=58299 Gull 3 Linux+Syzygy and Fathom released] by [[Basil Falcinelli]], [[CCC]], November 20, 2015</ref> . Unlike the original tbprobe code, Fathom does not necessarily require the callee to provide [[Move Generation|move generation]] functionality. The new modifications and extensions to Ronald de Man's original code which can be "redistributed and/or modified without restrictions", are released under the permissive [[Massachusetts Institute of Technology#License|MIT License]]. The API consists of three functions <ref>[https://github.com/basil00/Fathom GitHub - basil00/Fathom: Syzygy TB probe tool] by [[Basil Falcinelli]]</ref> :
 +
* tb_init initializes the tablebase
 +
* tb_probe_wdl probes the Win-Draw-Loss (WDL) table for a given position
 +
* tb_probe_root probes the Distance-To-Zero (DTZ) table for the given position.
 +
 
 +
[[Jon Dart]] has a fork of Fathom with some bug fixes and enhancements <ref>[https://github.com/jdart1/Fathom jdart1/Fathom · GitHub] by [[Jon Dart]]</ref>, also supporting 7-man <ref>[http://www.talkchess.com/forum3/viewtopic.php?f=7&t=70568 7-man Syzygy support in Fathom] by [[Jon Dart]], [[CCC]], April 23, 2019</ref>.
 +
 
 +
==Pyrrhic==
 +
'''Pyrrhic''' is a cleaned up Fathom by [[Andrew Grant]], introduced in August 2020 <ref>[http://www.talkchess.com/forum3/viewtopic.php?f=7&t=74809  Pyrrhic, Fathom for Humanoids] by [[Andrew Grant]], [[CCC]], August 16, 2020</ref> <ref>[https://github.com/AndyGrant/Pyrrhic/ GitHub - AndyGrant/Pyrrhic: Fathom, for Humanoids]</ref>.
 +
 
 +
=Elo gained by Stockfish=
 +
==Stockfish with Classical [[Evaluation]]==
 +
[[Stockfish#Fishtest|Fishtest]] team revealed a test with [[Stockfish]] (SF10dev) at 10+0.1, with all Syzygy WDL files on RAM, testing using none(0), 4, 5, and 6 men TB in a round-robin tournament<ref>https://github.com/glinscott/fishtest/wiki/UsefulData Collection of useful data concerning SF</ref>.
 +
 
 +
{| class="wikitable"
 +
|-
 +
! Rank
 +
! Name
 +
! Elo
 +
! +/-
 +
! Games
 +
! Score
 +
! Draws
 +
|-
 +
! 1
 +
| style="text-align:right;" | syzygy6
 +
| style="text-align:right;" | 13
 +
| style="text-align:right;" | 2
 +
| style="text-align:right;" | 82591
 +
| style="text-align:right;" | 51.8%
 +
| style="text-align:right;" | 59.5%
 +
|-
 +
! 2
 +
| style="text-align:right;" | syzygy5
 +
| style="text-align:right;" | 2
 +
| style="text-align:right;" | 2
 +
| style="text-align:right;" | 82590
 +
| style="text-align:right;" | 50.3%
 +
| style="text-align:right;" | 59.4%
 +
|-
 +
! 3
 +
| style="text-align:right;" | syzygy4
 +
| style="text-align:right;" | -7
 +
| style="text-align:right;" | 2
 +
| style="text-align:right;" | 82591
 +
| style="text-align:right;" | 49.0%
 +
| style="text-align:right;" | 59.3%
 +
|-
 +
! 4
 +
| style="text-align:right;" | syzygy0
 +
| style="text-align:right;" | -7
 +
| style="text-align:right;" | 2
 +
| style="text-align:right;" | 82592
 +
| style="text-align:right;" | 48.9%
 +
| style="text-align:right;" | 59.4%
 +
|}
 +
 
 +
 
 +
==[[Stockfish NNUE]]==
 +
Elo gains by Stockfish NNUE with Sygyzy 6 men are clearly decreased<ref>[https://github.com/official-stockfish/Stockfish/wiki/Useful-data#elo-gain-using-syzygy Useful data · official-stockfish/Stockfish · GitHub] by [[Joost VandeVondele]], August 26, 2023</ref>. With version 15, it gains only 2.7 Elo.
 +
 
 +
[[FILE:Stockfish-syzygy-elo-gain.png|none|border|text-bottom|1024px]]
 +
 
 +
=Quotes=
 
by [[Ronald de Man]] in a reply to [[Guy Haworth]], April 06, 2013 <ref>[http://kirill-kryukov.com/chess/discussion-board/viewtopic.php?f=6&t=6971&start=6 Re: New 6-piece tablebase generator] by [[Ronald de Man|syzygy]], [[Computer Chess Forums|CCRL Discussion Board]], April 06, 2013</ref> :
 
by [[Ronald de Man]] in a reply to [[Guy Haworth]], April 06, 2013 <ref>[http://kirill-kryukov.com/chess/discussion-board/viewtopic.php?f=6&t=6971&start=6 Re: New 6-piece tablebase generator] by [[Ronald de Man|syzygy]], [[Computer Chess Forums|CCRL Discussion Board]], April 06, 2013</ref> :
 
  I create both WDL and DTZ in one go, so I don't use WDL in the creation of DTZ. The algorithm used is the [[Retrograde Analysis#Algorithm|grandfather algorithm]] with 2 plies per iteration (I think [[Harm Geert Muller|HGM]] calls this leapfrogging, but I might be wrong). I tried the outcounting method, but it didn't seem to be competitive (and it makes things more complicated). <ref>[http://www.open-chess.org/viewtopic.php?f=5&t=779 Retrograde tablebase methods] by [[Mark Watkins|BB+]], [[Computer Chess Forums|OpenChess Forum]], November 26, 2010</ref> <ref>[http://home.hccnet.nl/h.g.muller/EGT7/retro.html Leapfrog: Retrograde Analysis] from [http://home.hccnet.nl/h.g.muller/EGT7/7-men.html Leapfrog tablebase generator] by [[Harm Geert Muller]]</ref>
 
  I create both WDL and DTZ in one go, so I don't use WDL in the creation of DTZ. The algorithm used is the [[Retrograde Analysis#Algorithm|grandfather algorithm]] with 2 plies per iteration (I think [[Harm Geert Muller|HGM]] calls this leapfrogging, but I might be wrong). I tried the outcounting method, but it didn't seem to be competitive (and it makes things more complicated). <ref>[http://www.open-chess.org/viewtopic.php?f=5&t=779 Retrograde tablebase methods] by [[Mark Watkins|BB+]], [[Computer Chess Forums|OpenChess Forum]], November 26, 2010</ref> <ref>[http://home.hccnet.nl/h.g.muller/EGT7/retro.html Leapfrog: Retrograde Analysis] from [http://home.hccnet.nl/h.g.muller/EGT7/7-men.html Leapfrog tablebase generator] by [[Harm Geert Muller]]</ref>
  A pure WDL/DTZ pair is not of much use for creating WDL50+/DTZ50+. I create tables in RAM that have all the information necessary for WDL50+ and DTZ50+, then permute them to different indexing schemes and compress. I do test runs on subsets of the data to find good permutations. (The idea to try permutations is from [[Jesper Torp Kristensen|Jesper Torp Kristensen's]] master thesis.) <ref>[[Jesper Torp Kristensen]] ('''2005'''). ''[http://jespertk.dk/Jesper/public/master_thesis/ Generation and compression of endgame tables in chess with fast random access using OBDDs]''. Master thesis, Supervisor [[Mathematician#Miltersen|Peter Bro Miltersen]], [https://en.wikipedia.org/wiki/Aarhus_University Aarhus University], [http://jespertk.dk/Jesper/public/master_thesis/report/report.pdf pdf]</ref> <ref>[https://en.wikipedia.org/wiki/Binary_decision_diagram OBDD - Ordered Binary Decision Diagram from Wikipedia]</ref>  
+
  A pure WDL/DTZ pair is not of much use for creating WDL50+/DTZ50+. I create tables in RAM that have all the information necessary for WDL50+ and DTZ50+, then permute them to different indexing schemes and compress. I do test runs on subsets of the data to find good permutations. (The idea to try permutations is from [[Jesper Torp Kristensen|Jesper Torp Kristensen's]] master thesis.) <ref>[[Jesper Torp Kristensen]] ('''2005'''). ''[https://issuu.com/jespertk/docs/master_thesis Generation and compression of endgame tables in chess with fast random access using OBDDs]''. Master thesis, supervisor [[Mathematician#Miltersen|Peter Bro Miltersen]], [https://en.wikipedia.org/wiki/Aarhus_University Aarhus University]</ref> <ref>[https://en.wikipedia.org/wiki/Binary_decision_diagram OBDD - Ordered Binary Decision Diagram from Wikipedia]</ref>  
 +
 
 +
[[Ronald de Man]] in a reply to [[Pham Hong Nguyen|Nguyen Pham]], April 15, 2020 <ref>[http://talkchess.com/forum3/viewtopic.php?f=7&t=73598 Re: Almost perfect DTM tablebase] by [[Dann Corbit]], [[Computer Chess Forums|CCRL Discussion Board]], April 15, 2020</ref> :
 +
Syzygy WDL is double sided, DTZ is single sided.
 +
So to know whether a 7-piece position is winning, losing or drawn (or cursed), the engine needs to do only a single probe of a 7-piece WDL table. (It may in addition have to do some probes of 6-piece WDL tables if any direct captures are available.)
 +
If the engine needs to know the DTZ value (which is only necessary when a TB root position has been reached), the probing code may have to do a 1-ply search to get to the "right" side of the DTZ table.
 +
For 6-piece TBs, DTZ is 81.9GB when storing only the smaller side of each table. Storing both sides might require perhaps 240GB.
 +
 
 +
[[Morgan Houppin]] explained why his chess engine [[Stash]] doesn't have Syzygy probe code, March 25, 2021 <ref>[http://talkchess.com/forum3/viewtopic.php?f=2&t=76927#p888045 Re: Stash has lost 2 game because of NO EGTB] by [[Morgan Houppin]], [[CCC]], March 25, 2021</ref> :
 +
Syzygy probing is a hell of a mess, and I don't want to plug two thousand lines of foreign code that I don't understand, nor do I have the motivation to fully understand how Syzygy files are stored, and then write the additional two thousand lines of code to read them for a mere 5 Elo gain at TCEC.
  
 
=Endgame News=  
 
=Endgame News=  
Line 49: Line 234:
  
 
and further gives some news about early software bugs and glitches concerning [[ChessBase]] products, and the importance of [https://en.wikipedia.org/wiki/MD5 MD5] to check the EGT integrity.
 
and further gives some news about early software bugs and glitches concerning [[ChessBase]] products, and the importance of [https://en.wikipedia.org/wiki/MD5 MD5] to check the EGT integrity.
<span id="Fathom"></span>
 
=Fathom=
 
'''Fathom''' is a stand-alone Syzygy based probing tool and [https://en.wikipedia.org/wiki/Application_programming_interface API] by [[Basil Falcinelli]], introduced in November 2015 along with his [[GullChess|Gull 3]] release <ref>[http://www.talkchess.com/forum/viewtopic.php?t=58299 Gull 3 Linux+Syzygy and Fathom released] by [[Basil Falcinelli]], [[CCC]], November 20, 2015</ref> . Unlike the original tbprobe code, Fathom does not necessarily require the callee to provide [[Move Generation|move generation]] functionality. The new modifications and extensions to Ronald de Man's original code which can be "redistributed and/or modified without restrictions", are released under the permissive [[Massachusetts Institute of Technology#License|MIT License]]. The API consists of three functions <ref>[https://github.com/basil00/Fathom GitHub - basil00/Fathom: Syzygy TB probe tool] by [[Basil Falcinelli]]</ref> :
 
* tb_init initializes the tablebase
 
* tb_probe_wdl probes the Win-Draw-Loss (WDL) table for a given position
 
* tb_probe_root probes the Distance-To-Zero (DTZ) table for the given position.
 
 
[[Jon Dart]] has a fork of Fathom with some bug fixes and enhancements <ref>[https://github.com/jdart1/Fathom jdart1/Fathom · GitHub] by [[Jon Dart]]</ref>.
 
  
 
=See also=
 
=See also=
Line 72: Line 249:
 
* [[Guy Haworth]] ('''2014'''). ''Chess Endgame News''. [[ICGA Journal#37_2|ICGA Journal, Vol. 37, No. 2]]
 
* [[Guy Haworth]] ('''2014'''). ''Chess Endgame News''. [[ICGA Journal#37_2|ICGA Journal, Vol. 37, No. 2]]
 
* [[Guy Haworth]] ('''2014'''). ''Chess Endgame News''. [[ICGA Journal#37_3|ICGA Journal, Vol. 37, No. 3]] » [[Fritz|Fritz 14]]
 
* [[Guy Haworth]] ('''2014'''). ''Chess Endgame News''. [[ICGA Journal#37_3|ICGA Journal, Vol. 37, No. 3]] » [[Fritz|Fritz 14]]
 +
* [[Guy Haworth]] ('''2018'''). ''Chess Endgame News: 7-man ‘Syzygy’ DTZ50 EGTs''. [[ICGA Journal#40_4|ICGA Journal, Vol. 40, No. 4]]
  
 
=Forum Posts=  
 
=Forum Posts=  
Line 112: Line 290:
 
* [http://www.talkchess.com/forum/viewtopic.php?t=54928 Komodo 8 - 5-men Syzygy tablebases] by [[Andreas Strangmüller]], [[CCC]], January 10, 2015 » [[Komodo#8|Komodo 8]]
 
* [http://www.talkchess.com/forum/viewtopic.php?t=54928 Komodo 8 - 5-men Syzygy tablebases] by [[Andreas Strangmüller]], [[CCC]], January 10, 2015 » [[Komodo#8|Komodo 8]]
 
* [http://www.talkchess.com/forum/viewtopic.php?t=55846  Problem with SF6 and Syzygy TB] by Forrest Hoch, [[CCC]], April 01, 2015 » [[Stockfish]]
 
* [http://www.talkchess.com/forum/viewtopic.php?t=55846  Problem with SF6 and Syzygy TB] by Forrest Hoch, [[CCC]], April 01, 2015 » [[Stockfish]]
* [http://www.talkchess.com/forum/viewtopic.php?t=56124 SD Syzygy] by [http://www.talkchess.com/forum/profile.php?mode=viewprofile&u=608 Ted Summers], [[CCC]], April 26, 2015
+
* [http://www.talkchess.com/forum/viewtopic.php?t=56124 SD Syzygy] by [[Ted Summers]], [[CCC]], April 26, 2015
 
* [http://www.talkchess.com/forum/viewtopic.php?t=56289 5 men Syzygy on USB 3.0 Flash Drive] by [[Kai Laskos]], [[CCC]], May 09, 2015
 
* [http://www.talkchess.com/forum/viewtopic.php?t=56289 5 men Syzygy on USB 3.0 Flash Drive] by [[Kai Laskos]], [[CCC]], May 09, 2015
 
* [http://www.talkchess.com/forum/viewtopic.php?t=56363&start=3 Re: how to probe egtb from console?] by [[Ronald de Man]], [[CCC]], May 15, 2015 » [[python-chess]]
 
* [http://www.talkchess.com/forum/viewtopic.php?t=56363&start=3 Re: how to probe egtb from console?] by [[Ronald de Man]], [[CCC]], May 15, 2015 » [[python-chess]]
* [http://www.talkchess.com/forum/viewtopic.php?t=58299 Gull 3 Linux+Syzygy and Fathom released] by [[Basil Falcinelli]], [[CCC]], November 20, 2015 » [[GullChess]], [[Syzygy Bases#Fathom|Fathom]]
+
* [http://www.talkchess.com/forum/viewtopic.php?t=58299 Gull 3 Linux+Syzygy and Fathom released] by [[Basil Falcinelli]], [[CCC]], November 20, 2015 » [[Gull]], [[Syzygy Bases#Fathom|Fathom]]
 
* [http://www.talkchess.com/forum/viewtopic.php?t=58488 Syzygy probing code: DTZ in some cursed endgames off by one?] by [[Niklas Fiekas]], [[CCC]], December 06, 2015
 
* [http://www.talkchess.com/forum/viewtopic.php?t=58488 Syzygy probing code: DTZ in some cursed endgames off by one?] by [[Niklas Fiekas]], [[CCC]], December 06, 2015
 
'''2016'''
 
'''2016'''
Line 140: Line 318:
 
* [http://www.talkchess.com/forum/viewtopic.php?t=60312&start=240 Natural TB (take 2)] by [[Marco Costalba]], [[CCC]], August 22, 2017 »  [[Stockfish]]
 
* [http://www.talkchess.com/forum/viewtopic.php?t=60312&start=240 Natural TB (take 2)] by [[Marco Costalba]], [[CCC]], August 22, 2017 »  [[Stockfish]]
 
* [http://www.talkchess.com/forum/viewtopic.php?t=65284 Probing tablebases through USB 3.0] by [[Jon Fredrik Åsvang]], [[CCC]], September 25, 2017 » [[Memory#USB3|USB 3.0]]
 
* [http://www.talkchess.com/forum/viewtopic.php?t=65284 Probing tablebases through USB 3.0] by [[Jon Fredrik Åsvang]], [[CCC]], September 25, 2017 » [[Memory#USB3|USB 3.0]]
* [http://www.talkchess.com/forum/viewtopic.php?t=65390 understanding DTZ] by [[Alexandru Mosoi]], [[CCC]], October 06, 2017 » [[Endgame Tablebases#DTZ|DTZ]], [[Syzygy Bases#Fathom|Fathom]]
+
* [http://www.talkchess.com/forum/viewtopic.php?t=65390 understanding DTZ] by [[Alexandru Mosoi]], [[CCC]], October 06, 2017 » [[Endgame Tablebases#DTZ|DTZ]], [[#Fathom|Fathom]]
 
* [http://www.talkchess.com/forum/viewtopic.php?t=65713 Is there now coming changes to syzygy databases?] by [[Jouni Uski]], [[CCC]], November 13, 2017 » [[Endgame Tablebases#DTM|DTM]], [[CFish]]
 
* [http://www.talkchess.com/forum/viewtopic.php?t=65713 Is there now coming changes to syzygy databases?] by [[Jouni Uski]], [[CCC]], November 13, 2017 » [[Endgame Tablebases#DTM|DTM]], [[CFish]]
 
* [http://www.talkchess.com/forum/viewtopic.php?t=66096 How to Download Syzygy Endgame Tablebase Files] by Daniel Johnson, [[CCC]], December 23, 2017
 
* [http://www.talkchess.com/forum/viewtopic.php?t=66096 How to Download Syzygy Endgame Tablebase Files] by Daniel Johnson, [[CCC]], December 23, 2017
Line 146: Line 324:
 
* [http://www.talkchess.com/forum/viewtopic.php?t=66754 The history of Syzygy tablebases] by Isaac Haïk Dunn, [[CCC]], March 06, 2018
 
* [http://www.talkchess.com/forum/viewtopic.php?t=66754 The history of Syzygy tablebases] by Isaac Haïk Dunn, [[CCC]], March 06, 2018
 
* [http://www.talkchess.com/forum/viewtopic.php?t=66797 7-men Syzygy attempt] by [[Bojun Guo]], [[CCC]], March 10, 2018
 
* [http://www.talkchess.com/forum/viewtopic.php?t=66797 7-men Syzygy attempt] by [[Bojun Guo]], [[CCC]], March 10, 2018
 +
: [http://talkchess.com/forum3/viewtopic.php?f=7&t=66797&start=472 Re: 7-men Syzygy attempt] by [[Bojun Guo]], [[CCC]], August 19, 2018
 
* [http://www.talkchess.com/forum/viewtopic.php?t=66830 Syzygy implementations of top engines] by [[Kai Laskos]], [[CCC]], March 14, 2018
 
* [http://www.talkchess.com/forum/viewtopic.php?t=66830 Syzygy implementations of top engines] by [[Kai Laskos]], [[CCC]], March 14, 2018
* [http://www.talkchess.com/forum/viewtopic.php?t=67122 Probing the Syzygy tablebase - beginners question] by Andreas Matthies, [[CCC]], April 16, 2018
+
* [http://www.talkchess.com/forum/viewtopic.php?t=67122 Probing the Syzygy tablebase - beginners question] by [[Andreas Matthies]], [[CCC]], April 16, 2018
 +
* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=67536 DTM50] by [[Ronald de Man]], [[CCC]], May 22, 2018
 +
* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=68236 Re-Pair compression questions] by [[Rein Halbersma]], [[CCC]], August 17, 2018
 +
* [http://www.talkchess.com/forum3/viewtopic.php?f=2&t=68265 BIG NEWS! The 7 man syzygy tablebase files are complete] by [[Dann Corbit]], [[CCC]], August 20, 2018
 +
* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=68755 Technical reason why probing N-men syzygy will also probe N-1 men?] by [[Sven Schüle]], [[CCC]], October 28, 2018
 +
* [http://www.talkchess.com/forum3/viewtopic.php?f=2&t=69309 7 Man Syzygy and SSD] by [[Michael Byrne|Michael B]], [[CCC]], December 18, 2018
 +
'''2019'''
 +
* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=70074 Testing the implementation of Syzygy] by [[Vincent Tang]], [[CCC]], March 02, 2019
 +
* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=70568 7-man Syzygy support in Fathom] by [[Jon Dart]], [[CCC]], April 23, 2019 » [[#Fathom|Fathom]]
 +
* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=71397 Simplest use of syzygy table] by [[Vivien Clauzon]], [[CCC]], July 28, 2019
 +
* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=71512 SYZYGY question] by [[Robert Hyatt]], [[CCC]], August 11, 2019 » [[Crafty]], [[En passant]]
 +
* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=71604 Syzygy 7-piece - several questions] by [[Andreas Matthies]], [[CCC]], August 19, 2019
 +
* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=71625 Syzygy 7 man advice please] by Barry Clements, [[CCC]], August 21, 2019
 +
* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=71896 Syzygy DTZ data explaination?] by [[Pham Hong Nguyen|Nguyen Pham]], [[CCC]], September 23, 2019
 +
==2020 ...==
 +
* [https://talkchess.com/forum3/viewtopic.php?f=7&t=73598 Almost perfect DTM tablebase] by [[Dann Corbit]], [[CCC]], April 08, 2020
 +
* [https://talkchess.com/forum3/viewtopic.php?f=2&t=74185 What is the best way to obtain the 7-piece tablebases?] by [[Mark Thellen]], [[CCC]], June 15, 2020
 +
* [https://talkchess.com/forum3/viewtopic.php?f=7&t=74809 Pyrrhic, Fathom for Humanoids] by [[Andrew Grant]], [[CCC]], August 16, 2020 » [[#Pyrrhic|Pyrrhic]]
 +
* [https://talkchess.com/forum3/viewtopic.php?f=7&t=75396 EGTB compression] by [[Dann Corbit]], [[CCC]], October 14, 2020 <ref>[https://en.wikipedia.org/wiki/LZ4_(compression_algorithm) LZ4 (compression algorithm) from Wikipedia]</ref> <ref>[https://en.wikipedia.org/wiki/Zstandard Zstandard from Wikipedia]</ref>
 +
* [https://talkchess.com/forum3/viewtopic.php?f=2&t=75487 Syzygy bases ... question to "Syzygy Probe Depth"] by [[Frank Quisinsky]], [[CCC]], October 21, 2020
 +
* [https://talkchess.com/forum3/viewtopic.php?f=7&t=75906 Fathom and 7-men] by [[Joshua Shriver]], [[CCC]], November 24, 2020 » [[#Fathom|Fathom]]
 +
* [https://talkchess.com/forum3/viewtopic.php?f=7&t=75973 Problem with Syzygy tablebase] by [[Elias Nilsson]], [[CCC]], December 03, 2020
 +
* [https://talkchess.com/forum3/viewtopic.php?f=7&t=76010 Can EGTB storage requirements be reduced using this scheme?] by mmt, [[CCC]], December 07, 2020
 +
'''2021'''
 +
* [https://talkchess.com/forum3/viewtopic.php?f=7&t=77267 Syzygy Tablebase Names: A very stupid exercise] by [[Andrew Grant]], [[CCC]], May 08, 2021
 +
* [https://talkchess.com/forum3/viewtopic.php?f=7&t=77358 syzygy implementation] by [[Michael Hoffmann|Desperado]], [[CCC]], May 23, 2021
 +
* [https://talkchess.com/forum3/viewtopic.php?f=2&t=77400 When will 8 piece tablebase be ready?] by [[Agustin Jorge Pichardo]], [[CCC]], May 29, 2021
 +
* [https://talkchess.com/forum3/viewtopic.php?f=7&t=77499 Syzygy bases from memory] by [[Ed Schroder|Ed Schröder]], [[CCC]], June 16, 2021 » [[KPK]]
 +
* [https://talkchess.com/forum3/viewtopic.php?f=2&t=78080 Syzygy benefit for current SF] by [[Jouni]], [[CCC]], September 02, 2021
 +
'''2022'''
 +
* [https://talkchess.com/forum3/viewtopic.php?f=7&t=79938 Definite occurance ranking of 7-Man EGTB] by [[Daniel Infuehr]], [[CCC]], May 24, 2022
 +
* [https://talkchess.com/forum3/viewtopic.php?f=7&t=80522 Fathom, munmap issue] by [[Pawel Osikowski]], [[CCC]], August 19, 2022
 +
* [https://talkchess.com/forum3/viewtopic.php?f=2&t=80608 Are tablebases useless for Stockfish15?] by [[Jouni]], [[CCC]], September 02, 2022
 +
* [https://talkchess.com/forum3/viewtopic.php?f=7&t=80696 endgame table generation] by [[Dave Gomboc]], [[CCC]], September 17, 2022
  
=External Links=  
+
'''2023'''
 +
* [https://talkchess.com/forum3/viewtopic.php?f=7&t=82115 very strange behavior with syzygy tables] by [[Carbec]], [[CCC]], June 02, 2023
 +
* [https://talkchess.com/forum3/viewtopic.php?f=7&t=82613 EGTB encoding] by [[Edmund]], [[CCC]], September 18, 2023
 +
* [https://talkchess.com/forum3/viewtopic.php?f=2&t=82937 Need a tip for 6-man syzygy ...] by [[Frank Quisinsky]], [[CCC]], December 02, 2023
 +
 
 +
=External Links=
 
==Tablebase==  
 
==Tablebase==  
 
* [https://github.com/syzygy1/tb syzygy1/tb · GitHub] by [[Ronald de Man]]
 
* [https://github.com/syzygy1/tb syzygy1/tb · GitHub] by [[Ronald de Man]]
Line 155: Line 372:
 
* [https://github.com/niklasf/python-chess/blob/master/chess/syzygy.py python-chess/syzygy.py at master · niklasf/python-chess · GitHub] by [[Niklas Fiekas]], [[Python]] implementation of probing code » [[python-chess]]
 
* [https://github.com/niklasf/python-chess/blob/master/chess/syzygy.py python-chess/syzygy.py at master · niklasf/python-chess · GitHub] by [[Niklas Fiekas]], [[Python]] implementation of probing code » [[python-chess]]
 
* [https://github.com/niklasf/syzygy-tables.info niklasf/syzygy-tables.info · GitHub] by [[Niklas Fiekas]], [[GUI]] and public API for Syzygy probing
 
* [https://github.com/niklasf/syzygy-tables.info niklasf/syzygy-tables.info · GitHub] by [[Niklas Fiekas]], [[GUI]] and public API for Syzygy probing
* [https://syzygy-tables.info/ Syzygy endgame tablebases Web Interface] - Syzygy online lookup
 
 
* [http://oics.olympuschess.com/tracker/index.php OICS Chess and EGTB Tracker Statistics] by [[Joshua Shriver]] <ref>[https://en.wikipedia.org/wiki/BitTorrent BitTorrent from Wikipedia]</ref>
 
* [http://oics.olympuschess.com/tracker/index.php OICS Chess and EGTB Tracker Statistics] by [[Joshua Shriver]] <ref>[https://en.wikipedia.org/wiki/BitTorrent BitTorrent from Wikipedia]</ref>
 
* [http://kirill-kryukov.com/chess/tablebases-online/ Endgame Tablebases Online] by [[Kirill Kryukov]]
 
* [http://kirill-kryukov.com/chess/tablebases-online/ Endgame Tablebases Online] by [[Kirill Kryukov]]
Line 163: Line 379:
 
* [https://github.com/basil00/Fathom GitHub - basil00/Fathom: Syzygy TB probe tool] by [[Basil Falcinelli]]
 
* [https://github.com/basil00/Fathom GitHub - basil00/Fathom: Syzygy TB probe tool] by [[Basil Falcinelli]]
 
* [https://github.com/jdart1/Fathom jdart1/Fathom · GitHub] by [[Jon Dart]] (with some bug fixes and enhancements)
 
* [https://github.com/jdart1/Fathom jdart1/Fathom · GitHub] by [[Jon Dart]] (with some bug fixes and enhancements)
 +
* [https://github.com/ljgw/syzygy-bridge GitHub - ljgw/syzygy-bridge: Java bridge to use the Syzygy Tablebases via JNI] by [[Laurens Winkelhagen]] » [[FrankWalter]]
 +
==Pyrrhic==
 +
* [https://github.com/AndyGrant/Pyrrhic/ GitHub - AndyGrant/Pyrrhic: Fathom, for Humanoids] by [[Andrew Grant]]
 +
==Online Lookup==
 +
* [https://www.chessdb.cn/queryc_en/ Chess Cloud Database Query Interface] by [[Bojun Guo|noobpwnftw]] <ref>[http://www.talkchess.com/forum3/viewtopic.php?f=2&t=71764 ChessDBCN] by [[Bojun Guo|noobpwnftw]], [[CCC]], September 09, 2019</ref>
 +
* [https://syzygy-tables.info/ Syzygy endgame tablebases Web Interface]
 
==ChessBase==  
 
==ChessBase==  
 
* [http://en.chessbase.com/post/syzygy-tablebases-newest-fastest-smallest Syzygy Tablebases: newest, fastest, smallest] by [[Albert Silver]], [[ChessBase|ChessBase News]], February 08, 2015
 
* [http://en.chessbase.com/post/syzygy-tablebases-newest-fastest-smallest Syzygy Tablebases: newest, fastest, smallest] by [[Albert Silver]], [[ChessBase|ChessBase News]], February 08, 2015
Line 173: Line 395:
 
* [https://en.wikipedia.org/wiki/Syzygy_%28mathematics%29 Syzygy (mathematics) from Wikipedia]
 
* [https://en.wikipedia.org/wiki/Syzygy_%28mathematics%29 Syzygy (mathematics) from Wikipedia]
 
* [https://en.wikipedia.org/wiki/Caledonian_Antisyzygy Caledonian Antisyzygy from Wikipedia]
 
* [https://en.wikipedia.org/wiki/Caledonian_Antisyzygy Caledonian Antisyzygy from Wikipedia]
* [[Videos#MichaelBrecker|Michael Brecker Band]] - [https://en.wikipedia.org/wiki/Michael_Brecker_(album) Syzygy], August 16, 1987 at [https://en.wikipedia.org/wiki/Newport_Jazz_Festival Newport Jazz Festival], [https://en.wikipedia.org/wiki/YouTube YouTube] Video
+
* [[:Category:Michael Brecker|Michael Brecker Band]] - [https://en.wikipedia.org/wiki/Michael_Brecker_(album) Syzygy], [https://www.bjd.sk/archiv-1979-2001#1987 BJD 1987], [https://en.wikipedia.org/wiki/Bratislava Bratislava], [https://en.wikipedia.org/wiki/YouTube YouTube] Videos
: [[Videos#MichaelBrecker|Michael Brecker]], [[Videos#MikeStern|Mike Stern]], [https://de-de.facebook.com/JeffAndrewsBassPage/ Jeff Andrews], [https://en.wikipedia.org/wiki/Adam_Nussbaum Adam Nussbaum], [https://en.wikipedia.org/wiki/Joey_Calderazzo Joey Calderazzo]
+
: [[:Category:Michael Brecker|Michael Brecker]], [[:Category:Mike Stern|Mike Stern]], [https://de-de.facebook.com/JeffAndrewsBassPage/ Jeff Andrews], [[:Category:Adam Nussbaum|Adam Nussbaum]], [https://en.wikipedia.org/wiki/Joey_Calderazzo Joey Calderazzo]
: {{#evu:https://www.youtube.com/watch?v=lgVxFM7cONQ|alignment=left|valignment=top}}
+
: {{#evu:https://www.youtube.com/watch?v=cU9aDLOG89o|alignment=left|valignment=top}}
 +
: {{#evu:https://www.youtube.com/watch?v=BUcz1YD0Nr8|alignment=left|valignment=top}}
  
 
=References=  
 
=References=  
 
<references />
 
<references />
 
 
'''[[Endgame Tablebases|Up one level]]'''
 
'''[[Endgame Tablebases|Up one level]]'''
 +
[[Category:Quotes]]
 +
[[Category:Michael Brecker]]
 +
[[Category:Adam Nussbaum]]
 +
[[Category:Mike Stern]]

Latest revision as of 12:35, 7 December 2023

Home * Knowledge * Endgame Tablebases * Syzygy Bases

Syzygy - Three Planets Dance Over La Silla [1]

Syzygy Bases, [2]
a compact six piece endgame database developed by Ronald de Man, published on April 01, 2013. Since August 2018, seven piece Syzygy Bases are available after an effort by Bojun Guo started in March 2018 [3]. The generator is released under the GNU General Public License Version 2, the thread safe probing code is released without restrictions [4] .

Data

File Types

Syzygy Bases consist of two sets of files, WDL files (extension .rtbw) storing win/draw/loss information considering the fifty-move rule for access during search, and DTZ files (extension .rtbz) with distance-to-zero information for access at the root. WDL has full data for two sides but DTZ50 omitted data of one side to save space. Each endgame has a pair of those types.

File Sizes

Men WDL DTZ Total
3-5 378.1 MiB 561.9 MiB 939.0 MiB
6 67.8 GiB 81.4 GiB 149.2 GiB
7 8.5 TiB 8.3 TiB 16.7 TiB

Comparision

Syzygy EGTB is significantly smaller than any existent DTM EGTB. It is 7 times as small as Gaviota for 5 men, 8 times as small as Nalimov for 6 men, 8 times as small as Lomonosov for 7 men. However, when all DTM EGTBs have full data of two sides, Syzygy EGTB omits data of one side for DTZ data to save space. Ronald de Man estimated if keep them all, the Syzygy's 6 men size may increase 158 GB, become 307 GB in total, double on size, be 4 times as small as Nalimov 6 men.

Generation

Up to 6-man

On the first release (Apr 01, 2013) the generator was ready to generate all endgames up to 6 men. It is multithreading and processes completely in RAM. To generate all 6 men, it requires a system with at least 32 GB of RAM and may run in 5 days (the period was measured with a computer 6-core i3930K @ 4.2Ghz, 64 GB).

7-man

Ronald de Man wasn't initially interested in the creation of 7-men Syzygy Based [5] since the generation would require about 1 TB of RAM, too expensive at that time. Generation time would be about 64 x per table, which means around 175 x time total [6].His original generator could not create them. But in 2018, he supported Bojun Guo in his 5-month attempt to generated them [7]. His hardware was estimated at over US$ 90K. In August of 2018 their creation was completed [8][9].

8-man

After the completion of 7-man, many people start being curious about the feasibleness of building 8-man. Ronald de Man made an estimation that task requires computers with 64 TB RAM and 2000 TB hard disks[10] (costed about $640K and $40K respectively in 2020 [11]).

Checksums

Syzygy endgame files may contain 128-bit checksum keys at the end of those files. It also has its own code for checksums (based on Google's cityhash library).

Search

During the Search

During the search, with the WDL tables stored on SSD [12] , it is possible to probe the tables at all depths without much slowdown. They have been tested in Ronald de Man's engine Sjaak (playing on FICS as TrojanKnight(C)) a couple of months quite successfully, don't probing in quiescence search.

At the Root

At the root, since pure DTZ50-optimal play (i.e. minimaxing the number of moves to the next capture or pawn move by either side) can be very unnatural, it might be desirable to let the engine search on the winning moves until it becomes clear that insufficient progress is being made and only then switch to DTZ-optimal play (e.g. by detecting repetitions and monitoring the halfmove clock) [13].

Pros & Cons

Pros

  • Small sizes. It is about 8 times as small as the second-best EGTBs. Having small sizes is the main success key of Syzygy Bases since it is much easier to create, store, provide, download than other EGTBs
  • Free and more popular (than other EGTBs) to find on the Internet
  • Support DTZ50 metric. That metric can help engines to have better results than DTM which is supported widely by other EGTBs

Cons

  • Hard to integrate with chess engines. Ronald de Man has not provided probing code as an independent library but as a part of Stockfish chess engines. The probe code has integrated too deeply with that chess engine code and it requires a lot of effort to de-integrate, modify and integrate with other chess projects
  • Hard to understand and contribute to the project. ETGB itself is a hard topic. Syzygy EGTB has also integrated with many advanced techniques/tricks. It is written in old-style C language. All make it become very hard to understand and/or modify to improve or for other purposes
  • DTZ50 metric may lead the engines to win in much longer and ugly ways, compared with DTM one

Data publish

Ronald de Man has provided only open source code for generators/probers but not endgame files themselves. Using his tools some people have generated endgames and published them via DVD or online.

DVD

As of February 2015, all 3-5 and most important 6-men Syzygy Bases are commercially available on 4 DVDs by ChessBase as Endgame Turbo 4 for their products Deep Fritz 14, Komodo Chess 8, Houdini 4 or ChessBase 12/13 [14] [15].

Free Download

There are some free FTP servers for downloading such as Bojun Guo and Lichess servers.

3-6 men

The sizes of those men are small enough to download and store on modern computers. Users should download them in full sets (3, 4, 5, 6 men).

7 men

All 7 men files' size is over 16.7 TiB, over storages of typical modern computers. They also require a long time to download too. Thus some users choose to download one or a few endgames only, based on their statistics of use in endgames. Bellow is the top 20 of those endgames by their order. The first one, KRPPvKRP, has a significantly higher frequency of use than the others and should be always downloaded:

Rank Name Rank Name Rank Name Rank Name
1 KRPPvKRP 6 KNPPvKNP 11 KBPPvKNP 16 KRBPPvKR
2 KBPPvKBP 7 KNPPvKBP 12 KRPPvKRB 17 KBPPvKPP
3 KPPPvKPP 8 KRBPvKRP 13 KRPPvKPP 18 KRPPPvKP
4 KRPPPvKR 9 KQPPvKPP 14 KBPPvKRP 19 KRBPvKRB
5 KQPPvKQP 10 KQPPPvKP 15 KRNPvKRP 20 KRPPvKRN

Probe Code and Tools

Stockfish

Ronald de Man did not provide the probe code as an independent library. Instead, he published it firstly as an already integrated code for Stockfish chess engines. It is c++ code and it has been rewritten, updated several times by Stockfish team.

Fathom

Fathom is a stand-alone Syzygy based probing tool and API by Basil Falcinelli, introduced in November 2015 along with his Gull 3 release [16] . Unlike the original tbprobe code, Fathom does not necessarily require the callee to provide move generation functionality. The new modifications and extensions to Ronald de Man's original code which can be "redistributed and/or modified without restrictions", are released under the permissive MIT License. The API consists of three functions [17] :

  • tb_init initializes the tablebase
  • tb_probe_wdl probes the Win-Draw-Loss (WDL) table for a given position
  • tb_probe_root probes the Distance-To-Zero (DTZ) table for the given position.

Jon Dart has a fork of Fathom with some bug fixes and enhancements [18], also supporting 7-man [19].

Pyrrhic

Pyrrhic is a cleaned up Fathom by Andrew Grant, introduced in August 2020 [20] [21].

Elo gained by Stockfish

Stockfish with Classical Evaluation

Fishtest team revealed a test with Stockfish (SF10dev) at 10+0.1, with all Syzygy WDL files on RAM, testing using none(0), 4, 5, and 6 men TB in a round-robin tournament[22].

Rank Name Elo +/- Games Score Draws
1 syzygy6 13 2 82591 51.8% 59.5%
2 syzygy5 2 2 82590 50.3% 59.4%
3 syzygy4 -7 2 82591 49.0% 59.3%
4 syzygy0 -7 2 82592 48.9% 59.4%


Stockfish NNUE

Elo gains by Stockfish NNUE with Sygyzy 6 men are clearly decreased[23]. With version 15, it gains only 2.7 Elo.

Stockfish-syzygy-elo-gain.png

Quotes

by Ronald de Man in a reply to Guy Haworth, April 06, 2013 [24] :

I create both WDL and DTZ in one go, so I don't use WDL in the creation of DTZ. The algorithm used is the grandfather algorithm with 2 plies per iteration (I think HGM calls this leapfrogging, but I might be wrong). I tried the outcounting method, but it didn't seem to be competitive (and it makes things more complicated). [25] [26]
A pure WDL/DTZ pair is not of much use for creating WDL50+/DTZ50+. I create tables in RAM that have all the information necessary for WDL50+ and DTZ50+, then permute them to different indexing schemes and compress. I do test runs on subsets of the data to find good permutations. (The idea to try permutations is from Jesper Torp Kristensen's master thesis.) [27] [28] 

Ronald de Man in a reply to Nguyen Pham, April 15, 2020 [29] :

Syzygy WDL is double sided, DTZ is single sided.
So to know whether a 7-piece position is winning, losing or drawn (or cursed), the engine needs to do only a single probe of a 7-piece WDL table. (It may in addition have to do some probes of 6-piece WDL tables if any direct captures are available.)
If the engine needs to know the DTZ value (which is only necessary when a TB root position has been reached), the probing code may have to do a 1-ply search to get to the "right" side of the DTZ table.
For 6-piece TBs, DTZ is 81.9GB when storing only the smaller side of each table. Storing both sides might require perhaps 240GB.

Morgan Houppin explained why his chess engine Stash doesn't have Syzygy probe code, March 25, 2021 [30] :

Syzygy probing is a hell of a mess, and I don't want to plug two thousand lines of foreign code that I don't understand, nor do I have the motivation to fully understand how Syzygy files are stored, and then write the additional two thousand lines of code to read them for a mere 5 Elo gain at TCEC.

Endgame News

In his 2014 Chess Endgame News in ICGA Journal, Vol. 37, No. 2 [31] , Guy Haworth classified Syzygy Bases as new data in three ways:

  1. 5-valued scale for evaluating positions in the context of the FIDE 50-move rule (50mr) which constrains the length of phases of play
    1. +2 ≡ unconditional win for the side to move
    2. +1 ≡ ‘win’ which can be frustrated by best play and a 50mr draw-claim
    3. _0 ≡ unconditional draw
    4. -1 ≡ ‘loss’ saved by a 50mr draw-claim
    5. -2 ≡ unconditional loss
  2. depths for ‘50mr draw’ positions with value ±1
  3. depths in symmetric, information-preserving ply ‘p’

and further gives some news about early software bugs and glitches concerning ChessBase products, and the importance of MD5 to check the EGT integrity.

See also

Publications

Forum Posts

2013 ...

2014

2015 ...

2016

Re: Stockfish 7 and partial 6 piece syzygy problem? by Marco Costalba, CCC, September 01, 2016

2017

2018

Re: 7-men Syzygy attempt by Bojun Guo, CCC, August 19, 2018

2019

2020 ...

2021

2022

2023

External Links

Tablebase

Fathom

Pyrrhic

Online Lookup

ChessBase

Misc

Michael Brecker, Mike Stern, Jeff Andrews, Adam Nussbaum, Joey Calderazzo

References

  1. It’s a real treat for photographers and astronomers alike: our skies are currently witnessing a phenomenon known as a syzygy — when three celestial bodies (or more) nearly align themselves in the sky. When celestial bodies have similar ecliptic longitude, this event is also known as a triple near-conjunction. Of course, this is just a trick of perspective, but this doesn't make it any less spectacular. In this case, these bodies are three planets, and the only thing needed to enjoy the show is a clear view of the sky at sunset. Luckily, this is what happened for ESO photo ambassador Yuri Beletsky, who had the chance to spot this spectacular view from ESO's La Silla Observatory in northern Chile on Sunday May 26 , 2013. Above the round domes of the telescopes, three of the planets in our Solar SystemJupiter (top), Venus (lower left), and Mercury (lower right) — were revealed aftersunset, engaged in their cosmic dance. An alignment like this happens only once every several years. The last one took place in May 2011, and the next one will not be until October 2015. This celestial triangle was at its best over the last week of May, but you may still be able to catch a glimpse of the three planets as they form ever-changing arrangements during their journey across the sky - source Three Planets Dance Over La Silla | ESO, Syzygy (astronomy) from Wikipedia
  2. Re: New 6-piece tablebases by Ronald de Man, CCC, April 10, 2013
  3. Re: 7-men Syzygy attempt by Bojun Guo, CCC, August 19, 2018
  4. New 6-piece tablebases by Ronald de Man, CCC, April 01, 2013
  5. Re: 7-piece syzygy by syzygy, CCRL Discussion Board, July 03, 2014
  6. Re: Syzygy tb generator for windows by syzygy, CCRL Discussion Board, June 01, 2014
  7. 7-men Syzygy attempt by Bojun Guo, CCC, March 10, 2018
  8. Powered by Ronald de Man's Syzygy endgame tablebases, 7-piece tables generated by Bojun Guo and a public API hosted by lichess.org, August 19, 2018
  9. Index of /tables/standard/7/ on lichess, August 19, 2018
  10. Re: What is the best way to obtain the 7-piece tablebases? by syzygy, CCC, June 22, 2020
  11. Re: What is the best way to obtain the 7-piece tablebases? by Dann Corbit, CCC, June 23, 2020
  12. Re: SSD and the use of Tablebases by Ronald de Man, CCC, May 08, 2013
  13. Re: New 6-piece tablebases by Ronald de Man, CCC, April 05, 2013
  14. Endgame Turbo 4 by ChessBase
  15. Syzygy Tablebases: newest, fastest, smallest by Albert Silver, ChessBase News, February 08, 2015
  16. Gull 3 Linux+Syzygy and Fathom released by Basil Falcinelli, CCC, November 20, 2015
  17. GitHub - basil00/Fathom: Syzygy TB probe tool by Basil Falcinelli
  18. jdart1/Fathom · GitHub by Jon Dart
  19. 7-man Syzygy support in Fathom by Jon Dart, CCC, April 23, 2019
  20. Pyrrhic, Fathom for Humanoids by Andrew Grant, CCC, August 16, 2020
  21. GitHub - AndyGrant/Pyrrhic: Fathom, for Humanoids
  22. https://github.com/glinscott/fishtest/wiki/UsefulData Collection of useful data concerning SF
  23. Useful data · official-stockfish/Stockfish · GitHub by Joost VandeVondele, August 26, 2023
  24. Re: New 6-piece tablebase generator by syzygy, CCRL Discussion Board, April 06, 2013
  25. Retrograde tablebase methods by BB+, OpenChess Forum, November 26, 2010
  26. Leapfrog: Retrograde Analysis from Leapfrog tablebase generator by Harm Geert Muller
  27. Jesper Torp Kristensen (2005). Generation and compression of endgame tables in chess with fast random access using OBDDs. Master thesis, supervisor Peter Bro Miltersen, Aarhus University
  28. OBDD - Ordered Binary Decision Diagram from Wikipedia
  29. Re: Almost perfect DTM tablebase by Dann Corbit, CCRL Discussion Board, April 15, 2020
  30. Re: Stash has lost 2 game because of NO EGTB by Morgan Houppin, CCC, March 25, 2021
  31. Guy Haworth (2014). Chess Endgame News. ICGA Journal, Vol. 37, No. 2
  32. Found an interesting research by Kirill Kryukov, April, 12, 2013
  33. Wine (software) from Wikipedia
  34. N. Jesper Larsson, Alistair Moffat (1999). Offline Dictionary-Based Compression. DCC'99, pdf
  35. LZ4 (compression algorithm) from Wikipedia
  36. Zstandard from Wikipedia
  37. BitTorrent from Wikipedia
  38. ChessDBCN by noobpwnftw, CCC, September 09, 2019

Up one level