# Difference between revisions of "Hash Table"

GerdIsenberg (talk | contribs) |
GerdIsenberg (talk | contribs) |
||

Line 91: | Line 91: | ||

* [[John Tromp]] ('''2014'''). ''[https://github.com/tromp/cuckoo/blob/master/cuckoo.pdf Cuckoo Cycle: a memory-hard proof-of-work system]''. [http://www.informatik.uni-trier.de/~ley/db/journals/iacr/iacr2014.html#Tromp14 IACR Cryptology ePrint Archive] <ref>[https://bitcointalk.org/index.php?topic=405483.0 Cuckoo Cycle: a new memory-hard proof-of-work system] by [[John Tromp]], [https://bitcointalk.org/index.php Bitcoin Forum], January 08, 2014</ref> <ref>[https://en.wikipedia.org/wiki/Cuckoo_hashing Cuckoo hashing from Wikipedia]</ref> | * [[John Tromp]] ('''2014'''). ''[https://github.com/tromp/cuckoo/blob/master/cuckoo.pdf Cuckoo Cycle: a memory-hard proof-of-work system]''. [http://www.informatik.uni-trier.de/~ley/db/journals/iacr/iacr2014.html#Tromp14 IACR Cryptology ePrint Archive] <ref>[https://bitcointalk.org/index.php?topic=405483.0 Cuckoo Cycle: a new memory-hard proof-of-work system] by [[John Tromp]], [https://bitcointalk.org/index.php Bitcoin Forum], January 08, 2014</ref> <ref>[https://en.wikipedia.org/wiki/Cuckoo_hashing Cuckoo hashing from Wikipedia]</ref> | ||

* [[David Eppstein]], [[Mathematician#MTGoodrich|Michael T. Goodrich]], [[Mathematician#MMitzenmacher|Michael Mitzenmacher]], [https://dblp.uni-trier.de/pers/hd/p/Pszona:Pawel Paweł Pszona] ('''2014'''). ''Wear Minimization for Cuckoo Hashing: How Not to Throw a Lot of Eggs into One Basket''. [https://arxiv.org/abs/1404.0286 arXiv:1404.0286] | * [[David Eppstein]], [[Mathematician#MTGoodrich|Michael T. Goodrich]], [[Mathematician#MMitzenmacher|Michael Mitzenmacher]], [https://dblp.uni-trier.de/pers/hd/p/Pszona:Pawel Paweł Pszona] ('''2014'''). ''Wear Minimization for Cuckoo Hashing: How Not to Throw a Lot of Eggs into One Basket''. [https://arxiv.org/abs/1404.0286 arXiv:1404.0286] | ||

+ | * [[Thomas Dybdahl Ahle]], [[Mathematician#MAumueller|Martin Aumüller]], [[Mathematician#RPagh|Rasmus Pagh]] ('''2016'''). ''Parameter-free Locality Sensitive Hashing for Spherical Range Reporting''. [https://arxiv.org/abs/1605.02673 arXiv:1605.02673] <ref>[https://en.wikipedia.org/wiki/Locality-sensitive_hashing Locality-sensitive hashing from Wikipedia]</ref> | ||

+ | * [[Thomas Dybdahl Ahle]] ('''2017'''). ''Optimal Set Similarity Data-structures Without False Negatives''. Master thesis, [https://en.wikipedia.org/wiki/University_of_Copenhagen University of Copenhagen], [http://www.itu.dk/people/thdy/papers/minhash.pdf pdf] <ref>[https://en.wikipedia.org/wiki/MinHash MinHash from Wikipedia]</ref> | ||

=Forum Posts= | =Forum Posts= |

## Revision as of 16:37, 9 August 2019

**Home * Programming * Data * Hash Table**

A **Hash Table**, or a Hash Map, is a data structure that associates identifiers or keys (names, chess positions) with values (i. e. phone number, score of a position). A hash function is used for turning the key into a relatively small integer, the hash, that serves as an index into an array.

In a well-dimensioned hash table, the average cost for each lookup is independent of the number of elements stored in the table. In general programming as well in computer chess, hash tables often serve as cache for once calculated values, to save relative expensive computations over and over again.

## Contents

# Perfect Hashing

If all of the keys are known at compile or initialization time and their cardinality is reasonable small, a perfect hash table can be created, in which there will be no collisions, since each key has an unique index. Opposed to Minimal Perfect Hashing, the lookup array contains either gaps or multiple entries.

Applications in computer chess are hashing of masked occupied bitboards, to map for instance attack sets of sliding pieces (rooks, bishops) on a particular square, or pawn shield stuff. Another application, despite a reversible hash function, is a precomputed table indexed by some material key, likely an incremental updated dot-product of all piece-type counts in some fixed order, by a vector of cumulated maximum count products of pieces ordered below, usually ignoring unusual material configurations due to promotions ^{[3]} . Persistent, or on the fly generated Endgame Tablebases and Bitbases containing perfect knowledge by retrograde analyses of certain material constellations with a few pieces in late endings might be considered as a kind of perfect hashing as well.

- Walter Faxon's Magic BitScan
- BitScan by Modulo
- Kindergarten Bitboards
- Magic Bitboards
- Hashing Dictionaries
- Congruent Modulo Bitboards
- Material Tables
- Pawn Shield Tables
- Endgame Tablebases

# Minimal Perfect Hashing

If the hash array has no gaps and unique, distinct values, so called Minimal Perfect Hashing is applied, like in following bitboard hashing for bitscan purpose or determining pre-calculated move lists by move target sets of knights and king.

- De Bruijn Multiplication
- Matt Taylor's Folding trick
- Hashing Multiple Bits for Bitboard Serialization

# Search Tables

The classical hash table implementation in computer chess are the transposition tables, indexed by Zobrist- or BCH keys of the position, modulo number of entries inside the table, to store search results, evaluation, pawn structure related stuff and repetitions.

- Transposition Table
- Pawn Hash Table
- Evaluation Hash Table
- Material Hash Table
- Persistent Hash Table
- Repetition Hash Table
- Shared Hash Table

# Class Libraries

- QMap Class Reference, C++ from Qt On-line Reference Documentation
- CMap Class, C++ from MFC Library Reference
- Interface Map from the Java package java.util
- UserDict - Class wrapper for dictionary objects from Python Library Reference
- defaultdict objects from Python Library Reference

# Publications

## 1968

- Ward Douglas Maurer (
**1968**).*An Improved Hash Code for Scatter Storage*. Communications of the ACM, Vol. 11, No. 1 - Elwyn Berlekamp (
**1968**).*Algebraic Coding Theory*, New York: McGraw-Hill. Revised ed., Aegean Park Press, (**1984**), ISBN 0894120638. amazon - F R A Hopgood (
**1968**).*A Solution to the table overflow problem for Hash Tables*. Literature: Reports hosted by Atlas Computer Laboratory

## 1970 ...

- Albert Zobrist (
**1970**).*A New Hashing Method with Application for Game Playing*. Technical Report #88, Computer Science Department, The University of Wisconsin, Madison, WI, USA. Reprinted (**1990**) in ICCA Journal, Vol. 13, No. 2, pdf - Burton H. Bloom (
**1970**).*Space/time trade-offs in hash coding with allowable errors*. Comm. of the ACM, Vol. 13, No. 7, pdf^{[5]} - F R A Hopgood, J. Davenport (
**1972**).*The quadratic hash method when the table size is a power of 2*. Literature: Reports hosted by Atlas Computer Laboratory - F R A Hopgood (
**1974**).*Computer Animation used as a Tool in Teaching Computer Science*. Literature: Reports hosted by Atlas Computer Laboratory - Ole Amble, Donald Knuth (
**1974**).*Ordered Hash Tables*. The Computer Journal, Vol. 17 - Ward Douglas Maurer, Ted G. Lewis (
**1975**).*Hash Table Methods*. ACM Computing Surveys, Vol. 7, No. 1 - J. Lawrence Carter, Mark N. Wegman (
**1977**).*Universal classes of hash functions*. STOC '77

## 1980 ...

- Harry Nelson (
**1985**).*Hash Tables in Cray Blitz*. ICCA Journal, Vol. 8, No. 1 » Cray Blitz - Tony Warnock, Burton Wendroff (
**1988**).*Search Tables in Computer Chess*. ICCA Journal, Vol. 11, No. 1 - Ivan Damgård (
**1989**).*A Design Principle for Hash Functions*. CRYPTO '89 - Ralph C. Merkle (
**1989**).*One Way Hash Functions and DES*. CRYPTO '89^{[6]}

## 1990 ...

- Zbigniew J. Czech, George Havas, Bohdan S. Majewski (
**1992**).*An Optimal Algorithm for Generating Minimal Perfect Hash Functions*. Information Processing Letters, Vol. 43, pp. 257–264. pdf - Warren D. Smith (
**1992**).*Hash functions for Binary and Ternary Words*. NEC Research Institute, ps - Martin Dietzfelbinger, Anna Karlin, Kurt Mehlhorn, Friedhelm Meyer auf der Heide, Hans Rohnert, Robert E. Tarjan (
**1994**).*Dynamic Perfect Hashing: Upper and Lower Bounds*. SIAM Journal on Computing, Vol. 23, Nr. 4 - Zbigniew J. Czech, George Havas, Bohdan S. Majewski (
**1997**).*Perfect Hashing*. Theoretical Computer Science, Vol. 182, Nos. 1-2, pp. 1-143 - Bob Jenkins (
**1997**).*Hash functions*. Dr. Dobb's Journal, September 1997^{[7]} - Donald E. Knuth (
**1998**).*The Art of Computer Programming*. Volume 3 - Sorting and Searching, 6.4 about universal hashing, (second edition), amazom - Dennis Breuker (
**1998**).*Memory versus Search in Games*. Ph.D. Thesis, Universiteit Maastricht, The Netherlands. ISBN 90-9012006-8. - Charles E. Leiserson, Harald Prokop, Keith H. Randall (
**1998**).*Using de Bruijn Sequences to Index a 1 in a Computer Word*, pdf » BitScan

## 2000 ...

- Robert Hyatt, Tim Mann (
**2002**).*A lock-less transposition table implementation for parallel search chess engines*. ICGA Journal, Vol. 25, No. 1 » Shared Hash Table - Lock-less - Robert Hyatt, Anthony Cozzie (
**2005**).*The Effect of Hash Signature Collisions in a Chess Program*. ICGA Journal, Vol. 28, No. 3 - Sam Tannous (
**2007**).*Avoiding Rotated Bitboards with Direct Lookup*. ICGA Journal, Vol. 30, No. 2, pdf » Hashing Dictionaries - Kumar Chellapilla, Anton Mityagin, Denis Xavier Charles (
**2007**).*GigaHash: scalable minimal perfect hashing for billions of urls*. WWW 2007 - Trevor Fenner, Mark Levene (
**2008**).*Move Generation with Perfect Hashing Functions.*ICGA Journal, Vol. 31, No. 1, pdf » Congruent Modulo Bitboards

## 2010 ...

- Maurizio Monge (
**2010**).*On perfect hashing of numbers with sparse digit representation via multiplication by a constant*. arXiv:1003.3196 » Magic Bitboards - Jiao Wang, Si-Zhong Li, Xin-He Xu (
**2010**).*A Minors Hash Table in Chinese-Chess Programs*. ICGA Journal, Vol. 33, No. 1 - Mihai Pătrașcu, Mikkel Thorup (
**2011**).*The Power of Simple Tabulation Hashing*. arXiv:1011.5200v2 - Dan Anthony Feliciano Alcantara (
**2011**).*Effcient Hash Tables on the GPU*. Ph.D. thesis, University of California, Davis, pdf » GPU - John Tromp (
**2014**).*Cuckoo Cycle: a memory-hard proof-of-work system*. IACR Cryptology ePrint Archive^{[8]}^{[9]} - David Eppstein, Michael T. Goodrich, Michael Mitzenmacher, Paweł Pszona (
**2014**).*Wear Minimization for Cuckoo Hashing: How Not to Throw a Lot of Eggs into One Basket*. arXiv:1404.0286 - Thomas Dybdahl Ahle, Martin Aumüller, Rasmus Pagh (
**2016**).*Parameter-free Locality Sensitive Hashing for Spherical Range Reporting*. arXiv:1605.02673^{[10]} - Thomas Dybdahl Ahle (
**2017**).*Optimal Set Similarity Data-structures Without False Negatives*. Master thesis, University of Copenhagen, pdf^{[11]}

# Forum Posts

## 1998 ...

- Fast hash algorithm by John Scalo, CCC, January 08, 1998 » Zobrist Hashing
- Fast hash key method - Revisited! by John Scalo, CCC, January 14, 1998
- Hash tables and data-cache, some programmer stuff... by Ed Schröder, CCC, January 17, 1998

## 2000 ...

- A fast hash -- assuming you are not planning to do incremental updates by Dann Corbit, CCC, June 15, 2001
- Non power of two hash table sizes by Alvaro Jose Povoa Cardoso, CCC, February 18, 2002

## 2005 ...

- What is the Difference between Mainhash, Evalhash, Pawnhash? by Thomas Wallendik, Hiarcs Forum, October 13, 2007
- Cache pollution when reading/writing hash table by Marco Costalba, CCC, August 09, 2009

## 2010 ...

- Cache-friendier material index by Harm Geert Muller, CCC, March 31, 2010
- how to measure frequency of hash collisions by Daniel Shawul, CCC, June 16, 2012 » Key Collisions, Transposition Table
- Hashing based on move lists by Matthew R. Brades, CCC, June 30, 2013

## 2015 ...

- Hash cache by Harm Geert Muller, CCC, October 12, 2015 » Transposition Table
- On-the fly hash key generation? by Evert Glebbeek, CCC, January 12, 2016
- Crafty's four hash tables by Louis Zulli, CCC, January 17, 2016 » Crafty
- Hashed repetition table by J. Wesley Cleveland, CCC, September 04, 2016 » Repetition Hash Table
- Hashing a quadboard from scratch by Edoardo Manino, CCC, November 23, 2016 » Quad-Bitboards
- Magic end-game material hash? by Harm Geert Muller, CCC, November 30, 2017
- hashing in chess4j by James Swafford, CCC, December 30, 2017 » chess4j, Transposition Table

# External Links

- Hash Table from Wikipedia
- Hash function from Wikipedia
- Perfect hash function from Wikipedia
- Cryptographic hash function from Wikipedia

- gperf - GNU Project - Free Software Foundation
- Transposition table from Wikipedia
- Tabulation hashing from Wikipedia
- Zobrist hashing from Wikipedia
- Cuckoo hashing from Wikipedia
- Distributed hash table from Wikipedia

- Koorde based on Chord and De Bruijn Sequence

- Transposition-driven scheduling - Wikipedia
- Hashlife from Wikipedia by Bill Gosper
^{[12]} - Integer Hash Function by Thomas Wang
- Hash functions by Paul Hsieh
- Hash Functions and Block Ciphers by Bob Jenkins
- A Hash Function for Hash Table Lookup by Bob Jenkins
- Perfect Hashing by Bob Jenkins
- Jenkins hash function from Wikipedia
- Fowler–Noll–Vo hash function from Wikipedia
- General Purpose Hash Function Algorithms by Arash Partow
- Transposition Table and Zobrist Hashing by Adam Berent
- Hash Functions
- 7. Wahl einer geeigneten Hash-Funktion from Guugelhupf - Design und Implementation (German)
- Association P.C. - Soft Time in a Life Machine, Berliner Jazztage, November 7, 1971, NDR-RB-SFB 3 Broadcast, YouTube Video

# References

- ↑ Hash function from Wikipedia
- ↑ Hash function from Wikipedia
- ↑ Cache-friendier material index by Harm Geert Muller, CCC, March 31, 2010
- ↑ Hash function from Wikipedia
- ↑ Bloom filter from Wikipedia
- ↑ Merkle–Damgård construction from Wikipedia
- ↑ A Hash Function for Hash Table Lookup by Bob Jenkins, updated Dr. Dobb's article
- ↑ Cuckoo Cycle: a new memory-hard proof-of-work system by John Tromp, Bitcoin Forum, January 08, 2014
- ↑ Cuckoo hashing from Wikipedia
- ↑ Locality-sensitive hashing from Wikipedia
- ↑ MinHash from Wikipedia
- ↑ Gosper's Algorithm (Hashlife) explained