Crafty

From Chessprogramming wiki
Jump to: navigation, search

Home * Engines * Crafty

Crafty Owl [1]

Crafty, [2]
a portable open source engine supporting the Chess Engine Communication Protocol written by Robert Hyatt in ANSI C starting in the early 90s, loosely derived from Cray Blitz, winner of the 1983 and 1986 World Computer Chess Championships [3]. Crafty pioneered in using Rotated bitboards , parallel search and probing Nalimov Tablebases. It performs a principal variation search, null move pruning, LMR as well as a SEE swap algorithm for move ordering and to prune "bad" captures in quiescence search. In 2006/2007, Crafty switched from rotated to Magic bitboards [4], according to Robert Hyatt because it was not faster but simpler [5]. Crafty 25.1, released in October 2016, not only includes an increase in playing strength [6] but support for Syzygy bases by Ronald de Man aided by the coding contributions of Basil Falcinelli [7]. Crafty 25.3 features playing strength adjustment between 800 and 2600 Elo [8].

Team Members

as mentioned at WCRCC 2010 [9] and CCT15, 2013 [10]

Tournaments

Crafty participated at four World Microcomputer Chess Championships, the WMCCC 1996, WMCCC 1997, WMCCC 2000, and WMCCC 2001, three World Computer Chess Championships, the WCCC 2004, WCCC 2005 and WCCC 2006, the ACCA Americas' Computer Chess Championships, the ACCA World Computer Rapid Chess Championships, CCT and various other tournaments. Crafty won the CCT1 in 2000, CCT5 in 2003 and CCT6 in 2004. At the Fifth Annual ACCA Americas' Computer Chess Championships in 2010 Crafty was runner-up behind Thinker [11] .

Descriptions

1997

from the ICGA page [12] : 

Crafty is a "bitmapper" using 64 bit words to represent the chess board, along the lines of the famous Chess 4.x program from Northwestern University. It uses a traditional alpha/beta search with the PVS (null-window) enhancement, along with null-moves (R=2) and lots of search extensions including "fractional ply extensions" to drive the search deeper along interesting lines. It has a very simple quiescence search that only considers capture moves and is fairly selective about which captures are included. It does a full endpoint evaluation, with no root pre-processing nor incrementally updated scoring terms. It is currently about 37,000 lines of ANSI C with about 3,000 lines of that being evaluation.

Since Crafty uses bitmaps, much of the evaluation is significantly shorter than it would be in a more traditional (array-based) board representation, so that this 3,000 lines of code is somewhat misleading (for example, to ask "can this pawn run and promote before the opposing king can get there?" only takes one line of code in Crafty. It is still very fast, searching around 100,000 moves per second. At 60 seconds per move, it solves 297/300 of the Win At Chess tactical positions. It has a large opening database composed from 250,000 GM games, and uses 3-4-5 piece endgame databases during the search (not just at the root of the tree.) It has played over 100,000 games during the past two years, playing on various chess servers around the world, and has maintained ratings on these servers that are always near the very top.

2008

from a CCC post [13] : 

Crafty's basic search is pretty simple. Just PVS + null-move + LMR + check extension (all others have been removed after testing showed they hurt performance rather than helped), + simple q-search checks (I am playing with this as I write this however and it might change) + simple q-search. I've used killers and hashing since middle 70's so those are not new. Bitboards were around in the middle 70's so that's not new. I don't think there is anything remarkable in my evaluation, certainly nothing we were not doing 15 years ago or longer, other than tuning adjustments.

As far as tuning goes on LMR and null-move, I have not gone very far afield there. I use R=3 everywhere. LMR is a 1-ply reduction although I have plans to try a variable reduction so that for moves that look really ugly I reduce them even further (white playing Na1 for example, with king on the other side of the board, no passed pawns, etc...)

I have even run without the check extension, the only one that is left. Many think the purpose of this extension is to find deep mates. That's wrong. The purpose is to try to expose horizon-effect moves and avoid them when possible. I also want to test a restricted check extension, where it is only applied in the last N plies where the horizon effect is most notable, where right now I apply them everywhere with no limit, always +1 ply added to depth.

Generation II

Crafty 25.0, Generation II, December 2015 [14]

  • This version contains a major rewrite of the parallel search code, now referred to as Generation II. It has a more lightweight split algorithm, that costs the parent MUCH less effort to split the search. The key is that now the individual "helper" threads do all the work, allocating a split block, copying the data from the parent, etc., rather than the parent doing it all. Gen II based on the DTS "late-join" idea so that a thread will try to join existing split points before it goes to the idle wait loop waiting for some other thread to split with it. In fact, this is now the basis for the new split method where the parent simply creates a split point by allocating a split block for itself, and then continuing the search, after the parent split block is marked as "joinable". Now any idle threads just "jump in" without the parent doing anything else, which means that currently idle threads will "join" this split block if they are in the "wait-for-work" spin loop, and threads that become idle also join in exactly the same way. This is MUCH more efficient, and also significantly reduces the number of split blocks needed during the search. We also now pre-split the search when we are reasonably close to the root of the tree, which is called a "gratuitous split. This leaves joinable split points laying around so that whenever a thread becomes idle, it can join in at these pre-existing split points immediately. We now use a much more conservative approach when dealing with fail highs at the root. Since LMR and such have introduced greater levels of search instability, we no longer trust a fail-high at the root if it fails low on the research. We maintain the last score returned for every root move, along with the PV . Either an exact score or the bound score that was returned. At the end of the iteration, we sort the root move list using the backed-up score as the sort key, and we play the move with the best score. This solves a particularly ugly issue where we get a score for the first move, then another move fails high, but then fails low and the re-search produces a score that is actually WORSE than the original best move. We still see that, but we always play the best move now. One other efficiency trick is that when the above happens, the search would tend to be less efficient since the best score for that fail-high/fail-low move is actually worse than the best move/score found so far. If this happens, the score is restored to the original best move score (in Search()) so that we continue searching with a good lower bound , not having to deal with moves that would fail high with this worse value, but not with the original best move's value.
  • We also added a new method to automatically tune the new SMP parameters. The command is autotune and "help autotune" will explain how to run it.
  • In addition , we did a complete re-factor of pawn evaluation code. There were too many overlapping terms that made tuning difficult. Now a pawn is classified as one specific class, there is no overlap between classes, which simplifies the code significantly. The code is now easier to understand and modify. In addition, the passed pawn evaluation was rewritten and consolidates all the passed pawn evaluation in one place. The evaluation used to add a bonus for rooks behind passed pawns in rook scoring, blockading somewhere else, etc. All of this was moved to the passed pawn code to make it easier to understand and modify.
  • Added a limited version of late move pruning (LMP) for the last two plies. Once a set number of moves have been searched with no fail high, non-interesting moves are simply skipped in a way similar to futility pruning.
  • We had a minor change to history counters that now rely on a "saturating counter" idea. I wanted to avoid the aging idea, and it seemed to not be so clear that preferring history moves by the depth at which they were good was the way to go. I returned to a history counter idea I tested around 2005 but discarded, namely using a saturating counter. The idea is that a center value (at present 1024) represents a zero score. Decrementing it makes it worse, incrementing it makes it better. But to make it saturate at either end, I only reduce the counter by a fraction of its distance from the saturation point so that once it gets to either extreme value, it will not be modified further avoiding wrap-around. This basic idea was originally reported by Mark Winands in 2005. It seems to provide better results (slightly) on very deep searches. One impetus for this was an intent to fold this into a move so that I could sort the moves rather than doing the selection approach I currently use. However, this had a bad effect on testing, since history information is dynamic and is constantly changing, between two moves at the same ply in fact. The sort fixed the history counters to the value at the start of that ply. This was discarded after testing, but the history counter based on the saturating counter idea seemed to be OK and was left in even though it produced minimal Elo gain during testing.
  • We change to the way moves are counted, to add a little more consistency to LMR. Now Next*() returns an order number that starts with 1 and monotonically increases, this order number is used for LMR and such decisions that vary things based on how far down the move list something occurs. Root move counting was particularly problematic with parallel searching, now things are at least "more consistent". The only negative impact is that now the move counter gets incremented even for illegal moves, but testing showed this was a no-change change with one thread, and the consistency with multiple threads made it useful.
  • Added the "counter-move" heuristic for move ordering (Jos Uiterwijk, IJICCA) which simply remembers a fail high move and the move at the previous ply. If the hash, captures or killer moves don't result in a fail high, this move is tried next. No significant cost, seems to reduce tree size noticeably. Added a follow-up idea based on the same idea, except we pair a move that fails high with the move two plies back, introducing a sort of "connection" between them. This is a sort of "plan" idea where the first move of the pair enables the second move to fail high. The benefit is that this introduces yet another pair of good moves that get ordered before history moves, and is therefore not subject to reduction. I have been unable to come up with a reference for this idea, but I believe I first saw it somewhere around the time Fruit showed up, I am thinking perhaps in the JICCA/JICGA. Any reference would be appreciated.
  • A minor change to the way the PV and fail-hi/fail-low moves are displayed when pondering .
  • Crafty now adds the ponder move to the front of the PV enclosed in parentheses so that it is always visible in console mode. The depths are reaching such extreme numbers the ponder move scrolls off the top of the screen when running in console mode or when "tail -f" is used to watch the log file while a game is in progress. This is a bit trickier than you might think since Crafty displays the game move numbers in the PV.
  • The penalty for pawns on same color as bishop now only applies when there is one bishop.

Leading or Trailing Zeros

Crafty had always mapped square-index 0 to square 'a1', 7 to 'h1', and 63 to 'h8' respectively, but recently (crafty-20.6) reversed the bit-index versus square-index mapping from leading to trailing zero count based little-endian rank-file (LERF), using bitscan forward (LSB) to retrieve squares in a1-h8 order, as most often used in CPW bitboard samples.

LSB

Trailing zero count aka bitscan forward for non empty sets as used in bitboard serialization for move generation and evaluation purposes, is implemented with x86-64 bsf instruction via intrinsic or inline assembly if available (there are also 32-bit x86 bsf versions), and a conditional 16-bit byte lookup approach otherwise - Windows 64, Linux 64 and lookup versions with preprocessor instructions for conditional compiles omitted [15]: 

int LSB(BITBOARD arg1) {
  unsigned long index;
  if (_BitScanForward64(&index, arg1))
    return index;
  else
    return 64;
}

int static __inline__ LSB(long word)
{
  long dummy, dummy2;

asm("          bsfq    %1, %0     " "\n\t"
    "          jnz     1f         " "\n\t"
    "          movq    $64, %0    " "\n\t"
    "1:                           " "\n\t"
:   "=&r"(dummy), "=&r" (dummy2)
:   "1"((long) (word))
:   "cc");
  return (dummy);
}

unsigned char lsb[65536];

int LSB(BITBOARD arg1) {
  if ( arg1        & 65535) return (lsb[ arg1        & 65535]);
  if ((arg1 >> 16) & 65535) return (lsb[(arg1 >> 16) & 65535] + 16);
  if ((arg1 >> 32) & 65535) return (lsb[(arg1 >> 32) & 65535] + 32);
  return (lsb[arg1 >> 48] + 48);
}

Last One

Earlier Crafty versions prior to 20.6 had a leading zero count compliant, big-endian rank-file mapping. Left-bottom square (from White's point of view) 'a1' with square-index 0 was mapped to the leftmost, arithmetical most significant bit of an unsigned 64-bit integer with bit-index 63, while square 'h8' with square-index 63, was mapped to the rightmost, arithmetical least significant bit with bit-index 0. Bitscan forward and found index reversal was used in LastOne, to retrieve squares in h8-a1 order [16]: 

int LastOne(BITBOARD arg1)
{
  unsigned long index;
  if (_BitScanForward64(&index, arg1))
    return 63 - index;
  else
    return 64;
}

This one was found in 15.17 with Mac OS support [17] 

int LastOne(register BITBOARD a)
{
  register unsigned long i;
  if (i = a & 0xffffffff)
    return(__cntlzw(i ^ (i - 1)) + 32);
  if (i = a >> 32)
    return(__cntlzw(i ^ (i - 1)));
  return(64);
}

Selected Games

WCCC 2004

WCCC 2004, round 9, Falcon - Crafty [18] 

[Event "WCCC 2004"]
[Site "Ramat Gan, Israel"]
[Date "2004.07.11"]
[Round "9"]
[White "Falcon"]
[Black "Crafty"]
[Result "0-1"]

1.e4 e5 2.Nf3 Nc6 3.Bb5 Nf6 4.Nc3 Bd6 5.O-O O-O 6.d3 h6 7.Be3 a6 8.Ba4 b5 
9.Bb3 Na5 10.d4 exd4 11.Bxd4 Be7 12.e5 Ne8 13.Nd5 Nxb3 14.axb3 c5 15.Be3 
Bb7 16.b4 Bxd5 17.Qxd5 Nc7 18.Qb7 Rb8 19.Qe4 cxb4 20.Rfd1 Qc8 21.Nd4 Re8 
22.Nf5 Bf8 23.Qg4 Kh7 24.Bxh6 gxh6 25.Qh5 Kg8 26.Rd3 Re6 27.Rg3+ Rg6 28.Rxg6+ 
fxg6 29.Qxg6+ Kh8 30.Nxh6 Bxh6 31.Qxh6+ Kg8 32.Qg6+ Kf8 33.Rd1 Ne6 34.Rd3 
Ke7 35.Qf6+ Ke8 36.Rg3 Qc5 37.Rg8+ Nf8 38.Rg7 Rb6 39.Qf7+ Kd8  40.Rg8 Kc7 
41.Qxf8 Qxf8 42.Rxf8 a5 43.Ra8 a4 44.g3 Rb8 45.Ra7+ Rb7 46.Ra5 Kb6 47.Ra8 
Ra7 48.Rb8+ Kc6 49.Rc8+ Kd5 50.e6 dxe6 51.Rd8+ Kc5 52.Rc8+ Kb6 53.Re8 a3 
54.Rxe6+ Kc5 55.bxa3 0-1

WCCC 2006

WCCC 2006, round 8, Diep - Crafty [19] 

[Event "WCCC 2006"]
[Site "Turin, Italy"]
[Date "2006.05.30"]
[Round "8"]
[White "Diep"]
[Black "Crafty"]
[Result "0-1"]

1.e4 e5 2.Nf3 d6 3.d4 exd4 4.Nxd4 Nf6 5.Nc3 Be7 6.Bf4 O-O 7.Qd2 c6 8.O-O-O b5 
9.f3 b4 10.Nce2 c5 11.Nb3 Nc6 12.Bxd6 c4 13.Nc5 Qa5 14.Bxe7 Nxe7 15.Qd6 Nf5 
16.exf5 Qxa2 17.g4 a5 18.Rd4 Re8 19.Kd1 Qxb2 20.Nc1 b3 21.cxb3 cxb3 22.Bd3 a4 
23.Re1 Rxe1+ 24.Kxe1 Qxc1+ 25.Ke2 Bb7 26.Qd8+ Ne8 27.Qe7 Bc6 28.Bc4 Nf6 29.Bxf7+ 
Kh8 30.Nd3 b2 31.Rd8+ Rxd8 32.Qxd8+ Be8 33.Bxe8 Qc2+ 34.Ke3 Nd5+ 35.Kd4 Qc3+ 
36.Ke4 Qc7 37.Qxc7 Nxc7 38.Nxb2 a3 39.Bf7 axb2 40.Ba2 Nb5 41.Kd3 Na3 42.g5 b1=B+ 
43.Bxb1 Nxb1 44.h4 Kg8 45.h5 Kf8 46.f4 Ke7 47.Kd4 Nd2 48.Ke3 Nc4+ 49.Kd4 Nd6 
50.Ke5 Nf7+ 51.Kd5 h6 52.f6+ gxf6 53.g6 Nd8 54.f5 Nb7 55.Kc6 Na5+ 0-1

Copyright

Many programmers did not grasp Crafty's Copyright statement, but apparently took remarks by Robert Hyatt like in his reply to Ren Wu, January 26, 1999, concerning code reuse and not reinventing the wheel as alibi for their chess programming [20] : 

This is a basic tenet of software engineering called 'code reuse'. Why should I pay you to write something from scratch and take a year, if you can take something that exists and modify it to do the same thing in a month? And then I don't have as much trouble debugging and testing, since it is mostly already done...

that's not a bad side to this... Of course occasionally starting over is a good thing. But not starting from 'scratch'. IE if you don't know what has already been tried, you will re-invent the same bad wheels over and over and probably follow the same footsteps many before you did... software engineering wants to avoid that 'reinvention' problem...

Robert Hyatt further on the copyright problem [21] : 

My primary requirement is that if something is done to crafty to make it 'better', then that 'something' must be as public as the original code was. Because many have contributed bits and pieces... Eugene, George, Steffen, Mark, SJE, and many others that are to numerous to mention. Seems unfair that they modify what I did, then they make their stuff public, and then someone else takes _all_ of this and purports it to be 'original'.

I suppose it has to do with 'national morals' or whatever, ie the software piracy problem in China, to name but one.

Crafty Clones

See also

Publications

1997 ...

2000 ...

2010 ...

2020 ...

Forum Posts

1995 ...

2000 ...

2005 ...

2010 ...

2011

2012

2013

2014

2015 ...

2016

Re: Around Crafty dev. ... by Robert Hyatt, CCC, September 01, 2016

2017 ...

2020 ...

External Links

Chess Engine

Misc

References

  1. ↑ Two Girls Being Crafty: Owl Pin Cushion
  2. ↑ Re: Robert - How did you came up with the name Crafty by Robert Hyatt, CCC, June 16, 2011
  3. ↑ Crafty from Wikipedia
  4. ↑ Re: Fastest Magic Move Bitboard Generator ready to use by Robert Hyatt, Winboard Forum, November 09, 2006
  5. ↑ Re: BitBoard Tests Magic v Non-Rotated 32 Bits v 64 Bits by Robert Hyatt, CCC, August 25, 2007
  6. ↑ Crafty Chess managed by Tracy Riegle
  7. ↑ Crafty 25.1 Release by Michael B, CCC, October 04, 2016
  8. ↑ Crafty Play By Elo ( Crafty v25.3) by Michael B, CCC, January 23, 2017
  9. ↑ 2010 Fourth Annual ACCA World Computer Rapid Chess Championships - Participants
  10. ↑ CCT 15 Participants | CCT Events
  11. ↑ The 2010 Fifth Annual ACCA Americas' Computer Chess Championships - Results
  12. ↑ Crafty's ICGA Tournaments
  13. ↑ Re: Hardware vs Software - test result by Robert Hyatt, CCC, December 03, 2008
  14. ↑ Crafty 25.0 Release by Michael B, CCC, December 25, 2015
  15. ↑ Index of /downloads/source crafty-23.5.zip, inline64.h, boolean.c
  16. ↑ Index of /downloads/source, crafty-20.5.zip, boolean.c
  17. ↑ Index of /downloads/source, crafty-15.17.zip, boolean.c
  18. ↑ Ramat-Gan 2004 - Chess - Round 9 - Game 3 (ICGA Tournaments)
  19. ↑ Turin 2006 - Chess - Round 8 - Game 5 (ICGA Tournaments)
  20. ↑ Re: Bionic v Crafty - a possible solution by Robert Hyatt, CCC, January 26, 1999
  21. ↑ Re: crafty copyright problem by Robert Hyatt, rgcc, February 17, 1999
  22. ↑ Computers choose: who was the strongest player?, ChessBase News, October 30, 2006
  23. ↑ Computer analysis of world champions by Søren Riis, ChessBase News, November 02, 2006
  24. ↑ MTD(f) experiments with Crafty by Eric Stock, CCC, December 18, 2009 » MTD(f)
  25. ↑ Eric Stock, David J. King (2010). A new enhancement to MTD(f). Games and Arts, Abertay University
  26. ↑ Vafra by Robert Jurjević

Up one level

Retrieved from "https://www.chessprogramming.org/index.php?title=Crafty&oldid=24567"