Changes

NNUE, introduced in 2018 by [[Yu Nasu]] <ref>[[Yu Nasu]] ('''2018'''). ''&#398;U&#1048;&#1048; Efficiently Updatable Neural-Network based Evaluation Functions for Computer Shogi''. Ziosoft Computer Shogi Club, [https://github.com/ynasu87/nnue/blob/master/docs/nnue.pdf pdf] (Japanese with English abstract)[https://github.com/asdfjkl/nnue GitHub - asdfjkl/nnue translation] </ref>,

and [[Kristallweizen]] <ref>[https://github.com/Tama4649/Kristallweizen/ GitHub - Tama4649/Kristallweizen: 第29回世界コンピュータ将棋選手権 準優勝のKristallweizenです。]</ref>. He YaneuraOu's author [[Motohiro Isozaki]] made an unbelievable prediction that NNUE can help to increase Stockfish strength by around 100 points, almost one year before revealing<ref>[http://yaneuraou.yaneu.com/2019/06/24/%E5%B0%86%E6%A3%8B%E3%82%BD%E3%83%95%E3%83%88%E9%96%8B%E7%99%BA%E8%80%85%E3%81%8Cstockfish%E3%81%AB%E8%B2%A2%E7%8C%AE%E3%81%99%E3%82%8B%E6%97%A5/ 将棋ソフト開発者がStockfishに貢献する日 The day when shogi software developers contribute to Stockfish] by [[Motohiro Isozaki]], June 2019</ref> <ref>[https://www.reddit.com/r/chess/comments/cltich/shogi_engine_developer_claims_he_can_make/ shogi engine developer claims he can make Stockfish stronger], [[Computer Chess Forums|Reddit]], August 2019</ref>.

The [[Neural Networks|neural network]] consists of four layers. The input layer is heavily overparametrized, feeding in the [[Board Representation|board representation]] for all king placements per side<ref>[http://www.talkchess.com/forum3/viewtopic.php?f=7&t=74531&start=1 Re: NNUE accessible explanation] by [[Jonathan Rosenthal]], [[CCC]], July 23, 2020</ref>.  ==HalfKP==The so called '''HalfKP''' structure consists of two halves covering input layer and first hidden layer, each half of the input layer associated to one of the two [[King|kings]], cross coupled with the side to move or not to move halves of the first hidden layer.For each either black or white king placement, the 10 none king pieces on their particular squares are the boolean {0,1} inputs, along with a relict from Shogi piece drop (BONA_PIECE_ZERO), 64 x (64 x 10 + 1) = 41,024 inputs for each half, which are multiplied by a 16-bit integer weight vector for 256 outputs per half, in total, 256 x 41,024 = 10,502,144 weights.As emphasized by [[Ronald de Man]] in a [[CCC]] forum discussion <ref>[http://www.talkchess.com/forum3/viewtopic.php?f=7&t=75506&start=7 Re: NNUE Question - King Placements] by [[Ronald de Man|syzygy]], [[CCC]], October 23, 2020</ref>, the input weights are arranged in such a way, that [[Color Flipping|color flipped]] king-piece configurations in both halves share the same index.However, and that seems also a relict from Shogi with its [https://en.wikipedia.org/wiki/Rotational_symmetry 180 degrees rotational] 9x9 board symmetry, instead of [[Vertical Flipping|vertical flipping]] (xor 56), [[Flipping Mirroring and Rotating#Rotationby180degrees|rotation]] is applied (xor 63) <ref>[https://github.com/official-stockfish/Stockfish/issues/3021 NNUE eval rotate vs mirror · Issue #3021 · official-stockfish/Stockfish · GitHub] by [[Terje Kirstihagen]], August 17, 2020</ref>. 

where only a tiny fraction of its neurons need to be considered <ref>[http://www.talkchess.com/forum3/viewtopic.php?f=7&t=74531&start=1 Re: NNUE accessible explanation] by [[Jonathan Rosenthal]], [[CCC]], July 23, 2020</ref>in case of none king moves. The remaining three layers with 2x256x32, 32x32 and 32x1 weights are computational less expensive, hidden layer 1 and 2 with layers apply a [https://en.wikipedia.org/wiki/Rectifier_(neural_networks) ReLu activation] <ref>[https://github.com/official-stockfish/Stockfish/blob/master/src/nnue/architectures/halfkp_256x2-32-32.h#L42 Stockfish/halfkp_256x2-32-32.h at master · official-stockfish/Stockfish · GitHub]</ref> <ref>[https://github.com/official-stockfish/Stockfish/blob/master/src/nnue/layers/clipped_relu.h#L82 Stockfish/clipped_relu.h at master · official-stockfish/Stockfish · GitHub]</ref>, best calculated using appropriate [[SIMD and SWAR Techniques|SIMD instructions]] performing fast [[Byte|8-bit]]/[[Word|16-bit]] integer vector arithmetic, like [[MMX]], [[SSE2]] or [[AVX2]] on [[x86]]/[[x86-64]], or, if available, [[AVX-512]].

NNUE layers in action <ref>Image courtesy Roman Zhukov, revised version of the image posted in [http://talkchess.com/forum3/viewtopic.php?f=2&t=74059&start=139 Re: Stockfish NN release (NNUE)] by Roman Zhukov, [[CCC]], June 17, 2020, labels corrected October 23, 2020, see [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=75506&start=1 Re: NNUE Question - King Placements] by [[Andrew Grant]], [[CCC]], October 23, 2020</ref> Explanation by [[Ronald de Man]] <ref>[http://www.talkchess.com/forum3/viewtopic.php?f=7&t=75506&start=9 Re: NNUE Question - King Placements] by [[Ronald de Man|syzygy]], [[CCC]], October 23, 2020</ref>, who did the Stockfish NNUE port to [[CFish]] <ref>[https://github.com/syzygy1/Cfish/blob/master/src/nnue.c Cfish/nnue.c at master · syzygy1/Cfish · GitHub]</ref>: The accumulator has a "white king" half and a "black king" half, where each half is a 256-element vector of 16-bit ints, which is equal to the sum of the weights of the "active" (pt, sq, ksq) features plus a 256-element vector of 16-bit biases.  The "transform" step of the NNUE evaluation forms a 512-element vector of 8-bit ints where the first half is formed from the 256-element vector of the side to move and the second half is formed from the 256-element vector of the other side. In this step the 16-bit elements are clipped/clamped to a value from 0 to 127. This is the output of the input layer.  This 512-element vector of 8-bit ints is then multiplied by a 32x512 matrix of 8-bit weights to get a 32-element vector of 32-bit ints, to which a vector of 32-bit biases is added. The sum vector is divided by 64 and clipped/clamped to a 32-element vector of 8-bit ints from 0 to 127. This is the output of the first hidden layer.  The resulting 32-element vector of 8-bit ints is multiplied by a 32x32 matrix of 8-bit weights to get a 32-element vector of 32-bit ints, to which another vector of 32-bit biases is added. These ints are again divided by 64 and clipped/clamped to 32 8-bit ints from 0 to 127. This is the output of the second hidden layer.  This 32-element vector of 8-bits ints is then multiplied by a 1x32 matrix of 8-bit weights (i.e. the inner product of two vectors is taken). This produces a 32-bit value to which a 32-bit bias is added. This gives the output of the output layer.

[[Hisayori Noda|Nodchip]] explained on [https://en The output of the output layer is divided by FV_SCALE = 16 to produce the NNUE evaluation.wikipedia.org/wiki/Discord_SF's evaluation then take some further steps such as adding a Tempo bonus (even though the NNUE evaluation inherently already takes into account the side to move in the "transform" step) and scaling the evaluation towards zero as rule50_count(software) Discord] on June 2020 <ref>[http://talkchess.com/forum3/viewtopicapproaches 50 moves.php?f=7&t=74780 Don't understand NNUE] by Lucasart, [[CCC]], August 14, 2020</ref>:

41,024 = 64 * 641=HalfKA==In subsequent Stockfish versions the network architecture was further improved by [[Tomasz Sobczyk]] et al. 64 comes from the number of the cells where king may exist. 641 The '''HalfKA''' architecture uses 12x64x64 = 64 * 5 * 2 + 1. 64 here comes from the number 45056 inputs for each of the cells where a piece other than king may exist. 5 is the number of 12 piece types other than king. 2 is the number times 64 squares for each of the colors64 own king squares, white times two, for both the side to move and black. 1 is a captured piece. "+ 1" is BONA_PIECE_ZERO. Here "bona" means "other side perspective, further using the [[BonanzaVertical Flipping|bonanzavertical flip]]" which is a popular computer [[Shogi|shogi]] engine. It introduced instead of the feature "p" for the first timeHalfKP rotate. BonaPieces are contained '''HalfKAv2''' as applied in Stockfish '''14''' saves some space considering the evalListking square redundancy using 11x64x64 = 45056 inputs per side, mapped to a 2x520 linear feature transformer <ref>[https://github. It is updated by Position::do_move() aom/official-stockfish/Stockfish/commit/e8d64af1230fdac65bb0da246df3e7abe82e0838 New NNUE architecture and Positionnet · official-stockfish/Stockfish@e8d64af · GitHub]</ref> <ref>[https://github.com/official-stockfish/Stockfish/pull/3474 Update default net to nn-8a08400ed089.nnue by Sopel97 · Pull Request #3474 · official-stockfish/Stockfish · GitHub]</ref> <ref>[https:undo_move()//github.com/glinscott/nnue-pytorch/blob/master/docs/nnue.md#halfkav2-feature-set HalfKAv2 feature set | nnue-pytorch/nnue.md at master · glinscott/nnue-pytorch · GitHub]</ref>, and used by NNUE further feeding 8x2 outputs of this feature transformer directly to calculate the network parameters between the input layer and the first hidden layeroutput for better learning of unbalanced material configurations <ref>[https://github.com/glinscott/nnue-pytorch/blob/master/docs/nnue. About md#a-part-of-the calculation, the following text will be helpful. This text is sent -feature-transformer-directly-forwarded-to RocketMiningPoo on Twitter. "We add -the i-th COLUMN output A part of the W{0} feature transformer directly forwarded to the z{0} for each i, where the ioutput | nnue-pytorch/nnue.md at master · glinscott/nnue-pytorch · GitHub]</ref>. Another improvement was using eight 512x2->16->32-th element is set to >1. And we subtract the ioutput sub-networks discriminated by (piece_count-th COULMN of 1) div 4 in the W{0} from the z{0} for each i, where the ito 7 range<ref>[https://github.com/glinscott/nnue-pytorch/blob/master/docs/nnue.md#multiple-psqt-outputs-and-multiple-subnetworks Multiple PSQT outputs and multiple subnetworks | nnue-pytorch/nnue.md at master · glinscott/nnue-th element is set to 0pytorch · GitHub]</ref>. This operation is "accumulate" in your question [[FILE:HalfKAv2." may be rightpng|none|border|text-bottom|658px]] HalfKAv2 architecture by [[Tomasz Sobczyk]] <ref>[https://user-images.githubusercontent.com/8037982/118656988-553a1700-b7eb-11eb-82ef-56a11cbebbf2. I hope that someone will double checkpng HalfKAv2.png] Image courtesy by [[Tomasz Sobczyk]]</ref>

All networks are Networks were built by some volunteers but not by any big community (differs from [[Leela Chess Zero]]) and can be , uploaded into [[Stockfish#Fishtest|Fishtest]] for testing. Networks with good test results will be are released officially on the Fishtest website <ref>[https://tests.stockfishchess.org/nns Neural Net download and statistics]</ref>with average speed of 2 weeks per network <ref>[https://github.com/official-stockfish/Stockfish/discussions/3628#discussioncomment-1047323 One year of NNUE.... · official-stockfish/Stockfish · GitHub] by [[Joost VandeVondele]], July 26, 2021</ref>. After long discussing the best way to publish networks with Stockfish <ref>[https://github.com/official-stockfish/Stockfish/issues/3030 Improve dealing with the default net? Issue ##3030 · official-stockfish/Stockfish · GitHub] by [[Joost VandeVondele]], August 19, 2020</ref>, the developing team decided to embed the default network into Stockfish binaries, making sure NNUE always works as well as bringing more convenience to users.

After long discussing In late 2020, [[Gary Linscott]] started an implementation of the best way to publish networks with Stockfish NNUE training in [https://en.wikipedia.org/wiki/PyTorch PyTorch] <ref>[http://www.talkchess.com/forum3/viewtopic.php?f=7&t=75724 Pytorch NNUE training] by [[Gary Linscott]], [[CCC]], November 08, 2020</ref> <ref>[https://github.com/officialglinscott/nnue-pytorch GitHub -stockfishglinscott/Stockfishnnue-pytorch: NNUE (Chess evaluation) trainer in Pytorch]</issues/3030 Improve dealing ref> using [[GPU]] resources to efficiently train networks. Further, the collaboration with the default net? Issue ##3030 · official[[Leela Chess Zero]] team in February 2021 <ref>[https://groups.google.com/g/fishcooking/c/AzYDbbv-stockfish/Coo Stockfish · GitHub13] by [[Joost VandeVondele]], August [[Computer Chess Forums|FishCooking]], February 19, 20202021 </ref>payed off, the developing team decided in providing billions of positions to embed train the default network into new networks <ref>[https://stockfishchess.org/blog/2021/stockfish-14/ Stockfish 14], The Stockfish binariesTeam, July 02, making sure NNUE always works as well as bringing more convenience to users2021</ref>.

=Rotation vs FlipElo gain=Since the 9x9 [[ShogiJoost VandeVondele]] board has created a centered king file and [[Castling|castling]] is not known in Shogi, [[Color Flipping|color flip]] versus[[Flipping Mirroring and Rotating#Rotationby180degrees|180 degree rotate]] differs in a [[Horizontal Mirroring|horizontal mirrored]] position from the other side's point of view, graph to show how Stockfish gains Elo with otherwise identical playing options. The NNUE is trained and probed from the side to move point of view, where the used 180 degree rotation (xor 63 instead of 56) to flip sides looks rather strange for chess after a year:<ref>[https://github.com/official-stockfish/Stockfish/blob/615d98da2447e79ceceae205e0cd4e878115acc3/srcdiscussions/types.h3628#L323 Stockfish/types.h at 615d98da2447e79ceceae205e0cd4e878115acc3 · officialdiscussioncomment-stockfish/Stockfish · GitHub]</ref>1047728 One year of NNUE. i.e color flipping the black king from e8 to d1 rather than e1. Does it consider castling short to the queen side? It is a little bit unclear, how that rotation rather than flip affects the playing strength <ref>[https://github.com/official-stockfish/Stockfish/issues/3021 NNUE eval rotate vs mirror · Issue #3021 · official-stockfish/Stockfish · GitHub] by [[Terje KirstihagenJoost VandeVondele]], August 17July 26, 20202021</ref> and whether NNUE for chess suffers from [[httpsFILE://enNNEUOneYearEloGain.wikipedia.org/wiki/Associative_visual_agnosia associative visual agnosiapng|none|border|text-bottom|1024px]].Maybe Fishtest needs to play many games with color-flipped openings, i.e. 1.e4 e5 and 1.e3 e5 2.e4, to look whether results differ or not.Anyway, a fix from rotate to flip has to be done from producer and consumer sides, and is likely to void some training sessions.

===January ...===

* [http://www.talkchess.com/forum3/viewtopic.php?f=2&t=74666 SF NNUE Problem] by [[Stephen Ham]], [[CCC]], August 03, 2020

* [http://www.talkchess.com/forum3/viewtopic.php?f=2&t=74705 What happens with my hyperthreading?] by [[Kai Laskos]], [[CCC]], August 06, 2020 » [[Thread]]* [http://www.talkchess.com/forum3/viewtopic.php?f=2&t=74722 Stockfish NNUE style] by Rowen, [[CCC]], August 08, 2020* [http://www.talkchess.com/forum3/viewtopic.php?t=74739 SF NNUE training questions] by [[Jouni Uski]], [[CCC]], August 10, 2020* [http://www.talkchess.com/forum3/viewtopic.php?f=2&t=74765 Progress of Stockfish in 6 days] by [[Kai Laskos]], [[CCC]], August 12, 2020

===September ...===* [https://groups.google.com/d/msg/fishcooking/TJHsiI61yQ4/liQoZ-AzAgAJ Stockfish 12] by [[Joost VandeVondele]], [[Computer Chess Forums|FishCooking]], September 02, 2020

* [http://computerchessccrl.orgchessdom.ukcom/ccrl/404/cgi/engine_details.cgi?match_length=30&each_game=1&print=Details&each_game=1&eng=Stockfish%2014%2064-bit%208CPU#Stockfish_14_64-bit_8CPU Stockfish 14 64-bit 8CPU] in [[CCRL|CCRL Blitz]]* [http://ccrl.chessdom.com/ccrl/404/cgi/engine_details.cgi?print=Details&each_game=1&eng=Stockfish%2012%2064-bit#Stockfish_12_64-bit Stockfish 12 64-bit] in [[CCRL|CCRL Blitz]]* [http://computerchess.org.uk/ccrl/404/cgi/engine_details.cgi?print=Details&each_game=1&eng=Stockfish%2BNNUE%20150720%2064-bit%204CPU#Stockfish%2BNNUE_150720_64-bit_4CPU Stockfish+NNUE 150720 64-bit 4CPU] in [[CCRL|CCRL Blitz]]

* [[:Category:Hiromi UeharaSenri Kawaguchi|HiromiSenri Kawaguchi]] - [https://en.wikipediaameliaray.org/wikinet/Spectrum_(Hiromi_album) Spectrumquarantuned The Quarantuned Music Festival], 2019May 2020, [https://en.wikipedia.org/wiki/YouTube YouTube] Video: {{#evu:https://www.youtube.com/watch?v=A8RCz_RoefMjO8VlSqw7do|alignment=left|valignment=top}}

[[Category:Hiromi UeharaSenri Kawaguchi]]