Difference between revisions of "OpenBench"
GerdIsenberg (talk | contribs) |
GerdIsenberg (talk | contribs) |
||
| Line 2: | Line 2: | ||
'''OpenBench''',<br/> | '''OpenBench''',<br/> | ||
| − | an open source chess engine testing framework authored by [[Andrew Grant]], written in [[Python]] and licensed under the [[Free Software Foundation#GPL|GNU GPL]]. | + | an [[:Category:Open Source|open source chess engine]] testing framework authored by [[Andrew Grant]], written in [[Python]] and licensed under the [[Free Software Foundation#GPL|GNU GPL]]. |
It is inspired [[Stockfish#TestingFramework|Fishtest]] and makes use of [https://en.wikipedia.org/wiki/Distributed_computing distributed computing], | It is inspired [[Stockfish#TestingFramework|Fishtest]] and makes use of [https://en.wikipedia.org/wiki/Distributed_computing distributed computing], | ||
| − | allowing anyone to contribute [https://en.wikipedia.org/wiki/CPU_time CPU time] to further improve the development of chess engines. | + | allowing anyone to contribute [https://en.wikipedia.org/wiki/CPU_time CPU time] to further improve the development of open source chess engines. |
OpenBench uses the [https://en.wikipedia.org/wiki/Django_(web_framework) Django Web Framework] and [[Cute Chess|Cutechess]] and applies the [[Match Statistics#SPRT|sequential probability ratio test]] in engine versus engine matches, | OpenBench uses the [https://en.wikipedia.org/wiki/Django_(web_framework) Django Web Framework] and [[Cute Chess|Cutechess]] and applies the [[Match Statistics#SPRT|sequential probability ratio test]] in engine versus engine matches, | ||
| − | and supports [[Chess960]] <ref>[https://github.com/AndyGrant/OpenBench/blob/master/README.md OpenBench/README.md at master · AndyGrant/OpenBench · GitHub]</ref>. | + | and supports [[Chess960]]. OpenBench was the primary testing framework used for the development of [[Ethereal]] <ref>[https://github.com/AndyGrant/OpenBench/blob/master/README.md OpenBench/README.md at master · AndyGrant/OpenBench · GitHub]</ref>. |
| + | |||
| + | =Clients= | ||
| + | OpenBench client PCs need to be [https://en.wikipedia.org/wiki/POSIX POSIX] compliant - that is [[Windows]] clients require [https://en.wikipedia.org/wiki/MinGW MinGW] <ref>[https://github.com/AndyGrant/OpenBench#setting-up-the-client-for-windows GitHub - AndyGrant/OpenBench: OpenBench is a Distributed SPRT Testing Framework for Chess Engines - Setting Up The Client For Windows]</ref> - | ||
| + | to build chess engine binaries on the fly assuming that [https://en.wikipedia.org/wiki/GNU_Compiler_Collection gcc]/g++ and [https://en.wikipedia.org/wiki/Make_(software) make] are on the system path. | ||
| + | To get the sources, OpenBench will pull down a [https://en.wikipedia.org/wiki/ZIP_(file_format) zip file] from [https://en.wikipedia.org/wiki/GitHub GitHub]. | ||
| + | Further, the client PC requires [[Python|Python 3]] and finally the OpenBench Client software itself <ref>[https://github.com/AndyGrant/OpenBench/blob/master/Client/Client.py OpenBench/Client.py at master · AndyGrant/OpenBench · GitHub]</ref>. | ||
| + | The client is started with four arguments specifying the registered user name and password, the server [https://en.wikipedia.org/wiki/URL URL] and the number of [[Thread|threads]] used for engine matches <ref>[https://github.com/AndyGrant/OpenBench#running-the-client GitHub - AndyGrant/OpenBench: OpenBench is a Distributed SPRT Testing Framework for Chess Engines - Running The Client]</ref>: | ||
| + | python3 OpenBench.py -U username -P password -S http://chess.grantnet.us/ -T 4 | ||
=See also= | =See also= | ||
| + | * [[Ethereal]] | ||
* [[Stockfish#TestingFramework|Fishtest]] | * [[Stockfish#TestingFramework|Fishtest]] | ||
* [[Match Statistics#SPRT|SPRT]] | * [[Match Statistics#SPRT|SPRT]] | ||
| Line 21: | Line 30: | ||
* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=70383 Any testing framwork similair to Fishtest that can be run locally ?] by [[Mahmoud Uthman]], [[CCC]], April 02, 2019 | * [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=70383 Any testing framwork similair to Fishtest that can be run locally ?] by [[Mahmoud Uthman]], [[CCC]], April 02, 2019 | ||
* [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=74503&start=3 Re: Looking for automatic Engine Testing Software] by [[Jon Dart]], [[CCC]], July 19, 2020 | * [http://www.talkchess.com/forum3/viewtopic.php?f=7&t=74503&start=3 Re: Looking for automatic Engine Testing Software] by [[Jon Dart]], [[CCC]], July 19, 2020 | ||
| + | * [http://www.talkchess.com/forum3/viewtopic.php?f=2&t=75335 Final Release of Ethereal, V12.75] by [[Andrew Grant]], [[CCC]], October 09, 2020 » [[Ethereal]] | ||
=External Links= | =External Links= | ||
==OpenBench== | ==OpenBench== | ||
* [https://github.com/AndyGrant/OpenBench GitHub - AndyGrant/OpenBench: OpenBench is a Distributed SPRT Testing Framework for Chess Engines] | * [https://github.com/AndyGrant/OpenBench GitHub - AndyGrant/OpenBench: OpenBench is a Distributed SPRT Testing Framework for Chess Engines] | ||
| + | * [https://github.com/AndyGrant/OpenBench/blob/master/Client/Client.py OpenBench/Client.py at master · AndyGrant/OpenBench · GitHub] | ||
* [http://chess.grantnet.us/index/ OpenBench] | * [http://chess.grantnet.us/index/ OpenBench] | ||
==Misc== | ==Misc== | ||
Revision as of 14:34, 27 November 2020
Home * Engine Testing * OpenBench
OpenBench,
an open source chess engine testing framework authored by Andrew Grant, written in Python and licensed under the GNU GPL.
It is inspired Fishtest and makes use of distributed computing,
allowing anyone to contribute CPU time to further improve the development of open source chess engines.
OpenBench uses the Django Web Framework and Cutechess and applies the sequential probability ratio test in engine versus engine matches,
and supports Chess960. OpenBench was the primary testing framework used for the development of Ethereal [1].
Clients
OpenBench client PCs need to be POSIX compliant - that is Windows clients require MinGW [2] - to build chess engine binaries on the fly assuming that gcc/g++ and make are on the system path. To get the sources, OpenBench will pull down a zip file from GitHub. Further, the client PC requires Python 3 and finally the OpenBench Client software itself [3]. The client is started with four arguments specifying the registered user name and password, the server URL and the number of threads used for engine matches [4]:
python3 OpenBench.py -U username -P password -S http://chess.grantnet.us/ -T 4
See also
Forum Posts
- SPRT when not used for self testing by Andrew Grant, CCC, October 21, 2016 » SPRT
- A question about SPRT by Andrew Grant, CCC, December 25, 2016
- Testing A against B by playing a pool of others by Andrew Grant, CCC, June 24, 2017 » Match Statistics
- Basic automated testing by odomobo, CCC, September 28, 2018
- Re:Basic automated testing by Andrew Grant, CCC, September 30, 2018
- Any testing framwork similair to Fishtest that can be run locally ? by Mahmoud Uthman, CCC, April 02, 2019
- Re: Looking for automatic Engine Testing Software by Jon Dart, CCC, July 19, 2020
- Final Release of Ethereal, V12.75 by Andrew Grant, CCC, October 09, 2020 » Ethereal
External Links
OpenBench
- GitHub - AndyGrant/OpenBench: OpenBench is a Distributed SPRT Testing Framework for Chess Engines
- OpenBench/Client.py at master · AndyGrant/OpenBench · GitHub
- OpenBench
Misc
References
- ↑ OpenBench/README.md at master · AndyGrant/OpenBench · GitHub
- ↑ GitHub - AndyGrant/OpenBench: OpenBench is a Distributed SPRT Testing Framework for Chess Engines - Setting Up The Client For Windows
- ↑ OpenBench/Client.py at master · AndyGrant/OpenBench · GitHub
- ↑ GitHub - AndyGrant/OpenBench: OpenBench is a Distributed SPRT Testing Framework for Chess Engines - Running The Client
