25,161
edits
Changes
Created page with "'''Home * Programming * Debugging''' FILE:Max_und_moritz-05-12.gif|border|right|thumb|link=http://www.wilhelm-busch-seiten.de/werke/maxundmoritz/streich5...."
'''[[Main Page|Home]] * [[Programming]] * Debugging'''
[[FILE:Max_und_moritz-05-12.gif|border|right|thumb|link=http://www.wilhelm-busch-seiten.de/werke/maxundmoritz/streich5.html
| Hin und her und rundherum<br/>Kriecht es, fliegt es mit Gebrumm <ref>[https://en.wikipedia.org/wiki/Wilhelm_Busch Wilhelm-Busch], [http://www.wilhelm-busch-seiten.de/werke/maxundmoritz/streich5.html Max und Moritz - Fünfter Streich] from [http://www.wilhelm-busch-seiten.de/index.html Wilhelm-Busch Seiten von Jochen Schöpflin] (German)</ref> ]]
'''Debugging''' is a process of finding and reducing [[Engine Testing#bugs|bugs]] in a computer program. A [https://en.wikipedia.org/wiki/Debugger debugger], usually in software, allows to execute the program (debugee) under its control, to set [https://en.wikipedia.org/wiki/Breakpoint breakpoints], let the user step to single lines of his source or machine code, to inspect variables, memory and processor registers. Processors often provide special instructions for the purpose of debugging.
=x86 Breakpoints=
[[8086]], [[x86]] and [[x86-64]] have the [https://en.wikipedia.org/wiki/INT_%28x86_instruction%29#INT_3 int 3] one byte software [https://en.wikipedia.org/wiki/Interrupt interrupt] instruction with opcode 0xCC, which might be explicitly used, or implicitly in assertions. This instruction is also used, when setting a breakpoint from a debugger, where current opcode is (temporarily) replaced by the ''int 3'' opcode (0xCC), which when executed calls a special interrupt routine of the debugger or runtime system.
Breakpoint opcode may be inserted inside the code at compile time, for instance with x86 [[Assembly#InlineAssembly|inline assembly]] or compiler intrinsic like ''DebugBreak'' <ref>[http://msdn.microsoft.com/en-us/library/ms679297%28VS.85%29.aspx DebugBreak Function (Windows)] from [http://msdn.microsoft.com/en-us/default.aspx Develop for Windows 7 and Windows Vista | MSDN]</ref> :
<pre>
__asm int 3
</pre>
=Compiler Support=
Various compiler allow a special Debug build, which disables optimizations, default initialization of otherwise not initialized variables or memory, and/or enable runtime checking, like [https://en.wikipedia.org/wiki/Bounds_checking bounds checking] of [[Array|array]] access. Various [https://en.wikipedia.org/wiki/Integrated_development_environment integrated development environments] (IDE) provide an integrated debugger.
<span id="Search"></span>
=Debugging the Search=
A [[Recursion|recursive]] [[Search|search]] is quite hard to debug. Therefor chess programs may provide debug routines to use a sequence of certain [[Moves|moves]] or a [[Zobrist Hashing|zobrist key]] as a precondition to break the search if they occur. Here are [[Tord Romstad|Tord Romstad's]] suggestions in a reply to [[Patrice Duhamel]] <ref> [http://www.open-aurec.com/wbforum/viewtopic.php?f=4&t=5955&start=5 Re: Testing and debugging chess engines] by [[Tord Romstad]], [[Computer Chess Forums|Winboard Forum]], December 05, 2006</ref>:
* Extend the [[UCI]]/[[XBoard]] command set with a few commands of your own for use in debugging. In particular, it is useful to have a command for looking up the [[Root|current position]] in the [[Transposition Table|hash table]] and printing the information ([[Score|score]], [[Score#Type|score type]], [[Best Move|best move]] , etc). to the standard output. You can use this to browse the [[Search Tree|search tree]] after a search is finished. When you want to know why the program discarded some move, you can make the move and inspect the hash table entry for the corresponding position to find the score and refutation. I've found this to be a very valuable debugging technique, and even have a simple [[GUI]] app for browsing the tree (the GUI app communicates with the engine through pipes connected to the standard input and output).
* The technique above can be further enhanced by including lots of additional information in the hash table when debugging the program. I sometimes store complete [[Move List|move lists]] with information about [[Extensions|extension]], [[Reductions|reduction]] and [[Pruning|pruning]] decisions for each move in every transposition table entry. Of course this makes each entry huge and greatly slows down the search, but it can be useful when chasing bugs or looking for ways to make the search more efficient.
* Implement an [[MTD(f)]] search, even if you intend to use [[Principal Variation Search|PVS]]. MTD(f) is great for debugging the hash table; all sorts of obscure bugs which are very tricky to find in PVS or other conventional searches suddenly become easy to spot.
* Whenever you add some non-trivial new function to your program, try to write two versions: One which is very slow and stupid, but almost certainly correct, and one which is highly optimized. Verify on a huge number of positions that they give the same results. Remove the slow version only when you feel 100% sure that the fast version is correct.
* Always make [[Color Flipping#Debugging|symmetry tests]] when you add a new term to your [[Evaluation function|evaluation function]] .
* Run through a simple tactical test like [[Win at Chess|WAC]] at 5 seconds/move every time you change something important in your search. Don't try to optimize the results, but just make sure that the score doesn't suddenly drop dramatically.
* Check the quality of your [[Move Ordering|move ordering]] by measuring how often a [[Beta-Cutoff|beta cutoff]] occurs on the first move, and the frequencies with which the 1st, 2nd, 3rd, ... move turns out to be best at [[Node Types#PV|PV nodes]].
=See also=
* [[Bibob]]
* [[Engine Testing]]
* [[Engine Testing#bugs|Famous Bugs]]
* [[InBetween]]
* [[Logging]]
* [[Fidelity Electronics#SpracklensAppleICE|Spracklens debugging with Apple II ICE]]
=Publications=
* [[Zhonghua Yang]], [[Tony Marsland]] ('''1992'''). ''Global Snapshots for Distributed Debugging''. [http://www.informatik.uni-trier.de/~ley/db/conf/icci/icci1992.html#YangM92 ICCI 1992], pp. 436-440
* [[Zhonghua Yang]], [[Tony Marsland]] ('''1993'''). ''Distributed Debugging in the Large''. [http://webdocs.cs.ualberta.ca/~tony/RecentPapers/acsc17.pdf pdf]
* [[Chrilly Donninger]] ('''1999'''). ''Computer machen keine Fehler''. [[Computerschach und Spiele|CSS]] 2/99, [http://www.mustrum.de/chrilly/keine_fehler.pdf pdf] (German)
* [[Yngvi Björnsson]], [[Jónheiður Ísleifsdóttir]] ('''2006'''). ''Tools for debugging large game trees''. [http://www.computer-shogi.org/gpw/gpw11_e.html 11th Game Programming Workshop], [https://en.wikipedia.org/wiki/Hakone,_Kanagawa Hakone], [https://en.wikipedia.org/wiki/Japan Japan] » [[Search Tree]]
* [[Jónheiður Ísleifsdóttir]] ('''2007'''). ''GTQL: A Query Language for Game Trees''. M.Sc. thesis, [https://en.wikipedia.org/wiki/Reykjav%C3%ADk_University Reykjavík University], [http://www.ru.is/lisalib/getfile.aspx?itemid=9655 pdf]
* [[Jónheiður Ísleifsdóttir]], [[Yngvi Björnsson]]. ('''2008'''). ''[http://link.springer.com/chapter/10.1007/978-3-540-87608-3_20 GTQ: A Language and Tool for Game-Tree Analysis]''. [[CG 2008]], [http://www.ru.is/faculty/yngvi/pdf/IsleifsdottirB08.pdf pdf]
=Forum Posts=
==2000 ...==
* [https://www.stmintz.com/ccc/index.php?id=129676 Debug Help] by [[Georg von Zimmermann]], [[CCC]], September 16, 2000
* [https://www.stmintz.com/ccc/index.php?id=269311 Winboard.debug] by [[David Rasmussen]], [[CCC]], December 07, 2002 » [[WinBoard]]
* [https://www.stmintz.com/ccc/index.php?id=367469 bugs, Bugs and BUGS!] by [[Anastasios Milikas|milix]], [[CCC]], May 27, 2004
* [https://www.stmintz.com/ccc/index.php?id=400603 Q. Why might node count differ between DEBUG and RELEASE] by [[David B. Weller]], [[CCC]], December 13, 2004
==2005 ...==
* [https://www.stmintz.com/ccc/index.php?id=431392 General Tips and Tricks for debugging a search] by [[Eric Oldre]], [[CCC]], June 15, 2005
* [http://www.open-aurec.com/wbforum/viewtopic.php?f=4&t=5955 Testing and debugging chess engines] by [[Patrice Duhamel]], [[Computer Chess Forums|Winboard Forum]], December 03, 2006
: [http://www.open-aurec.com/wbforum/viewtopic.php?f=4&t=5955&start=5 Re: Testing and debugging chess engines] by [[Tord Romstad]], [[Computer Chess Forums|Winboard Forum]], December 05, 2006
==2010 ...==
* [http://www.talkchess.com/forum/viewtopic.php?t=39390 Debugging regression tests] by [[Onno Garms]], [[CCC]], June 16, 2011 <ref>[https://en.wikipedia.org/wiki/Regression_testing Regression testing from Wikipedia]</ref>
* [http://www.talkchess.com/forum/viewtopic.php?t=46968 DrMemory: memory debugger tool for Windows (and Linux)] by [[Martin Sedlak]], [[CCC]], January 22, 2013 » [[Memory]]
==2015 ...==
* [http://www.talkchess.com/forum/viewtopic.php?t=55578 OT: Finding the Line of the Assert Fail?] by [[Steve Maughan]], [[CCC]], March 07, 2015 » [[Maverick]]
* [http://www.talkchess.com/forum/viewtopic.php?t=59046 Best way to debug perft?] by Meni Rosenfeld, [[CCC]], January 25, 2016 » [[Perft]]
* [http://www.talkchess.com/forum/viewtopic.php?t=61551 Help with Debugging My Chess Engine] by Pranav Deshpande, [[CCC]], September 28, 2016
* [http://www.talkchess.com/forum/viewtopic.php?t=61565 Help with Debugging My Chess Engine - 2] by Pranav Deshpande, [[CCC]], September 30, 2016
* [http://www.talkchess.com/forum/viewtopic.php?t=63119 How to go about chasing a bug like this?] by [[Colin Jenkins]], [[CCC]], February 09, 2017
* [http://www.talkchess.com/forum/viewtopic.php?t=63454 How to find SMP bugs ?] by Lucas Braesch, [[CCC]], March 15, 2017 » [[Lazy SMP]]
* [http://www.talkchess.com/forum/viewtopic.php?t=65712 assert] by [[Folkert van Heusden]], [[CCC]], November 13, 2017 <ref>[https://en.wikipedia.org/wiki/Assertion_(software_development) Assertion (software development) from Wikipedia]</ref>
* [http://www.talkchess.com/forum/viewtopic.php?t=66124 How do I debug cutechess-cli engine input/output?] by [[Daniel Dugovic]], [[CCC]], December 25, 2017 » [[Cutechess-cli]]
* [http://www.talkchess.com/forum/viewtopic.php?t=66366 Debugging UCI engine] by Cadel Watson, [[CCC]], January 19, 2018 » [[InBetween]], [[UCI]]
=External Links=
* [https://en.wikipedia.org/wiki/Debug Debugging from Wikipedia]
* [https://en.wikipedia.org/wiki/Debugger Debugger from Wikipedia]
* [https://en.wikipedia.org/wiki/Assertion_(software_development) Assertion (software development) from Wikipedia]
* [https://en.wikipedia.org/wiki/Regression_testing Regression testing from Wikipedia]
* [http://www.gnu.org/software/gdb/ GDB: The GNU Project Debugger]
* [http://swik.net/gdb+eclipse gdb + eclipse - SWiK]
* [http://www.hex-rays.com/idapro/ IDA Pro Disassembler - multi-processor, windows hosted disassembler and debugger]
* [https://en.wikipedia.org/wiki/SoftICE SoftICE from Wikipedia]
* [http://research.microsoft.com/en-us/projects/chess/ CHESS - Microsoft Research] a tool for finding and reproducing [https://en.wikipedia.org/wiki/Unusual_software_bug Heisenbugs] in concurrent programs.
* [http://cm.bell-labs.com/cm/cs/tpop/debugging.html From Chapter 5, Debugging] excerpt from Chapter 5 of
: [https://en.wikipedia.org/wiki/Brian_Kernighan Brian W. Kernighan], [https://en.wikipedia.org/wiki/Rob_Pike Rob Pike] ('''1999'''). ''[https://en.wikipedia.org/wiki/The_Practice_of_Programming The Practice of Programming]''. [https://en.wikipedia.org/wiki/Addison-Wesley Addison-Wesley], ISBN: ISBN 0-201-61586-X
* [https://en.wikipedia.org/wiki/Electric_Fence Electric Fence (Memory Debugger) from Wikipedia] » [[Memory]]
=References=
<references />
'''[[Programming|Up one Level]]'''
[[FILE:Max_und_moritz-05-12.gif|border|right|thumb|link=http://www.wilhelm-busch-seiten.de/werke/maxundmoritz/streich5.html
| Hin und her und rundherum<br/>Kriecht es, fliegt es mit Gebrumm <ref>[https://en.wikipedia.org/wiki/Wilhelm_Busch Wilhelm-Busch], [http://www.wilhelm-busch-seiten.de/werke/maxundmoritz/streich5.html Max und Moritz - Fünfter Streich] from [http://www.wilhelm-busch-seiten.de/index.html Wilhelm-Busch Seiten von Jochen Schöpflin] (German)</ref> ]]
'''Debugging''' is a process of finding and reducing [[Engine Testing#bugs|bugs]] in a computer program. A [https://en.wikipedia.org/wiki/Debugger debugger], usually in software, allows to execute the program (debugee) under its control, to set [https://en.wikipedia.org/wiki/Breakpoint breakpoints], let the user step to single lines of his source or machine code, to inspect variables, memory and processor registers. Processors often provide special instructions for the purpose of debugging.
=x86 Breakpoints=
[[8086]], [[x86]] and [[x86-64]] have the [https://en.wikipedia.org/wiki/INT_%28x86_instruction%29#INT_3 int 3] one byte software [https://en.wikipedia.org/wiki/Interrupt interrupt] instruction with opcode 0xCC, which might be explicitly used, or implicitly in assertions. This instruction is also used, when setting a breakpoint from a debugger, where current opcode is (temporarily) replaced by the ''int 3'' opcode (0xCC), which when executed calls a special interrupt routine of the debugger or runtime system.
Breakpoint opcode may be inserted inside the code at compile time, for instance with x86 [[Assembly#InlineAssembly|inline assembly]] or compiler intrinsic like ''DebugBreak'' <ref>[http://msdn.microsoft.com/en-us/library/ms679297%28VS.85%29.aspx DebugBreak Function (Windows)] from [http://msdn.microsoft.com/en-us/default.aspx Develop for Windows 7 and Windows Vista | MSDN]</ref> :
<pre>
__asm int 3
</pre>
=Compiler Support=
Various compiler allow a special Debug build, which disables optimizations, default initialization of otherwise not initialized variables or memory, and/or enable runtime checking, like [https://en.wikipedia.org/wiki/Bounds_checking bounds checking] of [[Array|array]] access. Various [https://en.wikipedia.org/wiki/Integrated_development_environment integrated development environments] (IDE) provide an integrated debugger.
<span id="Search"></span>
=Debugging the Search=
A [[Recursion|recursive]] [[Search|search]] is quite hard to debug. Therefor chess programs may provide debug routines to use a sequence of certain [[Moves|moves]] or a [[Zobrist Hashing|zobrist key]] as a precondition to break the search if they occur. Here are [[Tord Romstad|Tord Romstad's]] suggestions in a reply to [[Patrice Duhamel]] <ref> [http://www.open-aurec.com/wbforum/viewtopic.php?f=4&t=5955&start=5 Re: Testing and debugging chess engines] by [[Tord Romstad]], [[Computer Chess Forums|Winboard Forum]], December 05, 2006</ref>:
* Extend the [[UCI]]/[[XBoard]] command set with a few commands of your own for use in debugging. In particular, it is useful to have a command for looking up the [[Root|current position]] in the [[Transposition Table|hash table]] and printing the information ([[Score|score]], [[Score#Type|score type]], [[Best Move|best move]] , etc). to the standard output. You can use this to browse the [[Search Tree|search tree]] after a search is finished. When you want to know why the program discarded some move, you can make the move and inspect the hash table entry for the corresponding position to find the score and refutation. I've found this to be a very valuable debugging technique, and even have a simple [[GUI]] app for browsing the tree (the GUI app communicates with the engine through pipes connected to the standard input and output).
* The technique above can be further enhanced by including lots of additional information in the hash table when debugging the program. I sometimes store complete [[Move List|move lists]] with information about [[Extensions|extension]], [[Reductions|reduction]] and [[Pruning|pruning]] decisions for each move in every transposition table entry. Of course this makes each entry huge and greatly slows down the search, but it can be useful when chasing bugs or looking for ways to make the search more efficient.
* Implement an [[MTD(f)]] search, even if you intend to use [[Principal Variation Search|PVS]]. MTD(f) is great for debugging the hash table; all sorts of obscure bugs which are very tricky to find in PVS or other conventional searches suddenly become easy to spot.
* Whenever you add some non-trivial new function to your program, try to write two versions: One which is very slow and stupid, but almost certainly correct, and one which is highly optimized. Verify on a huge number of positions that they give the same results. Remove the slow version only when you feel 100% sure that the fast version is correct.
* Always make [[Color Flipping#Debugging|symmetry tests]] when you add a new term to your [[Evaluation function|evaluation function]] .
* Run through a simple tactical test like [[Win at Chess|WAC]] at 5 seconds/move every time you change something important in your search. Don't try to optimize the results, but just make sure that the score doesn't suddenly drop dramatically.
* Check the quality of your [[Move Ordering|move ordering]] by measuring how often a [[Beta-Cutoff|beta cutoff]] occurs on the first move, and the frequencies with which the 1st, 2nd, 3rd, ... move turns out to be best at [[Node Types#PV|PV nodes]].
=See also=
* [[Bibob]]
* [[Engine Testing]]
* [[Engine Testing#bugs|Famous Bugs]]
* [[InBetween]]
* [[Logging]]
* [[Fidelity Electronics#SpracklensAppleICE|Spracklens debugging with Apple II ICE]]
=Publications=
* [[Zhonghua Yang]], [[Tony Marsland]] ('''1992'''). ''Global Snapshots for Distributed Debugging''. [http://www.informatik.uni-trier.de/~ley/db/conf/icci/icci1992.html#YangM92 ICCI 1992], pp. 436-440
* [[Zhonghua Yang]], [[Tony Marsland]] ('''1993'''). ''Distributed Debugging in the Large''. [http://webdocs.cs.ualberta.ca/~tony/RecentPapers/acsc17.pdf pdf]
* [[Chrilly Donninger]] ('''1999'''). ''Computer machen keine Fehler''. [[Computerschach und Spiele|CSS]] 2/99, [http://www.mustrum.de/chrilly/keine_fehler.pdf pdf] (German)
* [[Yngvi Björnsson]], [[Jónheiður Ísleifsdóttir]] ('''2006'''). ''Tools for debugging large game trees''. [http://www.computer-shogi.org/gpw/gpw11_e.html 11th Game Programming Workshop], [https://en.wikipedia.org/wiki/Hakone,_Kanagawa Hakone], [https://en.wikipedia.org/wiki/Japan Japan] » [[Search Tree]]
* [[Jónheiður Ísleifsdóttir]] ('''2007'''). ''GTQL: A Query Language for Game Trees''. M.Sc. thesis, [https://en.wikipedia.org/wiki/Reykjav%C3%ADk_University Reykjavík University], [http://www.ru.is/lisalib/getfile.aspx?itemid=9655 pdf]
* [[Jónheiður Ísleifsdóttir]], [[Yngvi Björnsson]]. ('''2008'''). ''[http://link.springer.com/chapter/10.1007/978-3-540-87608-3_20 GTQ: A Language and Tool for Game-Tree Analysis]''. [[CG 2008]], [http://www.ru.is/faculty/yngvi/pdf/IsleifsdottirB08.pdf pdf]
=Forum Posts=
==2000 ...==
* [https://www.stmintz.com/ccc/index.php?id=129676 Debug Help] by [[Georg von Zimmermann]], [[CCC]], September 16, 2000
* [https://www.stmintz.com/ccc/index.php?id=269311 Winboard.debug] by [[David Rasmussen]], [[CCC]], December 07, 2002 » [[WinBoard]]
* [https://www.stmintz.com/ccc/index.php?id=367469 bugs, Bugs and BUGS!] by [[Anastasios Milikas|milix]], [[CCC]], May 27, 2004
* [https://www.stmintz.com/ccc/index.php?id=400603 Q. Why might node count differ between DEBUG and RELEASE] by [[David B. Weller]], [[CCC]], December 13, 2004
==2005 ...==
* [https://www.stmintz.com/ccc/index.php?id=431392 General Tips and Tricks for debugging a search] by [[Eric Oldre]], [[CCC]], June 15, 2005
* [http://www.open-aurec.com/wbforum/viewtopic.php?f=4&t=5955 Testing and debugging chess engines] by [[Patrice Duhamel]], [[Computer Chess Forums|Winboard Forum]], December 03, 2006
: [http://www.open-aurec.com/wbforum/viewtopic.php?f=4&t=5955&start=5 Re: Testing and debugging chess engines] by [[Tord Romstad]], [[Computer Chess Forums|Winboard Forum]], December 05, 2006
==2010 ...==
* [http://www.talkchess.com/forum/viewtopic.php?t=39390 Debugging regression tests] by [[Onno Garms]], [[CCC]], June 16, 2011 <ref>[https://en.wikipedia.org/wiki/Regression_testing Regression testing from Wikipedia]</ref>
* [http://www.talkchess.com/forum/viewtopic.php?t=46968 DrMemory: memory debugger tool for Windows (and Linux)] by [[Martin Sedlak]], [[CCC]], January 22, 2013 » [[Memory]]
==2015 ...==
* [http://www.talkchess.com/forum/viewtopic.php?t=55578 OT: Finding the Line of the Assert Fail?] by [[Steve Maughan]], [[CCC]], March 07, 2015 » [[Maverick]]
* [http://www.talkchess.com/forum/viewtopic.php?t=59046 Best way to debug perft?] by Meni Rosenfeld, [[CCC]], January 25, 2016 » [[Perft]]
* [http://www.talkchess.com/forum/viewtopic.php?t=61551 Help with Debugging My Chess Engine] by Pranav Deshpande, [[CCC]], September 28, 2016
* [http://www.talkchess.com/forum/viewtopic.php?t=61565 Help with Debugging My Chess Engine - 2] by Pranav Deshpande, [[CCC]], September 30, 2016
* [http://www.talkchess.com/forum/viewtopic.php?t=63119 How to go about chasing a bug like this?] by [[Colin Jenkins]], [[CCC]], February 09, 2017
* [http://www.talkchess.com/forum/viewtopic.php?t=63454 How to find SMP bugs ?] by Lucas Braesch, [[CCC]], March 15, 2017 » [[Lazy SMP]]
* [http://www.talkchess.com/forum/viewtopic.php?t=65712 assert] by [[Folkert van Heusden]], [[CCC]], November 13, 2017 <ref>[https://en.wikipedia.org/wiki/Assertion_(software_development) Assertion (software development) from Wikipedia]</ref>
* [http://www.talkchess.com/forum/viewtopic.php?t=66124 How do I debug cutechess-cli engine input/output?] by [[Daniel Dugovic]], [[CCC]], December 25, 2017 » [[Cutechess-cli]]
* [http://www.talkchess.com/forum/viewtopic.php?t=66366 Debugging UCI engine] by Cadel Watson, [[CCC]], January 19, 2018 » [[InBetween]], [[UCI]]
=External Links=
* [https://en.wikipedia.org/wiki/Debug Debugging from Wikipedia]
* [https://en.wikipedia.org/wiki/Debugger Debugger from Wikipedia]
* [https://en.wikipedia.org/wiki/Assertion_(software_development) Assertion (software development) from Wikipedia]
* [https://en.wikipedia.org/wiki/Regression_testing Regression testing from Wikipedia]
* [http://www.gnu.org/software/gdb/ GDB: The GNU Project Debugger]
* [http://swik.net/gdb+eclipse gdb + eclipse - SWiK]
* [http://www.hex-rays.com/idapro/ IDA Pro Disassembler - multi-processor, windows hosted disassembler and debugger]
* [https://en.wikipedia.org/wiki/SoftICE SoftICE from Wikipedia]
* [http://research.microsoft.com/en-us/projects/chess/ CHESS - Microsoft Research] a tool for finding and reproducing [https://en.wikipedia.org/wiki/Unusual_software_bug Heisenbugs] in concurrent programs.
* [http://cm.bell-labs.com/cm/cs/tpop/debugging.html From Chapter 5, Debugging] excerpt from Chapter 5 of
: [https://en.wikipedia.org/wiki/Brian_Kernighan Brian W. Kernighan], [https://en.wikipedia.org/wiki/Rob_Pike Rob Pike] ('''1999'''). ''[https://en.wikipedia.org/wiki/The_Practice_of_Programming The Practice of Programming]''. [https://en.wikipedia.org/wiki/Addison-Wesley Addison-Wesley], ISBN: ISBN 0-201-61586-X
* [https://en.wikipedia.org/wiki/Electric_Fence Electric Fence (Memory Debugger) from Wikipedia] » [[Memory]]
=References=
<references />
'''[[Programming|Up one Level]]'''
