Fuego Wiki

Brought to you by: broderic, enz, herzbube, mmueller65

AdditiveKnowledgeTypeComparison

Summary

This page contains the results of comparing the different types of additive knowledge used by Fuego's UCT search. At the time of writing, the trunk version of Fuego is at revision [r1718] and implements these types of additive knowledge

KNOWLEDGE_NONE
KNOWLEDGE_GREENPEEP
KNOWLEDGE_RULEBASED

Builds

The following builds were used for the tests:

Build 1
- Fuego trunk [r1711]
- This build was used before KNOWLEDGE_NONE was implemented in [r1715]
- Available knowledge types: KNOWLEDGE_GREENPEEP (0) and KNOWLEDGE_RULEBASED (1)
Build 2
- Fuego trunk [r1718]
- This build was used after KNOWLEDGE_NONE was implemented in [r1715]
- Available knowledge types: KNOWLEDGE_NONE (0), KNOWLEDGE_GREENPEEP (1) and KNOWLEDGE_RULEBASED (2)
Build 3
- Fuego 1.1

Test environment

Build configuration

Mac OS X 10.9
boost 1.55 (installed via Homebrew)
autoconf 2.69 (installed via Homebrew)
Xcode 5.0.2, Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn)

Runtime configuration

MacBook Pro 15-inch, Late 2013
2.6 GHz Intel Core i7
16 GB RAM

Test runs

Common settings

The following configuration settings apply for all test runs:

The default opening book is not used (--nobook)
1000 games are played to have a good sample
Komi 6.5
No handicap
To make the tests somewhat reproducible on different test systems with different CPU speeds, the limiting factor for playing strength is the number of simulations calculated per move ("uct_param_player max_games"). To eliminate the time factor, all tests set "uct_param_player ignore_clock 1": This guarantees that the full number of "max_games" is played out regardless of how fast the machine that runs the tests is. If "ignore_clock" were not set, a slower machine might hit the time limit (default is 10 seconds) before all games are played out.
All tests are conducted by gogui-twogtp from [GoGui] release 1.4.9
gogui-twogtp is run with the command line option -alternate
GNU Go 3.8 is used as the referee program, with this command line: gnugo --mode gtp --never-resign --chinese-rules --capture-all-dead

Run 1: Fast, 19x19

boardsize 19
uct_param_player max_games 1000
uct_param_player ignore_clock 1

Run 2: Fast, 9x9

boardsize 9
uct_param_player max_games 1000
uct_param_player ignore_clock 1

Run 3: Medium, 19x19

boardsize 19
uct_param_player max_games 10000
uct_param_player ignore_clock 1
uct_max_memory 3221225472

Run 4: Medium, 19x19

boardsize 9
uct_param_player max_games 10000
uct_param_player ignore_clock 1
uct_max_memory 3221225472

Run 5: Slow, 19x19

boardsize 19
uct_param_player max_games 25000
uct_param_player ignore_clock 1
uct_max_memory 3221225472

Run 6: Slow, 9x9

boardsize 9
uct_param_player max_games 25000
uct_param_player ignore_clock 1
uct_max_memory 3221225472

Test results

This matrix tells which result set to look at for a specific combination of configurations/builds.

                      KNOWLEDGE_NONE  KNOWLEDGE_GREENPEEP  KNOWLEDGE_RULEBASED  Fuego 1.1
-----------------------------------------------------------------------------------------
KNOWLEDGE_NONE        -               B                    C                    F
KNOWLEDGE_GREENPEEP   B               -                    A                    D
KNOWLEDGE_RULEBASED   C               A                    -                    E
Fuego 1.1             F               D                    E                    -

Full disclosure result details are available here: http://www.herzbube.ch/fuego/additive-knowledge-type-comparison

Results test set A

Player 1: Build 1, KNOWLEDGE_GREENPEEP
Player 2: Build 1, KNOWLEDGE_RULEBASED

                                       19x19 board                        9x9 board
                             wins by    wins by    unknown     wins by    wins by    unknown
                             greenpeep  rulebased  results     greenpeep  rulebased  results
============================================================================================
Fast (max_games 1000)
- Referee (GNU Go 3.8)       73.1%      26.9%      -           66.9%      33.1%      -
- Greenpeep bot              73.1%      26.9%      -           66.2%      33.8%      5
- Rulebased bot              73.1%      26.9%      -           66.3%      33.7%      5

Medium (max_games 10000)
- Referee (GNU Go 3.8)       66.0%      34.0%      -           66.3%      33.7%      -
- Greenpeep bot              66.0%      34.0%      -           66.3%      33.7%      -
- Rulebased bot              66.0%      34.0%      -           66.3%      33.7%      -
Note: Almost all games are won by resignation.

Slow (max_games 25000)
- Referee (GNU Go 3.8)       63.6%      36.4%      -           62.7%      37.3%      -
- Greenpeep bot              63.6%      36.4%      -           62.4%      37.6%      2
- Rulebased bot              63.6%      36.4%      -           62.5%      37.5%      2
Note: Almost all games are won by resignation.

Results test set B

Player 1: Build 2, KNOWLEDGE_GREENPEEP
Player 2: Build 2, KNOWLEDGE_NONE

                                        19x19 board                          9x9 board
                             wins by    wins by       unknown     wins by    wins by       unknown
                             greenpeep  no-knowledge  results     greenpeep  no-knowledge  results
==================================================================================================
Fast (max_games 1000)
- Referee (GNU Go 3.8)       73.6%      26.4%         -           69.5%      30.5%         -
- Greenpeep bot              73.6%      26.4%         1           68.5%      31.5%         1
- No-knowledge bot           73.6%      26.4%         1           68.7%      31.3%         1

Medium (max_games 10000)
- Referee (GNU Go 3.8)       71.9%      28.1%         -           70.2%      29.8%         -
- Greenpeep bot              71.9%      28.1%         1           69.9%      30.1%         -
- No-knowledge bot           71.9%      28.1%         -           69.9%      30.1%         -
Note: Almost all games are won by resignation.

Slow (max_games 25000)
- No tests run

Results test set C

Player 1: Build 2, KNOWLEDGE_RULEBASED
Player 2: Build 2, KNOWLEDGE_NONE

                                        19x19 board                          9x9 board
                             wins by    wins by       unknown     wins by    wins by       unknown
                             rulebased  no-knowledge  results     rulebased  no-knowledge  results
==================================================================================================
Fast (max_games 1000)
- Referee (GNU Go 3.8)       45.0%      55.0%         -           53.7%      46.3%         -
- Rulebased bot              45.0%      55.0%         -           53.7%      46.3%         6
- No-knowledge bot           45.0%      55.0%         -           53.9%      46.1%         6

Medium (max_games 10000)
- Referee (GNU Go 3.8)       62.3%      37.7%         -           52.4%      47.6%         -
- Rulebased bot              62.3%      37.7%         -           52.4%      47.6%         2
- No-knowledge bot           62.3%      37.7%         -           52.5%      47.5%         1
Note: Almost all games are won by resignation.

Slow (max_games 25000)
- No tests run

Results test set D

Player 1: Build 1, KNOWLEDGE_GREENPEEP
Player 2: Build 3 (Fuego 1.1)

                                       19x19 board                        9x9 board
                             wins by    wins by    unknown     wins by    wins by    unknown
                             greenpeep  fuego-1.1  results     greenpeep  fuego-1.1  results
============================================================================================
Fast (max_games 1000)
- Referee (GNU Go 3.8)       93.4%      6.6%       -           81.8%      18.2%      -
- Greenpeep bot              93.4%      6.6%       5           80.8%      19.2%      7
- Fuego 1.1 bot              93.4%      6.6%       5           80.9%      19.1%      7

Medium (max_games 10000)
- Referee (GNU Go 3.8)       83.0%      17.0%      -           79.5%      20.5%      -
- Greenpeep bot              83.0%      17.0%      -           79.6%      20.4%      -
- Fuego 1.1 bot              82.9%      17.1%      -           79.4%      20.6%      1
Note: Almost all games are won by resignation.

Slow (max_games 25000)
- No tests run

Results test set E

Player 1: Build 1, KNOWLEDGE_RULEBASED
Player 2: Build 3 (Fuego 1.1)

                                       19x19 board                        9x9 board
                             wins by    wins by    unknown     wins by    wins by    unknown
                             rulebased  fuego-1.1  results     rulebased  fuego-1.1  results
============================================================================================
Fast (max_games 1000)
- Referee (GNU Go 3.8)       82.3%      17.7%      -           65.1%      34.9%      -
- Rulebased bot              82.3%      17.7%      1           64.2%      35.8%      4
- Fuego 1.1 bot              82.3%      17.7%      1           64.0%      36.0%      5

Medium (max_games 10000)
- Referee (GNU Go 3.8)       83.6%      16.4%      -           68.3%      31.7%      -
- Rulebased bot              83.6%      16.4%      -           68.3%      31.7%      -
- Fuego 1.1 bot              83.4%      16.6%      -           68.1%      31.9%      2
Note: Almost all games are won by resignation.

Slow (max_games 25000)
- No tests run

Results test set F

Player 1: Build 1, KNOWLEDGE_NONE
Player 2: Build 3 (Fuego 1.1)

                                         19x19 board                          9x9 board
                             wins by       wins by    unknown     wins by       wins by    unknown
                             no-knowledge  fuego-1.1  results     no-knowledge  fuego-1.1  results
==================================================================================================
Fast (max_games 1000)
- Referee (GNU Go 3.8)       83.8%         16.2%      -           65.8%         34.2%      -
- No-knowledge bot           83.7%         16.3%      4           64.5%         35.5%      11
- Fuego 1.1 bot              83.7%         16.3%      4           64.6%         35.4%      12

Medium (max_games 10000)
- Referee (GNU Go 3.8)       69.0%         31.0%      -           62.1%         37.9%      -
- No-knowledge bot           69.0%         31.0%      -           61.7%         38.3%      -
- Fuego 1.1 bot              68.3%         31.7%      -           61.8%         38.2%      2
Note: Almost all games are won by resignation.

Slow (max_games 25000)
- No tests run

Commit: [r1711]
Commit: [r1715]
Commit: [r1718]
Wiki: ExperimentalResults
Wiki: GoGui