Recent changes to Database Tool

WikiPage Database Tool modified by Björn Ståhl

Björn Ståhl — Thu, 04 Oct 2012 14:35:21 -0000

--- v6
+++ v7
@@ -1,6 +1,6 @@
 This is about the little tool hidden away behind the 'Build DB' link in Windows, or, as it's known on the other platforms, arcan_romman.rb.
 
-This tool is a command-line controlled set of scripts that helps changing or generating the database in a safe-way, but also to try and reach a "zero config" state for setting up and embedding most emulators into Arcan.
+This tool is a command-line controlled set of scripts that helps changing or generating the database in a safe-way, but also to try and reach a "zero config" approach for setting up and embedding most emulators into Arcan.
 
 --------
 Basic Use
@@ -23,7 +23,7 @@
  --rompath pathname    -- point to the resources/games folder
  --targetpath pathname -- point to the resources/targets folder
 
-Depending on which importer scripts are installed (resourcepath/scripts/importers/*.rb) the *commandargs* will be populated with options that each script takes. The reason for the --targetpath is that some importer scripts (notably mame) need to run the target to extract metadata about the games it supports. 
+Depending on which importer scripts are installed (resourcepath/scripts/importers/*.rb) the *commandargs* will be populated with options that each script takes. The reason for the --targetpath is that some importer scripts (notably mame) need to run the target to extract metadata about the games it supports. Running the tool without any arguments should generate a full list of detected options.
 
 In the list both on generic- arguments and importer-specific ones, you'll find those matching --genargs, --genintargs, --genextargs (similarly, --mameargs, --mameintargs, etc.).
 
@@ -35,22 +35,33 @@
 [gamepath] -- expand to resourcepath/games.
 [themepath] -- expand to themepath/themename.
 
-What the generic- importer (the ones used for targets where a specialized one cannot be found) does, is that it simply matches the filename of the target (excluding extension) with a subdirectory in rompath, and each entry in there is added as a *setname*, with some simple heuristics to generate a title.
+--------
+Building a Database
+--------
+The most useful / common command is builddb. 
 
---------
-Example Commands
---------
+By default, this is a destructive command, it will **wipe** your database unless you explicitly tell it not to with the --update command.
 
-To use the default builddb configuration on win32,
-builddb --generic --mamegood --mameskipclone --mameshorttitle --dbname ./resources/arcandb.sqlite --rompath ./resources/games --targetpath ./resources/targets
+It will scan your *resources/targets* subfolder for valid targets (executables, and libretro- compatible shared libraries (.dll on windows, .so on BSD and Linuxes)
 
-the --generic will just map resources/targets/* to resources/games/* and strip extensions and special characters for game title.
+When a target has been found, it first asks all known importer scripts if it can handle the target in question and, if no importer can, use a generic fallback. 
 
- --mamegood, --mameskipclone --mameshorttitle just keeps part of the mame game db that are listed as good, not a clone and removes all (...) from the titles.
+The matching importer will then be fed with the entries from the subfolder in resources/games with a matching subdirectory in games. 
+
+In addition to the specialized arguments that each importer accepts, there are a few arguments specific to the builddb command, these are:
+
+--nogeneric (don't use the generic fallback at all)
+--update (only add / update metadata, don't delete anything)
+--skipgroup targetname (can be repeated, don't scan targetname)
+--scangroup targetname (can be repeated, only scan targetname)
+
+In addition, the generic fallback importer will probe libretro- cores for supported extensions to use as a filter, it will also try and map a few common target names (snes9x => Super Nintendo (SNES)) to system names for easier filtering. 
+Finally, It can be set to (--genscrape, --genscrapemedia) scan theGamesDB for additional metadata, and download screenshots, boxart etc.
 
 -------
 Internals
 -------
+(mainly for curious developers)
 
 the arcan_romman.rb is just a simple wrapper that prints usage instructions, parses arguments and tries to locate the other scripts.

WikiPage Database Tool modified by Björn Ståhl

Björn Ståhl — Mon, 23 Jul 2012 07:52:19 -0000

--- v5
+++ v6
@@ -31,9 +31,9 @@
 
 There are also special strings you can add to the arguments you want,
 [romset] -- expand to the name of the romset (filename) of the game being launched.
-[romsetfull] -- expand to resourcepath/games/targetname/romsetname  
-[gamepath] -- expand to resourcepath/games
-[themepath] -- expand to themepath/themename
+[romsetfull] -- expand to resourcepath/games/targetname/romsetname.
+[gamepath] -- expand to resourcepath/games.
+[themepath] -- expand to themepath/themename.
 
 What the generic- importer (the ones used for targets where a specialized one cannot be found) does, is that it simply matches the filename of the target (excluding extension) with a subdirectory in rompath, and each entry in there is added as a *setname*, with some simple heuristics to generate a title.

WikiPage Database Tool modified by Björn Ståhl

Björn Ståhl — Mon, 23 Jul 2012 07:51:54 -0000

--- v4
+++ v5
@@ -4,40 +4,42 @@
 
 --------
 Basic Use
-----
+--------
 
 The prerequisites to using this tool is a ruby1.9 interpreter, and the gems/addons nokogiri and sqlite3.
 
-./arcan_romman.rb *genericargs* **command** *commandargs*
+./arcan_romman.rb **command** **genericargs** **commandargs**
 
 where command can be:
- builddb     -- create a new database and populate with games.
- list        -- search for entries in the specified database
- alterdb     -- update, insert or delete entries
- exec        -- build the commandline for a game and launch it
+ *builddb*     -- create a new database and populate with games.
+ *list*        -- search for entries in the specified database
+ *alterdb*     -- update, insert or delete entries
+ *exec*        -- build the commandline for a game and launch it
 
-so the most important command is builddb, and then the others can be used to test and fix.
+so the most important command is *builddb*, and then the others can be used to verify and fix your configuration. 
 
 the genericargs are:
  --dbname filename     -- set the full path/filename for the output database
  --rompath pathname    -- point to the resources/games folder
  --targetpath pathname -- point to the resources/targets folder
 
-most of the commandargs are pretty self-explanatory, but there are some hidden parts that could use some additional information.
+Depending on which importer scripts are installed (resourcepath/scripts/importers/*.rb) the *commandargs* will be populated with options that each script takes. The reason for the --targetpath is that some importer scripts (notably mame) need to run the target to extract metadata about the games it supports. 
 
 In the list both on generic- arguments and importer-specific ones, you'll find those matching --genargs, --genintargs, --genextargs (similarly, --mameargs, --mameintargs, etc.).
 
 The --???args are always used when launching a target, the --???intargs are added only when the target is launched in internal mode, and the ???extargs are added only when the target is launched in external mode. 
 
 There are also special strings you can add to the arguments you want,
-[romset] -- expand to the name of the romset of the game being launched.
+[romset] -- expand to the name of the romset (filename) of the game being launched.
 [romsetfull] -- expand to resourcepath/games/targetname/romsetname  
 [gamepath] -- expand to resourcepath/games
 [themepath] -- expand to themepath/themename
 
+What the generic- importer (the ones used for targets where a specialized one cannot be found) does, is that it simply matches the filename of the target (excluding extension) with a subdirectory in rompath, and each entry in there is added as a *setname*, with some simple heuristics to generate a title.
+
 --------
 Example Commands
----
+--------
 
 To use the default builddb configuration on win32,
 builddb --generic --mamegood --mameskipclone --mameshorttitle --dbname ./resources/arcandb.sqlite --rompath ./resources/games --targetpath ./resources/targets
@@ -46,21 +48,9 @@
 
  --mamegood, --mameskipclone --mameshorttitle just keeps part of the mame game db that are listed as good, not a clone and removes all (...) from the titles.
 
---------
-Planned Features
----
-While this tool isn't highly prioritized compared to other parts of the project, there are a bunch of ideas that would be neat to toy around with, so if anyone is up for the task, and got some Ruby programming experience, feel free to contribute ;-)
-
-+ expand the mame scripts to include mess/ume
-+ add other popular emulators (supermodel for instance)
-+ expand the general importers to have a heuristic for stripping "file-naming"
-+ use online databases and checksumming to properly identify games and download metadata
-+ add features to populate resources (snapshots, screenshots, models etc.) 
-+ merge some of the edit functionality into the Welcome script and the arcan_db.c/arcan_lua.c parts of the core
-
 -------
 Internals
----
+-------
 
 the arcan_romman.rb is just a simple wrapper that prints usage instructions, parses arguments and tries to locate the other scripts.

WikiPage Database Tool modified by Björn Ståhl

Björn Ståhl — Mon, 02 Jul 2012 04:00:32 -0000

--- v3
+++ v4
@@ -1,6 +1,6 @@
 This is about the little tool hidden away behind the 'Build DB' link in Windows, or, as it's known on the other platforms, arcan_romman.rb.
 
-In its essence, this tool is a command- line controlled set of scripts that primarily helps manipulating the database in a safe-way (as to not cause inconsistencies that would make scripts stop working) but also to try and reach a "zero config" state for setting up and embedding most emulators into Arcan.
+This tool is a command-line controlled set of scripts that helps changing or generating the database in a safe-way, but also to try and reach a "zero config" state for setting up and embedding most emulators into Arcan.
 
 --------
 Basic Use

WikiPage Database Tool modified by Björn Ståhl

Björn Ståhl — Sat, 16 Jun 2012 01:25:52 -0000

--- v2
+++ v3
@@ -2,18 +2,51 @@
 
 In its essence, this tool is a command- line controlled set of scripts that primarily helps manipulating the database in a safe-way (as to not cause inconsistencies that would make scripts stop working) but also to try and reach a "zero config" state for setting up and embedding most emulators into Arcan.
 
+--------
 Basic Use
----
+----
 
 The prerequisites to using this tool is a ruby1.9 interpreter, and the gems/addons nokogiri and sqlite3.
 
-*arcan_romman.rb genericargs* **command** *commandargs*
+./arcan_romman.rb *genericargs* **command** *commandargs*
 
+where command can be:
+ builddb     -- create a new database and populate with games.
+ list        -- search for entries in the specified database
+ alterdb     -- update, insert or delete entries
+ exec        -- build the commandline for a game and launch it
 
+so the most important command is builddb, and then the others can be used to test and fix.
 
+the genericargs are:
+ --dbname filename     -- set the full path/filename for the output database
+ --rompath pathname    -- point to the resources/games folder
+ --targetpath pathname -- point to the resources/targets folder
+
+most of the commandargs are pretty self-explanatory, but there are some hidden parts that could use some additional information.
+
+In the list both on generic- arguments and importer-specific ones, you'll find those matching --genargs, --genintargs, --genextargs (similarly, --mameargs, --mameintargs, etc.).
+
+The --???args are always used when launching a target, the --???intargs are added only when the target is launched in internal mode, and the ???extargs are added only when the target is launched in external mode. 
+
+There are also special strings you can add to the arguments you want,
+[romset] -- expand to the name of the romset of the game being launched.
+[romsetfull] -- expand to resourcepath/games/targetname/romsetname  
+[gamepath] -- expand to resourcepath/games
+[themepath] -- expand to themepath/themename
+
+--------
 Example Commands
 ---
 
+To use the default builddb configuration on win32,
+builddb --generic --mamegood --mameskipclone --mameshorttitle --dbname ./resources/arcandb.sqlite --rompath ./resources/games --targetpath ./resources/targets
+
+the --generic will just map resources/targets/* to resources/games/* and strip extensions and special characters for game title.
+
+ --mamegood, --mameskipclone --mameshorttitle just keeps part of the mame game db that are listed as good, not a clone and removes all (...) from the titles.
+
+--------
 Planned Features
 ---
 While this tool isn't highly prioritized compared to other parts of the project, there are a bunch of ideas that would be neat to toy around with, so if anyone is up for the task, and got some Ruby programming experience, feel free to contribute ;-)
@@ -23,7 +56,9 @@
 + expand the general importers to have a heuristic for stripping "file-naming"
 + use online databases and checksumming to properly identify games and download metadata
 + add features to populate resources (snapshots, screenshots, models etc.) 
++ merge some of the edit functionality into the Welcome script and the arcan_db.c/arcan_lua.c parts of the core
 
+-------
 Internals
 ---

WikiPage Database Tool modified by Björn Ståhl

Björn Ståhl — Sat, 16 Jun 2012 00:48:02 -0000

--- v1
+++ v2
@@ -5,7 +5,10 @@
 Basic Use
 ---
 
-The prerequisites to using this tool is a ruby1.9 interpreter, the 
+The prerequisites to using this tool is a ruby1.9 interpreter, and the gems/addons nokogiri and sqlite3.
+
+*arcan_romman.rb genericargs* **command** *commandargs*
+
 
 
 Example Commands

WikiPage Database Tool modified by Björn Ståhl

Björn Ståhl — Sat, 16 Jun 2012 00:38:43 -0000

This is about the little tool hidden away behind the 'Build DB' link in Windows, or, as it's known on the other platforms, arcan_romman.rb. In its essence, this tool is a command- line controlled set of scripts that primarily helps manipulating the database in a safe-way (as to not cause inconsistencies that would make scripts stop working) but also to try and reach a "zero config" state for setting up and embedding most emulators into Arcan. Basic Use --- The prerequisites to using this tool is a ruby1.9 interpreter, the Example Commands --- Planned Features --- While this tool isn't highly prioritized compared to other parts of the project, there are a bunch of ideas that would be neat to toy around with, so if anyone is up for the task, and got some Ruby programming experience, feel free to contribute ;-) + expand the mame scripts to include mess/ume + add other popular emulators (supermodel for instance) + expand the general importers to have a heuristic for stripping "file-naming" + use online databases and checksumming to properly identify games and download metadata + add features to populate resources (snapshots, screenshots, models etc.) Internals --- the arcan_romman.rb is just a simple wrapper that prints usage instructions, parses arguments and tries to locate the other scripts. This is done by searching a few known paths (., ./scripts, ./resources/scripts, HOMEDIR/.arcan/resources/scripts, /usr/local/share/arcan/resources/scripts, /usr/share/arcan/resources/scripts ) for romman_base.rb Afterwards, it scans the subdir to romman_base called importers for .rb files and loads all of them, checks the contained classes for a few methods (set_defaults, accepted_arguments, check_games, check_target). the accepted arguments are added to the scanned arguments and the usage instructions. look at generic.rb and mame.rb for examples of importers.