#329 Create an automated tool to try to generate schemas

[Genreless]

Having an automated tool that can try to decode schemas would greatly cut down on the time taken to implement schema updates and support for new games. I have started upgrading the DbDecoding tool to follow a choice tree to try to find a functional schema for each table that fails decoding. After the tool starts being able to find functional schemas, it should be allowed to check the official modding tools to try to get column names, key columns, and column references.

[ole kristian homelien]

This is a really cool idea and I am actually working on something kinda related. How far along have you gotten?

Could you elaborate on:

it should be allowed to check the official modding tools to try to get column names, key columns, and column references.

Where is this data available?

[daniu]

You may want to look into https://sourceforge.net/p/packfilemanager/code/HEAD/tree/branches/table-decoder/DbDecoding/ where I started something similar.
The issue is that there is no one to one mapping between tables in the official editor files and the ones in the pack files - there are fields that don't go into the corresponding pack db files, and there are columns in the pack db files without corresponding sources in the editor files. I assume those fields are filled from other tables during pack compilation.
I'm pretty pessimistic about an automated mapping being possible, but you're free to try of course and I wish you all the success.

[ole kristian homelien]

Thanks for the info. I have some ideas on how to do it, especially for tables with large number of entires. I keep seeing references to offical editor files, but I am unsure what this really is? Where can I find more info about it or maybe download it?

[Genreless]

I just pushed the changes I had made to DbDecoding to the table-decoder branch. Unfortunately, I haven't finished either of the two new classes I was implementing to allow for the automation so they are currently mostly a reference for the route I was going and a partial implementation.

As for the official data, what I know of is in the Assembly Kit's files in the "assembly_kit\raw_data\db folder". This contains two kinds of helpful xml files: the table files, which contain the table data in readable text, and files in the format TWaD%tablename%.xml, which specify column formatting and attributes for the table. Where this isn't reliable is the Assembly Kit doesn't output the columns in the same order and it skips some columns in the final table in a way predictable from the data in the Assembly Kit, but not from what I have found in its external files. This means it should be possible to compare tables' contents to the raw files to find which columns are unique matches to help check that the automated system got the right format, pull the names of columns, their table lookup references, and which are key value columns for ~80% of the columns rather than doing them all by hand.

[ole kristian homelien]

Just wanted to share some progress of the new hex editor with automatic table deletection.

So far it support brute force decoding and I am working on adding a smarter brute force decoder using the meta data provided in TWaD files. I have also massivly increase the speed of table parsing and replaced the old hex edior with a new and fancier one.