Aubrey McAuley - 2006-11-21

I've posted something along these same lines in an ESF thread already, but I'm posting here a more technical description to get more technical feedback on the feasibility of my idea.

Here's the thread: http://www.elderscrolls.com/forums/index.php?showtopic=593155

The are two somewhat distinct but related concepts here:

1) A plugin splitter -- automatically splits up an ESP or ESM based on which records are modifying a master and which records are original (i.e., new records). This is sort of the inverse of what ScripterRon's TES4 Plugin Utility does (merging plugins together).

2) A leveled list merger for multiple masters -- lets you select several ESM files containing items or creatures/NPCs and generate an ESP containing leveled lists for for those master files.

The Splitter

The criteria for the split is simple:

1) If a record in the mod has a local FormID (i.e., it is not modifying a master record), then it should be written to a new master file (say "mymod.esm").

2) If a record in the mod has a foreign FormID (i.e., it is a modification to a record in a master file, such as Oblivion.esm) then it should be written out to an ESP (say "mymod.esp"). This new ESP should be made dependent on all masters it references (i.e., "mymod.esm" and "Oblivion.esm").

This tool is needed to resolve problems with dirty masters. Eh? What's that? Let me explain.

It seems that the ESM format is being used in many cases without regard for whether the mod changes records in a master file or not. The reasons for this are fairly obvious: it's easy to do with current tools, it works just fine in-game and usually works fine in the CS, and it's the easy way out of problems where you're trying to add original content to vanilla objects (such as leveled lists). Modders typically start out by building an ESP that includes changes to Oblivion.esm and then later decide they want to convert their mod to a master for whatever reason (size limits, extensibility, and compatibility are the only good reasons that come to mind). We're using fan-made tools to do this because Bethesda removed it from the CS, but our fan-made tools don't pay any heed to the design methodology Bethesda intended for the ESM/ESP structure, and as a result we're seeing problems in the CS when we try to use these hacked master files.

Most of the time, we're converting an entire ESP into an ESM, when the truth is that there is absolutely no benefit to be gained for many (maybe even most) of the records in that mod since they are merely changes to existing master records in another ESM rather than new content. In fact, the only thing we gain by doing this is a headache for anyone who then wants to use this shiny new master content.

If we really do want other modders to be able to build on, reuse, expand, and enhance our new content, then putting it in an ESM is the way to go. But if we also muddy up the ESM with duplicate records from another master (especially Oblivion.esm), then we're shooting ourseves in the foot at the same time.

It's quite understandable given the limits of the tools we have available, but it doesn't have to be this way. We could and should create tools that do a much better job of making a master -- by separating the new records from the change records, putting the new records in an ESM and putting the change records where they belong, in an ESP.

Of course, it is quite feasible to make and use clean masters right now. In fact, the only mods that should have any real problem with this are ones make large-scale changes to Oblivion.esm. It's quite feasible to make a bunch of new creatures or items, new leveled lists to contain them, and new spawn points for them to appear (even in the existing Tamriel worldspace), then convert all this to a master. This would be a clean master. Even though it is referencing Oblivion.esm, it's only adding new content. However, as soon as we want to add new creatures or lists to existing spawn points in Oblivion.esm, then we need to make these changes separately in an ESP that is tied to both masters.

However, this sort of design methodology requires a lot of planning, so while it can be done, it may not be reasonable to expect most mod-makers to have things thought out this well during the creative process of building a mod. Fortunately, it does seem completely feasible to make a tool which can solve the problem for us by sorting individual records into the appropriate place depending on whether they are new or not.

Some good examples of mods that currently suffer from this problem include Francesco's leveled creatures-items mod.esm and Mart's Monster Mod.esm, although MMM also includes a separate ESP file for leveled list changes. I'm guessing the same problem would be likely to occur in Oscuro's Oblivion Overhaul once it becomes available in ESM format. It also seems certain that many more mods will be made this way in the future if something isn't done to help with the situation.

Mixing changed content with new content in an ESM defeats the main purpose of using the ESM format in the first place, which is to let other mods use and/or change what you've created. I say this because the CS does not like it when you load two master files that both try to change the same things in another master. It will issue a warning and change the EditorID for one of these records (presumable the last one) to include the phrase "DUPLICATE" (I'm not sure what it does to the duplicate FormIDs).

This means that if I want to make a mod that uses objects from multiple masters, I'm going to run into problems if both of those masters make changes to the same records in a common master (i.e., Oblivion.esm) -- in other words, if both of the masters are "dirty". However, I'll be fine as long as at least one of the masters is "clean". A master should only be considered "clean" if it does not include any changes to another master. All such changes can and should be done in an ESP, which can reference both masters and resolve any integration issues between them (such as altering a leveled list from one master to include items created in the other master).

The List Merger

Once we have clean master/plugin pairs for our mods, suddenly it becomes potentially feasible to also create merged leveled lists for items and/or creatures/NPCs in those mods. This could go a long way toward solving one of the biggest drawbacks of oblivion: mod isolation.

The basic merging capability is already provided in TES4 Plugin Utility, but it is incomplete in several respects. Because it is normally used for merging typical ESPs that only have one master (Oblivion.esm), it seems to try to fix all other FormIDs without regard for whether they point to some other master. In other words, it specifically seems to lack support for handling multiple masters, and that is exactly what's most needed in this case.

This may sound like a wild fantasy, but I do think it is feasible to create a merged leveled lists tool like we had in Morrowind, allowing the user to actually specify which clean masters to draw from and automatically generate an ESP that uses all these masters and draws creatures or items from each.

The goal here would be twofold: let mod creators distribute their new items or creatures along with a template defining how the mod should be merged into the player's leveled lists, and then let the player easily do this merging after selecting a number of compliant item/creature mods to use. 

The exact blueprint for how this would work will need to be fleshed out, but the rough idea is to start with two master lists -- one for items and one for creatures/NPCs:

modname_itemsLL.esp
modname_creaturesLL.esp

These could initially be structured based on extracted copies of the vanilla lists, but most people would probably want to use a template designed more like Francesco's lists are (rare items stay rare even when your character reaches a high level).

If either of these templates is already available, it would be used in the merge. If the template is not available, then it would be extracted from mymod.esp (if the esp contains an such a list). If neither is found, the mod would not be included in the merge (based on the assumption that its items or creatures are meant to be unique, hand-placed objects). Of course, many players may devise their own merge templates for popular mods and share them.

Aside from all this planning and structure, the actual merger would really only require a straightforward merge that preserves the master-file relationships for each mod. Just to reiterate, this merger routine would only support mods that have already been cleaned using the splitter. It absolutely will not work for dirty mods or mods that only have an ESP.

What do you think about this idea?