Menu
▾
▴
1 Attachments
#100 Support for UMAPINFO Rev 2.2
Specification:
https://github.com/kraflab/umapinfo/blob/master/docs/spec.md
The proposed implementation uses libdoom-umapinfo (written in C90) as parser.
An archive with a release candidate of the library is attached.
Running "make" should build a static library sufficient for testing.
For documentation point your browser to "doc/Documentation.html".
An internal representation of UMAPINFO data for the engine is implemented in "umapinfo.c".
The patches for the engine (SVN 1648) to use UMAPINFO data are contained in the archive "patches_1648.tar.bz2".
UMAPINFO is for Doom only and not loaded for other games like Heretic.
UMAPINFO supports at most 8 episodes for Doom, the engine is currently limited to 5 episodes (consistent with Doom + Thy Flesh Consumed + Sigil).
UMAPINFO supports unlimited map numbers. The engine is currently limited to the map numbers of the original games. The patches of this FR intentionally do not limit the map numbers, hopefully the limits of the engine can be raised (see FR #99).
Files.
Last edit: Michael Bäuerle 2023-04-03
Test WADs:
- For Ultimate Doom
"umapinfo1.wad" attached.
Uses episode crossing progression.
- For Doom II
"umapinfo2.wad" attached.
Real world WADs that use UMAPINFO:
- 2022 A Doom Odyssey (for Ultimate Doom)
https://www.doomworld.com/idgames/levels/doom/Ports/megawads/2022ado
The 5th episode should work, but levels ExM10 do not work because of engine
limitations unrelated to UMAPINFO.
- Solar Struggle (for Ultimate Doom)
https://www.doomworld.com/idgames/levels/doom/Ports/megawads/sstruggle
Last edit: Michael Bäuerle 2023-04-03
I downloaded your files.
I had already started with getting mapinfo from doomlegacy 2, but was interrupted.
I will have to reconcile the two sources.
For some keys, e.g. "intertext", the UMAPINFO specification 2.2 allows for multiple lines.
No formal syntax is given, only an example that uses multiple, comma separated, string values.
Some real world WADs use a single string value with a "\n" semantic for line breaks instead. Example:
https://www.doomworld.com/idgames/levels/doom2/Ports/megawads/poogers
The attached file "umapinfo.c" was modified to accept this semantic.
I have been working on this and having some difficulty. I did notice your name in several place, and I expect you may have spent some time developing this.
I am not sure the copyright is compatible with ours. I would prefer to be able to modify a copy of the code, adapt it to our ZMalloc, and the DoomLegacy coding.
I also have kept DoomLegacy compatible with many systems, some of which I really doubt have the meson compiling environment. I see no need to keep that part, and would just integrate the compilation in with the rest.
The use of yacc, Lex/Flex, Bison, seems to be overkill for such a simple syntax. I would prefer to take the output of those tools and just build the correct code, and avoid their run-time problems.
I have a project that I tried using those, and ended up creating a simple substitute tools so that I did not have to be stuck satisfying their requirements.
I was already working on integrating the mapinfo from legacy2. It has some problems of its own, which I would have to deal with. At least I would be able to use the same lexer that we already use, and would not have to support a second lexer. It is already compatible with our ZMalloc.
So I am debating much.
I feel that parts from each would be best. Would there be a copyright option that allowed modifying a copy of the library under GPL.
Some words about the library:
I have first taken a look at the reference implementation in PrBoom+.
It is written in C++ and I expected that you don't like this for Legacy 1 (I assume you would work with Legacy 2 code otherwise).
There is another implementation in Woof, that is written in C.
Maybe you should consider using this code, if you don't like my implementation.
Modifiying it for this project means that code for parsing UMAPINFO data already was implemented at least three times.
The idea for the library was to provide a documented API for such a parser and a documented syntax that it accepts.
The library is shipped with prebuild lexer and parser, this means flex and yacc are required only for development.
A C90 conformant compiler is sufficient to build the library.
The library has two builds systems:
A Makefile for POSIX make, similar to what Doom Legacy already uses.
This is intended for embedding the library. Import the code into a subdirectory, let the toplevel Makefile execute "make" in this subdirectory and link against the static library.
In this case the user need no separate installation of the library.
The Meson build system is intended to build and install an independent shared library. The toplevel Makefile checks for and links against the shared library.
If the library was not found, the feature (UMAPINFO support in this case) is not available.
This case is similar to e.g. how zlib is currently used.
For old systems without Meson, the Makefile of the library can install an independent static library too.
Some words about the legal stuff, that is always a bit difficult.
All the code was written by me, so I can change the license if you want that.
The BSD license should already be compatible with the GPL. I have used it to be more liberal than the GPL, not to be more restrictive.
The copyright is more difficult. E.g. here in Germany it is legally impossible to drop or transfer the property that you have created a work (even if you want that), you can only give permission to use it.
For the file "umapinfo.c" I have used "Copyright(C) 2023 by DooM Legacy Team", as this was already present in other files (even if I am no real member of this team). I consider this file as part of the patch for the engine, that is not large enough to justify different copyright or license terms.
For the library, as a separate work with a different license, I used my name for the copyright.
If you habe problems with the BSD license, I can dual license the library code for GPL too.
In Version 1.0.0rc4 of the library, the API function doom_umi1_register_mmanager() was exported with wrong prefix.
This is fixed in the attached version 1.0.0rc5.
The attached version of "umapinfo.c" let the library use zone memory of the engine.
Last edit: Michael Bäuerle 2023-08-25
I am programmer on the project, not owner of DoomLegacy. Messing with copyright issues would by necessity bring other people into this. That is good way to waste several months having endless discussions. The issue would lead me to use one of the alternatives.
From my understanding, you cannot make that version with that copyright cease to exist, as someone else may have relied upon what that copyright says.
However, by editing even a few lines, you can create a new work, and put any copyright you want on that new work. I believe you could also give permission to incorporate a copy and modify that copy under a different copyright.
It will take me some time to figure out what impact the rest of that has, as I have several other serious issues right now. It likely will take months to move and setup again.
Even if flex and yacc are not run by the compilation that I need to support for so many diverse systems, it sounds like they have they introduced code and requirements that have become problems. It was that way on the previous projects where I tried to use them, which led me to avoid them. Even though I am specifically designing a computer language, that compiler will not use them.
I would take the tables output and translate it into direct C-code, eliminating the need for supporting anything from flex and yacc. I have done that on my projects before.
How many keywords is it supporting, 20, and only a few unique syntax.
That translation is actually easier for me to do that than try to deal with general flex and yacc code. It uses what is already supported on all my target systems, I take no risks that way.
I have not looked in that library yet to see what support, just found the note in the DoomLegacy code talking about requiring a realloc in ZMalloc, so I cannot be more specific.
I have not looked at Prboom+ umapinfo, as that is now under the control of someone who has made it clear he does not approve of how DoomLegacy does things. That they are better positioned to capture the Windows users means that very many wads will be made that cater to their Doom ports, and that will continue.
Do not have Woof code.
I generally do not browse around in other port's code, except PrBoom (but not PrBoom+), Heretic, MBF, or eternity.
Last edit: Wesley Johnson 2023-08-26
No, my comment was about the law in some countries.
The final 1.0.0 release of the library (with BSD license) is now available here:
https://micha.freeshell.org/libdoom-umapinfo/
Attached is a version containing the same code, but without the copyright line and a literal copy of the GPLv2 license from Doom Legacy.
This version should allow you to rip out code pieces and merge them, without taking care of the BSD license.
Compatibility with old systems was intended from the start.
For this reason I have used Berkeley yacc instead of GNU bison for better portability.
Yesterday I have successfully tested that a really old Unix system from the last century (SuSE Linux 6.1, Kernel 2.2.7, GCC 2.91.66 and GNU make 3.76.1 ) can build the library version 1.0.0.
I don't know how you do the Windows builds. MSYS2 should provide an environment for POSIX make and Meson should support Windows directly.
Should be possible with the GPL version of the library code, see above.
The code generated by flex/yacc uses realloc(). Therefore the library only supports memory managers with reallocation capability for the API function doom_umi1_register_mmanager().
The zone memory of the Doom Legacy engine currently does not support Z_Realloc(), but maybe this API can be added.
If this is not desired, it already supports Z_Datasize(), that allows to register a wrapper function like UMI_Realloc(). See my updated file "umapinfo.c" for details.
This means it is already possible to use zone memory for the parser.
Thank you for that. I downloaded it. Probably explore that tomorrow.
Because of the way that library header does not give names about who is giving the copyright, I have to ask explicitly. Does this discussion explicitly cover the content of the library, or just the interface between the library and DoomLegacy, that extra code with your name on it ??
I still do not know if YOU wrote the library, or some other group, with some unknown relationship between you and library group. Everytime I think I got it, another sentence goes by that confuses me. So I ask explicitly.
As for the realloc. There is no need at all for the library to be using ZMalloc. ZMalloc is only needed by map based storage that is automatically destroyed when a map is exited. It is mostly pointless when used with STATIC. If something like the library has its own destruction, it can use alloc, and realloc, and not bother with ZMalloc.
Was working on the mapinfo code from DoomLegacy 2. Note that it is mapinfo, and NOT umapinfo. Do not know the implication of that yet. It is painful work, as it is all in C++ with doxygen, and it is impossible to track anything back to where it got used. Got to search around where Hexen stuff got inserted and follow it back to the classes.
The worst part was that I found some flex and yacc stuff in DoomLegacy 2. Uggh.
In a separate compilation module. Note, that this is in DoomLegacy 2 and there is NO corresponding support in DoomLegacy 1.
Still not worth the pain.
In the long run, want to get Hexen from the DoomLegacy 2 code and option it into DoomLegacy 1. Note that, as a code OPTION. If that is possible.
I also need to address the game mode problem, Doom1, Doom2, MBF, Heretic, not just being selected by which map got loaded. Have been considering some kind of "revert to Vanilla" switch for years, but cannot figure one that is safe to give to innocent users.
Any explicit control is going to conflict with automatic detection, and automatic detection will have to override explicit selections all the time, or else the player will constantly have some MBF mode selection they were using, trying to be active when they loaded a heretic map.
Not exactly keen on the idea of having a "revert to Vanilla" push button, that changes a bunch of other settings around. This should be a "revert to Vanilla" switch, that when active overrides their usual settings, and when deactivated, their usual settings return.
Last edit: Wesley Johnson 2023-08-29
All code proposed in this FR was written by me (the library itself, the glue code for the engine in "umapinfo.c" and the patches for the engine to use the UMAPINFO data).
Nobody else has a copyright on (parts of) this code.
You have explicit permission to cherry pick and integrate the code (including the library code) under the Doom Legacy license (GPLv2), as this seems what you prefer.
Optionally you can link against (a bundled or external copy of) the library as a separate work under BSD license. This was originally intended by me.
You are the maintainer of Doom Legacy, it is your choice. I don't want to stay in your way.
I have been working on this, for a few months.
I am reminded how much I dislike FLEX and YACC. I did try to just modify, but had to replace the lexer, and then could not do anything with the YACC, so in a couple days I wrote a new parser from your yacc sources.
I give you my permission to use any of my code in the doomlegacy umapinfo code in your library. I strongly urge you to abandon using YACC and FLEX as they produce horrible unmanageable code.
Last edit: Wesley Johnson 2023-10-25
Here is an overall tar file. It is based on SVN 1647.
Is there any reason that the parser makes an intermediary database. It looks to me like there is no interaction at all between umapinfo keyword assigns. The parser, at the end of each keyword statement, could just put the values into the final assignments and not allocate any value stacks, no keyword stacks, nor map stacks.
The existing value of any map keyword cannot be accessed by umapinfo, so there is no need for keeping that pristine.
All interactions in the umapinfo data is by a string, like "MAP3". The umapinfo does NOT check for existence of this map at parsing time, so why have this intermediate database.
The intermediate database does cause one fault.
If a map is mentioned 2 or more times in the umapinfo lump, I believe that any keyword assignments in both of the data blocks should become effective. I see nothing in the syntax that would imply that this is invalid, or that there would be data hiding. With this intermediate database, only the first map entry will be found when searching for the map. It is a detail of the append whether this only honors the first appearance of a MAP, or only the last appearance of a MAP.
The reason for the intermediate data is to provide a thread-safe API.
The parser itself is not thread-safe.
The library selects the last (not the first) entry, if there are multiple entries.
The UMAPINFO specification is not very precise what should happen in such cases.
I know, many sources tout thread safe as being some achievement. But, most programs, like DoomLegacy, and every other Doom port, are monolithic with highly global data structures. They have no good place to even run more than one thread.
The dehacked parsers are not going to get called as a separate thread, mostly because nothing else can run in parallel with them. All the dehacked parsing must be done in the correct sequence, and it is dependent upon the previous wad and the previous dechacked results. Setup is inherently serial.
I do not think that intermediate database accomplished anything in making it capable of being multi-threaded. The global database would still have to be locked during the modifications to the global database. It does not matter if it is direct from the UMAPINFO parsers, or some UMAPINFO intermediate database.
The UMAPINFO parsing is simple and direct, so it is already happening as fast as it could be done. The intermediate database is not speeding up the update of the Doom global data, but is actually making it slower, by having to repeatedly search the intermediate database, and having to check all these new possible database access errors. The lump representation seems to be a faster and easier source of UMAPINFO than the intermediate database. I cannot think of any other representation that would simplify or speed up any of the Doom global data changes that dehacked does. Just storing the parser output sequentially would be simpler for at least would not have to do all that searching, and it will not be changing the UMAPINFO results due to database collisions. But then why store it in intermediate, just apply it in global update and be done with it.
No database searching was required when writing parse results direct to the global data.
Meanwhile, absolutely everything in any Doom port would have to stop and wait for it to finish.
I have been fighting with another free game that did use multi-threading. It is horrible code, and it segfaults often, and I cannot find where it is corrupting the database. It segfaults in the default destructor of a particular data structure, and I have done everything I can think of to guard and detect anything, added canaries, added tracking tags. Only in the renderer would any Doom port find it worth while to deal with this multi-threading level of pain.
I am sorry your code got shredded so bad.
I removed the intermediate database, and finished cleaning up the connection to DoomLegacy.
The code has shrunk. The umi malloc is no longer needed, as nothing uses it anymore.
Most of the error conditions have disappeared, as most of them were caused by bad results from searching that intermediate database.
The parser now directly calls a routine in umapinfo.c that stores the umapinfo line direct to the final DoomLegacy umapinfo structure.
I pass a single line structure that allows up to 64 values on the line. The only thing that I can think of that would test that would be multiple-line text. I do not think there should be any longer than that because of the limit of screen size.
If there were any really long texts, I suppose it could be made longer.
It is only a temporary, and there is only one in existance, so it does not impact memory usage.
The parameter structure is fixed size, so no dynamic memory allocation is needed in the parser, nor lexer anymore. Without all that searching, all those error tests were eliminated.
I give you permission to use any of my umapinfo code in your library.
I strongly suggest you not use FLEX nor YACC, as those are tools for educational exploring of LR grammars. I know most people consider them not fit for use in any final product.
Sorry for the late answer. Thank you for merging UMAPINFO support!
I have done a quick test with 1.48.14 and it seems to work as expected.
I have not looked at the details yet.