Re: [Freedos-devel] New freeware DOS memory manager: EMS Magic

[Bret J. <bre...@ju...>]

>> the card was indeed needed for 8088/80286 to access any sort of memory
>> > 640KB.
>
> Not for an 80286, no. The 80286 could access 16MB of RAM and many '286
> PCs shipped with 1MB, meaning that they had 384kB of XMS out of the
> box.

In fact, the 286 had a Protected Mode, but it was 16-bit PM instead of 32-bit.  There is even an EMM286.EXE driver for machines with a 286 CPU floating around the Internet that basically does the same thing EMM386 from MS does (including providing virtualized EMS).  HIMEM from MS also works with 286 CPU's.

Unfortunately, PM in a 286 has some issues.  The story is told that Intel thought Protected Mode was so superior to Real Mode that no one would ever want to use Real Mode again (I don't know if that's actually true or not, but it sounds plausible based on what Intel did with the 286).  In the 286, Intel provided a method to switch the CPU into Protected Mode but didn't provide a way to switch it back into Real Mode again (386+ CPU's, of course, have a way to do it).  People found a way to work around this by triple-faulting the CPU and forcing it to reset (do a partial reboot) since the CPU always boots into Real Mode.  Unfortunately, this takes a long time (in computer terms) to do so switching back and forth between Real and Protected Mode on a 286 should only be done sparingly.

It also turns out that 16-bit protected mode is EXTREMELY useful, especially for things like TSR's.  AFAIK, all CPUs that support 32-bit PM also support 16-bit PM, and it's possible to take advantage of that fact to do some interesting things.  A little more on that further below.

Just as an FYI, in 16-bit PM you can access the entire 4 GB of addressable memory just like 32-bit PM, but a 16-bit selector is limited to a 64k chunk of memory just like a Real Mode segment register.  In one sense this is a disadvantage (you can't address all 4 GB memory with a single selector), but in another sense it's an advantage -- you can use almost the exact same code from 16-bit PM that you can from Real Mode (they are not exactly the same, but the differences are relatively easy to account for and work through).  That is, if you want the same code to be able to work from either Real Mode or 32-bit PM, you really can't -- 32-bit PM code is compiled (and even written) very differently than Real Mode code.  If you want the same code to work from either Real Mode or 16-bit PM, it's relatively easy to do.  This lets you write programs that can be used from either Real Mode or (16-bit) Protected Mode.  For "regular" programs, this is normally not a problem.  For TSR's and Device Drivers, it can be very beneficial.

>> For me personally was more important that EMS would take away 64kB of upper
>> memory, which could be used to load MS lanmanaged network drivers, and no
>> programs that I would use supported EMS anyway.
>
> This was my thinking in the way I set up the DOS USB key, but one of
> my first ever users to contact me disagrees. :-) / :-(

I used to think that, too, until I actually thought about things a little more deeply and did some research.

Let me tell you a little of my story over the last several years.  Some of you may find it interesting, and others probably will think I'm an idiot.  So be it.

Let me first address the comment about leaving room for MS LANManager (or other big TSR's) in Upper Memory.  If LANManager was designed to use EMS, that wouldn't be a problem.  It is true that EMS requires a 64k Page Frame, and if it isn't used for anything, it is a waste of memory.  But, the Page Frame is a "shared resource" for all the programs that do use it.  That is, you can have literally dozens of programs using EMS (including TSR's), altogether requiring multiple MB of memory, but the 64k Page Frame never gets any bigger.  The other programs switch what they need in and out of the Page Frame when they need it and when the memory is unused it is "hidden" from Conventional/Upper Memory.  In DOS, the important thing is to minimize CONVENTIONAL memory use.  The easiest (and most common) way to do this (at least with TSR's) is to use Upper Memory, but EMS let's you do even more.  If only one or a few small programs use EMS, the Page Frame does waste memory.  But if a large program or several programs use it, the Page Frame can effectively "multiply" the amount of memory it is "managing" to much more than size of the 64k Page Frame and Conventional/Upper Memory is not affected.

Now the story.  As many of you know, I published the first round of my USB Drivers about 15 years ago.  I have been slowly working on updates since then, but it is a long process with lots of distractions (any volunteers?).  As some of you are also aware, I believe my drivers are "better" than the others in several respects, the main ones being that they support hot-plugging and unplugging of USB devices and there is an extensibility API so that additional drivers for all kinds of USB devices can be written.  It is not a monolithic USB driver that only supports a few types of USB device the way the others do.  Needless to say, that's tough to do in a single-tasking environment like DOS.

As I started working on the next round of drivers, a few things bothered me.  One was that I had been using the A86 Assembler (which I had been using for a LONG time), and was having problems with its limitations.  So, I've been converting to NASM (that in itself is not a trivial process, especially for complicated programs like the USB drivers).  The other large issue was that the drivers use a LOT of memory and I wanted to figure out a way to reduce the memory use.

There are a few different ways to approach this problem.  The way most people seem to approach it is in one of two methods: they either "tweak" the code to try and optimize it for size, or they remove features and turn the program into a minimalistic one that doesn't do a whole lot of what it could, and probably should, do.  Optimizing the size can make a little bit of difference, but on programs as large as the USB drivers that's not going to save enough memory to make any difference worth discussing.  As far as removing features, that's not something I was willing to do either.  The features that are complicated enough to make any real difference in size (like hot-plugging and an extensibility API) are too critical and important to remove.

So, I needed to find a way to decrease memory use some other way, while keeping the same (and potentially even adding more) features.  I looked at options like DPMI and XMS, but they aren't designed for TSR's and just have too many drawbacks to be usable in my particular situation.

Pretty much throughout my history of programming, I had always heard that EMS was "bad" but never really took the time to understand why.  I even remember seeing a program a long time ago called HYPEREAL which could take "regular" TSR's and load them into EMS using some sort of "wrapper" mechanism.  You can still find it at some of the Simtel.Net archives, like https://ftp.sunet.se/mirror/archive/ftp.sunet.se/pub/simtelnet/msdos/tsrutil/ (the file is called hyper_01.zip if you want to download and check it out).  The program is not free and the source isn't available AFAIK, but is still interesting.  I pretty much blew it off back when I came across it because of all the bad stuff I'd heard about EMS (and because of the 64k Page Frame that at the time *thought* was an insurmountable problem but now think differently).

I actually starting investigating what EMS actually was and discovered it certainly doesn't deserve the bad reputation it has garnered.  As I noted above, I think the fact that it needs a 64k Page Frame is just sort of an "automatic" turn-off because you could be using that memory for something else (even though on many machines it isn't actually used for anything at all).  But I also realized that with programs like my USB drivers, I would ultimately need WAY more than 64k of memory (especially as I added more drivers for specific devices), so "giving up" 64k in exchange for the possibility of many MB makes absolute sense.  I also realized that using this methodology, it would be possible to literally add features while affecting conventional memory use minimally, if at all.  Most of the code that actually does important work would be in EMS, and the code in conventional/upper memory is mostly just used to set up and tear down the switch to EMS, along with the code that must be in conventional/upper memory necessarily to be compatible with DOS and other TSR's.

The other useful "mechanism" I discovered in the research is DPMS (DOS Protected Mode Services).  You can think of DPMS sort of like "DPMI for TSR's and Device Drivers".  What DPMS does is to temporarily switch the CPU into protected mode, run your TSR code from Protected Mode (either (16-bit or 32-bit PM) and then switch back to Real Mode again (or V86 mode if you have something like HIMEM/EMM386 loaded).  This essentially lets you put your TSR code in XMS since DPMS handles the switch to PM for you.

The reason I mentioned 16-bit PM above comes into play here.  The way I'm writing the programs, they can run in either Real (or V86) Mode, from EMS, or from DPMS, all using the exact same code.  When I start DPMS, I tell it I want the CPU to be in 16-bit PM when my TSR code runs.  The "stub" of the TSR that remains in conventional or upper memory contains the set up and tear down code to switch to DPMS, EMS, or not switch at all, depending on how things are configured and how the user wants to use their memory.  You can also have both an EMS and a DPMS server installed at the same time.

A little history on DPMS.  I believe DPMS was originally written by Novell when they owned DR-DOS, and the DPMS servers are included with various releases of DR-DOS.  DR-DOS also included a couple of utilities that were compatible with DPMS (IIRC they were a disk caching program and an MSCDEX alternative, and maybe some others too).  Some later versions of IBM's PC-DOS also supposedly included a DPMS server (never tried it myself, and I think it works a little differently than the Novell version).  AFAIK the Novell DPMS source code has never been released, but Novell said you could distribute the DPMS server with any DPMS client.  The DPMS server only uses about 2k of memory, so is much more efficient than EMS (which needs a 64k page frame plus some "overhead" memory somewhere to manage everything).  The DPMS server also works on 286 CPU's, though I know most people won't care about that.

Anyway, that's the story for TSR's, EMS, DPMS, and USB.  Also, I didn't start the EMS/DPMS experiments with the USB drivers, but started with some simpler TSR's as a proof of concept to prove to myself that it was all viable -- and it is.  There are still some issues with slower CPU's (286 and earlier) because of the Protected Mode => Real Mode switching speeds mentioned above (and other items), but I'm working on ways around that.