Re: [Freedos-kernel] Any interest in 486, 586, 686 kernels?

[Eric A. <e....@jp...>]

Hi!

Your vision on DOS is somewhat, well, interesting ;-) So
there is a lot to chat about, although I am not sure if
the KERNEL list is the right place for this topic. DPMI:

> I would expect performance gain to be minimal.  Maybe there could be
> Low/HMA/UMB memory savings with a different architecture.  Hard to say.

Well there could, but why would it matter? More heavy DOS
software normally uses DOS extenders / DPMI anyway, which
means it does not care about how much low DOS RAM is free.

The HMA is mostly used for the kernel and buffers, so as
long as the kernel fits in there, no others have a heavy
need for it. Only a few drivers may use it "UMB style".
Also, caches put bigger buffers in XMS, not needing HMA.

Finally UMB is mostly used for drivers: You could write
drivers that use DOS extenders / DPMI if you really have
a lack of space. Actually the simulation of SoundBlaster
16 that comes with "SoundBlaster" PCI works like that.

Probably also some commercial NTFS drivers, because NTFS
is complex and you do not want to spend 100s of kilobytes
of DOS memory only for loading a filesystem driver...

USB, PXE and CD/DVD/BD drivers as drivers / in memdisk:

> I lean this way too w/respect to drivers.  Built-in's biggest advantage is

You still have to configure built-in drivers, you only
avoid the risk that you forget to include the file in
your boot disk ;-)

> simplicity in user configuration. However, networking seems to be lacking
> throughput speed with either mTCP or Watt32 apps.  I'm not sure if it's a
> packet driver, TCP/IP stack, app (doubt), or kernel (doubt) issue.

Networking in DOS means that your app has a compiled-in
network stack which communicates with a packet driver to
get the low-level hardware stuff done. You often have a
small buffer for that and little concurrency. There may
be a bit of IRQ and DMA, but "big" operating systems are
more relaxed about juggling with multiple streams of net
data with support from complex chips on your network card.
Note that this is just an educated guess: Ask our experts!

> guess is that it's a 16-bit MOV/LD* loops, the lack of zero-copy
> networking, and the switch between kernel and app.  I've seen some

No switch: The kernel does not network at all and there is
only one app at a time using the network. Depending on how
your packet driver and network stack library work, you do
not need many steps of copying either and the transfer to
or from a buffer is unlikely to be a big bottleneck with
modern CPU, I think. However, as said, you probably work
with little bits of data and small buffers in DOS, because
you may have less orchestration between stack and driver.

> reference to the same ... USB drivers as well (USB 1.1 speed from USB
> 2.0 devices or 3.0 devices).  And there seems to be 3 ways to get USB

That problem far much more trivial than you might think:
USB 2 and 3 are controlled in ways that differ a lot from
USB 1, so many drivers simply do not talk USB 2 or 3 at
all. This is not like AGP, PCI or PCIe where you flip a
few bits and suddenly I/O to your graphics card is fast.

It is more like the difference between paged graphics at
a000:0 and a linear framebuffer, to stay in the example.
However, there is no linear USB. Talking USB the 2 or 3
way is just a quite different language, but as you know,
at least one shareware driver speaks it and knows the
dialect of at least some relatively widespread chips...

> working on DOS: USBDOS, DOSUSB, Panasonic/ASPI Method. I need to play

Enjoy :-) And maybe do some benchmarks. Even the shareware
driver should work long enough for that - I think it just
blocks after a while after each boot, but is not locked in
terms of how many days or years you have it installed.

>> On the other hand, I think it would be an interesting experiment
>> to have a kernel which can load files from at least the root dir
>> of ISO9660 or UDF disks and similar (ext2? ntfs?) and which can
>> directly interpret GPT partition tables...

> Yes! I agree!  The FD Kernel needs to speak MBR, GPT, ISO9660 & UDF
> (including El Torito), NTFS, Ext2/3/4 to stay relevant. Probably HFS+ as
> well.  Linux and/or FUSE should be helpful here.

Uhm no. You do not agree ;-) Yes the kernel should speak both
MBR and GPT. Something about 4k sector size is also a good
idea. However, El Torito is only so-so as CD/DVD/BD driver
and Jack's drivers are probably better. It would be fun from
an academic point of view to have ElTorito-ISO9660-GPT in a
kernel, but even Linux works great with kernels-without-any-
disk-drivers when you put the drivers in the boot-ramdisk to
load them as separate files from there. DOS with MEMDISK is
basically the same.

Having (separately loaded) drivers for ISO9660 (we do, even
with long file name support) and UDF (only experimental and
abandoned??) is indeed important for DOS. Also, somebody may
want to undust the old EXT2 semi-drivers called LTOOLS, they
probably still have some sort of ancient limitation such as
lacking LBA support or getting confused by non-ancient EXT2
or even EXT3 or EXT4 filesystems. Note that LTOOLS, similar
to SMBCLIENT, works a bit like FTP clients: You type some
command and files get transferred. That is much easier than
making a drive letter, because a Windows or Linux drive is
probably way more complex than can be represented as a good
old DOS drive letter and the drivers would also be way more
complex than the amount of RAM that you want to invest for
a DOS driver (loadable or part of the kernel) would allow.

This also holds for HFS+ and NTFS and ExFAT and so on, of
course. Even if you take the effort to make a super-powered
DOS extended / DPMI driven driver, you would still end up
having your cool Windows or Linux drive with devices and
pipelines and long Unicode names and so on represented as
a boring, maybe even 8.3-named shim for the sake of letting
your 1980 GIF viewer access your Windows 8 clip-arts ;-)

If you access your Windows and Linux drive using some DOS
extended file manager with loadable filesystem modules,
you might get a better user experience with less headache
about DOS RAM than when you try to over-boost DOS itself.

> I'd want FD to be USB-ready (built-in or driver) for Floppy,
> Zip/Jaz/SparQ/LS-120 (similar drives), HDD, CD/DVD/BD, mice,
> gamepads/joysticks, keyboards.

A built-in driver in that sense would be any driver for any
DOS. It does not have to have any relation with the kernel
of FreeDOS in particular... If fancy USB drives such as ZIP
behave well, they should all be covered by the same storage
driver for DOS. But if you read the docs from Bret, you see
that USB storage devices kind of default to misbehave :-p

Similarily, CD/DVD/BD go in one category, as long as your
driver speaks the general protocol (or for non-USB: speaks
your controller such as SATA or IDE/ATA/ATAPI) and as long
as you have a CD-like / network-redirector-like driver for
understanding the ISO9660 or other "optical" filesystem.

I think your contributions would be welcome in writing some
components a la SHSU-UDF (SHSUCDX but then UDF) or modules
for Bret's drivers to understand GPT with all FAT sizes and
maybe maybe the most basic support for the most widespread
"modern" filesystems. Say some NTFS version and EXT2 with a
bit of resistence to EXT3 and EXT4 at the expense of being
only read-only or something... As long as you get a bit in
touch with your Windows and Linux data... I think writing
from DOS to those is a risky idea in the first place. That
would be like using a disk hex editor to fix your DirectX.

Luckily Bret already supports mice, keyboards and game :-)

> I think Jim Hall's suggestion of a virtualized OS is more applicable here.
>  Something like MOL (Mac On Linux), or QEMU User Space Emulation (ala Wine)
> with device redirection to generic hardware might be best.

That would be called dosbox and dosemu then, already exists ;-)
You could contribute by making a TINY Linux distro around them
which mostly serves as workhorse for all sorts of drivers, for
hardware and filesystems, network, showing your CGA game in
suitable form on your holographic tablet touch screen, etc...

>> Some automated or semi-automatic quality checks seem interesting.

> Just need people to start writing tests. :D

Feel invited. Note that with automated code quality checks you
will initially just find out that DOS is convoluted and ugly.
But it has to be like that to be compatible. The hard part is
to find out what is necessarily ugly and what are real bugs :-p

> I've seen cwsdpmi r7 supports upto 64GB address space.  Some new interfaces
> might be nice to program too, but I'd advocate step one to be making use of
> such extensions transparent to old programs.

How are your experiences with CWSDPMI r7? Can you help making
it more stable, at least by giving feedback to the maintainer
who probably does not really have but will enjoy it if people
enjoy the proof of concept? :-)

You are right about the transparency in some aspects: If your
interface can only allocate 100 MB for some reasons, you can
still offer ten programs to allocate 100 MB each as long as
you know that you actually have 1 GB and as long as you can
let the driver keep an overview which 100 MB is of which app.

> I think memcopy becomes more important the more we look to implement
> advanced hardware like USB1.1/2.0/3.0, 10/100/1000 ethernet.  They're
> all based on memory mapped IO.

How do you know? Maybe they can also just DMA to where you
need the data to be in the end? In particular with USB, the
problem seems to be more that USB 1 is not for big chunks
of data and requesting many small chunks has big overhead,
no matter how fast you memcopy them. You get similar issues
with disk I/O, DOS caches rarely pool your I/O to read-ahead
megabytes of your MPEG or combine several FAT changes before
writing them in some smart and safe way. Instead, expect DOS
apps to read your MPEG 32 kB at a time and update the FAT at
a ratio of a few bytes at a time. Combine that with the IOPS
rating of a typical USB2 stick and the overhead of only using
USB1 to talk to it and you see why sorting your family pics
on that stick or some SD card is not very smooth inside DOS.

You could try to write a layer of cleverness and pooling for
some cache, or a complete cache, but note that people might
actually prefer basic: You write it, it is slow, but you can
unplug that USB stick or even your whole PC after a fraction
of a second and the data already is there and consistent...

>> Maybe we should discuss the roadmap again in the first place?

> I'd like to see more discussion.  In the mean time, I'll keep
> experimenting.

Enjoy :-) As Tom said between the lines, the (more long term)
roadmap is more visionary and less based on consensus or some
practical "what do we NEED to add NEXT" considerations. The
roadmap for 1.0 and 1.1 was more like that. Today I would say
the question is more which bits you want to ADD and less what
you want to CHANGE DOS as a whole into. Because after all, it
is quite good that DOS is what it is. It does not need to be
a weak imitation of something else which already is good, too.

Regards, Eric :-)