The FreeDOS Project

Hi Bernd,

> I'm happy with whatever I can get. My real hardware has an Nvidia 
> chipset network driver for which no packet drivers exist, so sticking to 
> virtual machines. I wonder if any PCI (or even PCI-express or onboard) 
> network cards still support packet drivers.

You probably can still find many RTL8139 PCI cards almost for free
in second hand shops. Also, there are NDIS wrappers and ODI drivers
for example for at least older nFORCE MCP chipsets and I think you
could use them with generic NDIS or ODI to packet wrapper drivers.
Search for example for NDIS_452 or for NVODI (ca 2001 to 2005).

Eric

On 7/6/2011 7:10 PM, Bernd Blaauw wrote:
> Op 7-7-2011 1:32, Michael B. Brutman schreef:
>> mTCP FTP compares poorly to the native stack and FireFox there, but FTP
>> is working in a very limited environment:
>>
>>    * The TCP/IP socket receive buffer is tiny compared to the native
>>      network stack
>>    * You are doing filesystem writes 8KB at a time
>>    * You have a small virtualization penalty
>>    * The packet interface wasn't designed for high performance; every
>>      incoming packet requires a hardware interrupt and two software
>>      interrupts
> I'm happy with whatever I can get. My real hardware has an Nvidia
> chipset network driver for which no packet drivers exist, so sticking to
> virtual machines. I wonder if any PCI (or even PCI-express or onboard)
> network cards still support packet drivers.
>
> 8KB filesystem writes? odd. So it's:
> 1) download/transfer 8KB (8KB transfer buffer)
> 2) halt download, dump transfer buffer to disk and clear it
> 3) continue downloading.

Not so odd.  All comm code fills buffers and then processes the 
buffers.  Unless you have a multi-core system you are always halting the 
processing of TCP/IP protocol handling to do your disk writes.  Modern 
OSes with DMA support hide some of that by letting the DMA controller of 
the disk (and possibly the Ethernet controller if so equipped) do the 
byte copying work.  But in the absence of DMA the host CPU does 
everything, and does it in a single threaded manner.

> Easier at least compared to having a 8KB transfer buffer plus a 'huge'
> receive buffer (nearly size of all of machine's conventional memory, a
> multiple of 8KB?) followed by only clearing the buffer if it's full or a
> file has been downloaded completely (whichever comes first). Your single
> buffer might be more efficient compared to transfer buffer plus receive
> buffer.
>
> Or perhaps I should stay silent hehe, failed miserably while learning
> about OSI layers and TCP/IP.

I suspect that the Intel chipsets on PCI-X cards will work; their PCI 
chipset cards had working packet drivers and from a software standpoint 
PCI-X is identical to PCI.  On the wiki at the Google project page I 
have three different PCI cards listed that are known to work.  (And I'd 
like to hear about more.)

In this environment we are entirely single threaded, except for the 
hardware buffering that happens on the Ethernet card.  To receive a 
packet the path looks like this:

- the card receives and buffers the frame from the wire
- the card signals a hardware interrupt
- the packet driver responds and either interrogates the card or copies 
the contents of the frame
- the packet driver makes an upcall to the TCP/IP code
- the TCP/IP code either provides a buffer or says 'no room'
- the packet driver makes a second upcall to let the TCP/IP code know 
the frame is copied
- the interrupt ends and the interrupted code resumes
- the packet must now go through IP and TCP protocol processing

The buffering scheme works at three levels:

- Raw packet buffers (20 at 1514 bytes)
- TCP receive buffering (8KB)
- File read/write chunk size (8KB)

Raw packet buffers are used by the packet driver directly.  They are the 
critical resource; if you run out of those you start dropping frames 
coming in from the wire.  TCP buffering is designed to pull data from 
those packet buffers as quickly as possible so that they may be 
recycled.  (In the case where you have a lot of small incoming packets 
that is really critical because every incoming packet is allocated 1514 
bytes whether it needs it or not.)  The TCP buffer is organized as a 
ring buffer so it is more space efficient.

The application reads from the TCP buffer and writes to the filesystem.  
All of this is still single threaded and for most systems the bottle 
neck is the disk access time, not the copying of data from multiple 
buffers.  At a minimum all reads and writes to the filesystem should be 
done in multiples of 512 bytes; anything less requires DOS to do a 
read/modify/write as it writes data to the blocks of the filesystem.  
1KB reads and writes were very inefficient to do - after some 
experimenting I found that 8KB was good.  16 or 32KB are marginally better.

The buffer sizes generally are not larger because larger does not make 
that much of a difference in the performance of the filesystem writes 
and does have a negative impact on buffering.  Long writes delay TCP 
protocol processing, causing incoming buffers to run low and delaying 
the sending of ACK packets for the received data.  The major opportunity 
for improving performance is to send the ACK for the packet as quickly 
as possible, right after TCP goes through protocol processing but before 
the application tries to read the receive buffer and empty it.  Getting 
that ACK out early keeps the flow of data constant and hides some of the 
latency the 'receive data/write data' cycle that is occurring.

Another optimization I could make to this would be to have the FTP 
application handle the raw packet buffers directly.  TCP would continue 
to do protocol processing, but instead of copying the data to a TCP 
receive buffer it would just give FTP the raw packets and let FTP do the 
copying.  This was the original design and the first netcat code used 
this technique.  The technique removes some memcpy overhead, but the 
largest overhead comes from the filesystem write.  It made the end 
application code (FTP) more complex and error prone, and had a nasty 
habit of starving the packet driver for buffers if the disk write was 
too large.

Most of my testing is on lower-end machines, like a 386-40 and the 
various 8088 machines that I have.  The performance of memcpy is far 
better on the newer processors due to pipeline efficiency and levels of 
caching.  (Even the 386-40 has a 128K L2 cache.)  Disk becomes the major 
bottleneck on the faster machines.

>> But for older machines, it is more than adequate. I'll try a few tests
>> here - I also test under VMware, VirtualBox, and DOSBox but I never
>> thought to compare the performance to the native TCP/IP stack.
> It's more than adequate indeed, I'm not complaining hehe. Only wanted to
> start out by eliminating slow servers as potential bottleneck. The
> listed ftp://ftp.xs4all.nl/pub/test/10mb.bin should be same for everyone
> as it's intended by that ISP for speedtest purposes.
>

Just for comparison, on my local network here the AMD 80386-40 clone 
with an NE2000 adapter on a 10Mb segment is connected to a Pentium 233 
running Fedora 2.  My FTP receive speed on this network is 506KB/sec, 
which is similar to what you are reporting for your virtual machine.  My 
Pentium 133 running with a PCI Ethernet card receives at 1950KB/sec, 
which is almost four times faster than what you are reporting.  Your VM 
might be adding unnecessary overhead.


Mike

On 7/6/2011 7:10 PM, Bernd Blaauw wrote:
> For DHCP I can imagine various errorlevels for different reasons:
> * missing MTCPCFG variable
> * MTCPCFG variable points to non-existing file
> * missing PACKETINT keyword in file
> * unable to write to MTCPCFG file (yay hidden/readonly)
> * packetdriver not loaded yet
> * static config found in MTCPCFG file
> * time out (no DHCP server found)
> * what else? invalid MTU size?
>
> Bernd

I have resisted getting too detailed with the errorlevels because I 
would need to make them consistent across all of the applications. At 
some point I'm not going to control all of the applications, so I won't 
be able to enforce it. The user readable error messages are usually 
descriptive enough.  (I might address this by reserving a block of error 
codes for the library, most of which would be used during startup.)

We might have a small misunderstanding about the relationship between 
DHCP and the configuration file.  All of the mTCP applications think 
that they are using a static configuration.  They read the configuration 
file and use what they find.  DHCP is a special case; it is the only 
code that will alter the configuration file.  This allows me to keep all 
of the other code simple while still providing DHCP support; all of the 
other code thinks it is using static configuration and DHCP plays along 
with that fiction.  Therefore, DHCP will usually find a static config in 
the file because it wrote the static config there the last time it was 
run.  (If you are generating the config file each time before calling 
DHCP then finding a static config would seem like a reason to stop, but 
normally it is not.)



Mike

On 7/6/2011 7:10 PM, Bernd Blaauw wrote:
> That funny URI syntax was grafted on. What exact URI are you using? Are
>> you adding "ftp://" or just "//" ?
> FreeCOM (maybe also MSDOS command.com or 4DOS) seem to split arguments
> at the "/" level. See the "FOR LOOP" in my batchfile, with batchfile
> called as: ftp.exe ftp://ftp.xs4all.nl/pub/test/10mb.bin
>
> resulting in:
> ftp.exe //ftp.xs4all.nl
> (on which it chokes, not knowing how to handle //).
>
> Basicly there are 2 options:
> 1) find a way to strip the prepending "//" before feeding to FTP
> 2) allow FTP to accept them and strip it themselves, requiring code changes.
>

Ah - I forgot to take the command line parsing into account.  What 
happened to the "ftp:" before the "//" in this case?

Let me see how this behaves on my other systems before deciding what to 
do.  I don't want to build in workarounds for strange command line 
parsing into the code.  A more general solution might be to allow the 
parameter to be quoted on the command line so that I can just handle 
standard URI syntax.


Mike

Hi,
    (yes, I tried trimming the quote, but it almost all seems vaguely
relevant, ugh)

At risk of embarrassing myself (again) by showing my ignorance ...

Mike, you said this:  "Disk becomes the major bottleneck on the faster
machines."

Doesn't FTP know the filesize of file-to-be-downloaded ahead of time?
If so, then perhaps you can try Eric's idea:

* create/open, seek to end-1 (in the empty output file), write a
single byte, close, reopen

This apparently avoids having to update the FAT over and over again
redundantly. See the following thread for some examples (esp. nidud's
comment about Doszip:  "This reduced the compression time from 455 to
180 sec.").

http://www.bttr-software.de/forum/board_entry.php?id=8862#p8879

I may be seriously off-base here (no surprise), but I felt I should
mention it "just in case"!


On 7/7/11, Michael B. Brutman <mbbrutman@...> wrote:
> On 7/6/2011 7:10 PM, Bernd Blaauw wrote:
>> Op 7-7-2011 1:32, Michael B. Brutman schreef:
>>> mTCP FTP compares poorly to the native stack and FireFox there, but FTP
>>> is working in a very limited environment:
>>>
>>>    * The TCP/IP socket receive buffer is tiny compared to the native
>>>      network stack
>>>    * You are doing filesystem writes 8KB at a time
>>>    * You have a small virtualization penalty
>>>    * The packet interface wasn't designed for high performance; every
>>>      incoming packet requires a hardware interrupt and two software
>>>      interrupts
>>
>> 8KB filesystem writes? odd. So it's:
>> 1) download/transfer 8KB (8KB transfer buffer)
>> 2) halt download, dump transfer buffer to disk and clear it
>> 3) continue downloading.
>
> Not so odd.  All comm code fills buffers and then processes the
> buffers.  Unless you have a multi-core system you are always halting the
> processing of TCP/IP protocol handling to do your disk writes.  Modern
> OSes with DMA support hide some of that by letting the DMA controller of
> the disk (and possibly the Ethernet controller if so equipped) do the
> byte copying work.  But in the absence of DMA the host CPU does
> everything, and does it in a single threaded manner.
>
>> Easier at least compared to having a 8KB transfer buffer plus a 'huge'
>> receive buffer (nearly size of all of machine's conventional memory, a
>> multiple of 8KB?) followed by only clearing the buffer if it's full or a
>> file has been downloaded completely (whichever comes first). Your single
>> buffer might be more efficient compared to transfer buffer plus receive
>> buffer.
>>
> In this environment we are entirely single threaded, except for the
> hardware buffering that happens on the Ethernet card.  To receive a
> packet the path looks like this:
>
> - the card receives and buffers the frame from the wire
> - the card signals a hardware interrupt
> - the packet driver responds and either interrogates the card or copies
> the contents of the frame
> - the packet driver makes an upcall to the TCP/IP code
> - the TCP/IP code either provides a buffer or says 'no room'
> - the packet driver makes a second upcall to let the TCP/IP code know
> the frame is copied
> - the interrupt ends and the interrupted code resumes
> - the packet must now go through IP and TCP protocol processing
>
> The buffering scheme works at three levels:
>
> - Raw packet buffers (20 at 1514 bytes)
> - TCP receive buffering (8KB)
> - File read/write chunk size (8KB)
>
> Raw packet buffers are used by the packet driver directly.  They are the
> critical resource; if you run out of those you start dropping frames
> coming in from the wire.  TCP buffering is designed to pull data from
> those packet buffers as quickly as possible so that they may be
> recycled.  (In the case where you have a lot of small incoming packets
> that is really critical because every incoming packet is allocated 1514
> bytes whether it needs it or not.)  The TCP buffer is organized as a
> ring buffer so it is more space efficient.
>
> The application reads from the TCP buffer and writes to the filesystem.
> All of this is still single threaded and for most systems the bottle
> neck is the disk access time, not the copying of data from multiple
> buffers.  At a minimum all reads and writes to the filesystem should be
> done in multiples of 512 bytes; anything less requires DOS to do a
> read/modify/write as it writes data to the blocks of the filesystem.
> 1KB reads and writes were very inefficient to do - after some
> experimenting I found that 8KB was good.  16 or 32KB are marginally better.
>
> The buffer sizes generally are not larger because larger does not make
> that much of a difference in the performance of the filesystem writes
> and does have a negative impact on buffering.  Long writes delay TCP
> protocol processing, causing incoming buffers to run low and delaying
> the sending of ACK packets for the received data.  The major opportunity
> for improving performance is to send the ACK for the packet as quickly
> as possible, right after TCP goes through protocol processing but before
> the application tries to read the receive buffer and empty it.  Getting
> that ACK out early keeps the flow of data constant and hides some of the
> latency the 'receive data/write data' cycle that is occurring.
>
> Another optimization I could make to this would be to have the FTP
> application handle the raw packet buffers directly.  TCP would continue
> to do protocol processing, but instead of copying the data to a TCP
> receive buffer it would just give FTP the raw packets and let FTP do the
> copying.  This was the original design and the first netcat code used
> this technique.  The technique removes some memcpy overhead, but the
> largest overhead comes from the filesystem write.  It made the end
> application code (FTP) more complex and error prone, and had a nasty
> habit of starving the packet driver for buffers if the disk write was
> too large.
>
> Most of my testing is on lower-end machines, like a 386-40 and the
> various 8088 machines that I have.  The performance of memcpy is far
> better on the newer processors due to pipeline efficiency and levels of
> caching.  (Even the 386-40 has a 128K L2 cache.)  Disk becomes the major
> bottleneck on the faster machines.

On 7/7/2011 7:56 AM, Rugxulo wrote:
> Hi,
>      (yes, I tried trimming the quote, but it almost all seems vaguely
> relevant, ugh)
>
> At risk of embarrassing myself (again) by showing my ignorance ...
>
> Mike, you said this:  "Disk becomes the major bottleneck on the faster
> machines."
>
> Doesn't FTP know the filesize of file-to-be-downloaded ahead of time?
> If so, then perhaps you can try Eric's idea:
>
> * create/open, seek to end-1 (in the empty output file), write a
> single byte, close, reopen
>
> This apparently avoids having to update the FAT over and over again
> redundantly. See the following thread for some examples (esp. nidud's
> comment about Doszip:  "This reduced the compression time from 455 to
> 180 sec.").
>
> http://www.bttr-software.de/forum/board_entry.php?id=8862#p8879
>
> I may be seriously off-base here (no surprise), but I felt I should
> mention it "just in case"!
>

Interesting, and probably part of the reason why 8KB writes were much 
more efficient than 1KB writes.  Since DOS is adjusting the FAT each 
time the file grows there is more work to do for the smaller block sizes.

The mTCP client does not query the file size before starting a download; 
it requires another command to be inserted before the GET, kind of the 
way that PORT and PASV commands are inserted.  It's an involved change, 
but if it has the potential to make the disk writes faster I'd be happy 
to try it.

(I'll see if it makes a noticeable difference first by receiving to a 
file that is already created and is at the right size.  In theory that 
transfer will happen faster than if the file is not created yet.)

For error recovery you would not want to leave a full sized by half 
written file laying around.  It looks like chsize is the right way to 
truncate the file to the correct size after the TCP socket is closed.


Mike

On Thu, 7 Jul 2011, Bernd Blaauw wrote:

> I'm happy with whatever I can get. My real hardware has an Nvidia
> chipset network driver for which no packet drivers exist, so sticking to
> virtual machines. I wonder if any PCI (or even PCI-express or onboard)
> network cards still support packet drivers.

I think the RTL-8139 chipset does and it's still quite common.

> FreeCOM (maybe also MSDOS command.com or 4DOS) seem to split arguments
> at the "/" level. See the "FOR LOOP" in my batchfile, with batchfile
> called as: ftp.exe ftp://ftp.xs4all.nl/pub/test/10mb.bin

Sounds like a bug: that should be handled at the program level, not the 
shell level.

-uso.

Op 7-7-2011 14:08, Michael B. Brutman schreef:
> I have resisted getting too detailed with the errorlevels because I
> would need to make them consistent across all of the applications. At
> some point I'm not going to control all of the applications, so I won't
> be able to enforce it. The user readable error messages are usually
> descriptive enough.  (I might address this by reserving a block of error
> codes for the library, most of which would be used during startup.)

Ah my interest was purely to DHCP.exe here, once that works all 
applications are fine with their semi-static file (except for a "lease 
expired, can't run, run DHCP.exe first again or stick to static IP)


> with that fiction.  Therefore, DHCP will usually find a static config in
> the file because it wrote the static config there the last time it was
> run.  (If you are generating the config file each time before calling
> DHCP then finding a static config would seem like a reason to stop, but
> normally it is not.)

Yeah I don't know DHCP.exe's behaviour, was just afraid it would be 
spamming the DHCP server everytime I run it, instead of analysing the 
configuration file's timestamp + lease.

VMware's DHCP gives 1800 seconds lease (NAT-mode), while my router's 
DHCP gives 1 day leases indeed (as does yours).

The tricky situation is a "hey lease is still valid, let's not query". 
with meanwhile lease only still being valid like 5 seconds or so.
Trouble ahead :)

Op 7-7-2011 15:37, Steve Nickolas schreef:
> On Thu, 7 Jul 2011, Bernd Blaauw wrote:
>

> I think the RTL-8139 chipset does and it's still quite common.
>

And there I was, wanting high-performant low-CPU network cards (with 
also packet drivers):
* Intel Gigabit PCI-card (10/100Mbit fallback)
* Intel Gigabit PCI-e card (10/100Mbit fallback)
* Intel 10Gigabit PCI-e card (only 1000mbit fallback, possibly only 
jumboframes supported?)

I'm afraid I've got no use for PCI-X cards, my old dual opteron machine 
is in the scrapyard.

>> FreeCOM (maybe also MSDOS command.com or 4DOS) seem to split arguments
>> at the "/" level. See the "FOR LOOP" in my batchfile, with batchfile
>> called as: ftp.exe ftp://ftp.xs4all.nl/pub/test/10mb.bin
>
> Sounds like a bug: that should be handled at the program level, not the
> shell level.

I don't know how other shells behave here.
doesn't matter if I do a:
* FOR %x in ( %path% ) do echo %x
* for %x in ( ftp://ftp.xs4all.nl/pub/test/10mb.bin ) do echo %x

I should probably see how MS Command.com and 4DOS parse this.

> -uso.

Bernd

Op 7-7-2011 13:58, Michael B. Brutman schreef:
> Most of my testing is on lower-end machines, like a 386-40 and the
> various 8088 machines that I have.  The performance of memcpy is far
> better on the newer processors due to pipeline efficiency and levels of
> caching.  (Even the 386-40 has a 128K L2 cache.)  Disk becomes the major
> bottleneck on the faster machines.

As FTP.exe doesn't occupy entire conventional memory anyway (though not 
all machines have 512KB/640KB) maybe a ramdisk in conventional memory is 
possible. I don't know any software besides TDSK being able to do so 
though. My own testing sometimes involves a solid state disk, other 
times a ramdrive. Bye latencies :)

> Just for comparison, on my local network here the AMD 80386-40 clone
> with an NE2000 adapter on a 10Mb segment is connected to a Pentium 233
> running Fedora 2.  My FTP receive speed on this network is 506KB/sec,
> which is similar to what you are reporting for your virtual machine.  My
> Pentium 133 running with a PCI Ethernet card receives at 1950KB/sec,
> which is almost four times faster than what you are reporting.  Your VM
> might be adding unnecessary overhead.

Yeah I'm not sure what's causing that, maybe the default emulated NIC is 
an AMD 10mb/s card. I'm sure there's some also the possibility of an 
emulated intel card, just need to find out how to configure it.
Not tried physical machine (no packet drivers, and officially there's no 
freely distributable NDIS/ODI files required to get a network card's 
NDIS/ODI drivers working followed by a shim/wrapper).
Not tried local network either yet, my router has some internal storage 
and a few USB2 ports for adding drives.

Just for grins I did a little speed testing comparing mTCP running in a 
VM to native FTP under Windows XP.  Here are the results:

File size: 32MB
Source: Linux 2.6.x running on a Pentium 233, local 100Mb/sec connection

Windows XP command line FTP client: ~8950KB/sec
LFTP running under Cygwin: ~8850KB/sec
mTCP, VMWare 3.13, PCNet emulation, standard buffer sizes: ~3950KB/sec
mTCP, VMWare 3.13, PCNet emulation, max buffer sizes: ~6826KB/sec


So the maximum buffer size settings do improve performance quite a bit.  
I need to go back and test my older/slower machines to see if they see 
similar results.  Performance characteristics have changed since I set 
those buffer sizes and some tweaking to the defaults might be 
appropriate.  (All of this can be set from the configuration file today, 
so no new code is needed.)



Mike

Op 7-7-2011 20:07, Michael B. Brutman schreef:
> Windows XP command line FTP client: ~8950KB/sec
> LFTP running under Cygwin: ~8850KB/sec
> mTCP, VMWare 3.13, PCNet emulation, standard buffer sizes: ~3950KB/sec
> mTCP, VMWare 3.13, PCNet emulation, max buffer sizes: ~6826KB/sec

Odd that my download sucks then at 500KB/s. Ah well, must be some local 
issue in Windows, VMware or the machine itself.

> So the maximum buffer size settings do improve performance quite a bit.
> I need to go back and test my older/slower machines to see if they see
> similar results.  Performance characteristics have changed since I set
> those buffer sizes and some tweaking to the defaults might be
> appropriate.  (All of this can be set from the configuration file today,
> so no new code is needed.)

Could you list a short version of your buffer settings in that file 
please? I settled for MTU 1500 and that was it.

Goodluck testing old machine behaviour :)

Hi Bernd,

> My own testing sometimes involves a solid state disk, other 
> times a ramdrive. Bye latencies :)

SSD have a quite noticeable per-write latency. The reason why
you do not notice it with Windows or Linux is that those may
use NCQ (queueing of concurrent disk I/O activities) and will
often use larger blocks... The latency only grows a bit while
blocks grow a lot, so writing e.g. the FAT or directory meta
data in units of 512 bytes without pooling can be still slow
in operating systems like DOS depending on your caching :-)

Eric