Re: XML was RE: [GD-General] RE: A portable preferences library but has nothing to do with them anymore

[J C L. <cl...@ka...>]

On Tue, 16 Dec 2003 22:17:07 +0100 
Alen Ladavac <ale...@cr...> wrote:

> Can you please explain this calculation in more detail? You get 10MB
> /1.5s = 6.7M/s ? But you said you load 40MB in 11s, what I would say
> is 3.6 MB/s. Please note that you should not substract IO time from
> the total loading time, as CPU should be working in parallel with the
> DMA, otherwise you are doing something _very_ wrong.

The code in question does:

  sz = get_size_of_file ()
  read (h, buf, sz)
  parse (buf)

As such load and parse are entirely separate.  And yeah, by arithmetic
was off.  I'm parsing and cross linking (there's a fair few URI links in
the data) in roughly 6 seconds.

> Anyway, either 7MB/s, or 3.6 MB/s, this is just awfully slow. 

This is an old slow machine, a PII-333 -- I said it was far from
cutting-edge.  As it is a multi-user host there's also a fair bit going
on otherwise (mostly network IO rather than CPU or disk), but I didn't
check for to see what extent the other loads may have had.  I wasn't
trying for anything rigorous, merely a key to incite people to test with
their actual loadings, not a quick hack with a set of gene AFFY data.

> A 40xCDROM...

I was reading from disk possibly NFS over 100bT (can't check right now,
don't have access to that machine from where I am now).  That machine
doesn't have a CD.

> ...  has transfer rate of 6MB/s. You say you are parsing data at that
> rate? Those are certainly not some impressive numbers. :(

Certainly they're not impressive.  They weren't intended to be.  I
suggest you check them on a more representative host with code that
wasn't written as quickly as possible and for data that more closely
matches your loads.

-- 
J C Lawrence
---------(*)                Satan, oscillate my metallic sonatas.
cl...@ka...               He lived as a devil, eh?
http://www.kanga.nu/~claw/  Evil is a name of a foeman, as I live.