[pclasses-devel] Fwd: [S11n-devel] FWD: 10^7

[stephan b. <st...@s1...>]

An interesting note about the s11n layer:

i've never tested s11n with more than about 150k data objects (PODs + 
full objects) at any given time. i didn't think it could hold more than 
a few hundred thousand objects before running out of memory because it 
internally uses at LEAST 1 std::map and lots of std::strings for each 
node.

However...

One user wrote last week to say he's using it for a data set of 10 
million objects(!!!). See the mail below for how he's using it.

He sent me some rough timing info: 3 minutes to load 1GB of s11n'd XML 
data (his 10M nodes), which adds up to 55k nodes per second. That is 
extremely close to my local timings (though on much smaller data sets), 
which range from 20k-50k/second, depending on the parser (expat is 
fastest, IIRC). Loading that same data set using the mysql_serializer 
would take... approximately 16.5 days, theoretically (it's 
implementation is EXTREMELY inefficient).

Anyway... i thought it was good news that the s11n layer which is now in 
P2 can hold up to those types of input amounts, and does so relatively 
quickly.


----------  Forwarded Message  ----------

Subject: [S11n-devel] FWD: 10^7
Date: Tuesday 15 February 2005 14:40
From: stephan beal <st...@s1...>
To: s11...@li...
Cc: James McCann <jm...@es...>

This impressed the hell out of me... i had to forward it to the list:

On Monday 14 February 2005 05:28, Paul Balomiri wrote:
>... By The way, Your Architecture has brought
> my program to a state where I successfully load/save a map of vienna,
> with all streets, and over 10^7 datapoints assigned to them.

WOW!

(@Paul, i'm gonna quote that on the web site if you don't mind.)

We're considering some options for custom NodeTypes to do that more
efficiently. i would never have thought someone could process 10^7
items in the lib, given it's heavy reliance on std::string and lexical
casts. That really says something about the std::string implementation,
actually.

10^7 == 10,000,000

The most i've ever tried is about 150k nodes/properties.

--
----- st...@s1...   http://s11n.net
"...pleasure is a grace and is not obedient to the commands
of the will." -- Alan W. Watts


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real
 users. Discover which products truly live up to the hype. Start
 reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
S11n-devel mailing list
S11...@li...
https://lists.sourceforge.net/lists/listinfo/s11n-devel

-------------------------------------------------------

-- 
----- st...@s1...   http://s11n.net
"...pleasure is a grace and is not obedient to the commands
of the will." -- Alan W. Watts