Re: [q-lang-users] Memory fun

[Tim H. <q...@st...>]

Albert Graef <Dr....@t-...> writes:

> Tim Haynes wrote:
>> Am I right in thinking that the generation of error strings including
>> object-dumps is non-lazy?
>
> Well, if you use a non-lazy data structure (lists rather than streams)
> then that's what you get...

Ah :) Yes, that's something else I've considered, but I'll start with a
smaller script to understand exactly how to convert them all across :)

>> It's a bit of a PITA trying to find the one or two list-elements on
>> which it can't do some function or other in the middle of such a large
>> list.
>
> I guess that you should really take a look at the new bytestring
> operations in clib which allow you to treat a bytestring as a mutable
> integer vector with fast random access. That should probably work much
> better for the application at hand. Or maybe tuples of references would
> be helpful (reftypes.q).

Oh? That's interesting. I discovered uint16_list and relatives the other
day, which has made life an awful lot nicer. Now one of the big slow-downs
is the conversion of the bitmap to triples and back - actually, I could use
a hand improving on this:

  //take flat list, return list of triples
  triplify [] = [];
  triplify Ds = [tuple $ sub Ds 0 2] ++ (triplify (sub Ds 3 ((#Ds)-1)));

  //take list of triples, return flat list
  detriplify = (foldr1 (++)) . (map list) ;

I was wondering if there was something smart I could be doing with zip3,
but memory is a serious concern at this point. 

I don't necessarily need random-access on the thing at the moment - a
simple map across the whole lot will suffice, so this is also a candidate
for streaming to reduce memory footprint later. Any advances welcome :)

> In Pure you'll hopefully be able to do serious number crunching like
> this, although it still won't be quite as fast as Fortran. ;-)

Sounds good to me :)

>> Is there any way to speed the process up a bit? Maybe set a few
>> exceptions or something?
>
> Yup, that's what they're for. Just add a few default rules throwing
> exceptions to the suspicious functions.

Right, I'll see what I can do. It's the occasional simple things like

             ...
             where C=hdict (R ++ [(G,G)])

except I first forgot about the [] there and wound up with a 600M pid for
the interpreter, a list of 64K elements plus a pair on the end in an error
message. Ooops :)

~Tim
-- 
<http://spodzone.org.uk/>