Re: [Oorexx-devel] System resources exhausted

[Mike C. <MF...@uk...>]

> The problem is caused not so much by running out of memory, but rather
> no longer being able to obtain a single contiguous piece of memory
> from the system that can contain the new string result.  The
> explanation is somewhat long and involved, but the biggest problem is
> the need to continually expand the ooRexx object heap to accommodate
> these progressively larger string objects.  These expansions are done
> by adding additional memory segments to the heap large enough to
> satisfy the request.  Generally, with smaller objects, an expansion is
> done by adding additional large chunks, with an understanding that
> this memory will be used for future requests.  This breaks down once
> you start dealing with larger objects, so the additional seguments
> start getting allocated in sizes closer to the actual request getting
> made. 

Chris Stevenson's algorithm might help with that .. always allocate pieces 
which are size 2^n.  I forget the details, but this made reconstructing 
bigger chunks much more possible/likely.

> Unfortunately, when dealing with a steadily expanding loop like
> that, the new segments are quickly too small for subsequent requests,
> and things need to get expanded again.  Eventually, the system hits a
> point where there's no longer room to add additional large chunks, and
> the out of memory error results.

But at that point the interpreter has a gigabyte or more of garbage ... 
but needs less than 5 MB.  And it has killed (stopped) every other process 
on the PC, too.  Surely fixable ...?

> If you know in advance how large the eventual object will be, a
> mutable buffer would be a better option.  You can then do the single
> allocation and just add additional data to it without hammering the
> garbage collector.  This version of your program zips right to the
> conclusion without running out of memory:

Well, this program predates Object Rexx.  It's basically geturl -- get any 
file from the Internet.  Usually the length would be known from the 
header, but not always.

(For now I've fixed it by writing to a temporary file, which is a bit 
messy, but much faster than doing it in memory(!).) 

Mike





Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU