#383 Regina 3.7RC1 uses much more memory

open
5
2012-09-07
2012-09-07
No

Using Regina 3.7RC1, a REXX program exhausted virtual memory, causing Regina to stop executing. Regina 3.3 and 3.6 both run OK.
The REXX program is reading a huge amount of records and storing them in stemmed variables.
The program is too large to post here, and the data is also too large (14.5 gigabytes, but not all of the files are read).
I realize that this problem may not be reproducible elsewhere due to the size of the input files, but I don't see that the program should
take that much more memory using a newer Regina as to cause it to run out of virtual memory.

This program failed on two different machines, both Windows 7.
I realize the information presented here is very lean, but I'll monitor this for any queuries and try to answer questions as best as I
can. ______________________________________________________________ Gerard Schildberger

Discussion

  • Mark Hessling

    Mark Hessling - 2012-09-09

    Yes this one will be difficult to replicate. I've compared a simple test script run under Windows XP using 3.3, 3.6 and 3.7RC1 reading a 12,488kb file:
    /**/
    fn = 'd:\dropbox\public\img_0252.mov'
    Do i = 1 While Chars( fn ) > 0
    block.i = Charin( fn, 1+((i-1)*1000), 1000 )
    End
    Say 'read' i-1 'blocks' 'Pausing'
    parse pull .

    The memory usage measured by Task Manager:
    3.3: 17272k
    3.6: 17,208k
    3.7RC1: 16784

    So with this test, 3.7RC1 is using LESS memory than previous versions, so a small test case that exhibits a significant memory increase over previous versions will be needed to investigate further.

     
  • Gerard Schildberger

    A couple of things on the overall execution of the "large" REXX program:

    It reads about 1,200 files which contain (for this particular failure) about
    14,500.000,000 bytes, not
    14,500,000 bytes.

    The format of the files is:

    nnnnnnnnnnnnnnnnnnnnnnn=yyy

    Where nnnnnnnnnnnnnnnnnn is a number, and it may have imbedded commas,
    but no leading zeroes. They are all positive well-formed integers (except for
    some, which have commas). Many of the numbers have blanks after the equal
    sign. The YYY string is viewable plain text.

    yyy is a string about 25 bytes on the average, ranging from 1 --> 31 bytes.

    Each record gets read via the LINEIN bif, with no 2nd argument. The records
    are spread (in numerical order) in about 1,200 files.

    Most of the first 68,000,000 records get stored (stuffed) into a series of
    100,000 byte records, each byte contains the length of yyy, stored as a
    binary value (base 256, if you will) if the number exists, 0 otherwise.
    The index into the string represents the number nnnnnn. The sum of all of
    these 100k records is exactly one billion bytes.

    "roster" format:
    About a half million of the 68m records (numbers) get stored as:
    @.xxxxx=yyy where xxxxx is the base 256 of the nnnnnnnnnn number.

    Numbers over 1 billion are all stored in "roster" format. The numbers
    go up to around 4,400 digits. Only a subset of these numbers are
    read because there isn't enough virtual memory to contain them all,
    and the "large" program is restricted to a range of numbers that are
    read (and stored).

    Older releases of Regina (the best is 3.3 for minimizing virtual storage),
    have no trouble reading the files, but Reginas 3.4 --> 3.6 take
    quite a bit more than Regina 3.3, and because of that, I'm restricted to
    using Regina 3.3 (for virtual storage concerns).

    Regina 3.7 exhausts virtual memory, however. Regina 3.7 is, on the
    other hand, twice as fast in the reading/processing of the 1,200 files
    than previous releases. I haven't yet got to the point in comparing
    the "computation" times of Regina 3.7RC1 with the older releases,
    the computations happen after all the numbers get read. It is this
    part that takes (for some cases), months.

    Each record is read by a subroutine which invoke many BIFs.
    I believe the problem is with Regina invoking these subroutines (tens
    of) millions of times, a belief yet unproven.

    I will endeavor to mimic the behavior of the large REXX program with
    a much smaller REXX program that can be ported to other machines for
    testing without the need of the large input files. I hope this will be
    successful. The other alternative would be to copy the REXX programs
    and it's data to a "flash" drive and mail you the device so you can test
    it on your machine. This is the method that I used to have a friend of
    mine to execute the same program(s).

    If you'd like to know how the numbers are spread across the 1,200
    files, I have histograms that show those things, as well as showing
    how many numbers are stored in the "roster" format.

    I believe this Regina bug is worth pursuing because this "memory
    leakage" has been around (at least) since Regina 3.4, and it may
    have been around earlier, but not to that large of an extent.

    It doesn't appear that the version of Windows matters. I've noticed
    this memory leakage (bug) in Windows/95/98/XP/7. All my Regina
    executions are done in a "DOS" window and use either CMD.EXE
    or COMMAND.EXE. I prefer to use CMD.EXE for almost all executions
    of Regina. ______________________________ Gerard Schildberger

     
  • Gerard Schildberger

    Two more notes:

    When I referred to Regina 3.7, I meant, of course, Regina 3.7RC1.

    Also, the nnnnnnnn number may have leadind and/or trailing blanks (in the file), and the
    equal sign (=) may be, in actuality, some other character, but it almost all cases, it's an
    equal sign. Other characters include: < > : ► ◄ ≡ ~ ¬ ≈

    ______________________________________________ Gerard Schildberger

     
  • Gerard Schildberger

    used to display/write tokens to screen/disk.

     
  • Gerard Schildberger

    used to issued error messages.

     
  • Gerard Schildberger

    used to express (and add/sub, convert) time units.

     
  • Gerard Schildberger

    determines the LINESIZE of the terminal.

     
  • Gerard Schildberger

    determines the screensize of the terminal.

     
  • Gerard Schildberger

    To recreate this problem, download all of the REXX (REX) files, then, using any
    version of REGINA, run the GENXXXXX.REX program, which will create the
    1,100+ files which mimics the real data.

    The 1,100+ files are written to the homedrive (most people use C:), and
    are placed in the folder:

    c:\#\XXXXX\

    and the files are named something like:

    c:\#\XXXXX\$XXXXX.nnn

    where nnn are for the most part, three digits, and some are a mixture of digits
    and letters, indicating (essentially) the size of the numbers (values) in the files.

    I hope this won't be much of an inconvenience to the test, but I wanted to re-create
    the real environment as much as possible. It won't be possible to re-create the
    actual data (files) due to their size and complexity. Only the files being read are
    actually created.

    The GENXXXXX.REX program will run for more than a few hours, but not more
    than 1/2 day.

    Then, using Regina 3.7RC1, execute the RUNXXXXX.REX program.
    It, among other things, invokes $XXXXX.REX program (the monster program),
    which, among other things, invokes the $XXXXXIM.REX program which builds
    a list of files to read (but it's main purpose is to issue informational messages).
    $XXXXX.REX then reads over 1,100+ files, parsing each record and storing
    them into a roster or a series of strings, the total size of those strings is
    exactly 1 billion bytes (in 10k chunks). Most of the data stored is in "two"
    dimensional sparse stemmed arrays. The rosters stemmed arrays are
    sequential arrays. To save real memory (virtual memory), the values are
    stored in base 256 instead of base 10.

    The $T.REX program is used to display all messages and also write
    messages to a INFormational or LOG file, among others. These files are
    all written to the C: (or homedrive) root directory, and will have a filename
    of $XXXXX and some 3-character filetype (file extension).

    It will run for 4 to 6 hours using a modern computer. The only place I've tested
    this program is on my Windows 7 PC, and my friend's computer, also using
    Windows 7, also roughly the same speed PC. If it runs like it runs on my PC,
    i(and/or my friend's PC), it will exhaust virtual memory [see the (3) messages
    below]. I also have a Frankenbox, but it would take another 3 or 4 days to
    run the test on that (slow) box (as it was built in the previous century).

    The last messages issued by Regina 3.7RC1 (or someONE/someTHING) are:

    System resources exhausted

    -------- there are 3 of these messages, followed by the issuance of a new
    Windows dialog window indicating that Regina has stopped responding or somesuch.

    It should be noted again that previous versions of Regina (notably 3.3) read all the
    1,100+ files and process the values without exhausting virtual memory. Regina 3.3
    was always used because successive releases of Regina read the data files
    slower [except for Regina 3.7RC1].

    If anybody wants to see the documentation for any of those REXX programs, I'd
    be happy to upload them (they are all flat text files).
    __________________________________________________ Gerard Schildberger

     
  • Gerard Schildberger

    I'm having problems in trying to accurately duplicate the data using a generator that mimics the
    real data. As soon as I get a good program for generating the data, I'll upload the GENXXXXX.REX
    program. The genXXXXX program takes about ten hours to run, plus another six hours to see if the
    RUNXXXXX.REX program fails. _____________________________________ Gerard Schildberger

     
  • Gerard Schildberger

    one of the real database files (too hard to mimic).

     
  • Gerard Schildberger

    the REXX program that does the bulk of the work.

     
  • Gerard Schildberger

    REXX program that issues informational message and defines the fileIDs to read.

     
  • Gerard Schildberger

    the REXX program used to drive the main REXX program.

     
  • Gerard Schildberger

    I tried to recreate the problem with a mimic version of the real data but essentially failed to make it good enough to fail in the same way (to exhaust virtual memory).

    So, I re-did the $XXXXX.REX program (the main program, it is invoked from RUNXXXXX.REX) and it has built-in logginf of memory usage using the Microsoft's MEMSNAP.EXE program (it is copyrighted, so I didn't upload here). I downloaded it from

    http://www.microsoft.com/en-us/download/details.aspx?id=18546

    It appears that its only for Windows/XP, but it also works under Windows 7 Pro for me.
    I ran it under both Windows, but since it was designed for Windowx/XP, I used that
    as the base for the test.

    The files:

    08/17/2001 14:59 6,144 memsnap.exe
    07/25/2004 19:15 264 MEMSNAP.HEL

    are what's needed (there are a host of other real good support tools, by the way).

    The suggested debugging is:

    Download (from Microsoft) and put MEMSNAP.EXE in the PATH.

    The C: drive is assumed to be the homedrive, if not, change the appropriate REXX programs.

    Run (using any REGINA): GENXXXXX

    This creates the folder: c:\#\XXXXX\

    and creates (copies) two files from the C:\ (root directory).

    Then:
    run (say, using Regina 3.3): RUNXXXXX OUT33

    and
    run ( using Regina 3.7): RUNXXXXX OUT37

    Each of the runs have unique output and the MEMSNAP output is qualified with
    the date stamp of the Regina (using the PARSE VERSION).

    It helps if you have a minimum of other applications running, and make sure
    there is only ONE REGINA running as MEMSNAP just shows "REGINA.EXE"
    for the Proc.Name in the output.

    A series of files are created and you can look at the two versions of the
    Regina invocations, and each of those has a "before and after" file, so you
    can compare what the increases are for the two versions of Regina.

    The two runs take less than a minute (each) to execute. The GENXXXXX
    takes a second or so.

    [Unfortunately, the MEMSNAP only writes to a file and I didn't want to
    write a filter which would just clutter up the debugging..
    ___________________________________ Gerard Schildberger

     
  • Nobody/Anonymous

    I've added a Regina bug report, 3603643 which may hopefully shed some
    light on this bug.

    I've written a much smaller REXX program that emulates the original (large)
    REXX program that fails and included/uploaded it in the 3603643 bug report.

    This bug report is getting too cluttered with my ineffective attempts at
    diagnosing the problem, so I thought a clean bug report would help the
    process of getting this bug addressed and/or fixed.
    _________________________________________________ Gerard Schildberger

     

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks