Menu

rawfile & malloc:Internal Error:cant...

sdaau
2011-11-28
2013-06-12
  • sdaau

    sdaau - 2011-11-28

    Hello again,

    Im trying to simulate a rather long simulation (600 ms - though not necesarilly a bit circuit); I run ngspice (latest from cvs) in batch mode:

    ngspice -b mycircuit.net
    

    Ngspice starts running, at around 3.8e-3 time it starts slowing down, Ubuntu Linux 11.04 starts swapping to hard disk very obviously, and in the end ngspice crashes with:

    malloc: Internal Error: can't allocate 1761942 bytes.

    Now, I try to take a look at the latest manual, and I see:

    16.8 Memory usage
    Ngspice started with batch option (-b) and rawfile output (-r rawfile) will store all simulation
    data immediately into the rawfile without keeping them in memory. Thus very large circuits
    may be simulated, the memory requested upon ngspice start up will depend on the circuit size,
    but will not increase during simulation.

    Then, I try both:

    ngspice --rawfile="ngs.raw" -b mycircuit.net
    

    and

    ngspice -r ngs.raw -b mycircuit.net
    

    and

    ngspice -b mycircuit.net -r ngs.raw
    

    … and the exact same thing happens - the OS starts swapping at around 3.8e-3; and ngspice eventually crashes with:

    malloc: Internal Error: can't allocate 1761942 bytes.

    .
    And - there is no 'ngs.raw' file generated at all; which along with the above message, tells me that `ngspice` ignored the raw file option completely, and went 'rendering' to RAM anyway, and crashed when memory ran out.

    My question is - what I am doing wrong? And could I somehow persuade `ngspice` to simulate to disk, instead of to RAM (as the text in the manual seems to imply)?

    Many thanks in advance for any answers, 
    Cheers!

     
  • sdaau

    sdaau - 2011-11-28

    Damn sourceforge forums without edits :)

    "though not necesarilly a bit circuit" -> "though not necessarily a biG circuit"

    … and post title is: "rawfile & malloc: Internal Error: can't allocate" :)

     
  • Holger Vogt

    Holger Vogt - 2011-11-28

    Sdaau,

    maybe you still have a .control … .ends section in your input file? If yes, this will probably lead to "interactive" simulation within a batch mode simulation.

    Delete that section and use .tran etc. instead.

    To avoid memory limits, compile ngspice with a 64Bit compiler on a 64 Bit OS.

    Regards

    Holger

     
  • Holger Vogt

    Holger Vogt - 2011-11-28

    Sdaau,

    the 'save' command followed by only the relevant nodes to be stored at the beginning of your .control section may also help to reduce memory usage, because only the mentioned vectors will be saved to memory.

    Holger

     
  • sdaau

    sdaau - 2011-11-28

    Hi @h_vogt,

    Many thanks for your prompt reply!

    maybe you still have a .control … .ends section in your input file?

    Yes, I did happen to have one..

    If yes, this will probably lead to "interactive" simulation within a batch mode simulation.

    Thanks for noting this - I was otherwise wandering why there exist dot commands, and "normal" commands; for reference, here is how the manual refers to `.tran` and `tran`:

    CHAPTER 15. ANALYSES AND OUTPUT CONTROL
    15.3 Analyses
    15.3.9 .TRAN: Transient Analysis
    CHAPTER 17. INTERACTIVE INTERPRETER
    17.4 Commands
    17.4.70 Tran*: Perform a transient analysis
    

    Delete that section and use .tran etc. instead.

    Thanks for that - I'm trying it, but I don't seem to have much success.

    This is how my control section looked like (the circuit is essentially the same as in VHDL sim'd .vcd data - as analog sim source; except with two PWM sources and opamps):

    .control
    ** without .control:
    ** Warning: Further gmin increment
    ** Trying gmin =   9.9996E-03 Warning: singular matrix:  check nodes  and
    ** Warning: Last gmin step failed
    ** Warning: singular matrix:  check nodes  and
    **
    ** Warning: Dynamic gmin stepping failed
    ** Note: Starting source stepping
    ** Supplies reduced to   0.0000% Warning: singular matrix:  check nodes  and
    **
    ** Trying gmin =   1.0000E-02 Warning: singular matrix:  check nodes  and
    ** Warning: gmin step failed
    ** Warning: source stepping failed
    ** Transient solution failed -
    echo ...........
    set hcopypscolor=1
    tran 1us 670730us
    ** without .control:
    ** Error on line 66 : display ; Unable to find definition of model  - default assumed
    display
    ** without .control:
    ** Error on line 77 ; Unable to find definition of model 3 - default assumed
    wrdata ghdl2ngspice_sim16b_pwm_d_source_rc V(5) V(3) V(6) V(4)
    .endc
    .end
    

    The error messages above show what would happen to those commands, ran outside a .control/.endc section.

    Replaced all with only:

    .tran 1us 670730us
    .end
    

    Ran again:

    ngspice --rawfile="ngs.raw" -b mycircuit.net
    

    Then again - runs, there is hard disk activity, and terminates with:

    malloc: Internal Error: can't allocate 1761942 bytes.
    

    Now, however, a `ngs.raw` file gets generated:

    $ ls -s --block-size=1 ngs.raw
    3829760 ngs.raw
    

    Then changed the part to:

    .save V(5) V(3) V(6) V(4)
    .tran 1us 670730us
    .end
    

    Again - it starts running, around 3.8e-3 swapping starts - finally crashes with malloc: Internal Error: can't allocate 1761942 bytes., .. even if the ngs.file has only the requested signals:

    $ grep -a 'No\.' ngs.raw
    No. Variables: 5
    No. Points: 0
    

    Ok, then changed again to:

    .control
    echo ...........
    *save V(5) V(3) V(6) V(4) * Error: no data saved for Transient analysis; analysis not run
    *write ngs.raw V(5) V(3) V(6) V(4) * Error: no such vector v(5)
    .save V(5) V(3) V(6) V(4)
    tran 1us 670730us
    display
    .endc
    .end
    

    And again - swapping starts, and malloc: Internal Error: can't allocate 1761942 bytes..

    To avoid memory limits, compile ngspice with a 64Bit compiler on a 64 Bit OS.

    Thanks for this as well - I always thought I'm running 32-bit, and so I started wandering where to find a 64-bit machine. Then, I followed this page: http://www.cyberciti.biz/faq/linux-how-to-find-if-processor-is-64-bit-or-not/ ; and ran these commands:

    $ uname -a
    Linux ljutntcol 2.6.38-12-generic #51-Ubuntu SMP Wed Sep 28 14:25:20 UTC 2011 i686 i686 i386 GNU/Linux
    $ grep flags /proc/cpuinfo
    flags       : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts aperfmperf pni dtes64 monitor ds_cpl est tm2 ssse3 cx16 xtpr pdcm movbe lahf_lm dts
    

    and it seems that I have a 64-bit processor (there is a `lm` flag in `cpuinfo`), yet I'm using 32-bit Linux OS on it (no `x86_64` in `uname`)… Could this be a part of the problem?

    Still, what puzzles me most is that - even when ngspice starts generating a raw file, it seems to start simulating/"rendering" to RAM first anyway - and at that, without any limits to RAM allocation, before flushing to disk starts? I would have thought, if `ngspice` knows it has to grab to disk, it would have "flushed" the data after each timestep (which in effect would be like a buffer limitation) - but it doesn't look like it behaves like that…

    Well, hope there is a way to get this simulation to complete :)

    Thanks again,
    Cheers!

     
  • sdaau

    sdaau - 2011-11-28

    Just experimenting some more, here are some more results - I tried to extract data (I need the ASCII in the end anyway) in a "piecewise" fashion, but still no dice:

    * segfault:
    .control
    echo ...........
    tran 1us 3ms
    wrdata fil1 V(5) V(3) V(6) V(4)
    tran 1us 6ms 3ms
    wrdata fil2 V(5) V(3) V(6) V(4)
    display
    .endc
    
    * segfault too:
    .save V(5) V(3) V(6) V(4)
    .tran 1us 3ms
    .control
    wrdata fil1 V(5) V(3) V(6) V(4)
    .endc
    .tran 1us 6ms 3ms
    .control
    wrdata fil2 V(5) V(3) V(6) V(4)
    .endc
    
    * malloc: Internal Error: can't allocate 1761942 bytes.
    .tran 1us 36ms 33ms
    .save V(5) V(3) V(6) V(4)
    .control
    wrdata fil2 V(5) V(3) V(6) V(4)
    .endc
    

    I also tried with `tran 1s 3ms` into `-rawfile="ngs1.raw"` (which completes); then `load ngs1.raw` with `tran 1s 6ms 3ms` into `-rawfile="ngs2.raw"` - this starts swapping, does not crash, but then there is only preamble in ngs2.raw (no data). It seems `tran 1s 6ms 3ms` runs the simulation from 0 to 3 ms in memory anyways, allocating as it goes - I hoped it would have "taken the last results" from the 0-3ms run in `ngs1.raw`, but it doesn't.

     
  • Holger Vogt

    Holger Vogt - 2011-11-28

    Sdaau,

    I am just running the venerable 51 stage ring oscillator stemming from the Berkeley BSIM3 web site. No .control section included, just using the .tran command. I am using MS Windows XP, 32 Bit, ngspice conmpiled with TDM mingw 32 Bit. 
    The command to invoke ngspice is:
    ngspice -b -r ro.raw -o ro.log ring51.cir
    The Windows task manager tells me that I am using less than 3 MBytes of main memory. Meanwhile the batch output file ro.raw is about 520 MBytes in size.

    Hmm, is there a difference between MS Windows and LINUX?

    I have to check that on another machine, maybe later today.

    Regards

    Holger

     
  • Holger Vogt

    Holger Vogt - 2011-11-28

    Sdaau,

    as soon as you have a .control section, like in
    .tran 1us 36ms 33ms
    .save V(5) V(3) V(6) V(4)
    .control
    wrdata fil2 V(5) V(3) V(6) V(4)
    .endc
    ngspice will store all output data in main memory and dumpp out the vectors only after the simulation.

    You may try:

    .control
    save 5 3 6 4
    tran 1us 36ms 33ms
    wrdata fil2 V(5) V(3) V(6) V(4)
    .endc

    Holger

     
  • Holger Vogt

    Holger Vogt - 2011-11-28

    Sdaau,

    same with LINUX 32 Bit, SUSE 12.4, memory usage < 3 MBytes, as told by the system monitor.

    Holger

     
  • sdaau

    sdaau - 2011-11-28

    Hi @h_vogt,

    Many, many thanks for your help and answers today!

    In brief - I'm starting to suspect the `d_source` in particular; but here are some more details:

    > I am just running the venerable 51 stage ring oscillator stemming
    > from the Berkeley BSIM3 web site. No .control section included,
    > just using the .tran command.

    Thanks for mentioning this; I just realized I need to deliver a proper test case (and sorry for not doing so earlier). So, here is a minimal version of the problematic circuit, along with data for it:

    http://sdaaubckp.sf.net/post/ngspice/ngspice_PWMexample.zip

    The zip contains `PWMexample.net` and the `d_source` data file `pwmexample_data.text`. The entire netlist (without .model, etc) is simply:

    A1 [3 4] input_vector
    abridge1 [3] [2] dac1
    R1 2 1 1K
    C1 0 1 10n
    

    … and always tested with

    ngspice -r ngs.raw -b PWMexample.net
    

    .

    Basically, as long as I do short transient runs (up to 3 ms or so), things go well - after that swapping starts (so for 6 ms it barely completes); 33 to 36 ms will fail with "malloc: Internal Error: can't allocate 1761942 bytes" (see comments in the .net file).

    > I am using MS Windows XP, 32 Bit, ngspice conmpiled with TDM mingw
    > 32 Bit. The command to invoke ngspice is: ngspice -b -r ro.raw -o
    > ro.log ring51.cir The Windows task manager tells me that I am
    > using less than 3 MBytes of main memory. Meanwhile the batch
    > output file ro.raw is about 520 MBytes in size.
    >
    > Hmm, is there a
    > difference between MS Windows and LINUX?

    Thanks for raising the issue - I fired up the old XP I have lying around, and I tried running `ngspice23_110605-2.zip` on the `PWMexample.net` file; that version says: "Unknown model type d_source - ignored" (and for the time being, I wanted to avoid rebuilding `ngspice` on Windows)

    as soon as you have a .control section, like in
    .tran 1us 36ms 33ms
    .save V(5) V(3) V(6) V(4)
    .control
    wrdata fil2 V(5) V(3) V(6) V(4)
    .endc
    ngspice will store all output data in main memory and dumpp out the vectors only after the simulation.

    You may try:

    .control
    save 5 3 6 4
    tran 1us 36ms 33ms
    wrdata fil2 V(5) V(3) V(6) V(4)
    .endc

    Well, all the way up to now, I was thinking `.save` and `save` are the same, and I kept feeding both of them V(1) etc as argument - and I was correspondingly wandering, why does `ngspice` tell me "Error: no such vector v(2)" when I try `save V(2)` - that's because it accepts a net name (not voltage of net) :) So thanks for that correction too!

    Unfortunately, that syntax too fails for me with "malloc: Internal Error: can't allocate 1761942 bytes"

    > I have to check that on another machine, maybe later today.
    > …
    > same with LINUX 32 Bit, SUSE 12.4, memory usage < 3 MBytes, as
    > told by the system monitor.

    Many thanks - that is in reference to the "51 stage ring oscillator" example, right? Would you mind giving the http://sdaaubckp.sf.net/post/ngspice/ngspice_PWMexample.zip a spin, and see how it performs on your machine?

    Many thanks again,
    Cheers!

     
  • Holger Vogt

    Holger Vogt - 2011-11-28

    Sdaau,

    in fact it is the XSPICE code model (probably the D_SOURCE). It eats up all your memory, than it fail by trying to allocate even more memory.

    This is difficult to track down, because I have no experience with this code.

    Regards

    Holger

     
  • sdaau

    sdaau - 2011-11-29

    Hi @h_vogt,

    > in fact it is the XSPICE code model (probably the D_SOURCE). It eats
    > up all your memory, than it fail by trying to allocate even more
    > memory. This is difficult to track down, because I have no experience
    > with this code.

    Many thanks for confirming the culprit!

    Btw, I also built CVS ngspice on Cygwin - and it too failed with "Internal Error: can't allocate 1761942 bytes."; memory was under control (15MB), but page file increased almost linearly until crash.

    Since we narrowed down the choices, it was relatively easy to find:

    http://www.gnu-darwin.org/www001/src/ports/cad/tclspice/work/tclspice/src/xspice/icm/digital/d_source/cfunc.mod

    … and see that `token = CNVget_token(&s, &type);` gets called a lot, where CNVget_token malloc's - yet there are no matching free's. By inserting some free's there I couldn't solve the "can't allocate" crash - however, the difference between alloc and free calls decreased by like a million, and it gave `valgrind` about enough headroom to be able to spit out a relatively decent log. To me it looks like XSPICE keeps event in a 'linked list', and it simply marks them 'removed' (apparently to allow the simulator to rewind if a step fails) - without freeing them; and thus "the bucket fills". However, it's a bit too difficult for me to solve this whole thing - if others wanna try (note, this is code from the 90's, apparently), I've uploaded my log and the valgrind log here:

    http://sdaaubckp.sf.net/post/ngspice/xspice_dbg_nov2011.zip

    Anyways, turns out - no biggie, since there's `filesource` :)

    The good thing is that the ASCII format (for data textfile) for `d_source` and `filesource` is nearly the same! Except I have '0s'/'1s' values in the d_source file - and `filesource` takes real numbers. However, it's relatively easy to change using `sed`:

    cp pwmexample_data.text pwmexample_data2.text
    sed -i 's/\*/#/' pwmexample_data2.text
    sed -i 's/0s/0.1/g' pwmexample_data2.text
    sed -i 's/1s/5.0/g' pwmexample_data2.text
    

    … and I just use this as a .net file now:

    * PWMexample2.net
    .model filesrc filesource (file="pwmexample_data2.text"
    + amploffset=[0 0] amplscale=[1 1]
    + timeoffset=0 timescale=1
    + timerelative=false amplstep=false)
    * note - [3] floats, that's intended
    A1 %vd([2 0 3 0]) filesrc
    R1 2 1 1K
    C1 0 1 10n
    .control
    save 2 1
    tran 1us 36ms
    wrdata fil2 V(2) V(1)
    .endc
    

    … and calling `-r ngs.raw -b PWMexample2.net` makes the entire 36ms transient pass in like 4.304 seconds (which is amazing for me, given how up to now d_source took tens of minutes, and crashes) :)

    Note - with the above `sed` data, if you let `filesource` interpolate, then one should get a triangular waveform; however, it seems there's a `amplstep=TRUE` option, which should preserve the square character of the PWM waveform (I guess, haven't tried it yet - but if worse come to worse, I guess I could insert samples with a script).

    Well, I'm glad this came to a conclusion - many thanks again @h_vogt for your kind help!

    Cheers!

     
  • sdaau

    sdaau - 2011-11-30

    PS: Be VERY careful when considering the above `filesource` example: namely, the PWM signal is derived from an engine with 20 ns clock. If (as in above post), the transient step is specified at 1 us - then the `ngspice` engine will skip over/"eat" some data point values; resulting with a completely unrecognizeable signal! Therefore, for accuracy, the transient sim line should be:

    tran 20ns ${SIMTIME}
    

    Of course, with a smaller timestep, the `ngspice` engine needs to do more steps - and so the simulation will take A LOT longer…

    (Note here, I didn't experience this with the `d_source` (during short runs, when it works) - because it marks its items as removed, and doesn't really remove them, that can serve as a signal to the `ngspice` engine to handle unhandled edges, it seems… However, that also seems to be the cause of the malloc problem… And I just realized filesource is also XSPICE code model - except its code is from 2011.)

    Note also that for 600ms transient @ 20 ns, without saving to raw file - the simulation completes; but `wrdata` fails with "malloc: Internal Error: can't allocate 268292248 bytes." With a `tran` command, `-r ngs.raw` will not help - because with `tran`, a raw file does *not* get generated (as the former posts imply). So, we need to replace that with a `.tran` command (however, `.tran` ignores the `save` command - so it needs `.save` too) - so the netlist becomes:

    * PWMexample2b.net
    .model filesrc filesource (file="pwmexample_data2.text"
    + amploffset=[0 0] amplscale=[1 1]
    + timeoffset=0 timescale=1
    + timerelative=false amplstep=false)
    * note - [3] floats, that's intended
    A1 %vd([2 0 3 0]) filesrc
    R1 2 1 1K
    C1 0 1 10n
    .control
    .endc
    * `.save 5 1 6 2` automatically resolves to V(5) V(1) V(6) V(2); binary format
    .save 5 1 6 2
    .tran 20ns 600ms
    

    … being called with `ngspice -r ngs.raw -b PWMexample2b.net`; in this case, `ngs.raw` increases as the simulation goes along.

    However, now that I need gnuplot data, I'd have to read back this *.raw file back into `ngspice`, and call `wrdata` on it… However, given that for my run of 100 ms @ 20 ns, the *binary* raw (with 5 vectors) file came out 200,001,449 bytes big - it's reasonable to expect that reading it back into `ngspice` will just cause memory allocation errors. So I wrote me a little perl script:

    http://sdaaubckp.svn.sourceforge.net/viewvc/sdaaubckp/single-scripts/spicebin2asc.pl?content-type=text%2Fplain

    … which will parse the binary data:

    perl spicebin2asc.pl ngs.raw > ngs.data
    

    … and output it in the same ASCII format as `wrdata`, usable for `gnuplot` (take note that the script excepts 1 time + 4 voltage (real) values, five columns in all - for anything else, you'd have to modify the script; but it looks almost all data is double-precision/8 bytes a piece).

    So, finally, with this workflow I can simulate 100 ms runs @ 20 ns timestep, with a `filesource` generating a PWM (originally from a .vcd)… Hope I won't run into memory allocation problems again! :) Cheers!

     
  • Holger Vogt

    Holger Vogt - 2012-08-20

    Hi,

    finally I have made an update to the d_source code:

    Originally the whole matrix of 'timepoints times input word length' was declared a state variable and stored once per accepted time event: not a very useful decision.

    Now the whole file is loaded into an extra memory space which is dynamically set up only once. The state vector now consists only of  the input vector for each specific time step.

    The tests in ngspice/tests/xspice/digital are simulated succesfully, as well as the example by sdaau with roughly 1.7 million entries (timepoints).

    Holger

     

Log in to post a comment.