When I try to import of 500Mb ASCII data SciDavis allocate 2-2.5Gb of memory. In the same case Origin consumes only 500Mb.
It is not important when you import 5-10Mb data, but for big input files it is significant.
I see some chance of improving the efficiency of ASCII import a bit. However, I'd like to stress once again that handling of large data sets isn't one of our primary design goals; in particular, you can easily run into memory consumption issues due to saving of undo information for all changes to a table.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I have slight improvement for it. To test it, increase table rows count to 10 000 000 (and 2 columns) on SciDAVis revision before [r1407]. In this case, SciDAVis consumes over 700 MB of memory.
After fix [r1407] it consumes about 100 MB.
I'll try to find another places, where else memory lost.
Got rid of many excess copies, and converted to floating point ASAP. ASCII import now uses a sane amount of memory for large data files - the data must still fit in memory, but only one copy, and in binary.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I see some chance of improving the efficiency of ASCII import a bit. However, I'd like to stress once again that handling of large data sets isn't one of our primary design goals; in particular, you can easily run into memory consumption issues due to saving of undo information for all changes to a table.
I have slight improvement for it. To test it, increase table rows count to 10 000 000 (and 2 columns) on SciDAVis revision before [r1407]. In this case, SciDAVis consumes over 700 MB of memory.
After fix [r1407] it consumes about 100 MB.
I'll try to find another places, where else memory lost.
Related
Commit: [r1407]
Last edit: PoZitron 2014-02-01
Got rid of many excess copies, and converted to floating point ASAP. ASCII import now uses a sane amount of memory for large data files - the data must still fit in memory, but only one copy, and in binary.