Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.

Close

#24 Option to store data in a temporary file

open
nobody
None
5
2012-10-18
2011-03-30
Andrii
No

Parsing large pages resulting in fatal error like this one:
"Allowed memory size of 67108864 bytes exhausted"
I assume this is because the DOM structure and data are being stored in RAM. This is OK for small pages, by I often run onto huge pages I need to parse.
Can the class have an option to write data to file, on the fly? And in the DOM structure within an object it'll have only pointers to that data (integers)?
So RAM will only hold information needed to search and traverse DOM tree, and action content (text nodes) will be stored in file.
If I'm not mistaking, this can significantly relieve the RAM usage.
I'd like to see that in Simple HTML DOM Parser as an option for parsing big HTML files.

Discussion

  • nickl-
    nickl-
    2012-09-11

    This has been improved with the new version

     
  • John Schlick
    John Schlick
    2012-10-18

    Start by doing an
    ini_set('memory_limit', '500M');
    to raise the limit of memory that you are allowed to use.