-
Rather. It was a 15 megabyte file, not 1.5. I can count.
2007-10-02 14:13:54 UTC in Infomap NLP Software
-
I noticed that in a very large multiple file corpus (circa 1 million files), the prepare_corpus process was running out of memory while allocating space to store the list of file names. This made little sense to me so I looked at what was happening:
For every filename, a new (system dependent value) 8k buffer was being allocated. This way, 1 million filenames required 8gb of buffers to be...
2007-10-02 13:13:16 UTC in Infomap NLP Software
-
This patch fixes the segmentation fault seen after calling mymalloc in 64-bit systems. This uses the relatively heavy handed solution of removing mymalloc, and only using malloc, though it does verify that malloc did not return NULL.
2007-10-02 12:55:59 UTC in Infomap NLP Software
