Though I'm an absolute beginner in CL I took a look to the code, I just missed a bit more of information for describing the problem, and also I couldn't really run the code since unit tests use some files not in the repo (currently myfavlibrary.exe).

I think the main problem is the file-to-bytes function, you are loading the entire file in memory (though I don't know the size of that specific file which made the test crash). Also the function bytes which splices the big vector into a list of bytes.
As I said I'm a beginner, maybe all that is not correct, I would use arrays instead of lists in the bytes function since you always pass the size (in count param). Maybe you dont need at all copying splices of the big array into new objects since they get unreachable soon (they got translated to ints, longs, or so).
As for avoiding load the entire file in mem I don't know a nice solution, you can always use file-position to seek in the file but I don't know if doing this the reads get buffered.

Also I noted you use loop with index for arrays, you can use across for sequences and subseq for splicing the sequences. I'm calling splice to a subsequence sorry if it sounds misleading. A simple test over a (make-array 10000000 :element-type 'integer) performed 20% faster with across than with the index approach summing all zeros in the array. Oh! also a tip, (loop for i to (1- x) == (loop for i below x, well maybe you tip more this way...


2014-02-04 Jacek Podkanski <ruby.object@googlemail.com>:
Adding this code at the end of my tests bring heap use back to normal.

     (setf bytes nil
           mem nil)
     (sb-ext:gc :full T)

------------------------------------------------------------------------------
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk
_______________________________________________
Sbcl-help mailing list
Sbcl-help@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sbcl-help