Though I'm an absolute beginner in CL I took a look to the code, I just missed a bit more of information for describing the problem, and also I couldn't really run the code since unit tests use some files not in the repo (currently myfavlibrary.exe).
I think the main problem is the file-to-bytes function, you are loading the entire file in memory (though I don't know the size of that specific file which made the test crash). Also the function bytes which splices the big vector into a list of bytes.
As I said I'm a beginner, maybe all that is not correct, I would use arrays instead of lists in the bytes function since you always pass the size (in count param). Maybe you dont need at all copying splices of the big array into new objects since they get unreachable soon (they got translated to ints, longs, or so).
As for avoiding load the entire file in mem I don't know a nice solution, you can always use file-position to seek in the file but I don't know if doing this the reads get buffered.