And you might get back more memory if you didn't have to have all the data in memory at once, but that may or may not help you. The only other suggestion I can make is to attempt to eliminate the overhead in the inner loop. Essentially, I would try making a single figure and a single AxesGrid object (before the outer loop). Then go over each subplot in the AxesGrid object and set the limits, the log scale, the ticks and the tick locater (I wouldn't be surprised if that is eating up cpu cycles). All of this would be done once before the loop you have right now. Then create the PdfPages object, and loop over all of the plots you have, essentially recycling the figure and AxesGrid object.
At end of the outer loop, instead of closing the figure, you should call "remove()" for each plot element you made. Essentially, as you loop over the inner loop, save the output of the plot() call to a list, and then when done with those plots, pop each element of that list and call "remove()" to take it out of the subplot. This will let the subplot axes retain the properties you set earlier.
I hope that made sense.