[Vorbb-tech-devs] FTP Server/Netflix Dataset

[Brian C. B. <bri...@gm...>]

Hey guys,

I created an FTP server on one of Jimmy's lab computers. This will allow us
to share large data files and stuff that's too big to go in the SVN (small
data files less than a meg are OK to put in the SVN). If you are on campus,
the IP is:

10.173.215.134

To get into FTP, the username is "vorbb-tech" and the password is again the
normal one. If you are off campus, grab Hamachi and join the network
VORBB-TECH with the password being the normal one. The computer is called
jsl-4500. 

I wrote up a program to convert the 2 GB ASCII text data files provided by
Netflix into a 500 MB packed binary data file. There is a program you can
use to do the conversion in the svn (takes about 15 minutes) or you can
download the already converted binary file from the FTP. 

One the code related side of things, the past two days I coded up some
classes that will let us easily access the Netflix data. It is memory
intensive, because it loads the 500 MB dataset into memory. However, once in
memory, everything is really fast. So you can go through the 100+ million
ratings in about 95 seconds. Not too shabby. I can probably optimize that
number a bit too. I haven't gotten a chance to test/comment very much, so if
you do take a look, be warned. If you find bugs, let me know and I'll fix
them ASAP. There is also a test program in there that demonstrates how to
load in the data. 

Well that's all for now folks. 

Sincerely,
Brian C. Becker @ www.BrianCBecker.com