#6 document how the dataset should be added to HDFS

[Anonymous]

It should be documented how the dataset can be added to HDFS.
Here's my version, but note that this does not do any optimisation of fileinput splits, I assume the mirex authors tinkered with this to achieve an optimal amount of input blocks, i.e. map jobs. I'm still working on figuring that out.

patch here http://pastie.org/3347444 or the attachment (same thing)

[Nobody/Anonymous]

[Dieter Plaetinck]

never mind the bit about input blocks. i found that increasing the blocksize to 512MB (as to have a block per file instead of about 3 for the default of 64MB blocks) didn't noticeable affect performance, as the warc files won't be split anyway. (?)
we could probably shave off some job setup overhead by using a sequencefile per directory and having the warc files as records. But most jobs seems to be mostly cpu-bound anyway.

[Robin Aly]

Thanks for the input