[Blocks-development] (no subject)

[Spencer Jr., M. <sp...@ac...>]

I'd like to begin implementing my earlier crazy idea for content
verification and certification.

Remember that the object is to make it possible to publish a file such that
you can still remain anonymous but the general public still has some idea
that the content you're publishing is meaningful and worth downloading.
That is...I can publish an mp3 and Joe Sixpack can easily tell that this mp3
file I published really contains the song that's listed in the title.  Mr.
RIAA Consultant can publish an mp3 with a popular song's name, containing a
'public service announcement' or dogs barking or bells ringing or something
like that...and Joe Sixpack can easily tell that this is a bogus file
without taking the time to download it, and without getting frustrated with
the network.

In case I haven't made it clear before, the idea is that data will still
replicate and be reshared as before...there could be 50 copies of The Matrix
parts 1 and 2 out there.  However, when someone with a trusted public key
downloads one of these copies and finds that it's legit content, he can sign
the content and publish his signature.  People can download the
signature...download enough key material to decide whether or not they trust
his signature...and then affirm that all 50 of these copies (which are all
exactly the same) are good copies.  They can also determine that the 5 or 10
bad copies...corrupted, or malicious planted files meant to waste peoples'
time...are really bad (or at least, uncertified) content.

The first step here is to have a way to uniquely identify a file without
downloading it -- a secure hash of the file.  I'd like to make this 'secure
hash' a feature request to implement rather quickly.

What's a secure hash?  Well, what's an insecure hash?

The object of all this is to save people some downloading time -- you don't
want them to download a file only to find out it's not what they thought it
was, especially if it's a several-hundred MB movie file.  We're going to be
applying PKI certificates to these signatures...so a user is going to look
at just the hash, not the file, and decide whether or not to download the
file.  An attacker is going to attempt to foil the system by returning
search results that match known-good content, but when you download the file
that supposedly matches the hash, it's not the same file.  Of course, when
you run the hash yourself after you've downloaded the file, you'll see the
delusion -- the hash you come up with doesn't match the hash the file
publisher provided...but the damage has already been done.  You downloaded
the file, wasting your bandwidth and frustrating yourself.

A secure hash would have to resist this.  How?  Block hashes.  If this works
as it's supposed to, you will only waste a maximum of one block's worth of
bandwidth downloading bogus files.

Here's my idea so far...if you see any weaknesses or problems, please help
us strengthen it -- we'll only get one shot to do this right before the
network starts going into widespread usage.

When you get a file advertisement you should also get a three-part hash with
the file.  The first part should be a whole-content hash (like normal), the
second part should be a long multiple-block hash with one hash value per
data block, and the third part should be a hash of all the multiple-block
hashes taken together.  (You're going to use the third part to check the
second part for corruption...and you're going to compute and use your own
value for this third one anyway when you search.)

This hash should provide two things:  a unique identifier you can use to
search for certificates on the network (probably the first and third hashes
concatenated), and a set of periodic checks you can run on the file while
you're downloading it, so you can tell if you're downloading what you're
supposed to be downloading and abort the download if necessary.

(Was that pretty clear on how this should be implemented?  Questions and
comments?  :) )

So this can resist different kinds of attacks:

1)  totally bogus content with a hash that matches good content.  This is
defended against because once the client downloads and checks the first
block, they can see the file is bad and immediately stop the download, after
only downloading one block.

2)  partially-bogus or halfway-completed content:  the client will continue
to download the content until it reaches the first bad block, and then
abort.  I would say this is a good condition, but at present Blocks doesn't
support resuming downloads.  (Once we have reliable content hashes we can
securely and confidently resume downloads)


On a related note...download resumes!  Once we have per-block and per-file
hashes that get sent around with the file adverts, we can trust other hosts
that claim to have the same data...or we can automatically download from
other hosts that have the same data if the current download fails.

Of course, all this is just 'application layer' stuff -- the network layer
is top priority, and I have no right to 'demand' a feature.  If a change
needs to be made for the health of the network (or for other peoples'
ideas...I'm not the only one here with good ideas and plans) then by all
means, implement that first.  But if there are no other changes in mind...

0.15:  implement sending whole-file and block-by-block hashes along with the
file adverts.  Add someplace in the client UI to save the hash information
to a text file, so we can do sample crypto stuff by hand.
0.16:  implement checking downloads against the advertised hashes, and
aborting downloads if a received block doesn't match the block hash.
0.17:  implement download resumes from the same and multiple sources, using
the (now debugged and tested) hash stuff.

Then we get to have fun with PKI crypto!  Hope I can find a few interested
bodies willing to learn how to use GPG, and do a LOT of cutting and pasting
into and out of text files.  We will need to get a general flowchart going
for how the client can automatically check and sign content and all that, so
we can later make the Blocks client do that automatically.  Then we need to
try to break it, so it doesn't die when the system comes under attack.

Sounds like fun, right?  Just think of the positive press we'll get later
on.  :)

(Just in case this project becomes famous later down the road...if you are
press or media viewing mailing list archives, these comments are owned by me
and I do *not* license anyone to snip bits and pieces of this post, looking
for things to quote out of context.  If you reproduce this post, reproduce
it in its entirety.  I encourage everyone else on this list to say something
like this at the end of their longish posts as well.)

--Michael Spencer
bl...@ms...