Re: [Blocks-development] Preventing duplicate file uploads, and feature recap

[craig <cr...@mi...>]

Michael Spencer Jr. wrote:

>Here I go, resurrecting a dead list and finding out who is still alive out
>there.  Remember Blocks?  It's still one-of-a-kind, still has all the
>disruptive-technology potential, yet is still pretty much useless because of
>all the convenience tradeoffs made to enhance security.  I'm interested in
>making some of those tradeoffs optional.
>
Hey Michael, long time no see. I noticed mspencer.net was running Redhat 
now?
I am still a Mandrake fan... hadnt heard of it until u recommended it...
ancient history *lol* :-)

I am not sure I agree with you about convenience tradeoffs against 
security.
The GUI just plain sucked, but the only reason for that was because the
focus was on hacking up the backend (which kind of sucked as well :-)

>
>
>Suppose two people have the same copy of a file they want to share.  Without
>collaborating, they both join a network and both share the file.  The same
>file data exists on the network with two different header blocks now, and
>encrypted in two different ways.  Someone could download part of one file,
>and the source would disconnect;  they would be unable to resume the file
>from the other source, because as far as the system is concerned, it's a
>different file altogether.
>
>Suppose we tried to prevent that redundancy.  When a node tries to share a
>second copy of an existing file, for which there already exists a file
>advert, the node should try to download just the header block from that
>existing copy of the file on the network.  If the file appears to be
>identical to the file this node is trying to share, the node should change
>how it caches its own copy of the file to match.  Now the existing file
>header b
>locks and file adverts on the network also refer to this file.
>
>What are the security implications of this?  Doesn't an attacker need to
>reverse RIPEMD160 to forge an impossible file header block?  And if an
>attacker publishes a nonexistent file header block and file advert, which
>eventually does point to a real file, won't the attacker have merely helped
>spread the new file?
>
I can imagine some nasty graph theory here where groups of nodes fight for
the one true key for the file. The file is not really encrypted anyway,
its just encoded using a key so that all the content blocks have unique
hashes (get rid of zero blocks etc). Why not just use a hash of the raw
file as a key. Then the blocks would be the same system wide. In fact,
why bother encoding it at all? Using blocks with the same content hash
even if they come from different files should be fine (maybe move to
a 256bit hash?)... this would be great for large files which differ
by a small amount (counterstrike 1.4 for instance :-) as previous
versions would already be commonly distributed across the net. Even
CD rips (with permission of course :-) may well have common blocks.
Reducing the block size to something sensible (4K?) would make this
even better.

>
>
>Or if the attacker is allowed to *choose* the on-disk encryption key used
>when he publishes his 'cooked' header block, will the attacker be able to
>then prove/disprove that someone was sharing a certain file by seizing the
>hard disk and looking for files encrypted with those keys?
>
On disk encryption is a different matter. My current thoughts are that this
was overkill. Paranoid people can use encrypted filesystems like scramdisk.
Hmmm... we would be able to create blocks on-the-fly from normal files if
we didnt use an encrypted cache. This would be way cool.

>
>
>----------
>
>Many months ago, I asked for some of Peter Gunn's time, and he helped me
>figure out FLTK, and walked me through some of the Blocks source.  I haven't
>done *anything* with that information yet.  Tonight I plan on printing out
>the Blocks source, to study it in my free time elsewhere.
>
>Maybe then I can add a few missing features to Blocks and make it useful for
>the masses.
>
>My ideal feature list:
>1)  add another layer of indirection to the on-disk block cache, so blocks
>can exist in the original files instead of only in new encrypted disk
>blocks.  This is a *major* killer, especially for movie downloads.  Make the
>disk cache recognize and not expire the user's own manual uploads.
>2)  make the 'download' function smarter, so it doesn't merely try to save
>one file stream, but intelligently chooses file streams and creates the
>destination file from many possible sources.
>3)  add node
> identity security -- if a node directly connects to me, I want
>to be able to verify whether or not it's a known trusted friend.  I might
>want to only publish my own file adverts to known trusted friend nodes.
>4)  Rate-limit the advert traffic.  Perhaps give 'advert-republication
>queue' priority to adverts from known trusted friend nodes.
>5)  GUI the hell out of it.
>
>What do you guys think?  Would you start using Blocks again if feature 1 was
>added, and you didn't have to devote copious amounts of disk space to
>running a Blocks node?
>
>Does disruptive filesharing technology interest the people still in this
>list any more?
>
>(I feel somewhat responsible for killing Blocks -- I kept going on and on
>with my web-of-trust drivel.  I'm sorry, guys.)
>
That wasnt drivel. Advert flooding is still a major concern. Perhaps
using a hashcash type of deal like the anonymous remailers do would
be a cheap and cheerful solution?

I am still playing with the idea of doing a Blocks2 project... over a
much larger timescale... year or two say. Maybe keep the basic idea
but build on different levels...

1) Node connection management. Have stand alone app 'noded' (or apace mod)
   that maintains a list of recent possible nodes (say 100). As new
   potential nodes query noded they are added to the list. This allows
   people to help to manage the blocknet without actaully taking part in
   any of the data sharing or relaying. Nodes could use ping times to
   choose favorable connections and drop those above a threshold. Nodes
   with low connection counts could repoll the noded periodically.
2) Same old DH key exchange for all network comms. Perhaps use Blowfish on
   top with a fixed key so that private blocknets (and noded) are possible.
3) Adverts. Same as before but with hashcash to avoid flooding.
4) Relaying data blocks. Smaller blocks (say 8Kb). Client requests a
   single header block for a file, then the header file blocks, then the
   blocks in the data file using the path from the advert. Intermediate
   nodes may be set to cache header files, or header and data files,
   and rebroadcast adverts for whatever they choose to keep.
5) Disk cache... just plain files. Perhaps just a set of 'root' cache
   directories. Create files in a temp dir containing the header blocks
   for each file and serve these as files themselves. Keep the header
   blocks for these header files in memory. (So 2Gb file, has a ~4Mb
   header file, which has a single 8K in memory header block, assuming a
   128bit block hash).
6) Simple interface for people to build GUIs against... maybe a library
   that has a minimal interface... connect_to_blocknet, advertise_file,
   query_adverts, start_file_download.

One day I might get my act together :-)

ttfn

PG.