Re: [Libbt-devel] Re: Features request

[Dakidd <da...@so...>]

>>From: Dakidd <da...@so...>
>> But the amount of time it takes to accomplish can be *VERY* discouraging,
>> particularly to "newcomers" to BitTorrent. Depending on the client (and the
>> exact SHA1 implementation being used by that particular client) and the
>> size of the file(s) involved, doing the "pre-check" on a freshly
>> (re)started .torrent can take anywhere from several minutes to more than an
>> hour. (potentially *MUCH* longer)
>>
>> Case in point: A 265 meg file I've got that takes well over an hour and a
>> half to be checked by the "official" Python-based client. Same file, using
>> my partially-hacked-into-operationality LibBT (more accurately, btList and
>> btCheck - I'm still hacking on networking capability so I can get btGet up
>> and running) and a standalone SHA1 implementation (Note 1) checks in a
>> little more than 75 seconds. In the Python-based client, that delay is
>> absolutely maddening - I can fire up to seed the file I mention, and it'll
>> be close to two hours before the leechers that flock to it get the first
>> byte of it from me.
>
>What the heck kind of system do you have?!

I'm using a PowerMac 7500, G4@450. In terms of "raw power", it compares
favorably to a mid-range P2 or P3 - when the code it's running is worth
anything... Python here in Mac-land doesn't qualify as "worth anything".

In a nutshell, Python on a Mac *SUCKS*. Not just a little, but HIDEOUSLY.
It'll take the chrome off a bumper at 50 paces, and still have power left
over to pull a golf ball through a garden hose. Apps written in Python
running on a Mac suck even harder.

>On a 233MHz P2 system I can run 256MB of data (/dev/zero since it is
>handy) through in 18 seconds! By this measure I'd guess you had less than
>a 66MHz system!
>
>If this is so, do you seriously expect such a system to be able to handle
>downloading and processing such a torrent? (during download you'll be
>clobbering your I/O system grabbing pieces, when looking at the file(s)
>you'll merely peg your processor)

My tests of the parts of LibBT that I've got ported and operational are
giving me every indication that BitTorrent will run just fine - If it ever
arrives as an actual compiled app, rather than the godforsaken clusterfsk
that is the "official" client written in Python.

>> I've been chasing some ideas about how to handle this problem around in my
>> head. I've got a few thoughts, but with all of them, I'm worried that if
>> they were put into wide practice, trackers would start getting hammered
>> into the ground by the traffic of the frequent "re-registers" that would be
>> needed during the time a particular client using them is "firing up".
>
>Hammering a tracker is a concern, but how would this scenario manage to
>cause that?

Well, the method I keep coming back to, then discarding, then coming back
to, then discarding again (lather-rinse-repeat) would involve checking a
block at a time and updating the tracker after each block, rather than
doing the checking as a "monolithic" action and talking to the tracker only
once.
It's hard for me to put it into words - perhaps I ought to sit down and
concentrate on the algorithm for a bit... Later... For now, I'd be happy
just getting this beast to WORK, let alone wrok WELL, or "The way I think
it ought to".

>> (Note 1) ReidSHA, if anyone is interested. Explicitly in the public domain,
>> small footprint in the library, and QUICK. Be aware that I'm coming at
>> things from another angle than most everybody else who's hacking on this
>> thing, since I'm trying to port the library over to Pre-OSX Macintosh
>> systems - "Canned" goodies like LibCURL, the OpenSSL libraries, etc, just
>> aren't "there" for me and my platform, so I'm having to "invent the
>> technology on the fly" - This has lead me to finding various substitutes
>> and workarounds, and discovery of what I believe to be better ways for
>> doing some things. ReidSHA is one definite Good Thing.
>
>I strongly disagree.

Disagree all you like. ReidSHA passes all of the NIST test data with no
problems, which is a fairly good indication that it's correct. It's
basically a speed optimization of the NIST code, unrolling loops, stomping
out unneccesary redundancy, and otherwise reorganizing to achieve better
speed.

>OpenSSL is very common.

Show me the port for Macs running pre-MacOS X systems, and I'll show you
the code that never existed. Maybe in Windows-Land it's very common. Here
in "Mac-world", it plain doesn't exist - Period. (And save your breath -
I'm *NOT* going to be the person who ports it across - I've already got
plenty on my plate already, thank you very much!)


>This means they've got the
>eyeballs to look at code and make sure it is correct and fast.

By all information I can find, that's already been dealt with in ReidSHA.

>Using an
>uncommon library is bad because you force people to go hunting to obtain
>it, and they may choose to go elsewhere and no bother.

Actually, it's not a library - Just some code that I'm rolling directly
into LibBT. Once I get things cranked up (if I ever manage to find enough
time to do so) there will be no need to "go hunting" for anything - I
intend to make LibBT for the Mac be a truly "standalone" library, not
dependent on anything except the standard C library (and as little of that
as I can possibly get away with - I'm still debating whether it ought to
have part of the functionality of "string.h" rolled directly in so that I
can get away from depending on even that much external support code) and
OS-supplied services.

For my nickel, the "ideal" LibBT port to Mac will have *NO* dependencies
other than OS-supplied stuff (Primarily networking and disk-access,
although a couple other minor things will also be needed - "print to the
screen", "read the keyboard", etc) rather than needing umpty-seven
libraries (several of which just plain don't exist for Macs) worth of
support code. That's the target I'm aiming at - Truly a standalone library.
Quite frankly, the concept of needing to drag half a dozen "foreign"
libraries into the linkage so that the library that's being built can run
offends my sensibilities in a big and ugly fashion. And that doesn't apply
only to LibBT - That goes for *ANY* library code. If it can't be written to
be standalone, requiring no support other than OS-level primitives for I/O
and networking, then it doesn't qualify as a "proper" library by my
standards.

>For crypto is an
>extremely bad idea as it is difficult to get right.

All reviews I've stumbled across give ReidSHA high marks for both
correctness and execution speed. Likewise, my own testing, limited though
it may be due to not having a "master cryptologist" degree, seems to
indicate that it has no significant flaws. It appears it's been "got right".



Don Bruder -  da...@so...      <--- Preferred Email - unmunged
I will choose a path that's clear: I will choose Free Will! - N. Peart