pydc-devel

[pyDC-devel] Latest pyDC updates

[Anakim B. <ab...@us...>]

pyDClib/AsyncSocket.py: more informative DNS error messages
pyDClib/DCHub.py: update info after logout
pyDClib/DCHub.py: more informative socket error messages
pyDClib/DCHublistRetriever.py: support for bz2-compressed lists
pyDClib/DCQueue.py: addQueueItem() accepts a 'priority' flag
pyDClib/DCQueue.py: avoid alternate source search for corrupted items
pyDClib/DCQueueItem.py: detect possible item corruption
pyDClib/DCUserList.py: use high priority queueing
pyDClib/DCWorker.py: signal upload banwidth limit via description tags
pyDClib/DCWorker.py: getLocalList() is now thread safe
pyDClib/DCXfer.py: minimal extended protocol support (makes happy many bots)
pyDClib/DCXfer.py: code cleanup
pyDClib/DNS.py: major rewrite; better error recovery using secondary 
nameservers
pyDClib/EventGenerator.py: use weakrefs to keep track of listeners
pyDClib/win32dns.py: many big fixes
pyDClib/win32dns.py: code cleanup
ChatViewer.py: support for away mode
HubViewer.py: fixed scrolling problems for Windows systems
HubViewer.py: fixed command processing
MainWnd.py: set focus to viewer when activating it
MainWnd.py: support for away messages
QueuePanel.py: status description table no longer needed
SearchViewer.py: setup combo box default value
Settings.py: accept bz2-compressed hublists
Settings.py: added various default values
Settings.py: bugfix: ip field must be enabled only if 'ForceIP' is true
Settings.py: refresh worker settings only when needed
XfersPanel.py: handle gracefully xfers with null item
XfersPanel.py: code cleanup
pydc.py: fix import path
pydc.py: handle IP detection failure
pyDC.sh: removed
pyDC.bat: removed

-- 
	Anakim Border
	ab...@us...
	http://pydc.sf.net

[pyDC-devel] Latest pyDC updates

[Anakim B. <ab...@us...>]

pyDClib/he3/he3.py: code cleanup;
pyDClib/AsyncSocket.py: code cleanup;
pyDClib/Channel.py: unicode support;
pyDClib/DCChannel.py: code cleanup;
pyDClib/DCChannel.py: flushCmd replaced by onDisconnection handler;
pyDClib/DCDownload.py: remove source on command error;
pyDClib/DCHub.py: cache password;
pyDClib/DCHub.py: block duplicate password requests;
pyDClib/DCHub.py: specify nick when starting DCXfers;
pyDClib/DCHub.py: nick and password as constructor args;
pyDClib/DCHub.py: call DCChannel.onDisconnection;
pyDClib/DCHublistRetriever.py: handle empty lists;
pyDClib/DCSettings.py: support for nick and password in favorites;
pyDClib/DCSettings.py: nick length limit set to 32 chars;
pyDClib/DCWorker.py: use PYDCLIB_ID and PYDCLIB_VERSION symbols when building 
description tags;
pyDClib/DCWorker.py: support different nicks on different hubs;
pyDClib/DCXfer.py: support for multiple nicks;
ChatViewer.py: get user nick from hub object, not from settings;
Favorite.py: initial import;
Favorite.py: set initial autoconnect value;
HubViewer.py: another attempt to fix user list sync;
HubViewer.py: clearAll is called through a wxPython event;
HubViewer.py: faster sorting algorithm;
HubViewer.py: fixed race condition;
HubViewer.py: (re)inserted user deletion code lost in last commit;
HublistPanel.py: new layout;
HublistPanel.py: validateFavorite requires 3 args;
JobViewer.py: notify viewer closure before Yielding;
JobViewer.py: catch PyAssertionError raise by nested Yield() calls;
MainWnd.py: bugfix: added self.viewer == viewer check in two event handlers;
MainWnd.py: support for automatic favorite hub authentication;
MainWnd.py: added JobInfos.remove();
MainWnd.py: center settings dialog before display;
pydc.py: added missing DCWorker.stop() call;
pydc.py: updated version banner;
pydc.py: added "#!/usr/bin/env python";
pydc.py: commented out debug code;
pydc.py: call wnd.Close() instead of returning False from OnInit();
pydc.py: center settings dialog before showing it.

----------------------

Since pyDC seems quite stable, I've prepared some packages (release candidate 
1). Here's the list:

http://pydc.sf.net/pydc-0.7-rc1-win.zip
http://pydc.sf.net/pydc-0.7-rc1-linux.tar.bz2
http://pydc.sf.net/pydc-0.7-rc1-src.zip
http://pydc.sf.net/pydc-0.7-rc1-src.tar.bz2

The first two are binary packages: this means they contain everything needed 
to run pyDC, including the Python interpreter.
If you try them, please let me know what happens.

Bye

-- 
	Anakim Border
	ab...@us...
	http://pydc.sf.net

[pyDC-devel] Latest pyDC updates

[Anakim B. <ab...@us...>]

pyDClib/he3/Makefile: cleanup;
pyDClib/DCHub.py: handle $HubName after login phase;
pyDClib/DCLocalList.py: keep track of scanned paths;
pyDClib/DCSettings.py: added isValid() member;
pyDClib/DCWorker.py: added PYDCLIB_DCPP_VERSION symbol;
pyDClib/DCWorker.py: refreshSettings will rebuild local file list only when 
really needed;
pyDClib/DCWorker.py: save valid settings only;
HubViewer.py: fixed chat text initial scrolling problem;
MainWnd.py: fixed addFavorite call;
Settings.py: detect socket errors;
pydc.py: load settings only once;
pydc.py: code cleanup;
pydc.py: fixed import paths order.

Bye

-- 
	Anakim Border
	ab...@us...
	http://pydc.sf.net

[pyDC-devel] Latest pyDC updates

[Anakim B. <ab...@us...>]

pyDClib/DCFileList.py: added userNick property;
pyDClib/DCHub.py: added timestamps to log rows;
pyDClib/DCHub.py: log invalid messages to chat;
pyDClib/DCQueueItem.py: fixed variable name;
ChatViewer.py: added timestamp support;
FileListViewer.py: fixed alternate source loop;
HubViewer.py: added timestamp support;
MainWnd.py: added timestamp support.

Bye

-- 
	Anakim Border
	ab...@us...
	http://pydc.sf.net

[pyDC-devel] Latest pyDC updates

[Anakim B. <ab...@us...>]

pyDClib/DCSettings.py: added custom properties support;
ChatViewer.py: bugfix: removed 'time' parameter from various functions;
ChatViewer.py: bugfix: added parent constructor to ChatViewerInfo;
ChatViewer.py: using custom properties;
HubsPanel.py: message text is now in row[2];
HubViewer.py: using custom properties;
MainWnd.py: set showTimestamps value inside setupViewer();
MainWnd.py: added 'Away' menu;
MainWnd.py: using custom properties;
QueuePanel.py: ask to delete incomplete items when dequeuing;
pydc.py: setup custom properties;
pydc.py: code cleanup.

Bye

-- 
	Anakim Border
	ab...@us...
	http://pydc.sf.net

[pyDC-devel] Latest pyDC updates

[Anakim B. <ab...@us...>]

pyDClib/DCDownload.py: use new DCQueueItemSource.getPath format;
pyDClib/DCHub.py: raise ValueError when sending PM to unknown user;
pyDClib/DCLocalListWriter.py: static methods;
pyDClib/DCLocalListWriter.py: bz2 compression support;
pyDClib/DCLocalListWriter.py: new cleanup() function;
pyDClib/DCQueueItem.py: new DCQueueItemSource.getPath prototype;
pyDClib/DCUpload.py: support for bz2 lists;
pyDClib/DCUserList.py: support for bz2 lists;
pyDClib/DCSettings.py: bugfix: accept favorite with None nick and password;
pyDClib/DCWorker.py: bugfix: send MyInfo on refresh settings;
pyDClib/DCWorker.py: use DCLocalList.cleanup();
pyDClib/DCWorker.py: new FILELIST_{HE3,BZ2} symbols;
pyDClib/DCXfer.py: proper support for protocol extensions;
ChatViewer.py: new timestamp format ('[stamp]') for better readability;
ChatViewer.py: disable away mode when sending message;
ChatViewer.py: different colors for incoming and outcoming messages;
HubViewer.py: new timestamp format ('[stamp]') for better readability;
HubViewer.py: chat textbox must be init differently when using GTK and 
Windows;
HubViewer.py: another scroll fix;
HubViewer.py: catch ValueError raised by DCHub.sendMsg();
HublistPanel.py: bugfix: addFavorite call;
HublistPanel.py: bugfix: GetNextItem call with named args;
MainWnd.py: added safeMode machinery;
QueuePanel.py: ask confirmation before removing item if in safeMode
Util.py: added boolFilter method;
XfersPanel.py: ask confirmation before stopping xfer if in safeMode;
pydc.py: bugfix: set showTimestamp on first run;
pydc.py: safeMode support.

-----------------

In a few hours I'll update RC1 packages.

Bye

-- 
	Anakim Border
	ab...@us...
	http://pydc.sourceforge.net

[pyDC-devel] Latest pyDC updates

[Anakim B. <ab...@us...>]

In the last two weeks I've made a number of changes to pyDC sources. Here's 
the list:

pyDClib/AsyncSocket.py: added unrecv() method;
pyDClib/AsyncSocket.py: added getsockname() method;
pyDClib/AsyncSocket.py: some speed optimizations;
pyDClib/AsyncSocket.py: use lambda function to avoid memory leak;
pyDClib/bloom.py: initial import;
pyDClib/Channel.py: added ChannelEventListener;
pyDClib/Channel.py: added onChannelClosed event notification;
pyDClib/DCChannel.py: avoid loosing commands when changing handler (call 
sock.unrecv());
pyDClib/DCChannel.py: replaced deprecated apply();
pyDClib/DCChannel.py: some speed optimizations;
pyDClib/DCChannel.py: new calling scheme for command handlers;
pyDClib/DCDownload.py: support for segmented zlib-compressed downloads;
pyDClib/DCDownload.py: support for new removeSource() signature;
pyDClib/DCFileList.py: no longer needed;
pyDClib/DCHub.py: match new DCLocalList interface;
pyDClib/DCHub.py: avoid memory leaks by using class.method instead of 
self.method references;
pyDClib/DCHub.py: better PM parsing;
pyDClib/DCHub.py: fixed registerListener signature;
pyDClib/DCHub.py: cleanup of socket event handlers;
pyDClib/DCHubEventListener.py: now subclass of ChannelEventListener;
pyDClib/DCItem.py: __slots__ to reduce memory consumption;
pyDClib/DCLocalList.py: full rewrite;
pyDClib/DCLocalList.py: Bloom filter support;
pyDClib/DCLocalListWriter.py: match DCLocalList changes;
pyDClib/DCQueue.py: new addItem() signature;
pyDClib/DCQueue.py: upper limit to the number of queue items processed by 
single poll() calls;
pyDClib/DCQueueItem.py: full rewrite;
pyDClib/DCQueueItem.py: new removeSource() signature;
pyDClib/DCSearchResult.py: __slots__ to reduce memory consumption;
pyDClib/DCSettings.py: moved default values to class constants;
pyDClib/DCUpload.py: support for segmented zlib-compressed uploads;
pyDClib/DCUpload.py: avoid memory leaks by using class.method instead of 
self.method references;
pyDClib/DCUser.py: __slots__ to reduce memory consumption;
pyDClib/DCUserList.py: DCUserList is no longer a Job;
pyDClib/DCUserList.py: removeSource() dequeues item;
pyDClib/DCUserList.py: moved UserDir and UserFile from DCFileList.py;
pyDClib/DCUserList.py: delete downloaded list after parsing;
pyDClib/DCUserList.py: new removeSource() signature;
pyDClib/DCWorker.py: call DCLocalList.freeze();
pyDClib/DCWorker.py: experimental IP change detection;
pyDClib/DCXfer.py: support for segmented zlib-compressed xfers;
pyDClib/DCXfer.py: avoid memory leaks by using class.method instead of 
self.method references;
pyDClib/DCXferWorker.py: xferBandwidth() returns 0 to signal unlimited 
bandwidth;
pyDClib/DNS.py: improved /etc/resolve.conf parsing;
pyDClib/DNS.py: avoid memory leak by using lambda functions;
ChatViewer.py: multirow message support;
FileListViewer.py: removed DCFileList references;
FileListViewer.py: support for delayed queue refresh;
HubViewer.py: added 'Copy nick' and 'Remove user from queue' commands;
HubViewer.py: support for multirow messages;
HubViewer.py: match DCUserList changes;
HubViewer.py: sendMsg displays error messages in main chat view;
HubsPanel.py: listen for Channel events;
HubsPanel.py: handle onChannelClosed instead of onHubDisconnection;
HubsPanel.py: bugfix: call deregisterListener() to free resources;
MainWnd.py: bugfix: searchChat is no longer messing with viewer stack;
MainWnd.py: removed DCFileList references;
MainWnd.py: added enableQueueRefresh() method;
MultirowMessage.py: initial import;
QueuePanel.py: match DCQueueItem changes;
QueuePanel.py: identify user lists by user nick;
QueuePanel.py: experimental support for delayed refresh;
SearchesPanel.py: call deregisterListener() to free resources;
SearchViewer.py: match DCUserList changes;
Settings.py: use default values from DCSettings;
XfersPanel.py: 3:2 ratio for downloads/uploads lists;
XfersPanel.py: added popup menu;
XfersPanel.py: '*' to signal zlib-compressed xfers;
XfersPanel.py: '!' to signal multi-source downloads;
pydc.py: catch all queue errors;
pydc.py: clear clipboard on exit to avoid segfaults;
pydc.py: don't try to save malformed settings;

-------------------

pyDC now supports the 'GetZBlock' protocol extension, thus allowing segmented 
zlib-compressed file transfers. To get the most, I rewrote the queue item 
manager from scratch. The new code is able to deal with segments of variable 
size and to rearrange them on the fly; each segment is independent from the 
others and that means that pyDC can download a single item from multiple 
sources at once. Since most hub owners forbid multidownloads, the feature is 
disabled by default: if you are allowed to and want to try it out, simply 
enter the following command in the Python interpreter:

DCQueue.MAX_CONCURRENT_DOWNLOADS=5

You may change the value depending on your needs, but keep it low (since your 
bandwidth is limited, setting it to 8 or to 100 yields no difference; in fact 
bigger values also means bigger overhead, so 100 may even be *worse* than 8).
Resetting MAX_CONCURRENT_DOWNLOADS to 1 will disable multidownloads (0 will 
stop downloads at all :-) ).
The GUI has been updated to inform you about the extrension: a '*' mark before 
the filename in the Xfers panel means that the xfer is using the zlib 
compression; a '!' means that the download refers to part of a 
multidownloaded item.

The other important area I've been working on is the search engine (the piece 
of code handling search request from other peers to you). Believe it or not, 
this is the place where pyDC spends most of its time; it is crucial to make 
"local" searches as fast as possibile.
The old code seemed fast; it was also severely bugged (it never returned 
results to queries containing more than one search term).
As a first task I wrote a new generic search engine; by replacing objects with 
tuples and by using a smarter algorithm, I was able to make the new code 
twice as fast.
Then I moved to size-constrained searches (you know: at least 1MB / at most 
1GB); measurements showed me that around 85% of all searches fall into this 
category. One can exploit that constraints to avoid filename pattern searches 
(slow) on files outside size limits; that reduces the search space and, 
hopefully, makes the search engine faster. The algorithm I used is the 
following: when the file list is initially created, maximum and minimum file 
size are noted. At that point all files are placed into one of two big lists, 
depending on their size:

[ all files with size below (max-min)/2 + min, all files above ]

The process continues by splitting each of those two lists in the same way; at 
the end all files will be grouped into classes having a fixed size range. 
When a search request with size constraints comes in, the search engine finds 
the first class matching the limits; only at that poing filename pattern 
search starts.

The last improvement is also the biggest. It is based on the genial idea that 
goes under the name of "Bloom filter". I won't cover the details here, since 
a number of guides on the subject already exist on the web (google is your 
friend) and since the implementation is straightforward (take a look at 
pyDClib/bloom.py).

As a last note I'd like to give you an idea of the impact of the 
optimizations. On my Athlon 1800 server, having 293 indexed files, the 
percentage of CPU usage by pyDC as reported by top before the changes was 
fluctuating around 15%; now it's around 0.7%.


As always you can get all the changes using CVS or you may download the 
packages from:

http://pydc.sourceforge.net/pydc/rc2.html

-- 
	Anakim Border
	ab...@us...
	http://pydc.sourceforge.net

[pyDC-devel] Latest pyDC updates [2]

[Anakim B. <ab...@us...>]

> http://pydc.sourceforge.net/pydc/rc2.html

Typo. The correct address is:

http://pydc.sourceforge.net/rc2.html

-- 
	Anakim Border
	ab...@us...
	http://pydc.sourceforge.net

Re: [pyDC-devel] Latest pyDC updates

[Luca M. <cb...@in...>]

Anakim Border wrote:
:

Ciao e Buon anno !
Le ultime modifiche mi danno questo errore...

[tlu@localhost pydc]$ python pydc.py

Gdk-WARNING **: locale not supported by C library
Traceback (most recent call last):
   File "pydc.py", line 14, in ?
     from MainWnd import *
   File "/u2/svil/python/pyDC/pydc/MainWnd.py", line 7, in ?
     from Settings import *
   File "/u2/svil/python/pyDC/pydc/Settings.py", line 12, in ?
     import DCWorker
   File "pyDClib/DCWorker.py", line 16, in ?
     from DCHub import *
   File "pyDClib/DCHub.py", line 12, in ?
     from DCChannel import *
   File "pyDClib/DCChannel.py", line 6, in ?
     from Channel import *
   File "pyDClib/Channel.py", line 6, in ?
     from AsyncSocket import *
   File "pyDClib/AsyncSocket.py", line 13, in ?
     class AsyncSocket(object, Job):
NameError: name 'Job' is not defined

Ho iniziato ad usare python al lavoro, questo mi permettera' di
poterti dare una mano in seguito.

luca