Re: [sparql-wrapper-devel] Supporting keep-alive in SPARQLWrapper

[Sergio F. <ser...@fu...>]

Dear Pēteris,

thanks you very much to point us such feature in urllib, it's good to
know it while that wrapper strong depends on that library.

We'll study how to include it, but I agree with you that should be
something optional. Meanwhile you can register it as a feature request
in SourceForge:

http://sourceforge.net/tracker/?group_id=217900&atid=1041490

Thanks you very much!

Best,

On Fri, 2010-01-08 at 13:18 +0000, Pēteris Caune wrote:
> Hello, 
> 
> 
> I recently ran into an issue when using SPARQLWrapper on Windows and
> executing many queries in quick succession. My script failed with:
> 
> 
>           (...)
>           File "E:\Python26\lib\site-packages
>         \sparqlwrapper-1.4.0-py2.6.egg\SPARQLWrappe
>         r\Wrapper.py", line 343, in query
>           File "E:\Python26\lib\site-packages
>         \sparqlwrapper-1.4.0-py2.6.egg\SPARQLWrappe
>         r\Wrapper.py", line 314, in _query
>           File "E:\Python26\lib\urllib2.py", line 124, in urlopen
>             return _opener.open(url, data, timeout)
>           File "E:\Python26\lib\urllib2.py", line 389, in open
>             response = self._open(req, data)
>           File "E:\Python26\lib\urllib2.py", line 407, in _open
>             '_open', req)
>           File "E:\Python26\lib\urllib2.py", line 367, in _call_chain
>             result = func(*args)
>           File "E:\Python26\lib\urllib2.py", line 1146, in http_open
>             return self.do_open(httplib.HTTPConnection, req)
>           File "E:\Python26\lib\urllib2.py", line 1121, in do_open
>             raise URLError(err)
>         urllib2.URLError: <urlopen error [Errno 10048] Only one usage
>         of each socket add
>         ress (protocol/network address/port) is normally permitted>
>         
>         
> Googling for the error message turned up the issue([1]). In short,
> every request creates a connection and leaves it in TIME_WAIT state,
> and after 4000 or so requests client machine runs out of free ports.
> The solution is either to tweak Windows Registry to allow more ports
> to be used or to use HTTP 1.1 keep-alive. I investigated the second
> option a bit, found an example how to do keep-alive with urllib2
> ([2]). So I tried it in practice: installed urlgrabber with
> easy_install, put the following lines before class definitions in
> Wrapper.py:
> 
> 
>         # Make urllib2 use keep-alive
>         from urlgrabber.keepalive import HTTPHandler
>         keepalive_handler = HTTPHandler()
>         opener = urllib2.build_opener(keepalive_handler)
>         urllib2.install_opener(opener)
> 
> 
> Running my script again and monitoring netstat confirmed that now only
> a single HTTP connection was used. 
> Could this feature be somehow optionally integrated in SPARQLWrapper?
> Maybe a new method "setUseKeepAlive" (or similar) could be added to
> SPARQLWrapper class that executes these lines (if urlgrabber is
> available)? Or would this be outside the scope of SPARQLWrapper... I
> asume urllib2 needn't be configured right in Wrapper.py, this could
> also be done in my client code.
> 
> 
> Peteris
> 
> 
> [1] http://blogs.msdn.com/dgorti/archive/2005/09/18/470766.aspx
> [2] http://stackoverflow.com/questions/1037406/python-urllib2-with-keep-alive
> ------------------------------------------------------------------------------
> This SF.Net email is sponsored by the Verizon Developer Community
> Take advantage of Verizon's best-in-class app development support
> A streamlined, 14 day to market process makes app distribution fast and easy
> Join now and get one step closer to millions of Verizon customers
> http://p.sf.net/sfu/verizon-dev2dev 
> _______________________________________________ sparql-wrapper-devel mailing list spa...@li... https://lists.sourceforge.net/lists/listinfo/sparql-wrapper-devel

-- 
Sergio Fernández
Research Assistant
ser...@fu...
CTIC - Technological Center
Parque Científico y Tecnológico de Gijón
C/ Ada Byron, 39
Edificio Centros Tecnológicos
33203 Gijón - Asturias - Spain
Phone: +34 984 29 12 12 - Fax: +34 984 39 06 12
www.fundacionctic.org