imaplib2-devel — imaplib2 development list

Re: [Imaplib2-devel] imaplib2 debugging

[Sebastian S. <seb...@ss...>]

On Mon, 09 May 2011 06:31:01 +1000, Piers Lauder <pi...@ja...> wrote:
> No, that can be restated as:
>   (debug is not None and debug) or (Debug is not None and Debug) or 0
> Which is the same as:
>   debug or Debug or 0
> which is an int (provided debug and Debug are).  Deprecated Python idiom :-)

DOH, and that was me thinking boolean arithmetic leads to boolean
results :). I am clearly a python rookie ;)

Thanks for the patience with me...

Sebastian

Re: [Imaplib2-devel] default argument choice

[Sebastian S. <seb...@ss...>]

On Mon, 09 May 2011 08:56:01 +1000, Piers Lauder <pi...@ja...> wrote:
> I've fixed the code to avoid the ambiguous-looking booleans.
> Thanks for the heads-up :-)

Much appreciated ;). Now, on to using the latest offlineimap2 in our
project.

BTW; is there any reason why imaplib2 can not go into python as imaplib
replacement? imaplib there seems to be kind of maintainerless.

Sebastian

[Imaplib2-devel] default argument choice

[Piers L. <pi...@ja...>]

I've fixed the code to avoid the ambiguous-looking booleans.
Thanks for the heads-up :-)

Piers.

Re: [Imaplib2-devel] imaplib2 debugging

[Piers L. <pi...@ja...>]

On Sat, 07 May 2011 11:31:06 +0200, Sebastian Spaeth wrote:
  > 
  > On Sat, 07 May 2011 10:37:45 +1000, Piers Lauder <pi...@ja...> wrote:
  > > I have committed a new version of imaplib2 with extra logging, and a
  > > new paremeter to the class 'debug_buf_lvl' (default: 3), that controls
  > > log buffering.
  > > 
  > > So, to invoke IMAP4 with output to stderr at logging level 3, one would
  > > invoke as:
  > >   IMAP4(..., debug=3, debug_buf_lvl=2)
  > 
  > Thanks, I see that  debug=3, debug_buf_lvl=2, we would not actually log
  > _log(3,... messages, but we would add them to the circular buffer of 20
  > last debug messages, right?

Right.

  > > More lock traces now come out at debug level 3, which involved much less
  > > output than level 5 (which traces actual I/O).
  > 
  > Uhh, we probably should reduce our default logging level to 4 (it is 5
  > now in cases we have debug turned on).

4 would be ok as well (and add more useful info than 3).

  > > Hope that helps...
  > 
  > One question, in _log() you have:
  >                 self.debug_lock.acquire()
  >                 self._mesg(line, tn)
  >                 self.debug_lock.release()
  > 
  > So you have a per IMAP-instance lock protecting from concurrent writes
  > to stderr. However, given our multithreaded, multiconnection nature, we
  > can (and do) and multiple IMAP instances running simultanously, and they
  > share the same log file (debug_file). I am not sure this could cause
  > hangs, but could certainly lead to corrupted debug output, can't it?

It would mean than any one imaplib2 instance synchronises its own threads
when writing to the logging instance. If multiple objects are going to
write to the one logger, than that loogger itself needs to institute
synchronisation. There would be no corruption and no conflicts provided
it uses it's own semaphore in code like that above. (In which case of course,
the imaplib2 synchronisation becomes redundant.)


  > Sorry, threads and locks give me the creeps and make my head hurt :).
  > 
  > I will propose a patch to offlineimap, reducing our default debug level
  > and updating imaplib2 to the latest and greatest. As we plan to
  > introduce the IDLE support very soon. I see you have locking fixes for
  > IDLE too, are they crucial?

They are because the IDLE can be cancelled from more than one thread,
and though not crucial, I've included them for thoroughness sake. Ie:
the lack wouldn't cause a hang, just that the IDLE might not start
properly (only to be immediately cancelled by some activity). So a
piece of code added for the sake of correctness, rather than essential.

Piers.

Re: [Imaplib2-devel] imaplib2 debugging

[Piers L. <pi...@ja...>]

On Sat, 07 May 2011 11:42:32 +0200, Sebastian Spaeth wrote:
  > 
  > On Sat, 07 May 2011 10:37:45 +1000, Piers Lauder <pi...@ja...> wrote:
  > > I have committed a new version of imaplib2 with extra logging, and a
  > > new paremeter to the class 'debug_buf_lvl' (default: 3), that controls
  > > log buffering.
  > > 
  > > So, to invoke IMAP4 with output to stderr at logging level 3, one would
  > > invoke as:
  > >   IMAP4(..., debug=3, debug_buf_lvl=2)
  > 
  > One more question:
  > 
  > Calling IMAP4(debug=3,...) will call:
  > 
  > _init_debug(self, debug=3,...): 
  >   self.debug = debug is not None and debug or Debug is not None and 
               Debuggor 0
  > 
  > so self.debug becomes a Boolean, rather than an int, right?

No, that can be restated as:
  (debug is not None and debug) or (Debug is not None and Debug) or 0
Which is the same as:
  debug or Debug or 0
which is an int (provided debug and Debug are).  Deprecated Python idiom :-)

  > Because in def _log() and elsewhere you use:
  > 
  > if lvl>self.debug:
  >      <actually log something>
  > 
  > which would be true for any value >1 of lvl. Or am I grossly misreading
  > things?

See above :-)

Re: [Imaplib2-devel] imaplib2 debugging

[Nicolas S. <nic...@la...>]

On Fri, May 06, 2011 at 11:00:14AM +0200, Sebastian Spaeth wrote:
> On Fri, 06 May 2011 13:10:14 +1000, Piers Lauder <pi...@ja...> wrote:
> > Maybe it would help to be able to turn on less verbose debugging, so how
> > about introducing another debugging parameter that controls the level at
> > which logging is redirected from the internal buffer to immediate output
> > (currently 4). Then one can turn on P/V logging at, say, level 3, and
> > be able to trace it when looking for a (possible) lock conflict.
> 
> Yes, implementing a logging level is very much part of my
> plans. Nicolas, you also favored that recently, right?

Yes, it would be a usefull feature for OfflineIMAP, I guess.


-- 
Nicolas Sebrecht

Re: [Imaplib2-devel] imaplib2 debugging

[Sebastian S. <Seb...@SS...>]

On Sat, 07 May 2011 10:37:45 +1000, Piers Lauder <pi...@ja...> wrote:
> I have committed a new version of imaplib2 with extra logging, and a
> new paremeter to the class 'debug_buf_lvl' (default: 3), that controls
> log buffering.
> 
> So, to invoke IMAP4 with output to stderr at logging level 3, one would
> invoke as:
>   IMAP4(..., debug=3, debug_buf_lvl=2)

One more question:

Calling IMAP4(debug=3,...) will call:

_init_debug(self, debug=3,...): 
  self.debug = debug is not None and debug or Debug is not None and 
               Debuggor 0

so self.debug becomes a Boolean, rather than an int, right?

Because in def _log() and elsewhere you use:

if lvl>self.debug:
     <actually log something>

which would be true for any value >1 of lvl. Or am I grossly misreading
things?

Sebastian

Re: [Imaplib2-devel] imaplib2 debugging

[Sebastian S. <Seb...@SS...>]

On Sat, 07 May 2011 10:37:45 +1000, Piers Lauder <pi...@ja...> wrote:
> I have committed a new version of imaplib2 with extra logging, and a
> new paremeter to the class 'debug_buf_lvl' (default: 3), that controls
> log buffering.
> 
> So, to invoke IMAP4 with output to stderr at logging level 3, one would
> invoke as:
>   IMAP4(..., debug=3, debug_buf_lvl=2)

Thanks, I see that  debug=3, debug_buf_lvl=2, we would not actually log
_log(3,... messages, but we would add them to the circular buffer of 20
last debug messages, right?

> More lock traces now come out at debug level 3, which involved much less
> output than level 5 (which traces actual I/O).

Uhh, we probably should reduce our default logging level to 4 (it is 5
now in cases we have debug turned on).

> Hope that helps...

One question, in _log() you have:
                self.debug_lock.acquire()
                self._mesg(line, tn)
                self.debug_lock.release()

So you have a per IMAP-instance lock protecting from concurrent writes
to stderr. However, given our multithreaded, multiconnection nature, we
can (and do) and multiple IMAP instances running simultanously, and they
share the same log file (debug_file). I am not sure this could cause
hangs, but could certainly lead to corrupted debug output, can't it?

Sorry, threads and locks give me the creeps and make my head hurt :).

I will propose a patch to offlineimap, reducing our default debug level
and updating imaplib2 to the latest and greatest. As we plan to
introduce the IDLE support very soon. I see you have locking fixes for
IDLE too, are they crucial?

Thanks,
Sebastian

Re: [Imaplib2-devel] imaplib2 debugging

[Piers L. <pi...@ja...>]

I have committed a new version of imaplib2 with extra logging, and a
new paremeter to the class 'debug_buf_lvl' (default: 3), that controls
log buffering.

So, to invoke IMAP4 with output to stderr at logging level 3, one would
invoke as:
  IMAP4(..., debug=3, debug_buf_lvl=2)

More lock traces now come out at debug level 3, which involved much less
output than level 5 (which traces actual I/O).

Hope that helps...

Re: [Imaplib2-devel] imaplib2 debugging

[Sebastian S. <Seb...@SS...>]

On Fri, 06 May 2011 13:10:14 +1000, Piers Lauder <pi...@ja...> wrote:
> Maybe it would help to be able to turn on less verbose debugging, so how
> about introducing another debugging parameter that controls the level at
> which logging is redirected from the internal buffer to immediate output
> (currently 4). Then one can turn on P/V logging at, say, level 3, and
> be able to trace it when looking for a (possible) lock conflict.

Yes, implementing a logging level is very much part of my
plans. Nicolas, you also favored that recently, right? It
would be irony if the debugging log is responsible for lock conflicts.

I have been rerunning gmail syncs a few times but couldn't reproduce
hangs so far. I probably need to kill my local store to retransmit a few
hundred messages at once.

Sebastian

Sebastian

Re: [Imaplib2-devel] imaplib2 debugging

[Piers L. <pi...@ja...>]

Maybe it would help to be able to turn on less verbose debugging, so how
about introducing another debugging parameter that controls the level at
which logging is redirected from the internal buffer to immediate output
(currently 4). Then one can turn on P/V logging at, say, level 3, and
be able to trace it when looking for a (possible) lock conflict.

Helpful?

Piers.