POPFile skips messages?

2008-01-31
2013-04-15
  • Paul_A._Rubin
    Paul_A._Rubin
    2008-01-31

    Every once in a while, a message will slip through into my in box with no POPFile classification header, and with no trace of it in POPFile's history.  The message is inevitably spam.  It's not big deal -- they're few and far between -- but I'm just curious how this can happen.  The server setting on my mail client (Thunderbird) is 127.0.0.1:110, so I have to believe that any message that reaches it comes through POPFile.  Has anybody else seen this, and does anyone have an explanation?

    Thanks,
    Paul

     
    • Wm
      Wm
      2008-01-31

      do you perhaps have something else that deals with mail on localhost that you are unaware of?  An example might be a pre-installed virus scanner.
      --
      Wm

       
      • Paul_A._Rubin
        Paul_A._Rubin
        2008-02-01

        Yes.  This happens on three different machines, all with Thunderbird as the mail client.  Two of them run Norton Antivirus; the third runs Avast.  I know Avast has an e-mail scanner, and I'm pretty sure NAV does as well.  I'm also running the ZoneAlarm firewall on all three, and it has some form of e-mail protection.

        I could see one of those blocking an e-mail message.  What I can't figure out is how they could feed it to T-bird while bypassing POPFile.  FWIW, the messages that bypass POPFile never seem to have attachments.  Mostly they're plain text, but there may be an occasional HTML spam-o-gram among them.

        /Paul

         
        • Wm
          Wm
          2008-02-04

          my guess is something like the following:

          PF is doing its hourly housekeeping (cleaning up history, etc.); AIUI PF will not respond to TB during this period; one of the other incoming mailscanners responds thus bypassing PF.

          Apart from disabling the (presumably unused) incoming mail checkers (you actually need to uninstall and re-install AVG w/o mail scanning to turn it off completely for example) others things to look out for a very large number of messages being processed (causing PF to take longer than expected tidying history) PLUS an agressively short delay between TB collecting from PF
          --
          Wm

           
          • Manni
            Manni
            2008-02-04

            "PF is doing its hourly housekeeping (cleaning up history, etc.); AIUI PF will not respond to TB during this period;"

            Should you ever catch POPFile taking so long for it's housekeeping that a mail clients expriences a time-out, please file a bug report ASAP.

            "one of the other incoming mailscanners responds thus bypassing PF."

            Responds to what? The request that was sent to the POPFile port? And the other mailscanner thinks "hmm, nobody's answering that connection attempt, I better have a go even if I have to guess the actual POP3 host?". Hardly.

             
            • Wm
              Wm
              2008-02-05

              From Manni:

              Wm:
              "PF is doing its hourly housekeeping (cleaning up history, etc.); AIUI PF will not respond to TB during this period;"

              Manni:
              Should you ever catch POPFile taking so long for it's housekeeping that a mail clients expriences a time-out, please file a bug report ASAP.

              ===

              It happens often enough here that it is common.  I don't see the need for a bug report.  PF is busy; I leave it alone for a few minutes; it works again.  I also stagger my requests to PF for mail from various servers.

              The PF time-outs don't bother me much, I know it will work eventually.

              My history is 2 days and my timeout 300 seconds; I do not poll my servers very often, my shortest period (through PF) is 17 minutes.  The next longest period is independant of PF as it is private.  Nevertheless TP times out at times.
              --
              Wm

               
          • Paul_A._Rubin
            Paul_A._Rubin
            2008-02-04

            Thanks for the suggestions, but I don't think it's POPFile timing out.  I had a single message slide by POPFile about half an hour ago.  It was the only message in its incoming batch, and TBird is set to check at five minute intervals.  The previous few batches had been 0-2 messages each, so POPFile should not have been too busy, to put it mildly.

            As best I can tell, the Avast mail scanner is sitting between TBird and POPFile.  The message that bypassed POPFile did go through Avast (which adds a header), as did all the messages that POPFile scanned.  The .ini file for Avast does not contain the true URL for the POP server, though (it's set to 127.0.0.1:110), so unless Avast guessed that the SMTP server (which it has) was also the POP server, I don't see how it could have downloaded the mystery message without the help of POPFile.

            Very peculiar.

            /Paul

             
    • Manni
      Manni
      2008-02-04

      Every once in a while, we get reports like this one, but we were never able to figure out what was really going on.

      If you have really set-up your mail client to proxy _all_ accounts through POPFile, than what you are expriencing is clearly a bug in POPFile.

      If you want to help us track down that bug, please do the following:

      Go to POPFile's advanced tab and in the tall table on the right, find the variable logger_level. Set it to a value of 2 and click the Apply button at the bottom of the page. The POPFile log will then be much more verbose.

      As soon as a message shows up in your mail client that apparently was not handled by POPFile, please send us the POPFile log from the time of the arrival of that message and the email or at least it's headers.

      Thanks!
      Manni

       
      • Paul_A._Rubin
        Paul_A._Rubin
        2008-02-04

        I've set the logging level -- unfortunately just after one of these messages slid through.  I'll send you the log file and headers the next time this happens, but Murphy's Law of Monitoring says that now we're watching, it won't happen for a long while (not all bad, I suppose).

        Thanks,
        Paul

         
        • Manni
          Manni
          2008-02-04

          ;-)

          Thank you!
          Manni

           
      • Paul_A._Rubin
        Paul_A._Rubin
        2008-02-15

        Ok, I caught a rogue message today.  It came in somewhere between 11:00 and 11:20 local time.  I'll try sending the message and log to your Sourceforge e-mail address (might be a bit long to post here).

        /Paul

         
        • Manni
          Manni
          2008-02-15

          Hey cool.

          Sending mail to a sourceforge address can be a real pain. If it bounces, you can just send it to <manni AT lxxi.org>.

          Regards,
          Manni

           
        • Manni
          Manni
          2008-02-15

          OK, I got the message and the log, Paul. Thank you.

          The problem was easy to spot because you pointed your finger at it: Your mail client didn't download the message because it was too large. Instead, it used the TOP command to retrieve only part of the message. By default, POPFile will not classifiy messages that were retrieved using the TOP command. However, there is an option that will make POPFile classify messages like that. You can find more information here:
          http://getpopfile.org/docs/howtos:toptoo

          If you want to enabled toptoo, go to POPFile's advanced tab, find the pop3_toptoo variable in the table on the right and set it's value to 1, click Apply and restart POPFile. Depending on the exact behavior of your mail client this can result in messages showing up twice in POPFile's history. I guess it's your decision then whether you prefer an occasional unfiltered message or a cluttered POPFile history.

          Regards,
          Manni

           
          • Paul_A._Rubin
            Paul_A._Rubin
            2008-02-16

            Thanks for the explanation, Manni.  I should have realized sooner that the message truncation was involved.  It turns out that most of the lengthy messages I receive belong in the normal (in box) bucket, so it probably makes more sense for me to live toptoo set to 0 and just deal with an occasional message that slips through.  I'm relieved to know it's not gremlins.  :-)

            Cheers,
            Paul

             
    • Brian Smith
      Brian Smith
      2008-02-05

      Paul asked: >> does anyone have an explanation? <<

      Yes, I think I do :)

      It is a "Windows problem" which I think you can easily solve.

      >> The server setting on my mail client (Thunderbird) is 127.0.0.1:110 <<

      This shows that you are using POPFile's default POP3 port setting (110).

      You've mentioned that the problem exists on three different systems, each of which also has installed an anti-virus package which is scanning email messages. Since port 110 is the standard port used for POP3 email I think these anti-virus packages will also be using port 110.

      I expect you are hoping your email follows a path like this:

      Thunderbird -- POPFile -- anti-virus email scanner -- internet -- mail server

      But your email client, POPFile and the anti-virus package are all using port 110 on your Windows machine (127.0.0.1) to handle POP3 email. Unfortunately Windows does not handle this sort of situation reliably or consistently. For example an _incoming_ message could follow the following path now and again (i.e. whenever Windows decides to be inconsistent):

      Thunderbird -- anti-virus email scanner -- internet -- mail server

      Although Thunderbird send a "retrieve" command to port 110 which POPFile intercepted and passed on to the email server, when the message is received Thunderbird might get it direct from the anti-virus email scanner instead of via POPFile.

      If you are using an anti-virus email scanner it is recommended that you change the POP3 port used by POPFile to communicate with the email client to make sure that your email is processed properly.

      The POPFile wiki recommends setting POPFile's POP3 listen port to 123. This would then ensure that Thunderbird only sends and receives email via POPFile:

      Thunderbird -- port 123 -- POPFile -- port 110 -- anti-virus email scanner -- internet -- mail server

      There is now no way for Thunderbird to receive email directly from the email scanner.

      To make this change you need to change the POPFile-enabled Thunderbird account(s) to use 127.0.0.1 and port 123 and also change POPFile's POP3 listen port to 123 (on the UI's CONFIGURATION page).

      Proxy chaining is explained in some detail here: http://popfile.sourceforge.net/wiki/howtos:proxy_chaining

      On my system I have used this configuration for many years without any problem:

      email client -- port 321 -- POPFile -- port 110 -- anti-virus email scanner -- internet -- mail server

      Brian

       
      • Paul_A._Rubin
        Paul_A._Rubin
        2008-02-06

        Brian,

        Thanks for the info.  I'm always happy to blame Windows for pretty much anything.  :-)

        Actually, I was wondering about the port issue.  I took a look at Avast's help file.  For manual configuration, it suggests using an alternate port number, but there's an automatic configuration feature in recent versions and apparently that does not switch the port.  I figured there must be a way to daisy-chain programs monitoring the same port, the way we used to daisy chain programs sitting on the same IRQ line (and perhaps still do -- been a long time since I had to confront system-level programming).  I couldn't find any details on how Windows handled that, though, and your message suggests that however Windows handles it is subject to some randomness.

        I'm going to leave my home machine set to port 110 and use an alternate port on my office machine.  The problem is intermittent and pretty rare, but if I don't see any more unrated messages at the office over an extended period, that would suggest that this was indeed the problem.

        Thanks again.

        /Paul

         
        • Brian Smith
          Brian Smith
          2008-02-06

          >> I took a look at Avast's help file. For manual configuration, it suggests using an alternate port number, but there's an automatic configuration feature in recent versions and apparently that does not switch the port. <<

          I think you may have misunderstood what I was trying to say. I was _not_ suggesting you change the port used by your anti-virus package's email scanner. If you look at the end of my message you will see that I have left my anti-virus email scanner port in automatic mode, so it uses the default port (110).

          These days most anti-virus packages can scan email "transparently", i.e. they do it without requiring any changes to the email client's settings. They do this by listening on port 110, so the proxy chain is like this:

          email client -- port 110 -- anti-virus email scanner -- internet

          What I was suggesting was that you insert POPFile into this chain _and_ make sure it _always_ appears between the email client and the anti-virus email scanner:

          email client -- port 123 -- POPFile -- port 110 -- anti-virus email scanner -- internet

          To do this you need to change the account settings in the email client and POPFile's POP3 listen from the default setting of 110 to the new setting of 123. In other words, all communication between the email client and POPFile goes via port 123.

          If you do not make this change then the proxy chain will be:

          email client -- port 110 -- POPFile -- port 110 -- anti-virus email scanner -- internet

          and in this case there is no guarantee that outgoing data from the email client will reach POPFile first or that the email client will receive incoming data directly from POPFile.

          Brian

           
        • Paul_A._Rubin
          Paul_A._Rubin
          2008-02-06

          Sorry, didn't mean to imply that I was going to shift NAV (the AV client on my office machine) to a new port.  In fact, I'm not sure I can (our system guys own the policies relating to NAV -- I have pretty much no local control.)  I'm going to shift T-bird and POPFile (client side) to a different port, as you suggested. 

          I was just commenting that Avast's help suggests that it has a way of "sharing" port 110 with a proxy server (which is effectively what POPFile looks like).  Apparently with earlier versions of Avast sitting in a chain with a proxy server, you had to manually configure both the e-mail client and Avast to a different port.  I infer from this that there is some sort of (alleged) port-sharing capability in Windows (some way that multiple programs can listen to the same address:port and have Windows sort out who gets traffic first).

          Thanks,
          Paul

           
          • Brian Smith
            Brian Smith
            2008-02-06

            >> Sorry, didn't mean to imply that I was going to shift NAV (the AV client on my office machine) to a new port. <<

            Thanks for the clarification. I was not sure what you were planning to do.

            >> I'm going to shift T-bird and POPFile (client side) to a different port, as you suggested. <<

            Hope it helps. If it does not then we'll have to try something else :)

            Brian