popfile does not scan IMAP-folders/mails

Extensions
alpal
2005-05-21
2013-04-15
  • alpal

    alpal - 2005-05-21

    Hi there,

    intrigued by the fact that one can now use popfile with IMAP, I installed 0.22.2 and imap-addon. Activated it under Advanced-tab, entered the server-IP, username etc. and got - no connection. It cost me a few minutes before I figured out the problem: Initially I entered the info for the user whose email I plan to fetch, i.e. whose account I want to access. That didn't work, although with that info I can access my mailbox via my email-client just fine (of course). To get a connection with popfile I had to enter the info of my imap-admin (I run cyrus under my own linux-server in a local network). That I don't understand, I must admit.

    But now my real problem:
    I have a connection and can see my folders. I chose my inbox as watch-folder (or several), chose my folders for my buckets, apply and access my account. Mails are fetched alright but no mail, not a single one, is scanned by popfile (nothing shows up in the history; it does alright with POP3-Accounts). I have my mail-client set up as I had when I used popfile with POP3: "mail-server" is 127.0.0.1, username is "mail-server:username".Everything else is non-fancy. The strange thing is that popfile is apparently working otherwise with this configuration I wouldn't be able to fetch new mails. Yet they don't show up in the history, no header is altered (that the module cannot do yet, though, IIRC) and they are not moved to any folder at all (unless my mail-client filters them). Any idea what I'm doing wrong?

     
    • alpal

      alpal - 2005-05-21

      What I forgot: also, when Imap is activated in popfile, I cannot use it with POP3 (a second account in my mail-client). I have to deactivate before usage. That shouldn't be, and I think there was no solution in a thread about that same problem. Anyone experiencing that?

      Greets,
      alpal

       
    • Manni

      Manni - 2005-05-21

      Hi alpal,

      I don't understand either why you have to logon as admin to the server. I guess this one is server-specific and I can't help you with that. The question is whether this is related to the problem you are having with POPFile's IMAP module or not. But I'm sure that we'll be able to find out.

      Let's start with your second post: What exactly do you mean when you say that you cannot use POP3 and IMAP? What exactly are the symptons? Any error messages? From the mail client?

      Back to your first post: You seem to be able to make the IMAP module log on to your server and retrieve the list of folders just fine. Now we need to find out why the module isn't seeing any of your new mail.

      Please go to the POPFile UI and flip to the Advanced tab. Find the variable 'logger_level' and set its value to 2. Click the 'Update' button at the bottom of the screen.

      Now shut down POPFile and delete any old POPFile log files (popfile*.log).

      Use your client to move mail into a watched folder on your IMAP server (or make sure that there is new mail, that arrived after you changed the logging level). Restart POPFile and wait a little (depending on the update interval you have set for the IMAP module) and then go to POPFile's Configuration tab and download the current log file.

      Post the log.

      Manni

       
    • James E Lang

      James E Lang - 2005-05-21

      "intrigued by the fact that one can now use popfile with IMAP, I installed 0.22.2 and imap-addon."

      I am running this as well.

      "Activated it under Advanced-tab, entered the server-IP, username etc. and got - no connection. It cost me a few minutes before I figured out the problem: Initially I entered the info for the user whose email I plan to fetch, i.e. whose account I want to access. That didn't work, although with that info I can access my mailbox via my email-client just fine (of course). To get a connection with popfile I had to enter the info of my imap-admin (I run cyrus under my own linux-server in a local network). That I don't understand, I must admit."

      I also run cyrus on my stand alone system. Here is how I have POPFile configured for IMAP:

      On the Configuration page:
      IMAP server host name: localhost
      IMAP server port: 143
      IMAP account login: my user name in cyrus
      Password for IMAP account: my password in cyrus

      Once you have that you need to apply those parameters.

      The remainder depends on your configuration of folders within cyrus. You will most certainly have an INBOX folder. That is the folder that you will most likely want to have POPFile watch. Under special circumstances you may want to add one or more additional watched folders but that is purely optional. You should have a folder defined within cyrus for each bucket that you have defined. In theory you can get by with no specific folder for ONE of the buckets leaving the messages in INBOX but I do not recommend that. Once you have the folders created in cyrus, then associate them with your buckets.

      I recommend that you check the box "Expunge deleted messages from watched folders" so that the messages that POPFile moves to folders associated with the buckets don't remain in INBOX (for example) after that. I also recommend a relatively short "update interval." I use 25 seconds for my update interval.

      "But now my real problem:
      I have a connection and can see my folders. I chose my inbox as watch-folder (or several), chose my folders for my buckets, apply and access my account. Mails are fetched alright but no mail, not a single one, is scanned by popfile (nothing shows up in the history; it does alright with POP3-Accounts). I have my mail-client set up as I had when I used popfile with POP3: "mail-server" is 127.0.0.1, username is "mail-server:username".

      The IMAP service is not a proxy. With POP3 mail passes through POPFile on its way from your cyrus server to your mail client. With IMAP your mail client accesses your cyrus server directly as does POPFile working in parallel with it.

      "Everything else is non-fancy. The strange thing is that popfile is apparently working otherwise with this configuration I wouldn't be able to fetch new mails. Yet they don't show up in the history, no header is altered"

      The IMAP service does not alter headers. It moves messages to associated folders instead.

      "(that the module cannot do yet, though, IIRC) and they are not moved to any folder at all (unless my mail-client filters them). Any idea what I'm doing wrong?"

      I hope my comments help you. If not, maybe you can restate what is happening and I'll give it another try.

      --
      Jim

       
    • alpal

      alpal - 2005-05-21

      Hi Manni,

      Too bad in this forum one cannot really quote things properly...

      > Let's start with your second post: What exactly do you mean when you say that you cannot use POP3 and IMAP? What exactly are the symptons? Any error messages? From the mail client?

      Apparently, popfile crashes?! I just found out that it was not running even though I had started it. After having started it again, I could use POP3 with my mail-client (POP3-account), i.e. fetch mails alright and scans them etc. even with imap set to 1. So it looks like my second post is solved. Will keep an eye on whether POPfile crashes again.

      The log is a partial and altered to not show personal data. It does make it quite obvious that there is a problem, though.

      "2005/5/21 13:56:36 1752: html: 182: HTTP handle_url returned code 1[0a]
      2005/5/21 13:56:36 1752: html: 190: Close HTTP connection on IO::Socket::INET=GLOB(0x148d194)[0a]
      2005/5/21 13:56:36 1752: imap: 416: Trying to connect to [IP] for folder user.xxx.Known.
      2005/5/21 13:56:36 1752: imap: 889: Connecting to [IP]:143
      2005/5/21 13:56:36 1752: imap: 936: Connected to [IP]:143 timeout 60
      2005/5/21 13:56:36 1752: imap: 659: Read slurp data * OK servername Cyrus IMAP4 v1.5.19 server ready[0d][0a]
      2005/5/21 13:56:36 1752: imap: 940: >> * OK servername Cyrus IMAP4 v1.5.19 server ready[0d][0a]
      2005/5/21 13:56:36 1752: imap: 975: Logging in
      2005/5/21 13:56:36 1752: imap: 1051: << A00000 LOGIN "xxxxx" "xxxxx"[0d][0a]
      2005/5/21 13:56:36 1752: imap: 659: Read slurp data A00000 OK User XXXXXX
      2005/5/21 13:56:36 1752: imap: 1195: >> A00000 OK User XXXXXX
      2005/5/21 13:56:36 1752: imap: 1051: << A00001 SELECT "user.xxx.Known"[0d][0a]
      2005/5/21 13:56:36 1752: imap: 659: Read slurp data A00001 NO Permission denied[0d][0a]
      2005/5/21 13:56:36 1752: imap: 1191: >> A00001 NO Permission denied[0d][0a]
      2005/5/21 13:56:36 1752: imap: 433: Could not SELECT folder user.xxx.Known.
      2005/5/21 13:56:36 1752: imap: 1051: << A00002 LOGOUT[0d][0a]
      2005/5/21 13:56:36 1752: imap: 659: Read slurp data * BYE LOGOUT received[0d][0a]A00002 OK Completed[0d][0a]
      2005/5/21 13:56:36 1752: imap: 1207: >> * BYE LOGOUT received[0d][0a]
      2005/5/21 13:56:36 1752: imap: 1195: >> A00002 OK Completed[0d][0a]
      2005/5/21 13:56:36 1752: imap: 416: Trying to connect to [IP] for folder user.xxx.SPAM.
      2005/5/21 13:56:36 1752: imap: 889: Connecting to [IP]:143
      2005/5/21 13:56:36 1752: imap: 936: Connected to [IP]:143 timeout 60
      2005/5/21 13:56:36 1752: imap: 659: Read slurp data * OK servername Cyrus IMAP4 v1.5.19 server ready[0d][0a]
      2005/5/21 13:56:36 1752: imap: 940: >> * OK servername Cyrus IMAP4 v1.5.19 server ready[0d][0a]
      2005/5/21 13:56:36 1752: imap: 975: Logging in
      2005/5/21 13:56:36 1752: imap: 1051: << A00000 LOGIN "xxxxx" "xxxxx"[0d][0a]
      2005/5/21 13:56:36 1752: imap: 659: Read slurp data A00000 OK User XXXXXX
      2005/5/21 13:56:36 1752: imap: 1195: >> A00000 OK User XXXXXX
      2005/5/21 13:56:36 1752: imap: 1051: << A00001 SELECT "user.xxx.SPAM"[0d][0a]
      2005/5/21 13:56:36 1752: imap: 659: Read slurp data A00001 NO Permission denied[0d][0a]
      2005/5/21 13:56:36 1752: imap: 1191: >> A00001 NO Permission denied[0d][0a]
      2005/5/21 13:56:36 1752: imap: 433: Could not SELECT folder user.xxx.SPAM.
      2005/5/21 13:56:36 1752: imap: 1051: << A00002 LOGOUT[0d][0a]
      2005/5/21 13:56:36 1752: imap: 659: Read slurp data * BYE LOGOUT received[0d][0a]A00002 OK Completed[0d][0a]
      2005/5/21 13:56:36 1752: imap: 1207: >> * BYE LOGOUT received[0d][0a]
      2005/5/21 13:56:36 1752: imap: 1195: >> A00002 OK Completed[0d][0a]
      2005/5/21 13:56:36 1752: imap: 416: Trying to connect to [IP] for folder user.xxx.
      2005/5/21 13:56:36 1752: imap: 889: Connecting to [IP]:143
      2005/5/21 13:56:36 1752: imap: 936: Connected to [IP]:143 timeout 60
      2005/5/21 13:56:36 1752: imap: 659: Read slurp data * OK servername Cyrus IMAP4 v1.5.19 server ready[0d][0a]
      2005/5/21 13:56:36 1752: imap: 940: >> * OK servername Cyrus IMAP4 v1.5.19 server ready[0d][0a]
      2005/5/21 13:56:36 1752: imap: 975: Logging in
      2005/5/21 13:56:36 1752: imap: 1051: << A00000 LOGIN "xxxxx" "xxxxx"[0d][0a]
      2005/5/21 13:56:36 1752: imap: 659: Read slurp data A00000 OK User XXXXXX
      2005/5/21 13:56:36 1752: imap: 1195: >> A00000 OK User XXXXXX
      2005/5/21 13:56:36 1752: imap: 1051: << A00001 SELECT "user.xxx"[0d][0a]
      2005/5/21 13:56:36 1752: imap: 659: Read slurp data A00001 NO Permission denied[0d][0a]
      2005/5/21 13:56:36 1752: imap: 1191: >> A00001 NO Permission denied[0d][0a]
      2005/5/21 13:56:36 1752: imap: 433: Could not SELECT folder user.xxx.
      2005/5/21 13:56:36 1752: imap: 1051: << A00002 LOGOUT[0d][0a]
      2005/5/21 13:56:36 1752: imap: 659: Read slurp data * BYE LOGOUT received[0d][0a]A00002 OK Completed[0d][0a]
      2005/5/21 13:56:36 1752: imap: 1207: >> * BYE LOGOUT received[0d][0a]
      2005/5/21 13:56:36 1752: imap: 1195: >> A00002 OK Completed[0d][0a]"

      Mabye just a problem with my server, although the IMAP itself works just fine.

      Greets,
      alpal

       
      • David Lang

        David Lang - 2005-05-21

        Ok, this is telling you that the user you are logging in to cyrus as does not have permissions on the folder

        the message telling you this is
        2005/5/21 13:56:36 1752: imap: 659: Read slurp data A00001 NO Permission denied[0d][0a]

        this is probably related to you useing the cyrus admin user to login.

        you should user the same userid that you use from your mail client. the main folder you will use will be INBOX (which maps to the user.username folder you see with the cyrus admin user)

        you will then want to create folders under your INBOX for your various buckets. at that point things should just work.

        the other possibility is to grant permission for the userid that you put in popfile to list, read, post, and delete messages in whatever folders you want (and at this point the folders don't need to be below your INBOX)

        useing the cyradmin tool you will need to set acl's on the various folders, you will need to grant the userid at least l r p d permissions, possibly some others I'm not thinking of at the moment.

        once you have done this things should work, and this has the advantage that you can use a different userid for popfile then for your normal access (if you care to keep the two independant for whatever reason)

         
    • alpal

      alpal - 2005-05-22

      Hi,

      thanks for your answer! I still couldn't connect to the mailbox via popfile using my mailbox-account. So what I did is I gave the cyrus-admin full rights - that did it. Mails are processed through popfile now, show up in the log and history. However, they are not sorted. :-/ The SPAM-mails all stay in my inbox, although I told Popfile to move anything that is classified as spam to the folder "user.xxx.SPAM". Ham goes to user.xxx, i.e. Inbox from where it would be filtered by my mail program in to subfolders. My Mail-client now also cannot filter the spam mails (as I did with POP3), since popfile doesn't set any x-classifications etc. anymore when using IMAP. So something is still not working. Now it really seems to be a popfile problem. Update interval is at 20 (btw, when you set 10, it brings an error message that the number needs to be between 10 and ... and sets it back to 20 - bug?). Any ideas?

      Greets,
      alpal

       
      • David Lang

        David Lang - 2005-05-22

        Ok, a few things here

        1. unable to connect as user
          to fix this we would need to see the logs from Cyrus. I wouldn't be surprised if it required an authentication method we don't support (newer versions do by default, we only support plain text passwords)

        2. the client not being able to sort the messages
          when you use IMAP the client doesn't sort the messages, the messages get sorted by putting them directly into different folders. the user popfile accesses things as will need full permissions on the spam and ham folders as well.

        3. popfile isn't sorting anything
          popfile need some pointers to get started. so either through the history or by just moving some recent messages into your spam/ham folders you need to tell popfile what you consider belonging in each catagory.

        once you have done at lease one message into each popfile will start working

        4. when you make an error report about things not working we need enough details to see what's really going on.
          in this case it means that you will need to post a trace showing what happens when things don't work. the last trace you posted showed us that you were having permission problems, but becouse of those problems nothing else would work and so after fixing those we will need additional traces.

        David Lang

         
    • alpal

      alpal - 2005-05-22

      @Jim: after having begun to understood how this whole setup works, I also realized that I had to put my server in there directly and not have popfile for imap work as a proxy. Now I can connect ok. Thanks though!

      @David: I don't consider myself to be stupid but it never occured to me that the user popfile needs access to the mailbox too! D'oh, makes perfect sense, of course, yet was nowhere documented. I think I should maybe write a howto for numbnuts like me. :-)

      Several issues: maybe my cyrus does not like plaintext-pw as long as I don't put a setting there allowing them in my /etc/imapd.conf. I gave that a try, even before I read your post. I still do not understand why it did connect before and inbetween, so this might actually not be a required setting with Cyrus 1.5.xx. I will try to disable it again to find that out.

      Giving the user popfile lrsi access to my mailbox made it look in there and indeed sort (correct permissions?). It doesn't work properly yet, but maybe it in fact still needs training: Although it does recognize the spam-mails correctly as spam, it simply doesn't always move them to the spam-folder, for whichever reason (even though that one is a) on my watched-folder-list and b) the place where the spam-bucket mails are supposed to go to).

      You mention the history: is it better to have a longer history for popfile-imap? I usually clear that list a couple times a day (as I skim through for misclassifications) - not recommended anymore?
      I now moved all my spam-mails from my inbox manually to my spam-folder to give popfile something to munch on.

      I will also try to figure out what the connection problem with my actual mail-user is.

      I'm getting there, with everybody's excellent help here! Thanks.

      Greets,
      alpal

       
      • David Lang

        David Lang - 2005-05-22

        to be able  to move a message to a new folder the user popfile runs as will need post privilages

        to be able to delete the message from the origional folder the user popfile runs as will need delete privilages

        as for the authentication problem, I thought that the default disallowing of plaintext passwords in Cyrus took place in the 2.x series and that you should be Ok, in the 1.5 series

        try logging in manually

        telnet to your mail server on port 143 and type
        a login userid password
        b capability
        c select inbox
        d logout

        and see what happens

        you should get something like

        root@mobile:~/adastra# telnet mail.invendra.net 143
        Trying 66.139.76.16...
        Connected to mail.invendra.net.
        Escape character is '^]'.
        * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION STARTTLS] Courier-IMAP ready. Copyright 1998-2004 Double Precision, Inc.  See COPYING for distribution information.
        a login userid password
        a OK LOGIN Ok.
        a capability
        * CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION
        a OK CAPABILITY completed
        a select inbox
        * FLAGS (\Draft \Answered \Flagged \Deleted \Seen \Recent)
        * OK [PERMANENTFLAGS (\* \Draft \Answered \Flagged \Deleted \Seen)] Limited
        * 915 EXISTS
        * 0 RECENT
        * OK [UIDVALIDITY 1104579119] Ok
        * OK [MYRIGHTS "acdilrsw"] ACL
        a OK [READ-WRITE] Ok
        a logout
        * BYE Courier-IMAP server shutting down
        a OK LOGOUT completed
        Connection closed by foreign host.

        if you get error messages during this process it will identify what isn't working

         
    • alpal

      alpal - 2005-05-23

      Hi,

      "telnet to your mail server on port 143"

      This is what I get:

      a OK User logged in
      b capability
      * CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS X-NON-HIERARCHICAL-RENAME NO_ATOMIC_RENAME UNSELECT X-NETSCAPE
      b OK Completed
      c select inbox
      * FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
      * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)]
      * 102 EXISTS
      * 0 RECENT
      * OK [UNSEEN 17]
      * OK [UIDVALIDITY 1116424063]
      c OK [READ-WRITE] Completed
      d logout
      * BYE LOGOUT received
      d OK Completed
      Connection closed by foreign host.

      No errors, as far as I can tell. I still cannot log in via popfile ("Could not login. Verify your login name and password, please." OR sometimes: "Failed to connect to server. Please check the host name and port and make sure you are online." - of course I'm online and the IP and ports never change) with that very username I had used above to successfully log into cyrus via telnet. That is strange.

      Just to clarify something: I run popfile on a windows XP and cyrus on a different linux-machine. The user "running" popfile, i.e. the one that is named in the field "IMAP_account_login:" under the configuration-tab is the one that needs to have the appripriate rights to the mailbox, correct? Those are set to "all" in my case. Or is there another user I need to set up under linux? I do have the user popfile whose cyrus-permissions for the (only) mailbox I now have set to lriwsp. Although that user shouldn't have anything to do with my popfile on my windows-machine?!

      Again, using the cyrus-admin account-info, I can login via popfile just fine.

      Greets,
      alpal

       

Log in to post a comment.