Links no longer working - Unhandled exception in callback

I couldn't use the action links either for the last day or so. I restarted CheckGmail from its own menu to see if it would fix it, but it didn't. I went to Gmail and marked a message as "Unread" to have something to act on, and then I started CheckGmail directly from a terminal to try to get the error output of the same form as you got.

The problem now was that the "Mark as read" link in CheckGmail just worked as normal and no error message was therefore printed... I'll see if I can reproduce the error again, though. In best case it was just a temporary service outage at Google.

Ah, "Mark as read" works, but e.g. clicking on a message header to expand the text does not. If I do that, the program produces the error text mentioned, hangs and won't take e.g. the "Mark as read" action afterwards.

The problem is in the XMLin() call on the given line. Normally, the Atom feed fetched by CheckGmail that resides in the $atom_txt variable, is well formed XML and XMLin() has no problems with it. For some reason, when one clicks on the headline of a message, though, the contents of $atom_txt gets mangled to a very weird string that looks like:

)]}'

[[["v","a5aQBlbcIQ2.en.","8","15ab5f6c884d5e66"]
,["di",255,,,,,[]
,[]
,,,[]
,[]
,[]
]

and continues like this for a while (and contains bits and pieces of relevant email data). I don't recognize the structure of this (see the error message when clicking on an email for a full example of the contents of $atom_txt).

Why does XML::Simple complain about that it can't find the filename? Because it tries to be very "clever" <http://search.cpan.org/~grantm/XML-Simple-2.20/lib/XML/Simple.pm#XMLin()>, and the first argument can _either_ be a string (as it is supposed to be used here) _or_ a filename. If the string passed contains the '<>' signs, it assumes it is XML. If not, it assumes it is a filename. Since the contents get mangled, XMLin() assumes we want to stat a filename that is completely bananas, and therefore produces the weird error message.

So, the question is why $atom_txt gets weird when we click on a header. And why it started getting weird just now. It probably has something to do with the way Google serves full messages. When one clicks on the header, the short summary XML should be replaced by the full contents. This replacement apparently currently completely garbles the XML structure to the form given in the error message.

From what I can find, it is the `if ($atom =~ m/while\(1\);/) ` line around 1076 (I have done some other modifications, so it might differ some lines from your version) that is a part of the problem.

This is supposed to catch the $atom variable and preprocess it if a fulltext request has been made. The $atom variable does not contain valid XML here, but some strange datapack format that within the if construct is supposed to be massaged into the $messages_ext variable and then return, leaving the poll loop completely.

The first problem is that the datapack format seem to have changed, so `m/while\(1\);/` does not match. The second problem is that the subsequent regexpes within that if clause does not pick out the correct parts of the message, since the datapack format has changed.

Naively, the first step would be to match the if clause on e.g. `m/LABEL=.*\n\)]}/` which seems to work, and then change the regexp that is run on $atom a couple of rows later to correspond to the new datapack format. Perhaps it is the same as the old one that resides in the commented code in this part. I don't have time to test now.

while(1); was added by Google at the beginning of each json response to protect against cross-site request forgery:
http://stackoverflow.com/questions/2669690/why-does-google-prepend-while1-to-their-json-responses

It seems that google changed their format so that while(1); is no longer prepended to the atom.

I replaced the regex:
if ($atom =~ m/while\(1\);/) {
with a check on "v" field in json:
if ($atom =~ m/\[\[\"v\",/) {
and it seems to work, although I'm not sure there are no side effects. Hopefully somebody would be able to come with a better idea (should the "if" be completely removed?)

There are still things not working: attachment part, mark all as read, mark as read (when the unread email is not the newest), but if i recall correctly these problems were there before.

Yes, Google has changed their JSON return format.

No, the "if" clause can't be removed, since it is the mechanism that checks if the atom response is of the form that indicates that we have received a full text message the way the program is written. We just need to get it to match the correct form of the fulltext atom message.

These two changes seem robust for me:

if ($atom =~ m/while\(1\);/) {
was changed to
if ($atom =~ s/LABEL=.*\n\)]}'//) {

while ($atom =~ m/\["ms",.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,\[.*?"\].*?\[.*?\[.*?,.*?,.*?,.*?,.*?,"(.*?)",\[/sg) {
was changed to:
while ($atom =~ m/\["ms",.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,\[.*?"\].*?\[.*?\[.*?,.*?,.*?,.*?,.*?,".*?","(.*?)",\[/sg) {
There is a minimum difference in the end of the regexp, if one looks closely.

One could do it more robust with real JSON parsing, but my initial attempt with the standard Perl module returned errors on malformed JSON (though it seemed fine in other parsers), and it would add more dependencies to the checkgmail script, so I'll let regexp handle the parsing (even though I don't like it in principle :-) ).

Here is a diff of the fix in my last post: http://pastebin.com/RaJmQA3e

Note that the line numbers might have some minor offsets from other people's versions, since there have been a couple of hot fixes by this point.

A minor comment on justagirl20's last comment: I have no problem marking messages as read even if it is not the newest one, and I don't believe I've ever had it.

I noted that the regexp to pick out the full text worked on some messages but not others (which has always been the case for me with checkgmail, for that matter). At least it does not crash on seemingly any messages.

Perhaps a "real" JSON solution would be nice to actually use the response as it is supposed to be done. I will look at it some other time if no one else does it first. For now it seems like checkgmail is back at its old functionality for me.

This is how I do the if{} clause now: http://pastebin.com/gjrA4cCS

The old part is wholly commented out. The replacement starts from the comment "### JSON parsing instead of above regexps."

It works for me, and it gets attachments working again.

The JSON rewriting looks good (I always hated the regexp hack), and I like getting back the attachments; unfortunately some of the messages show up just empty when clicked on (which did not happen when using regexps.
"mark as read" and "mark all as read" for multiple unread messages malfunctions when conversational style is off, which is my case.
I think we should fork this project, something along the lines of "checkgmail-ng", since we can't keep on patching it without having an official version

There is also a parsing error when message body fails to display:

Unable to parse markup: Error on line 36 char 8: Element 'span' was closed, but the currently open element is 'img'

If $mb is picked up using the regexp, it works fine.

I have uploaded on github gist a modified checkgmail version with two factor support (config, UI, integration), login refactoring (requests, UI), refactored kwallet integration, + additional refactoring/bugfixing.

It includes a similar fix for this.

Instructions on getting this modified version are available here :
https://gist.github.com/romainguinot/5065977.