[eGroupWare-tracker] Bugs #3212: Sometimes json_encode failure in html-view of emails.

[<tr...@eg...>]

Ticket modified by Uwe Redecker at 2013/02/26 10:17 
Tracking System - Bugs 
Category - FelamiMail 
Version - Trunk 
Status - Open 
Resolution - None 
Completed - 0% 
Priority - 5 - medium 
Created by - Uwe Redecker 
Created on - 2013/02/24 11:37 
Assigned to - Klaus Leithoff 
Summary - #3212 - Sometimes json_encode failure in html-view of emails. 
Hi, sometimes viewing of some emails generates an error : In the peview windows and the email-window is only shown : null !!! In the php-error-log is the following message : [24-Feb-2013 09:33:06 Europe/Berlin] PHP Warning: json_encode(): Invalid UTF-8 sequence in argument in G:\Programme\Apache24\htdocs\egroupware\felamimail\inc\class.uidisplay.inc.php on line 1130. The relevant code : " var email_content_$hash = ".json_encode($frameHtml)."; function $funcname(_tar) { if (typeof _tar != \"undefined\" && _tar && typeof _tar.contentWindow != \"undefined\" && _tar.contentWindow != null && typeof _tar.contentWindow.egw_instant_load != \"undefined\") { _tar.setAttribute(\"scrolling\", \"no\"); // Workaround for FF 3.5 _tar.contentWindow.egw_instant_load(email_content_$hash); _tar.setAttribute(\"scrolling\", \"auto\"); } } "; After restarting the web-server everything is OK, the email is displayed normaly .... for a small time ... then the error comes back. The error occurs in every
browser except in normal email programs such as Thunderbird or Outlook ... The system is installed on a Windows-Server 2k3. The Patchlevel of eGroupware is r41830. Apache 2.4.3 (win32) ; PHP 5.4.11. Any hints or suggestion to solve this? Grettings, Uwe Redecker 
Comment by Uwe Redecker at 2013/02/26 10:17: 
Hi Klaus,
 i attached 2 files.
 The eml-file of the "bad" eMail,
 and the PHP-ERROR.LOG if line 1507 uncommented.

 Uwe 
Comment by Klaus Leithoff at 2013/02/26 09:23: 
could you test for me, if the mail processing passes by line 1507
 by uncommenting the error_log statement there.
 depending on your php version json_last_error is available or not (PHP 5 >= 5.3.0)
 If its not comming past that line, the displayCharset is not UTF-8.
 Then you should check what is returned by translation->charset() in line 99.

 Would you mind attaching the eml file of that problemmail?
 seems to be a newsletter. 
Comment by Uwe Redecker at 2013/02/26 08:15: 
Hi Klaus,
 I tested out a few things:

 - If I forward or answer a broken email I can see the content !
 ( but the forwarding text has text coding errors )
 See attached files : forwarding-bad-email.jpg, bad-email-view.jpg.

 Funny ! 

 Greetings
 Uwe Redecker 
Comment by Uwe Redecker at 2013/02/25 20:53: 
Hi Klaus.

 I just tested the hint with "return "";"

 No change! The error must be somewhere else. :( 
Comment by Klaus Leithoff at 2013/02/25 12:06: 
does not seem to hold non compatible UTF-8 chars.
 Try returning an empty string, instead of the additionalStyle stuff.
 return "";
 just after the error log.
 If the problem persists, we must look somewhere else. 
Comment by Uwe Redecker at 2013/02/25 11:52: 
Hi Klaus,
 I enabled the logging in line 1061, the result is attached as file "20130225 php-errors.log"

 I'm not sure if dieses helps.

 A little perplexed,
 Uwe Redecker 
Comment by Klaus Leithoff at 2013/02/25 09:25: 
Hi Uwe,
 getdisplayableBody already tests this (non UTF8 Chars in Body).
 Line 1503 to 1516

 There is only one way to traverse this: by adding the styles that
 may be included by the eMail.
 But: if you restart your WebServer, and everything works, for that particular eMail,
 the problem seems to be connected not only to the content of the Mail, but some other factors too.
 There is a comment for error_logging in Line 1061. Please uncomment that, so you may see what is the possible input causing the problem.
 Best regards
 Klausi 
Comment by Ralf Becker at 2013/02/24 12:11: 
Hi Uwe,
 problem is probably mails which claim to be utf-8, but are not, at least not entirely.
 PHP JSON encode will not return anything, if there's a single non utf-8 char in it!
 To fix that situation for mail (dont think we need or want it in general), fmail could check if encoding failed (json_last_error() == JSON_ERROR_UTF8) and then clean, mail content with regular expression described in 

 [magp.ie/2011/01/06/remove-non-utf8-characters-from-string-with-php/ -> http://magp.ie/2011/01/06/remove-non-utf8-characters-from-string-with-php/]

 Maybe for external mail content, we could also think about doing that anyway, as mailformed utf-8 chars have some exploit potential.

 Ralf 
 
Linked entries:
https://community.egroupware.org/egroupware/index.php?menuaction=tracker.tracker_ui.edit&tr_id=3212&no_popup=1