#299 too many carriage returns in ics files

[Campbell Morrison]

[Transferred from the mailing list]

Hi

There may be a bug in MRBS' way of how timezone data is stored in mrbs_zoneinfo.vtimezone.
When I run a mysql mrbs -e 'select * from mrbs_zoneinfo' > output.txt I receive a one-liner that looks like this in vi:

id timezone outlook_compatible vtimezone last_updated
1 Europe/Zurich 1 BEGIN:VTIMEZONE^M^M\nTZID:Europe/Zurich^M^M\nTZURL:http://tzurl.org/zoneinfo-outlook/Europe/Zurich^M^M\nX-LIC-LOCATION:Europe/Zurich^M^M\nBEGIN:DAYLIGHT^M^M\nTZOFFSETFROM:+0100^M^M\nTZOFFSETTO:+0200^M^M\nTZNAME:CEST^M^M\nDTSTART:19700329T020000^M^M\nRRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=-1SU^M^M\nEND:DAYLIGHT^M^M\nBEGIN:STANDARD^M^M\nTZOFFSETFROM:+0200^M^M\nTZOFFSETTO:+0100^M^M\nTZNAME:CET^M^M\nDTSTART:19701025T030000^M^M\nRRULE:FREQ=YEARLY;BYMONTH=10;BYDAY=-1SU^M^M\nEND:STANDARD^M^M\nEND:VTIMEZONE 1396361075
2 Europe/Zurich 0 BEGIN:VTIMEZONE^M^M\nTZID:Europe/Zurich^M^M\nTZURL:http://tzurl.org/zoneinfo/Europe/Zurich^M^M\nX-LIC-LOCATION:Europe/Zurich^M^M\nBEGIN:DAYLIGHT^M^M\nTZOFFSETFROM:+0100^M^M\nTZOFFSETTO:+0200^M^M\nTZNAME:CEST^M^M\nDTSTART:19810329T020000^M^M\nRRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=-1SU^M^M\nEND:DAYLIGHT^M^M\nBEGIN:STANDARD^M^M\nTZOFFSETFROM:+0200^M^M\nTZOFFSETTO:+0100^M^M\nTZNAME:CET^M^M\nDTSTART:19961027T030000^M^M\nRRULE:FREQ=YEARLY;BYMONTH=10;BYDAY=-1SU^M^M\nEND:STANDARD^M^M\nBEGIN:STANDARD^M^M\nTZOFFSETFROM:+003408^M^M\nTZOFFSETTO:+002944^M^M\nTZNAME:BMT^M^M\nDTSTART:18480912T000000^M^M\nRDATE:18480912T000000^M^M\nEND:STANDARD^M^M\nBEGIN:STANDARD^M^M\nTZOFFSETFROM:+002944^M^M\nTZOFFSETTO:+0100^M^M\nTZNAME:CET^M^M\nDTSTART:18940601T000000^M^M\nRDATE:18940601T000000^M^M\nEND:STANDARD^M^M\nBEGIN:DAYLIGHT^M^M\nTZOFFSETFROM:+0100^M^M\nTZOFFSETTO:+0200^M^M\nTZNAME:CEST^M^M\nDTSTART:19410505T010000^M^M\nRDATE:19410505T010000^M^M\nRDATE:19420504T010000^M^M\nEND:DAYLIGHT^M^M\nBEGIN:STANDARD^M^M\nTZOFFSETFROM:+0200^M^M\nTZOFFSETTO:+0100^M^M\nTZNAME:CET^M^M\nDTSTART:19411006T020000^M^M\nRDATE:19411006T020000^M^M\nRDATE:19421005T020000^M^M\nRDATE:19810927T030000^M^M\nRDATE:19820926T030000^M^M\nRDATE:19830925T030000^M^M\nRDATE:19840930T030000^M^M\nRDATE:19850929T030000^M^M\nRDATE:19860928T030000^M^M\nRDATE:19870927T030000^M^M\nRDATE:19880925T030000^M^M\nRDATE:19890924T030000^M^M\nRDATE:19900930T030000^M^M\nRDATE:19910929T030000^M^M\nRDATE:19920927T030000^M^M\nRDATE:19930926T030000^M^M\nRDATE:19940925T030000^M^M\nRDATE:19950924T030000^M^M\nEND:STANDARD^M^M\nBEGIN:STANDARD^M^M\nTZOFFSETFROM:+0100^M^M\nTZOFFSETTO:+0100^M^M\nTZNAME:CET^M^M\nDTSTART:19810101T000000^M^M\nRDATE:19810101T000000^M^M\nEND:STANDARD^M^M\nEND:VTIMEZONE 1397650421

Please note the double ctrl-M. According to this the ics file looks like this. When this is sent to an IBM Domino server the mail automatically receives status 'deleted' (this in turn may be a Domino bug).

BEGIN:VCALENDAR
PRODID:-//MRBS//NONSGML MRBS 1.4.10 //EN
VERSION:2.0
CALSCALE:GREGORIAN
METHOD:REQUEST
BEGIN:VTIMEZONE

TZID:Europe/Zurich

TZURL:http://tzurl.org/zoneinfo-outlook/Europe/Zurich

X-LIC-LOCATION:Europe/Zurich

BEGIN:DAYLIGHT

TZOFFSETFROM:+0100

TZOFFSETTO:+0200

TZNAME:CEST

DTSTART:19700329T020000

RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=-1SU

END:DAYLIGHT

BEGIN:STANDARD

TZOFFSETFROM:+0200

TZOFFSETTO:+0100

TZNAME:CET

DTSTART:19701025T030000

RRULE:FREQ=YEARLY;BYMONTH=10;BYDAY=-1SU

END:STANDARD

END:VTIMEZONE
BEGIN:VEVENT
UID:MRBS-5357582d39564-fb15b8fe@booking.business.uzh.ch
DTSTAMP:20140423T065523Z
LAST-MODIFIED:20140423T060533Z
DTSTART;TZID=Europe/Zurich:20140430T070000
DTEND;TZID=Europe/Zurich:20140430T080000
SUMMARY:Angewandte Relativitätstheorie
DESCRIPTION:Beschreibung: Das ist ein Test\nBestätigungsstatus: vorläufig
\nGenehmigungsstatus: Genehmigung ausstehend\nRaum: PLM - 103/104\nAnfangs
datum: 07:00:00 - Mittwoch 30 April 2014\nDauer: 1 Stunden\nEnddatum: 08:0
0:00 - Mittwoch 30 April 2014\nKategorie: Intern\nErstellt von: hmust\nLet
zte Änderung: 08:05:33 - Mittwoch 23 April 2014\nLehrstuhl: Einstein\nTel
efon: 45913\nKontakt Name: Alfred Einstein\nSerientyp: Keine\n
X-ALT-DESC;FMTTYPE=text/html:

\n\n\n

Beschreibung:	Das ist ein Test
Bestätigungsstatus:	vorläufig
Genehmigungsstatus:	Genehmigu ng ausstehend
Raum:	PLM - 103/104
Anfangsdatum:	07:00:00 - Mittwoch 30 April 2014< /td>\n
Dauer:	1 Stunden
E nddatum:	08:00:00 - Mittwoch 30 April 2014
Intern
Erstellt von:
Letzte Änderung:	08:05:33 - Mit twoch 23 April 2014
Lehrstuhl:	Einstein\n
Telefon:	45913
Kont akt Name:	Alfred Einstein
Serientyp:	Keine

LOCATION:PLM - 103/104 SEQUENCE:0 STATUS:TENTATIVE ORGANIZER;CN="hmust":mailto:my@email.adress.ch END:VEVENT END:VCALENDAR I believe RFC 2445 states that lines are separated by one CRLF and not by a double CR. Many thanks for looking into this.

[Campbell Morrison]

Strange. Yes, you're right, the RFC requires one CRLF, but as far as I can see MRBS is just using one, correctly. I've run the attached test program on my system and get the output I expect, ie

66 (B)
69 (E)
71 (G)
73 (I)
78 (N)
58 (:)
86 (V)
84 (T)
73 (I)
77 (M)
69 (E)
90 (Z)
79 (O)
78 (N)
69 (E)
13
10
84 (T)
90 (Z)
73 (I)
etc.

Can you run it on your system please (you'll need to put it in the MRBS directory as it uses some MRBS include files) and let me know what the output is, ie whether you're getting a double CR? If there is a double CR then we'll have to work out how they got there. If not, then how they're being inserted later.

Campbell

[Anonymous]

Hi Campell

thanks for your quick response. Please find the output of test.php attached.

Thanks,
Philipp

[Campbell Morrison]

Thanks. So, yes, the double CR is already in the database. Two things: (1) what versions of MRBS, PHP and MySQL are you using (answers on the Help page)? and (2) can you empty the vtimezone table and then export an entry from MRBS, which will force a new entry to be inserted in the table, and then re-run the test and let me know the output.

Campbell

[Anonymous]

System Info:
Raumbuchungssystem: MRBS 1.4.10
Datenbank: MySQL 5.5.35-0ubuntu0.12.04.2-log
System: Linux ibwcloud 3.2.0-60-generic #91-Ubuntu SMP Wed Feb 19 03:54:44 UTC 2014 x86_64
Serverzeit: Mit 23 Apr 2014 13:03:12 CEST
PHP: 5.3.10-1ubuntu3.11

Please find attached the new output of test.php.

Philipp

[Campbell Morrison]

Here's a new test program. This time it checks the contents of the timezone definition in the filesystem. Could you let me know the output please?

Thanks,

Campbell

[Anonymous]

I've attached the output.

thank you.

Philipp

[Campbell Morrison]

Thanks. The problem is that the timezone definitions in the standard release distribution have the double CR. I suspect that this is caused by an incorrect EOL property setting in Subversion. I will look into this and fix it for the next release. In the meantime you can patch your system by replacing the tzurl directory on your webserver with the one attached here. Then empty the vtimezone table again in your database in order to force the new definitions to be used.

Campbell

[Anonymous]

Thank you, Campbell!
Now I guess it's up to IBM to fix the still not arriving ics emails generated by MRBS.

[Campbell Morrison]

This may help https://sourceforge.net/p/mrbs/support-requests/317/

I never did look into the problem with Lotus Notes in detail, but from what I remember it seemed that Lotus isn't obeying the RFC. It's probably going to be quicker to put a circumvention into MRBS. I might take another look at it somehow, the only problem being that I don't have a Lotus system to test with.

[Anonymous]

Hello all,

I have not studied in detail Philipp’s post.

Either indirectly or directly related, I would like to point out that when I import the .ics file included in the MRBS 1.4.9 email notifications into Google Calendar, the file is successfully imported instantly and all content including time and timezone are displayed correctly.

Regards,
Robert

[Anonymous]

I hope I did the right changes. Pls see below. If yes, then there's no change. MRBS mails containing ics files don't arrive in my Notes Mailbox.
However, this is only true for emails with ics files.
MRBS mails without ics file arrive just well as well as emails that were sent to another mail system with ics files attached and then forwarded to the Domino server.

--- functions_mail.inc-orig 2014-04-23 15:28:33.006939691 +0200
+++ functions_mail.inc 2014-04-23 15:30:01.548129355 +0200
@@ -976,7 +976,7 @@ function notifyAdminOnBooking($data, $ma
$attachment = array();
if ($mail_settings['icalendar'] && !$enable_periods)
{

• $attachment['method'] = "REQUEST";
• $attachment['method'] = "PUBLISH";
  $ical_components = array();
  $ical_components[] = create_ical_event($attachment['method'], $data, $text_body, $html_body, $addresses, $series);
  $attachment['content'] = create_icalendar($attachment['method'], $ical_components);

--- functions_ical.inc-orig 2014-04-23 15:28:22.638800363 +0200
+++ functions_ical.inc 2014-04-23 15:29:04.831367291 +0200
@@ -1217,7 +1217,7 @@ function export_icalendar(&$res, $keep_p
}
}
// Build the iCalendar from the array of events and output it

• $icalendar = create_icalendar("REQUEST", $ical_events);
• $icalendar = create_icalendar("PUBLISH", $ical_events);
  echo $icalendar;
  }

[Campbell Morrison]

Yes, those were the right changes. Strange that forwarding via another mail system works OK.

[Anonymous]

Hi Robert

I agree. I made the same experience as you sending to google mail or to my own mail service which is dovecot/postfix based. It's a different experience however with Domino. I believe it's a Domino problem. Anyhow, CRCRLF in ical files is not RFC compliant. Fixing this is necessary, I believe.

Philipp

[Campbell Morrison]

The files were correct in MRBS 1.4.9, which may be why Robert did not see the problem. The extra CR was introduced in 1.4.10 somehow.

Campbell

[Campbell Morrison]

Well I've installed the IBM Notes client Social Edition Release 9.0.1 and everything works OK there when I point it at a non-Domino server. So I suspect the problem is indeed a Domino problem of some kind. Over to IBM.

[Campbell Morrison]

It would be helpful to know if you get the same problem with Domino if you generate an appointment using another calendar program, eg Google Calendar or Outlook. If you do, then it's almost certainly a Domino problem. If not, then it may be an MRBS problem.

Campbell

[Anonymous]

Hi

It's the last body part of multipart/alternative.
When I comment this code in functions_mail.inc emails arrive in my Domino mailbox. Since this code works with other mail systems do believe it's a Domino bug. Maybe you have an idea what could be wrong?

1262 // Otherwise we need to carry on and add the text version of the iCalendar
1263 $mime_params['content_type'] = "text/calendar; method=" . $attachment['method'];
1264 // The encoding needs to be base64, because Postfix will otherwise not preserve
1265 // CRLF sequences and these are mandatory terminators in the iCalendar file
1266 $mime_params['encoding'] = "base64";
1267 $mime_inner->addSubPart($attachment['content'], $mime_params);

Thanks,
Philipp

[Campbell Morrison]

Could you try creating an event in Google Calendar and inviting somebody with a Domino mailbox and see if that works? If it does then I can try and narrow down the differences between a Google and an MRBS .ics file. If it doesn't, then it looks like a Domino problem, though some circumvention might be possible along the lines you suggest above. (If you don't have a Google account let me have your email address off-list and I'll generate an event).

Campbell

[Anonymous]

I did as you asked and yes, it works with Google Calendar and Domino.

Philipp

[Campbell Morrison]

OK, thanks. I'll try and work out what the difference is then between a Google file and an MRBS file.

[Anonymous]

Let me know if you need a domino account for testing.

[Campbell Morrison]

Yes please, that would save a lot of time.

[Anonymous]

Ok, please send me your email address to [snipped]. I will send you the credentials to the university's system.

[Campbell Morrison]

Well I now understand what's happening. The problem is that if the ORGANIZER of the VEVENT has the same email address as the Domino recipient then Domino doesn't bother showing the email or the calendar entry - presumably because it assumes you know about the event already. If you make the booking under another name and set say the Domino email address as the MRBS admin then the email arrives correctly.

I'll have a think about what can be done to get round this problem.

Campbell

[Campbell Morrison]

I think the easiest thing to do is to make the organizer MRBS, or the 'from' address. If you're running MRBS 1.4.10 then you can patch your system by commenting out line 920 in functions_ical.inc:

  // $organizer_email = get_email_address($data['create_by']);

Campbell