#180 Normalize the bodies of REST API submissions

[Ahasuerus]

Normalize the bodies of REST API submissions. Currently, all manual submissions go through a normalization process which replaces a large number of Unicode and Latin-1 characters with their ASCII or Latin-1 equivalents based on a set of rules in common/library.py. On the other hand, submissions created using the Web/REST API do not go through this normalization process. It can result in disallowed Unicode character getting added to the database, which then requires cleanup.

Also, the manual interface uses HTML escaping for the apostrophe character ("’") which is not XML-compliant. The REST API doesn't convert XML-escaped apostrophes to "’", which results in discrepancies in the submission table. This behavior needs to be standardized.

[Ahasuerus]

Implemented in:

common/library.py
rest/submission.py

Installed in SVN 540 on 2020-05-20. Closing the SR.