Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.

Close

JSON Template

2010-03-02
2013-05-30
  • Micah Stetson
    Micah Stetson
    2010-03-02

    I've been working on something my last two OpenBiblio days.  I just made the first commit of the new code, and I'd like to talk a bit about it.  There are a number of different little template systems OpenBiblio is using right now, and I think this is contributing to confusion and insecurity.  Here's what we have:

    Translation with substituted variables: echo T('Hello, %name%', array('name'=>$name))
    SQL with substituted values: $sql = $db->mkSQL('SELECT * FROM table WHERE date=%Q', $date)
    Values in HTML (and such): <h2><?php echo H($title) ?></h2>
    Insecure values in quoted strings: echo "<h2>$title</h2>"
    Parameters in reports: WHERE date=%date%

    Several of these notations are there to protect the system from various code injection vulnerabilities, but they're all different, and if you're going to write secure code in OpenBiblio, you have to understand all of them.  Bad thing.  I think the lack of a single, simple syntax has encouraged new code to take the easy way out and substitute variables into double-quoted strings.  That's no good because it takes us back to having SQL, HTML, and JavaScript injection vulnerabilities - basically making OpenBiblio completely insecure.

    I think these problems can be solved with the use of a simple template package.  JSON Template is a nice one that's implemented in several languages and isn't too complex.  I want us to replace all of these systems with JSON Template, so the examples above will look like this:

    echo T('Hello, {name}', array('name'=>$name))
    $sql = SQL('SELECT * FROM table WHERE date={date}', array('date'=>$date))
    echo HTML('<h2>{title}</h2>', array('title'=>$title))
    In a report: WHERE date={date}

    One syntax for variable substitution, and it's used everywhere.  No double-quoted string substitutions should ever appear in any part of OpenBiblio.

    Moving forward, I'd also like us to get away from having our application code produce HTML directly.  If it produces a simple data structure, then that structure can either be marshaled to JSON and handed to JavaScript code on the client, or it can be given to JSON Template to produce HTML using a template file.  This will enable further separation of the presentation from the back-end logic.

    I realize that this is a big change to make at this stage in the game, but I don't want to release an insecure 1.0, and I don't want to force you guys to learn 4 different notations for escaping.  I think this is the right way to go.  I'll be posting articles on the wiki and more code in the repo as time goes on.  I'm sure my initial commit breaks translation (at least), and I know there will be questions and concerns, so fire away.

    Micah

     
  • Micah Stetson
    Micah Stetson
    2010-03-03

    I just converted shared/biblio_view.php to use JSON Template pretty extensively.  It now spends most of its time constructing an array $page with the information to be handed to the template, then it simply passes that info to biblio_view.jsont where all the HTML lives.  There's still a lot of room for improvement, but I think this is a decent proof of concept.

    As a particular note, I am not currently interested in trying to allow each theme to use custom templates.  I know this wouldn't be really hard, but that's not the point here.  The point is to have cleaner code while still making sure everything is properly escaped.

    Please take a look at shared/biblio_view.php and shared/biblio_view.jsont and let me know what your thoughts are.  My day is pretty-well done, so I'll have to pick up on OpenBiblio again in two weeks.  I may be able to find time to answer questions between now and then, but if not, please bear with me.

    Micah

     
  • Fred LaPlante
    Fred LaPlante
    2010-03-04

    Completly over my head I'm afraid. Not a CS person, just a simple electrical engineer!

    I presume this is for use with reports as otherwise biblio_vies isn't being used anywhere I can recall.

    Fred

     
  • Micah Stetson
    Micah Stetson
    2010-03-04

    I think to some extent this gets back to what's working for you versus what's working for me.  For some reason, the new search and view stuff doesn't work well with my database - titles don't show up, weird stuff is missing, I get SQL errors when I try to edit.  I'll look into that, but in the mean time I've been falling back on what I know - the old stuff.  It's also something I can completely change and mess up without it ruining usability for you and Luuk.  That's why I've been using it as a playground.

    This doesn't have to do with reports, per se.  And I haven't tried to apply it to the reports system at all yet, though that's coming at some point.  Let me come at it from a different angle:

    Web apps are all about producing HTML and other text to be sent to a browser.  PHP has two easy ways for you to do this.  There's HTML in PHP code:

    echo "<h2>$title</h2>";
    

    and there's PHP code in HTML:

    <h2><?php echo $title ?></h2>
    

    The same idea works for producing JSON or SQL or any other structured text.  For JSON:

    echo "{\"title\":\"$title\"}";
    

    All of these suffer from the same basic problem: the value isn't properly escaped.  What happens if $title has a <  or & in it?  Or for JSON, what about " or \?  This leads to weird bugs and even security holes.  The PHP designers tried to fix this with something called "magic quotes", but that was an unmitigated disaster and did far more harm than good.  We need a way to process $title so it's safe to embed in HTML (or SQL or JSON or whatever) before we echo it.  One correct way for HTML is this:

    $title = htmlspecialchars($title);
    echo "<h2>$title</h2>";
    

    This gets old really fast, and it's also hard to audit.  What happens if the call to htmlspecialchars() is not right next to the echo?   It leads to all kinds of bugs.

    This problem crops up often, and it doesn't always look like the same problem.  I thought of a way to fix the problem for producing HTML, a different way for producing SQL, one for doing values in translations, and another for reports.  All of them have different syntax, and it's confusing.  I did it; it's my fault; and it's awful.

    The realization I had two weeks ago was that all of the little escaping tricks I had in OpenBiblio are special cases of the same problem that can be solved using a template language.  JSON Template is a nice little template language.  Here's the basic idea of using it to produce HTML:

    First, you need to make an array (well, not necessarily, but let's pretend it only works with arrays for now) representing the information you want to send to the browser, for example:

    $book = array('title'=>'Spot < Dick & Jane', 'author'=>'Ben Bozo');
    

    Then you need a template that shows the HTML you're going to make.  For a long template, you might put it in its own file, but a string is better for something short, say '<h2>{title} by {author}</h2>'.  So the template says where each value should be substituted into the string.  It's just like "<h2>$title by $author</h2>" only it won't be PHP doing the substitution.

    Now we let JSON Template do its thing:

    echo HTML('<h2>{title} by {author}</h2>', $book)
    

    Because the HTML function tells JSON Template to do HTML escaping by default, the output will be this:

    <h2>Spot &lt; Dick &amp; Jane by Ben Bozo</h2>
    

    JSON Template also allows for more complicated things, which I'm experimenting with in the old biblio_view and biblio_search code.  More on that later.

    Micah