Performance on parsing BibTex

  • Joseph Reagle
    Joseph Reagle

    I recently noted that a script I use was getting kind of laggy. It was taking almost 2s to parse my very large bibtex file using bibstuff (which uses SimpleParse). I thought, why not write something in pyparsing and see how that fares? It took about a 60s! (Though I'm sure I've done some clumsy stuff, including the need to flatten.) So I thought, what if I just wrote a simple regexp bibtex parse? 0.24s!

    I then read that psyco and packrat can speed things up. So I report on the various values below. All of these functions return dict[ident][field] = value. My bibtex is machine generated, so I can make a lot of simplifying solutions in the regexp parse.

    regexGet = 0.24 seconds
    bibstuffGet = 1.75 seconds
    pypGet = 59.83 seconds
    pypGet+psyco = 53.20 seconds
    pyGet+packrat = 102.27 seconds
    pypGet+psyco+packrat = 96.97 seconds

    The test code is here:

    • Paul McGuire
      Paul McGuire

      I'm realizing that I should offer a non-grouping form of nestedExpr, that just returns the entire nested expression as a single string.

      Try my modified version of your test program:

      -- Paul

    • Joseph Reagle
      Joseph Reagle

      For some reason, that version chokes on the first bibentry:

      test = '''@online{1722005u1,
         author = {172, User},
         shorttitle = {{User:172} (Version 10702240)},
         title = {{User:172}},
         day = {2},
         year = {2005},
         urldate = {2005-03-24},
         url = {},
         month = {3},
         custom1 = {20050324},
         organization = {Wikimedia},
         author = {zenspider},
         title = {{Burnout} and the Late Night Rant},
         day = {28},
         year = {2007},
         urldate = {2007-03-01 14:26Z},
         url = {},
         journal = {Polishing Ruby},
         month = {2},
         custom1 = {20070301 14:26 UTC},
         keyword = {frustration},
      Traceback (most recent call last):
        File "./", line 177, in <module>
          print "pypGet = %.2f seconds\n" % float(tp.timeit(number=rep)/rep)
        File "/usr/lib/python2.5/", line 161, in timeit
          timing = self.inner(it, self.timer)
        File "<timeit-src>", line 6, in inner
        File "./", line 125, in pypGet
          bibtex = pypParse(text)
        File "./", line 118, in pypParse
        File "/home/reagle/bin/lib/python2.5/site-packages/pyparsing-1.5.2-py2.5.egg/", line 1076, in parseString
          raise exc
      pyparsing.ParseException: Expected "}" (at char 48), (line:3, col:4)

      btw: I see you moved a packrat test before psyco, but that didn't make a difference in timing, except to hurt the subsequent psyco test.

      • Paul McGuire
        Paul McGuire

        Well, that would be because of my bug in nestedExprWithoutGrouping!  Now things run much better (

        Some other notes:
        - ignoreExpr should not be set to None.  I assume that if you had an article with the title "How to use a }", that should not be interpreted as a closing brace.  Now that I am using originalTextFor, the ignore command wont suppress the returned text
        - Your setKey/setValue parse actions are a valiant stab at converting a list of pairs into a dict-like set of entries.  If only you had some pyparsing documentation, the Dict class might have suggested an alternative.  The modified version uses results names and a Dict to dynamically give named results in the returned ParseResults structure.  By calling dump() we get:

        [u'article', '{', u'Ball2007mws',...
        - data: [[u'author', u'Ball, Philip'], [u'shorttitle',...
          - author: Ball, Philip
          - custom1: 20070301
          - journal:
          - month: 2
          - shorttitle: {The} More, the Wikier
          - url:
          - year: 2007
        - ident: Ball2007mws

        And you can reference these fields like dict entries (results["ident"], or results["data"]["author"]) or like object attributes (results.ident or

        -- Paul

    • Joseph Reagle
      Joseph Reagle

      Thanks so much for your response Paul.

      1. I see that you moved the nestedExpr from the outer structure, to just the values, makes sense.
      2. I was using ingoreExpr because even though the open/close chars I was used was "{}" I believe the function still uses default quoteStrings, and hence causing me problems with a field like:
        title = {'The {Spinster} and the {Prophet'} by {A.B.} {McKillop}},
      (I shouldn't be creating bibtex like that, but for the meaningtime...) So I turned it back on.
      3. Is there any difference in settings a ResultsName at the assignment versus within an expression? e.g.:
        pairs = Dict(OneOrMore(pair))("data")
        pairs = Dict(OneOrMore(pair)) ... + pairs("data") + '}')
      4. Trust me, I've read HowToUsePyparsing, PyCon2006, module docs, and OnLamp many times -- not that I understand it :) -- and did want to use Dict(), but still can't figure how to implement what I want directly. (BTW: the HowToUsePyparsing on wikispaces is out of date.) Part of this goes back to my continuing confusion of the structure of the ParseResults given a partiuclar grammar. In any case, I thought constructing from within the using ParseActions would be faster than creating a dictionary later and it made it easy for me to debug -- which is what I do now:

              entries = {}
              for result in results:
                  entries[result.ident] = {}
                  for field, value in
                      entries[result.ident][field] = value
              return entries
      5. Results with current code
      regexGet = 0.22 seconds
      bibstuffGet = 1.63 seconds
      pypGet = 48.83 seconds
      pypGet+psyco = 41.12 seconds
      pypGet+packrat = 81.06 seconds