Christian Kulas - 2013-03-27

Parses PDF files of scientific articles based on naive bayes and sophisticated heuristics. The output is a XML file that contains the parsed data. Meta data is detected and marked as such.

The meta data contains the following elements:

  • Title
  • Authors
  • Abstract
  • Text
  • Headlines
  • Enumerations
  • References (Literature)

In the first step, the text elements are divided into blocks (similar to paragraphs) and after that, predictions for each element are made.

The project contains three runnable classes that can work on given PDFs in batch mode via threading:

a) BatchHeuristic: A parser that uses defined heuristics and rules. Especially applicable for articles with a broad set of layouts (e.g. PeDocs, http://www.pedocs.de/).
b) BatchHybrid: A parser that uses machine learning (Naive Bayes) to find the correct element.
c) ModelGenerator: Generates a training model, used by BatchHybrid, from given PDF and XML files. The training PDF Files must be in the same folder as the manually prepared XML files. The XML files do not have to be complete, but they must be in the correct order. An example XML file is also provided.

The whole project was written in Java and makes use of the library "PDFBox". The code is available under the Apache License 2.0.

It can be used in the following fashion:

a) java -Xms512M -Xmx2048M -classpath parser.jar de.yones.pdf.runnable.BatchHeuristic <pdf folder> <output folder> (<config file>)
b) java -Xms512M -Xmx2048M -classpath parser.jar de.yones.pdf.runnable.BatchHybrid <pdf folder> <output folder> <model file> (<config file>)
c) java -Xms512M –Xmx2048M –classpath parser.jar de.yones.pdf.runnable.ModelGenerator <training folder> <output model file> (<config file>)

Be aware of the large amount of resources that are required, therefore the heap size was dramatically increased in these examples.

A complete manual is at https://sourceforge.net/p/scientificpdfpa/code/11/tree/help/manual_german.docx .

You can also find a short summary and an evaluation at https://sourceforge.net/p/scientificpdfpa/code/11/tree/help/summary_german.ppt .

 

Last edit: Christian Kulas 2013-03-27